# Adaptive Compensation for Robotic Joint Failures Using Partially Observable Reinforcement Learning
## 3 Methodology
### Reinforcement Learning Framework
#### 3.2.2 Agent Training  
The agent's training process involves several key steps, with each being critical for optimizing the policy and value networks. The workflow for training the agent using the PPO algorithm is as follows:  
**Initialization**: Initialize the policy network \(\pi_{\theta}\) with parameters \(\vartheta\). Initialize the value network \(V_{\phi}\) with parameters \(\phi\). Set the initial parameters for the learning rate, discount factor \(\gamma\), and clipping parameter \(\epsilon\).  
**Collecting Trajectories**: Interact with the environment to collect trajectories of states \(s_{t}\), actions \(a_{t}\), and rewards \(r_{t}\). The interaction involves executing actions sampled from the policy network and observing the resulting states and rewards.  
Figure 1: Our framework addresses the issue of joint malfunction in robot manipulators using the PPO algorithm. The actor takes in the observation from the environment and outputs action for joints. The critic estimates the value of a state, helping the actor learn by providing feedback on the quality of its actions.  
**Computing Returns and Advantages**: Calculate the discounted returns \(R_{t}\) from each time step \(t\):  
\[R_{t}=\sum_{k=0}^{T-t}\gamma^{k}r_{t+k}, \tag{12}\]  
where \(R_{t}\) is the return at time step \(t\), \(\gamma\) is the discount factor, and \(r_{t+k}\) is the reward at time step \(t+k\).  
Compute the advantage estimates \(A_{t}\) using the Generalized Advantage Estimation (GAE):  
\[A_{t}=\delta_{t}+(\gamma\lambda)\delta_{t+1}+\ldots+(\gamma\lambda)^{T-t+1} \delta_{T}, \tag{13}\]  
where  
\[\delta_{t}=r_{t}+\gamma V_{\phi}(s_{t+1})-V_{\phi}(s_{t}). \tag{14}\]  
Here, \(A_{t}\) is the advantage estimate at time step \(t\), \(\lambda\) is the GAE smoothing parameter, \(\delta_{t}\) is the temporal difference error, \(V_{\phi}(s_{t})\) is the value estimate at state \(s_{t}\), and \(r_{t}\) is the reward at time step \(t\).  
To calculate the objective loss function of the policy, first of all, we need to calculate the ratio of action probabilities \(r_{t}(\theta)\) between the new and old policies:  
\[r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{t }|s_{t})} \tag{15}\]  
where \(\pi_{\theta}(a_{t}|s_{t})\) is the probability of action \(a_{t}\) given state \(s_{t}\) under the new policy, and \(\pi_{\theta_{\text{old}}}(a_{t}|s_{t})\) is the probability under the old policy. The clipped surrogate objective \(L^{CLIP}(\phi)\) is as follows:  
\[L^{CLIP}(\theta)=\mathbb{E}_{t}[\min(r_{t}(\theta)A_{t},\text{clip}(r_{t}( \theta),1-\epsilon,1+\epsilon)A_{t})] \tag{16}\]  
where \(\mathbb{E}_{t}\) denotes the expectation over time steps \(t\), and \(\text{clip}(r_{t}(\theta),1-\epsilon,1+\epsilon)\) restricts \(r_{t}(\theta)\) to the range \([1-\epsilon,1+\epsilon]\).  
The total loss for PPO includes both the policy loss (clipped objective) and the squared-error value function loss, as well as an entropy bonus to encourage exploration. The squared-error value function loss is defined as  
\[L^{value}(\phi)=\mathbb{E}_{t}\Big{[}(V_{\phi}(s_{t})-R_{t})^{2}\Big{]}, \tag{17}\]  
\[\mathcal{L}(\theta)=\mathcal{L}^{\text{CLIP}}(\phi)-c_{1}L^{value}(\phi)+c_{2 }\mathbb{E}_{t}[\text{Entropy}(\pi_{\theta}(s_{t}))]. \tag{18}\]  
Here, \(c_{1}\) and \(c_{2}\) are coefficients that balance the value loss and entropy bonus.  
After calculating the loss function, we update the policy network parameters by computing the gradients of the total loss \(\mathcal{L}(\theta)\). Similar to the policy network, we also update the parameters in the value network based on the value loss \(L^{value}(\phi)\).  
\[\phi\leftarrow\theta-\alpha_{\phi}\
abla_{\phi}\mathcal{L}(\theta) \tag{19}\]  
\[\phi\leftarrow\phi-\alpha_{\phi}\
abla_{\phi}\mathcal{L}^{value}(\phi). \tag{20}\]  
Repeat the update process for a fixed number of epochs or until convergence.