# Monocular Camera-Based Robotic Pick-and-Place in Fusion Applications
## 3 Proposed Method
### Distributed Proximal Policy Optimization Algorithms
```
1: Take action \(a\sim\pi_{\theta}(a|s)\), get \(\{s,a,r,s^{\prime}\}\)
2: Update \(\hat{v}^{\pi}_{\phi}(s)\) using target \(r+\gamma\hat{v}^{\pi}(s^{\prime})\) {The function approximation idea is used here, \(\hat{v}^{\pi}_{\phi}(s)\) is the approximation of \(\hat{v}^{\pi}(s)\)}
3: Evaluate \(\hat{A}^{\pi}(s_{i},a_{i})=r(s_{i},a_{i})+\gamma\hat{v}^{\pi}_{\phi}(s^{\prime })-\hat{v}^{\pi}_{\phi}(s)\)
4:\(\
abla I(\theta)=\
abla_{\theta}\log\pi_{\theta}(a_{i}|s_{i})\hat{A}^{\pi}(s_ {i},a_{i})\)
5:\(\theta\leftarrow\theta+a\
abla_{\theta}I(\theta)\)
```  
**Algorithm 1** A2C. The A2C algorithm excels at managing continuous control problems. However, it inherits the drawback of policy-based algorithms, which makes it challenging to converge to the global optimal solution. The primary reason for this challenge is the difficulty in determining the appropriate step size/learning rate during the policy gradient. To address this issue, the proximal policy optimization (PPO) [22] algorithm is introduced based on the idea of trust region policy optimization (TRPO). The objective of TRPO is illustrated below:  
\[Maximize_{\theta}E_{s}\bigg{[}\frac{\pi_{\theta}(a|s)}{\pi_{\theta old}(a|s)} \hat{A}\bigg{]} \tag{4}\]  
To limit the difference between the new policy and the old one, TRPO introduces the Kullback-Leibler (KL) divergence constraint:  
\[E_{t}[KL(\pi_{\theta old}(\cdot|s),\pi_{\theta}(\cdot|s))]\leq\delta \tag{5}\]  
where \(\delta\) is the size of the constraint region, then the final target function of the actor turns into  
\[Maximize_{\theta}E_{s}\bigg{[}\frac{\pi_{\theta}(a|s)}{\pi_{\theta old}(a|s)} \hat{A}-\beta KL[\pi_{\theta old}(\cdot|s),\pi_{\theta}(\cdot|s)]\bigg{]} \tag{6}\]  
where the \(\beta\) is the fixed penalty coefficient. However, research revealed that the KL penalty update performs less effectively in reality compared to the clipping option, as illustrated below:  
\[L^{CLIP}(\theta)=E_{t}\big{[}\min(r_{t}(\theta)\hat{A},clip(r_{t}(\theta),1- \epsilon,1+\epsilon)\hat{A})\big{]} \tag{7}\]To summarize, the PPO algorithm can be presented as Algorithm 2. On the other hand, the distributed proximal policy optimization (DPPO) algorithms employ multiple threads based on PPO to gather state, action, reward, and other data from numerous agents trained in a distributed manner. These data are then utilized to update parameters synchronously. While the theoretical model is consistent with PPO, the challenge lies primarily in the engineering implementation. ```
1:Run policy \(\pi_{\theta}\) for \(T\) timesteps, collecting \(\{s_{t},a_{t},r_{t}\}\)
2:Estimate advantages \(\hat{A}_{t}=\sum_{t^{\prime}>t}\gamma^{t^{\prime}-t}r_{t^{\prime}}-V_{\phi}(s_ {t})\)
3:Update the policy by PPO-Clip objective: \(\theta_{k+1}=arg\max_{\theta}\sum_{t=0}^{T}min(R_{t}(\theta)\hat{A}_{t},\ clip(R_{t}(\theta),1-e,1+e)\hat{A}_{t})\), where \(R_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{old}(a_{t}|s_{t})}\)
4:Updating the value function: \(\phi_{k+1}=arg\min\sum_{t=0}^{T}(\sum_{t^{\prime}=t}^{T}\gamma^{t^{\prime}-t} r_{t^{\prime}}-V_{\phi}(s_{t}))^{2}\)
```  
**Algorithm 2** Proximal policy optimization (PPO).