# Monocular Camera-Based Robotic Pick-and-Place in Fusion Applications
## 3 Proposed Method
### Distributed Proximal Policy Optimization Algorithms  
RL is a Markov decision process (MDP), which could be represented by \((S,\mathcal{A},\mathcal{P},\mathcal{R},\gamma)\). Here, \(S\) represents the state space, \(\mathcal{A}\) denotes the action space, \(\mathcal{P}\) refers to the state transition possibility, \(\mathcal{R}\) stands for the reward space, and \(\gamma\) represents the discount factor. The objective of all RL algorithms is to maximize the cumulative rewards, formulated as \(max\sum_{t=1}^{T}\gamma^{t}r_{t}\), where the \(t\) is the time step and \(r_{t}\in\mathcal{R}\) is the reward at step \(t\). Although the goals are the same, the accumulated rewards have different equivalent forms in different algorithms. One of the two basic forms is the state value function \(V(s)\), which is defined in Formula (1):  
\[ \begin{split} V(s)&=E\Bigg{[}\sum_{k=0}^{\infty} \gamma^{t}r_{t+k+1}|S_{t}=s\Bigg{]}\\ &=E\Big{[}r_{t+1}+\gamma r_{t+2}+\gamma^{2}r_{t+3}+\ldots|S_{t}=s \Big{]}\\ &=E[r_{t+1}+\gamma V(S_{t+1})|S_{t}=s]\end{split}  \tag{1}\]  
where \(S_{t}\in S\) is the state in time step \(t\), \(r\in\mathcal{R}\) is the reward, and all subscripts indicate the time step. Another one is the state-action value function \(Q(s,a)\), as shown in Formula (2):  
\[ \begin{split} Q_{\pi}(s,a)&=E_{\pi}\Bigg{[}\sum_{k= 0}^{\infty}\gamma^{t}r_{t+k+1}|S_{t}=s,A_{t}=a\Bigg{]}\\ &=E_{\pi}\Big{[}r_{t+1}+\gamma r_{t+2}+\gamma^{2}r_{t+3}+\ldots| S_{t}=s,A_{t}=a\Big{]}\\ &=E_{\pi}[r_{t+1}+\gamma Q_{\pi}(S_{t+1},A_{t+1})|S_{t}=s,A_{t}=a ]\end{split}  \tag{2}\]  
In comparison to Formula (1), Formula (2) incorporates an additional variable \(\pi\in\mathcal{P}\), which represents the policy. The simplest policy is to employ the greedy algorithm, which sets \(\pi(s)=arg\max_{a}Q(s,a)\). However, to strike a balance between exploration and exploitation, a more intricate policy must be utilized. Furthermore, the following equations demonstrate the relationship between \(Q(s,a)\) and \(V(s)\):  
\[ \begin{split}& V(s)=\sum_{a\in\mathcal{A}}\pi(a|s)Q_{\pi}(s,a)\\ & Q_{\pi}(s,a)=r_{s}^{a}+\gamma\sum_{s\in S}\mathcal{P}_{ss^{ \prime}}^{a}V(s^{\prime})\end{split}  \tag{3}\]  
Figure 1: A sub-environment in our Isaac Gym simulation environment. where the \(s^{\prime}\) is the new state, \(\pi(a|s)\) is the policy and \(\mathcal{P}^{a}_{ss^{\prime}}\) is the probability that the agent takes action \(a\) in the current state \(s\) and moves to the new state \(s^{\prime}\). Thus, RL algorithms can be divided into value-based and policy-based groups. The value-based algorithms focus on \(Q(s,a)\) and guide the agent to the final goal by updating the \(Q(s,a)\) value with a fixed policy, such as the previously mentioned greedy strategies. The policy-based algorithms modify the actions' probability, e.g., \(\mathcal{P}^{a}_{ss^{\prime}}\) in Formula (3) by parameterizing and updating the policy function \(\pi(a|s)\) to ensure that the agent obtains the maximum accumulated rewards. The two aforementioned methodologies possess their respective merits and demerits. Generally speaking, value-based algorithms tend to converge at a slower rate, but they have the ability to reach the global optimal solution. Additionally, they tend to exhibit subpar performance when dealing with continuous space problems. Conversely, policy-based algorithms tend to converge at a faster rate; however, during later stages, these algorithms tend to oscillate around the optimal value function with minimal fluctuations and are unable to attain the optimal value result. Therefore, an important DRL algorithm called advantage actor-critic (A2C) [20] which depends on the actor-critic [21] is presented. A2C combines the value function and policy together, the actor decides which action to take, and the critic tells the actor how good its action was and how it should adjust. The brief process is shown in Algorithm 1.