# Reinforcement Learning-Based Algorithm to Avoid Obstacles by the Anthropomorphic Robotic Arm
## 2 Materials and Methods
### Algorithms
#### 2.2.1 Deep Deterministic Policy Gradient  
DDPG is classified as a model-free, off-policy RL algorithm for continuous action spaces that simultaneously learns a Q function and a policy. This algorithm uses the agent, which interacts in discrete time steps \(t\) with an environment. In every step, the agent receives an observation \(s_{t}\), performs an action \(a_{t}\), and receives a reward \(r_{t}\). The algorithm utilizes the Bellman equation [12] to learn the Q function and utilizes this Q function to learn its policy. DDPG uses four following neural networks: a Q network \(Q(s,a|\theta^{Q})\), a deterministic policy network \(\mu(s|\theta^{\mu})\), a target Q network \(Q^{\prime}(s,a|\theta^{Q^{\prime}})\), and a target policy network \(\mu^{\prime}(s|\theta^{\mu^{\prime}})\). Artificial neural networks are characterized by parameters \(\theta\) with superscript that indicates which network it is.  
DDPG utilizes also replay buffer \(R\) in which the experience is stored and samples from it to update the neural networks parameters. During each trajectory rollout, all tuples \((s_{t},a_{t},r_{t},\ s_{t+1})\) of the experience are saved and stored in a finite-sized cache. The parameters of the tuples are: \(s_{t}\) -- state in time \(t\), \(a_{t}\)--an action in time \(t\), \(r_{t}\)--a reward in time \(t\), \(s_{t+1}\)--the next state in time \(t+1\). In order to update the neural network parameters, a random minibatch of \(N\) tuples \((s_{i},a_{i},r_{i},\ s_{i+1})\) is taken (sampled) from the replay buffer \(R\). Subscripts \(i\) mean that tuples are sampled from the replay buffer as past experience.  
The target \(y_{i}\) for the Q network is determined using the following equation:  
\[y_{i}(r_{i},s_{i+1})=r_{i}+\gamma Q^{\prime}\Big{(}s_{i+1},\mu^{\prime}\Big{(} s_{i+1}\Big{|}\theta^{\mu^{\prime}}\Big{)}\Big{|}\theta^{Q^{\prime}}\Big{)} \tag{1}\]  
where \(\gamma\) is a discount factor in the range 0 and 1; it is usually close to 1. The Q network is trained by minimizing the mean-squared loss between the main Q function and the target \(y_{i}\):  
\[L=\frac{1}{N}\sum\,\big{(}y_{i}(r_{i},s_{i+1})-Q(s_{i},a_{i}\ \Big{|}\theta^{Q}) \big{)}^{2} \tag{2}\]  
The policy must derive the action that maximizes \(Q(s,a|\theta^{Q})\); thus, the policy is updated with a policy gradient using the following formula:  
\[\
abla_{\theta^{\mu}}J(\theta)\approx\frac{1}{N}\sum\Bigl{[}\
abla_{a}Q\Big{(} s_{t},\mu(s_{i}|\theta^{\mu})\Big{|}\theta^{Q}\Big{)}\
abla_{\theta^{\mu}}\mu(s_{i}| \theta^{\mu})\Bigr{]} \tag{3}\]  
The target network Q is copied from the main network every fixed number of steps. It is updated once per main network update using Polyak averaging:  
\[\begin{array}{l}\theta^{Q^{\prime}}\leftarrow\rho\theta^{Q}+(1-\rho)\theta ^{Q^{\prime}}\\ \theta^{\mu^{\prime}}\leftarrow\rho\theta^{\mu}+(1-\rho)\ \theta^{\mu^{\prime}}\end{array} \tag{4}\]  
where \(\rho\in\langle 0,1\rangle\).  
Noise is added to every action to provide exploration in DDPG with deterministic policy. The time-correlated Ornstein-Uhlenbeck noise [45] is used in the original paper, but later results show that uncorrelated, mean-zero Gaussian noise works superbly. The final action is determined using the following formula in which the noise \(N_{t}\) in time \(t\) is included:  
\[a_{t}=\mu\Big{(}s_{t}\Big{|}\theta_{t}^{\mu}\Big{)}+N_{t}. \tag{5}\]