# Reinforcement Learning-Based Algorithm to Avoid Obstacles by the Anthropomorphic Robotic Arm
## 2 Materials and Methods
### Methodology of the Research
#### 2.3.2 Description of a Research Experiment
```
1: Initialize neural networks with parameters \(\theta^{\mu},\theta^{Q}\) and \(\theta^{Q^{\prime}}\leftarrow\theta^{Q}\), \(\theta^{\mu^{\prime}}\leftarrow\theta^{\mu}\)
2: Initialize buffer \(D\)
3:for episode = 1, M do
4: if random_uniform(0, 1) > \(p_{obsa}\)
5: hide obstacle
6: generate goal position \(g\) gripper start position \(s_{0}\) from uniform distribution
7: else
8: generate obstacle position
9: generate goal position \(g\) from uniform distribution
10:#not inside obstacle and not in a forbidden zone near to the obstacle
11: if random_uniform(0, 1) > \(p_{gptp}\)
12: generate gripper start position \(s_{0}\) on the same side as the goal position \(g\)
13: #lycoordinates have the same sign
14:else
15: generate gripper start position \(s_{0}\) on the opposite side of the goal pos. \(g\)
16:#lycoordinates have different signs
17:for\(t\) = 0, \(T-1\)do
18: select action \(a_{t}\) according to the current policy:
19:\(a_{t}=\mu\Big{(}s_{t}||g|\theta_{t}^{\mu}\Big{)}+N_{t}\)\(\qquad\) symbol \(||\) denotes concatenation
20: execute action \(a_{t}\) and observe new state \(s_{t+1}\)
21:for\(t\) = 0, \(T-1\)do
22:\(r_{t}:=R(s_{t},\ a_{t},\ g)\)
23: store transition \((s_{t}||g,a_{t},r_{t},s_{t+1}||g,d_{obs}||c)\) in \(D\)
24: sample a set of additional goals for replay \(G:=S\)
25:for\(g^{\prime}\in G\)do
26:\(r^{\prime}:=R(s_{t},\ a_{t},\ g^{\prime})\)
27: store transition \((s_{t}||g^{\prime},a_{t},r^{\prime},s_{t+1}||g^{\prime},d_{obs}||c)\) in \(D\)
28:if update neural networks parameters then
29: sample a random minibatch \(B=\{(s_{i}||g,a_{t},r_{i},s_{i+1}||g,d_{obs}||c)\}\) from \(D\)
30: Calculate target \(y\)
31:\(y_{i}(r_{i},s_{i+1})=r_{i}+\gamma Q^{\prime}\Big{(}s_{i+1}||g,\mu^{\prime} \Big{(}s_{i+1}||g|\theta^{\mu}\Big{)}\Big{)}|\theta^{Q^{\prime}}\Big{)}\)
32: update neural network parameters \(\theta^{Q}\):
33:\(\
abla_{\theta^{Q}}L=\
abla_{\theta^{Q}}\frac{1}{|B|}\sum\big{[}y_{i}(r_{i},s _{i+1})-Q\big{(}s_{i}||g,a_{i}|\theta^{Q}\big{)}\big{]}^{2}\ \theta^{Q}\leftarrow\theta^{Q}-\alpha_{Q}\
abla_{\theta^{Q}}L\)
34: update neural network parameters \(\theta^{\mu}\)
35:\(\
abla_{\theta^{Q}}J=\
abla_{\theta^{Q}}\frac{1}{|B|}\sum\big{[}Q(s_{i}||g, \mu(s_{i}||g|\theta^{\mu})|\theta^{Q})\big{]}\qquad\qquad\theta^{\mu}\leftarrow \theta^{\mu}+\alpha_{\mu}\
abla_{\theta^{Q}}J\)
36: update the target networks:
37:\(\theta^{Q^{\prime}}\leftarrow\rho\theta^{Q^{\prime}}+(1-\rho)\theta^{Q}\)
38:\(\theta^{\mu^{\prime}}\leftarrow\rho\theta^{\mu^{\prime}}+(1-\rho)\theta^{\mu}\)
```  
**Algorithm 1** RL-based algorithm for avoiding obstacles by robotic arm  
There are two main parameters in the learning process. The first, \(p_{obsa}\), is the probability of the presence of the obstacle in the workspace, which is set to 0.9. The second, \(p_{gptp}\), is the probability that the start position of the robot's TCP is on the same side of the obstacle as the goal position (signs of the \(y\) coordinates of the start position of the robot's TCP and goal position are the same), which is set to 0.1. The first parameter \(p_{obsa}\) (line 4--Algorithm 1) is to ensure such a generalization of the environment that, in case of no obstacle nearby the robot, it will move toward the goal along the optimal trajectory (trajectory close to a straight line, without curvation caused by an obstacle). This is done to avoid overfitting. The research has shown that if the first parameter is too small, the robot cannot avoid the obstacle correctly in some cases or even in every case. We believe that in this situation, the agent gained too little experience in avoiding obstacles (learning time did not help to improve it). When the agent is trained in a scenario where the first parameter is too large or even if it is equal to \(1\) (in all episodes an obstacle appeared), after the learning process the robot tends to move along a non-optimal trajectory, i.e., the robot does not move towards the goal as quickly as possible.