# Reinforcement Learning-Based Algorithm to Avoid Obstacles by the Anthropomorphic Robotic Arm
## 2 Materials and Methods
### Reinforcement Learning and Its Applications
The research was performed only in the simulation domain. Another example of path planning using the RL algorithm was proposed by [PERSON] et al. [26]. Their algorithm combines the RL algorithm with the path planning algorithm of the mobile robot to compensate for the disadvantages of conventional methods by learning the situations where each robot has mutual influence. In most cases, known path planning algorithms are highly dependent on the environment. Other examples of applications of RL methods in mobile robots are presented in [27; 28]. In [29], the authors proposed a learning algorithm based on cooperative multi-agent RL to control the [PERSON] robot. The algorithm was developed using a simulated planar 6-DOF with a discrete action-set. The results showed that all the points reached by the manipulator have an average accuracy of 5.6 mm. The algorithm was found to be repeatable. The authors implemented the concept and tested it on the Baxter robotic arm to obtain accuracy up to 8 mm. Deep Deterministic Policy Gradient (DDPG) methods have been successfully used in several applications in different robotic control tasks. In [30], the obstacle avoidance and navigation problem in the robotic Unmanned Ground Vehicles (UGVs) control area was investigated. Revised DDPG and Proximal Policy Optimization (PPO) algorithms with improved reward shaping methods are proposed. The performances between these methods were validated in simulation. [PERSON] et al. in [31] presented a set of continuous control tasks, including the movement of a 3D humanoid robot with partial observations and tasks with a hierarchical structure. They attempted to address the evaluation by presenting a benchmark consisting of 31 continuous control tasks. The authors compared DDPG with batch algorithms, concluding that on certain tasks, DDPG was able to converge significantly faster, but it was less stable than batch algorithms. It was also more susceptible to scaling of the reward. [PERSON] et al. [32] have demonstrated that the Deep Deterministic Policy Gradient algorithm (DDPG), which is an off-policy deep Q-function-based algorithm, can achieve training times suitable for real robotic systems. Authors have shown in experimental evaluation that this method can teach different 3D manipulation skills, such as door opening, to a real robot any special demonstrations. In [33], the results of several RL tasks with three commercially available robots are presented. The tasks present different levels of difficulty and repeatability. An experimental study of multiple policy learning algorithms, e.g., TRPO, PPO, DDPG, and Soft-Q, was made. The results showed that these methods can be easily applied to real robots. RL algorithms were also used in other areas of robotics such as pick and place [34], push tasks [35], tossing arbitrary objects [36], and dexterous manipulation of the objects [37; 38; 39; 40; 41]. The use of RL methods in the positioning of the robots and avoiding obstacles opens the way to introducing self-learning methods in robot programming and control. The use of the RL algorithm allows a robot to achieve a certain degree of autonomy by learning to move by itself in order to perform a task, e.g., positioning by trying and gathering experience. Thanks to this, a robot can adapt to changes in the environment such as the appearance of an obstacle, which is difficult for classic control methods. The main contribution of this paper is an RL-based algorithm designed to avoid obstacles and control the positioning of the robotic arm. The authors applied a Deep Deterministic Policy Gradient combined with Hindsight Experience Replay in order to build a multi-agent system to avoid an obstacle and reach the goal position as fast as possible. In order to adapt the algorithms to the task of avoiding obstacles, changes in the operation of the RL algorithms have been made. In case of laboratory investigations to detect the obstacle, a dedicated head with optical (laser) sensors attached to the wrist of the robot was designed and built.