# Reinforcement Learning-Based Algorithm to Avoid Obstacles by the Anthropomorphic Robotic Arm
###### Abstract  
In this paper, the application of the policy gradient Reinforcement Learning-based (RL) method for obstacle avoidance is proposed. This method was successfully used to control the movements of a robot using trial-and-error interactions with its environment. In this paper, an approach based on a Deep Deterministic Policy Gradient (DDPG) algorithm combined with a Hindsight Experience Replay (HER) algorithm for avoiding obstacles has been investigated. In order to ensure that the robot avoids obstacles and reaches the desired position as quickly and as accurately as possible, a special approach to the training and architecture of two RL agents working simultaneously was proposed. The implementation of this RL-based approach was first implemented in a simulation environment, which was used to control the 6-axis robot simulation model. Then, the same algorithm was used to control a real 6-DOF (degrees of freedom) robot. The results obtained in the simulation were compared with results obtained in laboratory conditions. obstacle avoidance; positioning; robotic arm; reinforcement learning +
Footnote †: journal: Applied Sciences  
1
Footnote 1: email: {[[https://www.mdpi.com/journal/applsci/code](https://www.mdpi.com/journal/applsci/code)]([https://www.mdpi.com/journal/applsci/code](https://www.mdpi.com/journal/applsci/code))  
sensors, the control system was able to follow the object outline smoothly until the robot reached the target position. [PERSON] and [PERSON] [7] introduced a real-time obstacle avoidance approach in the case of robot motion described by dynamical systems and convex obstacles. The method allowed the avoidance of multiple obstacles. One of the drawbacks of the global strategy approach is that it assumes that the environment is not changing. This is because the algorithms' computational complexity usually does not enable any re-configuration in the required response time of a controller. Therefore, the global strategy approach cannot be implemented in real-time systems [8]. In [9], a trajectory planning algorithm with autonomous obstacle avoidance features for a manipulator was proposed. This was an improved Rapidly-exploring Random Tree (RRT), called Smoothly (S-RRT). To detect obstacles, authors used an RGBD Kinect V2 camera mounted on a tripod, so it was not integrated with a robot. The proposed method allowed the manipulator to avoid both static and suddenly appearing obstacles by re-planning the path of movement. The authors used one type of obstacle during the investigation. The RRT algorithm performed a correct obstacle avoidance maneuver 12 times out of 20 attempts, and the S-RTT algorithm performed it 20 times out of 20 attempts. The navigation tasks--path planning and obstacle avoidance--must cooperate together to solve possible conflicts, e.g., to reach the target position as fast as possible and avoid obstacles during the movement. In a real indoor environment, there are known static obstacles such as tables, chairs, and furniture, and unknown dynamic obstacles such as other robot arms, or people approaching the controlled robot. In some cases, the velocity or shapes of these objects can also change in an unpredictable way. Planning the trajectory of movement, manipulating objects, and avoiding obstacles are among the most important challenges that face each implementation of robots in real environments. [PERSON] et al. provided a broad overview of different control approaches to obstacle avoidance for industrial robots [10]. The described methods use the reconfiguration of the robot without changing the task motion, which requires a robot to have sufficient redundant degrees of freedom. First, the theoretical background of redundant serial manipulators and the strategies for obstacle avoidance are presented. The basic strategy is to consider the problem at the kinematic level, which consists of identifying the point on the robot arm that is closest to the obstacles and changing the speed of this point in such a way that the manipulator moves away from the obstacle.