# Reinforcement Learning-Based Algorithm to Avoid Obstacles by the Anthropomorphic Robotic Arm
## 2 Materials and Methods
### Reinforcement Learning and Its Applications
A very interesting application of standard RL algorithms for control of agents that compete in a simple game of hide-and-seek is presented in [19]. In this game, a mixed cooperative and competitive physics-based environment is introduced. In a learning process, only a visibility-based reward function and competition are used. The results showed that agents learned many emergent skills and strategies. It is worth pointing out that they learned the use of collaborative tools and intentionally changed their environment to suit their needs. In the process of learning, the agents created a self-controlling curriculum that contained many distinct rounds of different strategies, many of which required sophisticated tool use and coordination. In [20], a framework for RL for mobile robots is described. Example trajectories are used in the learning process, which consist of two phases. In the first example, the robot was controlled by an exemplary trajectory for a given task or was controlled directly by an operator. The RL observes the actions, states, and rewards obtained by the robot, and uses this information for the approximation of a value-function. If the learned policy is assessed as acceptable to control the robot, the second phase of learning is used, in which the RL method is used to control the robot. In this step, the Q-learning framework is applied to progress the learning. During the investigations, the robot translation speed is constant. The goal is to lead the robot to its goal point while avoiding obstacles on its path. A reward equal to 1 is used if the robot has reached the goal, and a reward equal to \(-1\) is given if the robot has hit an obstacle. A reward equal to 0 is given in all other cases. The final efficiency obtained did not differ significantly (at the level of 95%) compared to the results obtained with direct robot control. [PERSON] et al. [21] investigated robot navigation ability using the RL algorithm. Task performance is improved by solving a problem of maximizing cumulative reward and jointly considering unsupervised and self-supervised tasks. The authors used two auxiliary tasks: the first for learning obstacle avoidance by unsupervised reconstruction of a simple depth map, and the second for short-term trajectory planning by solving a self-supervised loop closure classification task. The results were validated in complex 3D mazes using raw sensory input. The robot behaved similarly to a human, including when the start and goal locations were changed. The article [22] describes the use of Twin Delayed Deep Deterministic policy gradient (TD3) to avoid obstacles by a 3-axis robotic arm. The authors compared its operation with the Probabilistic Roadmap (PRM) algorithm. A 90% success rate was achieved for the gain learning algorithm and a 3% shorter motion trajectory compared to the PRM algorithm. However, the algorithm was only implemented in the simulation. The same authors in [23] tested the proposed system in a laboratory on a real robot. Additionally, they implemented Soft Actor Critic (SAC) and Hindsight Experience Replay (HER). Due to various combinations of algorithms, the trajectory was 10% shorter compared to the PRM algorithm. [PERSON] et al. [24] proposed the use of a DDPG algorithm to plan the trajectory of motion of the Nao robotic arm in simulation. They used the reward function, depending on the target position and the actual position of the manipulator. The agent trained in this way was able to position the TCP (tool center point) of the robotic arm in a given point, but it was not able to avoid obstacles without collision. In order to improve the learning procedure, the authors formulated a reward function that also depends on the distance of the manipulator to the obstacle. The results of simulation tests confirmed that the modified algorithm allowed the robot to successfully avoid obstacles. A similar approach was used by the authors of [25], which also used an RL algorithm called Q-learning to plan the robot's path and avoid obstacles.