# Reinforcement Learning-Based Algorithm to Avoid Obstacles by the Anthropomorphic Robotic Arm
## 2 Materials and Methods
### Methodology of the Research
#### 2.3.2 Description of a Research Experiment  
The goal of the algorithm was to learn to achieve the target position of the TCP of the robot as accurately and quickly as possible. In case of an obstacle, the robot had to avoid it and still achieve the desired position. The robot was working without any payload. At first, the agent was trained in a simulation environment to learn the optimal strategy to ensure the completion of the task. After training the optimal policy in simulation, the strategy was tested on a simulated model of the robot and on the physical robot. The algorithm was trained for \(5\times 10^{4}\) steps. The number of steps was selected so that the value of the reward in the function of the number of steps during the training reaches a plateau--an almost constant, unchanged value. In order to find optimal hyperparameters, we used the OPTUNA hyperparameter optimization framework [47]. Selected hyperparameters of the algorithm are presented in Table 1 in Appendix 0.A. The observations of the agent creating the state \(s\) were: current TCP position (\(x,\ y,\ z\)) in meters; linear velocity of the TCP (\(v_{x},\ v_{y},\ v_{z}\)) in \(\frac{m}{s}\); target position (\(g_{x},\ g_{y},\ g_{z}\)) in meters; distance of the TCP of the robot from the obstacle \(d_{obs}\) in meters; and contact binary information \(c\) whether any part of the robotic arm has touched an obstacle or not (true/false). The last information was only provided in simulation investigation and helped to teach the agent the kinematics of the robotic arm. Altogether, there are 11 observations. In every step of control, the algorithm generated an action \(a\), which is a 3-dimensional TCP displacement described by increments in meters (\(\Delta x,\ \Delta y,\ \Delta z\)). In order to make sure that in the event of an obstacle appearing in the workspace, the robot can bypass it without collision and reach the desired position as quickly and accurately as possible, a special training approach was proposed. The whole process of training was divided into episodes with a fixed maximum number of steps (moves) that the robot can perform. The maximum number of steps for each episode was 50. Each episode during training differed from the others. While the obstacle appeared in the workspace with some chosen probability, the position of the obstacle, the start position of the TCP of the robot, and the goal position were sampled from a uniform distribution. The start and goal positions were sampled from workspace with the following limits, \(x\)-axis: 0.7 m-1.05 m; \(y\)-axis: \(-\)0.45 m-0.45 m; \(z\)-axis: 0.6 m-0.9 m. Coordinates of the obstacle position were sampled with the following limits, \(x\)-axis: 0.85 m-1.05 m; \(y\)-axis: \(-\)0.1 m-0.1 m; \(z\)-axis: _const_. Data were given in relation to the robot's coordinate system-origin (Figure 2b). The workspace in the simulation environment (Figure 3) has the same dimensions as the workspace of the actual robot arm installed in the laboratory. In simulation investigations, the obstacle had a cuboid shape and a size of 0.07 \(\times\) 0.1 \(\times\) 1.2 m; in an investigation  
Figure 2: The Mitsubishi RV-12 SDL-12S used in the research: (**a**) the physical robot with marked joints; (**b**) the 3D model of the robot in a virtual environment with marked axes X, Y, and Z.  
conducted in the laboratory, Styrofoam blocks were used as obstacles of different sizes. The RL-based algorithm for avoiding obstacles by the robotic arm is presented in Algorithm 1.