# Reinforcement Learning-Based Algorithm to Avoid Obstacles by the Anthropomorphic Robotic Arm
## 2 Materials and Methods
### Methodology of the Research
#### 2.3.2 Description of a Research Experiment
The second parameter \(p_{gptp}\) (line 10--Algorithm 1) is introduced in order to ensure that the robot could achieve its main objective, which was achieving a specific position in space without interacting with or moving too close to an obstacle. If the \(p_{gptp}\) is too small, i.e., less than 0.1, the robot is not able to achieve the goal position. The robot does not move towards the goal because when it is too close to the obstacle, the agent receives a larger penalty, which prevents it from exploring the workspace, finding a goal, and thereby obtaining a larger reward after the correct maneuver bypassing the obstacle. In the case when \(p_{gptp}\) is too larger, the robot cannot learn how to avoid obstacles. During testing, different approaches were tested in order to achieve the desired behavior of the robot. An application of a much more complicated reward function than the one described in Section 2.3.4 does not seem to improve the policy of the agent. Tests showed that an agent trained in an environment with obstacles to both avoid the obstacles and reach the goal position as quickly as possible, when in an environment without obstacles, does not achieve the target as quickly as possible compared to an agent trained only to achieve the goal position as quickly as possible. To address this problem, the application of two agents working in parallel is proposed. The scheme of such a system is presented in Figure 4. The agent trained to avoid obstacles is called Agent#1 and the agent trained to achieve the goal position as fast as possible moving along an optimal trajectory (trajectory close to a straight line created based on the start and goal position) is called Agent#2. Observations of Agent#1 are the same as Agent#2. Both agents generated actions as 3-dimensional TCP movements (\(\Delta x\), \(\Delta y\), \(\Delta z\)). During the tests, both agents operate in parallel, and an additional block called the Decision Block (DB) chooses which agent should perform an action in the environment and move the TCP of the robot. DB is simply a conditional \(if\) statement that, based on the distance between the TCP of the robot and the obstacle, decides which agent should take an action. The threshold distance \(d_{obs}\) was set to 0.25 m, i.e., if the distance between the TCP and the obstacle is smaller than this value, Agent#1 takes an action; if the value is larger, Agent#2 does. Figure 3: The workspace of the Mitsubishi robotic marked with a red cuboid. We performed research on the simulated model of the robot and on the physical one. For every algorithm, the results obtained in simulation and during deployment on the physical robot were compared for the same 50 start gripper positions and 50 goal positions to ensure a reliable comparison between different attempts at training. As for the position of the obstacle, in the simulation investigation, 50 different, known positions were tested; in laboratory investigation, there were also 50 positions, but they were not the same as in simulation investigation. This was because it is impossible to set the obstacle in the same position as used in simulation and change its position in each episode. During laboratory investigation, an obstacle was placed inside the workspace by the operator. All 50 points were sampled from the uniform distribution in a 3D space box. The parameters presented and compared in Tables 1 and 2 are the distance (error \(e\)) of the robot's TCP from the target position, standard deviation \(\sigma_{e}\) of the distance (error \(e\)), maximum error \(e\), and error \(e\) for different axis \(x\), \(y\), \(z\). Error \(e\) is calculated from the formula:  
\[e=\sqrt{\Delta x_{T}^{2}+\Delta y_{T}^{2}+\Delta z_{T}^{2}}. \tag{8}\]  
where \(\Delta x_{T},\ \Delta y_{T},\ \Delta z_{T}\) are the difference between the coordinates of the robot's TCP position in the last (terminal) step of the episode and the goal position.