# Reinforcement Learning-Based Algorithm to Avoid Obstacles by the Anthropomorphic Robotic Arm
## 2 Materials and Methods
### Methodology of the Research
#### 2.3.3 Detection of the Obstacle  
In the case of simulation investigation, the position of the obstacle is known and it is easy to establish the distance \(d_{obs}\) between the TCP of the robot and the obstacle in every step. The obstacle has a cuboid shape; hence, distance \(d_{obs}\) is calculated as a distance between the coordinates of the TCP of the robot and the coordinates of the closest point on the surface of the cuboid (obstacle). Therefore, the distance \(d_{obs}\) is the smallest distance between the TCP and the obstacle.  
In the case of laboratory investigations, determining the distance \(d_{obs}\) is not so simple. In order to address this problem, a dedicated head with optical (laser) sensors attached to the wrist of the robot was designed and built. A total number of ten VL53L1 time-of-flight sensors were used (Figure 5a). These sensors measure the time for emitted light to travel from the sensor to the object in front of it and back again. Due to the fact that the speed of light is a constant value independent of changing environmental conditions, such sensors are characterized by high measurement precision independent of the color and the degree  
Figure 4: Multi-agent system.  
of reflection of light of the tested object. The sensor is equipped with an infrared laser of the Vertical-Cavity Surface-Emitting Laser (VCSEL) type with wavelength of 940 nm and a SPAD (Single Photon Avalanche Detector) avalanche photodiode. The field of view of each optical sensor is equal to a 27\({}^{\circ}\) cone. The sensors are placed around the ring on the head. Visualization of cones and field of view for every sensor is presented in Figure 6. Even though sensors do not cover all space in front of the head, they are still able to correctly detect obstacles smaller than the human hand. The range of obstacle detection by the head is constrained to 1.5 m. The CAD project of the head is presented in Figure 5a and a photo of the robot with the head is shown in Figure 5b.  
In order to collect distances from sensors and preliminary data analysis, a PCB with microcontroller STM32F446 was designed and made. The board communicates with the computer via a serial interface--UART-USB. The simple electronic scheme of the board is presented in Figure 7. All data obtained from the sensors are sent to the computer,  
Figure 5: The head designed to detect obstacles: (**a**) CAD project; (**b**) head attached to the wrist of the robot.  
Figure 6: Visualization of cones and field of view for every VL53L1 time-of-flight sensor.  
which analyzes them in order to determine the shortest distance between the head and the obstacle. For the determination of \(d_{obs}\), +the smallest distance transmitted by one of the sensors is selected.  
Due to the time and method of communication of the microcontroller with VL53L1 sensors, to ensure the required sampling rate (15 Hz) of data from VL53L1 sensors, it is necessary to use two separate I2C buses operating in Fast mode. Five VL53L1x sensors are connected to each master bus.  
The head that was been built is only a prototype; therefore, its cost is very low and it aims to check whether an investigated method and algorithms can properly work in laboratory conditions using a real robot. Thanks to this, it is possible to verify the results of the simulation tests in reality. The head is designed in such a way that it is possible to extend its functionality and add more sensors. New sensors can be added to the existing ring. An additional ring can also be added in which the sensors are pointing at different angles. The head is empty and it is also possible to insert the gripper. The head can be optimized by taking into account its size.