# An Object Detection and Localization Method Based on Improved YOLOv5 for the Teleoperated Robot
## 2 Method
### Object Localization
#### 2.2.2 Distance Estimation  
In our work, we use the ZED2 depth camera and the YOLOv5_Tel model to achieve real-time distance estimation between the object and the camera left eye. The ZED2 depth camera can obtain the 3D point cloud coordinates (x, y, z) of any pixel in the image based on binocular stereo vision. The object class, the confidence score, and the predicted box coordinate (x\({}_{\text{min}}\), y\({}_{\text{min}}\), x\({}_{\text{max}}\), y\({}_{\text{max}}\)) are returned when the object is detected by the YOLOv5 network model, where (x\({}_{\text{min}}\), y\({}_{\text{min}}\)) is the coordinate of the top left-most of the predicted box and (x\({}_{\text{max}}\), y\({}_{\text{max}}\)) is the coordinate of the down right-most of the predicted box. It is possible to determine an object's pixel point coordinates within an image as well as the matching three-dimensional point cloud coordinates when an object is detected. The distance between the object and the left eye of the camera can be calculated through the Euclidean distance formula, as shown in Equation (7).  
\[distance=\sqrt{x^{2}+y^{2}+z^{2}} \tag{7}\]  
Researchers usually only solve the distance between the center point of the object and the left eye of the camera as the distance estimation result [37]. However, the depth information of some pixels cannot be obtained due to occlusion or anomaly in the standard  
Figure 5: Binocular camera ranging geometric model.  
depth sensing mode of the ZED2. If the depth information of the center point cannot be exactly obtained, it will cause the abnormal distance estimation of the object. In our work, we used the center point of the prediction box and its surrounding 8 points to solve the distance estimation to improve the accuracy and the robustness to lighting conditions and occlusions, and their position distribution is shown in Figure 6.  
In the teleoperation system, when the distance is too close or too far between the robot and the object, the operation task cannot be done so that there is no need for distance estimation to the object. In addition, the predicted box and label information of the object are prone to occlusion in the image, which is not conducive to the feedback of environmental information. Therefore, the object is detected and estimated distance in the range of 0.3-3 m between the object and the left eye of the camera in our work. The algorithm of the distance estimation method is shown in Algorithm 1.  
```
0: Self-trained YOLOv5 model and ZED2 depth camera
0: Objects class and their distance from the camera's left eye
1: Initialize the depth camera(HD720p, 60 FPS, depth minimum distance = 0.3 m)
2: Load the self-trained YOLOv5 model
3:While(true)
4: Stereo image is captured for each frame (3D image)
5: The image is pre-processed and resized to (640 \(\times\) 640)
6: Input the image into the self-trained YOLOv5 model for object detection and return (\(x_{min}\), \(y_{min}\), \(x_{max}\), \(y_{max}\))
7:foreachobjectdo
8:if(object == 1)then
9: Calculate the coordinates of 9 points
10:\(distance=0\), \(n=0\)
11:foreach\(point_{i}\)do
12:if(point,isValidMeasure())then
13:\(\mathrm{dis_{i}=\sqrt{x_{i}{}^{2}+y_{i}{}^{2}+z_{i}{}^{2}}}\)
14:\(distance=(distance*n+dis_{i})/(n+1)\)
15:\(n=n+1\)
16:endif
17:endfor
18:if(0.3 \(\leq\) distance and distance \(\leq\) 3)then
19: Plot predicted box, class, confidence, distance labels
20:endif
21:endif
22:endfor
```  
**Algorithm 1** Object Detection and Distance Estimation  
Figure 6: The positional relationship of the 9 points used for distance estimation as indicated by the red dots.