## 2 Materials and Methods
### Systems for Hand Tool Tracking
#### 2.1.2 System Based on Deep Detection Model (YOLO)  
The system that uses a deep neural network, i.e., YOLO, is based on two subsequent steps: (1) a deep neural network is trained to localize within a 2D RGB image the specific tool's area (or point) that has to be tracked, and (2) the 3D position of the tracked tool's point is estimated by computing the spacial centroid of the point cloud underlying the ROI found in the previous step.  
You Only Look Once (YOLO) [23; 24] is a family of convolutional neural networks (CNN) that achieve near state-of-the-art results with a single end-to-end model that can perform object detection in real-time. Compared to the approach taken by previous object detection algorithms, YOLO proposes the use of an end-to-end neural network that makes predictions of bounding boxes and class probabilities all at once (Figure 3). Starting from YOLO's pre-trained models, it is possible to re-train the last layers to introduce new classes that are not available in the original dataset to fit the object detector capabilities to new objects. In the re-training process, it is also possible to tune settings of the net and to adjust accuracy over performance, training time and batch iterations, choosing the best weight candidate. YOLO's performance has improved over time, starting from the v1 version. The original YOLO v1 was born as the first object detection network to combine the problem of drawing bounding boxes and identifying class labels in one end-to-end differentiable network. YOLO v4 outperforms most of the other object detection models [31]  
Figure 2: Example of ArUco markers pose estimation: (**Left**) three markers placed on a desk in different poses; (**Right**) three markers placed on three different objects.  
by a significant margin, keeping frame-rate high and making it the best opportunity to detect objects in real-time.  
In this study, YOLO was implemented in its fourth version. The last layers of the model were re-trained to recognize the power drill chuck as a new class, and then the performance of the model inference was evaluated. In order to train the model, a dataset consisting of 735 images was created. The images were acquired by Azure Kinect RGB camera with the power drill in different poses, with heterogeneous light conditions and in different environments. Considering the objective of study, the model was trained with images containing only the particular power drill used for our experiment, leading the net to recognize only the chuck of this tool and not that of other power drills. Data augmentation was performed in order to improve the performance of the model using flip upside-down, flip left-right, rotation and Gaussian blur. The training process was based on transfer learning and the CNN model was initialized with the weights retrieved from the GitHub page of YOLO v4 based on darknet framework. The net was trained with images of 512 \(\times\) 512 resolution and max batches set to 2000. The starting weights were obtained from the training on the COCO dataset, containing 80,000 images and 80 different object classes.  
The trained model was able to recognize the power drill chunk in every frame of real-time acquisition, returning the pixel coordinates x and y of the top-left corner of a box that contains it, the width and the height of the box and the confidence as the probability that the object was classified correctly. In order to estimate the 3D pose of the power drill chuck, the center point (x,y) of the box returned by YOLO model was calculated and the 3D coordinates of that point in the scene were computed by using the functions of Azure Kinect SDK.