## 2 Materials and Methods
### Systems for Hand Tool Tracking
#### 2.1.4 System Based on OpenPose  
This system, like the one based on the AKBT, uses the information of the operator upper limb pose to obtain the tool's point position. In detail, it relies on the 3D position of some hand's keypoints. OpenPose [27; 28; 29; 30] can be considered as the state-of-the-art approach for real-time human pose estimation. It is the first framework that can jointly detect human body, hand, facial and foot keypoints on single images. OpenPose is a multi-stage CNN that uses a bottom-up approach to find every instance of a key point and then attempts to assemble groups of key points into skeletons of distinct humans. The deep learning OpenPose model is based on a CNN and follows a precise pipeline: in the first step, the model computes confidence maps for every body part detection; in the second part, it predicts part affinity fields (PAF) in order to associate every key point of every person in the frame; in the third step, bipartite matching is computed, so a several-graph connection is evaluated in order to choose the best performing PAF; in the last step, results are parsed and the skeleton is reconstructed. A confidence map is the 2D representation of the belief that a particular body part can be located. A single body part will be represented on a single map. Therefore, the number of maps is the same as the total number of body  
Figure 4: Azure Kinect Body Tracking detectable joints: (**Left**) joints with reference systems; (**Right**) joints’ hierarchy.  
parts, and is a number that depends on the dataset the model is trained on. Instead, PAF is a set of 2D vector fields that encode the location and orientation of limbs over the frame domain. The framework integrates three different trained models for body, hands and face keypoint estimation. They return, respectively, 25 keypoints for the body, 21 \(\times\) 2 keypoints for hands and 70 keypoints for the face (135 keypoints).  
For this study, the authors focused on OpenPose capability to track up to twenty keypoints of the hand, consisting of wrist, finger's knuckles and phalanges. The adopted configuration for the framework has the following characteristics: Body Network input size equal to \(160\times 160\), Hands Net input size equal to \(368\times 368\) and Face Net disabled. The keypoints are just recognized in the 2D frame and the output is expressed in pixel coordinates within the image. Hence, a registered 3D point cloud is used to obtain the 3D position of the keypoints.  
Three out of all detected keypoints of the hand have been considered to compute the estimated hand's pose (Figure 5):  
* Keypoint n.0 (wrist);
* Keypoint n.9 (middle finger knuckles);
* Keypoint n.17 (little finger knuckles).  
The reference frame of the operator's hand was computed as follows. First, two vectors are computed: one from the wrist point (\(kp_{0}\)) to the middle finger knuckle point (\(v_{1}\)) (Equation (1)) and one from the wrist point to the little finger knuckle point (\(v_{2}\)) (Equation (2)). The cross-product between \(v_{1}\) and \(v_{2}\) returns \(v_{3}\), as shown in Equation (3); it is an orthogonal vector to \(v_{1}\) and \(v_{2}\) with a direction given by the right-hand rule outgoing from the back of the hand. Then, it was possible to calculate the vector whose direction is toward the thumb (\(v_{4}\)) as cross-product between \(v_{1}\) and \(v_{3}\) (Equation (4)).  
\[\overline{v}_{1}=[kp_{yx}-kp_{0x}\,\ kvp_{yy}-kp_{0y}\,\ kvp_{yz}-kp_{0z}] \tag{1}\]  
\[\overline{v}_{2}=[kp_{17x}-kp_{0x}\,\ kvp_{17y}-kp_{0y}\,\ kvp_{17z}-kp_{0z}] \tag{2}\]  
\[\overline{v}_{3}=\overline{v}_{1}\times\overline{v}_{2} \tag{3}\]  
\[\overline{v}_{4}=\overline{v}_{1}\times\overline{v}_{3} \tag{4}\]  
Figure 5: OpenPose keypoints for the hand, where the ones used for the calculation are highlighted. (Keypoint n.9) Middle finger knuckles; Keypoint n.0) wrist; (Keypoint n.17) little finger knuckles.  
with  
\[x\equiv v_{1}\,\ y\equiv v_{3}\,\ z\equiv v_{4} \tag{5}\]