## 4 Methodology
### Gesture Acquisition
#### 4.2.2 Gesture Acquisition and Interpretation in HoloLens 2  
Gesture recognition for the palm-based control in HoloLens 2 utilized the MRTK Unity3D plugin and more specifically, the Hand-tracking API. The MRTK hand-tracking API did not provide (at least during the time of the development) high-level functions for recognizing extended fingers or an open/closed palm. The main values provided are hand-edness (left or right hand), joint transform (the position and orientation in the 3D space), and joint type (e.g., index tip, index distal, metacarpals, etc.). As a result, the development of palm navigation had to start by implementing a Gesture Recognizer component able to detect an open palm gesture in real-life conditions where fingers are quite often occluded or not completely straight, etc. The gesture recognizer component is based on the work provided by [PERSON] ([[https://github.com/[PERSON]/MRTK_HL2_HandPose/](https://github.com/[PERSON]/MRTK_HL2_HandPose/)]([https://github.com/[PERSON]/MRTK_HL2_HandPose/](https://github.com/[PERSON]/MRTK_HL2_HandPose/)), accessed on 20 December 2021), but is heavily modified to adjust to our case needs. The code takes into account finger-eye alignment and co-linearity between finger pairs to cope with finger occlusions. It also uses ratios between different hand's parts instead of absolute measurements to be able to handle different hand sizes. Since in the palm-based control we are only interested in two classes of gestures, open/extended palm and closed palm,the gesture detector does not have to be very precise in recognizing gesture details. For that reason, the gesture classification criteria are modified to match the needs of the application: The criterion for an open palm classification is the detection of an extended thumb and at least three extended fingers. This way, the recognizer is more sensible in recognizing an open palm gesture, which is the main state of interest.  
When the Gesture Recognizer module detects an open palm, the hand joints' information (position and rotation) is processed by the Palm Navigation module that calculates the navigation commands (pitch, yaw, roll, throttle, and take off/landing) and their amplitude (or velocity values) based on the hand joints' current and past rotation angles. As mentioned in Section 4.1.2, in order to avoid physical stress, the reference null position can change every time the user performs the reset command (closed palm gesture). The next open palm position after the reset command corresponds to the reference position and the joint angles and position at that point are stored as the neutral (or zero) transforms for the future commands.  
In particular, three components of the hand's position are relevant: the hand's Up normal vector, perpendicular to the palm; the hand's Forward normal vector, perpendicular to Up, and facing forward, along the direction of the middle finger; and the hand's vertical position. An overview of the vectors and their correspondence with the drone control values is shown in Figure 7. Interpretation is relative to the reference hand position. The current position's normals are translated to the y-axis and analyzed into components. Pitch is computed by the angle of the Forward vector to the xz-plane. Yaw is tied to the angle of the Forward vector's xz-component to the x-axis. Roll is computed by the angle of the Up vector's yz-component to the y-axis. Throttle is proportionate to the vertical (y-axis) distance between the current position and the reference. All commands allow for a \"neutral zone\", so that minor or involuntary motions are not translated to a drone command.