# Implementation Details for Controlling Contactless 3D Virtual Endoscopy
## 4 Results  
During the surgery, based on our experience, we found that the best modality for surgeon-computer interaction is contactless interaction by hand. The next issue we needed to resolve was the ability for real-time interaction on a medium-range computer system (due to the cost issues). We used a disposal computer system with 4-core Intel Ice Lake processor i5 series, 1.5 GHz Core speed, 8 MB L3 Cache memory, 4 \(\times\) 1280 KB L2 Cache memory, and 4 \(\times\) 48 L1 Data Cache memory; 8 GB DDR4 RAM and Intel Xe built-in graphics. For capturing the hand motions, we used an Intel RealSense D435 stereo vision camera that has the depth perception capability needed for capturing 3D hand and finger motions. Since our computer system specification did not allow adding any external graphics solution and computer power, we focused on lightweight hand-tracking software solutions. Since MediaPipe Hands [12] enables precise hand and finger tracking solution that does not require any resource-powerful computer system for hand tracking solutions and is based on machine learning (ML) software, we implemented and optimized that solution. MediaPipe Hands 213] used in the API utilize an ML pipeline consisting of:  
* a palm detection model for defining the ROI in the picture captured and returning a hand bounding box, as shown later in Section 4.2;
* a hand landmark that operates on the cropped region of interest defined by the palm detector and returning precise 3D hand keypoints.  
To detect initial hand locations, a single-shot detector is used. This approach uses a single deep neural network that can detect left and right various hand sizes and even can detect hands overlapping. In hand detection, the lack of high contrast patterns (like for example eyes on the face) makes it comparatively difficult to detect them reliably from their visual features alone. Instead, providing additional contexts, such as arm, body, or person features, aids accurate hand localization. The presented model achieves an average precision of 96% in palm detection. We measured this rate by re-initiating the model with different hands and initial positions for tracking in a row of 100 attempts with four people (eight different hands) serving as a testbed. All four failed attempts occurred when the palm was too near to the person's head. We indicated a successful attempt with the successful palm detection within three seconds from the initial positioning of the hand in front of the camera, and detecting the palm in the region of interest frame (more details in Section 4.2).  
We implemented MediaPipe 21 Hand Landmarks for precise hand detection by identifying 21 3D hand-knuckle coordinates [13]. The implemented MediaPipe algorithm provides up to 20 FPS hand/gesture detection on a provided system. The outputs from the algorithm are:  
* Center of hand in 2D Image Pixel coordinates;
* Center of hand in 3D world/camera coordinates;
* Each finger state (open or closed): thumb, first, second, third, and fourth finger.  
We implemented the following gestures: ONE, TWO, THREE, FOUR, FIVE, FIST, YEAH, ROCK, SPIDERMAN, and OK, as shown in Figure 1. Each of the gestures is defined by a finger state. The full list of gestures is presented in the following section.