# AnyGesture: Arbitrary One-Handed Gestures for Augmented, Virtual, and Mixed Reality Applications
## 3 Materials and Methods
### Feature Extraction  
The proposed framework requires two primary features for recognizing one-handed gestures:  
* **Joint Positions** for static hand shape detection and matching;
* **Finger Tip Positions** for recording spatial information required to perform dynamic gestures. Hand poses and shapes can be stored for later recognition of static hand gestures, e.g., while a user is performing the hand movement, it can be matched against a predefined set of hand shapes. In order to increase the recognition performance for users with hands below or above the average human hand size, the hand positions are adjusted by a scaling factor. This normalization is necessary to increase the recognition accuracy for gestures that were recorded by a different user than the user performing the gestures. To achieve this, each joint position is divided by the hand scaling factor. The raw hand shape does not take hand rotation or orientation into account and therefore has rotation invariance. Performing a hand shape that resembles a \"thumbs up\" will therefore be indistinguishable from a \"thumbs down\", but it should likely have a different meaning. Furthermore, for some gestures, it is necessary to know whether the hand is facing the user. For example, if a menu should be opened when an open hand gesture in the direction of the user is performed. To solve this problem, the orientation of the hand in relation to the head of the user is utilized. The direction the user is looking at is calculated by using the forward vector of the HMD. Then, the angles between head direction and the X/Y/Z directions perpendicular to the hand at its current position are calculated. This feature can clearly identify the orientation of a hand relative to the VR HMD. This allows for gestures that should be activated when the hand is facing the user or the palm should be upwards, etc. A static gesture is then formed by storing joint positions provided by the hand tracker and the hand orientation relative to the user's facing direction. Static gestures can be captured by pressing a single button on the keyboard. To be more precise, the user shapes their hand as the gesture should be defined and presses a key, and then this shape is added to the set of predefined gestures that can be recognized by the system (see Algorithm 1). Figure 2: Hand points used to capture static gestures. Dynamic gestures, on the other hand, use spatial information for specific points on the hand, e.g., finger tip positions or the palm. The proposed framework allows the recording of spatial information for any joint with arbitrary movements that can be stored for later recognition. This process is depicted in Figure 3. Each dynamic gesture requires a \"start\" shape, i.e., the system needs to know when the user intends to perform a dynamic gesture. For this reason, each dynamic gesture has a static gesture attached to it, which is later matched to the hand shape from the tracking device during run time. A dynamic gesture can be considered a more complex variant of a static gesture. While a static gesture is performed as soon as it is recognized, it can be used as a cue to activate a dynamic gesture. In order to record and store a dynamic gesture, the user shapes their hand as the gesture should be defined and presses a key to start the recording process. The gesture recording component attaches itself to predefined points on the hand, such as the index finger tip, and follows the path of it until the key is pressed again. These predefined points should be chosen by the user before a gesture is being recorded. A gesture that is primarily conducted with the index finger should record the index finger, a swipe gesture might be recorded best if the palm is observed, etc. Which joints should be tracked for which gesture can be decided by the user of the framework. Recording will save a path consisting of 3D points for chosen joints. These 3D points are stored by computing the spatial differences between subsequent points.