# AnyGesture: Arbitrary One-Handed Gestures for Augmented, Virtual, and Mixed Reality Applications
## 3 Materials and Methods
### Framework Architecture  
This section provides a general overview of the implementation of the proposed framework. A crucial component of the framework is the hand data provider, which was implemented to decouple the framework from specific hand tracking hardware and SDKs. It is designed as a wrapper to allow all relevant framework components access to hand tracking data without relying on a specific implementation for hand tracking. In essence, it allows access to hand joints, hand scale, and has support functions to provide data to other components. It generalizes incoming hand tracking data by only relying on joint positions as input. Thus, there is no dependency on specific implementations by manufacturers, and users of the framework only have to make a few adjustments to support a particular device.  
The shape of the hand, which is composed of the positions of each individual hand joint, can be stored for the recognition of a gesture. Common hand tracking devices are able to track the Distal, Middle, and Proximal Phalanges of the fingers and provide information about the palm and wrist positions.  
While a static gesture is recognized with a previously stored hand shape, dynamic hand gestures need a subset of these points. As an example, a dynamic gesture can be recorded by tracking the position of single or multiple chosen joints. In the process of implementing the framework, it was found that the most important joints are the finger tips, palm, and wrist. Invisible 3D objects are attached to these points in order to allow more interaction possibilities. These invisible objects are basically spheres attached to key positions that can be used to record a spatial path or to perform collision detection (e.g., detect if something was touched by a joint). Collision detection within the hand was found to be particularly useful for some gestures, e.g., the pinch gesture, since a pinch can be reliably detected when the thumb and index finger collide.  
To enhance robustness, a subsystem was implemented that observes the movement state of the key positions. It can be considered a post-processing step for the provided hand tracking data. While developing the proposed framework, it was found that such a component is crucial to reliably recognizing gestures since it can prevent many undesired hand gestures. Such undesired hand gestures often occur if too many possible gestures are stored in the system, the user is moving the hands, and no preventive measures are implemented. The position of a hand joint is observed over time, and previously chosen criteria determine whether a specific joint is currently moving or not. These criteria define the following behavior:  
* At which threshold a joint position can be considered still. This is necessary to eliminate noisy data from the hand tracking device.
* How long the joint should be still until it is considered to be not moving.  
As mentioned earlier, the motion path of joints is stored. The sampling rate as well as the amount of stored information can be configured. Depending on the desired behavior, a gesture can only be recognized if one or multiple joints are not moving. Each joint is individually observed, but a higher-level component is used to monitor the state of the whole hand.  
A rough overview of the main parts that make up the framework is shown in Figure 1. The framework supports an easy swap of the _Hand Tracker_ component, allowing many hand tracking solutions to be used with the framework. The _Gesture Recorder_ allows recording and saving gestures that are then stored in a set of known gestures. These gestures are then recognized by the _Gesture Recognizer_ by comparing stored data with live data from the hand tracker. A _Gesture Interpreter_ is used to communicate with the desired application using events that inform when gestures are performed.