# Recognition of American Sign Language Gestures in a Virtual Reality Using Leap Motion
## 2 Methods
### Feature Extraction and Pre-Processing  
The Leap Motion Controller returns the data in units of real-world coordinates (measured in millimetres) that represents positions within the Leap Motion's frame of reference. The data represents the x, y and z coordinates of key hand features (wrist position, palm position, positions of metacarpal, proximal, intermediate and distal bones of fingers and fingertip positions). The 3D positional coordinates from the Leap Motion's Hand Hierarchy are illustrated in Figure 3.  
The general approach for feature extraction presented here is shown in Figure 4. We extract 4 types of hand features, that is, the 3D positions of the, fingertip distances from the hand centroid, elevations of fingertips above the plane of the palm, angles between the fingertips-to-palm-centre vectors. The fingertip angles (adopted from [41]) are angles representing the fingertip orientation projected on the palm. The Leap Motion Controller includes the 3D positions of 11 finger joints. For each gesture we calculate the Euclidian distances between the seven main hand vertices, representing the tip positions of thumb, index, middle, ring and pinky fingers, palm position and wrist position. In all  
Figure 3: Positional data available from Leap Motion. there are 21 distances between the 7 vertices. Additionally, the angular features were generated, representing the angles between any of three different vertices, representing another 35 features. In total, 56 features (21 distance and 35 angular) are extracted. To make all features uniform, the z-score based normalization is applied, which normalizes the data by subtracting the mean and dividing it by standard deviation. Following [42], we describe the kinematic model of the hand movement as follows:  
\[\left[ \begin{array}{c}x\\ y\\ z\end{array} \right]=\left[ \begin{array}{ccc}\cos\theta\cos\psi&\sin\varphi \sin\theta\cos\psi-\cos\varphi\sin\psi&\cos\varphi\sin\theta\cos\psi+\sin\varphi \sin\psi\\ \cos\theta\sin\psi&\sin\varphi\sin\theta\sin\psi+\cos\varphi\cos\psi&\cos\varphi \sin\theta\sin\psi-\sin\varphi\cos\psi\\ -\sin\theta&\sin\varphi\cos\theta&\cos\varphi\cos\theta\end{array} \right] \left[ \begin{array}{c}u\\ v\\ w\end{array} \right] \tag{1}\]  
here \(X,Y,Z\) are the 3D coordinate components, \(u,v,w\) are the velocity components, \(\theta\) is the roll angle, \(\psi\) is the pitch angle and \(\varphi\) is the yaw angle of the hand. The total velocity \(V\) of the hand's centroid is calculated as:  
\[V=\sqrt{u^{2}+v^{2}+w^{2}} \tag{2}\]  
The angles are calculated as follows:  
\[\alpha=\arctan\frac{w}{u} \tag{3}\]  
\[\beta=\arctan\frac{v}{\sqrt{v^{2}+w^{2}}} \tag{4}\]  
\[\gamma=\arccos(\cos\alpha\cos\phi) \tag{5}\]  
here \(\alpha\) is the angle of attack, \(\beta\) is the angle of sideslip and \(\gamma\) is the angle of total attack. The fingertip distances represent the distance of fingertips from the centre of the palm and are defined as:  
\[D_{i}=|F_{i}-C|,i=1,\ldots 5 \tag{6}\]  
here \(F_{i}\) are the 3D positions of each fingertip; and \(C\) is the 3D position associated with the centre of the hand palm in the 3D frame of reference. Figure 4: Pipeline of our approach. We define \(\alpha_{q}(t)=P(y_{1}^{t},q)\) then the above equation is written as:  
\[\alpha_{q}(t)=P(y_{1}|Q_{t}=q)\sum_{r}\left(Q_{t}=q|Q_{t-1}=r\right)\alpha_{r}(t -1) \tag{14}\]  
here \(Q_{t}\) is the state space at time \(t\). We calculate the partial probability from time \(t+1\) to the end of the sequence, given \(q_{t}\) as:  
\[\beta_{q}(t)=\sum_{r}\beta_{r}(t+1)P(y_{t+1}|Q_{t+1}=r)P(Q_{t+1}=r|Q_{t}=q) \tag{15}\]  
here \(\beta_{q}(t)=P\big{(}y_{t+1}^{T}|Q_{t}=q\big{)}\) and \(Q_{t}\) is the state at time \(t\).