# Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction
## 3 Dynamic Gesture Recognition in Augmented Reality Assisted Assembly Training
### Action Recognition
#### 3.2.1 Feature Extraction  
The actions in ARAAT are all dynamic gestures with movements. It is difficult to distinguish gestures with only 2D static features, so both static and dynamic features are extracted simultaneously to provide greater accuracy of action recognition. The static features are mainly the 2D representation characters of gestures and the dynamic features are the trajectories of hand motions. 1. Static Features  
In this paper, Zernike moments [30] and histogram of oriented gradient (HOG) are used to extract static features [31]. Zernike moments were first proposed by [PERSON] in 1934 to uniquely describe functions on the unit disk and were then extended to describe images for feature extraction. Zernike moments have the properties of shift-invariant, scale-invariant, and rotation-invariant, and are always used as descriptors for gestures. Therefore, differently sized and shaped artifacts will not have a great influence on the gesture recognition results. However, Zernike moments cannot achieve good results in texture recognition. HOG is a feature descriptor for object detection in the static image that counts occurrences of gradient orientation in localized portions of an image. Thus, HOG is used to compensate for the recognition of local texture features in images. For \(\forall f_{t}\in A_{i,j}\subset V,i=1,2,\dots,N,j=1,2,\dots,M_{i},t=2,3,\dots\), the static feature matrix from \(f_{1}\) to \(f_{t}\) can be defined as  
\[F_{t}^{\text{Static}}=\left[ \begin{array}{c}f_{2}^{\text{Static}}\\ f_{2}^{\text{Static}}\\ \vdots\\ f_{t}^{\text{Static}}\end{array} \right] \tag{6}\]  
where \(f_{t}^{\text{Static}}=(Z_{t}^{nm},V_{t}^{\text{HOG}},f_{t}^{\text{Type}})\) is the static feature vector of frame \(f_{t}\), \(Z_{t}^{nm},V_{t}^{\text{HOG}}\) and \(f_{t}^{\text{Type}}\) are Zernike moments, HOG feature vector and the gesture type, respectively. Figure 5: The types of standard actions for assembly: (**a**) rotating, (**b**) moving, (**c**) grasping and releasing, (**d**) pointing, and (**e**) scaling. Before calculating Zernike moments, each frame must first be normalized. Let \(p_{t}(x_{t},y_{t})\) is a pixel of frame \(f_{t}\). \(m_{f}\) and \(n_{f}\) are the length and width of \(f_{t}\), and \(0\leq x\leq m_{f}\), \(0\leq y\leq n_{f}\). Then, we conduct a mapping transformation of \(f_{t}\) to a normalized polar image \(f_{t}^{\prime}\). We define \(p_{t}^{\prime}(x_{t},y_{t})\) as the pixel of \(f_{t}^{\prime}\), where \(-n_{f}/2\leq x,y\leq n_{f}/2\). The Zernike moments of order \(n\) with repetition \(m\) for gestures are calculated as follows,  
\[Z_{t}^{mm}=\frac{n+1}{\pi}\sum_{x_{t}}\sum_{y_{t}}p_{t}^{\prime}(x_{t},y_{t})R _{t}^{mm*}(\rho_{t})e^{jm\theta_{t}} \tag{7}\]  
\[\theta_{t}=\arctan(y_{t}/x_{t}) \tag{8}\]  
\[\rho_{t}=\sqrt{{x_{t}}^{2}+{y_{t}}^{2}} \tag{9}\]  
where \(n\) and \(m\) are nonnegative integers, \(m\leq n\) and \(n-m\) is even. \(R_{t}^{mm}\) is a radial polynomial and \(R_{t}^{mm*}\) is the complex conjugate. \(\rho_{t}\) is polar value and \(\theta_{t}\) is polar angle. We define the gesture type \(f_{t}^{\text{Type}}\) as the finger number used in each gesture, that is,  
\[f_{t}^{\text{Type}}=\left\{ \begin{array}{rl}1,&if\text{ performing gestures only using the index finger}\\ 2,&if\text{ performing gestures using both the thumb and index finger}\\ 5,&if\text{ performing gestures using all fingers}\end{array} \right. \tag{10}\]  
2. Dynamic Features In addition to the static representation of gestures, the dynamic motions of gestures also need to be acquired. We define \(p\big{(}x_{t}^{C},y_{t}^{C}\big{)}\) as the centroid of hand in each frame \(f_{t}\in A_{i,j}\subset V,i=1,2,\ldots,N\), \(j=1,2,\ldots,M_{i},t=2,3,\ldots\). We construct two vectors by \(p\big{(}x_{t}^{C},y_{t}^{C}\big{)}\), namely \(X_{t}=\big{(}x_{1}^{C},x_{2}^{C},\ldots,x_{t}^{C}\big{)}\) and \(Y_{t}=\big{(}y_{1}^{C},y_{2}^{C},\ldots,y_{t}^{C}\big{)}\).