# Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction
## 3 Dynamic Gesture Recognition in Augmented Reality Assisted Assembly Training
### Action Recognition
#### 3.2.2 Gesture Classification  
We use the feature matrix \(F_{t}\) as the input of the classifier \(C\) and recognize the gestures. In other words, classifier \(C\) can be seen as a mapping for \(\forall f_{t}\in A_{i,j}\subset V\), \(i=1,2,\ldots,N\)\(j=1,2,\ldots,M_{i},t=2,3,\ldots\). By calculating the feature matrix \(F_{t}\), we can obtain the recognition result \(C(f_{t},F_{t})\in\mathbf{A}\). The classifier \(C\) can be defined as:  
\[C:f_{t}\times F_{t}\to\mathbf{A} \tag{14}\]  
In this paper, the directional pulse-coupled neural network (DPCNN) is chosen as the classifier. The DPCNN can classify and recognize dynamic gestures by template matching and is often applied to real-time applications, which is verified in our previous work [32]. The DPCNN can select the neuron firing directions by different excitations to reduce computational complexity and time of traditional PCNN. The DPCNN can also improve recognition accuracy by the choice of reasonable firing directions. For each gesture in the standard basic action set \(\mathbf{A}\), that is, (Pointing, Moving, Grasping, Releasing, Scaling, Rotating, Null), we construct a single gesture video dataset that is used as a training template to train parameters of the DPCNN classifier. We input the feature matrix \(F_{t}\) of each single gesture into the DPCNN and then train the classifier.  
To increase the efficiency of gesture recognition, an action unit is introduced in the process of feature classification. We define \(L\) as the length of the action unit. An \(L\) that is too small will lead to poor computational efficiency, while an \(L\) that is too large will increase computational complexity. To determine the value of \(L\), we collected assembly gesture video data online and conducted an experimental statistical analysis. By trial and error, we determined that \(L=20\) is the best length for the action unit. Because of the use of the action unit, the dimension of features is reduced to 20 instead of all frames. For \(\forall f_{t}\in A_{i,j}\subset V,i=1,2,\ldots,N,j=1,2,\ldots,M_{i},t=20,21, \ldots\), \(f_{t}\) and 19 frames before \(f_{t}\) construct an action unit denoted by \(U_{t}\), that is,  
\[U_{t}=\{f_{t-19},f_{t-18},\ldots,f_{t}|f_{t-19}\oplus f_{t-18}\oplus\ldots \oplus f_{t-1}\oplus f_{t}\} \tag{15}\]  
The multiple features of the action unit \(U_{t}\) are then gained by the feature extractor. Based on the features, the classifier assigns each frame in the action unit \(U_{t}\) to the appropriate action category and gives scores to each action category, which are shown in Figure 5(a). This reduces the computational cost and time, as well as the impact of video length on complexity, allowing even long operation videos to be processed quickly and efficiently. The scores then produce a unit probability distribution \(p_{U}(A)\) for the action class. The distribution suggests the probability of actions assigned to each class. As the action unit moves forward along frames, the probability distribution \(p(A)\) on the whole video can be produced, as illustrated in Figure 5(b).  
Figure 6: The output of the classifier. (**a**) Scores of each action in an action unit, and (**b**) the score distributions of each action on an input video.