# Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction
## 3 Dynamic Gesture Recognition in Augmented Reality Assisted Assembly Training
### Action and Operation Prediction  
The purpose of ARAAT is to improve the standard for trainees by evaluating performance of hand operations. Thus, the prediction of actions and operations is as necessary as action recognition. The prediction algorithm can identify the assembly operations of trainees early by recognizing actions and the judgement of current orders, allowing for the evaluation of the standard order and achievement of the following actions.  
In ARAAT, trainees are required to use their hands or various virtual equipment to complete different assembly training tasks. As mentioned in Section 3.1, \"inserting\", \"fastening\", and \"screwing\" are typical practical assembly operations. To conduct these training operations, trainees need to hold some equipment as assistance, such as a wrench or screwdriver. By moving  
Figure 8: The algorithm of boundary segmentation.  
the workpieces with their hands, trainees can carry out the \"inserting\" operation; by rotating the wrench, trainees can perform the \"fastening\" operation; by rotating the screwdriver, trainees can perform the \"screwing\" operation. In ARAAT, trainees also need to complete the assembly training through controlling the virtual workpiece by their hands or equipment.  
Each assembly operation in ARAAT contains several actions in a standard order, concluded as follows (refer to Figure 9):  
Inserting: grasping (the workpieces) \(\rightarrow\) moving \(\rightarrow\) rotating/scaling \(\rightarrow\)releasing (the workpieces), that is,  
\[\text{Inserting }=\text{grasping}\oplus\text{moving}\oplus\text{rotating}/ \text{scaling}\oplus\text{releasing} \tag{18}\]  
Fastening: grasping (the wrench) \(\rightarrow\) moving \(\rightarrow\) moving around \(\rightarrow\)releasing (the wrench), that is,  
\[\text{Fastening}=\text{grasping}\oplus\text{moving}\oplus\text{moving}\text{ around}\oplus\text{releasing} \tag{19}\]  
Screwing: grasping (the screwdriver) \(\rightarrow\) moving \(\rightarrow\) rotating \(\rightarrow\) releasing (the screwdriver), that is  
\[\text{Screwing}=\text{grasping}\oplus\text{moving}\oplus\text{rotating} \oplus\text{releasing} \tag{20}\]  
We can evaluate the performance of actions and operations by recognizing the gestures and predicting the action order which the trainee is performing. Furthermore, when the trainee carries out an operation in ARAAT, the virtual object needs to give the corresponding reaction. If there is a delay in the reaction, the operation will not be smooth and the interaction will be unnatural. In order to avoid inconsistencies with the operation, it is necessary to predict the current and next actions and give the computer enough time to make real-time feedback reactions. By the recognition of frames in the action unit, we can predict the uncompleted actions and reduce the response time, providing trainees with a smooth and natural human-machine interaction experience in ARAAT.  
Based on the above discussion, the framework of action and operation prediction is given in Figure 10.  
Figure 10: Action and operation prediction in ARAAT.  
Figure 9: The action orders of three typical operation “inserting”, “fastening”, and “screwing” in ARAAT.