# Augmented Reality Assisted Assembly Training Oriented Dynamic Gesture Recognition and Prediction
## 4 Experiments and Discussion  
### Experimental Design and Datasets  
The experiments are divided into two sections to evaluate the proposed algorithm in this paper. The first section is to validate the recognition accuracy and efficiency of the proposed dynamic gesture recognition algorithm. We conduct the experiments on 4 datasets compared with other algorithms which contain 4 parts: frame recognition, action recognition, action boundary segmentation, and the effect of image resolution on recognition. The second section is to validate the effectiveness of proposed algorithm for operation recognition and prediction and the reliability of proposed ARAAT system. We invite participants to take real-time ARAAT tasks on HoloLens device to evaluate the naturalness of interactions in ARAAT.  
In the first section, the proposed algorithm for dynamic gesture recognition is evaluated on two public datasets (Sheffield Kinect Gesture (SKIG) dataset [33] and Sebastien Marcel Dynamic Hand Posture Dataset [34]) and two homemade datasets (Assembly Gesture Video Dataset and HoloLens ARAAT Dataset).  
The SKIG Dataset was captured by a Kinect device that included a RGB camera and a depth camera. It contains 10 gestures for Circle (clockwise), Triangle (anti-clockwise), Up-Down, Right-Left, Wave, \"Z\", Cross, Comehere, Turn-Around, and Pat. Six subjects performed each gesture 18 times and there are 1080 RGB image sequences in total.  
The Sebastien Marcel Dynamic Hand Posture Dataset contains 4 hand gestures: Clic, Rotate, Stop-Grasp-Ok, and No. Each gesture in the dataset was performed 13-15 times and recorded by image sequences.  
The sample image frames of the SKIG Dataset and Sebastien Marcel Dynamic Hand Posture Dataset are presented in Figures 11 and 12. These two datasets both comprise common gestures which are not very relevant to assembly. Thus, we introduced two new homemade datasets, called the Assembly Gesture Video Dataset and HoloLens ARAAT Dataset, to validate our approach.  
The Assembly Gesture Video Dataset contains 437 assembly operation gesture video sequences collected online from different uploaders on YouTube demonstrated in Figure 13. The dataset has 6 standard assembly actions: \"Pointing\", \"Moving\", \"Grasping\", \"Releasing\", \"Scaling\", \"Rotating\", and the transition action \"Null\". These 6 standard actions were generalized according to the ASME standard operations [28]. The resolution for each video is 640 \(\times\) 480.  
The HoloLens ARAAT Dataset is constructed from various action videos from HoloLens RGB cameras. The videos of ARAAT tasks used in the experiments were captured by HoloLens and the resolution of the image is 1028 \(\times\) 720 with 30 frames per second (fps). The dataset is divided into two parts: the single action dataset and the ARAAT task dataset. The single action dataset contains 6 types of actions: \"Pointing\", \"Moving\", \"Grasping\", \"Releasing\", \"Scaling\", and \"Rotating\". Videos were recorded of 30 participants, who conducted the actions of real assembly operations based on the ASME standard operations. Each participant performed 6 actions 10 times each, which created 60 videos for each participant and 1800 videos in total. Each video has approximately 100-150 frames. The single action dataset is split into the training and testing sets. The ARAAT task dataset was recorded by 30 participants and is used for testing sets in the boundary segmentation experiments. Each trainee performed the ARAAT tasks, containing 6 types of actions: \"Pointing\", \"Moving\", \"Grasping\", \"Releasing\", \"Scaling\", and \"Rotating\", in any order at will for 10 times, producing 300 testing task videos.