# Dynamic Gesture Recognition System with Gesture Spotting Based on Self-Organizing Maps
## 4 Simulation and Experiment
### Real-Time Gesture Recognition  
To conduct the experiment, real-time gesture recognition system was developed in software that ran on a PC. The input gesture was taken by the USB camera, and fed to the system. The recognition result was outputted only when the spotting function detected the end of the gesture. Figure 5 is a screen shot of the implemented real-time gesture recognition system.  
The system was tested with the same gesture that was used in the simulation. The system was trained off-line by using pre-captured gesture data set. The number of training data for the off-line training was 50 samples for each gesture class. Each training sample was acquired by simply capturing 10 consecutive frames. Neither setting up the key posture that represented the gesture, nor manual selection of key frame was done in the acquisition of the training data set. The recognition system then classified the dynamic gesture presented  
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Method**} & \multicolumn{8}{c}{**Gesture Class**} & \multicolumn{8}{c}{**Average**} \\ \cline{2-10}  & **1** & **2** & **3** & **4** & **5** & **6** & **7** & **8** & **9** & \\ \hline JRNN & 100\% & 100\% & 82\% & 89\% & 72\% & 60\% & 99\% & 99\% & 46\% & 83.0\% \\ \hline This work & 82\% & 66\% & 100\% & 89\% & 98\% & 100\% & 100\% & 100\% & 100\% & 92.8\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Gesture recognition accuracies of various classifiers.  
Figure 4: Nine types of gesture.  
in real-time using the weight vectors obtained from the off-line training. In the experiment, recognition test was carried out 100 times for each gesture by the same person who had provided the training data set. Most of the gesture ends were correctly detected even though the gesture groups 1-2-3, 4-5-6, and 7-8-9 were performed in succession (video of the experiment (10 fps) is available on [[http://www2.kansai-u.ac.jp/hikawa/ichikawa.mp4](http://www2.kansai-u.ac.jp/hikawa/ichikawa.mp4)]([http://www2.kansai-u.ac.jp/hikawa/ichikawa.mp4](http://www2.kansai-u.ac.jp/hikawa/ichikawa.mp4)), 19 February 2021). Since the gesture in our method is defined by the fixed number of posture types, the input gesture must be made of the same number of posture types. Therefore, the speed of gesture to be recognized depends on the speed of gesture that has been captured as the training data. The recognition tests were carried out with two speeds, i.e., 5 frames per second (fps) and 10 fps. Since \(F=20\) (10 frames for one gesture), the gesture speeds were 2 second/gesture in case of 5 fps, and 1 second/gesture in case of 10 fps.  
Table 2 shows the experimental results of 5 fps frame rate. The average accuracy of the recognition was 96.6%. NS in the table is the number of cases where no spotting was detected, and MS is the number of cases where multiple spotting occurred. Both cases were counted as errors. Table 3 summarizes the experimental results of 10 fps frame rate. The average accuracy of the recognition rate was 97.0%, and no significant difference is found due to the speed difference.  
\begin{table}
\begin{tabular}{c c c c c c c c c c c} \hline \hline
**Input** & \multicolumn{5}{c}{**Recognition Results**} & \multicolumn{5}{c}{**Spotting**} & **Accuracy** \\ \cline{2-10}
**Gesture** & **1** & **2** & **3** & **4** & **5** & **6** & **7** & **8** & **9** & **NS** & **MS** & **\%** \\ \hline
1 & 98 & 2 & & & & & & & & & 98 \\
2 & & 97 & & 3 & & & & & & & 97 \\
3 & & & 91 & 2 & & & & 2 & 5 & 91 \\
4 & & & 89 & & & & & & 5 & 6 & 89 \\
5 & & & & 100 & & & & & & 100 \\
6 & & & & 100 & & & & & & 100 \\
7 & & & & & 96 & & & 2 & 2 & 96 \\
8 & & & & & & & 100 & & & 100 \\
9 & & & & & & & 98 & & 2 & 98 \\  & & & & & & & & & & Average & 96.6 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Confusion table (5 fps).  
Figure 5: Real-time gesture recognition system.