## 5 Results of Experiments on Human-Robot Interaction
### Human-Robot Interaction Using Hand Postures and Speech

Table 12: Average accuracies of different types of speech instructions. are not detected, as shown in Figures 21a and 22a, incorrect results of gesture-speech combination are obtained. Furthermore, to show the results of the integration of deictic hand postures and speech more intuitively, a graphical user interface based on Qt was designed. Moreover, additional images are captured in another scene which is not in the training data set. Some results are shown in Figures 23-26. In addition, if hand postures are incorrectly classified, deictic hand postures and speech cannot be correctly integrated, as shown in Figures 27-30, perhaps because the training dataset does not include images in this scene, which are shown in Figures 27-30. Moreover, appearances of the same type of hand postures vary greatly, and appearances of different kinds of hand postures are similar. Specifically, although hand postures in Figure 27a,b are the same type of hand postures, they differ in appearances greatly. However, the appearances of hand postures in Figures 27a and 28b are similar. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**POS of the** & & **Accuracy of** \\
**Demonstrative** & **Speech Instructions** & **Experiments** & **Gesture-Speech** \\
**Word** & & & **Resolution Results (\%)** \\ \hline \hline \multirow{4}{*}{pronoun} & Go over there quickly & 12 & 91.7 \\  & Go there & 12 & 100.0 \\  & Check whether there are dangerous goods & 12 & 75.0 \\  & Check for flames over there & 12 & 91.7 \\  & Move over there & 12 & 83.3 \\ \hline \multirow{4}{*}{adjective} & Cut the wire over there & 12 & 50.0 \\  & Put the speculum into the crevice of the door over there slowly & 12 & 41.7 \\  & Examine the circumstance over there & 12 & 100.0 \\  & Check the situation over there & 12 & 100.0 \\ \cline{1-1}  & Go to the elevator over there & 12 & 100.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 13: Experimental results using voice commands and hand postures. \begin{table}
\begin{tabular}{c c c} \hline \hline
**POS of the** & & **Accuracy of** \\
**Demonstrative** & **Instructions** & **Recognition (\%)** & **Gesture-Speech** \\
**Word** & & & **Combination Results (\%)** \\ \hline \hline \multirow{4}{*}{pronoun} & Go over there quickly & 91.7 & 91.7 \\  & Go there & 100.0 & 100.0 \\  & Check whether there are dangerous goods & 100.0 & 100.0 \\  & over there quickly & 100.0 & 100.0 \\  & Check for flames over there & 100.0 & 100.0 \\  & Move over there & 91.7 & 91.7 \\ \hline \multirow{4}{*}{adjective} & Cut the wire over there & 100.0 & 100.0 \\  & Put the speculum into the crevice of the & 100.0 & 100.0 \\ \cline{1-1}  & door over there slowly & 100.0 & 100.0 \\ \cline{1-1}  & Examine the circumstance over there & 100.0 & 100.0 \\ \cline{1-1}  & Check the situation over there & 100.0 & 100.0 \\ \cline{1-1}  & Go to the elevator over there & 100.0 & 100.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 14: Experimental results using hand postures and the text of voice commands. Figure 20: Sample images used in gesture–speech combination. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**Hand Postures**} & \multicolumn{4}{c}{**Prediction**} & \multirow{2}{*}{**Accuracy (\%)**} \\ \cline{2-2} \cline{4-6}  & **1 Left** & **1 Right** & **PalmLeft** & **PalmRight** & \\ \hline
1 Left & 30 & 0 & 0 & 0 & 100.0 \\
1 Right & 0 & 28 & 0 & 0 & 93.3 \\ PalmLeft & 0 & 0 & 30 & 0 & 100.0 \\ PalmRight & 30 & 30 & 0 & 30 & 100.0 \\ \hline \hline \end{tabular}
\end{table} 
Table 15: A confusion matrix of deictic hand postures. Figure 21: Example of comparison when the system correctly classifies the hand posture and when it fails. (**a**) The hand posture is not detected and recognized; (**b**) the hand posture is correctly classified. Figure 22: Example of comparison when the system correctly classifies the hand posture and when it fails. (**a**) The hand posture is not detected and recognized; (**b**) the hand posture is correctly classified.