# 3D Behavior Recognition Based on Multi-Modal Deep Space-Time Learning
## 4 Experimental Results
### Experimental Results on MSR-Action3D Dataset  
During the experiment, we store the features of the depth sequence from the 3D spatial-temporal convolutional neural network, the second 3D convolutional neural network and the 3D maximal pooling layer as corresponding 3D convolutional features for later SVM classification usage.  
Similarly, we also use the depth motion maps corresponding to three subsets AS1, AS2 and AS3 and feed them into another 3D space-time convolution neural network for extracting the 3D convolution features.  
In the fusion phase, we train the model with 3D convolutional features from the depth map sequence, depth motion map and 3D human skeleton manifold representation features corresponding to each subset respectively using linear SVM. In the test phase, three corresponding probability vectors are obtained as the SVM output. Assuming that there are \(n\) action categories, the probability vector is an \(n\)-dimensional vector and can be expressed as \(P_{i}=\{p_{i1},p_{i2},\ldots,p_{in}\}\), where \(p_{in}\) denotes the probability that this action is belonging to the \(n\)th action class for the \(i\)th model (\(i\) = 1, 2, 3). Given the fusion weights \(w_{i}\) (\(i\) = 1, 2, 3), the recognized action category index can be calculated from the following equation:  
\[ci=\underset{j}{\text{max}}\bigg{(}\underset{i=1}{\overset{3}{\sum}}w_{i}p_{ ij}\bigg{)}\quad(j=1,2,\ldots,n) \tag{2}\]  
In MSR-Action3D dataset, we use the same setup as the method of [43]. The results of our proposed methods and other state-of-the-art methods are shown in Table 2. As shown in this table, it can be concluded that our proposed method achieves the best results on all three subsets AS1, AS2 and AS3. This shows that our fusion method not only learns the structure and motion information through the 3D convolutional neural network, but also the details of the motion through the skeleton sequence.  
Figures 3-5 show the confusion matrix of our proposed methods in AS1, AS2 and AS3 respectively. In Figure 3, it can be found that the recognition accuracy of the actions \"hammer\" and \"forward punch\" is relatively low. This is because these two actions are very similar to each other. Similar reasons also apply to the actions \"pick and throw\" and \"bend\". Similar problem could also be observed in Figure 4 for actions \"hand catch\" and \"high arm wave\". Observing the confusion matrix in Figure 5, we can find out that the recognition accuracy is better than the first two subsets since the actions in AS3 are complex and not similar, which further proves that our approach can extract the entire structure and motion information as well as the fine features of motion.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Method of [43]** & **Method of [44]** & **Method of [31]** & **Our Method** \\ \hline
**AS1** & 72.9\% & 87.98\% & 74.5\% & **95.89\%** \\
**AS2** & 71.9\% & 85.48\% & 46.1\% & **87.68\%** \\
**AS3** & 79.2\% & 63.45\% & 96.4\% & **97.32\%** \\
**Whole dataset** & 74.7\% & 78.97\% & 82.3\% & **94.15\%** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparisons of classification accuracy with different methods.  
Table 3 compares the average classification accuracy of our proposed method on the MSR-Action3D dataset against other state-of-the-art methods. From Table 3, it can be concluded that the average classification accuracy of our proposed method is higher than all other competitors. This reflects the superiority of our proposed method, which can extract more space-time characteristics to enhance the discriminative power.  
Figure 4: Confusion matrix of our proposed method in AS2.  
Figure 5: Confusion matrix of our proposed method in AS3.  
Figure 3: Confusion matrix of our proposed method in AS1.