# 3D Behavior Recognition Based on Multi-Modal Deep Space-Time Learning
## 4 Experimental Results  
### Datasets  
In this paper, experiments are conducted on three datasets: the UTD-MHAD dataset [42], the UTKinect-Action3D dataset [43] and the MSR-Action3D dataset [44]. All three datasets were captured by a fixed Microsoft Kinect sensor. All of the datasets provide the 3D coordinates of the corresponding joints. The results of our proposed method are compared against the current state-of-the-art on all 3 datasets.  
The UTD-MHAD dataset was collected in an indoor environment. The dataset contains 27 actions which were performed by 8 actors (4 males and 4 females). Each actor performed 4 times for each action. This dataset includes 3D coordinates of 20 human joints. The dataset contains 861 image sequences.  
The UTKinect-Action3D dataset consists of 10 actions, each of which was performed by 10 different actors. This dataset contains 3D coordinates of 20 human body joints. These 10 actions are \"walk\", \"sit down\", \"stand up\", \"pick up\", \"lift\", \"throw\", \"push\", \"pull\", \"wield arm\" and \"clap\". Each actor executed twice for each action. The dataset contains 199 valid action sequences. For the sake of processing convenience, a total of 200 motion sequences were used in this paper. The missing \"lift\" motion of the tenth actor was interpolated with the 1242 th to the 1300 th frames. The main challenge of UTKinect-Action3D dataset is the perspective and the large intra-class variations.  
The MSR-Action3D dataset contains 20 actions which were performed by 10 different actors. Each actor performed 2 or 3 times for each action. This dataset contains 3D coordinates of 20 human joints. These 20 actions are \"high arm swing\", \"level arm swing\", \"hammer strike\", \"grip\", \"forward punch\", \"high pitch\", \"drawing X\", \"painted hook\". \"circle\", \"high clap\", \"waving hands\", \"flank\", \"bend\", \"kick\", \"side kick\", \"jog\", \"tennis swing\", \"tennis serve\" \"golf swing\" and \"throw after picking\". There are 567 sequences in this dataset. According to the test environment of the current method [43], we divided these 20 actions into three action subsets AS1, AS2 and AS3 (as shown in Table 1). Each subset contains eight actions. The actions in AS1 and AS2 are a group of actions with similar motions, and the actions in AS3 are a complex set of actions. The main challenge in the MSR-Action3D dataset is that some of the actions are similar, especially in AS1 and AS2.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**AS1** & **AS2** & **AS3** \\ \hline horizontal arm wave & high arm wave & high throw \\ hammer & hand catch & forward kick \\ forward punch & draw X & side kick \\ high throw & draw tick & jogging \\ hand clap & draw circle & tennis swing \\ bend & two hand wave & tennis serve \\ tennis serve & forward kick & golf swing \\ pickup and throw & side boxing & pickup and throw \\ \hline \hline \end{tabular}
\end{table}
Table 1: Actions in actions subset AS1, AS2 and AS3.  
In order to reduce the sensitivity of our proposed method against different testing sequences, we propose a preprocessing step for each depth image sequence. It mainly includes the foreground extraction, body-centered bounding box detection, normalization of spatial and temporal dimensions and normalization of depth values. After foreground extraction, only the depth values of the foreground pixels are preserved. Other pixels are set to zero. In the process of clipping the bounding box, the maximum values of the top, bottom, left and right boundaries of the foreground pixels in the entire input sequence are found, and this boundary is used as the bounding box for the entire video. Then the resize method is applied to normalize the temporal and spatial dimensions of all the motion videos to the same size. Finally, the depth value is normalized to the range of 0 1.