# Multi-Head TrajectoryCNN: A New Multi-Task Framework for Action Prediction
## 4 Experiments  
### Dataset and Implementation Settings  
Dataset. Our method was evaluated on the NTU RGB+D dataset [42]. NTU RGB+D was collected by Microsoft Kinect v2 sensors, consisting of depth, 3D joints, RGB, and IR modalities. In this paper, we only use the 3D joint modalities. The dataset consists of 60 action classes and 56,880 samples collected by 40 subjects and cameras from different views, and each action class contains around 940 samples. The 40 subjects show varieties of age (i.e., 10\(\sim\)35), gender, and height. The views contain three horizontal angles: \(-45^{\circ}\) (i.e., Camera 1), \(+0^{\circ}\) (i.e., Camera 2), and \(+45^{\circ}\) (i.e., Camera 3). To test the robustness of the algorithms to scale and perspective, two official protocols were adopted for evaluations, respectively, i.e., cross-subject evaluation and cross-view evaluation. (1) For the cross-subject evaluation, the training set comprised the samples of subject IDs 1, 2, 4, 5, 8, 9, 13, 14, 15, 16, 17, 18, 19, 25, 27, 28, 31, 34, 35, and 38. The testing set comprised the samples of the remaining subjects. As a result, 40,320 and 16,560 samples were selected for training and testing sets, respectively. (2) For the cross-view evaluation, the samples of Camera 1 were used for testing, and the remaining samples were for training. As a result, 37,920 and 18,960 samples were selected for training and testing sets, respectively. For more details, refer to [42]. To efficiently evaluate the effectiveness of our method for both action semantic and human motion prediction, the lengths of all samples were processed to a fixed length of 64 frames by the uniform sampling strategies.  
Implementation settings. All models were implemented with TensorFlow. Our learning rate was set to 0.0001 and the batch size was set to 256. All models were conducted on one GeForce GTX 1080 GPU. In experiments, the sequences on NTU RGB+D were unified sampled to a fixed length of 64. The lengths of partially observed motion sequence \(\mathbf{X}_{1}\) (i.e., \(N\)) and future motion sequence \(\mathbf{X}_{2}\) (i.e., \(M\)) were set to 32 (i.e., \(N=M=32\)), and scale factor \(\alpha\) was set to 0.01 empirically. \(n^{\star}\) was set to 2 in the encoders and decoders. The channels of Encoder1 were set to 256. The channels of RB in Decoder1 were set to 256, and the channels of the convolutional layers in Decoder2 were set to 32 (i.e., \(M\)). The channels of CBs in Decoder2 were set to 256 and 128 and the channels of FCs in Decoder2 were set to 256 and 60 (i.e., the action classes). Accuracy and mean per-joint position error (MPJPE) are introduced for our metrics. Accuracy is used for the evaluation of action semantic predictions, which indicates the percentage of correctly predicted categories. The MPJPE \(E\) is used for the evaluation of future motion predictions, which can be formulated by Equation (5).  
\[E{=}\frac{1}{J\times M}\sum_{n=1}^{M}\sum_{k=1}^{J}\left\|\hat{\jmath}_{nk}- \widehat{\jmath}_{nk}\right\|_{2} \tag{5}\]  
where \(M\) is the length of the future motion sequence, \(J\) is the number of joints of the human body, \(\widehat{\jmath}_{nk}\) denotes the \(k\)-th predicted joint at the \(n\)-th pose, \(j_{nk}\) denotes the ground truth joint, and \(\left\|\cdot\right\|_{2}\) denotes the \(L_{2}\) norm loss.