SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We validate our framework on four benchmark datasets. Human3.6M[17] is a large dataset that contains 3.6 million 3D human pose data. 15 types of actions performed by 7 actors(S1, S5, S6, S7, S8, S9 and S11) are included in this dataset. Each actor is represented by a skeleton of 32 joints. However, following the data preprocessing method proposed in [28, 31], we only use 22 joints. The global rotations and translations of poses are removed, and the frame rate is downsampled from 50 fps to 25 fps. For testing and validation, we use actors S5 and S11, while training is conducted on the remaining sections of the dataset. CMU-MoCap is a smaller dataset that has 8 different action categories. The global rotations and translations of the poses are also removed. Each pose contains 38 joints, but following the data preprocessing methods in [28, 31], we only use 25 joints. 3DPW[44] is a challenging dataset that contains human motion data captured from both indoor and outdoor scenes. Poses in this dataset are represented in 3D space, with each pose containing 26 joints. However, only 23 of these joints are used, as the other three are redundant. The Archive of Motion Capture as Surface Shapes (AMASS)[29] dataset gathers 18 existing mocap datasets. Following [39], we select 13 from those and take 8 for training, 4 for validation and 1 (BMLrub) as the test set. We consider forecasting the body joints only and discard those 4 static ones, leading to an 18-joint human pose.
Following the benchmark protocols, we use the Mean Per Joint Position Error (MPJPE) in millimeters (ms) as our evaluation metric for 3D coordinate errors and Euler Angle Error (EAE) for Euler angle representations. The performance is better if this metric is smaller.
Implementation Details. Following [28, 30, 39], we set the input length to 10 frames and the predictive output to 25 frames for Human3.6M, AMASS and CMU-Mocap datasets, respectively. For the 3DPW dataset, we predict 30 frames conditioned on the observation of the preceding 10 frames. We choose PGBIG as our backbone model by default. In order to learn the PCF, we add an extra dimension to the output of the backbone model and calculate PCF through an MLP network whose hidden dimension is set to 512. We partitioned the future sequences into three segments with lengths of 3, 9, and 13. The training process was conducted on an NVIDIA RTX 3090 GPU for 120 epochs, allocating 50, 90, and 120 epochs for each respective stage.
Table 1: Results on Human3.6M. *Using EAE as the metric.
dataset	H36M
ms	80	160	320	400	560	1000
POTR*	0.235	0.581	0.990	1.143	1.362	1.826
POTR+Ours	0.233	0.549	0.923	1.056	1.290	1.746
R.S.	29.9	55.7	94.5	108.7	130.1	168.0
R.S.+Ours	28.0	52.8	91.3	105.6	127.3	166.3
LTD	12.7	26.1	52.3	63.5	81.6	114.3
LTD+Ours	10.8	23.0	48.2	59.3	77.5	111.2
MM	12.7	26.4	53.4	65.0	83.6	117.6
MM+Ours	11.5	25.0	52.0	63.7	82.8	117.1
siMLPe	10.7	23.9	50.7	62.6	82.0	116.0
siMLPe+Ours	10.0	22.9	49.4	61.2	80.6	114.6
PGBIG	10.3	22.7	47.4	58.5	76.9	110.3
PGBIG+Ours	9.4	21.3	45.7	56.8	75.4	108.8
Table 2: Results on CMU-MoCap, 3DPW and AMASS.
dataset	CMU
ms	80	320	400	560	1000
R.S.	24.4	80.1	93.1	112.5	141.5
R.S.+Ours	21.4	72.7	85.6	105.6	136.1
LTD	9.3	33.0	40.9	55.8	86.2
LTD+Ours	9.1	31.0	38.5	53.1	84.9
PGBIG	7.6	29.0	36.6	50.9	80.1
PGBIG+Ours	7.5	28.3	35.4	48.6	78.4
dataset	3DPW
ms	200	400	600	800	1000
R.S	99.3	129.7	142.9	161.1	171.7
R.S+Ours	91.8	120.0	139.5	156.2	169.1
LTD	35.6	67.8	90.6	106.9	117.8
LTD+Ours	33.1	64.1	86.1	100.4	110.0
PGBIG	29.3	58.3	79.8	94.4	104.1
PGBIG+Ours	21.4	47.1	67.7	83.5	96.0
dataset	AMASS
ms	80	320	400	560	1000
STSGCN	11.4	37.2	43.8	53.8	69.7
STSGCN+Ours	10.9	36.7	42.8	52.4	68.1
We apply our method on the following approaches on four benchmark datasets: Res.Sup.(R.S.) [32], LTD [31], POTR [33], STSGCN [39], MotionMixer(MM) [3], siMLPe [13] and the current state-of-the-art PGBIG [28]. Res.Sup. is an RNN-based model. LTD, STSGCN and PGBIG are GCN-based models. POTR is a transformer-based method. MotionMixer and siMLPe are MLP-based models. All these methods have released their code publicly. We employ their pre-trained models or re-train their models using the suggested hyper-parameters for a fair comparison. And we also follow the metric they used to evaluate the results.
Table 1 presents experimental results of different backbone models before and after applying our training strategy on Human3.6M dataset. Our frameworks outperform the corresponding backbone models by a considerable margin (more than 1.5). Specifically, when applying our strategy to PGBIG, we obtained a performance of 64.97, which is much better than the performance of the original PGBIG (66.52). Table 2 shows the quantitative comparisons of prediction results on the CMU-MoCap, 3DPW and AMASS datasets in which our proposed framework also achieves the best results. We can observe that the improvement in long-term prediction is greater compared to short-term prediction, indicating that the prior information provided by short-term prediction is more crucial for challenging long-term prediction tasks. It is worth noticing that our framework outperforms the others by a significant margin on 3DPW dataset. Specifically, our strategy outperforms PGBIG by 8.9 (63.59 vs 72.49). We attribute this to our framework's ability to leverage prediction prior for the subsequent prediction in this challenging dataset.
Predicting results. Figure 4 provides some visualization examples of predicted motions, demonstrating that our framework achieves more accurate results. Specifically, Figure 4a represents the action "directions", with the person maintaining an upright position throughout the sequence. The results of PGBIG exhibit a bent posture during long-term predictions, whereas our method can predict states closer to the ground truth position. While in Figure 4b, the person first bends and then stands upright. The PGBIG maintains the bent posture throughout the long-term prediction and our method can accurately predict the changes in posture. It is evident that our method demonstrates an improvement in long-term prediction effectiveness.
$\alpha$  at different stages. As Figure 5 displays, the value of  $\alpha$  progressively increases with each training stage. When a new task is added, the model focuses on the new objective without considering
(a) Predicting result of the action "directions".
(b) Predicting result of the action "takingphoto".
Figure 4: Some visualization results on Human3.6M dataset.
Figure 5:  $\alpha$  at different stages.
the preservation of prior information, resulting in the forgetting of prior knowledge. Here, we utilize  $\alpha$  to mitigate the loss of prior information, thereby improving the overall training of the model.
We conduct several ablation experiments to further verify the effectiveness of our proposed framework.
Evaluation on the number of tasks. As shown in Table 3, the model's performance improves as the number of tasks gets larger from 1 to 3, and it remains stable when the number of tasks becomes larger than 3.
Table 3: The average error of different numbers of tasks.
number of tasks	1	2	3	5	8
avg error	66.95	66.02	65.00	65.05	65.03
Evaluation on different implementations. As Figure 6 shows, we compare the results of four different implementations. PGBIG is our baseline model. "w/o  $\alpha$  " means that we only divide the training process into several stages to train each task without using the PCF. "HC" represents using a hand-crafted coefficient that changes its values similar to our PCF at each epoch. Specifically, in stage  $S_{1}$ , the value of  $\alpha$  is set to 1. In stage  $S_{2}$ ,  $\alpha$  is initially set to 0.1 and increased by 0.05 at each epoch until reaching 0.5, where it remains constant. The same pattern applies to stage  $S_{3}$ . In each subfigure, we show the results of  $Z_{k}$  on the validation set which is obtained at stage  $S_{k}$ .
Figure 6: Comparison of different approaches. PGBIG is a baseline model that is trained without multi-stage. "w/o  $\alpha$  represents training multi-stage process without PCF. "HC" means using a hand-designed coefficient. "Ours" is a multi-stage training process with PCF.
In stage  $S_{1}$ , where no prior information is available, the results of "w/o  $\alpha$ ", "HC", and "Ours" are identical, but superior to the baseline. This validates the effectiveness of decomposing multiple-moment predictions, as it alleviates the constraint of long-term predictions on short-term predictions and enhances the model's ability to learn short-term predictions. In the following stages, the performance of "w/o  $\alpha$ " is worse than "Ours", which indicates that the prior knowledge exploited by our framework benefits the prediction model training. Moreover, as the training period progressed, the performance gap becomes larger. However, the method without  $\alpha$  can still achieve better performance than "PGBIG", indicating that training model in a multi-stage manner can also exploit some useful prior information for prediction. We also note that the performance of "Ours" is much better than "HC", which conducts temporal continual learning with fixed and manually defined PCF. It demonstrates that joint training PCF and model parameters is beneficial.
The forgetting of prior knowledge. As shown in Table 4, introducing the prior compensation factor alleviates the performance degradation from stage  $S_{1}$  to stage  $S_{3}$  of task  $Z_{1}$ 's predictions. Specifically, without PCF, the prediction error of  $Z_{1}$  increases by 0.83, whereas with PCF, it only increases by 0.27. This result suggests that PCF can effectively alleviate the forgetting issue. As a result,  $Z_{1}$  can offer more comprehensive priors for  $Z_{2}$  and  $Z_{3}$  predictions, resulting in better prediction performance.
Table 4: The average error of different tasks at the end of each stage.  ${Z}_{i}$  represents task  $i$  .
(a) Without PCF.
Z1	Z2	Z3
S1	9.03	-	-
S2	9.44	45.33	-
S3	9.86	45.70	92.80
(b) Using PCF.
Z1	Z2	Z3
S1	9.03	-	-
S2	9.10	44.43	-
S3	9.30	44.62	91.37
In this paper, we introduced the temporal continual learning framework for addressing the challenges in human motion prediction. Our framework addresses the constraint between long-term and short-term prediction, allowing for better utilization of prior knowledge from short-term prediction to enhance the performance of long-term prediction. Additionally, we introduced the prior compensation factor to mitigate the issue of forgetting information. Extensive experiments demonstrated our framework's effectiveness and flexibility.
**Limitation.** Our proposed training framework may slightly increase training time. However, the testing time remains unchanged compared to the backbone model.
Broader Impact. We believe our work has value for not only human motion prediction but also for more general prediction tasks and backbone models [41, 24, 16, 34]. This has benefits in various areas such as security monitoring, robotics, and autonomous driving.
This work was supported partially by the NSFC (U21A20471, U22A2095, 62076260, 61772570), Guangdong Natural Science Funds Project (2020B1515120085, 2023B1515040025), Guangdong NSF for Distinguished Young Scholar (2022B1515020009), and Guangzhou Science and Technology Plan Project (202201011134).
[1] E. Aksan, M. Kaufmann, P. Cao, and O. Hilliges. A spatio-temporal transformer for 3d human motion prediction. In 2021 International Conference on 3D Vision (3DV), pages 565-574. IEEE, 2021.
[2] E. Aksan, M. Kaufmann, and O. Hilliges. Structured prediction helps 3d human motion modelling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7144-7153, 2019.
[3] A. Bouazizi, A. Holzbock, U. Kressel, K. Dietmayer, and V. Belagiannis. Motionmixer: Mlp-based 3d human body pose forecasting. arXiv preprint arXiv:2207.00499, 2022.
[4] H.-k. Chiu, E. Adeli, B. Wang, D.-A. Huang, and J. C. Niebles. Action-agnostic human pose forecasting. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1423–1432. IEEE, 2019.
[5] E. Corona, A. Pumarola, G. Alenya, and F. Moreno-Noguer. Context-aware human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6992-7001, 2020.
[6] Q. Cui and H. Sun. Towards accurate 3d human motion prediction from incomplete observations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4801-4810, 2021.
[7] Q. Cui, H. Sun, and F. Yang. Learning dynamic relationships for 3d human motion prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6519-6527, 2020.
[8] K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE international conference on computer vision, pages 4346-4354, 2015.
[9] P. Ghosh, J. Song, E. Aksan, and O. Hilliges. Learning human motion models for long-term predictions. In 2017 International Conference on 3D Vision (3DV), pages 458-466. IEEE, 2017.
[10] A. Gopalakrishnan, A. Mali, D. Kifer, L. Giles, and A. G. Ororbia. A neural temporal model for human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12116-12125, 2019.
[11] L.-Y. Gui, Y.-X. Wang, X. Liang, and J. M. Moura. Adversarial geometry-aware human motion prediction. In Proceedings of the European conference on computer vision (ECCV), pages 786-803, 2018.
[12] L.-Y. Gui, Y.-X. Wang, D. Ramanan, and J. M. Moura. Few-shot human motion prediction via meta-learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 432-450, 2018.
[13] W. Guo, Y. Du, X. Shen, V. Lepetit, X. Alameda-Pineda, and F. Moreno-Noguer. Back to mlp: A simple baseline for human motion prediction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4809-4819, 2023.
[14] X. Guo and J. Choi. Human motion prediction via learning local structure representations and temporal dependencies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2580-2587, 2019.
[32] J. Martinez, M. J. Black, and J. Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2891-2900, 2017.
[33] A. Martínez-González, M. Villamizar, and J.-M. Odobez. Pose transformers (potr): Human motion prediction with non-autoregressive transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2276-2284, 2021.
[34] A. Mohamed, K. Qian, M. Elhoseiny, and C. Claudel. Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14424-14432, 2020.
[35] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[36] M. B. Ring. Child: A first step towards continual learning. Machine Learning, 28(1):77-104, 1997.
[37] H.-F. Sang, Z.-Z. Chen, and D.-K. He. Human motion prediction based on attention mechanism. Multimedia Tools and Applications, 79(9):5529-5544, 2020.
[38] H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
[39] T. Sofianos, A. Sampieri, L. Franco, and F. Galasso. Space-time-separable graph convolutional network for pose forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11209-11218, 2021.
[40] J. Sun, Z. Lin, X. Han, J.-F. Hu, J. Xu, and W.-S. Zheng. Action-guided 3d human motion prediction. Advances in Neural Information Processing Systems, 34:30169-30180, 2021.
[41] J. Sun, J. Xie, J.-F. Hu, Z. Lin, J. Lai, W. Zeng, and W.-s. Zheng. Predicting future instance segmentation with contextual pyramid convlstms. In Proceedings of the 27th acm international conference on multimedia, pages 2043-2051, 2019.
[42] J. Tang, J. Wang, and J.-F. Hu. Predicting human poses via recurrent attention network. Visual Intelligence, 1(1):18, Aug 2023.
[43] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[44] T. Von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference on computer vision (ECCV), pages 601-617, 2018.
[45] S. Xu, Y.-X. Wang, and L.-Y. Gui. Diverse human motion prediction guided by multi-level spatial-temporal anchors. In European Conference on Computer Vision, pages 251-269. Springer, 2022.