Temporal Continual Learning with Prior Compensation for Human Motion Prediction
Authors: Jianwei Tang
Jiangxin Sun
Xiaotong Lin
lifang zhang
Wei-Shi Zheng
Jian-Fang Hu
Venue: NeurIPS 2023 poster (2023)
Keywords: Human Motion Prediction; Temporal Continual Learning; Prior Compensation Factor

Abstract: Human Motion Prediction (HMP) aims to predict future poses at different moments according to past motion sequences. Previous approaches have treated the prediction of various moments equally, resulting in two main limitations: the learning of short-term predictions is hindered by the focus on long-term predictions, and the incorporation of prior information from past predictions into subsequent predictions is limited. In this paper, we introduce a novel multi-stage training framework called Temporal Continual Learning (TCL) to address the above challenges. To better preserve prior information, we introduce the Prior Compensation Factor (PCF). We incorporate it into the model training to compensate for the lost prior information. Furthermore, we derive a more reasonable optimization objective through theoretical derivation. It is important to note that our TCL framework can be easily integrated with different HMP backbone models and adapted to various datasets and applications. Extensive experiments on four HMP benchmark datasets demonstrate the effectiveness and flexibility of TCL. The code is available at https://github.com/hyqlat/TCL.

Temporal Continual Learning with Prior Compensation for Human Motion Prediction
Jianwei Tang Sun Yat-sen University tangj [ang]a70mai12. sysu. edu. en
Jiangxin Sun
Sun Yat-sen University sunjx50mai12 sysu. edu. cn
Xiaotong Lin
Lifang Zhang
Sun Yat-sen University 1inxt298mai12. sysu. edu. cn
Dongguan University of Technology 20170288dgut, edu. cn
Wei-Shi Zheng
Jian-Fang Hu"
Sun Yat-sen University wszheng@ieee.oo org
Sun Yat-sen University hujf5 sysu. edu. cn
Abstract
Human Motion Prediction (HMP) aims to predict future poses at different moments according to past motion sequences. Previous approaches have treated the predic- tion of various moments equally, resulting in two main limitations: the learning of short-term predictions is hindered by the focus on long-term predictions, and the incorporation of prior information from past predictions into subsequent predic- tions is limited. In this paper, we introduce novel multi-stage training framework called Temporal Continual Learning (TCL) to address the above challenges. To introduce the Prior Compensation Factor
better preserve prior information, (PCF). We incorporate it into the model training to compensate for the lost prior information. Furthermore, derive more reasonable optimization objective through theoretical derivation. It is important to note that our TCL framework be easily integrated with different HMP backbone models and adapted various datasets and applications. Extensive experiments on four HMP benchmark datasets demonstrate the effectiveness and flexibility of TCL. The code is available at
cttiib..ooomhhhaae www.commm////.eee Bonn/hyllat/TCII.
Introduction
Human Motion Prediction (HMP) aims to predict future poses at varied temporal moments based on the observed motion sequences. The accurate prediction of human motion plays vital role in many applications, such as autonomous driving, human- robot interaction, and security monitoring, enabling the anticipation and mitigation of risks. This task is challenging due its requirement for predicting multiple moments, including short-term predictions for the "near-future" and long-term predictions for the "far-future".
Previous approaches address this task by autoregressively forecasting using recurrent neural networks (RNNs) and transformer architectures [1 4 9112 26
37
40
42
parallelly generating all
frames with graph convolution networks (GCNs) 2
30
31
28 These methods employ
"Jian-Far Fang Hu is the corresponding author. Hu is also with Guangdong Province Key Laboratory of Information Security Technology, Guangzhou, China and Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

11
15
35
1
11
35
learned target
learned target
condition
short
condition
short
long
short only
short+long
11
15
35
learned target
Stage 1
condition
short
learned target
Stage
condition
long
(c) short then short-long
Figure 1: Illustration of three different prediction settings. "short+long" and "short only" represent optimizing prediction for the entire sequence and prediction only for the first 5 frames, respectively. "short then short+long" denotes training the entire sequence after pretraining for the first frames.
110.4
10.5
Valid ava. errer
110.1
10.2
.avg.
109.8
9.9
9.6
short+long only short
short+long
short>long
shorte
(a) Bhort-tomm prediction result
(b) Long-tomn prediction result
Figure 2: Preliminary experiment results of three different prediction settings (lower values indicate better performance). (a) shows the short-term prediction results (predicting the 2-nd frame), while (b) illustrates the long-term prediction results(predicting the 25-th frame).
the one-stage training strategy to directly train model that can predict both the short-term prediction and the long-term prediction. However, the long-term motion prediction more challenging since the future motion can vary greatly (i.e., the prediction space is large), which would increase the uncertainty and ambiguity future prediction. As the prediction length increases, the fitting of high-uncertainty long-term prediction will gradually dominate the learning process of the prediction model, which hinders the learning short-term predictions and further limits the full potential of leveraging the prior knowledge learned from short-term inputs to facilitate long-term predictions.
This motivates us to exploit proper training strategies to better learn and utilize the prior knowl- edge. We further investigate this by conducting preliminary experiments on the following settings "short+long", "short only", and "short then short+long", which are illustrated in Figure ! The results are summarized in Figure We observe that "short then short+long" outperforms "short+long" on long-term prediction, which implies that the knowledge learned in short-term prediction can serve as prior to facilitating the learning of "far-future" prediction. This is intuitive and thus we can use progressive learning approach, where the model is trained to predict increasing numbers frames over multiple training stages, e.g.. starting with 5 frames in the first stage and 10 frames in the second stage, and so on. However, we also observe that "short then short+long" performs worse than "short only" by considerable margin for short-term prediction, which demonstrates the joint learning of short-term and long-term prediction results in knowledge forgetting for short-term prediction.
To overcome these problems, we introduce the Prior Compensation Factor (PCF) into the multi-stage training method obtain sequential continuous learning framework, which is named Temporal Continual Learning (TCL). is multi-stage training framework that alleviates constraint posed by long-term prediction on short-term prediction and effectively utilize prior information from short-term prediction. Specifically, divide the future sequence into segments and divide the training process

---

into multiple stages accordingly. We incrementally increase the number of prediction segments, allowing us to leverage the prior knowledge acquired from earlier stages for predicting the subsequent ones. Upon completion of each stage's s training. the learned prior knowledge is saved in the model parameters. However, with the changing of optimization objective when switching stages, the prior knowledge gradually fades away. In order to overcome this forgetting problem of prior knowledge, we further introduce the PCF, which designed as a learnable variable. Then, we derive a more reasonable optimization objective for this regression problem through theoretical derivation.
The proposed training framework is flexible and can be easily integrated with various HMP backbones or adapted to different datasets. To validate the effectiveness and flexibility of our framework, we conduct experiments on four popular HMP benchmark datasets by integrating TCL with several HMP backbones. Our contributions can be summarized as follows:1) We identify certain limitations in existing HMP models and propose a novel multi-stage training strategy called Temporal Continual Learning to obtain more accurate motion prediction model. 2) We introduce Prior Compensation Factor to tackle the forgetting problem of prior knowledge, which can be learned jointly with the prediction model parameters. 3) We obtain an easily optimized and more reasonable objective function through theoretical derivation.
2
Related Work
Autoregressive prediction approaches for HMP. Motivated by natural language processing, many researchers have adopted esquence-to-sequee sequences HMP, which includes RNNs, Long Short-Term Memory Networks (LSTMs) [1]] and Transformer temporal aspect, while Jain et al. 18
models to exploit the temporal information of pose
43 For instance, ERD
combined LSTMs with an incoder-decode to model the proposed Structural-RNe to capture spatiotemporal features applied sequence- coceenceett--seeeee io-seuunnce architecture for modeling the
of human motion. Martinez et al. 33
human motion structure. Aksan et al. 1 used Transformer to autoregressively predict future poses. Sun al. [40] designed query-read process to retrieve some motion dynamics from the memory bank. Lucas et al. (27] proposed a GPT-like (35 autoregressive method to generate human poses. Tang al. [2] combined attention mechanism and LSTM to complete the human motion prediction task. However, autoregressive methods are difficult to train and suffer from error accumulation problem.
Parallel prediction approaches for HMP.
Some researchers employed parallel prediction methods 28 The works of (31 used GCN to encode
to address HMP problem [2] 6 7
23
25 30
feature or to decode it, which associates different joints' information. Mao er al. [I viewed a pose as fully connected graph and used GCN to extract hidden information between any pair of joints. Martinez et al. (33] devised a transformer-baser network to predict human poses. Sofianos el al. [39] proposed method to extract spatiotemporal features using GCNs. And Ma et al. (28) tried to achieve better prediction results using a progressive manner. Xu et al. [45 used multi-level anchors make diverse predictions.
Continual learning. Although Deep Neural Networks (DNNs) have demonstrated impressive performance on specific tasks, their limitations in handling diverse tasks hinder their broader applica- bility. Therefore, some researchers introduced the concept of Continual Learning (36) to DNNs to ensure that models retain the knowledge of previous tasks while learning new tasks. Kirkpatrick et al. [00 proposed the Elastic Weight Consolidation (EWC) method to overcome the catastrophic forgetting problem and improve the performance of multi-task problems. Shin al. (38] introduced a method that addresses catastrophic forgetting in sequential learning scenarios by using generative model to replay data from past tasks during the training of new tasks. It is important note that the traditional CL approaches do not account for temporal correlation and are unable to leverage data from previous tasks.
3 Method
The problem of human motion prediction involves predicting future motion sequences by utilizing previously observed motion sequences. Formally, let X L.TT. x,, XT,] E JxXTh denotes the observed motion sequence of length Th where X, indicates motion of time i, and ,T,+ E R JXDT, represents the motion sequence of
X.....
X +I,
[T *2
3

---

length TN that needs to be predicted. Note that J is the number of joints for each pose, and D is the dimension of coordinates.
be regarded as a composite task consisting of multiple sequential prediction tasks, which involves predicting the future poses at varied moments conditioned on the motion sequences observed in the past.
To accomplish this multiple sequential prediction task, we first model the HMP problem as solving the following optimization problem:
0- arg max P(XI,+1 X XZ,+2, X1, X2,
X2, 0).
Thus our target is to find the optimal model that maximizes Equation ① To achieve this, we propose a framework called Temporal Continual Learning. Specifically, we partition the entire prediction interval into several smaller segments and perform multi-stage training. We claim that this enables the utilization of prior information from previous segments as knowledge for predicting the subsequent segments. Further, as the optimization objective changes in each training stage, we find that the prior knowledge, i.e., information learned in previous training stages, will be forgotten to a certain degree. To mitigate this problem, introduce a Prior Compensation Factor and accordingly derive more reasonable optimization objective at each stage.
3.1 Multi-stage Training Process.
We initially decouple the future sequence into K segments with time boundaries Tz, Tzz:" Tzx where Tz = Th Tp- And we denote the prediction of segment as task Zk. which can be expressed as follows:
Task Z1
XITh XIa+1:T2,
Task Z2
X1Th. XTz, +1:T2z
Task Zk:
X1TT XT +1:T2x
To be specific, the target of task Z1 is to predict X TT,+ conditioned on X and task Zμ aims to predict XTy, +1.T%, with XIT7 as condition. Therefore, by leveraging bayesian formulation, optimization problem P(X TT,+ XT,
X X1 X..... .I, 0) can be formulated as:
P(Z Zx; 0) = P(Z Zi ... Zk-1; P Z2 -x 0) P(Z (2)
where 0 is model parameters to be learned. In the following, we denote "71 72 7z" as "Z1:1". Our target maximize Equation 2 which means finding optimal model accomplish tasks.
For the purpose of transferring the prior knowledge in preceding tasks to their subsequent prediction task, we progressively increase the number of tasks in temporal order and train them successively. More precisely, our training is decomposed into K stages. In each stage Sk, we leverage the optimal model parameters 0 trained the previous stage S to initialize the parameters 9, and then update it based on the prediction tasks Z1, Z2, ... Ze (maximizing P(Z 0)). Since the model is trained to optimize the prediction tasks - in training stage S the knowledge of tasks Z can be implicitly involved in its well-trained parameters 0; 0% Initializing 02 as 0; can exploit the prior knowledge learned in previous tasks to assist the prediction of the next task.
1-
3.2 Definition of Prior Compensation Factor.
With training different optimization objectives stage by stage, the prior knowledge provided by previous tasks can be effectively exploited to predict the subsequent task. However, the change the optimization objective in different training stages could also bring about the knowledge forgetting problem. To mitigate the problem, we introduce 2- +2, to estimate the extent of forgotten knowledge when utilizing prior knowledge from tasks Z1:k-1 predict task Zg, which refer to as the "Prior Compensation Factor".
aZ = P(Zz P(Z
(3)
Here, 21.-11 is regarded as the prior knowledge that is reserved and can be still provided for predicting task Zg. So 2 initially represents the prior knowledge reserved in 0% in every stage

---

Ss and would get somewhat corrupted gradually during training. (1:-111; indicates the most ideal case, where the current prediction task Z5 can fully leverage the prior information provided by previous prediction tasks 71:k-1- Consequently, the loss of the prior knowledge is non-negative,
implying that 0 S a2 I 1 P(Zx 1:k- 0).. Specifically, we can observe that 0 = P(Zx) 11 0), which implies that all the prior knowledge of previous
when 11
tasks is completely exploited although the 0 changes. By substituting Equation (3) into Equation 2
and taking the negative logarithm, we can obtain:
log P(Z 0) = log P(Z1:
Σ log 1og(P(Z(/21=11:0) +a azm--122,)
Our objective is to minimize -
log (Zi1: 0) with respect to the model parameter 0 and prior compensation factors (02 =233 k).
3.3 Optimization Objective
Optimizing Equation (4) directly is challenging due to the presence of the PCF and P inside the logarithm. By applying Lemma 3.1 (details provided in the appendix), we can obtain an upper bound for Equation (4), which can be expressed as:
UB = log P(Z1;0) Σ (11 0Z aZ (- -- log P(Z
(5)
+ +11 1-2,- log(1 a7 log(1 + azn--1-2,)),
Hence, we can turn to minimize the upper bound of Equation (4p. It appears that in the optimization
objective, 0 a 1-2 serves factor control the weights of different tasks, which mitigates the loss of prior information and thus compensates for the lost prior knowledge. The Lemma 3.2 indicates that the largest difference between log P(Zi: 0) and the upper bound UB would not
exceed log(3/2) * (k 1) when 2 1; 0) 2 1/2, i (2, ... .k.. kk)
Lemma 3.1. For 0 v a < 1 - b and 0 < b < 1, the inequality - log(a b) S (1 a) (og log (1 (1-a) a) log(1 a) + log(1 (1+ + a) holds. The equality holds and only if 0.
Lemma 3.2. The absolute difference between the target objective (Equation
4h and the upper bound
(Equation 55) is not larger than 13/(3/2) (k- 1) when P(Z: -1:-1;
1;
0)
2 1/2, i E 3,, ,.).
This bound is achieved when P(Zz 2 1: 0) 1/2 and az -2; 1/2,1 E (2.3. k).
Due to the space limitation, we present the proofs of Lemma 3.1 and 3.2 in the appendix.
An intuitive explanation. Figure 3 illustrates the comparison between the term + 02 in the actual optimization objective, the term - log P(Z 21x=1; in the naive optimization objective merely leveraging multi-stage training strategy, and the corresponding approximate term in our optimization objective. It can be observed that our approximation method has a smaller difference from the actual optimization objective compared to the naive method. This is important as a more reasonable objective function can improve the accuracy of the optimization process. When the prior compensation factor 4'7,. 1- is zero, which means P(Zz) Zi 1 0) = P(Zz [[:x|21:k-1;: 0), the prior prediction information from previous tasks Zi.k-1 is not lost. As the value of a increases, the discrepancy between the actual objective Z1 and the objective of the naive method P(Zx| 1 0) becomes more evident, indicating greater degree of forgetting prior knowledge. In contrast, our approach narrows this gap by effectively mitigating the loss of prior information obtained from tasks Z1:k-1-
- log
3.4 Optimization Strategy
We train the model in a multi-stage manner, in which an initial stage and K 11 TCL. stages are involved. The initial stage S, aims to forecast the motion in the foremost segment, while each TCL stage Sg, k (2,3, ... K performs prediction for segments Zi. and simultancously trains
5

---

-log 2i
-logP(Z
Naive objective
Our objective
Actual objective
P(Zx) 1:
Figure 3:
A toy example illustrating the optimization objectives. Our approximate optimization objective is closer the actual objective compared to the naive optimization method.
1- Once the stage Sy is completely trained, we then estimate the factors á 11t Zx which would be used in the optimization of the following stages. This process is repeated until the final stage SK is reached.
Learning of initial stage St. Following the implementations of previous methods 19 we can train the initial stage Z₁ with Mean Suqared Error (MSH) loss:
L1
X
(6)
(=
where X; and X, represent the ground truth and predicted motion of the i-th frames respectively.
Temporal Continual Learning at stage St. In stage S₂ (k 2), need update the model parameters O corresponding to tasks Z1. and the PCF a7 According to Equation (5) the loss function in this stage can be calculated as follows:
Tz
X 1|
L% =(1 0% 2-11--1-22 +22
X
+(1 02,2-1 log(1 - -2216-1-2o)
i=TTy
+1
(7)
k-1
T2 (1 02
+ log(1 + -2 1-
X, -x,-x(1' +Li
x1|²
721511177
1=2
where the parameters -Z, ... _ are dertermined in the learning of previous stages.
Once the model parameters for stage S₁ are determined, then calculate az
1
M
where M represents number of samples and a 07 PCF estimated for the m-th sample.
continue TCL training process by predicting stage Sk+1 and updating the model parameter 0 as well as PCF a' - We repeat this TCL process until we reach the final stage Sx- In practice, we require the backbone model output an extra dimension and pass it through an MI.P head to obtain a. The algorithm flow is summarized in Algorithm 1.

---

Algorithm 1 Training procedure of proposed TCL framework.
Require:
observed frames XIT,, ground truth future frames XT,,+ +17T, model parameters 0, stage number K, training epoch for k-th stage Eg, learning rate X, training sample number M. for i = to E do
+1T+
fg (1(11))))
0- 0 X VoLi (XT Xn.+1T2,)
end for A 0)
for to K do
for i = 1 to Eig do
X X1,+1:12 02 = (:
0 0 0 X VoL* (ITT+ XTu+1:T2 aZ a2 A)
end for
.Z 1 = 0
for m M do
OZ -he = fel (X I:T,,
aZ = 7Z +Zx + a
711t-11224
end for iZ -2
M 1Z
A = A U Uâzzz 1
end for
4 Experiments
4.1 Experimental Setup
We validate our framework four benchmark datasets. Human3.6M is large dataset that contains 3.6 million 3D human pose data. 15 types of actions performed by 7 actors(S1, S5, S6, S7, S8, S9 and S11) are included in this dataset. Each actor is represented by skeleton of 32 joints. However, following the data preprocessing method proposed in [28] 31, we only use 22 joints. The global rotations and translations of poses removed, and the frame rate is downsampled from 50 fps to 25 fps. For testing and validation, we use actors S5 and S1 1, while training is conducted on the remaining sections of the dataset. CMU-MoCap a smaller dataset that has 8 different action categories. The global rotations and translations of the poses are also removed. Each pose contains 38 joints, but following the data preprocessing methods in 28 3TI we only use 25 joints. 3DPW is challenging dataset that contains human motion data captured from both indoor and outdoor scenes. Poses in this dataset are represented only 23 of these joints are used, as the other three are redundant. The Archive of Motion Capture as Surface Shapes (AMASS) 29 dataset gathers existing mocap datasets. Following 39), we select 13 from those and take 8 for training, 4 for validation and 1 (BMI.rub) the test set. We consider forecasting the body joints only and discard those 4 static ones, leading to an 18-joint human pose.
3D space, with each pose containing 26 joints. However,
Following the benchmark protocols, we use the Mean Per Joint Position Error (MPJPE) in millimeters (ms) as our evaluation metric for 3D coordinate errors and Euler Angle Error (EAB) for Euler angle representations. The performance is better if this metric is smaller.
we set the input length to 10 frames and the predictive output to 25 frames for Human3.66 36M AMASS and CMU-Mocap datasets, respectively. For the 3DPW dataset, we predict 30 frames conditioned on the observation of the preceding 10 frames. We choose PGBIG as our backbone model by default. In order to learn the PCF, we add an extra dimension to the output of the backbone model and calculate PCF through an MLP network whose hidden dimension is set to 512. We partitioned the future sequences into three segments with lengths of 3, 9, and 13. The training process was conducted on an NVIDIA RTX 3090 GPU for 120 epochs, allocating 50, 90, and 120 epochs for each respective stage.
Implementation Details. Following 28
30
39
7

---

Table 2: Results on CMU-MoCap, 3DPW and AMASS.
Table I: Results on Human3.6M, the metric.
*Using EAE
dataset
CMU
dataset
H36M
ms
80
320
400
560
1000
ms
160
320
400
560
1000
R.S.
24.4
80.1
93.1
112.5
141.5
R.S.+Ours
21.4
72.7
85.6
105.6
136.
POTR'
0.235
0.581
0.990
1.143
1.362
1.826
LTD
9.3
33.0
40.9
55.8
86.2
POTR+Ours
0.233
0.549
0.923
1.056
1.290
1.746
LTD+Ours
9.1
31.0
38.5
53.1
84.9
PGBIG
7.6
29.0
36.6
50.9
80.1
R.S.
29.9
55.7
94.5
108.7
130.1
168.0
PGBIG+Ours
7.5
28.3
35.4
48.6
78.4
R.S.+Ours
28.0
52.8
91.3
105.6
127.3
166.3
dataset
3DPW
LTD
12.7
26.1
52.3
63.5
81.6
114.3
ms
200
400
600
800)
1000
LTD+Ours
23.0
48.2
59.3
77.5
111.2
R.S
99.3
129.7
142.9
161.1
1717
R.S+Ours
91.8
120.0
139.5
156.2
169.1
MM
12.7
26.4
53.4
65.0
83.6
117.6
LTD
35.6
67.8
90.6
106.9
117.8
MM+Ours
11.5
25.0
52.0
63.7
82.8
117.1
LTD+Ours
33.1
64.1
86.1
100.4
110.0
PGBIG
29.3
58.3
79.8
94.4
104.T
siMLPe
10.7
23.9
50.7
62.6
82.0
116.0
PGBIG+Ours
21.4
47.1
67.7
83.5
96.0
siMLPe+Ours
10.0
22.9
49.4
61.2
80.6
114.6
dataset
AMASS
PGBIG
10.3
22.7
47.4
58.5
76.9
110.3
ms
80
320
400
560
1000
STSGCN
11.4
37.2
43.8
53.8
69.7
PGBIG+Ours
9.4
21.3
45.7
56.8
75.4
108.8
STSGCN+OurS
10.9
36.7
42.8
52.4
68.1
4.2
Experimental Results
We apply our method on the following approaches on four benchmark datasets: Res.Sup. [32. LTD 31, POTR [33], STSGCN [39], MotionMixer(MMM) (3) siMLPe (13) and the current state-of-the-an art PGBIG (28). Res.Sup. is an RNN-based model. LTD, STSGCN and PGBIG are method. MotionMixer and siMLPe are MLP-based
GCN-based models. POTR is a transformer-base models. All these methods have released their code publicly. We employ their pre-trained models or re-train their models using the suggested hyper-ypramee pper-parameters for a fair comparison. And we also follow the metric they used evaluate the results.
Table presents experimental results of different backbone models before and after applying our training strategy on Human3.6M 3.6M dataset. Our frameworks outperform the corresponding backbone models by a considerable margin (more than 1.5). Specifically, when applying our strategy to PGBIG, we obtained a performance of 64.97, which is much better than the performance of the original PGBIG (66.52). Table shows the quantitative comparisons of prediction results on the CMU-MoCap. 3DPW and AMASS datasets in which proposed framework also achieves the best results. We can observe that the improvement in long-term prediction is greater compared to short-term prediction, indicating that the prior information provided by short-term prediction more crucial for challenging long-term prediction tasks. It is worth noticing that our framework outperforms the others by a significant margin on 3DPW dataset. Specifically, attribute this to our framework's 's ability to leverage prediction prior for the subsequent prediction in this challenging dataset.
strategy outperforms PGBIG by 8.9 (63.59 vs 72.49). We
4.3
Visualization
Predicting results. Figure provides provides some visualization examples of predicted motions, demon- strating that our framework achieves more accurate results. Specifically, Figure represents the action "directions", with the person maintaining results PGBIG exhibit a bent posture during long-term predictions, whereas our method can predict states closer to the ground truth position. While in Figure 4b the person first bends and then stands upright. The PGBIG maintains the bent posture throughout the long-term prediction and our method can accurately predict the changes in posture. It is evident that our method demonstrates improvement long-term prediction effectiveness.
upright position throughout the sequence. The
at different stages. As Figure displays, the value of a progressively increases with each training stage. When new task is added, the model focuses on the new objective without considering

---

PGBIG
Ours
(a) Predicting result of the action "directions".
GT
Ours
(b) Predicting result the action "takingphoto".
Figure 4: Some visualization results Human3.6M, dataset.
0.375
0.350
0.325
0.300
0.250
51
5.
5,
5r
80
Stage
Figure 5: a at different stages.
the preservation of prior information, resulting in the forgetting of prior knowledge. Here, we utilize mitigate the loss of prior information, thereby improving the overall training of the model.
4.4
Ablation Studies
We conduct several ablation experiments further verify the effectiveness of proposed framework.
Evaluation on the number of tasks. As shown in Table the model's performance improves as the number of tasks gets larger from to 3, and it remains stable when the number of tasks becomes larger than 3.
Table 3: The average error of different numbers of tasks.
number tasks
avg error
66.95
66.02
65.00
65.05
65.033
Evaluation on different implementations. As Figure Ssows,, shows, we compare the results of four different implementations. PGBIG is our baseline model. "w/o a" means that we only divide the training process into several stages to train each task without using the PCF. "HC" represents using a hand-crafted coefficient that changes its values similar to our PCF at each epoch. Specifically, in stage Si, the value of a is set to 1. In stage S2, is initially set to 0.1 and increased by 0.05 at each epoch until reaching 0.5, where remains constant. The same pattern applies to stage Sz. In each subfigure, we show the results of Zş the validation set which is obtained stage Sk.

---

10.5
result
46.5
Z, result
93.0
result
67.00
Avg result
Valld arror
10.0
46.0
Valld avg arror
92.5
66.55
45.5
9.5
66.00
85..5
44.5
8.5
PGBIG w/o
HC
Ours
44.0
PGBIG w/o
MC
Ours
91.0
PGBIG w/o
HC
Ours
65.0
PGBIG w/o
HC
Ours
Figure 6: Comparison of different approaches. PGBIG is baseline model that is trained without stati- "w/o a" represents training multi-stage process without PCE. "HC" means using hand-designed coefficient. "Ours" is a multi-stage training process with PCF.
In stage S1, where no prior information available, the results of "w/o a", "HC", and "Ours" are identical, but superior to the baseline. This validates the effectiveness of decomposing multiple- moment predictions, as it alleviates the constraint of long-term predictions on short-term predictions and enhances the model's ability to learn short-term predictions. In the following stages, the performance of "woo a" is worse than "Ours", which indicates that the prior knowledge exploited by our framework benefits the prediction model training. Moreover, the training period progressed, the performance gap becomes larger. However, the method without a can still achieve better performance than "PGBIG", indicating that training model in multi- stage manner can also exploit some useful prior information for prediction. We also note that the performance of "Ours" is much better than "HC", which conducts temporal continual learning with fixed and manually defined PCF. It demonstrates that joint training PCF and model parameters is beneficial.
The forgetting of prior knowledge. As shown in Table introducing the prior compensation factor alleviates the performance degradation from stage Si to stage S3 of task Z1's predictions. Specifically, without PCF, the prediction error of Z1 increases by 0.83, whereas with PCF, it only increases by 0.27. This result suggests that PCF can effectively alleviate the forgetting issue. As result, Zi can offer more comprehensive priors for Zo and Z3 predictions, resulting in better prediction performance.
Table 4: The average error of different tasks at the end of each stage. Z, represents task i.
(a) Without PCF.
(b) Using PCF
Z
Zz
Z3
Z1
Zz
Z3
S1
9.03
S
9.03
S2
9.44
45.33
S2
9.10
44.43
83
9.86
45.70
92.80
53
9.30
44.62
91.37
5
Conclusion
In this paper, introduced the temporal continual learning framework for addressing the challenges in human motion prediction. Our framework addresses the constraint between long-term and short- term prediction, allowing for better utilization of prior knowledge from short-term prediction to enhance the performance of long-term prediction. Additionally, we introduced the prior compensation factor mitigate the issue of forgetting information. Extensive experiments demonstrated our framework's 's effectiveness and flexibility.
Limitation. Our proposed training framework may slightly increase training time. However, the testing time remains unchanged compared to the backbone model.
Broader Impact. We believe our work has value for not only human motion prediction but also for more general prediction tasks and backhone models areas such as security monitoring, robotics, and autonomous driving.
34 This has benefits in various
10

---

Acknowledgments and Disclosure of Funding
This work was supported partially by the NSFC (U2IA20471, U22A2095, 62076260, 61772570), Guangdong Natural Science Funds Project 1512100515120085 202381515040055) Guangdong NSF for Distinguished Young Scholar (2022B151 and Guangzhou Science and Technology Plan Project (202201011134).
References
[1] E. Aksan, M. Kaufmann, P. Cao, and O. Hilliges. A spatio-temporal transformer for 3d human motion prediction. In 2021 International Conference on 3D Vision (3DV), pages 565-574, IEEE, 2021.
[2] E. Aksan, M. Kaufmann, and O. Hilliges. Structured prediction helps 3d human motion modelling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7144-7153, 2019.
[3] A. Bouazizi, A. Holzbock, U. Kressel, K. Dietmayer, and V. Belagiannis. Motionmixer: Mlp- hip-based 3d human body pose forecasting, arXiv preprint (XXi::22770009990 2022.
[4] H.-k. Chiu, E. Adeli, B. Wang, D.-A. Huang, and C. Niebles. Action-agnostic human pose forecasting. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1423-1433 IEEE, 2019.
[5] E. Corona, A. Pumarola, G. Alenya, and F. Moreno-Noguer Context-aware human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6992-7001, 2020.
[6] Q. Cui and H. Sun. Towards accurate 3d human motion prediction from incomplete observations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4801-4810, 2021.
[7] Q. Cui, H. Sun, and F. Yang. Learning dynamic relationships for 3d human motion prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6519 0027, 2020.
[8] K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE international conference computer vision, pages 4354, 2015.
[9] P. Ghosh, J. Song. E. Aksan, and O. Hilliges. Learning human motion models for long-term predictions. 2017 International Conference on 3D Vision (3DV), pages 458 188.466. IEEE, 2017.
[10] A. Gopalakrishnan, A. Mali, D. Kifer, L. Giles, and A. G. Ororbia. A neural temporal model for human motion prediction. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 121 121 12-1225,, 2019.
[11] L.-Y. Gui, Y.-X. Wang, X. Liang, and J. M. Moura. Adversarial geometry-aware human motion prediction. In Proceedings of the european conference on computer vision (ECCV), pages 786-803 2018.
[12] L.-Y. Gui, Y.-X. Wang, D. Ramanan, and J. M. Moura. Few-shot human motion prediction via meta-learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 432 450, 2018
[13] W. Guo, Y. Du. X. Shen, V. Lepetit, X. Hameeda-Pinedas and F. Moreno-Nogune Back to mip: A simple baseline for human motion prediction. In Proceedings the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 4809-4819, 2023.
[14] X. Guo and J. Choi. Human motion prediction via learning local structure representations and temporal dependencies. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2580-2587, 2019.
11

---

[15] S. Hochreiter and J. Schmidhuber, Long short-term memory. Neural computation, 98):17335 1780, 1997.
[16] J.-F. Hu, J. Sun, Z. Lin, J.-H. Lai, W. Zeng. and W.-S. Zheng. Apanet: Auto-path aggregation for future instance segmentation prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(7):3386-3403, 2021.
[17] C. lonescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3, 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-133) 2013.
[18] A. Jain, A. R. Zamir, S. Savarese, and A. Saxena. Structural-rmn: Deep learning on spatio- temporal graphs. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 5308-5300 5318-537,777 2016.
[19] A. Kendall, Y.
Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 7482-7491, 2018.
[20] J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. RUsu, K. Milan, et al. Overcoming catastrophic forgetting in
J. Quan, T. Ramalho, A. neural networks. Proceedings of the national academy of sciences, 114(13):3521-355 2017.
[21] Lebailly, S. Kiciroglu, M. Salzmann, P. Fua, and W. Wang. Motion prediction using temporal inception module. In Proceedings of the Asian Conference on Computer Vision, 2020.
[22] M. Li, S. Chen, X. Chen, Y. Zhang, Y. Wang, and Q. Tian. Symbiotic graph neural networks for 3d skeleton- based human action recognition and motion prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(6))3316 (166)3316--333, 2021.
[23] M. Li, S. Chen, Y. Zhao, Y. Zhang, Y. Wang. and Q. Tian. Dynamic multiscale graph neural networks for 3d skeleton based human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 214-223, 2020.
[24] Z. Lin, J. Sun, J.-F. Hu, Q. Yu, J.-H. Lai, and W.-S. Zheng. Predictive feature learning for future segmentation prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7365-7374, 2021.
[25] J. Liu and J. Yin. Multi-grained trajectory graph convolutional networks for habit-unrelated human motion prediction. arXiv preprint arXiv:2012.12556 2020.
[26] Z. Liu, S. Wu, S. Jin, Q. Liu, S. Lu, R. Zimmermann, and L. Cheng. Towards natural and accurate future motion prediction of humans and animals. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10004-10012, 2019.
[27] T. Lucas, F. Baradel, P. Weinzaepfel, and G. Rogez. Posegpt: Quantization- baastd 3d human motion generation and forecasting. In European Conference on Computer Vision, pages 417-435. Springer, 2022.
[28] T. Ma. Y. Nie, C. Long, Q. Zhang, and G. Li. Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6437-6446, 2022.
[29] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black. Amass: Archive of motion capture surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5442-544 5451, 2019.
[30] W. Mao, M. Liu, and M. Salzmann. History repeats itself: Human motion prediction via motion attention. In European Conference on Computer Vision, pages 4744899 Springer, 2020.
[31] W. Mao, M. Liu, M. Salzmann, and H. Li. Learning trajectory dependencies for human motion prediction, Proceedings of the IFFF/CVE International Conference on Computer Vision, pages 9489-9497, 2019.
12

---

[32] J. Marttinez M. J. Black, and J. Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 2891-2900, 2017.
[33] A. Maríinez-González, M. Villamizar, and J.-M. Odobez. Pose transformers (potr): Human motion prediction with non-autoregreessvve transformers. In Proceedings of the IEEE/CVF International Conference Computer Viston, pages 2276-22665 2021.
[34] A. Mohamed, K. Qian, M. Elhoseiny, and C. Claudel. Social-stgenn: A social spatio-tempone graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 14424-14432, 2020.
[35] A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training, 2018.
[36] M. B. Ring. Child: A first step towards continual learning, Machine Learning, 28(1):77-104, 1997.
[37] H.-F. Sang, Z.-Z. Chen, and D.-K. He. Human motion prediction based on attention mechanism. Multimedia Tools and Applications, 19(9):5529-554) 2020.
[38] H. Shin, K. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017.
[39] T. Sofianos, A. Sampieri, L. Franco, and F. Galasso. Space-time-scparable graph convolutional network for pose forecasting. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1209-1111 1 1218, 2021.
[40] J. Sun, Z. Lin, X. Han, J.-F. Hu, J. Xu, and W.-S. Zheng. Action-guided 3d human motion prediction. Advances in Neural Information Processing Systems, 34:30169-30180, 2021.
[41] Sun, J. Xie, J.-F. Hu, Z. Lin, J. Lai, W. Zeng, and W.-s. Zheng. Predicting future instance segmentation with contextual pyramid convistms. In Proceedings of the 27th acm international conference on multimedia, pages 2043-2051, 2019.
[42] J. Tang, J. Wang, and J.-F. Hu. Predicting human poses via recurrent attention network. Visual Intelligence, (1)::88, Aug 2023.
[43] A. Vaswani, N. Shazeer, N. Parmar, Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[44] T. Von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and moving camera. In Proceedings of the European conference on computer vision (ECCV), pages 601-617, 2018.
[45] S. Xu, Y.-X. Wang, and L.-Y. Gui. Diverse human motion prediction guided by multi-level spatial-temporal anchors. In European Conference on Computer Vision, pages 251-269 Springer, 2022.
13


Reviews:
[{"review_id": "bKuaMNsFLv", "reviewer": "Reviewer_skai", "summary": "The paper addresses the common trade-off between the short- and long-term prediction quality of 3D human motion prediction models. The proposed training technique improves the performance of the underlying models both on short- and long-term prediction horizons, where the models generally prioritize one and suffer from the other. It is a multi-stage training scheme resembling curriculum learning with an increasing prediction horizon. The prediction horizon is split into consecutive chunks of frames (i.e., stages). The model is trained iteratively for every stage by initializing the weights from the previous stage and considering all the stages so far. Different from curriculum learning, the proposed technique aims to preserve and leverage the prior information of the past stages by alleviating catastrophic forgetting in the future stages. To do so, the paper incorporates the \"Prior Compensation Factor\" in the training objective. Intuitively, for every training sample, the model predicts a weight parameter that dynamically distributes the total loss weight across stages. The proposed technique is evaluated using three different architectures. Experiments on various benchmarks show that it is highly effective.", "questions": "I’ve raised my concerns and asked my questions in the weaknesses section.", "limitations": "The paper does not have a limitations section. I think it is clear that the proposed technique significantly increases training complexity. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**originality**\nThe paper addresses a common and often neglected problem in the 3D human motion prediction domain. In fact, most prior works prefer to ignore the short- and long-term prediction trade-off and present results with separate model configurations. I find this effort valuable.\n\n**quality**\nThe proposed technique is well-motivated and sound. I have some concerns regarding the theoretical analysis (see below), but the experimental results are solid.\n\n**clarity**\nThe motivation is clear. The paper provides enough background to understand the problem setting. However, it is not straightforward to intuitively understand what the proposed technique does.\n\n**significance**\nI find the paper interesting and useful for the community. The experiments show that the proposed technique could improve the performance of the out-of-the-box models.\n", "weaknesses": "There is a potential problem with lemma 3.1. It expects `b` to be between 0 and 1, a likelihood value from a probability mass function. For the proposed training objective (Eq. 7) to be held, the loss functions should be chosen carefully. In this work, it is implemented as a MSE loss (Eq. 6 and 7), which does not have an upper-bound. How about implementing the objective as a log-likelihood with a Gaussian output model? It may have negative values. \n\nI had to spend some time getting an intuitive understanding of Eq. 7 (no, the “An intuitive explanation” section does not help). It boils down to a dynamic weighting scheme where the network learns to distribute the loss weights across prediction horizons (i.e. stages). I think this could be explained better in the paper. In fact, the training dynamics is very straightforward. If the MSE loss is high, then the predicted alpha should be higher to optimize the objective. We can say that the model learns to “assess the difficulty of the task”. Naturally, alpha values get higher in the future stages as it becomes more difficult. If you run the following code snippet, you get a very similar plot to the one in Fig. 4 ($\\alpha$ values at different stages). It gives you the optimal $\\alpha$ value for different MSE values. I kindly ask the authors to clarify if I am wrong or share their thoughts on this analysis.\n\n```\nimport matplotlib.pyplot as plt\nfrom scipy.optimize import minimize_scalar\nimport math\nimport numpy as np\n\ndef objective(mse):\n    return lambda alpha: (1 - alpha) * mse + (1-alpha) * math.log(1-alpha) + math.log(1 + alpha)\n\nmse_values = np.arange(0, .5, 0.01)\nbest_alphas = []\nbest_res = []\nfor mse in mse_values:\n    opt_res = minimize_scalar(objective(mse), bounds=(0, 1), method=\"bounded\")\n    best_alphas.append(opt_res.x)\n    best_res.append(opt_res.fun)\n\nplt.plot(mse_values, best_alphas)\nplt.xlabel(\"MSE values (i.e., stages)\")\nplt.ylabel(\"Alpha values\")\nplt.show()\n```\nThe authors can also try setting the $\\alpha$ values for the “HC” ablation using this code snippet.\n\nAn ablation on the number of stages is missing. Similarly, its cost is not discussed. While it improves the performance of the underlying model, it introduces a significant overhead at training time. \n\nIt would be interesting to see the evolution of the performances on the previous stages as training continues on the future stages. A triangular matrix reporting the performance of stage K after training every stage > K.\n\nInstead of following a dynamic approach (i.e., predicting an $\\alpha$ per training sample), would it still work if a single, stage-wise $\\alpha$ variable was trained? \n\nThis is merely a suggestion for the presentation. I think it would be more interesting to report the performance for the pairs (baseline, baseline + ours) in all benchmarks (as in Table 5). \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the constructive comments. We have carefully addressed your concerns and provided detailed responses for each review.\n\n**Q1:** **There is a potential problem with lemma 3.1. It expects b to be between 0 and 1, a likelihood value from a probability mass function. For the proposed training objective (Eq. 7) to be held, the loss functions should be chosen carefully. In this work, it is implemented as a MSE loss (Eq. 6 and 7), which does not have an upper-bound. How about implementing the objective as a log-likelihood with a Gaussian output model? It may have negative values.**\n\nRe: We would like to clarify that in our implementation, the objective is formulated as log-likelihood with a Gaussian output, in which the model output is defined by a Gaussian distribution with value from 0 to 1. Thus, the requirements of lemma 3.1 are explicitly satisfied. Then, we take the negative logarithm of the Gaussian distribution and derive the MSE loss, whose value range between [0, +∞). \n\n**Q2: α** **values get higher in the future stages as it becomes more difficult. It's like the model is learning to evaluate task difficulty. I kindly ask the authors to clarify if I am wrong or share their thoughts on this analysis.** \n\nRe: Our perspective on this differs slightly from yours. Rather than evaluating difficulty of the current task itself, our proposed strategy estimates the extent of prior information loss, which can also be viewed as estimating the difficulty of transferring prior information from previous tasks to the current one. Our motivation is to effectively utilize the prior knowledge acquired from past tasks. To achieve this, we employ a multi-stage training strategy and introduce a prior compensation factor to estimate the extent of information loss.It is worth noting that the multi-stage training method allows us to estimate the degree of prior information loss. This estimation can also be seen as measuring the difficulty of transferring prior knowledge of previous tasks to the current one. In contrast, evaluating task difficulty to assign varying task weights can be achieved in one stage. This is a significant difference.\n\n**Q3:** **The authors can also try setting the α values for the “HC” ablation using this code snippet.** \n\nRe: Thank you for your suggestion. We have conducted an experiment, referred to as \"HC2\", using the α values calculated with the code you provided to validate this viewpoint. The results are shown in Table 1, from which we can obtain two observations. First, the analytical α calculated by “HC2” also helps to mitigate the loss of prior information. Second, our learning strategy performs better than “HC2”. It is because that in our model learning process, α is treated as a distribution that estimates the loss of prior information for previous stages. Based on the α definition, given that the model's output is inherently a distribution, the α should be considered as a random variable satisfying a certain distribution as well. Therefore, using the model to estimate the distribution of α is a more reasonable way. \n\n**Q4: An ablation on the number of stages is missing.** \n\nRe: Following your suggestion, we have conducted experiments with different numbers of stages, and the results are presented in Table 2. As can be seen, the model's performance improves as the number of stages gets larger from 1 to 3, and it remains stable when the number of stages becomes larger than 3.\n\n**Q5: It would be interesting to see the evolution of the performances on the previous stages as training continues on the future stages. A triangular matrix reporting the performance of stage K after training every stage > K.**\n\nRe: Based on your suggestion, we conducted experiments on the Human3.6M dataset, as shown in the Table 3 (lower values indicate smaller errors). It can be observed that as the training continues, the performances of previous prediction tasks slightly decrease. Taking Z1 task for example, the performances decrease from 9.03 to 9.30.\n\n**Q6: Instead of following a dynamic approach (i.e., predicting an α per training sample), would it still work if a single, stage-wise α variable was trained?**\n\nRe: We conduct experiments based on your suggestion, using the code you provided to calculate the average α value as stage-wise α. The results are shown in the Table 4. It shows that the stage-wise α helps in mitigating the loss of prior information and aiding the model's training process. However, this approach isn't as effective as our strategy, largely because the model approximates the loss of prior information at each stage in a coarse-grained manner, leading to imprecise control over the training process.\n\n**Q7: I think it would be more interesting to report the performance for the pairs (baseline, baseline + ours) in all benchmarks (as in Table 5).** \n\nRe: Thank you for your suggestion. We will report the performances in the form of **baseline and baseline+ours** **pairs**. \n\n**Q8：Its cost** **is not discussed. While it improves the performance of the underlying model, it introduces a significant overhead at training time.**\n\nRe: Thank you for your comments. We would like to point out that our multi-stage training approach does not lead to a significant increase in the overall training time as compared with the single stage training approaches (16h vs. 19h). The detailed training time on Human3.6M dataset can be found in the Table 5. In practice, training models for short-term predictions is easier and fast to converge. Training long-term predictions in our multi-stage training strategy also converge fast since the model trained for earlier stages are employed as pretrain model to provide prior information. We will include the discussion in the revision."}, {"review_id": "uc9vJWAWIe", "reviewer": "Reviewer_hcCQ", "summary": "This paper aims to enhance human motion prediction. The main contributions of this paper are:\n1. The paper presents a multi-stage training strategy named Temporal Continual Learning to incorporate the learning of both short-term prediction and long-term prediction.\n2. The paper introduces Prior Compensation Factor to better preserve prior information during the process of Temporal Continual Learning.\nThese optimizations are given through theoretical derivation.\n", "questions": "1. typo in L119, should be T_{Z_{K-1}}\n2. Figure 4 in the ablation study is a bit confusing. What is ablated here? Is the alpha value the same across different experiments? \n3. The implementation defines a specific partition of sequence for the multi-task. I think it would be interesting to see the ablation on different partitions, e.g. different number of sub-tasks\n", "limitations": "It would be helpful if the authors could provide videos to demonstrate the quality of generated human motion. I wonder if the multi-task learning for different motion segments will cause the discontinuity", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper has several strengths:\n1. Overall, the paper is well-written with a clear and well-motivated introduction. The storyline to leverage the prior knowledge learned from short-term inputs to facilitate long-term predictions makes sense.\n2. Secondly, the proposed method is flexible and demonstrates good performance when applied to different human motion predictors, outperforming state-of-the-art on different datasets.\n", "weaknesses": "The paper could benefit from a more thorough discussion of related work on Human Pose Prediction, such as in L23-24, transformer architectures include\n- PoseGPT: Quantization-Based 3D Human Motion Generation and Forecasting. ECCV 2022\n\n\tand graph convolution networks include\n\n- Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchors. ECCV 2022\n\nThe key insight is supported by L32-42 and Figure 1. I'm wondering if the comparison can be represented by a clearer illustration, e.g. using figures to explain different terms such as “short+long” and “short then short + long”. The current version looks a bit rough.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the constructive comments. We have carefully addressed your concerns and provided detailed responses for each review.\n\n**Q1:** **The paper could benefit from a more thorough discussion of related work on Human Pose Prediction, such as in L23-24, transformer architectures.**\n\nRe: Thank you for the suggestion. Some works attempt to develop human motion prediction models based on Transformer architectures. For instance, PoseGPT [1] focus on generating diverse future poses with GPT-like model. [2] introduces spatial-temporal anchor-based sampling to generate diverse human poses. We will include these works and provide a more thorough discussion in the revision.\n\n**Q2:** **The key insight is supported by L32-42 and Figure 1. I'm wondering if the comparison can be represented by a clearer illustration, e.g. using figures to explain different terms such as “short+long” and “short then short + long”. The current version looks a bit rough.**\n\nRe: \"short+long\" means that the model are trained for both short-term and long-term predictions together. \"short only\" indicates that the model is only trained for short-term predictions without considering long-term predictions. \"short then short+long\" means that after training the model for short-term prediction, the model is further trained by combining both short-term and long-term predictions together. We will follow your suggestion and provide a figure to illustrate the differences of these terms.\n\n**Q3:** **typo in L119, should be T_{Z_{K-1}}**\n\nRe: Thank you. We will correct the writing error.\n\n**Q4:** **Figure 4 in the ablation study is a bit confusing. What is ablated here? Is the alpha value the same across different experiments?**\n\nRe: The purpose of the figure is to visualize the average value of α for each stage. As can be seen, as the training stages gets larger, more prior information loss can be observed, and our multi-stage training strategy can effectively reduce the information loss. Since alpha is learned in the data-driven manner, its value could vary in different experiments. \n\n**Q5:** **The implementation defines a specific partition of sequence for the multi-task. I think it would be interesting to see the ablation on different partitions, e.g. different number of sub-tasks.**\n\nRe: Thank you for your suggestion. We have conducted experiments with different numbers of tasks, and the results are shown in the following table. As can be observed, the model's performance improves as the number of tasks gets larger from 1 to 3, and it remains stable when the number of tasks becomes larger than 3.\n\n| **number of tasks** | **1** | **2** | **3** | **5** | **8** |\n| ------------------- | ----- | ----- | ----- | ----- | ----- |\n| **Avg_err⬇**        | 66.95 | 66.02 | 65.00 | 65.05 | 65.03 |\n\n**Q6:** **It would be helpful if the authors could provide videos to demonstrate the quality of generated human motion. I wonder if the multi-task learning for different motion segments will cause the discontinuity.**\n\nRe: In the proposed multi-stage learning strategy, the training process does not change the way of model prediction (parallel or auto-regressive). We only changed the way of determining parameters of the model. As a result, we did not break the continuity of the results generated by the original model. We also examine the generated motion sequence and find that it is visually continuous. And we have provided a link of our demo to AC.\n\n**reference:**\n\n[1] Lucas, T., Baradel, F., Weinzaepfel, P., & Rogez, G. (2022, October). Posegpt: Quantization-based 3d human motion generation and forecasting. In European Conference on Computer Vision (pp. 417-435). Cham: Springer Nature Switzerland.\n\n[2] Xu, S., Wang, Y. X., & Gui, L. Y. (2022, October). Diverse human motion prediction guided by multi-level spatial-temporal anchors. In European Conference on Computer Vision (pp. 251-269). Cham: Springer Nature Switzerland."}, {"review_id": "hwB4WP8ZdR", "reviewer": "Reviewer_GqjD", "summary": "This paper proposes to train human motion prediction networks by gradually increasing the prediction horizon.\nThis encourages the network to learn short-term predictions first and then leverage the learned to predict longer horizons.\nThe easy-to-hard curriculum makes the network learn more efficiently, as evidenced by the comparison to networks that learn all horizons at the same time.\nGiven the continual setting, forgetting prevention is needed and dealt with by the introduced prior compensation factors.\nExperiments are carried out on three major benchmarks, and effectiveness is observed compared with selected baselines.", "questions": "1) In lines 114-117, all predictions are conditioned on X_1:T_h, however, in Eq. 2, the conditions consist of previous predictions.\n2) Eq. 3 assumes that the actual distribution is a degenerated distribution plus a positive bias, why is the bias always positive? This seems not reasonable as the degenerated one could be larger than the actual one.\n3) Eq. 4 shows that under the bias assumption, the final loss can be treated as a linear combination of the loss terms for different horizons, however, in Eq. 5 the weights for each term is changed from \\alpha to 1-\\alpha, is this really valid?\n4) Are the terms involving only \\alphas really optimized by Eq. 8?\n5) Not sure whether the presentation/derivation is really necessary given that a lot of approximation is needed, yet what we want is just a loss that weights the prediction error at different horizons, hopefully the adaptiveness is the key?", "limitations": "The limitation is addressed in the broader impact section.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written. The idea of training sequence prediction with gradually increased horizons is well instantiated in the context of human motion prediction.\nThe motivation is also clearly conveyed by the ablation in Figure 1.\nBetter performance is achieved when compared with two recent methods.\nIt is good that some derivation is shown to arrive at the final combined loss of predictions at different horizons.\nThe adaptive scheme from the derivation shows better performance than a hand-crafted fixed set of weights.", "weaknesses": "The comparison is a bit weak. Shall compare with more recent methods, for example, \"Back to MLP: A Simple Baseline for Human Motion Prediction, 2023.\"\nAlso, since the method proposed is not backbone-dependent, so more evaluation is needed, for example, transformer-based architectures.\nMoreover, there are some questionable parts in the derivation, even though it seems that the final result may not be heavily affected.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Part 1 (Part 2 is in global rebuttal)**\n\nThanks to the reviewer for the constructive comments. We have carefully addressed your concerns and provided detailed responses for each review.\n\n**Q1: The comparison is a bit weak. Shall compare with more recent methods, for example, \"Back to MLP: A Simple Baseline for Human Motion Prediction, 2023.\"** \n\nRe: The mentioned approach is a very recent work, and considering your suggestion, we have also conducted experiments with the Human3.6M dataset on this model. The experimental results are shown in the Table 6. It can be observed that our proposed training strategy consistently improves the performances of corresponding models. The average error of the baseline **siMLPe** **[1]** model is 68.76. When training it with our multi-stage strategy, the average error reduces to 67.57. This also verifies that our approach can be flexibly applied to various backbone models for human pose prediction, enhancing their performance for human motion prediction.\n\n**Q2: Since the method proposed is not backbone-dependent, so more evaluation is needed, for example, transformer-based architectures.**\n\nRe: Yes. Our training strategy is not dependent on the backbone and can be flexibly applied to other models. In the manuscript, we have tested it with three different backbones: PGBIG [3] (GCN-based), LTD [4] (GCN-based), and Res. Sup. [5] (LSTM-based). Following your suggestion, we also included a transformer-based backbone and tested it on the Human3.6M dataset. Specifically, we used the POTR [2] as our backbone model, which leverages a transformer as the primary framework for parallel prediction manner. The same as [2], we used Euler Angle Error as the evaluation metric. The experimental results are presented in the Table 7. As expected, our method shows consistent improvements with this backbone as well, further validating that our proposed training strategy is quite flexible and effective to improve different prediction models.\n\n**Q3:There are some questionable parts in the derivation, even though it seems that the final result may not be heavily affected.**\n\nRe: Thank you for the suggestion. We have thoroughly validated our approach both theoretically and experimentally. We will carefully examine the details of the derivations to ensure the accuracy of our conclusions.\n\n**Q4:In lines 114-117, all predictions are conditioned on X_1:T_h, however, in Eq. 2, the conditions consist of previous predictions.**\n\nRe: Actually, the conditions in lines 114-117 and Eq.2 have different meanings. Specifically, in line 114-117, we define the prediction task $Z_k$ as predicting segment $k$, i.e., $X_{T_{Z_{k-1}}+1:T_{Z_k}}$, conditioned on the history $X_{1:T_h}$. While in Eq.2, the probability of $Z_k$ conditioned on $Z_1, …, Z_{k-1}$ suggests exploiting the beneficial knowledge of previous tasks $Z_1, …, Z_{k-1}$ to predict task $Z_k$. We will improve the writing to ensure a more concise and understandable presentation.\n\n**Q5:Eq. 3 assumes that the actual distribution is a degenerated distribution plus a positive bias, why is the bias always positive? This seems not reasonable as the degenerated one could be larger than the actual one.**\n\nRe: Indeed, this bias is always positive. The $P(Z_{k}|Z_{1}Z_{2}\\cdots Z_{k-1}\\theta)$ represents the most ideal scenario, where the current prediction task can fully leverage the prior information provided by previous prediction tasks. While $P(Z_{k}|\\hat{Z}_{1:k-1}\\theta)$ indicates completing the current prediction task with incomplete prior information. Hence, utilizing complete predictive priors would yield more accurate predictions than using incomplete information. \n\n**Q6: In Eq. 5 the weights for each term is changed from \\alpha to 1-\\alpha, is this really valid?**\n\nRe: Yes, it is valid. We have double checked the derivations presented in the appendix material.\n\n**Q7: Are the terms involving only \\alphas really optimized by Eq. 8?**\n\nRe: We actually optimize $\\alpha$s by leveraging Eq.7 as the loss function. Eq.8 is used to estimate the $\\hat{\\alpha}_{Z_{1:k-1}\\rightarrow Z_{k}}$  involved in Eq. 7 for the training in subsequent stages. The detailed training process can be found in Algorithm 1.\n\n**Q8: Not sure whether the presentation/derivation is really necessary given that a lot of approximation is needed, yet what we want is just a loss that weights the prediction error at different horizons, hopefully the adaptiveness is the key?**\n\nRe: Thank you for your insightful comments. The derivation is necessary. Aiming to compensate for the loss of prior knowledge when switching stages, we introduce α and obtain Eq. 3. Only by relying on Eq.3 can we derive the final objective function with the form of Eq.5. Although the objective function appears to be dynamic weighting control, the underlying mechanism is derived from rigorous theoretical analysis. Specifically, our strategy promote the model training process by estimating the extent of prior information loss. This estimation can only be achieved through a multi-stage training process. In contrast, the assignment of different task weights can be accomplished through a one-stage training approach. Moreover, arbitrarily designed dynamic weight control often fails to improve the training process effectively.\n\nTo validate the effectiveness of our derived objective function, we also conducted an experiment using a simple dynamic weighting approach in one-stage training manner. The results demonstrate that our strategy achieves better performance, which is shown in Table 8.\n"}, {"review_id": "WKP9BIrOoj", "reviewer": "Reviewer_vWPd", "summary": "This paper introduces the continual learning insight into human motion prediction. By analysis of the performance relationship between the short and long-term prediction, a compensatory method is proposed in a multi-stage learning setting. On several widely-used benchmarks, the proposed method is cooperated with different backbones and methods and compared with previous methods. Some decent progress is shown according to the analyses.", "questions": "1. Though there are some \"traditional\" settings of the length definition of the long and short terms. I wonder if we change the 5 and 15 frames setting, is the analysis of Fig. 1 kept or changed? \n\n2. Learning rate in Eq. 9: choice discussion.", "limitations": "Please add a discussion.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ Splitting the long-term prediction into multi-stage and using the continuous learning insight is interesting and non-trivial.\n\n+ The method proposal looks sound and designed well.\n\n+ A good comparison and discussion are given to support the proposed method.", "weaknesses": "- What is the additional cost of using the proposed method? Please discuss the efficiency and the other possible cost.\n\n- Lacking a vivid figure to illustrate the whole method pipeline before the detailed method introduction. Besides, the introduction part, especially the method description can be organized better and more logically.\n\n- L128-130: Though embedding the “knowledge” into the trained parameters seems very reasonable, this is just an intuitive discussion. A more detailed and clear explanation is essential to illustrate what is knowledge really, or what sign we can get to observe the knowledge utilized or not, to avoid a metaphysical or empirical discussion only.\n\n- Fig. 3: hard to read and discover the difference between methods.\n\n- Lacking direct and clear experiments to show the effectiveness of the continual learning design, e.g., better avoidance of forgetting, better balance of the shot and long predictions, etc.\n\n- typo: Fig. 1: lacking space before the (", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks to the reviewer for the constructive comments. We have carefully addressed your concerns and provided detailed responses for each review.\n\n**Q1:What is the additional cost of using the proposed method? Please discuss the efficiency and the other possible cost.**\n\nRe: The extra cost only occurs during training. However, the testing time is exactly the same as that of the baseline model. In practice, training models for short-term predictions is easier and fast to converge. Training long-term predictions in our multi-stage training strategy also converge fast since the model trained for earlier stages are employed as pretrained model to provide prior information. We will include the discussion in the revision.\n\n|                   | **stage 1** | **stage 2** | **stage 3** | **total** |\n| ----------------- | ----------- | ----------- | ----------- | --------- |\n| **Baseline**      | 16h         | -           | -           | 16h       |\n| **Baseline+Ours** | 8h          | 6h          | 5h          | 19h       |\n\n**Q2:Lacking a vivid figure to illustrate the whole method pipeline before the detailed method introduction. Besides, the introduction part, especially the method description can be organized better and more logically.**\n\nRe: Thank you for your suggestion. We will provide a figure to illustrate our pipeline and improve our presentation.\n\n**Q3:A more detailed and clear explanation is essential to illustrate what is knowledge really, or what sign we can get to observe the knowledge utilized or not.**\n\nRe: The knowledge refers to the information involved in the model, which is exploited not only for completing the current prediction task but also enhancing the performance of subsequent tasks. We can assess whether this knowledge has been utilized through the accuracy of long-term predictions. More details can be found in Q5.\n\n**Q4:Fig. 3: hard to read and discover the difference between methods.**\n\nRe: In Fig. 3, the upper image represents the actions of directions, with the person maintaining an upright position throughout the sequence. The results of PGBIG exhibits a bent posture during long-term predictions, whereas our method can predict states closer to the ground truth (GT) position. While in the lower image, the person first bends and then stands upright again. The PGBIG method maintains the bent posture throughout the long-term prediction and our method can accurately predict the changes in posture. It is evident that our method demonstrates a improvement in long-term prediction effectiveness. Thanks for your suggestion and we will highlight the major differences in visualization results among different methods. \n\n**Q5:Lacking direct and clear experiments to show the effectiveness of the continual learning design, e.g., better avoidance of forgetting, better balance of the short and long predictions, etc.**\t\n\nRe: Following your suggestion, we have conducted experiments to evaluate the evolution of the performances on the previous stages as training continues on the future stages on the Human3.6M dataset. The results are shown in the table below, where lower values indicate more accurate prediction. As shown, introducing prior compensation factor alleviates the performance degradation from stage 1 to stage 3 of Z1 predictions. Specifically, without prior compensation, the prediction error of Z1 increases by 0.83, whereas with prior compensation, it only increases by 0.27. This result suggests that the prior compensation factor can effectively alleviate the forgetting issue. As a result, Z1 can offer more comprehensive priors for Z2 and Z3 predictions, resulting in better prediction performance compared to the training approach without prior compensation.\n\n**without prior compensation:**\n\n|             | **Z1** | **Z2** | **Z3** |\n| ----------- | ------ | ------ | ------ |\n| **Stage 1** | 9.03   | -      | -      |\n| **Stage 2** | 9.44   | 45.33  | -      |\n| **Stage 3** | 9.86   | 45.70  | 92.80  |\n\n**Ours:**\n\n|             | **Z1** | **Z2** | **Z3** |\n| ----------- | ------ | ------ | ------ |\n| **Stage 1** | 9.03   | -      | -      |\n| **Stage 2** | 9.10   | 44.43  | -      |\n| **Stage 3** | 9.30   | 44.62  | 91.37  |\n\n**Q6: typo: Fig. 1: lacking space before the (**\n\nRe: Thank you for pointing out this. We will add space before the missing ( in Fig.1.\n\n**Q7:Though there are some \"traditional\" settings of the length definition of the long and short terms. I wonder if we change the 5 and 15 frames setting, is the analysis of Fig. 1 kept or changed?**\n\nRe: If we set the short-term to 15 frames, the analysis depicted in Figure 1 still holds. The detailed results are shown in the table. They are consistent with the conclusion drawn from Figure 1, indicating that short-term predictions offer valuable prior information for improving long-term prediction performance.\n\n|            | **short+long** | **short only** | **short then short+long** |\n| ---------- | -------------- | -------------- | ------------------------- |\n| **80ms**   | 10.53          | 9.88           | 10.20                     |\n| **1000ms** | 110.37         | -              | 109.86                    |\n\n**Q8: Learning rate in Eq. 9: choice discussion.**\n\nRe: We have exactly followed the implementation details of the backbone PGBIG and maintained consistency with its learning rate (initialize to 5e-3 and decrease the learning rate exponentially)."}, {"review_id": "13c0NSxhVn", "reviewer": "Reviewer_aXof", "summary": "The paper proposes a novel multi-stage training framework called Temporal Continual Learning (TCL) for Human Motion Prediction (HMP) to address the challenges of short-term and long-term predictions and the incorporation of prior information from past predictions into subsequent predictions. The Prior Compensation Factor (PCF) is introduced to compensate for the lost prior information, and an optimization objective is derived through theoretical derivation. The TCL framework can be easily integrated with different HMP backbone models and adapted to various datasets and applications. Extensive experiments on three HMP benchmark datasets demonstrate the effectiveness and flexibility of TCL.", "questions": "+ It is clear that \"our objective\" is between the \"actual objective\" and \"naive objective\". In Figure 2, is \"our objective\" close to the \"actual objective\"? Why isn't it closer to the \"naive objective\"? Any intuition or proof?\n+ Missing baselines or related work: [1, 2, 3, 4]. I would like to discuss with the authors about the choices of baselines. \n+ Efficiency and multi-stage pipeline. Recent researches [5] suggest predicting motions in one stage, which is easier to train. Will the multi-stage training be harder to tune or train? Will it be more time-consuming? I would like to discuss with the authors about it. \n+ It will be great if authors can discuss or provide zero-shot adaptation experiments on other datasets. \n\n\n\n**I would like to discuss with the authors according to the authors' rebuttal. The experiments are not sufficient and the presentation of the paper is not good enough. However, the theoretical insights are interesting. Therefore, I provide a weak accept score here. I will adjust my score according to the authors' response. I will carefully check the authors' responses. I would like to discuss with the authors with details of the paper and improve the quality of the papers jointly.**\n\n\n\n[1]: Zhong, Chongyang, et al. \"Spatio-temporal gating-adjacency GCN for human motion prediction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2]: Sofianos, Theodoros, et al. \"Space-time-separable graph convolutional network for pose forecasting.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[3]: Bouazizi, Arij, et al. \"MotionMixer: MLP-based 3D Human Body Pose Forecasting.\"IJCAI 2022.\n\n[4]: Guo, Wen, et al. \"Back to mlp: A simple baseline for human motion prediction.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n\n[5]: Chen et al. \"HumanMAC: Masked Motion Completion for Human Motion Prediction.\" arXiv preprint arXiv:2302.03665 (2023).\n\n---\n\n\nI revise my rating to borderline accept. See [detail](https://openreview.net/forum?id=v0GzRLvVp3&noteId=FWQ456OsEQ).", "limitations": "Authors discussed limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ Introducing the Temporal Continual Learning (TCL) framework, a multi-stage training framework that addresses the constraint between short-term and long-term prediction in Human Motion Prediction (HMP) and allows for better utilization of prior knowledge from short-term prediction to enhance the performance of long-term prediction.\n+ Introducing the Prior Compensation Factor (PCF) to mitigate the issue of forgetting information during multi-stage training, and deriving a more reasonable optimization objective through theoretical derivation.\n+ Exploring a new pipeline for the HMP task.", "weaknesses": "+ Writing:\n  + Without periods in lines 113-117.\n  + $P\\left(Z_{k} \\mid Z_{1} Z_{2} \\cdots Z_{k-1} \\theta\\right)$ -> $P\\left(Z_{k} \\mid Z_{1} Z_{2} \\cdots Z_{k-1}; \\theta\\right)$.\n  + **The following concepts are confusing: short+long, short only, short then shorr long. The explanation in Figure 1 is not easy to understand. I suggest providing a figure to illustrate the difference between the three concepts.** It is essential to your motivation.\n  + **Please provide more experimental settings about the toy example (Figure 1).**\n  + Methods part should be carefully written. Otherwise, it is somewhat confusing. \n+ It is not clear about the motivation for designing $\\alpha$s. The authors name it Prior Compensation Factors in Section 3,2. Why can it compensate for forgotten knowledge when leveraging prior knowledge? It is a scalar and why can it reflect so much knowledge of complex motions?\n+ For experiments:\n  + Datasets: I suggest authors provide the results on AMASS datasets, which will make your work more solid. \n  + Authors choose PGBIG  as the backbone. More choices and comparisons of backbones should be presented in the main result (not in ablation). This verifies the main contribution of the paper. \n  + I did not find the codes and demo videos in the supplementary. (Not necessary. If provided, more convincing. Pretrained models with inference codes are acceptable.)\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Part 1 (Part 2 is in global rebuttal)**\n\nThanks to the reviewer for the constructive comments. We have carefully addressed your concerns and provided detailed responses for each review.\n\n**Q1:Some issues with the wording.**\n\nRe: Thank you for pointing out them. We will correct them.\n\n**Q2: Some questions of fig.1.**\n\nRe: Thank you for your suggestion. \"short+long\" represents the training approach of the baseline model, where both short-term and long-term predictions are trained together. \"short only\" indicates that only short-term predictions are trained on the baseline model without considering long-term predictions. \"short then short+long\" means that after training the model using the \"short only\" approach, the model is further trained by combining both short-term and long-term predictions together. We will provide a figure to illustrate the difference between different concepts and express it in a clearer manner. As for the experimental setting of Fig. 1, we followed the implementation details of the backbone PGBIG.\n\n**Q3: It is not clear about the motivation for designing αs. The authors name it Prior Compensation Factors in Section 3,2. Why can it compensate for forgotten knowledge when leveraging prior knowledge? It is a scalar and why can it reflect so much knowledge of complex motions?**\n\nRe: As we conduct a preliminary experiment and find that knowledge learned in short-term prediction can serve as a prior to facilitate the learning of far-future, this motivates us to formally model the motion prediction and propose a multi-stage training strategy to better exploit this prior knowledge. However, since we discover that the prior knowledge acquired from previous tasks diminishes when switching stages, our motivation for designing αs is to address this issue and estimate this lost prior knowledge to assist in predicting subsequent task. By applying a series of derivations, we obtain the final objective function as Eq. 5, where αs are formally presented as controlling the weighted combination of losses at different stages. Since the fitting of long-term prediction dominates the learning process, the prediction model struggles to preserve the prior knowledge learnt from previous short-term predictions. Consequently, the loss function assigns lower weights to current task to mitigate the loss of forgotten information. Thus, αs which is a vector does play a compensatory role for the forgotten knowledge and demonstrates effectiveness.\n\n**Q4: Datasets: I suggest authors provide the results on AMASS datasets, which will make your work more solid.**\n\nRe: We follow your suggestion and conduct experiment on the AMASS dataset. However, due to the unavailability of some sub-dataset (\"Eyes_Japan_Dataset\" and \"BioMotionLab_NTroje\") in AMASS, we could only perform experiments on its subset. We use the STSGCN [2] model you mentioned as backbone. Detailed results can be found in Q8. The experimental results indicate that our strategy effectively enhances model training on this dataset.\n\n**Q5: Authors choose PGBIG as the backbone. More choices and comparisons of backbones should be presented in the main result (not in ablation). This verifies the main contribution of the paper.**\n\nRe: We select the PGBIG as the backbone of our main experiments as it is the state-of-the-art architectures for human motion prediction under the common setting. However, we have also followed your suggestion to test the results of our approach with other backbones such as STSGCN [2] (GCN-based), MotionMixer [3] (MLP-based), siMLPe [4] (MLP-based) and POTR [6] (Transformer-based). The results can be found in Q8. As can be seen, our training method still improves the performance of the original prediction backbone model.\n\n**Q6: I did not find the codes and demo videos in the supplementary. (Not necessary. If provided, more convincing. Pretrained models with inference codes are acceptable.)**\n\nRe: We have provided a link of demo videoto AC. Due to NIPS' official policy that prohibits providing external links in the rebuttal, we would release our codes and pre-trained models after this work is accepted for publication. \n\n**Q7: It is clear that \"our objective\" is between the \"actual objective\" and \"naive objective\". In Figure 2, is \"our objective\" close to the \"actual objective\"? Why isn't it closer to the \"naive objective\"? Any intuition or proof?**\n\nRe: Fig. 2 is a illustrative diagram illustrating the relative positioning among \"naive objective\", \"our objective\" and \"actual objective\". This serves to demonstrate that our objective is a superior upper bound compared to the naive approach. \n\n**Q8: Missing baselines or related work: [1, 2, 3, 4]. I would like to discuss with the authors about the choices of baselines.**\n\nRe: Our approach can be flexibly applied to various backbones, such as [1, 2, 3, 4]. [1] enhances the generalization ability of GCN using a gating network. [2] utilizes space-time separable GCN to extract features from different dimensions. [3] presents an MLP-based architecture that effectively leverages spatiotemporal aggregated features. [4] is a recent MLP-based model. Due to the lack of open-source code for [1], we can only report experimental results for [2], [3] and [4]. The experiment of baseline [2] was conducted on the AMASS dataset with mean per joint position error as the evaluation metric. The experiments of baselines [3, 4] were conducted on the Human3.6M dataset, and the evaluation metric used was mean per joint position error. We also conducted experiments on POTR [6] (a Transformer-based model). And the results for POTR are shown using euler angle error as the evaluation metric. From the experimental results in the Table 6, 7, 9 and 10, it can be observed that our strategy consistently improves the performance of these backbones, validating the effectiveness of our proposed strategy. We will provide discussions on [1, 2, 3, 4] in subsequent versions of the manuscript."}]

Author rebuttal:
**(Part 2 for reviewer GqjD)**

**reference:**

[1] Guo, W., Du, Y., Shen, X., Lepetit, V., Alameda-Pineda, X., & Moreno-Noguer, F. (2023). Back to mlp: A simple baseline for human motion prediction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 4809-4819).

[2] Martínez-González, A., Villamizar, M., & Odobez, J. M. (2021). Pose transformers (potr): Human motion prediction with non-autoregressive transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2276-2284).

[3] Ma, T., Nie, Y., Long, C., Zhang, Q., & Li, G. (2022). Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6437-6446).

[4] Mao, W., Liu, M., Salzmann, M., & Li, H. (2019). Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 9489-9497).

[5] Martinez, J., Black, M. J., & Romero, J. (2017). On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2891-2900).


**(Part 2 for reviewer aXof)**

**Q9: Efficiency and multi-stage pipeline. Recent researches [5] suggest predicting motions in one stage, which is easier to train. Will the multi-stage training be harder to tune or train? Will it be more time-consuming? I would like to discuss with the authors about it.**

Re: The multi-stage training approach we proposed does not require complex adjustment techniques. What's more, it offers certain advantages over single-stage training. The temporal multi-stage training process leverages prior predictive information from earlier stages more effectively, thereby guiding the learning of challenging long-term prediction. Simultaneously, training the short-term prediction separately is easier to learn and results in better short-term predicting capabilities. Although our proposed strategy incurs increase in training time, the testing time remains consistent with that of the backbone model. Regarding the training process, due to the faster convergence of short-term predictions, fewer iterations are needed for training the first stage. Additionally, because the model can better utilize the prior information provided by short-term predictions, the subsequent stages of training also converge more easily. As a result, the overall training time does not increase significantly. The training time of different stages are demonstrated in the Table 5. We will include this discussion in the subsequent manuscript.

**Q10: It will be great if authors can discuss or provide zero-shot adaptation experiments on other datasets.**

Re: We conducted zero-shot experiments based on your advice. Due to variations in the number of key-points and annotations across different human motion datasets, we conduct the zero-shot experiments on Human3.6M dataset with backbone only trained on stage 1 (task Z1). The results can be found in the table below. First, the zero-shot performance is not as good as training all tasks. Second, there are variations in performance between different backbones, with PGBIG outperforming others.

|                                | **Z2** | **Z3** |
| :----------------------------: | :----: | :----: |
|       **Res. Sup.+Ours**       | 83.23  | 146.80 |
| **Res. Sup.+Ours (zero-shot)** | 86.26  | 155.66 |
|         **PGBIG+Ours**         | 44.39  | 91.11  |
|   **PGBIG+Ours (zero-shot)**   | 76.84  | 122.59 |

**reference:**

[1] Zhong, C., Hu, L., Zhang, Z., Ye, Y., & Xia, S. (2022). Spatio-temporal gating-adjacency gcn for human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6447-6456).

[2] Sofianos, T., Sampieri, A., Franco, L., & Galasso, F. (2021). Space-time-separable graph convolutional network for pose forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 11209-11218).

[3] Bouazizi, A., Holzbock, A., Kressel, U., Dietmayer, K., & Belagiannis, V. (2022). Motionmixer: Mlp-based 3d human body pose forecasting. arXiv preprint arXiv:2207.00499.

[4] Guo, W., Du, Y., Shen, X., Lepetit, V., Alameda-Pineda, X., & Moreno-Noguer, F. (2023). Back to mlp: A simple baseline for human motion prediction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (pp. 4809-4819).

[5] Chen, L. H., Zhang, J., Li, Y., Pang, Y., Xia, X., & Liu, T. (2023). HumanMAC: Masked Motion Completion for Human Motion Prediction. arXiv preprint arXiv:2302.03665.

[6] Martínez-González, A., Villamizar, M., & Odobez, J. M. (2021). Pose transformers (potr): Human motion prediction with non-autoregressive transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 2276-2284).

[7] Ma, T., Nie, Y., Long, C., Zhang, Q., & Li, G. (2022). Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 6437-6446).

[8] Martinez, J., Black, M. J., & Romero, J. (2017). On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2891-2900).

Meta-review: The manuscript proposes a multi-stage training scheme resembling curriculum learning with an increasing prediction horizon for human motion prediction. 

The reviewers acknowledge that the paper addresses a common but often neglected problem in human motion prediction, that the proposed technique is well-motivated and sound, and that the proposed technique improves the performance of various models. 

The reviewers had some questions regarding the interpretation of the $\alpha$ term and Figure 1. They suggested a more thorough discussion of related work and requested to include more recent methods and datasets in the comparison.   

During the discussion, the authors answered the questions of the reviewers and resolved their concerns. While there was some confusion regarding the experiments on the AMASS dataset and the submitted results in the PDF of the rebuttal due to missing captions, the additional results reported during the discussion support the results reported in the paper and show the usefulness of the proposed training scheme, which can be applied to a large range of approaches for human motion prediction. The authors are asked to include the additional results in the camera-ready version and address the clarity issues raised by the reviewers.
Decision: Accept (poster)