Title: Temporal Continual Learning with Prior Compensation for Human Motion Prediction
Venue: NeurIPS 2023
Accepted: yes
Keywords: Human Motion Prediction; Temporal Continual Learning; Prior Compensation Factor
Abstract: Human Motion Prediction (HMP) aims to predict future poses at different moments according to past motion sequences. Previous approaches have treated the prediction of various moments equally, resulting in two main limitations: the learning of short-term predictions is hindered by the focus on long-term predictions, and the incorporation of prior information from past predictions into subsequent predictions is limited. In this paper, we introduce a novel multi-stage training framework called Temporal Continual Learning (TCL) to address the above challenges. To better preserve prior information, we introduce the Prior Compensation Factor (PCF). We incorporate it into the model training to compensate for the lost prior information. Furthermore, we derive a more reasonable optimization objective through theoretical derivation. It is important to note that our TCL framework can be easily integrated with different HMP backbone models and adapted to various datasets and applications. Extensive experiments on four HMP benchmark datasets demonstrate the effectiveness and flexibility of TCL. The code is available at https://github.com/hyqlat/TCL.

Temporal Continual Learning with Prior Compensation for Human Motion Prediction

Abstract

Human Motion Prediction (HMP) aims to predict future poses at different moments according to past motion sequences. Previous approaches have treated the prediction of various moments equally, resulting in two main limitations: the learning of short-term predictions is hindered by the focus on long-term predictions, and the incorporation of prior information from past predictions into subsequent predictions is limited. In this paper, we introduce a novel multi-stage training framework called Temporal Continual Learning (TCL) to address the above challenges. To better preserve prior information, we introduce the Prior Compensation Factor (PCF). We incorporate it into the model training to compensate for the lost prior information. Furthermore, we derive a more reasonable optimization objective through theoretical derivation. It is important to note that our TCL framework can be easily integrated with different HMP backbone models and adapted to various datasets and applications. Extensive experiments on four HMP benchmark datasets demonstrate the effectiveness and flexibility of TCL. The code is available at https://github.com/hyqlat/TCL .

1 Introduction

Human Motion Prediction (HMP) aims to predict future poses at varied temporal moments based on the observed motion sequences. The accurate prediction of human motion plays a vital role in many applications, such as autonomous driving, human-robot interaction, and security monitoring, enabling the anticipation and mitigation of risks. This task is challenging due to its requirement for predicting multiple moments, including short-term predictions for the 'near-future' and long-term predictions for the 'far-future'.

Previous approaches address this task by autoregressively forecasting using recurrent neural networks (RNNs) and transformer architectures [1, 4, 5, 9-12, 14, 26, 37, 40, 42], or parallelly generating all frames with graph convolution networks (GCNs) [2, 6, 7, 23, 25, 30, 31, 28]. These methods employ

condition

learned target

11

35

1

long

short

(b) short+long

condition

short

learned

target

11

35

1

15

(a) short only

condition

condition

short

learned

target

long

learned target

11

35

1

15

Stage 1

Stage 2

short

(c) short then short+long

short+long

short only

short then

short+long

9.6

9.9

10.2

10.5



short+long

short then

short+long

109.5

109.8

110.1

110.4







the one-stage training strategy to directly train a model that can predict both the short-term prediction and the long-term prediction. However, the long-term motion prediction is more challenging since the future motion can vary greatly (i.e., the prediction space is large), which would increase the uncertainty and ambiguity of future prediction. As the prediction length increases, the fitting of the high-uncertainty long-term prediction will gradually dominate the learning process of the prediction model, which hinders the learning of short-term predictions and further limits the full potential of leveraging the prior knowledge learned from short-term inputs to facilitate long-term predictions.

This motivates us to exploit proper training strategies to better learn and utilize the prior knowledge. We further investigate this by conducting preliminary experiments on the following settings 'short+long', 'short only', and 'short then short+long', which are illustrated in Figure 1. The results are summarized in Figure 2. We observe that 'short then short+long" outperforms 'short+long' on long-term prediction, which implies that the knowledge learned in short-term prediction can serve as a prior to facilitating the learning of 'far-future' prediction. This is intuitive and thus we can use a progressive learning approach, where the model is trained to predict increasing numbers of frames over multiple training stages, e.g., starting with 5 frames in the first stage and 10 frames in the second stage, and so on. However, we also observe that 'short then short+long' performs worse than 'short only' by a considerable margin for short-term prediction, which demonstrates the joint learning of short-term and long-term prediction results in knowledge forgetting for short-term prediction.

To overcome these problems, we introduce the Prior Compensation Factor (PCF) into the multi-stage training method to obtain a sequential continuous learning framework, which is named Temporal Continual Learning (TCL). It is a multi-stage training framework that alleviates constraint posed by long-term prediction on short-term prediction and effectively utilize prior information from short-term prediction. Specifically, we divide the future sequence into segments and divide the training process

into multiple stages accordingly. We incrementally increase the number of prediction segments, allowing us to leverage the prior knowledge acquired from earlier stages for predicting the subsequent ones. Upon completion of each stage's training, the learned prior knowledge is saved in the model parameters. However, with the changing of optimization objective when switching stages, the prior knowledge gradually fades away. In order to overcome this forgetting problem of prior knowledge, we further introduce the PCF, which is designed as a learnable variable. Then, we derive a more reasonable optimization objective for this regression problem through theoretical derivation.

The proposed training framework is flexible and can be easily integrated with various HMP backbones or adapted to different datasets. To validate the effectiveness and flexibility of our framework, we conduct experiments on four popular HMP benchmark datasets by integrating TCL with several HMP backbones. Our contributions can be summarized as follows:1) We identify certain limitations in existing HMP models and propose a novel multi-stage training strategy called Temporal Continual Learning to obtain a more accurate motion prediction model. 2) We introduce a Prior Compensation Factor to tackle the forgetting problem of prior knowledge, which can be learned jointly with the prediction model parameters. 3) We obtain an easily optimized and more reasonable objective function through theoretical derivation.

2 Related Work

Autoregressive prediction approaches for HMP. Motivated by natural language processing, many researchers have adopted sequence-to-sequence models to exploit the temporal information of pose sequences in HMP, which includes RNNs, Long Short-Term Memory Networks (LSTMs) [15] and Transformer [43]. For instance, ERD [8] combined LSTMs with an encoder-decoder to model the temporal aspect, while Jain et al. [18] proposed Structural-RNN to capture spatiotemporal features of human motion. Martinez et al. [32] applied a sequence-to-sequence architecture for modeling the human motion structure. Aksan et al. [1] used Transformer to autoregressively predict future poses. Sun et al. [40] designed a query-read process to retrieve some motion dynamics from the memory bank. Lucas et al. [27] proposed a GPT-like[35] autoregressive method to generate human poses. Tang et al. [42] combined attention mechanism and LSTM to complete the human motion prediction task. However, autoregressive methods are difficult to train and suffer from error accumulation problem.

Parallel prediction approaches for HMP. Some researchers employed parallel prediction methods to address HMP problem [2, 6, 7, 23, 25, 30, 31, 28]. The works of [21-23] used GCN to encode feature or to decode it, which associates different joints' information. Mao et al. [31] viewed a pose as a fully connected graph and used GCN to extract hidden information between any pair of joints. Martinez et al. [33] devised a transformer-based network to predict human poses. Sofianos et al. [39] proposed a method to extract spatiotemporal features using GCNs. And Ma et al. [28] tried to achieve better prediction results using a progressive manner. Xu et al. [45] used multi-level spatial-temporal anchors to make diverse predictions.

Continual learning. Although Deep Neural Networks (DNNs) have demonstrated impressive performance on specific tasks, their limitations in handling diverse tasks hinder their broader applicability. Therefore, some researchers introduced the concept of Continual Learning (CL)[36] to DNNs to ensure that models retain the knowledge of previous tasks while learning new tasks. Kirkpatrick et al. [20] proposed the Elastic Weight Consolidation (EWC) method to overcome the catastrophic forgetting problem and improve the performance of multi-task problems. Shin et al. [38] introduced a method that addresses catastrophic forgetting in sequential learning scenarios by using a generative model to replay data from past tasks during the training of new tasks. It is important to note that the traditional CL approaches do not account for temporal correlation and are unable to leverage data from previous tasks.

3 Method

The problem of human motion prediction involves predicting future motion sequences by utilizing previously observed motion sequences. Formally, let X 1: T h = [ X 1 , X 2 , · · · , X T h ] ∈ R J × D × T h denotes the observed motion sequence of length T h where X i indicates motion of time i , and X T h +1: T h + T p = [ X T h +1 , X T h +2 , · · · , X T h + T p ] ∈ R J × D × T p represents the motion sequence of

length T p that needs to be predicted. Note that J is the number of joints for each pose, and D is the dimension of coordinates. It can be regarded as a composite task consisting of multiple sequential prediction tasks, which involves predicting the future poses at varied moments conditioned on the motion sequences observed in the past.

To accomplish this multiple sequential prediction task, we first model the HMP problem as solving the following optimization problem:

Thus our target is to find the optimal model that maximizes Equation (1). To achieve this, we propose a framework called Temporal Continual Learning. Specifically, we partition the entire prediction interval into several smaller segments and perform multi-stage training. We claim that this enables the utilization of prior information from previous segments as knowledge for predicting the subsequent segments. Further, as the optimization objective changes in each training stage, we find that the prior knowledge, i.e., information learned in previous training stages, will be forgotten to a certain degree. To mitigate this problem, we introduce a Prior Compensation Factor and accordingly derive a more reasonable optimization objective at each stage.

3.1 Multi-stage Training Process.

We initially decouple the future sequence into K segments with time boundaries T Z 1 , T Z 2 , · · · , T Z K , where T Z K = T h + T p . And we denote the prediction of segment k as task Z k , which can be expressed as follows:

Task Z 1 : X 1: T h → X T h +1: T Z 1

Task Z 2 : X 1: T h → X T Z 1 +1: T Z 2 · · ·

Task Z K : X 1: T h → X T Z K -1 +1: T Z K

To be specific, the target of task Z 1 is to predict X T h +1: T Z 1 conditioned on X 1: T h , and task Z k aims to predict X T Z k -1 +1: T Z k with X 1: T h as condition. Therefore, by leveraging bayesian formulation, optimization problem P ( X T h +1 , X T h +2 , · · · , X T h + T p | X 1 , X 2 , · · · , X T h ; θ ) can be formulated as:

where θ is model parameters to be learned. In the following, we denote ' Z 1 Z 2 · · · Z k ' as ' Z 1: k '. Our target is to maximize Equation (2) which means finding optimal model to accomplish all tasks.

For the purpose of transferring the prior knowledge in preceding tasks to their subsequent prediction task, we progressively increase the number of tasks in temporal order and train them successively. More precisely, our training is decomposed into K stages. In each stage S k , we leverage the optimal model parameters θ ∗ k -1 trained in the previous stage S k -1 to initialize the parameters θ , and then update it based on the prediction tasks Z 1 , Z 2 , · · · , Z k (maximizing P ( Z 1: k ; θ ) ). Since the model is trained to optimize the prediction tasks Z 1: k -1 in training stage S k -1 , the knowledge of tasks Z 1: k -1 can be implicitly involved in its well-trained parameters θ ∗ k -1 . Initializing θ k as θ ∗ k -1 can exploit the prior knowledge learned in previous tasks to assist the prediction of the next task.

3.2 Definition of Prior Compensation Factor.

With training different optimization objectives stage by stage, the prior knowledge provided by previous tasks can be effectively exploited to predict the subsequent task. However, the change of the optimization objective in different training stages could also bring about the knowledge forgetting problem. To mitigate the problem, we introduce α Z 1: k -1 → Z k to estimate the extent of forgotten knowledge when utilizing prior knowledge from tasks Z 1: k -1 to predict task Z k , which we refer to as the 'Prior Compensation Factor'.

Here, ˆ Z 1: k -1 is regarded as the prior knowledge that is reserved and can be still provided for predicting task Z k . So ˆ Z 1: k -1 initially represents the prior knowledge reserved in θ ∗ k -1 in every stage

S k and would get somewhat corrupted gradually during training. P ( Z k | Z 1: k -1 ; θ ) indicates the most ideal case, where the current prediction task Z k can fully leverage the prior information provided by previous prediction tasks Z 1: k -1 . Consequently, the loss of the prior knowledge is non-negative, implying that 0 ≤ α Z 1: k -1 → Z k ≤ 1 -P ( Z k | ˆ Z 1: k -1 ; θ ) . Specifically, we can observe that α = 0 when P ( Z k | Z 1: k -1 ; θ ) = P ( Z k | ˆ Z 1: k -1 ; θ ) , which implies that all the prior knowledge of previous tasks is completely exploited although the θ changes. By substituting Equation (3) into Equation (2) and taking the negative logarithm, we can obtain:

Our objective is to minimize -log P ( Z 1: k ; θ ) with respect to the model parameter θ and prior compensation factors { α Z 1: i -1 → Z i , i = 2 , 3 , · · · , k } .

3.3 Optimization Objective

Optimizing Equation (4) directly is challenging due to the presence of the PCF α and P ( Z k | ˆ Z 1: k -1 ; θ ) inside the logarithm. By applying Lemma 3.1 (details provided in the appendix), we can obtain an upper bound for Equation (4), which can be expressed as:

Hence, we can turn to minimize the upper bound of Equation (4). It appears that in the optimization objective, α Z 1: i -1 → Z i serves as a factor to control the weights of different tasks, which mitigates the loss of prior information and thus compensates for the lost prior knowledge. The Lemma 3.2 indicates that the largest difference between -log P ( Z 1: k ; θ ) and the upper bound UB would not exceed log( 3 / 2 ) ∗ ( k -1) when P ( Z i | ˆ Z 1: i -1 ; θ ) ≥ 1 / 2 , i ∈ { 2 , 3 , · · · , k } .

Lemma 3.1. For 0 ≤ a ≤ 1 -b and 0 < b ≤ 1 , the inequality -log( a + b ) ≤ (1 -a )( -log b ) + (1 -a ) log(1 -a ) + log(1 + a ) holds. The equality holds if and only if a = 0 .

Lemma3.2. The absolute difference between the target objective (Equation (4)) and the upper bound (Equation (5)) is not larger than log( 3 / 2 ) ∗ ( k -1) when P ( Z i | ˆ Z 1: i -1 ; θ ) ≥ 1 / 2 , i ∈ { 2 , 3 , · · · , k } . This bound is achieved when P ( Z k | ˆ Z 1: k -1 ; θ ) = 1 / 2 and α Z 1: i -1 → Z i = 1 / 2 , i ∈ { 2 , 3 , · · · , k } .

Due to the space limitation, we present the proofs of Lemma 3.1 and 3.2 in the appendix.

An intuitive explanation. Figure 3 illustrates the comparison between the term -log( P ( Z k | ˆ Z 1: k -1 ; θ ) + α Z 1: k -1 → Z k ) in the actual optimization objective, the term -log P ( Z k | ˆ Z 1: k -1 ; θ ) in the naive optimization objective merely leveraging multi-stage training strategy, and the corresponding approximate term in our optimization objective. It can be observed that our approximation method has a smaller difference from the actual optimization objective compared to the naive method. This is important as a more reasonable objective function can improve the accuracy of the optimization process. When the prior compensation factor α Z 1: k -1 → Z k is zero, which means P ( Z k | Z 1: k -1 ; θ ) = P ( Z k | ˆ Z 1: k -1 ; θ ) , the prior prediction information from previous tasks Z 1: k -1 is not lost. As the value of α increases, the discrepancy between the actual objective P ( Z k | Z 1: k -1 ; θ ) and the objective of the naive method P ( Z k | ˆ Z 1: k -1 ; θ ) becomes more evident, indicating a greater degree of forgetting prior knowledge. In contrast, our approach narrows this gap by effectively mitigating the loss of prior information obtained from tasks Z 1: k -1 .

3.4 Optimization Strategy

We train the model in a multi-stage manner, in which an initial stage and K -1 TCL stages are involved. The initial stage S 1 aims to forecast the motion in the foremost segment, while each TCL stage S k , k ∈ { 2 , 3 , · · · , K } performs prediction for segments Z 1: k and simultaneously trains

α

0

-𝐥𝐨𝐠𝑷

(

𝒁𝒌

|

𝒁

𝟏

:

𝒌-𝟏

;

𝜽

)

-𝐥𝐨𝐠𝑷

(

𝒁𝒌

|

𝒁𝟏

:

𝒌-𝟏

;

𝜽

)

𝟏 - 𝑷

(

𝒁𝒌

|

𝒁

𝟏

:

𝒌-𝟏

;

𝜽

)

Our objective

Naive objective

Actual objective

α Z 1: k -1 → Z k . Once the stage S k is completely trained, we then estimate the factors ˆ α Z 1: k -1 → Z k which would be used in the optimization of the following stages. This process is repeated until the final stage S K is reached.

Learning of initial stage S 1 . Following the implementations of previous methods [19], we can train the initial stage Z 1 with Mean Suqared Error (MSE) loss:

where X i and ˆ X i represent the ground truth and predicted motion of the i -th frames respectively.

Temporal Continual Learning at stage S k . In stage S k ( k ≥ 2 ), we need to update the model parameters θ corresponding to tasks Z 1: k and the PCF α Z 1: k -1 → Z k . According to Equation (5), the loss function in this stage can be calculated as follows:

where the parameters ˆ α Z 1 → Z 2 , · · · , ˆ α Z 1: k -2 → Z k -1 are dertermined in the learning of previous stages. Once the model parameters for stage S k are determined, we then calculate ˆ α Z 1: k -1 → Z k as:

where M represents number of samples and ˆ α m Z 1: k -1 → Z k is PCF estimated for the m -th sample.

We continue the TCL training process by predicting stage S k +1 and updating the model parameter θ as well as PCF α Z 1: k → Z k +1 . We repeat this TCL process until we reach the final stage S K . In practice, we require the backbone model to output an extra dimension and pass it through an MLP head to obtain α . The algorithm flow is summarized in Algorithm 1.

Algorithm 1 Training procedure of proposed TCL framework.

4 Experiments

4.1 Experimental Setup

We validate our framework on four benchmark datasets. Human3.6M[17] is a large dataset that contains 3.6 million 3D human pose data. 15 types of actions performed by 7 actors(S1, S5, S6, S7, S8, S9 and S11) are included in this dataset. Each actor is represented by a skeleton of 32 joints. However, following the data preprocessing method proposed in [28, 31], we only use 22 joints. The global rotations and translations of poses are removed, and the frame rate is downsampled from 50 fps to 25 fps. For testing and validation, we use actors S5 and S11, while training is conducted on the remaining sections of the dataset. CMU-MoCap is a smaller dataset that has 8 different action categories. The global rotations and translations of the poses are also removed. Each pose contains 38 joints, but following the data preprocessing methods in [28, 31], we only use 25 joints. 3DPW[44] is a challenging dataset that contains human motion data captured from both indoor and outdoor scenes. Poses in this dataset are represented in 3D space, with each pose containing 26 joints. However, only 23 of these joints are used, as the other three are redundant. The Archive of Motion Capture as Surface Shapes (AMASS)[29] dataset gathers 18 existing mocap datasets. Following [39], we select 13 from those and take 8 for training, 4 for validation and 1 (BMLrub) as the test set. We consider forecasting the body joints only and discard those 4 static ones, leading to an 18-joint human pose.

Following the benchmark protocols, we use the Mean Per Joint Position Error (MPJPE) in millimeters (ms) as our evaluation metric for 3D coordinate errors and Euler Angle Error (EAE) for Euler angle representations. The performance is better if this metric is smaller.

Implementation Details. Following [28, 30, 39], we set the input length to 10 frames and the predictive output to 25 frames for Human3.6M, AMASS and CMU-Mocap datasets, respectively. For the 3DPW dataset, we predict 30 frames conditioned on the observation of the preceding 10 frames. We choose PGBIG as our backbone model by default. In order to learn the PCF, we add an extra dimension to the output of the backbone model and calculate PCF through an MLP network whose hidden dimension is set to 512. We partitioned the future sequences into three segments with lengths of 3, 9, and 13. The training process was conducted on an NVIDIA RTX 3090 GPU for 120 epochs, allocating 50, 90, and 120 epochs for each respective stage.

4.2 Experimental Results

We apply our method on the following approaches on four benchmark datasets: Res.Sup.(R.S.) [32], LTD [31], POTR [33], STSGCN [39], MotionMixer(MM) [3], siMLPe [13] and the current state-of-the-art PGBIG [28]. Res.Sup. is an RNN-based model. LTD, STSGCN and PGBIG are GCN-based models. POTR is a transformer-based method. MotionMixer and siMLPe are MLP-based models. All these methods have released their code publicly. We employ their pre-trained models or re-train their models using the suggested hyper-parameters for a fair comparison. And we also follow the metric they used to evaluate the results.

Table 1 presents experimental results of different backbone models before and after applying our training strategy on Human3.6M dataset. Our frameworks outperform the corresponding backbone models by a considerable margin (more than 1.5). Specifically, when applying our strategy to PGBIG, we obtained a performance of 64.97, which is much better than the performance of the original PGBIG (66.52). Table 2 shows the quantitative comparisons of prediction results on the CMU-MoCap, 3DPW and AMASS datasets in which our proposed framework also achieves the best results. We can observe that the improvement in long-term prediction is greater compared to short-term prediction, indicating that the prior information provided by short-term prediction is more crucial for challenging long-term prediction tasks. It is worth noticing that our framework outperforms the others by a significant margin on 3DPW dataset. Specifically, our strategy outperforms PGBIG by 8.9 (63.59 vs 72.49). We attribute this to our framework's ability to leverage prediction prior for the subsequent prediction in this challenging dataset.

4.3 Visualization

Predicting results. Figure 4 provides some visualization examples of predicted motions, demonstrating that our framework achieves more accurate results. Specifically, Figure 4a represents the action 'directions', with the person maintaining an upright position throughout the sequence. The results of PGBIG exhibit a bent posture during long-term predictions, whereas our method can predict states closer to the ground truth position. While in Figure 4b, the person first bends and then stands upright. The PGBIG maintains the bent posture throughout the long-term prediction and our method can accurately predict the changes in posture. It is evident that our method demonstrates an improvement in long-term prediction effectiveness.

α at different stages. As Figure 5 displays, the value of α progressively increases with each training stage. When a new task is added, the model focuses on the new objective without considering

dataset ms

R.S.

R.S.+Ours

LTD

LTD+Ours

PGBIG

PGBIG+Ours dataset

ms

R.S

R.S+Ours

LTD

LTD+Ours

PGBIG

PGBIG+Ours dataset

ms

STSGCN

STSGCN+Ours

CMU

400

93.1

85.6

40.9

38.5

36.6

35.4

3DPW

400

600

800

1000

129.7 142.9 161.1 171.7

120.0 139.5 156.2 169.1

67.8

90.6

106.9 117.8

64.1

58.3

47.1

320

37.2

36.7

86.1

79.8

100.4 110.0

94.4

104.1

67.7

83.5

AMASS

400

560

43.8

42.8

53.8

52.4

96.0

1000

69.7

68.1

80

24.4

21.4

9.3

9.1

7.6

7.5

200

99.3

91.8

35.6

33.1

29.3

21.4

80

11.4

10.9

320

80.1

72.7

33.0

31.0

29.0

28.3

560

1000

112.5 141.5

105.6 136.1

55.8

86.2

53.1

50.9

48.6

84.9

80.1

78.4

GT

PGBIG

Ours

(a) Predicting result of the action 'directions'.

GT

PGBIG

Ours

S

1

S

2

S

3

S

4

S

5

S

6

S

7

S

8

S

9

S

10

S

11

0.250

0.275

0.300

0.325

0.350

0.375

Value

Stage number

the preservation of prior information, resulting in the forgetting of prior knowledge. Here, we utilize α to mitigate the loss of prior information, thereby improving the overall training of the model.

4.4 Ablation Studies

Weconduct several ablation experiments to further verify the effectiveness of our proposed framework.

Evaluation on the number of tasks. As shown in Table 3, the model's performance improves as the number of tasks gets larger from 1 to 3, and it remains stable when the number of tasks becomes larger than 3.

Evaluation on different implementations. As Figure 6 shows, we compare the results of four different implementations. PGBIG is our baseline model. 'w/o α ' means that we only divide the training process into several stages to train each task without using the PCF. 'HC' represents using a hand-crafted coefficient that changes its values similar to our PCF at each epoch. Specifically, in stage S 1 , the value of α is set to 1. In stage S 2 , α is initially set to 0.1 and increased by 0.05 at each epoch until reaching 0.5, where it remains constant. The same pattern applies to stage S 3 . In each subfigure, we show the results of Z k on the validation set which is obtained at stage S k .

PGBIG w/o

HC

Ours

8.5

9.0

9.5

10.0

10.5

Valid_avg_error

Z

1

result

PGBIG w/o

HC

Ours

44.0

44.5

45.0

45.5

46.0

46.5

Valid_avg_error

Z

2

result

PGBIG w/o

HC

Ours

91.0

91.5

92.0

92.5

93.0

Valid_avg_error

Z

3

result

PGBIG w/o

HC

Ours

65.0

65.5

66.0

66.5

67.0

Valid_avg_error

Avg result

In stage S 1 , where no prior information is available, the results of "w/o α ", "HC", and "Ours" are identical, but superior to the baseline. This validates the effectiveness of decomposing multiplemoment predictions, as it alleviates the constraint of long-term predictions on short-term predictions and enhances the model's ability to learn short-term predictions. In the following stages, the performance of 'w/o α ' is worse than 'Ours', which indicates that the prior knowledge exploited by our framework benefits the prediction model training. Moreover, as the training period progressed, the performance gap becomes larger. However, the method without α can still achieve better performance than 'PGBIG', indicating that training model in a multi-stage manner can also exploit some useful prior information for prediction. We also note that the performance of 'Ours' is much better than 'HC', which conducts temporal continual learning with fixed and manually defined PCF. It demonstrates that joint training PCF and model parameters is beneficial.

The forgetting of prior knowledge. As shown in Table 4, introducing the prior compensation factor alleviates the performance degradation from stage S 1 to stage S 3 of task Z 1 's predictions. Specifically, without PCF, the prediction error of Z 1 increases by 0.83, whereas with PCF, it only increases by 0.27. This result suggests that PCF can effectively alleviate the forgetting issue. As a result, Z 1 can offer more comprehensive priors for Z 2 and Z 3 predictions, resulting in better prediction performance.

(a) Without PCF.

(b) Using PCF.

5 Conclusion

In this paper, we introduced the temporal continual learning framework for addressing the challenges in human motion prediction. Our framework addresses the constraint between long-term and shortterm prediction, allowing for better utilization of prior knowledge from short-term prediction to enhance the performance of long-term prediction. Additionally, we introduced the prior compensation factor to mitigate the issue of forgetting information. Extensive experiments demonstrated our framework's effectiveness and flexibility.

Limitation. Our proposed training framework may slightly increase training time. However, the testing time remains unchanged compared to the backbone model.

Broader Impact. We believe our work has value for not only human motion prediction but also for more general prediction tasks and backbone models [41, 24, 16, 34]. This has benefits in various areas such as security monitoring, robotics, and autonomous driving.

Acknowledgments and Disclosure of Funding

This work was supported partially by the NSFC (U21A20471, U22A2095, 62076260, 61772570), Guangdong Natural Science Funds Project (2020B1515120085, 2023B1515040025), Guangdong NSF for Distinguished Young Scholar (2022B1515020009), and Guangzhou Science and Technology Plan Project (202201011134).

References

E. Aksan, M. Kaufmann, P. Cao, and O. Hilliges. A spatio-temporal transformer for 3d human motion prediction. In 2021 International Conference on 3D Vision (3DV) , pages 565-574. IEEE, 2021.

E. Aksan, M. Kaufmann, and O. Hilliges. Structured prediction helps 3d human motion modelling. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7144-7153, 2019.

A. Bouazizi, A. Holzbock, U. Kressel, K. Dietmayer, and V. Belagiannis. Motionmixer: Mlp-based 3d human body pose forecasting. arXiv preprint arXiv:2207.00499 , 2022.

H.-k. Chiu, E. Adeli, B. Wang, D.-A. Huang, and J. C. Niebles. Action-agnostic human pose forecasting. In 2019 IEEE Winter Conference on Applications of Computer Vision (WACV) , pages 1423-1432. IEEE, 2019.

E. Corona, A. Pumarola, G. Alenya, and F. Moreno-Noguer. Context-aware human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6992-7001, 2020.

Q. Cui and H. Sun. Towards accurate 3d human motion prediction from incomplete observations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4801-4810, 2021.

Q. Cui, H. Sun, and F. Yang. Learning dynamic relationships for 3d human motion prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 6519-6527, 2020.

K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE international conference on computer vision , pages 4346-4354, 2015.

P. Ghosh, J. Song, E. Aksan, and O. Hilliges. Learning human motion models for long-term predictions. In 2017 International Conference on 3D Vision (3DV) , pages 458-466. IEEE, 2017.

A. Gopalakrishnan, A. Mali, D. Kifer, L. Giles, and A. G. Ororbia. A neural temporal model for human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12116-12125, 2019.

L.-Y. Gui, Y.-X. Wang, X. Liang, and J. M. Moura. Adversarial geometry-aware human motion prediction. In Proceedings of the european conference on computer vision (ECCV) , pages 786-803, 2018.

L.-Y. Gui, Y.-X. Wang, D. Ramanan, and J. M. Moura. Few-shot human motion prediction via meta-learning. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 432-450, 2018.

W. Guo, Y. Du, X. Shen, V. Lepetit, X. Alameda-Pineda, and F. Moreno-Noguer. Back to mlp: A simple baseline for human motion prediction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 4809-4819, 2023.

X. Guo and J. Choi. Human motion prediction via learning local structure representations and temporal dependencies. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 33, pages 2580-2587, 2019.

S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural computation , 9(8):17351780, 1997.

J.-F. Hu, J. Sun, Z. Lin, J.-H. Lai, W. Zeng, and W.-S. Zheng. Apanet: Auto-path aggregation for future instance segmentation prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(7):3386-3403, 2021.

C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence , 36(7):1325-1339, 2013.

A. Jain, A. R. Zamir, S. Savarese, and A. Saxena. Structural-rnn: Deep learning on spatiotemporal graphs. In Proceedings of the ieee conference on computer vision and pattern recognition , pages 5308-5317, 2016.

A. Kendall, Y. Gal, and R. Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7482-7491, 2018.

J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences , 114(13):3521-3526, 2017.

T. Lebailly, S. Kiciroglu, M. Salzmann, P. Fua, and W. Wang. Motion prediction using temporal inception module. In Proceedings of the Asian Conference on Computer Vision , 2020.

M. Li, S. Chen, X. Chen, Y. Zhang, Y. Wang, and Q. Tian. Symbiotic graph neural networks for 3d skeleton-based human action recognition and motion prediction. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(6):3316-3333, 2021.

M. Li, S. Chen, Y. Zhao, Y. Zhang, Y. Wang, and Q. Tian. Dynamic multiscale graph neural networks for 3d skeleton based human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 214-223, 2020.

Z. Lin, J. Sun, J.-F. Hu, Q. Yu, J.-H. Lai, and W.-S. Zheng. Predictive feature learning for future segmentation prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7365-7374, 2021.

J. Liu and J. Yin. Multi-grained trajectory graph convolutional networks for habit-unrelated human motion prediction. arXiv preprint arXiv:2012.12558 , 2020.

Z. Liu, S. Wu, S. Jin, Q. Liu, S. Lu, R. Zimmermann, and L. Cheng. Towards natural and accurate future motion prediction of humans and animals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10004-10012, 2019.

T. Lucas, F. Baradel, P. Weinzaepfel, and G. Rogez. Posegpt: Quantization-based 3d human motion generation and forecasting. In European Conference on Computer Vision , pages 417-435. Springer, 2022.

T. Ma, Y. Nie, C. Long, Q. Zhang, and G. Li. Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6437-6446, 2022.

N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision , pages 5442-5451, 2019.

W. Mao, M. Liu, and M. Salzmann. History repeats itself: Human motion prediction via motion attention. In European Conference on Computer Vision , pages 474-489. Springer, 2020.

W. Mao, M. Liu, M. Salzmann, and H. Li. Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9489-9497, 2019.

J. Martinez, M. J. Black, and J. Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2891-2900, 2017.

A. Martínez-González, M. Villamizar, and J.-M. Odobez. Pose transformers (potr): Human motion prediction with non-autoregressive transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2276-2284, 2021.

A. Mohamed, K. Qian, M. Elhoseiny, and C. Claudel. Social-stgcnn: A social spatio-temporal graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 14424-14432, 2020.

A. Radford, K. Narasimhan, T. Salimans, I. Sutskever, et al. Improving language understanding by generative pre-training. 2018.

M. B. Ring. Child: A first step towards continual learning. Machine Learning , 28(1):77-104, 1997.

H.-F. Sang, Z.-Z. Chen, and D.-K. He. Human motion prediction based on attention mechanism. Multimedia Tools and Applications , 79(9):5529-5544, 2020.

H. Shin, J. K. Lee, J. Kim, and J. Kim. Continual learning with deep generative replay. Advances in neural information processing systems , 30, 2017.

T. Sofianos, A. Sampieri, L. Franco, and F. Galasso. Space-time-separable graph convolutional network for pose forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11209-11218, 2021.

J. Sun, Z. Lin, X. Han, J.-F. Hu, J. Xu, and W.-S. Zheng. Action-guided 3d human motion prediction. Advances in Neural Information Processing Systems , 34:30169-30180, 2021.

J. Sun, J. Xie, J.-F. Hu, Z. Lin, J. Lai, W. Zeng, and W.-s. Zheng. Predicting future instance segmentation with contextual pyramid convlstms. In Proceedings of the 27th acm international conference on multimedia , pages 2043-2051, 2019.

J. Tang, J. Wang, and J.-F. Hu. Predicting human poses via recurrent attention network. Visual Intelligence , 1(1):18, Aug 2023.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

T. Von Marcard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and a moving camera. In Proceedings of the European conference on computer vision (ECCV) , pages 601-617, 2018.

S. Xu, Y.-X. Wang, and L.-Y. Gui. Diverse human motion prediction guided by multi-level spatial-temporal anchors. In European Conference on Computer Vision , pages 251-269. Springer, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a novel multi-stage training framework called Temporal Continual Learning (TCL) for Human Motion Prediction (HMP) to address the challenges of short-term and long-term predictions and the incorporation of prior information from past predictions into subsequent predictions. The Prior Compensation Factor (PCF) is introduced to compensate for the lost prior information, and an optimization objective is derived through theoretical derivation. The TCL framework can be easily integrated with different HMP backbone models and adapted to various datasets and applications. Extensive experiments on three HMP benchmark datasets demonstrate the effectiveness and flexibility of TCL.\n\nSTRENGTHS:\n+ Introducing the Temporal Continual Learning (TCL) framework, a multi-stage training framework that addresses the constraint between short-term and long-term prediction in Human Motion Prediction (HMP) and allows for better utilization of prior knowledge from short-term prediction to enhance the performance of long-term prediction.\n+ Introducing the Prior Compensation Factor (PCF) to mitigate the issue of forgetting information during multi-stage training, and deriving a more reasonable optimization objective through theoretical derivation.\n+ Exploring a new pipeline for the HMP task.\n\nWEAKNESSES:\n+ Writing:\n  + Without periods in lines 113-117.\n  + $P\\left(Z_{k} \\mid Z_{1} Z_{2} \\cdots Z_{k-1} \\theta\\right)$ -> $P\\left(Z_{k} \\mid Z_{1} Z_{2} \\cdots Z_{k-1}; \\theta\\right)$.\n  + **The following concepts are confusing: short+long, short only, short then shorr long. The explanation in Figure 1 is not easy to understand. I suggest providing a figure to illustrate the difference between the three concepts.** It is essential to your motivation.\n  + **Please provide more experimental settings about the toy example (Figure 1).**\n  + Methods part should be carefully written. Otherwise, it is somewhat confusing. \n+ It is not clear about the motivation for designing $\\alpha$s. The authors name it Prior Compensation Factors in Section 3,2. Why can it compensate for forgotten knowledge when leveraging prior knowledge? It is a scalar and why can it reflect so much knowledge of complex motions?\n+ For experiments:\n  + Datasets: I suggest authors provide the results on AMASS datasets, which will make your work more solid. \n  + Authors choose PGBIG  as the backbone. More choices and comparisons of backbones should be presented in the main result (not in ablation). This verifies the main contribution of the paper. \n  + I did not find the codes and demo videos in the supplementary. (Not necessary. If provided, more convincing. Pretrained models with inference codes are acceptable.)\n\nQUESTIONS:\n+ It is clear that \"our objective\" is between the \"actual objective\" and \"naive objective\". In Figure 2, is \"our objective\" close to the \"actual objective\"? Why isn't it closer to the \"naive objective\"? Any intuition or proof?\n+ Missing baselines or related work: [1, 2, 3, 4]. I would like to discuss with the authors about the choices of baselines. \n+ Efficiency and multi-stage pipeline. Recent researches [5] suggest predicting motions in one stage, which is easier to train. Will the multi-stage training be harder to tune or train? Will it be more time-consuming? I would like to discuss with the authors about it. \n+ It will be great if authors can discuss or provide zero-shot adaptation experiments on other datasets. \n\n\n\n**I would like to discuss with the authors according to the authors' rebuttal. The experiments are not sufficient and the presentation of the paper is not good enough. However, the theoretical insights are interesting. Therefore, I provide a weak accept score here. I will adjust my score according to the authors' response. I will carefully check the authors' responses. I would like to discuss with the authors with details of the paper and improve the quality of the papers jointly.**\n\n\n\n[1]: Zhong, Chongyang, et al. \"Spatio-temporal gating-adjacency GCN for human motion prediction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2]: Sofianos, Theodoros, et al. \"Space-time-separable graph convolutional network for pose forecasting.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n\n[3]: Bouazizi, Arij, et al. \"MotionMixer: MLP-based 3D Human Body Pose Forecasting.\"IJCAI 2022.\n\n[4]: Guo, Wen, et al. \"Back to mlp: A simple baseline for human motion prediction.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2023.\n\n[5]: Chen et al. \"HumanMAC: Masked Motion Completion for Human Motion Prediction.\" arXiv preprint arXiv:2302.03665 (2023).\n\n---\n\n\nI revise my rating to borderline accept. See [detail](https://openreview.net/forum?id=v0GzRLvVp3&noteId=FWQ456OsEQ).\n\nLIMITATIONS:\nAuthors discussed limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the continual learning insight into human motion prediction. By analysis of the performance relationship between the short and long-term prediction, a compensatory method is proposed in a multi-stage learning setting. On several widely-used benchmarks, the proposed method is cooperated with different backbones and methods and compared with previous methods. Some decent progress is shown according to the analyses.\n\nSTRENGTHS:\n+ Splitting the long-term prediction into multi-stage and using the continuous learning insight is interesting and non-trivial.\n\n+ The method proposal looks sound and designed well.\n\n+ A good comparison and discussion are given to support the proposed method.\n\nWEAKNESSES:\n- What is the additional cost of using the proposed method? Please discuss the efficiency and the other possible cost.\n\n- Lacking a vivid figure to illustrate the whole method pipeline before the detailed method introduction. Besides, the introduction part, especially the method description can be organized better and more logically.\n\n- L128-130: Though embedding the “knowledge” into the trained parameters seems very reasonable, this is just an intuitive discussion. A more detailed and clear explanation is essential to illustrate what is knowledge really, or what sign we can get to observe the knowledge utilized or not, to avoid a metaphysical or empirical discussion only.\n\n- Fig. 3: hard to read and discover the difference between methods.\n\n- Lacking direct and clear experiments to show the effectiveness of the continual learning design, e.g., better avoidance of forgetting, better balance of the shot and long predictions, etc.\n\n- typo: Fig. 1: lacking space before the (\n\nQUESTIONS:\n1. Though there are some \"traditional\" settings of the length definition of the long and short terms. I wonder if we change the 5 and 15 frames setting, is the analysis of Fig. 1 kept or changed? \n\n2. Learning rate in Eq. 9: choice discussion.\n\nLIMITATIONS:\nPlease add a discussion.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes to train human motion prediction networks by gradually increasing the prediction horizon.\nThis encourages the network to learn short-term predictions first and then leverage the learned to predict longer horizons.\nThe easy-to-hard curriculum makes the network learn more efficiently, as evidenced by the comparison to networks that learn all horizons at the same time.\nGiven the continual setting, forgetting prevention is needed and dealt with by the introduced prior compensation factors.\nExperiments are carried out on three major benchmarks, and effectiveness is observed compared with selected baselines.\n\nSTRENGTHS:\nThe paper is well-written. The idea of training sequence prediction with gradually increased horizons is well instantiated in the context of human motion prediction.\nThe motivation is also clearly conveyed by the ablation in Figure 1.\nBetter performance is achieved when compared with two recent methods.\nIt is good that some derivation is shown to arrive at the final combined loss of predictions at different horizons.\nThe adaptive scheme from the derivation shows better performance than a hand-crafted fixed set of weights.\n\nWEAKNESSES:\nThe comparison is a bit weak. Shall compare with more recent methods, for example, \"Back to MLP: A Simple Baseline for Human Motion Prediction, 2023.\"\nAlso, since the method proposed is not backbone-dependent, so more evaluation is needed, for example, transformer-based architectures.\nMoreover, there are some questionable parts in the derivation, even though it seems that the final result may not be heavily affected.\n\nQUESTIONS:\n1) In lines 114-117, all predictions are conditioned on X_1:T_h, however, in Eq. 2, the conditions consist of previous predictions.\n2) Eq. 3 assumes that the actual distribution is a degenerated distribution plus a positive bias, why is the bias always positive? This seems not reasonable as the degenerated one could be larger than the actual one.\n3) Eq. 4 shows that under the bias assumption, the final loss can be treated as a linear combination of the loss terms for different horizons, however, in Eq. 5 the weights for each term is changed from \\alpha to 1-\\alpha, is this really valid?\n4) Are the terms involving only \\alphas really optimized by Eq. 8?\n5) Not sure whether the presentation/derivation is really necessary given that a lot of approximation is needed, yet what we want is just a loss that weights the prediction error at different horizons, hopefully the adaptiveness is the key?\n\nLIMITATIONS:\nThe limitation is addressed in the broader impact section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to enhance human motion prediction. The main contributions of this paper are:\n1. The paper presents a multi-stage training strategy named Temporal Continual Learning to incorporate the learning of both short-term prediction and long-term prediction.\n2. The paper introduces Prior Compensation Factor to better preserve prior information during the process of Temporal Continual Learning.\nThese optimizations are given through theoretical derivation.\n\nSTRENGTHS:\nThe paper has several strengths:\n1. Overall, the paper is well-written with a clear and well-motivated introduction. The storyline to leverage the prior knowledge learned from short-term inputs to facilitate long-term predictions makes sense.\n2. Secondly, the proposed method is flexible and demonstrates good performance when applied to different human motion predictors, outperforming state-of-the-art on different datasets.\n\nWEAKNESSES:\nThe paper could benefit from a more thorough discussion of related work on Human Pose Prediction, such as in L23-24, transformer architectures include\n- PoseGPT: Quantization-Based 3D Human Motion Generation and Forecasting. ECCV 2022\n\n\tand graph convolution networks include\n\n- Diverse Human Motion Prediction Guided by Multi-Level Spatial-Temporal Anchors. ECCV 2022\n\nThe key insight is supported by L32-42 and Figure 1. I'm wondering if the comparison can be represented by a clearer illustration, e.g. using figures to explain different terms such as “short+long” and “short then short + long”. The current version looks a bit rough.\n\nQUESTIONS:\n1. typo in L119, should be T_{Z_{K-1}}\n2. Figure 4 in the ablation study is a bit confusing. What is ablated here? Is the alpha value the same across different experiments? \n3. The implementation defines a specific partition of sequence for the multi-task. I think it would be interesting to see the ablation on different partitions, e.g. different number of sub-tasks\n\nLIMITATIONS:\nIt would be helpful if the authors could provide videos to demonstrate the quality of generated human motion. I wonder if the multi-task learning for different motion segments will cause the discontinuity", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the common trade-off between the short- and long-term prediction quality of 3D human motion prediction models. The proposed training technique improves the performance of the underlying models both on short- and long-term prediction horizons, where the models generally prioritize one and suffer from the other. It is a multi-stage training scheme resembling curriculum learning with an increasing prediction horizon. The prediction horizon is split into consecutive chunks of frames (i.e., stages). The model is trained iteratively for every stage by initializing the weights from the previous stage and considering all the stages so far. Different from curriculum learning, the proposed technique aims to preserve and leverage the prior information of the past stages by alleviating catastrophic forgetting in the future stages. To do so, the paper incorporates the \"Prior Compensation Factor\" in the training objective. Intuitively, for every training sample, the model predicts a weight parameter that dynamically distributes the total loss weight across stages. The proposed technique is evaluated using three different architectures. Experiments on various benchmarks show that it is highly effective.\n\nSTRENGTHS:\n**originality**\nThe paper addresses a common and often neglected problem in the 3D human motion prediction domain. In fact, most prior works prefer to ignore the short- and long-term prediction trade-off and present results with separate model configurations. I find this effort valuable.\n\n**quality**\nThe proposed technique is well-motivated and sound. I have some concerns regarding the theoretical analysis (see below), but the experimental results are solid.\n\n**clarity**\nThe motivation is clear. The paper provides enough background to understand the problem setting. However, it is not straightforward to intuitively understand what the proposed technique does.\n\n**significance**\nI find the paper interesting and useful for the community. The experiments show that the proposed technique could improve the performance of the out-of-the-box models.\n\nWEAKNESSES:\nThere is a potential problem with lemma 3.1. It expects `b` to be between 0 and 1, a likelihood value from a probability mass function. For the proposed training objective (Eq. 7) to be held, the loss functions should be chosen carefully. In this work, it is implemented as a MSE loss (Eq. 6 and 7), which does not have an upper-bound. How about implementing the objective as a log-likelihood with a Gaussian output model? It may have negative values. \n\nI had to spend some time getting an intuitive understanding of Eq. 7 (no, the “An intuitive explanation” section does not help). It boils down to a dynamic weighting scheme where the network learns to distribute the loss weights across prediction horizons (i.e. stages). I think this could be explained better in the paper. In fact, the training dynamics is very straightforward. If the MSE loss is high, then the predicted alpha should be higher to optimize the objective. We can say that the model learns to “assess the difficulty of the task”. Naturally, alpha values get higher in the future stages as it becomes more difficult. If you run the following code snippet, you get a very similar plot to the one in Fig. 4 ($\\alpha$ values at different stages). It gives you the optimal $\\alpha$ value for different MSE values. I kindly ask the authors to clarify if I am wrong or share their thoughts on this analysis.\n\n```\nimport matplotlib.pyplot as plt\nfrom scipy.optimize import minimize_scalar\nimport math\nimport numpy as np\n\ndef objective(mse):\n    return lambda alpha: (1 - alpha) * mse + (1-alpha) * math.log(1-alpha) + math.log(1 + alpha)\n\nmse_values = np.arange(0, .5, 0.01)\nbest_alphas = []\nbest_res = []\nfor mse in mse_values:\n    opt_res = minimize_scalar(objective(mse), bounds=(0, 1), method=\"bounded\")\n    best_alphas.append(opt_res.x)\n    best_res.append(opt_res.fun)\n\nplt.plot(mse_values, best_alphas)\nplt.xlabel(\"MSE values (i.e., stages)\")\nplt.ylabel(\"Alpha values\")\nplt.show()\n```\nThe authors can also try setting the $\\alpha$ values for the “HC” ablation using this code snippet.\n\nAn ablation on the number of stages is missing. Similarly, its cost is not discussed. While it improves the performance of the underlying model, it introduces a significant overhead at training time. \n\nIt would be interesting to see the evolution of the performances on the previous stages as training continues on the future stages. A triangular matrix reporting the performance of stage K after training every stage > K.\n\nInstead of following a dynamic approach (i.e., predicting an $\\alpha$ per training sample), would it still work if a single, stage-wise $\\alpha$ variable was trained? \n\nThis is merely a suggestion for the presentation. I think it would be more interesting to report the performance for the pairs (baseline, baseline + ours) in all benchmarks (as in Table 5).\n\nQUESTIONS:\nI’ve raised my concerns and asked my questions in the weaknesses section.\n\nLIMITATIONS:\nThe paper does not have a limitations section. I think it is clear that the proposed technique significantly increases training complexity.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}