Future Motion Dynamic Modeling via Hybrid Supervision for Multi-Person Motion Prediction Uncertainty Reduction

Authors: yan  zhuang
yanlu  cai
weizhong  zhang
cheng  jin
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-14 05:29:58

Abstract: Multi-person motion prediction remains a challenging problem due to the intricate motion dynamics and complex interpersonal interactions, where uncertainty escalates rapidly across the forecasting horizon. Existing approaches always overlook the motion dynamic modeling among the prediction frames to reduce the uncertainty, but leave it entirely up to the deep neural networks, which lacks a dynamic inductive bias, leading to suboptimal performance. This paper addresses this limitation by proposing an effective multi-person motion prediction method named Hybrid Supervision Transformer (HSFormer), which formulates the dynamic modeling within the prediction horizon as a novel hybrid supervision task. To be precise, our method performs a rolling predicting process equipped with a hybrid supervision  mechanism, which  enforces the model to be able to predict the pose in the next frames based on the (typically error-contained) earlier predictions. Addition to the standard supervision loss, two  self and auxiliary supervision mechanisms, which minimize the distance of the predictions with error-contained inputs and the predictions with error-free inputs (ground truth) and guide the model to make accurate predictions based on the ground truth, are introduced to improve the robustness of our model to the input deviation in inference and stabilize the training process, respectively. The optimization techniques, such as stop-gradient, are extended to our model to improve the training efficiency.

Main paper: 
 introduction: 
 Human motion prediction is a crucial task, which predicts future motion trends based on the previous observations, and has wide applications in the fields of 3D character animation [5, 7, 25, 44, 46] , surveillance systems [6, 22, 52] , and autonomous driving [8, 24, 50] . Thanks to the Transformer's powerful capability in modeling sequential relationships, recent approaches [33, 42, 45] , employing Transformers for multi-person interaction modeling and implicitly temporal relationship modeling, achieve promising results.However, due to the significant motion dynamic uncertainty within the forecasting horizon, i.e., the steps with motion to be predicted, multi-person motion prediction remains a challenging problem. These uncertainties mainly arise from the following two sources. One is the intricate and varying interactions among the individuals. Figure 1b (1) shows that the left wrist joint of the player in red obstructs several joints in the torso area of the player in blue, while the left wrist joint of the player in blue obstruct the hip joint joint of the player in red. The other is the perplexing evolution patterns in human motions. For example, the motion characteristics of different keypoints differ greatly. As shown in Figure 1b (2), the movement range of the player's knee joint is much smaller than that of the ankle joint. More importantly, the uncertainties can grow rapidly over the forecasting horizon. In this work, we argue that explicitly modeling the motion dynamic within the forecasting horizon should be an indispensable module to reduce the uncertainty and finally improve the motion prediction accuracy, which is always overlooked in the previous studies as they usually leave it entirely up to the decoder neural networks.To address the above issues, we propose a novel Transformer based framework Hybrid Supervision Transformer (HSFormer). Our key idea is to explicitly model the motion dynamics in the forecasting horizon by constructing the interdependence among the adjacent frames. We first propose a rolling predicting process, which predicts the poses in the next frames based on the (typically error-contained) earlier predictions. Notice that training the model under this framework is highly nontrivial. For example, with vanilla optimization algorithms, the model would be susceptible to the accumulated prediction errors in the previous prediction. Therefore, addition to the standard supervision loss, which minimizes the distance between the ground truth and our rolling prediction output, two additional supervision mechanisms, i.e., the self and auxiliary supervision mechanisms, are introduced to improve the robustness of our model to the input deviation in inference and stabilize the training process, respectively. To be precise, the self supervision mechanism minimizes the distance of the predictions with error-contained inputs and the predictions with error-free inputs (ground truth), while the auxiliary supervision mechanism guides the model to make accurate predictions based on the ground truth. These standard, self and auxiliary supervision mechanisms together form our novel hybrid supervised training framework. Additional optimization techniques, such as stop-gradient and parameter sharing, are extended to our HSFormer to improve the training efficiency. Additionally, in order to reduce the uncertainties arising from the intricate and varying interactions, HSFormer utilizes a feature learning component, namely Spatio-Temporal Encoder (STE) block, for fine grained modeling. The STE block consists of the Spatial Joint Encoder (SJE) module and the Temporal Joint Encoder (TJE) module, which is responsible for capturing the relationships between body joints within each individual frame and the dynamic evolution of the joints, respectively.We evaluate our method on 5 different datasets with varying scales and complexities. The results demonstrate the significant superiority of our HSFormer over the state-of-the-art methods. Notably, our method outperforms the current state-of-the-art approaches for both short-and long-term predictions, with 1%∼23% accuracy improvement for the short-term (0s∼1.0s) and 2%∼8% accuracy improvement for the long-term prediction (1.0s∼3.0s).The main contributions of this work can be summarized as follows: 1) We design a rolling prediction scheme to effectively reduce the prediction uncertainty, which explicitly models the motion dynamic within the forecasting horizon by learning the interdependence between the motions in adjacent frames. 2) As training the model under our rolling prediction scheme is highly nontrivial, we develop a hybrid supervision mechanism to guarantee the training efficiency and the generalization capability of the learned model. 3) We introduce the STE block, specifically designed to address the uncertainty arising from interaction details. STE enables a meticulous learning of the body joint relations in each frame and the global temporal correlation of each joint. 4) The experimental results on multiple multi-person motion datasets demonstrate that the proposed model significantly outperforms state-of-the-art methods.We reconstruct the decoder, referred to as the HScoder block, which is designed to guide the model to learn the dynamics of the predicted frames through a rolling prediction process to reduce uncertainty. As depicted in Figure 3 , this process divides the forecasting horizon into multiple intervals and predicts the motions sequentially. To be precise, let 𝑓 be the length of the divided predicton interval, our rolling process would be comprised of 𝑑 = 𝐹 /𝑓 prediction rounds. In the 𝑙 𝑑 -th round with 𝑙 𝑑 ∈ {1, . . . , 𝑑 }, our HScoder first constructs a body query token 𝑄 𝑙 𝑑 ∈ R 𝑁 ×𝐽 ×𝐶 based on the neighbor motion dynamics of the recent ℓ frames, denoted as N 𝑙 𝑑 ∈ R 𝑁 ×ℓ ×𝐽 ×𝐶 , that is,EQUATIONwhere 𝐶𝑜𝑛𝑣1𝑑 (•) is a 1D Convolution layer with kernel size of ℓ. Then HScoder leverages a Transformer Decoder to predict the future pose features of 𝑓 frames, denoted as ŷ𝑙 𝑑 ∈ R 𝑁 ×𝑓 ×𝐽 ×𝐶 , based on 𝑄 𝑙 𝑑 and the guidance of the history context H 𝑙 𝑑 , that is,EQUATIONwhere FC(•) is linear layer to convert the single token into 𝑓 frames.Our rolling prediction scheme takes F 𝐸𝑚𝑏𝑒𝑑 and F 𝐿 2 𝑇 𝐽 𝐸 to obtain the initial neighbor motion dynamics N 1 and history context H 1 , respectively. And then leverage the previous prediction ŷ𝑙 𝑑 together with previous N 𝑙 𝑑 and H 𝑙 𝑑 to build the new ones, i.e., N 𝑙 𝑑 +1 and H 𝑙 𝑑 +1 . Specifically,EQUATIONwhere 𝑆𝑙𝑖𝑐𝑒 (•) is to get the last ℓ-frame feature. As for the history context H 𝑙 𝑑 , we re-encode the prediction ŷ𝑙 𝑑 into the feature space of the history context by employing a STE module, that is,EQUATIONA TJE module is adopted after concatenation to extract the temporal relationship between previous history context and the new prediction. Thus, it can be formulated asEQUATIONwhereH 𝑙 𝑑 ∈ R 𝑁 × (𝑇 +(𝑙 𝑑 -1) * 𝑓 ) ×𝐽 ×𝐶 .After 𝑑 rounds rolling prediction, all the prediction feature ŷ𝑙 𝑑 ∈ R 𝑁 ×𝑓 ×𝐽 ×𝐶 will be concatenated and projected to obtained the predicted pose sequence Ŷ , that is, Ŷ𝑇 +1:𝑇 +𝐹 = IDCT FC 𝐶𝑜𝑛𝑐𝑎𝑡 ŷ1 , ŷ2 , . . . , ŷ𝑙 𝑑 ,where the linear layer FC(•) is adopted to project the feature into the pose sequence, and IDCT(•) stands for Inverse Discrete Cosine Transformation. Finally, Ŷ𝑇 +1:𝑇 +𝐹 is transformed back to X𝑇 +2:𝑇 +𝐹 +1 . 
related work 2.1 single-person pose prediction: 
 Single-person motion prediction [10, 11, 20, 22, 27, 37, 39, 41] involves the task of predicting the future motion of an individual based on historical motion. Compared to multi-person motion, single-person motion prediction has relatively simpler spatiotemporal dependencies and can be effectively captured using model architectures based on Recurrent Neural Networks (RNNs) [12-15, 17, 19, 26, 29, 32, 36, 49] . RNNs have been proven successful in sequence-to-sequence prediction tasks [21] . However, due to frame-by-frame prediction, issues of discontinuity and error accumulation often arise as the model independently generates predictions for each frame without considering the continuity with preceding and subsequent frames. To address the problem of error accumulation when using RNN-based models for long-term prediction, some research [4, 23] start exploring the use of fully connected or convolutional networks to better capture long-term dependencies and reduce error accumulation. In addition to RNNs, temporal convolutional networks have also shown promising results in modeling long-term motion [3, 4, 23, 28] . For instance, [23] construct a convolutional sequence-to-sequence model for human motion prediction. Unlike previous chain-based RNN models, the hierarchical structure of convolutional neural networks enables them to naturally model and learn spatial dependencies and longterm temporal dependencies. However, human motion is influenced by the surrounding environment, making it inherently uncertain, which becomes more evident in long-term prediction. Recent research begin to address this issue by jointly predicting human pose and world coordinate trajectories [6, 43, 47, 48, 51] . For example, [6] leverage scene context to tackle the challenges of long-term prediction. Currently, single-person motion prediction has achieved promising results, while multi-person motion prediction is more complex due to factors such as crowd interaction. Our work extends to the simultaneous prediction of multi-person motion, including 3D pose and trajectories.Multi-person motion prediction aims to predict the movement among multiple individuals in a scene [31, 35, 38] . This task requires considering interactions between individuals, making it more complex in a multi-person context compared to single-person motion prediction. The challenges arise because individual movements are influenced not only by their own dynamics but also by dynamic interactions with other individuals. To solve above issues, [1] introduce multi-person motion prediction, proposing a novel approach based on graph attention networks to model the dynamics of trajectories and poses, simulating interactions between individuals in both input space and output space. [42] introduces local-range encoders for individual motions and global-range encoders for social interactions. [16] introduce a cross-interaction attention mechanism that utilizes the historical information of two individuals, learning to predict cross-dependencies between two pose sequences. [33] introduce TBIFormer, proposing a novel Social Body Interaction Multi-Head Self-Attention (SBI-MSA) that learns dynamic body part interactions inter and intra individuals, capturing complex interaction dependencies. However, these methods predict all future frames directly from the input sequence, neglecting the dynamic changes within the predicted frames, which increases the uncertainty in the prediction process, especially in long-term predictions. In this paper, we propose a new multi-person motion prediction model framework based on Transformer to address the aforementioned issue. 
preliminaries 3.1 problem formulation: 
 Given the historical motion trajectory of multiple individuals, our objective is to predict the future motion for these individuals. Formally, with 𝑁 individuals in the scene, the history motion trajectory of the 𝑛-th individual with 𝑇 + 1 time steps can be presented as𝑋 𝑛 1:𝑇 +1 = [𝑥 𝑛 1 , 𝑥 𝑛 2 , . . . , 𝑥 𝑛 𝑇 +1 ] with each 𝑥 𝑛 𝑖 ∈ R 𝐽 ×3being the coordinates of 𝐽 skeleton joints. Following the related work [42] , we transform𝑋 𝑛 1:𝑇 +1 into 𝑌 𝑛 1:𝑇 = [𝑦 𝑛 1 , . . . , 𝑦 𝑛 𝑇 ]by taking the difference between the adjacent entries to expose the motion trends to the model, i.e., 𝑦 𝑛 𝑖 = 𝑥 𝑛 𝑖+1 -𝑥 𝑛 𝑖 , 𝑖 = 1, . . . ,𝑇 . Our goal is to predict the 3D pose sequence of the next 𝐹 time steps, i.e., Ŷ𝑛 𝑇 +1:𝑇 +𝐹 = [ ŷ𝑛 𝑇 +1 , . . . , ŷ𝑛 𝑇 +𝐹 ], and transform it back to X𝑛 𝑇 +2:𝑇 +𝐹 +1 with 𝑛 = 1, . . . , 𝑁 .Given a 3D pose sequence 𝑌 ∈ R 𝑁 ×𝐽 ×𝑇 ×3 with N persons, J joints, and T frame as input, the vanilla pipeline apply a Discrete Cosine Transformation (DCT) [2] to encode motion into the frequency domain, creating a more compact representation. This representation is then projected into a high-dimensional feature space to obtain the embedded feature F 𝐸𝑚𝑏𝑒𝑑 ∈ R 𝑁 ×𝐽 ×𝑇 ×𝐶 , where 𝐶 is the feature dimension. Subsequently, F 𝐸𝑚𝑏𝑒𝑑 is fed into an encoder to learn sequence features followed by a decoder for predicting future sequences. Finally, the pipeline utilizes a fully connected (FC) layer and an Inverse Discrete Cosine Transformation (IDCT) to obtain the future motion Ŷ𝑇 +1:𝑇 +𝐹 for each individual. In this work, we reconstruct both the encoder and decoder block to reduce the prediction uncertainty, which are the main contributions of this paper. To be precise, we propose  
fine-grained correlation learning: 
 For the convenience of presenting our method HSFormer, we first introduce our fine-grained correlation learning block STE as it is integrated into both the encoding and decoding components. STE block is designed to assist the feature learning to reduce the uncertainties arising from the intricate and varying interactions among the individuals. As shown in Figure 2 , it consists of a SJE module and a TJE module, which learn the joint relations in each frame and the global temporal correlation for each joint in different frames, respectively. Spatial Correlation Learning. SJE aims to learn the spatial relationships across body joints for all individuals via attention mechanism. Suppose the SJE module consists of 𝐿 1 layers, we embed F 𝐸𝑚𝑏𝑒𝑑 with the learnable position embedding 𝐸 𝑠𝑝𝑜𝑠 ∈ R (𝑁 ×𝐽 ) ×𝐶 to obtain the embedded feature F 0 𝑆 𝐽 𝐸 ∈ R 𝑇 × (𝑁 ×𝐽 ) ×𝐶 before the first layer of SJE module. SJE module takes 𝑁 × 𝐽 tokens of size C from a certain frame as inputs and models relationship across all joints. After this, tokens will have knowledge about the joints of other individuals in the same frame. The output of the final layer can be represented as F 𝐿 1 𝑆 𝐽 𝐸 . Temporal Correlation Learning. Due to the diversity of joint movements, it is crucial to learn the independent characteristics of each joint. The TJE module treats the motion trajectory of each joint as an independent unit, capable of identifying the unique motion characteristics of each joint. We first reshape the output feature F 𝐿 1 𝑆 𝐽 𝐸 from SJE into R (𝑁 ×𝐽 ) ×𝑇 ×𝐶 and then combine it with the learnable temporal positional encoding 𝐸 𝑡𝑝𝑜𝑠 ∈ R (𝑁 ×𝐽 ) ×𝑇 ×𝐶 to obtain the feature F 0 𝑇 𝐽 𝐸 . It is then fed into TJE module to learn the contextual dependencies for each joint in parallel. After processed by our TJE module, F 0 𝑇 𝐽 𝐸 is transformed into the final output F 𝐿 2 𝑇 𝐽 𝐸 , where 𝐿 2 is the number of TJE layers.  
hybrid supervision mechanism: 
 Note that it is indeed not trivial to train such a network under rolling prediction framework, since the errors contained in previous predictions can accumulate over time, thereby adversely affecting subsequent predictions and potentially leading to the failure of training. This issue is particularly severe in the early training stage. To solve the above issue, we propose a gt-augment branch together with a hybrid supervision mechanism, which provides a more distinct optimization path unaffected by the accumulation of errors, thereby enhancing the training stability. The details are presented as follows.The gt-augmented branch is designed for training process only and will be removed in inference. It shares the same architecture and weights with the main branch to make the model in two branches be consistent, while it takes ground truth rather than previous predictions as inputs to predict the next frames. Specifically, it replaces the prediction feature ŷ𝑙 𝑑 in Eqn. (3) For simplicity, we let 𝑋 , X and Xgt be the ground truth, the output of the main branch and the gt-augmented branch, respectively. By denoting MSE-loss to beEQUATION, we train our two-branched model with the following hybrid supervision mechanism:• Standard Supervision. It is adopted to enforce the model to be able to make accurate predictions even with typically error-contained previous predictions as inputs and the loss takes the form of L 𝑒𝑟𝑟𝑜𝑟 = L MSE (𝑋, X ). • Auxiliary Supervision. To avoid the potential failure of training caused by error accumulation, especially in the early training stage, we adopt the following auxiliary supervision loss to guide the model to make accurate predictions taking the ground truth poses in the previous frames as inputs. That is L 𝑔𝑡 = L MSE (𝑋, Xgt ). • Self Supervision. A self-supervised contrastive loss L 𝑐𝑜𝑛 is adopted to minimize the distance of the prediction with errorcontained inputs and the prediction with error-free inputs, enhancing the robustness of the model to the prediction errors in the previous frames, that is,L 𝑐𝑜𝑛 = L MSE ( X, 𝑠𝑔( Xgt )),where 𝑠𝑔(•) stands for stop-gradient operation [9] .Thus, the overall loss becomes:EQUATIONwhere the parameters 𝜆 ∈ (0, 1] and 𝛾 > 0 are adopted to tune the weights of the above three losses during training. We observe that our model is insensitive to 𝛾 and we can achieve good performance with a common value 𝛾 = 1 in all our experiments. For 𝜆, we gradually increase it from 0 to 1 during the training process for two reasons: 1) In the early training stage, a small 𝜆 is preferred as the inputs of our first branch, i.e., the previous predictions, always contain pronounced errors, making the standard supervision mechanism ill posed; 2) Unlike our second branch does in training, we cannot make predictions by taking the ground truth as inputs in inference. Therefore, we require 𝜆 to be close to 1 in the end of training to gradually discard our second branch to guarantee the consistency between the training and inference process. 
experiments 6.1 implementation details: 
 Our experiments use PyTorch framework on two Nvidia GeForce RTX 3090 GPUs. We train with the batch size of 32 for 150 epochs. The learning rate is set to 0.0003. The number of stacked encoder layers, 𝐿 1 and 𝐿 2 , are both set to 2, while the transformer decoder is stacked with 3 layers. The kernel size of the 1D Convolution layer ℓ is set to 10. For predictions within 1 second, 𝑑 is set to 5; for predictions within 1 to 3 seconds, 𝑑 is set to 9. 𝜆 1 = 𝑒𝑝𝑜𝑐ℎ 𝑖 /150, where "𝑒𝑝𝑜𝑐ℎ 𝑖 " represents the current epoch number. 1.0 2.0 3.0 Average 1.0 2.0 3.0 Average 1.0 2.0 3.0 Average 1.0 2.0 3.0 Average MRT [42] 148 Dataset. Following the state-of-the-art method TBIFormer [33] , we conduct our experiments on five widely-used datasests with varying scales and complexities. CMU-Mocap dataset consists of a training set with 6,000 sequences and a test set with 800 sequences. MuPoTS-3D (2 to 3 persons) [30] is a dataset of 3D human poses collected by a multi-view unmarked motion capture system. Mix1 (6 persons) and Mix2 (10 persons) are composed by blending data from CMU-Mocap, 3DPW [40] , and MuPoTs-3D [30] datasets. On the above datasets, we follow TBIFormer [33] for fair comparison, i.e., the model takes 50 frames (2.0s) as inputs to predict 25 frames (1.0s) for short-term prediction, while the model takes 15 frames as inputs (1.0s) to predict 45 frames (3.0s) for long-term prediction. The MI-Motion dataset is a large-scale dataset, which includes 167k frames of multi-person skeleton poses and covers five different everyday activity scenarios: indoor, park, street, special locations, and crowded scenes. On MI-Motion dataset, the model takes 25 frames (1.0s) as inputs to predict 25 frames (1.0s), which is consistant with SocialTGCN [34] . Baseline. We choose the most recent state-of-the-art multi-person prediction methods as baselines, i.e., MRT [42] , TBIFormer [33] , SocialTGCN [34] and JRFormer [45] . Metrics. We evaluate our results with following three metrics, which are consistent with those in TBIFormer [33] . The detailed definitions of these metrics are given below: 1) JPE. We use the Mean Per Joint Position Error (MPJPE) [18] metric to measure the poses of all the individuals, including body trajectory:EQUATION, where 𝑋 𝑛 𝑖,𝑗 and X𝑛 𝑖,𝑗 are the ground truth and the prediction of the 𝑗-th joint for the 𝑛-th individual in the 𝑖-th frame.2) APE. We remove global movement and use Aligned mean per joint Position Error (APE) to measure pure pose position error:APE(𝑋 , X ) = 1 𝑁 × 𝐹 × 𝐽 𝑁 ∑︁ 𝑛=1 𝑇 +𝐹 +1 ∑︁ 𝑖=𝑇 +2 𝐽 ∑︁ 𝑗 =1 𝑋 𝑛 𝑖,𝑗 -𝑋 𝑛 𝑟 -X 𝑛 𝑖,𝑗 -X 𝑛 𝑖,𝑟 2 ,where 𝑋 𝑛 𝑖,𝑟 and X𝑛 𝑖,𝑟 are the ground truth and prediction of the root joint for the 𝑛-th individual in the 𝑖-th frame.3) FDE. In order to evaluate the global motion of all individuals, we introduce the Final Displacement Error (FDE), that isFDE 𝑋 𝑇 +𝐹 +1 ,𝑟 , X𝑇 +𝐹 +1 ,𝑟 = 1 𝑁 𝑁 ∑︁ 𝑛=1 𝑋 𝑛 𝑇 +𝐹 +1 ,𝑟 -X𝑛 𝑇 +𝐹 +1 ,𝑟 2 ,where 𝑋 𝑛 𝑇 +𝐹 +1 ,𝑟 and X𝑛 𝑇 +𝐹 +1 ,𝑟 represent the ground truth and prediction of the root joint at the last frame, i.e., the (𝑇 + 𝐹 + 1)-th frame, for the 𝑛-th individual.Quantitative Results. Table 1 presents the results of JPE, APE and FDE on the datasets CMU-Mocap, MuPoTS-3D, Mix1 and Mix2 (cross-dataset validation) in short term (0.2s∼ 1.0s). Our model provides rolling guidance for the prediction sequence of multiple frames and establishes long-term temporal dependencies. Compared to the previous best method, our model demonstrates an improvement in the JPE metric ranging from 1% to 23% in the four datasets. Furthermore, Table 2 validates the effectiveness of our model over longer time intervals (1.0s ∼ 3.0s). Compared to the previous best method, our model exhibits an improvement of 2% ∼ 8%. Additionally, as shown in Table 3 , we evaluate the performance of our model on the large-scale dataset MI-Motion. Our model outperforms previous model, with improvement of 3% ∼ 22%, in terms of the JPE metric across five different scenes. Qualitative Results. Figure 5 shows our visualization results compared to the baselines and ground truth. TBIFormer [33] demonstrates the trend of converging to a static pose in long-term predictions, with less pronounced motion amplitudes. JRFormer [45] does not perform well in learning the climbing pattern, as indicated by the red and black boxes in the figure . In contrast, our model generate dynamic and accurate prediction results. Compared to other methods, our approach is closer to the ground truth.We further conduct extensive ablation studies on CMU-Mocap to investigate the contribution of technical components in CONSFormer, with results given in Table 4 and Table 5 .Effectiveness of Rolling Design. The purpose of Rolling Design is to provide rolling guidance for the prediction sequence of multiple frames and establish long-term temporal dependencies. As illustrated in the first row of Effectiveness of Hybrid Supervision. The purpose of Hybrid Supervision Mechanism is to reduce the accumulation and propagation of errors in each rolling prediction. We employ the Hybrid Supervision Mechanism by introducing ground truth as the predicted frames. When we remove Hybrid Supervision, the performance drops substantially as in the second row of Table 5 . Apparently, our model equipped with Hybrid Supervision can enhance the model's robustness to noise. Additionally, to verify the importance of the hybrid supervision in our framework, we compare the training loss and test loss on the CMU-Mocap dataset under three different training conditions with: hybrid supervision, i.e., L ℎ𝑦𝑏𝑟𝑖𝑑 , standard supervision only, i.e., L 𝑒𝑟𝑟𝑜𝑟 and auxiliary supervision only, i.e., L 𝑔𝑡 . As shown in Figure 4 , it can be observed that the final result with hybrid supervision (0.6) is much smaller than the result with standard supervision (0.7) and auxiliary supervision (1.7) only, indicating the effectiveness of the hybrid supervision.Effectiveness of SJE module. SJE is designed to learn the spatial relationships between all joints. If the SJE is removed, the performance drops substantially as shown in the third row of Table 5 .Effectiveness of TJE module.TJE is designed to learn the independent variations of each joint over time and captures the motion characteristics of each joint. As shown in the last row of Table 5 , if the TJE is removed, the experimental results become worse.We visualize the attention score between individuals' query and key features in the attention mechanism in Figure 6 . The figure shows the motion of two people, in the left figure, we can see person1 and person2 hugging each other, with higher attention weights on their hand and neck joints. In the right figure, as person1 takes person2's hand, the hands of person1 and person2 have higher attention weights. 
conclusion: 
 In this paper, we present a transformer-based framework designed to effectively reduce uncertainty for multi-person pose prediction. We first introduce Hybrid-Supervised coder (HScoder) block that provides rolling guidance for multi-frame prediction. We also develop a hybrid supervision mechanism to ensure the training efficiency and the robustness of the final learned model to the accumulated prediction errors. In addition, we integrate the STE block into HScoder for fine-grained modeling to learn spatiotemporal correlations. Experiments demonstrate that our method outperforms state-of-the-art methods on multiple motion datasets. 
 
 Appendix:  


Decision: The paper addresses multi person motion modelling. Reviews are mixed with one negative score, lowered after the rebuttal, one positive score raised and one kept. The use of teacher forcing does not seem to invalidate the approach and regarding novelty two reviewers are recognizing the value of the proposed approach.

Reviews: review: paper_summary: paper_summary: The authors proposed a Hybrid Supervision Transformer with hybrid supervision mechanism to address the problem of multi-person motion prediction.
strength_weakness: strength_weakness: ### Evaluation:

The quantitative results presented in the paper look pretty good.

### Technical Correctness:

The technical details sounds correct in the main paper.
limitations: limitations: ### Novelty

(1) The STE block looks to be a product of the stitching of TBIFormer[1] (TBPM), motion-transformer[2] (spatial-temporal attention), which contributes a little and be over-claimed in the Intro section.

(2) I am very confused about the motivation behind the Auxiliary Supervision. One of the biggest challenges of the prediction problem is resolving uncertainty. Leaking ground-truth directly in early stage feels like it's all about a trick to fit datasets. Please explain your motivation for doing this. Did you test this stragety in inference stage for the unseen interaction or situation (may adopt cross-dataset validation).

### Experiments:

Although quantitative results look competitive, there are no more qualitative results on the other datasets (except for MI-Motion), even in the appendix.

### Writing and Drawing:
- Abstract and some sections need further rewording.
- The Figure 2 looks distracting and contains too much elements. I recommend authors to redraw it and improve its resolution. 


### Conclusion:

Overall, some of the ideas are poorly motivated and have limited contributions. Therefore, I am inclined to reject this paper.

*Reference:*
>[1] Trajectory-Aware Body Interaction Transformer for Multi-Person Pose Forecasting;
>[2] A Spatio-temporal Transformer for 3D Human Motion Prediction;
paper_summary: paper_summary: This paper focuses on the multi-person human motion prediction. It designs a novel spatial-temporal Transformer module that explicitly  models the motion dynamics by constructing the interdependence among the adjacent frames. Furthermore, it proposes a hybrid supervision mechanism that effectively reduce the overfitting problem.
strength_weakness: strength_weakness: - The proposed model is novel and effective, the emphasis on motion dynamic modeling is well motivated and is proven to be effective.

- The concept of hybrid supervision is both intriguing and technically sound. It has the potential to introduce fresh perspectives to the community

- The experiments are comprehensive and solid.
limitations: limitations: - The writing should be further improved, the formula definition and mathematical notations are quite messy. For example, in Eq (1), how the neighbor motion dynamics $N^{l_d} \in \mathbb{R} ^{N \times l \times J \times C}$ construct the query $Q^{l_d} \in \mathbb{R} ^{N \times J \times C}$ with a 1-D conv? Applying a  Conv1d will change the length of tensor dimensions, and also the last dimension should be the length, not the channel. In addition, the definition of neighbor frames $N^{l_d}$ in Eq (3) and history context $H^{l_d}$ in Eq (4) should be introduced before Eq (1). Same problem appears in Section 5.2, some equations are not even with an index number 

- I'm wondering how the "hybrid supervision mechanism" compares with standard data augmentation technics in motion prediction, such as motion sequence reverse https://arxiv.org/abs/2110.07495

- Since this paper focuses on multi-person motion prediction, I'm curious how this model performs on multi-person dataset with high interaction, such as ExPI dataset in ref [15].

- Typos. $X_r^n$ -> $X_{i, r}^n$ in APE definition.

- This work present a unimodal application, i.e. predicting the 3d human motion skeleton based on historical observation, which may not be very consistent with the theme of ACM MM
paper_summary: paper_summary: The manuscript "Future Motion Dynamic Modeling via Hybrid Supervision for Multi-Person Motion Prediction Uncertainty Reduction" presents a novel approach to the problem of multi-person motion prediction. The authors introduce the Hybrid Supervision Transformer (HSFormer), a method that addresses the challenge of uncertainty in predicting future movements of individuals in a group. The proposed model incorporates a rolling prediction process and a hybrid supervision mechanism to enhance the robustness of predictions and mitigate the accumulation of errors. The manuscript also introduces a Spatio-Temporal Encoder (STE) block to capture fine-grained spatio-temporal correlations, which is crucial for improving prediction accuracy. The method has been evaluated on multiple datasets and has demonstrated state-of-the-art results in both short- and long-term predictions.
strength_weakness: strength_weakness: [-] The introduction of the Hybrid Supervision Transformer (HSFormer) is a significant contribution to the field of human motion prediction. 

[-] The concept of hybrid supervision is particularly novel and addresses a key challenge in the domain.

[-] The paper is well-structured, with clear explanations of the methodology and the mathematical formulations underlying the proposed model.

[-] The paper demonstrates that the model is robust to input deviation, which is a critical aspect when dealing with real-world applications where data can be noisy or incomplete.
limitations: limitations: [-] In the introduction, the author should summarize the limitations of the existing work. 

[-] the contributions of the proposed approach should be strengthened by highlighting the unique contributions of each component rather than merely describing them. Each component should be presented in terms of its specific contribution to the overall approach.

[-] While the model performs well on the evaluated datasets, it is unclear how it would generalize to scenarios with different types of motion or in the presence of occlusions. 

[-] Addressing the specific challenges and potential improvements for long-term prediction would make the manuscript more comprehensive.

[-] More diverse datasets should be considered.