Multi-Transmotion: Pre-trained Model for Human Motion Prediction
Authors: Yang Gao
Po-Chien Luan
Alexandre Alahi
Venue: CoRL 2024 (2024)
Keywords: Human motion prediction
Pre-training
Transformer

Abstract: The ability of intelligent systems to predict human behaviors is essential, particularly in fields such as autonomous vehicle navigation and social robotics. However, the intricacies of human motion have precluded the development of a standardized dataset and model for human motion prediction, thereby hindering the establishment of pre-trained models. In this paper, we address these limitations by integrating multiple datasets, encompassing both trajectory and 3D pose keypoints, to further propose a pre-trained model for human motion prediction. We merge seven distinct datasets across varying modalities and standardize their formats. To facilitate multimodal pre-training, we introduce Multi-Transmotion, an innovative transformer-based model capable of cross-modality pre-training. Additionally, we devise a novel masking strategy to learn rich representations. Our methodology demonstrates competitive performance across various datasets on several downstream tasks, including trajectory prediction in the NBA and JTA datasets, as well as pose prediction in the AMASS and 3DPW datasets. The code will be made available upon publication.

Multi-Transmotion: Pre-trained Model for Human Motion Prediction
Yang Gao, Po-Chien Luan, Alexandre Alahi
Visual Intelligence for Transportation (VTTA) laboratory
EPFL, Switzerland
(firstname.h I hatatmmmep..lstttmmmeeeee......
Abstract: The ability of intelligent systems to predict human behaviors is crucial, particularly in fields such as autonomous vehicle navigation and social robotics. However, the complexity of human motion have prevented the development of standardized dataset for human motion prediction, thereby hindering the estab- lishment of pre-trained models. In this paper, we address these limitations by integrating multiple datasets, encompassing both trajectory and 3D pose keypoints, to propose pre-trained model for human motion prediction. We merge seven distinct datasets across varying modalities and standardize their formats. To fa- cilitate multimodal pre-training, we introduce Multi- Transmotion, an innovative transformer-base model designed for cross-modes mods-tt pre-training. Additionally, we present novel masking strategy to capture rich representations. Our method- ology demonstrates competitive performance across various datasets several downstream tasks, including trajectory prediction in the NBA and JTA datasets, well as pose prediction in the AMASS and 3DPW datasets. The code is publicly available: https://gtthu..co// http:/huwww.ooo/iittt////////a/e http///lli-ttaasmotio....
Keywords: Human motion prediction, Trajectory prediction, Pose prediction, Multimodal pre-trained model, Multitask pre-trained model
1
Introduction
The research community has witnessed substantial advancements through the adoption of pre-trained models. In natural language processing (NLP), large language models (LLMs) have demonstrated remarkable interdisciplinary proficiency, excelling across a variety of downstream tasks [1, 2]. In contrast, pre-trained models in computer vision (CV) typically exhibit greater degree of task speci- ficity, which can be attributed the multimodal nature of visual data [3, 4]. This specificity typically results in reduced efficiency when compared to NLP models. Nevertheless, recent developments in multitask pre- pre-training, exemplified by MultiMAE [5] and 4M [6], have shown promising capabilities in transferring knowledge across diverse array of CV tasks. However, this critical gap remains in the field of human motion prediction. Unlike fields such as NLP and CV, human motion incorporates rich representations and manifests through diverse modalities, including keypoints, trajectories, and bounding boxes, each reflecting different aspects of human movement. Despite this complexity, there currently exists no multimodal pre-trained model for accurately predicting human motion. Intuitively, human motion cannot be fully expressed by single modalities. Thus, we argue that each modality can benefit from the others by integrating multiple modalities into the models. Consequently. the development of a multitask pre-trained model is imperative for this domain.
Three principal challenges must overcome to effectively pre-train a model for human motion prediction. First, the field lacks comprehensive, large-scale dataset that encompasses various modalities of human motion. Second, versatile framework required to handle these diverse modalities, in contrast to previous approaches that typically addressed each modality in isolation. Third, the model must be robust when confronted with incomplete noisy input data. To address
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Inpert histooical frames
Output
Tragectory
AV
Multii Transemtiionnn
3DPW
3D PCD Poose keggositts
hoypolnnt
Srb Tiis Broundinng bosse
Transmotion
Trijectory
Triretee
NBA
Human3.6M Hamann.8M
Tranamotion
Fine
Fow
Multi Tran-Transmttion
Figure I: Overview. We propose unified human data framework by standardizing the data format and frame settings. Based on this framework, we introduce a pre-trained transformer model with specialized masking techniques, validating its effectiveness and flexibilityy across different scenarios.
these challenges, this paper proposes a novel approach that integrates multiple datasets, develops a flexible network architecture, and demonstrates its effectiveness handling noisy data.
Due to the absence of large-scale multimodal human motion prediction dataset, we have undertaken the task of merging several existing datasets, namely Joint Track Auto (JTA) [7], Trajnet++ [8], IRDB-Pose [9], NBA [10], Human3.6M [11], AMASS [12] and 3DPW [13]. Each dataset was originally created with different data formats and frame settings. To streamline the training process, we have unified the data framework. This framework standardizes the observation and prediction horizons, as well as the frame rates, ensuring consistency across the merged datasets. This unified framework enables more efficient and effective pre-training of the model, addressing the complexities of multimodal human motion prediction. For the model design, we implement tokenization strategy that maintains spatial-temporal information across all modalities by applying moodality-speciff linear projection layers to convert coordinates into hidden dimensions. To further enhance the model's s robustness and adaptability, we employ up-sampling padding, sampling masks, and bi-directional temporal encoder. The up-sampling padding facilitates casy fine- mune-tunnng across different frame rates, while the sampling masks simulate various frame rates by masking tokens with different chunk sizes.
Figure/ 1 shows overview of our work. By unifying the datasets, are able to pre- peeitriin transformer-based model, Multi-Transmotion, that can predict future trajectories and informative 3D pose keypoints. The flexibility of our model architecture and data framework allows for easy fine-tuning to specific tasks with varying frame settings. Our model achieved competitive results on both tasks, demonstrating the effectiveness of our pre-training techniques, as shown by our ablation studies on few-shot learning and robustness.
We summarize the main contributions follows:
Dataset:
We create a unified human motion data framework by merging seven datasets with standardized settings. Additionally, this framework is flexible, allowing for the seamless addition of more datasets or adjustments to frame settings.
Method:
We propose multi-Transmotion. a pre-trained transformer-based model that flexi- bly adapts to different frame settings, demonstrating strong robustness and efficiency. This model outperforms previous models across several datasets.
Related Work
Human trajectory prediction involves forecasting the future positions and movements individuals based on their past and current trajectories [14]. Data-driven methods have demonstrated remarkable
efficacy in human trajectory prediction [15, 16, 17, 18]. Numerous studies have explored social
interactions to enhance prediction accuracy, such social pooling [8, 19, 20], graphs [21, 22, 23,

---

24, 25, 26], and attentions [27]. Diverse architectures have been explored including recurrent neural networks (RNNs) [15, 23, 8), generative adversarial networks (GANs) [19, 28, 29, 30], conditional variational autoencoders (CVAEs) [31, 32], diffusion models [33, 34, 35], and LLMs [36]. Moreover, integrating transformer architectures with positional encoding has become a useful tool for capturing long-range dependencies and is widely used in trajectory prediction tasks [37.38, 39, 40]. From a dataset perspective, trajdata [41] provides unified interface for handling trajectory and map data. Recent studies have also explored leveraging 2D body pose as visual cues for trajectory prediction in image space [42], with emphasis on the utility of an individual agent's 3D body pose for trajectory prediction [43, 44, 45]. Thus, foundational model for this task must adeptly leverage available visual cues, including pose information, to produce accurate predictions.
Human pose prediction involves predicting the future coordinates of pose keypoints. The pose keypoints can be extracted with monocular [46] stereo [47] cameras. When predicting the poses, recurrent neural networks have traditionally been prominent [48], capitalizing their ability to capture temporal dependencies in sequential data. Subsequent advancements introduced feed-forward networks as alternatives [49], while graph convolutional networks (GCNs) were proposed to better capture the spatial dependencies of body poses [50]. Noteworthy innovations include separating temporal and spatial convolution blocks [51] and introducing trainable adjacency matrices [52, 53 Diffusion [54] has also shown strong performance by repairing and refining pose. Transformer-base approaches have also gained traction for modeling human motion [55, 56], showcasing significant improvements with spatio-temporal modules [50]. Accordingly, the pre-trained model proposed in this work incorporates attention mechanisms to capture both temporal and spatial pose representations.
Masked prediction is a technique used to learn robust representations by reconstructing masked data. BERT [57] uses masked prediction in natural language processing by replacing random words in sentence with special mask token and training the model predict these masked words from the surrounding context. In computer vision, Masked Autoencoder (MAE) [58] applies similar concept by masking random patches of an input image and training an autoencoder to reconstruct the missing patches, thereby learning meaningful image representations. The masking strategy has also been extended to trajectory prediction. Forecast-MAE [59] and Traj-MAE [59] both leverage the concept of masked autoencoders to enhance the tasks of motion prediction and trajectory prediction, respectively. Traj-MAF employs diverse masking strategies to pre-train trajectory and map encoders, capturing social and temporal information while utilizing a continual pre-training framework to mitigate catastrophic forgetting. Similarly, Forecast-MAE utilizes novel masking strategy that considers the interconnections between agents" trajectories and road networks, demonstrating competitive performance and substantial improvements over previous self supervupe methods in motion prediction.
Method
3.1 Unified Human Motion Data Framework
As previously mentioned, one of the primary challenges in the human motion domain is the variety of datasets. Specifically, different datasets vary in terms of horizon, frame rates, data formats, and pose joint configurations. To address the dataset challenges. we propose unified human motion dataset framework with standardized settings. We generate data sequences under uniform conditions in a consistent format, specifically 2 seconds of observation and 4 seconds of prediction 5 frames per second (fps). Since we are handling human motion prediction tasks, all annotations are recorded as coordinates. Regarding pose data, different datasets have varying joint ID configurations. Our unified joint ID configuration is provided in Appendix 1. As depicted in Table 1, the dataset framework comprises datasets featuring diverse combinations of modalities. By combining synthetic and real-world data, this framework encompasses over million samples for trajectories and over million samples for 3D pose keypoints. To our best knowledge, it the largest data framework for human motion prediction currently available. Furthermore, this dataset framework allows for the flexible addition of new data and adjustments to the horizon with different frame rates.
3

---

Table Approximate number annotations of the training split our unified human motion data framework. The default setting predicts 4 seconds into the future with 2 seconds of observation fps, except for TrajNet++ [8]*, where the frame setting fixed to 2.5 fps.
Dataset
R(eal)/S(yt.)
Traj
3D BB
2D BB
3D Pose
2D Pose
Scene
NBA U [10]
R
430k
/
No
Trajnet++ [8] (w/ ETHUCCY [60, 61]}*
R+S
250k
/
No
JRDB-Pose [9]
R
284k
284k
284k
/
94k
Yes
JIA [7]
764k
764k
764k
764k
764k
Yes
Human3.66 6M []]]
256k
256k
256k
256k
256k
Yes
AMASS [12]
314k
314k
314k
3DPW [13]
Yes
SUM
R+S
2302k
1622k
1304k
1338k
1114k
3.2 Multi-Transmotion Pre-trained Transformer-based Model for Human Motion Prediction
Pre-trained models have demonstrated remarkable effectiveness across diverse domains, from lan- guage tasks [62] image tasks [63, 3]. However, significant gap remains in the development of pre-trained models specifically designed for human motion prediction. This gap stems from the absence of model architecture capable of managing multimodal motion prediction with the flexibility to accommodate different frame settings and keypoint configurations. To bridge this gap, we proposed Multi- Tulti-Transmotio (Figure 2), a multimodal pre-trained model for multi-task motion prediction, designed to seamlessly integrate available visual cues while adapting to varying hori- zons, frame rates, and pose keypoints. To achieve this, we devised dynamic spatial-temporal mask, sampling mask, and bi-directional temporal encoding strategies, enhancing the model's robustness and adaptability to different frame rates and observation horizons. Detailed math reasoning was shown in Appendix A.6.
Tokentzatin Upsemmping Sampling
Scoete Htmmpood
Posse head
Transformer Enooder
Modsiit-speoffe bi di-dirctions enooder
Tokeen
Tolion
Tekion for bound box
Leomno tojcotorn querios prodiction horizon prediction
Masked
sse gia redt motion takke for apant Fhne grainea motion totin for neightor
transformer-basse model that learns cross-modality represen- tations and social interactions. Sampling mask and bi-directional encoders make the model flexible different frame settings, while dynamic spatial-temporal mask make pre-training efficient and robust.
Figure 2:
Tokenization. To retain the spatial-temporal information across all modalities, we apply modality- specific linear projection layers tokenize the coordinates into the hidden dimension the trans- former. Specifically, we project 3D x x-y-z coordinates for trajectories and 3D bounding boxes/pose keypoints into the hidden dimension. Likewise, the pixel coordinates of 2D bounding boxes/pose keypoints in image space projected into the hidden dimension. The learned queries padded and initialized as future motion tokens. These tokens are concatenated with historical motion tokens, after which masking strategies are applied. The combined tokens are then processed by the transformer.
Up-sampling padding, sampling mask, and bi-directional encoder. ensure the model can be easily fine-tuned with different frame rates, we apply up-sampling padding the valid tokens during pre- tre-training, simulating the maximum fps for the model. The maximum fps is set 50, as it can be easily downsampled to commonly used fps settings for trajectory prediction tasks (2.5 fps and

---

Wald Fise:
Volid Fps: 10
Fiss: 25
Observelion Frames
Preciction trames
Trad temporal encoder
Transtormer Encoder
Transformer Encoder
Transforma Encoder
Volid loker
Obs encoding Predi encoding
Masked loker
Obsarvation frames Predi ction Bi-directional temporal encoder
Figure 3: Sampling mask and bi-directional encoder.
5 fps) [44, 8, 33] and pose prediction tasks (25 fps and 50 fps) [54, 50], As illustrated on the left of Figure 3, the sampling mask allows the model to mask different chunk sizes, simulating various frame rates. The right of Figure shows the difference between the traditional temporal encoder and our bi-directional temporal encoder. Rather than starting the encoding from the first observation, we begin from the last observation and encode observations and predictions separately, facilitating seamless adaptation to different observation and prediction horizons during the fine-tuning process.
Dynamic Spatial-temporal mask. After applying the bi-directional encoder and sampling mask, each valid token includes spatial-temporal information, allowing the model to learn detailed positional representations. To make the model more robust, we introduce a dynamic apatial-tempont mask, which randomly masks out a dynamic spatial-ratio (0 < T, < 1) of the total number of 3D/2D pose keypoints for all frames. Meanwhile, we apply temporal-ratio (r: 0.1) for trajectory and 3D/2D) bounding box modalities, randomly masking some frames of data. This dynamic spatial- temporal mask strategy has made pre-training more effective compared to related works [44, 5], as demonstrated in our ablation study shown in Appendix A.3.
Experiments
4.1 Datasets
We exclude Trajnet++ [8] and use the training split of the other 6 datasets to develop a pre-trained model, enabling us to perform few-shot learning on Trajnet++ [8]. For the evaluation of the trajectory prediction task, we employ the NBA [10] and JTT [7], as they represent the largest trajectory-only dataset and the largest dataset with 3D pose, respectively. To assess performance on pose prediction tasks, we use the widely adopted large-scale AMASS [12] and the smaller-scale 3DPW [13]. to thoroughly evaluate performance in both indoor and outdoor scenarios. Additionally, we also show the model's application on robot navigation task Appendix A.2 and the robustness against imperfect poses in Appendix A.4.
Regarding experimental settings, we follow the pioneering works [33, 44, 54) with the configurations below:
NBA [10]: Predict seconds of trajectory given 2 seconds of past trajectory at fps.
JTT [7]: Predict 4.8 seconds of trajectory given 3.6 seconds of past trajectory at 2.5 fps.
AMASS [12]: Predict second pose keypoints given 2 seconds of past pose at 25 fps.
3DPW [13]: Predict second of pose keypoints given 2 seconds of past pose at 25 fis.
4.2 Metrics
For evaluation, we use Average Displacement Error (ADE) and Final Displacement Error (FDE) for the deterministic trajectory prediction task on JTA [7], and MinFDDE best-of-k ADE/FDE for stochastic trajectory prediction NBA [10]. Mean Per Joint Position Error (MPJPE) is used for the pose prediction tasks on AMASS [12] and 3DPW [13].
5

---

Table 2: Quantitative results on NBA [10]. MinADE.g/MMinDEgg (meters) are reported.
Models
Venue
MinADE20
MinFDE2o)
Social-LSTM [15]
CVPR 16
1.65
2.98
Social-GAt [19]
CVPR 18
1.59
2.41
STGAT [21]
ICCV 19
1.40
2.18
Social-STGCNN [26]
CVPR 20
1.53
2.26
Trajectron++ [23]
ECCV 20
1.15
1.57
NPSN [64]
CVPR 22
1.31
1.79
GroupNet [24]
CVPR 22
0.96
1.30
MID [65]
CVPR 22
0.96
1.27
Leapfrog [33]
CVPR 23
0.81
1.10
Social-Transmotion [44]
ICLR 24
0.78
1.01
Multi- Multi-Transse Transmotion
0.75
0.97
4.3 Results
Trajectory prediction on NBA [10]. We selected the large-scale NBA [10] to evaluate our model on the pure trajectory prediction task, as this dataset does not provide visual modalities. To ensure a fair comparison with pioneering works, the test split remained hidden from the model during the pre-training process. We used the same test data provided by [33] for evaluation. Table 2 presents numerical comparison between method and several strong baselines. Our approach, which involves pre-training followed by fine-tuning, consistently outperformed all prior baselines. This result underscores the effectiveness of our proposed architecture, which integrates innovative masking and encoding techniques, establishing its value as a pre-trained model.
Figure illustrates the qualitative results of our model, showing that the predicted trajectory closely aligns with the ground truth. This provides further evidence of the model's strong performance in highly interactive sports scenarios.
Neighboring agents
Observation
GT
Prediction
Figure 4: Qualitative results on NBA [10]. The red, blue, and green dots represent the observed historical frames, predicted future frames, and ground truth, respectively. All neighboring players are shown in grey.
Trajectory prediction on JTA [7]. To demonstrate the effectiveness of 3D pose in augmenting trajectory prediction, we evaluate the model using pure trajectory (T) and trajectory combined with 3D pose (T+3D P) as input modalities on the JTA [7]. Since 3D pose is the most informative visual cue, we fine-tune our model with trajectory and 3D pose, then compare with Social-Transmont Transmotion [44] trained on the same modalities to ensure fairness. leverages knowledge from 3D poses enhance trajectory prediction. Furthermore, our model demonstrates superior performance in utilizing pose knowledge compared to the previous multimodal trajectory prediction model [44], which we attribute to our pre-training approach and novel masking strategy. Despite the different frame settings from pre-training, the model's performance remains robust through fine-tuning, demonstrating the flexibility of our pre-trained model.
shown in Table 3, our model effectively
To further examine how the model leverages 3D pose information, visualize the predictions of model using Trajectory-oni -nly and Trajectory 3D pose. Figure 5 shows the qualitative results of model. It is evident from this figure that the last frame of the pose provides valuable information

---

Table 3: Quantitative results on JTAA [7]. croisl-Transootion [44] and Multi-Transmodine are
trained on Trajectory and 3D Pose modalities since they can leverage pose. (T and 'P' abbreviate Trajectory, and Pose keypoints)
Models
Venue
Input modality inference
ADE
FDE
Vanilla-LSSTM [15]
CVPR 16
1.44
3.25
Directional-LSTM [8]
T-ITS
1.37
3.06
Social-LSTM [15]
CVPR 16
T
1.21
2.54
Autobots [39]
ICLR 22
1.20
2.70
Trajectron++ [23]
ECCV 20
T
1.18
2.53
EqMotion [40]
CVPR 23
1.13
2.39
[44]
ICLR 24
T
0.99
2.00
Docial-Transmottin [44]
ICLR 24
T 3D
0.94
1.94
MuattiTrransootion
T
0.97
1.97
Multi-Transmont
T P
0.91
1.89
about walking direction and body rotation, enabling the model to predict accurate future trajectory when pose information is available.
Mu Mut-Tranmmction IT
Mut Mut--aammetiion !T1 Ground truts Observation
Figure 5: Qualitative results on dataset. This visualization shows how our model can leverage 3D pose information augment trajectory prediction. The red trajectory denotes the prediction without using pose knowledge, and the blue trajectory denotes the prediction with help of leverage 3D pose. The green trajectory denotes the ground truth.
Pose prediction on AMASS [12] and 3DPW [13]. For the pose prediction task, we utilize the AMASS [12] and 3DPW [13] to thoroughly evaluate our model's performance in indoor and outdoor scenarios. Following previous works [54], we use the same test split and data preprocessing to normalize poses. Table presents the qualitative results on the AMASS and 3DPW, showing that our model consistently achieves the best or second-best performance across the entire prediction horizon.
Table 4: Quantitative results on AMASS [12] and 3DPW [13]. Numbers are reported by MPJPE in millimeter at specific predicted time-step. Best numbers are in bold and second-best numbers are underlined.
AMASS [12]
3DPW [13]
Model
160 ms
400 ms
720 ms
1000 ms
160 ms
400 ms
720 ms
1000
ConvSeq2Seq (CVPR
36.9
67.6
87
93.5
32.9
58.8
77
87.8
LTD-10-10 10 [66] (ICCV 19)
19.3
44,6
75.9
91.2
22
46.2
69.1
81.1
LTD-10-25 [66] (ICCV 19)
20.7
45.3
65.7
75.2
23.2
46.6
65.8
75.5
HRI [50] (ECCV 20)
20.7
422
58.6
67.2
22.8
45
62.9
72.5
ST-Trans [56] (Ra-L 24)
21.3
42.5
58.3
66.6
24.5
47.4
64.6
73.8
TransmTrannttooonn
19.3
41.4
58.6
66.9
22.5
45.6
64.2
73.7
Figure presents the qualitative results of our approach, with the actions of sitting, walking, and standing shown in separate rows. Benefiting from pre-training on the large-scale motion dataset, the walking scenario (second row) is handled effectively by our model. In the sitting scenario (first row), the prediction quality declines in the last few frames as the agent begins stand up, challenging transition even for humans to predict. Overall, model demonstrates competitive performance the pose prediction task, evaluated intensively across various scenarios in AMASS [12].

---

Ground Truth
R
R
K
2
$
Time
80ms
160ms
320ms
400ms
560ms
720ms
880ms
1000ms
Figure 6: Qualitative results on AMASS [12]. with sitting, walking, and standing shown by row.
Few-shot learning on small data under new frame settings To further explore the potential of our model on small datasets, we conduct few-shot learning on a maximum of samples and
evaluate approximately 4k samples from the real-world part of Trajnet++ [8]. We fine-tuned our
model to adapt to the Trajnet++ [8] frame settings and compared it with wansformer-baseee baseline model (Autobots(39)) and an LSTM-based bascline model (Social-LSTM) [15]). Figure shows that
Social-LSTM converges faster than Autobots, transformer networks are typically very data-hungry
However, our mulii-Transmotion model outperforms Social-LSTM, benefiting from the pre-trained knowledge.
Table 5: Comparison of Pre-trained model and Specific model. Specific model indicates the model is trained from scratch on one dataset. MinADEgg/MinFDEEz are used for NBA [10), ADE/FDE are used on Trajnet++ [8], and final
Sarial4STM
Mulii-Fansmmtion
MPJPE is used on AMASS [12] and 3DPW [13].
1.5
Dataset
Pre-trained model Specifie model
NBA [10]
0.75/U.97
0.77/0.98
1.0
Trajnet++ [8]
0.54/1.13
0.57/1.22
200
400
600
800
1000
AMASS [12]
66.91
69.58
Number training
3DPW [13]
73.74
76.77
Figure 7: Few-shot learning curve
Pre-trained model vs. Specific model To explore the generalization capabilities of models trained on different datasets, we conduct an ablation study to see if the model benefits from pre-trained knowledge. We observe the performance differences between pre-trained model trained with multiple datasets and specific models trained on individual datasets. Table 5 demonstrates that the pre-trained model consistently outperforms the specific models. This indicates that pre-training on large-scale data significantly enhances the generalization ability of the model for both human trajectory prediction and pose prediction.
5 Conclusion and Limitations
In this work, introduced unified human motion data framework and Nulti-Transsotion, the first pre-trained transformer-based model in the human motion prediction domain. By employing several novel masking strategies, our model can fine-tune on different frame settings and achieve competitive results on human motion prediction, underscoring its practical value. Additionally, the masking strategies enhance efficiency without sacrificing robustness against noisy input.
As for limitations and future work, incorporating additional modalities, such contextual images and human intentions, further empower the model and bring it closer to being foundational model for predicting human motion. We leave this for future exploration.

---

Acknowledgement
The authors would like to thank Saeed Saadatnejad, Lan Feng, Reyhaneh Hosseininejad, and Zimin Xia for their valuable feedback. This work was supported by Sportradar' (Yang's Ph.D.) and Honda R&D Co., Ltd.
References
[1] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, Barham, H. W. Chung, C. Sutton, Gehrmann, et al. Palm: Scaling language modeling with pathways. arxiv 2022. arXiv preprint arXiv:2204.02311 10, 2022.
[2] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
[3] M. Oquab, T. Darcet, T. Moutakanni, H. Vo, M. Szafraniee, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
[4] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8766 8763. PMLR, 2021.
[5] R. Bachmann, D. Mizrahi, A. Atanov, and A. Zamir. Multimae: Multi-modal multi-task masked autoencoders. In European Conference on Computer Vision, pages 348-367. Springer, 2022.
[6] D. Mizrahi, R. Bachmann, O. Kar, T. Yeo, M. Gao, A. Dehghan, and A. Zamir. 4m: Massively multimodal masked modeling. Advances in Neural Information Processing Systems, 36, 2024.
[7] M. Fabbri, F. Lanzi, S. Calderara, A. Palazzi, R. Vezzani, and R. Cucchiara. Learning detect and track visible and occluded body joints in virtual world, 2018.
[8] Kothari, S. Kreiss, and A. Alahi. Human trajectory forecasting in crowds: A deep learning perspective, 2021.
[9] E. Vendrow, D. T. Le, J. Cai, and H. Rezatofighi. Jrdb-pose: A large-scale dataset for multi- person pose estimation and tracking, 2023.
[10] K. Linou, Linou, and M. de Boer. Nba player movements. https://github.com com/ Maaouu23//BA-Plaaeee-ooooe
2016.
[11] C. Ionescu, D. Papava, V. Olaru, and C. Sminchisescu. Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence, 36(7):1325-133) 1339, 2013.
[12] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. Black. Amass: Archive of motion capture as surface shapes. In Proceedings of the IEEE/CVF international conference on computer vision, pages 5451,, 2019.
[13] T. Von Mareard, R. Henschel, M. J. Black, B. Rosenhahn, and G. Pons-Moll. Recovering accurate 3d human pose in the wild using imus and moving camera. In Proceedings of the European conference on computer vision (ECCV), pages 601-617, 2018.
[14] A. Rudenko, L. Palmieri, M. Herman, K. M. Kitani, D. M. Gavrila, and K. O. Arras. Human motion trajectory prediction: A survey. The International Journal of Robotics Research, 39(8): 895-935, 2020.
'nttss//sportradar.com//

---

[15] A. Alahi, K. Goel, V. Ramanathan, A. Robicquet, L. Fei-Fei, and S. Savarese. Social Istm: Human trajectory prediction in crowded spaces. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 961-971, 2016.
[16] J. Sun, Y. Li, L. Chai, H.-S. Fang, Y.-L. Li, and C. Lu. Human trajectory prediction with momentary observation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 6467-6476, 2022.
[17] Y. Liu, A. Rahimi, P.-C. Luan, F. Rajič, and A. Alahi. Sim-to-real causal transfer: A metric learn- ing approach causally-aware interaction representations. arXiv preprint arXtv:2312.04540. 2023.
[18] M. Bahari, Saadatnejad, A. A. Farsangi, S.-M. Moosavi-Dezfoolli, and A. Alahi. Certified human trajectory prediction. arXiv preprint arXiv:2403. 2024.
[19] A. Gupta, J. Johnson, L. Fei-Fei, S. Savarese, and A. Alahi. Social gan: Socially acceptable trajectories with generative adversarial networks, 2018.
[20] N. Deo and M. M. Trivedi. Convolutional social pooling for vehicle trajectory prediction. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pages 1468-1476, 1476, 2018.
[21] Y. Huang, H. Bi, 7. Li, T. Mao, and 7. Wang. Stgat: Modeling spatial-temporal interactions for human trajectory prediction. In Proceedings of the IEEE/CVF international conference computer vision, pages 6272-62222 6281, 2019.
[22] Li, Yang, M. Tomizuka, and C. Choi. Evolvegraph: Multi-agent trajectory prediction with dynamic relational reasoning. Advances in neural information processing systems, 33: 19783-19794, 2020.
[23] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-fcasib trajectory forecasting with heterogeneous data, In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pages 683-700. Springer, 2020.
[24] C. Xu, M. Li, Z. Ni, Y. Zhang, and S. Chen. Groupnet: Multiscale hypergraph neural networks for trajectory prediction with relational reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498-6507, 2022.
[25] A. Monti, A. Bertugli, S. Calderara, and R. Cucchiara. Dag-net: Double attentive graph neural network for trajectory forecasting. In 2020 25th International Conference on Pattern Recognition (ICPR), pages 2551-2558. IEEE, 2021.
[26] A. Mohamed, K. Qian, M. Elhoseiny, and C. Claudel. Social-stgenn: A social temporlt graph convolutional neural network for human trajectory prediction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14424-14432. 2020.
[27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[28] Y. Hu, S. Chen, Y. Zhang, and X. Gu. Collaborative motion prediction via neural motion message passing. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 6319 119662288 2020.
[29] A. Sadeghian, V. Kosaraju, Sadeghian, Hirose, H. Rezatofighi, and Savarese. Sophie: An attentive gan for predicting paths compliant to social and physical constraints. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 1 1349-1358. 2019.
10

---

[30] V. Kosaraju, A. Sadeghian, R. Martin- Martin, I. Reid, H. Rezatofighi, and S. Savarese. Social- bigat: Multimodal trajectory forecasting using bicycle-gan and graph attention networks. Ad- vances in neural information processing systems, 32, 2019.
[31] E. Schmerling, Leung, W. Vollprecht, and M. Pavone. Multimodal probabilistic model-based planning for human-robot interaction. In 2018 IEEE International Conference on Robolics and Automation (ICRA), pages 3399-3406. IEEE, 2018.
[32] B. Ivanovic, K. Leung, E. Schmerling, and M. Pavone. Multimodal deep generative models for trajectory prediction: A conditional variational autoencoder approach. IEEE Robotics and Automation Letters, 6(2):955-30,, 2020.
[33] W. Mao, C. Xu, Q Zhu, S. Chen, and Y. Wang- Leapfrog diffusion model for stochastic trajectory prediction. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5517-5526, 2023.
[34] T. Gu, G. Chen, J. Li, C. Lin, Y. Rao, J. Zhou, and J. Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 17113-17122, June 2022.
[35] I. Bae, Y.J. Park, and H.-G. Jeon. Singulartrajectoryy Universal trajectory predictor using diffusion model. arXiv preprint arXiv:2403. /8452, 2024.
[36] I. Bae, Lee, and H.-G. Jeon. Can language beat numerical regression? language-based multimodal trajectory prediction. arXiv preprint arXiv:2403.18444 2024.
[37] Giuliari, Hasan, M. Cristani, and F, Galasso. Transformer networks for trajectory forecasting. In 2020 25th international conference on pattern recognition (ICPR), pages 10335-10342. IEEE, 2021.
[38] Y. Yuan, X. Weng, Y. Ou, and K. M. Kitani. Agentformer: Agent-aware transformers for socio- temporal multi-agent forecasting. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9813-9823, 2021.
[39] R. Girgis, F. Golemo, F. Codevilla, M. Weiss, J. A. D'Souza, S. E. Kahou, F. Heide, and C. Pal. Latent variable sequential set transformers for joint multi-agent motion prediction. In International Conference on Learning Representations (ICLR), 2022. URL https:// openreview bpenreview.net/forumm wwwnmndddDDDddddaad_
[40] C. Xu, R. T. Tan, Tan, S. Chen, Y. Wang, Wang, and Wang. Eqmotion: Equivariant multi-agent motion prediction with invariant interaction reasoning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 1410-1420, 2023.
[41] B. Ivanovic, G. Song, I. Gilitschenski. and M. Pavone. trajdata: A unified interface to multiple human trajectory datasets. In Proceedings of the Neural Information Processing Systems (NeurIPS) Track on Datasets and Benchmarks, New Orleans, USA, Dec. 2023. URL https: //arxiv. www//b///////eoo/o//.o/
aa/abb//230 13924.
[42] K. Chen, X. Song, and X. Ren. Pedestrian trajectory prediction in heterogeneous traffic using pose keypoints- convolutional encoder-decoder network. IEEE Transactions on Circuits and Systemsfot for Video Technology, 3165):1764-1775. 2020.
[43] V. Kress, Jeske, S. Zernetsch, K. Doll, and B. Sick. Pose and semantic map based probabilistic forecast of vulnerable road users trajectories. IEEE Transactions on Intellignt Vehicles, 2022.
[44] S. Saadatnejad, Y. Gao, K. Messaoud, and A. Alahi. Social-tramss tranmmttrasmoooon: Promptable human trajectory prediction. In International Conference on Learning Representations (ICLR), 2024.

---

[45] Y. Gao. Social-pose: Human trajectory prediction using input pose, 2022.
[46] S. Kreiss, L. Bertoni, and A. Alahi. Openpifpaf: Composite fields for semantic keypoint detection and spatio-temporal association. IEEE Transactions on Intelligent Transporiation Systems,
2021.
[47] W. Deng. L. Bertoni, S. Kreiss, and A. Alahi. Joint human pose estimation and stereo 3d localization. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 2324-2330. IEEE, 2020.
[48] H.-k. Chiu, E. Adeli, B. Wang, D.-A. Huang, and J. C. Niebles. Action-agnostic human pose forecasting, In IEEE Winter Conference Applications of Computer Vision (WACV), pages 1423-1432. JEEE, 2019.
[49] C. Li, Z. Zhang, W. S. Lee, and G. H. Lee. Convolutional sequence to sequence model for human dynamics. In IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 5226-5234 (266,,2,,2200 2018.
[50] W. Mao, M. Liu, and M. Salzmann. History repeats itself: Human motion prediction via motion attention. In European Conference Computer Vision, pages 474-489, Springer, 2020.
[51] T. Ma. Y. Nie, C. Long, Q. Zhang, and G. Li. Progressively generating better initial guesses towards next stages for high-quality human motion prediction. In IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 6437-6446, June 2022.
[52] Sofianos, A. Sampieri, L. Franco, and F. Galasso. Space-time-separable graph convolutional network for pose forecasting. In IEEE/CVF International Conference Computer Vision (ICCV), pages 1209-11218, October 2021.
[53] C. Zhong, L. Hu, Z. Zhang, Y. Ye, and S. Xia. Spatio-temporal adjanneny gen for human motion prediction. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6447-6456, June 2022.
[54] S. Saadatnejad, A. Rasekh, M. Mofayezi, Y. Medghalchi, Rajabzadch, T. Mordan, and
A. Alahi. A generic diffusion- based approach for 3d human pose prediction in the wild. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 8246-8253. IEEE, 2023.
[55] M. Petrovich, M. J. Black, and G. Varol. Action-conditioned 3d human motion synthesis with transformer vae. In IEEE/CVF International Conference Computer Vision (ICCV), pages 10985-10995, October 2021.
[56] S. Saadatnejad, M. Mirmohammadi, M. Daghyani, P. Saremi, Y. Z. Benisi, A. Alimohammadi, Z. Tehraninasab, T. Mordan, and A. Alahi. Toward reliable human pose forecasting with uncertainty, IEEE Robotics and Automation Letters, 2024.
[57] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini arXiv: 181000488 2018.
[58] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[59] H. Chen, Wang, K. Shao, F. Liu. J. Hao, C. Guan, G. Chen, and P.-.. Heng. Traj-mae: Masked autoencoders for trajectory prediction. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 8351-8362, 2023.
[60] S. Pellegrini, A. Ess, K. Schindler, and L. Van Gool. You'll never walk alone: Modeling social behavior for multi-targe tracking. In 2009 IEEE 12th international conference computer vision, pages 261-268. IEEE, 2009.
12

---

[61] A. Lerner, Y. Chrysanthou, and D. Lischinski. Crowds by example. In Computer graphics forum, volume 26, pages 635-664. Wiley Online Library, 2007.
[62] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro. F. Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13371, 2023.
[63] A. Kirillov, E. Mintun, N. Ravi, H. Mao, Rolland, L. Gustafson, Xiao, S. Whitehead, A. C. Berg, W.-Y. Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4015-4026, 2023.
[64] Baa, J.-H. Park, and H.-G. Jeon. Non-probability sampling network for stochastic human trajectory prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6477-6487, 2022.
[65] T. Gu, G. Chen, J. Li, C. Lin, Y. Rao, J. Zhou, and J. Lu. Stochastic trajectory prediction via motion indeterminacy diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17113-17122, 2022.
[66] W. Mao, M. Liu, M. Salzmann, and H. Li. Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEE/CVE international conference on computer vision, pages 9489-9497, 2019.
[67] C. Chen, Y. Liu, S. Kreiss, and A. Alahi. Crowd-robot interaction: Crowd-aware robot navigation with attention-based deep reinforcement learning. In 2019 international conference on roboties and automation (ICRA), pages 6015-6022. IEEF, 2019.
[68] D. Helbing and P. Molnar. Social force model for pedestrian dynamics. Physical review E, 51 (5):4282, 1995.
[69] Grishchenko, V. Bazarevsky, Zanfir, E. G. Bazavan, M. Zanfir, R. Yee, K. Raveendran, M. Zhdanovich, M. Grundmann, and C. Sminchisescu. Blazepose ghum holistic: Real-time 3d human landmarks and pose estimation. arXiv preprint MrXiv:2266.11678 2022.
[70] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint 1xXiv:1412.6980, 2014.
13

---

A Appendix
A.1
Unified Human Motion Data Framework
Table 6 presents the detailed joint configurations and the mapping between unified IDs and the original datasets.
Table Joint ID Mapping
Unified ID
Unified joints
JTA
H36M
JRDB
AMASS
3DPW
Old ID
Old ID
Old ID
Old ID
Old ID
pelvis
15
0
0
0
1
right_hip
16
10
2
2
2
right_knee
17
2
13
5
5
3
right_ankle
18
3
15
8
8
4
right ffoo_arrch
nan
4
nan
11
11
5
right
nan
5
nan
nan
nan
6
left hip
19
6
11
1
left_ knee
20
14
4
8
left ankle
21
8
16
7
7
left_ fit_foot_arch
nan
nan
10
10
10
lefe_toes spine (H36M)
nan
10
nan
nan
12
nan
nan
nan
12
thorax/chest
nan
13
nan
nan
13
neck head center
12
14
1
15
15
15
15
left shoulder
8
17
16
16
left elbow left_wrist left_outer_thigg thigh
9
18
7
18
18
17
10
19
nan
20
20
18
21
19
left_hand right_shone shouldh
nan
22
12
nan
22
20
4
25
17
17
21
right_ right_elbow
5
26
6
19
19
22
right_wrist.
6
27
nan
21
21
23
__outer_thigh right hand head top right clavicle
nan
29
nan
nan
nan
24
nan
30
nan
23
25
0
nan
0
nan
nan
26
3
nan
nan
14
14
27
left clavicle spine (JTA)
7
nan nan
13
13
28
nan nan
nan
nan
29
spine (JTA)
12
nan nan
nan
nan
30
spine (JTA)
13
nan
nan nan
nan
31
spine 3 (JTA)
14
nan
nan
right_eye (JRDB)
nan nan
nan
nan
33
left_eye (JRDB)
nan nan
2
nan
nan
spine (AMASS/3DPW)
nan nan
nan
3
3
35
spine (AMASS/3DPW)
6
6
spine (AMASS/3DPW)
nan nan
nan
9
9
37
nose
38
forehead
nan nan
nan
nan
nan
A.2
Application in Robot Navigation
To further explore the potential application of our model in robotic tasks, we integrated our predictor into a robotic navigation system.
We applied the CrowdNav [67] simulator generate pedestrian trajectories with social interactions. Consequently, there approximately 800 training samples and 200 test samples. The social force
14

---

model [68] was used as the navigator, as it allowed us to easily incorporate the predicted trajectories by adding extra repulsive forces and validate the effectiveness of our predictor. As Table shows both the completion time and the collision rate were reduced after incorporating our predictor, which highlighted the effectiveness of our model in robot navigation scenarios. Additionally, we found that the improvement completion time could reach up 14% when increasing the social forces.
Table 7: Application of our model in robot navigation task
Completion time seconds (gain) Collision rate (gain)
robot navigation w/o our predictor
16.46
1.93%
robot navigation w/ predictor 16.20 (+2%)
0.39 % (+80%)
A.3 Comparison Between Different Masking Strategies
Masking has been shown to efficiently improve the robustness and generalization ability in transformer architectures [58, 44]. this ablation study, we pre-train three smaller models on JTA [7] to quickly examine how different masking implementations affect pre-training in human motion prediction. This ablation study is implemented on the same Tesla V100 GPU device. Table shows that while modality-mask and meta-mask yield strong performance, they come with high computational costs. Conversely, the fixed spatial-temorral mask used in [5] improves computational efficiency by consistently dropping a fixed number of tokens but leads to the lowest robustness. Our implementation employs a dynamic spatial-temporal mask that randomly drops tokens, maintaining high computational speed without sacrificing robustness during pre-training.
Table 8: Effect of different masking strategy.
Fixed and Mela-mask [44] Spatial-temporal mask [5]
Modality mask
Dynamic Spatial-temporal mask (ours)
Computational cost
Spatial cost Temporal cost
6864 MB
3128 MB
6142 MB
11.306 ms
7.631 ms
7.349 ms
Performance on JTA [7]
T
1/2./2
1,50/3.13
1.00/1,99
++3DB BDBBB 2DB T B 3D
1.04/2.08
1.13/2.23
0.99/1.98
0.96/1.94
0,95/1.93
0.96/1.94
A.4 Does Imperfect Pose Input Still Augment Trajectory Prediction?
In the real world, it is challenging to capture accurate 3D pose keypoint annotations due to occlusion or sensor noise. Therefore, it is crucial to assess whether the model remains reliable when faced with incomplete or noisy pose input. To better understand the robustness in such scenarios, we also compare with a specific model trained without any masking strategy. Table presents the performance comparison, showing that the masking strategy significantly enhances the model's robustness. Specifically, it reduces performance degradation from 46.7% to 11.0% when dealing with noisy 3D pose keypoints.
Table 9: Performance under imperfect pose input.
Input Modality at inference
Multi- Transmotion
Specife model w/o masking
ADE/ / FDE (degradation % 4)
ADE FDE (degradation %
T 100% 3D P
0.91 / 1.89
0.92 1.90
T + 50% 3D P
0,92/ 1,90 (1.1% 70,5%)
0.99 1,98 (7.6% 4422%)
+ 10% 3D P
0.95 / 1.94 (4.4% 2.6%)
1.17 2,.1 (27.2% 21.6%)
T + 3D P w/ Gaussian Noise (std=25)
0.97 / 1.98 (6.6% 448%%)
1.16 2.31 (26.1% 21.6%)
T 3D P w/ Gaussian Noise (std=50)
1.01 / 2.05 (11.0% 8.5%)
1.35 / 2.68 (46.7% 44.1%)
15

---

In addition to simulating noisy pose input, it is more realistic to handle estimated poses from existing pose estimators. In this study, we use the estimated 3D poses generated by an off-the-shelf pose estimator [69] on some clips from the JRDB-Pose [9]. Table 10 shows that our model can also generalize to pseudo 3D poses, improving trajectory prediction compared to using only trajectory data. Although the improvement is modest, still demonstrates the model's effectiveness a realistic scenario.
Table 10: Performance with using estimated Psudo 3D pose.
Model
ADE
FDE
Multi-Transmotiin (T)
0.13
0.20
Multi-Transmotion (T + Pseudo 3D P)
0.11
0.18
A.5 Implementation Details
The tual-trrasformer architecture has 4 heads in each transformer. The first transformer, used to learn multimodal features, has 6 layers, while the second transformer, focused on social interaction, has 4 layers. For the pre-training, we use the Adam optimizer [70] with a learning rate starting at x 10-1, which decays by factor of 0.1 after 80% of the 60 total epochs are completed. The pre-training is conducted on 8 NVIDIA A100 GPUs, each with 80GB of memory. L2 loss is applied both the trajectory prediction and pose prediction tasks. No supervision is applied masked keypoints (e.g.. missing joints) and the model is trained to predict the available pose keypoints annotated in the data.
A.6 Math Reasoning for Method
In this work, all terms regarding trajectory, 3D/2D bounding box, and 3D/2D pose refer to the 3D/2D coordinates of that modality. E.g., we use 2D x-y coordinate input/output for the bird-eye-view trajectory and 3D x-y-z coordinate input/output for the 3D pose. Specifically, denote the sequential Trajectory, 3D/2D bounding box and 3D/2D local pose of agent i as x; Tráj 22dB 2d? I 3dP and
a.doP respectively. The observed time-steps and predicted time-steps are defined as t = 1, ..... Toks and t = Tohe + I,..., Treed.
Considering a sample with pedestrians, the model input X x= _2,, XN). where X; (af,CE E {Traj, 3dB, 2dB, 3dP, 2dP]] depending on the availability of different modalities. The tensor 2t has shape of (Tots: f"), where e" represents the number of elements in specific cue (e.g., the number of 3D pose keypoints) and f denotes the number of features (e.g.. 3 for x-y-z dimension of 3D pose keypoints) for each element.
Lastly, the model output is the ego (i=l) pedestrian's 's motion Y Y1. where Yı = (Yara Yp} containing & future possible sequential trajectories and one future sequential deterministic 3D pose keypoints.
During the tokenization process, we use modality-specific MLPs to encode each input modality, resulting in the projected hidden dimension H; = (xF( Next, we apply Up-sampling Padding and a modality sppeciffc Bi-directional encoder to generate tensors with fighufreyeency positional information, i.e., H (HŞ) + B', where 3e is the Bi-directional encoder applying to modality e and U denotes Up-sampling padding. After this step. each token from Hद represents a specific element of one modality specific time step (e.g., token could contain information about the 3D neck keypoint the first observed frame).
The token length of each modality can be calculated as L. = c. t.r where e, is the number of elements in a modality and te is the number of frames considered for that modality. During the masking process, we first use the Sampling Mask to mask out specific chunk sizes based on the data configurations, allowing the model simulate different frame settings. Then, the Dynamic Spatial-temporal mask randomly drops dynamic spatial-ratio (0 r. < 1) of the total number of 3D/2D pose tokens for all frames. Mcanwhile, we apply temporal-ratio (r, 0.1) for tokens
16

---

related to trajectory and 3D/2D bounding box modalities, as each of these modalities contains only one element, randomly masking some frames of data. After applying the masking, the token length for the first transformer (I1) is given by the following formula:
4 = TTraj + Laan + LeaB + T<ar(( - ra) + 12ap(1 - ra)-
As the second transformer learns the fine tnee-graied motion interaction between pedestrians, the token length for the second transformer (T2) is calculated as follows:
L = Nirraaj + Laap - ra)):
where N the number of pedestrians the scene. After passing through the transformers, the fine-grained tensors are denoted F HHG where represents the output modalities of the model. Le.. (Traj, 3dP)
Finally, we use modality-specifie MLP heads to project the multi-modal x-y trajectories and deter- ministic x-y-z 3D pose keypoints for the ego (i=1) pedestrian:
Troj MLPPraj (FH)), Yp (FH
17


Reviews:
[{"review_id": "nY0lT0KYN1", "reviewer": "Reviewer_CsuV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Predicting human motion is crucial for applications like self-driving cars and robots, but existing models often struggle with the complexity and variety of human movement. This paper proposes a new approach:  First, they create a large, unified dataset by combining seven existing datasets that cover different types of motion (walking, sports, etc.).  Second, they develop Multi-Transmotion, a specialized transformer model that can learn from this diverse data. Finally, they use a novel \"masking\" strategy to train the model, making it more robust to noisy or incomplete data. Their results show that Multi-Transmotion outperforms previous methods and provides a strong foundation for future research in human motion prediction.", "summary_of_recommendation": "I recommend a weak accept given the strong performance of the model on human motion prediction benchmarks and novel methods.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "The method and results are very appealing but having result in a robotics task would make the paper even stronger.\n* One suggestion would be to do few-shot learning on a sim navigation task and show that the concepts learned can transfer and are applicable to robot navigation. HuNavSim could be a good benchmark: https://arxiv.org/pdf/2305.01303.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C05mJgrdnW", "reviewer": "Reviewer_4Ldu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper focuses on the problem of human motion prediction and aims to develop a large scale model for human motion prediction. The hope is that such a large scale pre-trained model can provide a good prior for training motion prediction model on smaller datasets. To train such a large pretrained model the paper collects existing datasets and pre-trains on them. Since these datasets have different sampling frequencies some special care has to be taken when using them together for pre-training. This is achieved by sampling to a fixed high-frequency rate (50 FPS) and then using extra masked tokens to represent intermediate missing data points.", "summary_of_recommendation": "The paper is interesting and focuses on an important topic. However, the effectiveness of merging disparate datasets for high-quality pre-training remains questionable.  While the results are encouraging, they do not present significant advance from previous approaches.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 2, "quality_of_limitations": 1, "questions_for_rebuttal": "please see above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fQNc7a78Gp", "reviewer": "Reviewer_6LnL", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 5, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper tackles the absence of a unified framework and pre-trained models for multimodal human motion prediction. It proposes a two-pronged approach: constructing a large-scale dataset by integrating and standardizing seven existing datasets with trajectory, pose and bounding box modalities. In addition, it proposes Multi-Transmotion, a transformer-based model pre-trained on this unified dataset. The authors evaluate the model on established benchmarks for trajectory prediction (NBA, JTA) and 3D pose prediction (AMASS, 3DPW), reporting competitive results on standard metrics. Additionally, they explore the model's performance in few-shot learning scenarios.", "summary_of_recommendation": "This work is of high quality and is quite novel. The evaluation is sufficient to demonstrate that the proposed method advances the state of the art in human motion prediction", "technical_quality": 5, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "How is the few-shot learning done? Can you show a scaling curve, i.e., the performance as a function of number of demonstration examples (shots)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "s4mp7qSWww", "reviewer": "Reviewer_ePRe", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper \"Multi-Transmotion: Pre-trained Model for Human Motion Prediction\" proposes a new transformer-based model to predict human motion more accurately by leveraging a novel integration of multiple datasets that include trajectory and 3D pose keypoints. This integration aims to address the lack of a standardized, large-scale multimodal dataset for human motion prediction. The authors developed a unified framework to standardize these datasets, which allowed them to pre-train their model effectively. The model, named Multi-Transmotion, utilizes novel masking strategies to learn rich representations and demonstrates competitive performance on several downstream tasks across different datasets. The study emphasizes the model's robustness and adaptability in handling various modalities and datasets, showing promising results in trajectory prediction for NBA and JTA datasets and pose prediction in the AMASS and 3DPW datasets.", "summary_of_recommendation": "I am currently leaning towards rejection of this paper with lack of method explaination and comprehansive comparision.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 2, "questions_for_rebuttal": "1. How the multi-transmotion works? It is hard to understand the novelty of the proposed framework without any mathematical reasoning.\n2. The comparison between methods did not report correctly and completely.\n3. The sampling is only to align with different fps, the novelty is limited.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: PRE REBUTTAL:

High level summary of reviews:

Strengths:

- Unified data framework and standardization: The paper presents a valuable contribution by unifying and standardizing seven existing datasets, enabling the training of a large-scale model for human motion prediction.
- Multi-Transmotion model architecture: The novel transformer-based architecture is considered a valuable contribution.
- Masking during training: The masking strategy employed during training enhances the model's robustness and efficiency.
- State-of-the-art results: The model achieves state-of-the-art results on several challenging human motion prediction tasks, demonstrating the effectiveness of the approach.
- Few-shot learning: The model's ability to perform well in few-shot learning scenarios, boosting performance in narrow domains with smaller amounts of data, is seen as a strength.

Weaknesses:

- Limited novelty: Some reviewers found limited technical novelty compared to prior works.
- Questionable dataset merging: The effectiveness of merging disparate datasets for high-quality pre-training is questioned.
- Marginal performance gains: Despite positive results, the improvements over baselines are considered marginal, raising questions about the practical significance of the findings.
- Lack of empirical evidence in robotics tasks: The absence of empirical evidence demonstrating the model's applicability to robotics tasks, such as navigation, is seen as a weakness.
- Limited relevance to CoRL community: The relevance of the paper to the robotics community is questioned, with concerns that CoRL attendees may not be the target audience.
- Lack of clarity and comparison: The paper lacks clarity in explaining the framework design and comparisons with other methods, making it difficult to understand the exact input, output, and addressed problem. Reviewer ePRe also says the framework is hard to understand without mathematical reasoning.

POST REBUTTAL:

While this paper marginally crossed the threshold for acceptance, the rebuttal arguments pertaining to low technical novelty and weak results are somewhat unconvincing. A novel composition of existing approaches is fair game for technical contribution (and some of the most impactful papers are along these lines!), however in such cases the final results are used to justify the engineered algorithm, and that is where I think this work is lacking. The authors argue that beating state of the art performance was not the main focus of the paper, but then the contribution that the resultant algorithm "can adapt" to various input characteristics definitely limits impact (since it's likely that small modifications to SoTA methods would also have this property). However, the authors did provide a thorough rebuttal and addressed many other of the reviewer concerns. While there was minimal reviewer engagement during the rebuttal period, the result is a stronger manuscript that just meets the bar for acceptance.
Decision: Accept