Multi-Transmotion: Pre-trained Model for Human Motion Prediction

Authors: yang  gao
po-chien  luan
alexandre  alahi
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 22:40:49

Abstract: The ability of intelligent systems to predict human behaviors is essential, particularly in fields such as autonomous vehicle navigation and social robotics. However, the intricacies of human motion have precluded the development of a standardized dataset and model for human motion prediction, thereby hindering the establishment of pre-trained models. In this paper, we address these limitations by integrating multiple datasets, encompassing both trajectory and 3D pose keypoints, to further propose a pre-trained model for human motion prediction. We merge seven distinct datasets across varying modalities and standardize their formats. To facilitate multimodal pre-training, we introduce Multi-Transmotion, an innovative transformer-based model capable of cross-modality pre-training. Additionally, we devise a novel masking strategy to learn rich representations. Our methodology demonstrates competitive performance across various datasets on several downstream tasks, including trajectory prediction in the NBA and JTA datasets, as well as pose prediction in the AMASS and 3DPW datasets. The code will be made available upon publication.

Main paper: 
 introduction: 
 The research community has witnessed substantial advancements through the adoption of pre-trained models. In natural language processing (NLP), large language models (LLMs) have demonstrated remarkable interdisciplinary proficiency, excelling across a variety of downstream tasks [1, 2] . In contrast, pre-trained models in computer vision (CV) typically exhibit a greater degree of task specificity, which can be attributed to the multimodal nature of visual data [3, 4] . This specificity typically results in reduced efficiency when compared to NLP models. Nevertheless, recent developments in multitask pre-training, exemplified by MultiMAE [5] and 4M [6] , have shown promising capabilities in transferring knowledge across a diverse array of CV tasks. However, this critical gap remains in the field of human motion prediction. Unlike fields such as NLP and CV, human motion incorporates rich representations and manifests through diverse modalities, including keypoints, trajectories, and bounding boxes, each reflecting different aspects of human movement. Despite this complexity, there currently exists no multimodal pre-trained model for accurately predicting human motion. Intuitively, human motion cannot be fully expressed by a single modalities. Thus, we argue that each modality can benefit from the others by integrating multiple modalities into the models. Consequently, the development of a multitask pre-trained model is imperative for this domain.Three principal challenges must be overcome to effectively pre-train a model for human motion prediction. First, the field lacks a comprehensive, large-scale dataset that encompasses various modalities of human motion. Second, a versatile framework is required to handle these diverse modalities, in contrast to previous approaches that typically addressed each modality in isolation. Third, the model must be robust when confronted with incomplete or noisy input data. To address 8th Conference on Robot Learning (CoRL 2024), Munich, Germany. these challenges, this paper proposes a novel approach that integrates multiple datasets, develops a flexible network architecture, and demonstrates its effectiveness in handling noisy data.Due to the absence of a large-scale multimodal human motion prediction dataset, we have undertaken the task of merging several existing datasets, namely Joint Track Auto (JTA) [7] , Trajnet++ [8] , JRDB-Pose [9] , NBA [10] , Human3.6M [11] , AMASS [12] and 3DPW [13] . Each dataset was originally created with different data formats and frame settings. To streamline the training process, we have unified the data framework. This framework standardizes the observation and prediction horizons, as well as the frame rates, ensuring consistency across the merged datasets. This unified framework enables more efficient and effective pre-training of the model, addressing the complexities of multimodal human motion prediction. For the model design, we implement a tokenization strategy that maintains spatial-temporal information across all modalities by applying modality-specific linear projection layers to convert coordinates into hidden dimensions. To further enhance the model's robustness and adaptability, we employ up-sampling padding, sampling masks, and a bi-directional temporal encoder. The up-sampling padding facilitates easy fine-tuning across different frame rates, while the sampling masks simulate various frame rates by masking tokens with different chunk sizes.Figure 1 shows an overview of our work. By unifying the datasets, we are able to pre-train a transformer-based model, Multi-Transmotion, that can predict future trajectories and informative 3D pose keypoints. The flexibility of our model architecture and data framework allows for easy fine-tuning to specific tasks with varying frame settings. Our model achieved competitive results on both tasks, demonstrating the effectiveness of our pre-training techniques, as shown by our ablation studies on few-shot learning and robustness.We summarize the main contributions as follows:• Dataset: We create a unified human motion data framework by merging seven datasets with standardized settings. Additionally, this framework is flexible, allowing for the seamless addition of more datasets or adjustments to frame settings.• Method: We propose Multi-Transmotion, a pre-trained transformer-based model that flexibly adapts to different frame settings, demonstrating strong robustness and efficiency. This model outperforms previous models across several datasets. 
related work: 
 Human trajectory prediction involves forecasting the future positions and movements of individuals based on their past and current trajectories [14] . Data-driven methods have demonstrated remarkable efficacy in human trajectory prediction [15, 16, 17, 18] . Numerous studies have explored social interactions to enhance prediction accuracy, such as social pooling [8, 19, 20] , graphs [21, 22, 23, 24, 25, 26] , and attentions [27] . Diverse architectures have been explored including recurrent neural networks (RNNs) [15, 23, 8] , generative adversarial networks (GANs) [19, 28, 29, 30] , conditional variational autoencoders (CVAEs) [31, 32] , diffusion models [33, 34, 35] , and LLMs [36] . Moreover, integrating transformer architectures with positional encoding has become a useful tool for capturing long-range dependencies and is widely used in trajectory prediction tasks [37, 38, 39, 40] . From a dataset perspective, trajdata [41] provides a unified interface for handling trajectory and map data.Recent studies have also explored leveraging 2D body pose as visual cues for trajectory prediction in image space [42] , with an emphasis on the utility of an individual agent's 3D body pose for trajectory prediction [43, 44, 45] . Thus, a foundational model for this task must adeptly leverage available visual cues, including pose information, to produce accurate predictions.Human pose prediction involves predicting the future coordinates of pose keypoints. The pose keypoints can be extracted with monocular [46] or stereo [47] cameras. When predicting the poses, recurrent neural networks have traditionally been prominent [48] , capitalizing on their ability to capture temporal dependencies in sequential data. Subsequent advancements introduced feed-forward networks as alternatives [49] , while graph convolutional networks (GCNs) were proposed to better capture the spatial dependencies of body poses [50] . Noteworthy innovations include separating temporal and spatial convolution blocks [51] and introducing trainable adjacency matrices [52, 53] . Diffusion [54] has also shown strong performance by repairing and refining pose. Transformer-based approaches have also gained traction for modeling human motion [55, 56] , showcasing significant improvements with spatio-temporal modules [50] . Accordingly, the pre-trained model proposed in this work incorporates attention mechanisms to capture both temporal and spatial pose representations.Masked prediction is a technique used to learn robust representations by reconstructing masked data. BERT [57] uses masked prediction in natural language processing by replacing random words in a sentence with a special mask token and training the model to predict these masked words from the surrounding context. In computer vision, Masked Autoencoder (MAE) [58] applies a similar concept by masking random patches of an input image and training an autoencoder to reconstruct the missing patches, thereby learning meaningful image representations. The masking strategy has also been extended to trajectory prediction. Forecast-MAE [59] and Traj-MAE [59] both leverage the concept of masked autoencoders to enhance the tasks of motion prediction and trajectory prediction, respectively. Traj-MAE employs diverse masking strategies to pre-train trajectory and map encoders, capturing social and temporal information while utilizing a continual pre-training framework to mitigate catastrophic forgetting. Similarly, Forecast-MAE utilizes a novel masking strategy that considers the interconnections between agents' trajectories and road networks, demonstrating competitive performance and substantial improvements over previous self-supervised methods in motion prediction.3 Method 
unified human motion data framework: 
 As previously mentioned, one of the primary challenges in the human motion domain is the variety of datasets. Specifically, different datasets vary in terms of horizon, frame rates, data formats, and pose joint configurations. To address the dataset challenges, we propose a unified human motion dataset framework with standardized settings. We generate data sequences under uniform conditions in a consistent format, specifically 2 seconds of observation and 4 seconds of prediction at 5 frames per second (fps). Since we are handling human motion prediction tasks, all annotations are recorded as coordinates. Regarding pose data, different datasets have varying joint ID configurations. Our unified joint ID configuration is provided in Appendix A.1. As depicted in Table 1 , the dataset framework comprises 7 datasets featuring diverse combinations of modalities. By combining synthetic and real-world data, this framework encompasses over 2 million samples for trajectories and over 1 million samples for 3D pose keypoints. To our best knowledge, it is the largest data framework for human motion prediction currently available. Furthermore, this dataset framework allows for the flexible addition of new data and adjustments to the horizon with different frame rates. Pre-trained models have demonstrated remarkable effectiveness across diverse domains, from language tasks [62] to image tasks [63, 3] . However, a significant gap remains in the development of pre-trained models specifically designed for human motion prediction. This gap stems from the absence of a model architecture capable of managing multimodal motion prediction with the flexibility to accommodate different frame settings and keypoint configurations. To bridge this gap, we proposed Multi-Transmotion (Figure 2 ), a multimodal pre-trained model for multi-task motion prediction, designed to seamlessly integrate all available visual cues while adapting to varying horizons, frame rates, and pose keypoints. To achieve this, we devised dynamic spatial-temporal mask, sampling mask, and bi-directional temporal encoding strategies, enhancing the model's robustness and adaptability to different frame rates and observation horizons. Detailed math reasoning was shown in Appendix A. Tokenization. To retain the spatial-temporal information across all modalities, we apply modalityspecific linear projection layers to tokenize the coordinates into the hidden dimension of the transformer. Specifically, we project 3D x-y-z coordinates for trajectories and 3D bounding boxes/pose keypoints into the hidden dimension. Likewise, the pixel coordinates of 2D bounding boxes/pose keypoints in image space are projected into the hidden dimension. The learned queries are padded and initialized as future motion tokens. These tokens are concatenated with historical motion tokens, after which masking strategies are applied. The combined tokens are then processed by the transformer.Up-sampling padding, sampling mask, and bi-directional encoder. To ensure the model can be easily fine-tuned with different frame rates, we apply up-sampling padding to the valid tokens during pre-training, simulating the maximum fps for the model. The maximum fps is set to 50, as it can be easily downsampled to commonly used fps settings for trajectory prediction tasks (2.5 fps and Observation frames Prediction frames 5 fps) [44, 8, 33] and pose prediction tasks (25 fps and 50 fps) [54, 50] . As illustrated on the left of Figure 3 , the sampling mask allows the model to mask different chunk sizes, simulating various frame rates. The right of Figure 3 shows the difference between the traditional temporal encoder and our bi-directional temporal encoder. Rather than starting the encoding from the first observation, we begin from the last observation and encode observations and predictions separately, facilitating seamless adaptation to different observation and prediction horizons during the fine-tuning process.Dynamic Spatial-temporal mask. After applying the bi-directional encoder and sampling mask, each valid token includes spatial-temporal information, allowing the model to learn detailed positional representations. To make the model more robust, we introduce a dynamic spatial-temporal mask, which randomly masks out a dynamic spatial-ratio (0 < r s < 1) of the total number of 3D/2D pose keypoints for all frames. Meanwhile, we apply a temporal-ratio (r t = 0.1) for trajectory and 3D/2D bounding box modalities, randomly masking some frames of data. This dynamic spatialtemporal mask strategy has made pre-training more effective compared to related works [44, 5] , as demonstrated in our ablation study shown in Appendix A.3. 
datasets: 
 We exclude Trajnet++ [8] and use the training split of the other 6 datasets to to develop a pre-trained model, enabling us to perform few-shot learning on Trajnet++ [8] . For the evaluation of the trajectory prediction task, we employ the NBA [10] and JTA [7] , as they represent the largest trajectory-only dataset and the largest dataset with 3D pose, respectively. To assess performance on pose prediction tasks, we use the widely adopted large-scale AMASS [12] and the smaller-scale 3DPW [13] , to thoroughly evaluate performance in both indoor and outdoor scenarios. Additionally, we also show the model's application on robot navigation task in Appendix A.2 and the robustness against imperfect poses in Appendix A.4.Regarding experimental settings, we follow the pioneering works [33, 44, 54] with the configurations below:• NBA [10] : Predict 4 seconds of trajectory given 2 seconds of past trajectory at 5 fps.• JTA [7] : Predict 4.8 seconds of trajectory given 3.6 seconds of past trajectory at 2.5 fps.• AMASS [12] : Predict 1 second of pose keypoints given 2 seconds of past pose at 25 fps.• 3DPW [13] : Predict 1 second of pose keypoints given 2 seconds of past pose at 25 fps.For evaluation, we use Average Displacement Error (ADE) and Final Displacement Error (FDE) for the deterministic trajectory prediction task on JTA [7] , and MinADE K /MinFDE K as best-of-k ADE/FDE for stochastic trajectory prediction on NBA [10] . Mean Per Joint Position Error (MPJPE) is used for the pose prediction tasks on AMASS [12] and 3DPW [13] . Trajectory prediction on NBA [10] . We selected the large-scale NBA [10] to evaluate our model on the pure trajectory prediction task, as this dataset does not provide visual modalities. To ensure a fair comparison with pioneering works, the test split remained hidden from the model during the pre-training process. We used the same test data provided by [33] for evaluation. Table 2 presents a numerical comparison between our method and several strong baselines. Our approach, which involves pre-training followed by fine-tuning, consistently outperformed all prior baselines. This result underscores the effectiveness of our proposed architecture, which integrates innovative masking and encoding techniques, establishing its value as a pre-trained model. Trajectory prediction on JTA [7] . To demonstrate the effectiveness of 3D pose in augmenting trajectory prediction, we evaluate the model using pure trajectory (T) and trajectory combined with 3D pose (T+3D P) as input modalities on the JTA [7] . Since 3D pose is the most informative visual cue, we fine-tune our model with trajectory and 3D pose, then compare it with Social-Transmotion [44] trained on the same modalities to ensure fairness. As shown in Table 3 , our model effectively leverages knowledge from 3D poses to enhance trajectory prediction. Furthermore, our model demonstrates superior performance in utilizing pose knowledge compared to the previous multimodal trajectory prediction model [44] , which we attribute to our pre-training approach and novel masking strategy. Despite the different frame settings from pre-training, the model's performance remains robust through fine-tuning, demonstrating the flexibility of our pre-trained model.To further examine how the model leverages 3D pose information, we visualize the predictions of our model using Trajectory-only and Trajectory + 3D pose. Figure 5 shows the qualitative results of our model. It is evident from this figure that the last frame of the pose provides valuable information Pose prediction on AMASS [12] and 3DPW [13] . For the pose prediction task, we utilize the AMASS [12] and 3DPW [13] to thoroughly evaluate our model's performance in indoor and outdoor scenarios. Following previous works [54] , we use the same test split and data preprocessing to normalize the poses. Table 4 presents the qualitative results on the AMASS and 3DPW, showing that our model consistently achieves the best or second-best performance across the entire prediction horizon.Table 4 : Quantitative results on AMASS [12] and 3DPW [13] . Figure 6 presents the qualitative results of our approach, with the actions of sitting, walking, and standing shown in separate rows. Benefiting from pre-training on the large-scale motion dataset, the walking scenario (second row) is handled effectively by our model. In the sitting scenario (first row), the prediction quality declines in the last few frames as the agent begins to stand up, a challenging transition even for humans to predict. Overall, our model demonstrates competitive performance on the pose prediction task, evaluated intensively across various scenarios in AMASS [12] . Few-shot learning on small data under new frame settings To further explore the potential of our model on small datasets, we conduct few-shot learning on a maximum of 1k samples and evaluate on approximately 4k samples from the real-world part of Trajnet++ [8] . We fine-tuned our model to adapt to the Trajnet++ [8] frame settings and compared it with a transformer-based baseline model (Autobots [39] ) and an LSTM-based baseline model (Social-LSTM [15] ). Figure 7 shows that Social-LSTM converges faster than Autobots, as transformer networks are typically very data-hungry. However, our Multi-Transmotion model outperforms Social-LSTM, benefiting from the pre-trained knowledge.NumbersFigure 7 : Few-shot learning curve Pre-trained model vs. Specific model To explore the generalization capabilities of models trained on different datasets, we conduct an ablation study to see if the model benefits from pre-trained knowledge. We observe the performance differences between a pre-trained model trained with multiple datasets and specific models trained on individual datasets. Table 5 demonstrates that the pre-trained model consistently outperforms the specific models. This indicates that pre-training on large-scale data significantly enhances the generalization ability of the model for both human trajectory prediction and pose prediction. 
conclusion and limitations: 
 In this work, we introduced a unified human motion data framework and Multi-Transmotion, the first pre-trained transformer-based model in the human motion prediction domain. By employing several novel masking strategies, our model can fine-tune on different frame settings and achieve competitive results on human motion prediction, underscoring its practical value. Additionally, the masking strategies enhance efficiency without sacrificing robustness against noisy input.As for limitations and future work, incorporating additional modalities, such as contextual images and human intentions, can further empower the model and bring it closer to being a foundational model for predicting human motion. We leave this for future exploration. 
 
 Appendix: A.1 Unified Human Motion Data Framework Table 6 presents the detailed joint configurations and the mapping between our unified IDs and the original datasets. To further explore the potential application of our model in robotic tasks, we integrated our predictor into a robotic navigation system.We applied the CrowdNav [67] simulator to generate pedestrian trajectories with social interactions. Consequently, there were approximately 800 training samples and 200 test samples. The social force model [68] was used as the navigator, as it allowed us to easily incorporate the predicted trajectories by adding extra repulsive forces and validate the effectiveness of our predictor. As Table 7 shows both the completion time and the collision rate were reduced after incorporating our predictor, which highlighted the effectiveness of our model in robot navigation scenarios. Additionally, we found that the improvement in completion time could reach up to 14% when increasing the social forces. Masking has been shown to efficiently improve the robustness and generalization ability in transformer architectures [58, 44] . In this ablation study, we pre-train three smaller models on JTA [7] to quickly examine how different masking implementations affect pre-training in human motion prediction. This ablation study is implemented on the same Tesla V100 GPU device. Table 8 shows that while modality-mask and meta-mask yield strong performance, they come with high computational costs. Conversely, the fixed spatial-temporal mask used in [5] improves computational efficiency by consistently dropping a fixed number of tokens but leads to the lowest robustness. Our implementation employs a dynamic spatial-temporal mask that randomly drops tokens, maintaining high computational speed without sacrificing robustness during pre-training. In the real world, it is challenging to capture accurate 3D pose keypoint annotations due to occlusion or sensor noise. Therefore, it is crucial to assess whether the model remains reliable when faced with incomplete or noisy pose input. To better understand the robustness in such scenarios, we also compare with a specific model trained without any masking strategy. Table 9 presents the performance comparison, showing that the masking strategy significantly enhances the model's robustness. Specifically, it reduces performance degradation from 46.7% to 11.0% when dealing with noisy 3D pose keypoints. In addition to simulating noisy pose input, it is more realistic to handle estimated poses from existing pose estimators. In this study, we use the estimated 3D poses generated by an off-the-shelf pose estimator [69] on some clips from the JRDB-Pose [9] . Table 10 shows that our model can also generalize to pseudo 3D poses, improving trajectory prediction compared to using only trajectory data. Although the improvement is modest, it still demonstrates the model's effectiveness in a realistic scenario. , and x 2dP i , respectively. The observed time-steps and predicted time-steps are defined as t = 1, ..., T obs and t = T obs + 1, ..., T pred .Considering a sample with N pedestrians, the model input is X = [X 1 , X 2 , ..., X N ], where X i = {x c i , c ∈ {T raj, 3dB, 2dB, 3dP, 2dP }} depending on the availability of different modalities. The tensor x c i has a shape of (T obs , e c , f c ), where e c represents the number of elements in a specific cue (e.g., the number of 3D pose keypoints) and f c denotes the number of features (e.g., 3 for x-y-z dimension of 3D pose keypoints) for each element.Lastly, the model output is the ego (i=1) pedestrian's motion Y = Y 1 , where Y 1 = {Y k T raj , Y P } containing k future possible sequential trajectories and one future sequential deterministic 3D pose keypoints.During the tokenization process, we use modality-specific MLPs to encode each input modality, resulting in the projected hidden dimension H c i = M LP c (x c i ). Next, we apply Up-sampling Padding and a modality-specific Bi-directional encoder to generate tensors with high-frequency positional information, i.e., H c i = U (H c i ) + B c , where B c is the Bi-directional encoder applying to modality c and U denotes Up-sampling padding. After this step, each token from H c i represents a specific element of one modality at a specific time step (e.g., a token could contain information about the 3D neck keypoint at the first observed frame).The token length of each modality can be calculated as L c = e c * t c , where e c is the number of elements in a modality and t c is the number of frames considered for that modality. During the masking process, we first use the Sampling Mask to mask out specific chunk sizes based on the data configurations, allowing the model to simulate different frame settings. Then, the Dynamic Spatial-temporal mask randomly drops a dynamic spatial-ratio (0 < r s < 1) of the total number of 3D/2D pose tokens for all frames. Meanwhile, we apply a temporal-ratio (r t = 0.1) for tokens related to trajectory and 3D/2D bounding box modalities, as each of these modalities contains only one element, randomly masking some frames of data. After applying the masking, the token length for the first transformer (L 1 ) is given by the following formula:L 1 = L T raj + L 3dB + L 2dB + L 3dP (1 -r d ) + L 2dP (1 -r d ).As the second transformer learns the fine-grained motion interaction between pedestrians, the token length for the second transformer (L 2 ) is calculated as follows:L 2 = N (L T raj + L 3dP (1 -r d )),where N is the number of pedestrians in the scene. After passing through the transformers, the fine-grained tensors are denoted as F H c i , where c represents the output modalities of the model. I.e., c ∈ {T raj, 3dP } Finally, we use modality-specific MLP heads to project the multi-modal x-y trajectories and deterministic x-y-z 3D pose keypoints for the ego (i=1) pedestrian:EQUATION).https://sportradar.com/ 


Decision: PRE REBUTTAL:

High level summary of reviews:

Strengths:

- Unified data framework and standardization: The paper presents a valuable contribution by unifying and standardizing seven existing datasets, enabling the training of a large-scale model for human motion prediction.
- Multi-Transmotion model architecture: The novel transformer-based architecture is considered a valuable contribution.
- Masking during training: The masking strategy employed during training enhances the model's robustness and efficiency.
- State-of-the-art results: The model achieves state-of-the-art results on several challenging human motion prediction tasks, demonstrating the effectiveness of the approach.
- Few-shot learning: The model's ability to perform well in few-shot learning scenarios, boosting performance in narrow domains with smaller amounts of data, is seen as a strength.

Weaknesses:

- Limited novelty: Some reviewers found limited technical novelty compared to prior works.
- Questionable dataset merging: The effectiveness of merging disparate datasets for high-quality pre-training is questioned.
- Marginal performance gains: Despite positive results, the improvements over baselines are considered marginal, raising questions about the practical significance of the findings.
- Lack of empirical evidence in robotics tasks: The absence of empirical evidence demonstrating the model's applicability to robotics tasks, such as navigation, is seen as a weakness.
- Limited relevance to CoRL community: The relevance of the paper to the robotics community is questioned, with concerns that CoRL attendees may not be the target audience.
- Lack of clarity and comparison: The paper lacks clarity in explaining the framework design and comparisons with other methods, making it difficult to understand the exact input, output, and addressed problem. Reviewer ePRe also says the framework is hard to understand without mathematical reasoning.

POST REBUTTAL:

While this paper marginally crossed the threshold for acceptance, the rebuttal arguments pertaining to low technical novelty and weak results are somewhat unconvincing. A novel composition of existing approaches is fair game for technical contribution (and some of the most impactful papers are along these lines!), however in such cases the final results are used to justify the engineered algorithm, and that is where I think this work is lacking. The authors argue that beating state of the art performance was not the main focus of the paper, but then the contribution that the resultant algorithm "can adapt" to various input characteristics definitely limits impact (since it's likely that small modifications to SoTA methods would also have this property). However, the authors did provide a thorough rebuttal and addressed many other of the reviewer concerns. While there was minimal reviewer engagement during the rebuttal period, the result is a stronger manuscript that just meets the bar for acceptance.

Reviews: review: title: title: Multi-Transmotion - Novel transformer training method and architecture applied to human motion prediction. Dataset creation and standardization was crucial.
paper_summary: paper_summary: Predicting human motion is crucial for applications like self-driving cars and robots, but existing models often struggle with the complexity and variety of human movement. This paper proposes a new approach:  First, they create a large, unified dataset by combining seven existing datasets that cover different types of motion (walking, sports, etc.).  Second, they develop Multi-Transmotion, a specialized transformer model that can learn from this diverse data. Finally, they use a novel "masking" strategy to train the model, making it more robust to noisy or incomplete data. Their results show that Multi-Transmotion outperforms previous methods and provides a strong foundation for future research in human motion prediction.
main_review: main_review: This paper represents a valuable contribution to human motion prediction research. Main contributions are as follows:
* The unified data framework across 7 datasets and standardization that included frame rate, common horizons, coordinate and pixel representations and unified pose joint configuration.
* Model architecture with modality specific tokenization, up-sampling padding and bi-directional temporal encoder.
* Masking during training to randomly hide parts of the motion data which enhances the model's robustness and efficiency.
* State of the art results on several challenging human motion prediction tasks, demonstrating the effectiveness of their approach.
* Few-shot learning on smaller amounts of data to boost performance in narrow domains. 

The main weakness of the paper is the lack of empirical evidence of the model's applicability on robotic's tasks like navigation. While it is understandable because it is hard to evaluate on real scenarios, it would be possible to leverage simulation benchmarks (see below).
questions: questions: The method and results are very appealing but having result in a robotics task would make the paper even stronger.
* One suggestion would be to do few-shot learning on a sim navigation task and show that the concepts learned can transfer and are applicable to robot navigation. HuNavSim could be a good benchmark: https://arxiv.org/pdf/2305.01303.
limitations: limitations: 2
review_summary: review_summary: I recommend a weak accept given the strong performance of the model on human motion prediction benchmarks and novel methods.
title: title: Review of Multi-Transmotion: Pre-trained Model for Human Motion Prediction
paper_summary: paper_summary: The paper focuses on the problem of human motion prediction and aims to develop a large scale model for human motion prediction. The hope is that such a large scale pre-trained model can provide a good prior for training motion prediction model on smaller datasets. To train such a large pretrained model the paper collects existing datasets and pre-trains on them. Since these datasets have different sampling frequencies some special care has to be taken when using them together for pre-training. This is achieved by sampling to a fixed high-frequency rate (50 FPS) and then using extra masked tokens to represent intermediate missing data points.
main_review: main_review: **Pros:**

The paper is well written and easy to understand. The overall aim of the paper is also interesting and the problem being tackled is definitely quite relevant. The experimental results do seem to show that the approach works well enough in most cases.  The overall idea of dynamic spatio-temporal masking seems interesting compared to modality specific masking and it seems to provide some improvements over it.

**Cons:**

I am not sure if simply combining a lot of existing datasets will be sufficient to train a large pre-trained model. This is especially because these existing datasets often represent very different activities being performed and hence the dynamics of human motion can vary. For instance, the existing paper combines NBA Sport VU with Human 3.6M (which is concerned with a diverse set of human motions). It is unclear if there is a huge positive transfer between these datasets which can be quite different. The other problem is that the joint mapping between some of these datasets is not consistent (and sometimes missing) and hence its unclear if naively combining all of this data is going to give consistent benefits.

*Limited novelty:* There is also limited technical novelty compared to prior works such as Social-Transmotion, traj-MAE which already proposed modality masking etc. 

*Results:* While the overall results seem positive they definitely don’t have a huge difference when compared to baselines. For example the difference between proposed approach and Social-Transmotion is only 0.02-0.03 (in terms of displacement error). Moreover, even for single-dataset training scenarios (i.e. comparing the benefits of pre-training vs training on a single downstream dataset) the difference is around 0.02-0.03 (displacement error for trajectory prediction). This seems like a huge of computational effort is spent for pre-training to get very little benefits. Similarly, even for few-shot learning examples very old LSTM based approaches (Social-LSTM) seem competitive with proposed approach (difference of only 0.01 in terms of displacement error). Overall, while the results are mostly positive, it’s unclear if they are practical since they require much larger compute and do comprehensively beat baselines. Do the authors have any thoughts on why this happens? Also, maybe the authors should consider expanding upon this more in their paper. 

*Relevance to robotics:* I think one big thing I see is the relevance of the paper for the robotics community. While human motion prediction is an important problem, its unclear if the target CoRL attendees will be the right audience for this paper.
questions: questions: please see above.
limitations: limitations: 1
review_summary: review_summary: The paper is interesting and focuses on an important topic. However, the effectiveness of merging disparate datasets for high-quality pre-training remains questionable.  While the results are encouraging, they do not present significant advance from previous approaches.
title: title: Nice unifying dataset and interesting architecture
paper_summary: paper_summary: This paper tackles the absence of a unified framework and pre-trained models for multimodal human motion prediction. It proposes a two-pronged approach: constructing a large-scale dataset by integrating and standardizing seven existing datasets with trajectory, pose and bounding box modalities. In addition, it proposes Multi-Transmotion, a transformer-based model pre-trained on this unified dataset. The authors evaluate the model on established benchmarks for trajectory prediction (NBA, JTA) and 3D pose prediction (AMASS, 3DPW), reporting competitive results on standard metrics. Additionally, they explore the model's performance in few-shot learning scenarios.
main_review: main_review: The unified human motion dataset is a very nice contribution to the community as evaluating new methods on standard datasets remains tedious and time consuming for researchers. In addition, the proposed transformer-based method that utilizes multiple modalities is a welcoming and novel addition. Overall, the paper provides a good contribution and is well-written.

To make the paper better, this reviewer recommends the following: 
- Explain how the impact of other humans can be accounted for in the proposed architecture, or, clearly state the lack thereof in the Limitations section.
- Explain how other important modalities can be added (this should be possible with the proposed architecture): Human motion prediction is nuanced, this work is missing some important features commonly found in other work. These include, but not limited to: the environment (e.g., occupancy maps), latent vectors encoding the influence of other agents (e.g., [1]), robot camera observation
- Explain how few-shot learning works: Since the proposed method only has transformer encoders, it is not directly clear how the model can do few-shot learning. The authors should explain this clearly.
- Explain the training loss for reproducibility.
 -There are quite a few design choices, e.g. bidirectional positional encoding, Dynamic Spatial-Temporal Mask. It would be great to see ablation studies on these as practitioners may want to know how important these are.

Here are some minor issues:
- It is very strange to call a projection to a continuous vector “tokenization” as it usually refers to converting strings to s sequence of integers. Perhaps projection is a better term. Also, how is the projection done? It is by a multiplying with a matrix? Pass through a MLP? Cross-attend with a learned query vector?
- It would be nice to state how many parameters the model has.
questions: questions: How is the few-shot learning done? Can you show a scaling curve, i.e., the performance as a function of number of demonstration examples (shots)
limitations: limitations: 1
review_summary: review_summary: This work is of high quality and is quite novel. The evaluation is sufficient to demonstrate that the proposed method advances the state of the art in human motion prediction
title: title: I think this paper is not prepared well
paper_summary: paper_summary: The paper "Multi-Transmotion: Pre-trained Model for Human Motion Prediction" proposes a new transformer-based model to predict human motion more accurately by leveraging a novel integration of multiple datasets that include trajectory and 3D pose keypoints. This integration aims to address the lack of a standardized, large-scale multimodal dataset for human motion prediction. The authors developed a unified framework to standardize these datasets, which allowed them to pre-train their model effectively. The model, named Multi-Transmotion, utilizes novel masking strategies to learn rich representations and demonstrates competitive performance on several downstream tasks across different datasets. The study emphasizes the model's robustness and adaptability in handling various modalities and datasets, showing promising results in trajectory prediction for NBA and JTA datasets and pose prediction in the AMASS and 3DPW datasets.
main_review: main_review: This paper summarizes its contribution from two perspectives: Dataset and Method. However, in the introduction, the authors state, "Due to the absence of a large-scale multimodal human motion prediction dataset, we have undertaken the task of merging several existing datasets," and all the experiments are conducted on single datasets. It seems this paper trains a multi-channel transformer with a combined dataset and tests the performance on single datasets. 

The proposed framework looks similar to [42] in the paper, while the key difference is sampling methods without a clear explanation of the framework design, and the comparison is not complete with other methods, where [42] reported the result with T+ 3D P is not the same as the paper reported. which is not show the advance in specific tasks. Because other scenarios such as T + 2D P, T + 2D BB, T + 3D P + 3D BB, and T + 3D P + 2D P + 3D BB + 2D BB is missing.

The proposed framework is hard to understand without any mathematical reasoning, I cannot understand the exact input and output of the proposed methods and the exact problem the proposed methods want to address. If it is only one pretrained framework can deal with multiple datasets, the novelty is not enough.
questions: questions: 1. How the multi-transmotion works? It is hard to understand the novelty of the proposed framework without any mathematical reasoning.
2. The comparison between methods did not report correctly and completely.
3. The sampling is only to align with different fps, the novelty is limited.
limitations: limitations: 2
review_summary: review_summary: I am currently leaning towards rejection of this paper with lack of method explaination and comprehansive comparision.