JointMotion: Joint Self-Supervision for Joint Motion Prediction

Authors: royden  wagner
ömer s ¸ahin tas
marvin  klemp
carlos  fernandez
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-06 11:13:23

Abstract: We present JointMotion, a self-supervised pre-training method for joint motion prediction in self-driving vehicles. Our method jointly optimizes a scene-level objective connecting motion and environments, and an instance-level objective to refine learned representations. Scene-level representations are learned via non-contrastive similarity learning of past motion sequences and environment context. At the instance level, we use masked autoencoding to refine multimodal polyline representations. We complement this with an adaptive pre-training decoder that enables JointMotion to generalize across different environment representations, fusion mechanisms, and dataset characteristics. Notably, our method reduces the joint final displacement error of Wayformer, HPTR, and Scene Transformer models by 3%, 8%, and 12%, respectively; and enables transfer learning between the Waymo Open Motion and the Argoverse 2 Motion Forecasting datasets.

Main paper: 
 introduction: 
 Self-supervised learning (SSL) [1] by design excels in applications with large amounts of unlabeled data and limited labeled data (e.g., [2, 3, 4] ). However, recent SSL methods combined with supervised fine-tuning outperform plain supervised learning with the same amount of data [5, 6] and shorter training times [7] . This makes SSL a versatile choice to improve existing methods in a wide range of applications.In this work, we focus on improving joint motion prediction for self-driving vehicles. Motion prediction aims to predict the future motion of traffic agents given their past motion and environment context, such as lane data and traffic light states. Therefore, motion prediction is essential for estimating future interaction among traffic agents [8] and subsequently assessing the risk of planned trajectories [9, 10] . The majority of recent motion prediction methods perform marginal prediction, where future motion is predicted for each agent individually (e.g., [11, 12] ). In contrast, joint motion prediction predicts scene-wide motion modes (i.e., capturing a joint set of future motion sequences for multiple agents), which enables interaction modeling.Since joint motion prediction requires scene-wide representations, we propose a scene-level SSL objective connecting motion and environment. We combine this with an instance-level SSL objective that refines learned representations, enhancing overall prediction accuracy. As scene-level objective, we match past motion sequences in a scene to the corresponding environment context, such as map data and traffic light states. Thereby, a model learns which sets of motion sequences are likely in a given environment, including interaction among agents. As instance-level objective, we reconstruct masked polylines representing past motion sequences, lanes, and past traffic light states. We complement this with an adaptive pre-training decoder, which leads to the high generalizability of our method.Overall, our main contributions are the design choices that enable our method to generalize across 2 Related work SSL for motion prediction in self-driving vehicles. Inspired by the success of SSL in computer vision (e.g., [3, 4, 6] ) and natural language processing (e.g., [2, 13, 14] ), related methods apply SSL to pre-train motion prediction models for self-driving. PreTraM [15] learns a joint embedding space for trajectory and map data via contrastive learning. During pre-training, the similarity of trajectory and map embeddings from the same traffic scene is maximized, while the similarity to embeddings from other scenes is minimized (i.e., negative examples). This objective is counterproductive when similar scenes are sampled in a mini-batch, e.g., the map embeddings of two four-way stops should be similar, but are erroneously used as negative examples against each other. The concurrent works Traj-MAE [16] , Forecast-MAE [17] , and RMP [18] are masked autoencoding methods for pre-training motion prediction models. They mask trajectory and/or lane tokens and learn to reconstruct them. Traj-MAE and Forecast-MAE represent motion sequences as trajectory polylines, where each polyline point represents only the agents' position omitting available features such as velocity and acceleration profiles, agent dimensions, yaw angles, and temporal order (see Fig. 1 (b)). Furthermore, they do not include traffic light states as environment context. SSL-Lanes [19] extends lane masking with the objectives of distance to intersection prediction, maneuver and success/failure classification. Although these objectives are well adapted to motion prediction, they require non-trivial heuristics (e.g., for clustering maneuvers).Joint motion prediction for self-driving vehicles. Joint motion prediction aims to predict the joint distribution of future motion sequences over multiple agents in a traffic scene. Thus, a predicted motion mode represents a scene-wide set of motion sequences with one sequence per agent. Scene Transformer [20] is an encoder-decoder transformer model for joint motion prediction. The encoder learns global scene-centric representations (i.e., agent, lane, and traffic light features), the decoder transforms these embeddings into joint motion modes. Global position embeddings are used for the learned scene-centric representations, which are more difficult to learn than pairwise relative position embeddings (cf. [21, 22] ). MotionLM [23] reformulates motion prediction as language prediction task and learns a vocabulary of discrete motion vectors. Joint prediction modes are generated by autoregressively decoding sequences of motion vectors for multiple agents. Mo-tionDiffuser [24] performs joint motion prediction as conditional denoising diffusion process. A set of noisy trajectories (i.e., positions disturbed by Gaussian noise) is transformed by a denoiser into a set of trajectories that approximates the ground truth joint prediction mode. The learned denoiser is conditioned on environment context such as lane data and traffic light states. Both MotionLM and MotionDiffuser use an agent-centric encoder [12] , which leads to repeated computation since a traffic scene is encoded for each agent individually. Furthermore, their best performing variants exhibit high inference latency (MotionLM: 250 ms2 , MotionDiffuser: 409 ms).Marginal motion prediction with auxiliary interaction prediction objectives. A marginal motion mode represents a single motion sequence for one agent. Many recent methods extend marginal prediction models with modules for interaction prediction objectives (i.e., dense [25, 26, 27] or conditional prediction [28, 29] ). In such methods, the main prediction modules are still trained in a marginal manner, where training targets are marginal motion modes per agent. The additional modules are trained to combine marginal predictions or the corresponding latent representations by minimizing overlap between them. Thus, these methods do not learn to model scene-wide joint motion modes from ground up, but how to combine marginal modes with less overlap. This limits interaction modeling to the learned re-combination of marginal motion predictions. 
method: 
 We present our method for self-supervised pre-training of motion prediction models in two steps. First, we describe our scene-level SSL objective (connecting motion and environments), then, our instance-level SSL objective (masked polyline modeling) and the adaptive pre-training decoder.Joint motion prediction requires scene-wide representations to decode joint modes, which include predictions of multiple agents within a scene. Therefore, we propose pre-training motion prediction models by connecting motion and environments (CME). This scene-level objective aims to learn a joint embedding space for motion sequences and environment context (see Figure 1 (a) ). Thus, a model implicitly learns which motion is likely in a given environment, including traffic rules and interaction among agents.In detail, we use the past motion of all agents within a scene to generate a scene-level motion embedding (Z M in Figure 1 ) and combine lane data and traffic light states to a corresponding environment embedding (Z E in Figure 1 ). These embeddings are generated by modality-specific encoders (i.e., for motion, lanes, and traffic lights) followed by global average pooling and an MLP-based projector (Proj in Figure 1 ). We perform average pooling on the intermediate embeddings (H M and H E in Figure 1 ) to be invariant to variations in the number of agents and the complexity of environments. Following [30] , we use two separate MLPs with LayerNorm and ReLU activations as projectors, each having a hidden dimension of 2048 and an output dimension of 256. We use the modality-specific encoders of recent motion prediction models (e.g., [20, 21] ) without modifications and remove the additional projector after pre-training. The joint embedding space is learned by similarity learning via redundancy reduction. Following [31, 32] , we reduce the redundancy of vector elements per embedding (i.e., for Z M and Z E individually) and maximize their similarity by approximating the cross-correlation matrix C of Z M and Z E to the corresponding identity matrix:L CME = λ red i j̸ =i C ij 2 + i (1 -C ii ) 2 ,where i, j index the vector dimension of the embeddings Z. The redundancy reduction term is scaled by λ red and ensures that individual embedding elements capture different features. Therefore, it prevents representation collapse with trivial solutions for all embeddings across all scenes (e.g., zero vectors). Non-trivial yet identical solutions are not explicitly prevented, but are unlikely, as empirically shown in [31] . In contrast to [15] , our scene-level objective does not require negative examples, which are difficult to define in this context. Unlike [32] , this objective maximizes the similarity of embeddings from different modalities (i.e., motion and environment) rather than augmented views of the same modality, removing the requirement to develop suitable augmentations.Scene-level representations are well suited to provide an overview of traffic scenes, but lack instancelevel details. For example, the exact position of traffic agents in the past or lane curvatures. Therefore, we combine our scene-level objective with the instance-level objective of masked polyline modeling (MPM) to refine learned representations.Inspired by masked sequence modeling [2, 6] , we mask elements of polylines representing past motion sequences, lanes, and past traffic light states and learn to reconstruct them from non-masked elements and environment context. As shown in Figure 1 , we represent traffic agents with 10 features rather than just past positions (cf. [16, 17] ). In detail, we reconstruct agent positions (x, y), dimensions (l, w, h), acceleration a, velocity v, yaw angle θ yaw , temporal order n t , and classes c A (i.e., vehicles, cyclists, and pedestrians). For lanes, we reconstruct positions (x, y) and lane classes c L . For traffic light state sequences, we reconstruct positions (x, y), temporal order n t , and state classes c TL (i.e., green, yellow, and red). Following [21] , positions, dimensions, accelerations, velocities, and yaw angles are represented as float values and temporal order and agent classes as boolean one-hot encodings.For models with late or hierarchical fusion mechanisms (e.g., [20, 21] ), we use the modality-specific encoders to generate embeddings per modality (see Figure 2 (a) ). Afterwards, we concatenate these embeddings and use a shared local decoder to reconstruct masked sequence elements from nonmasked elements and context from other modalities. As local decoder, we use transformer blocks with PreNorm [33] , local attention [34] with an attention window of 32 tokens, 8 attention heads, rotary positional embeddings [35] , and FeedForward layers with an input and hidden dimension of 256 and 1024.For models that employ early fusion mechanisms (e.g., [12] ), we use learned queries and a shared decoder to reconstruct the input sequences (see Figure 2 (b) ). Such models learn a compressed latent representation for multi-modal input. Therefore, we use learned queries in the same number as input tokens to decompress these representations and reconstruct the input sequences. We use a regular cross-attention mechanism between the learned queries and compressed latent representations and a local self-attention mechanism within the set of learned queries. The resulting transformer blocks have the same hyperparameters as in the late fusion setup.For both variants, we use random attention masks for masking and a masking ratio 60%. As training target, we minimize the Huber loss between the reconstructed polylines and the input polylinesL MPM = λ A L A + λ L L L + λ TL L TL . If not specified otherwise, we set λ A = λ L = λ TL = 1.Encoder A Encoder LEncoder TL[mask][mask][mask] In this experiment, we compare our method with recent self-supervised pre-training methods for autonomous driving using the WOMD dataset. Specifically, we compare with contrastive learning via PreTraM [15] and masked autoencoding with Forecast-MAE [17] and Traj-MAE [16] .Motion prediction models. We pre-train and fine-tune well-established Scene Transformer [20] models on the WOMD training split. We use the publicly available implementation by [21] with 3 modality-specific encoders (i.e., for agents, lanes, and traffic lights). Adapted to the complexity per modality, we encode traffic light features with 1, agent features with 3, and lane features with 6 transformer blocks.Pre-training. For all methods, we add a pre-training decoder for late fusion models (see Figure 2 ) with 3 transformer blocks and the hyperparameters described in Section 3.2. For our method, we additionally add two projectors for scene-level representations as described in Section 3.1. For Pre-TraM, we follow its trajectory-map contrastive learning configuration and add two linear projection layers as projectors for trajectory and map embeddings. For Forecast-MAE, we use a masking ratio of 60% and reconstruct positions of lane polylines and past trajectories by minimizing the MSE loss. We exclude future trajectories, since self-supervised learning by design does not use the same labels as the intended downstream task (cf. [1] ). For Traj-MAE, we use a masking ratio of 60% and reconstruct positions of lane polylines and past trajectories by minimizing the corresponding Huber loss. For our method, we minimize the joint loss of our proposed objectives L JointMotion = λ CME L CME + L MPM . We set λ CME = 0.01 and following [31] the weight of the redundancy reduction term λ red = 0.005.Fine-tuning. For all methods, we replace the pre-training decoder with a shared global decoder and learned anchors for k = 6 motion modes. We initialize the modality-specific encoders with the learned weights from pre-training and do not freeze any weights during fine-tuning. We fine-tune the model using its joint configuration and hard loss assignment. Accordingly, the loss is computed for the best scene-wide joint prediction mode. As post-processing, we follow [36] and adjust the confidences of redundant predictions.Training time, hardware, and optimizer. For all methods, we perform pre-training for 10 hours and fine-tuning for 23.5 hours using a training server with 4 A100 GPUs. For pre-training and finetuning, we use AdamW [37] with an initial learning rate of 1e-4 and a step learning rate scheduler with a reduction rate of 0.5 and a step size of 25 epochs.Results. Overall, pre-training with both proposed objectives (i.e., JointMotion) leads to the best scores across all agent types. This shows that the combination of our objectives works best. Figure 3 further highlights the complementary nature of our two objectives. Specifically, the reconstruction of past traffic light sequences are learned very similar with both configurations. The reconstruction loss for past agent motion converges more slowly with JointMotion pre-training, but reaches similar values as well. However, with our scene-level objective, the lane reconstruction loss likely converges to a higher value. We hypothesize that models pre-trained with our scene-level objective tend to focus more on the overall lane structure than on specific details of individual lane polylines. Figure 4 shows that Scene Transformer models pre-trained with our method achieve higher mAP scores on WOMD than models trained from scratch, even in a shorter wall training time. 
comparing scene-level self-supervision methods: 
 In this experiment, we further compare the scene-level objectives PreTraM [15] and JointMotion. We Scene Transformer, HPTR, and a joint configuration of Wayformer to cover all common types of environment representations in motion prediction (i.e., scene-centric, pairwise relative, and agent-centric).Experimental setup. For Scene Transformer, we use the same configuration as in the previous experiment. For HPTR, we analogously add 3 modality-specific encoders and use a shared decoder for k = 6 motion modes. For the joint configuration of Wayformer, we follow [24] and use a shared encoder for early fusion, which compresses the multi-modal input to 128 tokens. We concatenate multiple such agent-centric embeddings with positional and rotation information into a common reference frame and use a shared decoder to predict joint motion modes. For pre-training the Wayformer model, we add our decoder for early fusion configurations (see Figure 2 ). The Wayformer model is not pre-trainable with PreTraM since an instance-level objective is required to decode the modality-specific tokens from fused representations (cf. Section 3.2). For all models, we employ the same hardware, training time, optimizer, and learning rate scheduling as in the previous experiment.Evaluation is performed on the interactive validation split of WOMD and AV2.Results. Table 2 shows the results of this experiment. Our method consistently outperforms Pre-TraM using different models with varying environment representations and fusion mechanisms. Our method improves all models, while the improvement of the scene-centric Scene Transformer model (e.g., 12% lower minFDE) is most significant and the improvement of the agent-centric Wayformer model is least significant (e.g., 3% lower minFDE). Hence, the improvements are inversely proportional to the sample efficiency of the models. In scene-centric modeling, one sample is generated per scene, whereas in agent-centric modeling, one sample is generated for each traffic agent within a scene. This aligns with the finding that fine-tuning with more samples generally reduces the value of pre-training [38] . Unlike PreTraM, the proposed adaptive pre-training decoder combined with our instance-level objective enable our method to adapt to models with early fusion mechanisms (e.g., Wayformer). Specifically, modality-specific masked polyline modeling with learned queries in the same number as input tokens enables our method to decode modality-specific tokens (i.e., agent, lane, and traffic light) from compressed latent representations. This is particular relevant since the current stateof-the-art methods on the Argoverse 1 Forecasting (ProphNet [11] via AiP tokens) and Argoverse 2 Forecasting (QCNeXt [39] via query-centric modeling) benchmarks rely on fusion mechanisms with compressed latent representations as well.Furthermore, our pre-training leads to comparable improvements on the interactive and the regular validation splits (cf. Table 1 ), while pre-training with PreTraM leads to smaller improvements on the interactive validation split. We hypothesise that with our additional instance-level objective, more fine-grained trajectory details are learned, which is more important for close trajectories of interacting agents. The lowest block shows that our method leads to transferable representations. Specifically, pre-training on WOMD improves fine-tuning AV2.In this experiment, we compare our method with state-of-the-art methods for joint motion prediction in autonomous driving.Experimental setup. We pre-train HPTR (configured as in Section 4.2) for 10 hours using JointMotion and fine-tune for 100 hours. We evaluate the methods in test and validation splits of WOMD. We use the official challenge website to compute performance metrics.Results.  
conclusion: 
 In this work, we introduced a SSL method for joint motion prediction of multiple agents within a traffic scene. Our SSL framework integrates scene-level and instance-level objectives that operate complementarily, enhancing the training speed and accuracy of motion prediction models. Notably, JointMotion outperforms recent contrastive and autoencoding methods for pre-training in motion prediction. Moreover, our method generalizes across different environment representations, information fusion mechanisms, and dataset characteristics. Our evaluations demonstrate significant performance improvements over non-ensembling joint prediction methods or joint-prediction variants of marginal-prediction architectures, underscoring the robustness and effectiveness of our proposed method.Limitations. Our method is not as label-efficient as SSL methods in computer vision, which generate supervisory signals from raw images (e.g., [3] ). Our pre-training does not include the downstream labels for motion prediction (i.e., future motion), but relies on past motion and map data.The Waymo Open Motion Dataset (WOMD) [41] is comprised of over 1.1 million data points extracted from 103,000 urban suburban driving scenarios, spanning 20 seconds each. The state of object-agents includes attributes like position, dimensions, velocity, acceleration, orientation, and angular velocity. Each data point captures 1 second of past followed by 8 seconds of future data. We resample this time interval with 10Hz.The Argoverse 2 Motion Forecasting Dataset (AV2) [42] is comprised of 250,000 urban or suburban driving scenarios each spanning 11 seconds, with 5 seconds of past and a 6 seconds of future data. The datasets entails interactions in over 2,000 km of roadways across six geographically diverse cities. 
: 
 (b) (a) [mask] [mask]X M X E H M H E [mask][x, y, n t , c TL ]  
 
 Appendix: This latency is linearly extrapolated, as inference latency is only reported up to the second best/largest model (see Table8in the Appendix of[23]) 


Decision: ### Strengths:

- Re: Methodology: R-ffHe and R-ZYgU mention innovation in the scene-level self-supervised learning (SSL) that connects motion and environment without requiring negative examples, enhancing scene-wide representation learning; they highlight effective application across different environment representations and information fusion mechanisms, and they highlight that the instance-level SSL reconstructs masked polylines for motion sequences, lanes, and traffic lights, refining detailed representations. R-ffHe mentions that the Adaptive Pre-Training Decoder enables generalization across various datasets and environment representations, showing adaptability in models with early fusion mechanisms.
    
- Re: Experiments: All reviewers cite the manuscript for having extensive experiments that demonstrate that JointMotion outperforms recent SSL methods and state-of-the-art (SOTA) methods in joint motion prediction. They highlight significant improvements over non-ensembling joint prediction and joint-prediction variants of marginal-prediction architectures. They suggest that the method demonstrates generalizability across different backbone models.
    
- Re: Clarity: R-ZYgU mentions that the manuscript is generally well-written, allowing readers to follow through the sections and understand the methodology and results.
- Re: Contribution: R-XhvA highlights the work for combining scene-level similarity learning and masked autoencoding, which they say suggests complementary properties and contributing to the field.
    

### Weaknesses:

- Re: Methodology:  R-ZYgU and R-XhvA express concerns over some methodological details that require familiarity with referenced models, suggesting a need for summarizing these details in supplementary materials. They also mention a lack of clarity in the pre-training procedures and the relationship between contributions and improvements.
- Re: Experiments: R-ZYgU and R-XhvA cite concerns over inconsistent experimental settings in Tables 2 and 3. R-XhvA expressed concerns over missing ablation studies for masked autoencoding from JointMotion, affecting the robustness of the results.
- Re: Clarity: R-ZYgU and R-XhvA cite concerns over the lack of clarity in the Figure 1 and Figure 3 descriptions; they mention that Figure 4’s training time comparison lacks clarity and might mislead readers. R-ffHe and R-XhvA assert that major revisions needed to improve organization and clarity and that specific sections, such as experiment setup and pre-training procedures, need better descriptions.
- Re: Novelty: R-ffHe and R-XhvA mention issues in originality, mentioning that scene-level similarity learning is not considered a major contribution due to extensive prior research and that combining existing techniques is seen as incremental rather than groundbreaking.
    
- Re: Limitations: All reviewers point out that the Limitations section is not well addressed, that the impact is considered incremental with no significant advances in robotics or machine learning, and that the reported improvements fall short in various SOTA metrics.

### Post-rebuttal Meta Review Statement

I applaud the reviewers and authors for engaging in a productive and efficient discussion that resulted in several improvements to the paper and a notable increase in a reviewer score.

While concerns still remain for this work (e.g., incremental contribution and lacking detailed analysis of convergence properties, discussed by R-XhvA), I do feel as if the stated strengths of the manuscript, in sum, sufficiently outweigh the limitations.

I recommend for Accept as Poster.

I encourage the authors to follow through with all the promised changes and to incorporate all additional reviewer requests, to improve the paper even further.

Reviews: review: title: title: Review for submission 284
paper_summary: paper_summary: This paper presents JointMotion, a self-supervised pre-training method for joint motion prediction in self-driving vehicles. The main idea is to improve the prediction accuracy of traffic agents' future motion by learning joint scene representations and refining learned representations through complementary scene-level and instance-level objectives.
main_review: main_review: Strengths:
* Scene-level Objective: The paper proposes a scene-level SSL objective that connects motion and environment. By matching past motion sequences to the corresponding environment context, the model learns which sets of motion sequences are likely in a given environment, including the interaction among agents. This objective does not require negative examples and maximizes the similarity of embeddings from different modalities, enhancing the model's ability to learn scene-wide representations.
* Instance-level Objective: The paper combines the scene-level objective with the instance-level objective of masked polyline modeling. By reconstructing masked polylines representing past motion sequences, lanes, and past traffic light states, the model refines the learned representations and captures more details at the instance level.
* Adaptive Pre-training Decoder: The paper introduces an adaptive pre-training decoder that enables the method to generalize across different environment representations, information fusion mechanisms, and datasets. This decoder allows the model to adapt to models with early fusion mechanisms and decode modality-specific tokens from compressed latent representations.
* Experimental Results: The paper conducts extensive experiments to compare JointMotion with recent self-supervised pre-training methods and state-of-the-art methods for joint motion prediction. The results show that JointMotion outperforms other methods and leads to significant performance improvements over non-ensembling joint prediction methods or joint-prediction variants of marginal-prediction architectures.


Weaknesses:

* It appears that utilizing similarity learning to acquire the joint embedding space for motion and environments may not be considered a major contribution as this kind of approach has been extensively studied in [23].
* As listed in the limitation part, future trajectories should also be valuable for pre-training, at least same importance as history information.
questions: questions: * Could you kindly elaborate on why explicit scene-level objective methods typically tend to yield better and more balanced performance across all agent types in comparison to implicit methods?
* Could you please explain why there is a worse performance of OR (+3.75%) when pre-trained with WOMD and fine tune on AV2
* There is a typo in line 196: two without.
limitations: limitations: 1
review_summary: review_summary: I recommend for a weak accept of this paper.
title: title: Self-supervision learning for joint motion prediction
paper_summary: paper_summary: This paper proposes joint self-supervision learning for joint motion prediction. It employs scene-level self-supervised learning to connect motion and environment information from past trajectories of each agent and HD map polylines. The scene-level self-supervised learning is further combined with instance-level self-supervised learning to reconstruct masked trajectories and map polylines to improve prediction accuracy.
main_review: main_review: Strengths:
This paper explores self-supervised learning at both the scene level and instance level to combine motion and map information for joint motion prediction.

The paper shows that this self-supervised learning approach can be effectively applied across different environment representations and information fusion mechanisms on both commonly used benchmark datasets, including the Waymo open motion dataset and the Argoverse 2 motion forecasting dataset.

The paper is well-written and the results are quite promising.

Weaknesses:
The methodology lacks some details, which require readers to be familiar with the detailed settings of models [21], [22], [31], and [32]. It would be easier for readers to summarize such details in the supplementary material.

The description of Figure 3 is not accurate. I do not see a strong similarity in the reconstruction of past motion with both configurations. The explanation of the disparity in the lane reconstruction with both configurations is not clear and convincing to me.

In Figure 1, the training from scratch stops even though it still depicts an increasing trend. It is not clear why the training from scratch is not compared with fine-tuning up to the same total number of training steps.

It seems that the experimental settings for Tables 2 and 3 are different since the numbers for JointMotion (HPTR) on WOMD are different. The authors should clarify these differences.
questions: questions: Overall, the paper shows the potential of exploring self-supervised learning at both the scene level and instance level to combine motion and map information for joint motion prediction. But the weaknesses mentioned above should be addressed.
limitations: limitations: 2
review_summary: review_summary: The paper has some drawbacks but they can be probably well addressed in the rebuttal. Hence, my rating is a weak accept
title: title: Incremental Advancement of Using Self-supervision Learning for Joint Motion Prediction
paper_summary: paper_summary: This paper JointMotion brings another way to employ data-efficient self-supervised learning (SSL) pre-training method to enhance existing autonomous driving joint motion prediction models. Unlike previous works, in this work, SSL is applied in two-fold to: (1) non-contrastive similarity learning for past motion sequences and environmental context at the scene level, creating robust joint scene representations (in section 4 this is also called Connection motion and environment (CME)); and (2) masked autoencoding and adaptive decoding to refining multimodal polyline representations, including motion, lanes, and traffic lights at the instance level. The authors’ experiment show this method outperforms existing SSL methods and is able to generalize across backbone models.
main_review: main_review: Before the review content I’d like to add some background glossary to disambiguate for those unfamiliar. 
- Joint vs Marginal: The “Joint Motion Prediction” terms refers to motion prediction models that output non-conflict, interaction-aware trajectories for all agents in a scene. In contrary, marginal prediction refers to the models that output trajectory for each agent. Such models can still make predictions for each agent in the scene, but their consideration for interactions is limited and might introduce conflicts. With the terminology clarified, we can see the need for a good joint predictor, and part of this is a good scene centric representation, which the author claim can be improved with SSL.
- Scene level vs instance level representation:  In the context of this article, scene level representation encodes the interactions and relationships among all agents and environmental features, whereas instance level encodes detailed attributes and features of individual agents and environmental elements, focusing on fine-grained information. 

### Quality
- [Strength] Ample experiments: The author compared against SSL baselines who (1) only used scene-level similarity objective (3) only used masked autoencoding. The author also show JointMotion’s generalizability on the backbone model by experimenting with (1) Scene Transformer, (2) HPTR, and (3) Wayformer, which are industry recognized models.
- [Weakness] The author did not seem to experiment ablating the masked autoencoding from JointMotion.

### Clarity
- [Strength] The article is divided into clear sections, allow readers to follow through.
- [Weakness] The procedure for pre-train deserves more description. For example, when you use JointMotion on Scene Transformer, do you freeze any weights at fine-tune? What data do you use in pre-train vs fine-tune? During pre-train do you train the scene-level and instance-level separately then copy the weights to the backbone model to further fine-tune?
- [Weakness] In section 1 the clarity of your summary of contributions can be improved by drawing a clear relationship between each of your design choice and its improvement, such as saying the scene similarity learning improved balance performance across agent types.
- [Weakness] Section 4.2 Experiment setup needs clarity improvement: correct me if I’m wrong. I infer from the context that you base your JointMotion and the other baselines on top of the Scene Transformer by adding the CME, masked autoencoding and/or decoder. At line 173, you mentioned you used pre-training, is this only applied to JointMotion? By placing the sentence “For pre-training…” before the next sentence “For our method…”, you make the former sentence seem like pre-training is not part of your method but rather applied to the baselines as well.
### Originally
- [Strength] Though both scene level similarity learning and masked autoencoding methods have been applied to motion prediction, this work is the first to combine them and suggests their complementary property. This contributes to the field of autonomous driving.
- [Weakness] While the combination of techniques is novel, the individual components are extensions of existing methods. The originality might be seen as incremental rather than groundbreaking.

### Significance
- [Strength] The reported improvements in trajectory metrics (mAP, minADE etc) are significant as compared to other SSL methods, suggesting the effectiveness of the method.
- [Weakness] The reported improvement as compared to SOTA prediction methods falls short in various metrics. Thus in Table 3: It might be helpful to include the number of parameters in your model and the others to show the difference in model capacity.
questions: questions: 1. In section 1 contribution c: by saying your method can generalize across datasets, how many different datasets did you experiment? I only see WOMD and Argoverse 2. In addition, saying a model that can work on two datasets is generalizable is not rigor, unless you are talking about zero-shot generalizability, which does not seem the case in your work.
2. Page 2 line 27: you listed reference 11 (Motion Transformer) as an example of marginal prediction model without interaction modeling. However, in section 3.1 of that paper, the author stated their model perform interaction prediction as part of the scene context encoding. 
3. Page 5 line 181: typo `ration`->`ratio`.
4. Page 5 line 184, do you intent to include $\lambda_{red}$ in the total loss defined at line 183?
5. Page 5 line 196, there is a typo where you wrote 2x `without`.
6. Page 6 line 205: how do you conclude lane reconstructions are more accurate in this case? Is figure 3 a plot of the training time loss? Indeed the blue curve decreases slower but they do not seem fully converged yet. You can say without scene-level objective, the training loss of lane reconstruction converges slower than that of agent state and traffic light but I don’t think it is evident enough to conclude lane reconstruction is less accurate.
7. You have experiments ablating the CME from JointMotion. Do you also have experiments ablating masked autoencoding and compare with the scene level SSL method PreTraM?
8. Page 6 figure 4: I’m a little confused by this figure. If I understand correctly, the orange curve and blue curves are meant to represent the $mAP$ progress under the same training time? First, your x-axis is step rather than time. Second, the orange curve (train from scratch) takes $1.8e5$ steps whereas the blue curve (pre-train + fine-tune) takes $0.8e5$ steps for pre-train plus $1.8e5$ steps for fine-tuning. Only the fine-tuning part should have taken the same time as the entire orange part given the model is the same, right? Are you sure the train-from-scratch configuration is given the same amount of time as the pre-train-then-fine-tune configuration?
limitations: limitations: 3
review_summary: review_summary: The contribution is incremental. The article needs multiple clarifications.