Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The authors develop 4 self-supervision schemes for motion forecasting for autonomous driving. These are: lane masking, distance to intersection, maneuver classification, and success/failure classification. They show that, in general, these pretext tasks improve performance and allow them to achieve a good balance between performance and architectural complexity / inference time. 
strengths_and_weaknesses: Strengths:

Overall I find the work very solid. SSL is a popular approach but has not been successfully applied to the important motion forecasting problem in autonomous driving. The 4 approaches are well-motivated and explained and the results are convincing.  



Weaknesses / Questions:

 - I find Figure 1 very hard to parse / interpret. 
 - Perhaps one point to note is that most (all?) data-driven approaches to motion forecasting are self-supervised in the sense that no hand labeling is needed. What is unique about this work is the development of the pretext tasks. 
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
summary_of_recommendation: Overall I find the work to be a nice contribution to the area of motion forecasting that will likely have high impact.  
issues: Only small formating and presentations issues listed above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper proposes to use self-supervised learning to improve trajectory prediction models. More specifically, it proposes four novel self-supervised tasks for motion forecasting.
strengths_and_weaknesses: Strengths:
To my knowledge, this is the first approach to exploit self-supervised learning in motion prediction tasks. The presented results clearly show that self-supervised learning enables increasing the motion forecasting models' performance without a significant increase in the model's complexity and inference times. Overall, the paper is well written, it presents a good review of the SoA and strong experimental results.

Weaknesses:
There are some minor improvements to improve this paper's readability.
1 - In Fig.1., The baseline [2] does not seem to miss all the predictions. The second case seems to be a good prediction. I would recommend the authors revise this statement. Additionally, the figures are ver
2 - Also in Fig.1., please discuss the result for Task 4, the second sub-figure counting from the top. It seems that the proposed method missed predicting a correct mode close to the GT. Why is this the case? Please discuss.
3 - In Fig.2, the caption does not fully describe the figure. In my opinion, a figure should be self-explainable. I recommend the authors to explain the figure in the caption.
4 - Adding an accompanying video to present the qualitative results would increase this paper's contribution.
5 - In Eq. 2 and Eq.1. not all the variables are presented/introduced. Please make sure that all the variables are explained. The same for Eq. 4, what is L_sup?
6 - In Sec. 4.1.3., which maneuvers are considered? How is the output classification vector defined?
7 - In Sec. 6.3., line 251, the presented results do not support the statement "allows it to generalize to other classes better". Adding evaluation results of the proposed method on new datasets that were not used for training to evaluate how self-supervised learning enables improving generalization would be a significant result. The authors may consider to add such evaluation in the paper or appendix section.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: To my knowledge, this is the first approach to exploit self-supervised learning in motion prediction tasks. Although the contribution is incremental, I think the presented results are significant and send a clear message to the community that self-supervised learning can also be employed to improve motion forecasting models. Therefore, I would recommend to accept this paper.
issues: No major issues. Some minor improvements are described above.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This work leverages the potential of self-supervised learning (SSL) to boost representation learning for trajectory prediction. Four pretext tasks are proposed for joint training in trajectory prediction, including lane masking, distance to intersection learning, maneuver classification, and target hitting classification. By incorporating these SSL tasks with typical prediction frameworks, the prediction performance gains without increasing the model complexity and parameter number. The target-hitting classification shows the most prominent performance improvement among all the pretext tasks. 
strengths_and_weaknesses: Strengths: 
- This work exploits the SSL power to enhance the capability of representation learning from different aspects, which may inspire many more approaches for optimizing trajectory prediction frameworks in this direction. 
- The main structure is built upon Land-GCN. All of the pretext tasks bring some performance improvement, in which the success/failure classification gains most with a simple design. 

Weaknesses:
- The reviewer admits that this is the first work to study the SSL tasks in trajectory prediction systematically, but actually, some of the pretext tasks had been proposed in prior works without being termed as SSL. This is the primary weakness of this paper. Specifically, the lane masking task is used as an auxiliary task in VectorNet[12]. The maneuver classification together with "classification+regression" is proposed in CS-LSTM, Multipath[3], etc., to make multimodal predictions. Considering the existing similar tries and the fact that all these SSL tasks contribute to the prediction performance, it is better to make an effective combination of them so that this work would be more solid. 
- The reviewer believes that the tradeoff between performance and model complexity seems to be overclaimed. Boosting the model performance without increasing model complexity and parameter number should be the common advantage of self-supervised learning, but not from any specific design from this work. Secondly, the reviewer wonder if "the best accuracy-simplicity-efficiency" is overstated, as the critical ranking metrics in Argoverse are not the best while some methods are absent in the comparison of Model Params and Inference time.  


robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Considering the fact that 1) the Lane-masking task and the Maneuver classification task had been proposed in previous works, and 2) the best trade-off between complexity and efficiency are drawn from part of the methods, the reviewer believes this work still needs to improve. An ideal direction to make it solid and convincing is to combine different pretext tasks effectively. 
issues: Some minor issues are listed below:
- As I know CS-LSTM had used a similar set of maneuver classification for trajectory prediction in highway scenarios, but in the urban scenarios of Argoverse, is that really enough to cover the driving maneuvers in urban driving (overtaking, cut-in, nudging)? Have you tried more clusters and what about the results? 
-  It is suggested to draw Fig3.a with the performance on the Argoverse test set, which would reflect more difference. It's better to supplement more model results on Figure3.b and Figure3.c (for example, TNT results should be added) to make your conclusion convincing.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper considers motion forecasting for autonomous driving (i.e., predicting the future states of road users). The paper proposes four novel loss terms to be used in combination with a traditional loss (e.g., mean squared error of future agent positions). The four losses are: "lane masking" to learn lane structure, "distance to intersection", "maneuver classification", and "success/failure" to predict trajectory endpoints. Where traditional losses are supervised from the labeled/ground-truth data, the proposed losses are self-supervised in the sense that they leverage pseudolabels created by additional processing of the original data. The method assumes access to ground truth data of road user positions and road structure. Training with these losses increases accuracy in future motion forecasts on the Argoverse dataset; the utility of each individual loss is confirmed by an ablation study. Overall, these losses improve performance over the state of the art.
strengths_and_weaknesses: Strengths:

- The main strength of this paper is in improving performance by crafting novel losses out of existing data. This is important for two reasons: (1) no additional hand-labeling is required, and (2) inference time does not change because these losses could be used with any model architecture.

- Another key strength of this paper is that the text is fully self-contained, clear, and well-scoped. It outlines a high-level and relevant challenge in robot autonomy, proposes a method to address the challenge, and thoroughly discusses why the proposed method worked.

- A third strength of this paper is in its thorough ablation study. The paper clearly summarizes exactly what performance increase each novel loss provides. Furthermore, the paper presents some well-reasoned hypotheses (and more detailed supplementary discussion) about why these losses increase performance.

- Finally, from a quick look through the literature, I agree that there are no other works that thoroughly investigate self-supervision in motion forecasting. I found the following papers [R1,R2,R3].


Weaknesses:

- One weakness (potentially major) of this paper is that the results are limited to a single dataset. While the Argoverse dataset is "large scale," it would be helpful to add some discussion of why this one dataset is sufficient, or why other datasets may not be feasible to use (e.g., NuScenes or Waymo).

- A second weakness (minor) of this paper is that it ignores the case of motion forecasting for interaction simulation. Motion forecasting models are most useful when they can act as simulators for realistic interaction scenarios, which really requires them to generalize (which is, I realize, always a complaint with any neural network model). My understanding of the literature is that models, especially when trained on lane information and road structure, often memorize what they have seen without learning a generic model of how agents actually move around. The current losses explicitly encourage a motion forecasting model to stay near ground truth and potentially memorize the data, so could they inhibit the model's ability to generalize?

- A third (very minor) weakness of the paper is that its self-supervised losses still rely on extensive existing labeled data.

- Finally (major), there is no easy-to-find limitations section within the main document, though some discussion of limitations is spread throughout the results section. The supplementary material also provides a very brief discussion. It would be great if the paper could clearly discuss its limitations, and ideally refute the weaknesses I note above, within the main text.

Update: The paper has been revised to clearly address these concerns, so I am happy to upgrade to my recommendation.


References:

[R1] Janjoš, F., Dolgov, M. and Zöllner, J.M., 2021, July. Self-Supervised Action-Space Prediction for Automated Driving. In 2021 IEEE Intelligent Vehicles Symposium (IV) (pp. 200-207). IEEE.

[R2] Hu, P., Huang, A., Dolan, J., Held, D. and Ramanan, D., 2021. Safe local motion planning with self-supervised freespace forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 12732-12741).

[R3] Luo, C., Yang, X. and Yuille, A., 2021. Self-supervised pillar motion learning for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 3183-3192).
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations section not present
summary_of_recommendation: This paper basically proposes some task-specific loss functions to improve performance on a benchmark in simulation; in that regard, though the results are nice and the text is well written, the underlying story is not very exciting.

I believe the paper can be greatly improved (i.e., switch from reject to accept) by including a fuller discussion of the importance of this work in a broader context. What are the limitations? Can the proposed method be used for high-quality interaction simulation? Can the proposed method generalize to other datasets, or be demonstrated in real-world use?

Update: The authors addressed my concerns well. I think the paper is not revolutionary, but definitely useful and important, hence my "weak accept."
issues: Please see the "weaknesses" section for major issues. Minor issues as follows:

104: "BEV" (presumably bird's eye view) should be spelled out for unfamiliar readers

The lane masking and distance-to-intersection tasks use human-supplied labels of lane structure, so are they really self-supervised? One could certainly argue that labeling lanes is less work or easier than labeling individual objects in video frames, for example. Note, the maneuver classification and success/failure classification tasks do generate pseudolabels, so I would consider them both to be self-supervised.
review_title: null

Meta-review: metareview: The paper studies self-supervised learning pretext tasks for motion forecasting in autonomous driving. The reviewers find the paper to be well written and address an important problem. However, the reviewers raised major concerns, most of which have been thoroughly addressed in the rebuttal. I thank the authors for the engaging discussions during the rebuttal. Some minor concerns still exist. Nevertheless, I agree with the reviewers that the paper is an interesting contribution.
confidence: 5: The area chair is absolutely certain
Decision: Accept (Poster)