Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new pre-training task tailored for end-to-end self-driving. The idea is to take an unsupervised monocular depth model (for instance [1]) and distill the component that takes as input 2 images and predicts the pose between them into a single-image model. The resultant model therefore learns a representation that is useful for predicting short-term future motion of the ego vehicle. The authors get good results when using this network as an initialization for 3 imitation learning CARLA tasks, 1 RL CARLA task, and 1 imitation learning nuScenes task.

[1] "Digging Into Self-Supervised Monocular Depth Estimation"
strength_and_weaknesses: This paper presents an interesting and convincing application of unsupervised depth models for pre-training end-to-end self-driving policies. I found the empirical analysis very thorough; the authors include open-loop and closed-loop experiments in CARLA as well as an open-loop real-world task on nuScenes. I also like the paper at a conceptual level. Unsupervised depth models certainly learn valuable representations and I think it makes sense to probe the extent to which these representations can enable other tasks.

The main weakness that I see is primarily with respect to impact. I'm wondering if the authors have tested how well their model can be used as a pre-training stage for maybe object detection or segmentation? In terms of applicability to self-driving, demonstrating that the author's method boosts detection performance would be a huge win. In many ways, I think the results for motion prediction are very intuitive in that the pre-training stage is basically imitation learning on 1-second long future trajectories, so the main difference between the pre-training stage and testing stage is domain gap. To claim this method is useful for pre-training, I think it's important to show task generalization.

I have a few other comments and edits below:

- page 3 contribution 1 - authors should clarify the scope of this claim. I don't think the goal of this paper is significantly different enough from [1] to be able to claim their paper is the first to attempt this task. If the "without pseudo-labels" part is important, the authors should define pseudo-labels and explain why it's significant to avoid using them.

- page 4 - "casual" should be "causal" I think?

- I recommend visualizing Table 1 as a graph of performance vs. # training samples, with a curve for each pre-training method

- Table 3 and Table 5 - bold the numbers that are better to improve interpretability

- SelfD baseline - SelfD seems more general than the authors' approach in that it trains the encoder to predict the future pose for multiple future timesteps. Could the authors comment on why the authors choose to train their visual encoder to predict only the pose 1 second into the future?

- Have the authors ablated the choice of sampling the videos at 1 Hz for pre-training? I'm wondering if it's important that the pre-training frequency roughly matches the CARLA frequency. If they match, is it correct to say that the pre-training task is equivalent to imitation learning? My sense though is that ACO and SelfD also pre-train with different variants of imitation learning.

[1] "Learning to drive by watching youtube videos" Qihang Zhang, Zhenghao Peng, and Bolei Zhou.
clarity,_quality,_novelty_and_reproducibility: As the authors point out, there are plenty of other papers that also seek to pre-train self-driving models on unlabeled YouTube videos. The authors should correct me if I'm wrong, but my sense is the main novelty in this paper is that they are using the photometric error to train their future-pose prediction model, instead of using a pre-processing stage to estimate future-pose prediction then train the model to predict those labels. Beyond that, I think the experiments are high-quality and the method and experiments were presented clearly.
summary_of_the_review: I think this paper is a principled study in how one can leverage unsupervised depth prediction for pre-training driving policies. Ideally, the authors would demonstrate to what extent the model can be used for standard self-driving perception tasks such as object detection. For the current tasks, it seems mostly like a domain adaptation paper in which the authors train for motion prediction on YouTube videos then fine-tune on synthetic or nuScenes images, in which case I think it's quite intuitive that this pre-training strategy would outperform for instance image classification (Imagenet). If the authors can expand the set of tasks for which their method is a top-performing pre-training strategy, the impact of the paper will increase substantially.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes PPGeo - Policy Pre-training via Geometric modeling,  a driving policy paradigm which uses a self-supervised framework for policy pretraining in visuomotor driving. Policy representations are learnt by modeling 3D geometric scenes (pose and depth) on public datasets. This is in turn done in two stages. In the first stage, the method generates pose and depth predictions with two input consecutive frames. In the second stage, driving policy representations are learnt from a single image by predicting the future ego motion. Experiments are conducted on lots of datasets under different conditions and applications. 

strength_and_weaknesses: The proposed method will be an important contribution in the autonomous driving field. The use of self-supervised learning on large public datasets and utilizing them to learn driving policy is certainly useful. The usefulness of this general method to depth and odometry estimation is also a plus. 

Though the paper’s contribution is important, there are some issues in the presentation of the paper, which if fixed, will make this paper even better.

clarity,_quality,_novelty_and_reproducibility: It will be good if the authors do a better job in describing what policy decisions in autonomous driving conditions are. Though some examples are shown in Figure 1 and Figure 2, it will be better if these policies are explained in more detail with more examples. 

Lots of autonomous driving tasks are described in Section 3.2. It will help if these are explained better with examples (possibly visual illustrations wherever possible). 

It is not clear what metric is shown from Table 1 and Table 2. Many metrics are mentioned in Table 3,4,5, without any reference in the body of the paper. 

The activation maps from Figure 4 do not provide much insight. Almost all the maps of the proposed method are in the center. It will help if more diversity is presented.

It will be good to discuss some failure cases (or some case studies) of the proposed method. Since this paper is in the autonomous driving domain, knowing about failures is important.

summary_of_the_review: The proposed method in the paper appears to be of good value. With improved presentation, the paper can get better rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work targets the end-to-end autonomous driving task from monocular images.  
In particular, they propose **PPGeo**, and leverage large-scale unlabeled driving videos mined from the web in order to pre-train a visual encoder.  
To do this, they first train a DepthNet and PoseNet on their unlabeled dataset, using consecutive images $I_i, I_{i+1}$ to predicting camera intrinsics $K$, depth $D$, and ego-motion $T$ to minimize a photometric loss, following Monodepth2.
In the second stage of pre-training, they freeze the PoseNet and DepthNet, train a separate visual encoder to predict ego-motion from a *single image* and minimize the same loss.
By learning ego-motion, this visual encoder learns important features which are directly aligned with the downstream task (planning).  
They show improved results on the CARLA simulator and real-world nuScenes dataset (open-loop).
strength_and_weaknesses: 
## Strengths

* Good high-level goal - using large amounts of unlabeled data as pre-training
* Variety of experiments on real world datasets (nuScenes) in addition to the CARLA simulator
* Clear writing + notation, particularly method and description of baselines

## Weaknesses

* Table 1,2, 3, 4 show competitive reslts against other pre-training methods, but are missing **all** baselines from literature. 
* Visual encoder seems redundant with PoseNet. In Sec 2.2, rather than using the PoseNet which operates on consecutive frames, the authors use a new visual encoder on single camera images since the authors claim "single input setting aligns with downstream driving tasks". I would argue the contrary that driving should be done using more than one image. Additionally, operating on image sequences allow for the pre-trained PoseNet to directly act as the visual encoder, simplifying Stage 2 of the pipeline.
* Section 2.2 "ego-motion predicted from the PoseNet is too spare": This claim goes without evidence.

## Minor Issues

* Figure 1: Hard to see how this directly motivates the work.
* Figure 2: I am confused why "... Since a car is ahead, We need to STOP" is there.
* Page 7 is quite busy 
clarity,_quality,_novelty_and_reproducibility: **Clarity, Quality:** The high-level ideas are clear, but the figures and results section needs more polish.  
**Novelty:** I found the proposed method to be relatively novel. Even though the heavy lifting of their pre-training stage builds on established methods, showing this idea actually works in end-to-end planning was novel.  
**Reproducibility:** The authors provide thorough details about their method and experiments, I am fairly confident this could be reproduced.
summary_of_the_review: Leveraging large amounts of unlabeled data as a self-supervised pre-training tasks for autonomous driving is a great unexplored direction and well motivated. The writing is clear and I am glad to see experiments across a variety of benchmarks. However, there are a few issues that I currently have that leans me against acceptance - (1) we need to see how their method performs against the current SOTA, not just the self-supervised baselines and (2) question some parts of the design of the pipeline (throwing away the ego-motion from the PoseNet after stage 1).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a two state pre-training technique that uses
unlabelled, uncalibrated driving data. This pre-training is used to improve
performance of policy training networks. 

The two state consists of first, predicting the depth and pose
between consecutive frames by doing odometry and pose change estimation.
Then, as a second step, predict the motion based on a single frame directly.

The paper shows that those proxy tasks applied 
sequentially can be used as a pre-training  that benefits CARLA imitation and reinforcement
learning policies and can also be used for odometry prediction and nu-scenes offline motion prediction.
strength_and_weaknesses: Strengths

1. The paper proposes a novel and general pre-training method that can be used on different contexts
and domains. This includes end-to-end driving in CARLA and odometry prediction on KITTI. It is capable of doing
so by training the backbone network to predict the whole photometric transformation and that correlates
more with actions than previous methods used in literature. The presented results have superior performance
than ImageNet pre-training, largely used on this domain.

2. The experimental section is very vast and clear. The experiments done in CARLA are complemented by
nu-scenes and KITTI. 

3. Pre-trainings for the autonomous driving context are useful since the amount of data available is vast
and methods like this helps to improve diverse applications in autonomous driving which can benefit from
more effective pre-trainings than ImageNet.

Weaknesses 

1.  I had several issues understanding exactly what was being proposed on this paper. The technical session is constituted by around only a single page with no supplementary material for a 3-steps algorithm. I think being concise on explanations is key but I felt I missed some details during this explanation. I was able to get the general idea easily but I still don't think I am able to get all the details even after reading the methodological session a few times. 
     + The lack of a clear mathematical formulation for the whole system really hindered my understanding. For example how precisely, the phase one is used for phase 2 and then adapted for phase 3 (downstream tasks). 
     + As quoted on page for "Then, in the second stage, we replace the PoseNet for ego-motion estimation with the visual encoder prepared for downstream driving policy learning tasks". This description in my opinion is kind of vague, and generate some doubts. What is does " prepared for downstream driving" means exactly ? This is replacing which parts of the network ?
    +  I understood that there are issues if we train the second phase directly but those were not clearly described. Why the second phase is so benefited by the first phase as it was shown on the ablation studies ? Could you provide more details on that ? 
    + I missed some extra justification on using the photometric loss function. Isn't predicting pixels harder than pseudo-actions ? Maybe the paper would benefit going a bit deeper on this discussion. Maybe the fact that there is 2 phase pre-training helps the photometric prediction and makes this method more beneficial.


2. Following a bit the weakness 1 , the paper has, in my opinion, some reproducibility problems.
    +  I couldn't find the description of the architectures used for the different networks. I know that the policy network uses CIRLS architecture but how does that fit exactly with the other steps and the pre-training networks. I guess you have to be pre-training only the resnet-34 for all stages, but that is not clearly specified.
   + From the  metadriverse datatase that was was used.  How the 0.8 million images were chosen ? I feel the dataset has a big impact on performance for this case and there was little discussion on this.

3.  I missed comparison with papers that use action labels like [1] which can also be obtained freely on many of the domains the authors analysed. 



[1] Xiao, Yi, et al. "Action-Based Representation Learning for Autonomous Driving." Conference on Robot Learning. PMLR, 2021.
clarity,_quality,_novelty_and_reproducibility: ### Quality

The general quality of the work is sufficient. 

### Novelty

The method proposed is novel in terms of the whole 3 steps framework that was built. 
The individual steps can be found on previous works and the dataset used is also from a previous
work from the literature.

### Clarity and Reproducibility

One of the main drawbacks of this paper is exactly related to clarity and reproducibility. The paper lacks a clearer explanation of the method and implementation details which hinders its reproducibility. Details for that can be found on points one and two from the weaknesses section.
summary_of_the_review: The paper provides a useful and novel method for pre-training backbone networks
to be used on different autonomous driving tasks.
I would recommend clear acceptance, however I think there are reproduction and clarity issues that 
still need to be addressed. With this I am recommending marginal acceptance.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a new pre-training method tailored for end-to-end autonomous driving from monocular images, PPGeo, which learns policy representations from large-scale unlabeled and uncalibrated YouTube driving videos in a two-stage fashion. The method is novel and useful and has the potential to be applied to different contexts and domains. The empirical analysis and results are very thorough and validate the effectiveness of the proposed approach, and self-supervised pre-training is indeed a promising direction and necessary step for the research field of autonomous driving. On the other hand, reviewers also noted that the paper is not well polished and the method is worth it for appearing at ICLR but not very exciting. Overall, the AC and the reviewers are positive about the contributions of this work and would like to see it in the conference if space allows. 
justification_for_why_not_higher_score: However, all the reviewers noted that (1) the paper is not well polished (which may be fixed in camera ready though), and (2) the method is ok for an acceptance but not exciting. So we would like to recommend an accept (poster) for this paper.
justification_for_why_not_lower_score: The AC and reviewers agreed on the strengths of this paper summarized above and reached a consensus that the paper could be a solid contribution to the community and worth it for appearing at ICLR.
Decision: Accept: poster