Policy Pre-training for Autonomous Driving via Self-supervised Geometric Modeling
Authors: Penghao Wu
Li Chen
Hongyang Li
Xiaosong Jia
Junchi Yan
Yu Qiao
Venue: ICLR 2023 poster (2023)
Keywords: Policy Pre-training
End-to-end Autonomous Driving
Self-supervised Learning

Abstract: Witnessing the impressive achievements of pre-training techniques on large-scale data in the field of computer vision and natural language processing, we wonder whether this idea could be adapted in a grab-and-go spirit, and mitigate the sample inefficiency problem for visuomotor driving. Given the highly dynamic and variant nature of the input, the visuomotor driving task inherently lacks the view and translation invariance, and the visual input contains massive irrelevant information for decision making, resulting in predominant pre-training approaches from general vision less suitable for the autonomous driving task. To this end, we propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pre-training in visuomotor driving. We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos. The proposed PPGeo is performed in two stages to support effective self-supervised training. In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input. In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only. As such, the pre-trained visual encoder is equipped with rich driving policy related representations and thereby competent for multiple visuomotor driving tasks. As a side product, the pre-trained geometric modeling networks could bring further improvement to the depth and odometry estimation tasks. Extensive experiments covering a wide span of challenging scenarios have demonstrated the superiority of our proposed approach, where improvements range from 2% to even over 100% with very limited data.

Published as a conference paper at ICLR 2023
POLICY PRE-TRAINING FOR AUTONOMOUS DRIVING VIA SELF-SUPERVISER GEOMETRIC MODELING
Penghao Wu Li Chen', .onnyyng Li - Xiaosong Jia Junchi Yan Yu Qiao'
"Shanghai AI Laboratory 'University of California at San Diego
3 MoE Key Lab of Artificial Intelligence, Shanghai Jiao Tong University {wupenghao, lichen, lihongyang, qiaoyu (royye@ppe @epilapbbbbp. org.cn (jiaxiaosong, yanjunchi jtu. Code: https://githuts com/ wpp.nrrveeaabb/PGee
ABSTRACT
Witnessing the impressive achievements of pre-training techniques on large-scale data in the field of computer vision and natural language processing, we won- der whether this idea could adapted grab-and-go spirit, and mitigate the sample inefficiency problem for visuomotor driving. Given the highly dynamic and variant nature the input, the visuomotor driving task inherently lacks view and translation invariance, and the visual input contains massive irrelevant in- formation for decision making, resulting in predominant pre-training approaches from general vision less suitable for the autonomous driving task. To this end, we propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully sele-supervised framework curated for the policy pre- training in visuomotor driving. We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scal scale unlabeled and uncalibrated YouTube driving videos. The proposed PPGeo is performed in two stages to support effective self-supervised training. the first stage, the geo- metric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input. In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and op- timizing with the photometric error based on current visual observation only. As such, the pre-trained visual encoder is equipped with rich driving policy related representations and thereby competent for multiple visuomotor driving tasks. As side product, the pre-trained geometric modeling networks could bring further improvement to the depth and odometry estimation tasks. Extensive experiments covering a wide span of challenging scenarios have demonstrated the superiority of our proposed approach, where improvements range from 2% to even over 100% with very limited data.
INTRODUCTION
Policy learning refers to the learning process an autonomous agent acquiring the decision-making policy to perform a certain task in particular environment. Visuomotor policy learning (Mnih et al., 2015; Levine et al., 2016; Hessel et al., 2018; Laskin et al., 2020; Toromanoff et al., 2020) takes as input sensor observations and predicts the action, simultaneously cooperating and training the perception and control modules in an end-to-end fashion. For visuomotor policy models, learning tabula rasa diffieult, where it usually requires prohibitively large corpus of labeled data or en- vironment interactions to achieve satisfactory performance (Espeholt et al., 2018; Wijmans et al., 2019; Yarats al.. 2020).
To mitigate the sample efficiency caveat in visuomotor policy learning, pre-training the visual per- ception network in advance promising solution. Recent studies (Shah & Kumar, 2021; Parisi
'Hongyang Li is the correspondence author. This work was in part supported by NSFC (62206172, 62222607), Shanghai Municipal Science and Technology Major Project (021SH2DDX00102) and Shanghai Committee of Science and Technology (21DZ1 100100).

---

Published as a conference paper at ICLR 2023
Irrelevant Object
Deterministic Signal
Light Weather Variation
(b)
(c)
Uniqueness of visuomotor driving policy learning. The planned trajectory is shown as red points. (a) static obstacles and background buildings (objects in yellow rectangles) are irrelevant to the driving decision; (b) the traffic signal in the visual input (marked with the green box) is extremely difficult to recognize and yet deterministic for control outputs; (c) the pre-trained visual encoder has to be robust to different light and weather conditions. Photo credit from (Caesar et al., 2020).
Figure 1:
al., 2022; Xiao al., 2022; Radosavovic et al., 2022; Shah et al., 2022) have demonstrated that applying popular visual pre- -ee-traiiing approaches, including ImageNet (Deng et al., 2009) classifica- tion, contrastive learning (He et al., 2020; Chen et al., 2020c), masked image modeling (MIM) (He et al., 2022; Xie et al., 2022; Xue et al., 2022), and language-vision pre-training (Radford et al., 2021), could guarantee superior representation for robotic policy learning tasks, e.g., dexterous ma- nipulation, motor control skills and visual navigation. However, for one crucial and challenging visuomotor task in particular, namely end-to-end autonomous driving', the aforementioned pre- dominant pre-training methods may not be the optimal choice (Yamada et al., 2022; Zhang et al., 2022b).
In this paper, we aim to investigate why ever-victorious pre-training approaches for general computer vision tasks and robotic control tasks are prone to fail in case of end-to-end autonomous driving- For conventional pre-training methods in general vision tasks, e.g., classification, segmentation and detection, they usually adopt a wide range of data augmentations to achieve translation and view invariance (Zhang et al., 2016; Wu et al., 2018). For robotic control tasks, the input sequence is generally of small resolution; the environment setting is simple and concentrated on objects (Parisi et al., 2022; Radosavovic et al., 2022). We argue that the visuomotor driving investigated in this paper, is sensitive to geometric relationships and usually comprises complex scenarios.
As described in Fig- (a)), the input data often carry irreeevan information, such background buildings, far-away moving vehicles, nearby static obstacles, etc., which are deemed as noises for the decision making task. To obtain good driving policy, we argue that the desirable model should only concentrate on particular parts/patterns of the visual input. That is, taking heed of direct or deterministic relation to the decision making, e.g., traffic signals in Fig. l(b). However, concurrent pre- tr-iiriinig approaches fail to fulfill such a requirement. There comes a natural and necessary demand to formulate a pre-traaee tre-training scheme curated for end-to-end autonomous driving. We attempt to pre-train a visual encoder with a massive amount of driving data crawled freely from the web, such that given limited labeled data, downstream applications could generalize well and quickly adapt various driving environments depicted in Fig. I(c).
The pivotal question is how to introduce driving-decision awareness into the pre-training process to help the visual encoder concentrate crucial visual cues for driving policy. One may resort to directly predicting ego-motion based on single frame sensor input, constraining the network on learning policy-related related features. Previous literature tackles the supervision problem with pseudo labeling training on either an open dataset (Zhang et al., 2022b) or the target domain data (Zhang al., 2022a). However, pseudo labeling approaches suffer from noisy predictions from poorly calibrated models this is true especially when there exists distinct domain gap such as geographical locations and traffic complexities (Rizve al., 2020).
To address the bottleneck aforementioned, we propose PPGeo (Policy Pre-training via Geometric modeling), fully self-supervised driving policy pre-training framework learn from unlabeled and uncalibrated driving videos. It models the 3D geometric scene by jointly predicting ego- ggo-motion, depth, and camera intrinsics. Since directly learning ego-motion based on single frame input along with depth and intrinsics training from scratch is too difficult, it is necessary to separate the visual en- coder pre- traiiinnn from depth and intrinsics learning two stages. In the first stage, the ego-motion
We use end-to-end autonomous driving and visuomotor autonomous driving interchangeably in this paper.
2

---

Published as a conference paper at ICLR 2023
supe-sue Visuomotor Policy Pre- Pre-triining
Downstream Tasks
Contosation ispas
Siece hanely change
Mocion
PoseNeet
Vivaal Input
Photomatric
Depth D,
DepthNet
a. Stage One
Visual Focooler
fissum
Egoo Mation
Phonomatric Reconsimction
Visuzal Encedler
Siygl Jeanss ispas
PooK))
Stage Two
Policy Learning
Figure 2: Overview PPGeo. (a) We focus on pre-training an effective visual encoder to encode driving policy related information by predicting ego-motion based on single frame input (a.2 Stage Two). As achieving such a goal without labels is non-trivial, the visual encoder is obtained with aid of preceding procedure (a.1 Stage One) with temporal inputs and two sub-networks (pose and depth). In this illustrative example, the ego-vehicle needs to take action of STOP. The ego-motion in (a. is inferred by judging two consecutive frames barely change; whilst the ego- ago-motion in (a.2) is predicted based on single visual input focusing on driving policy related information. As such, the visual encoder could be fine-tuned and applied wide span of downstream tasks in (b).
predicted based on consecutive frames as does in conventional depth estimation frameworks (Go- dard et al., 2017; 2019). In the second stage, the future ego-motion is estimated based the single frame by a visual encoder, and could be optimized with the depth and camera intrinsics network well-learned in the first stage. As such, the visual encoder is capable of inferring future ego-motion based on current input alone. The pre-trained visual encoder could be well adopted for downstream driving tasks since captures driving policy related information. As side product, the depth and pose networks could be utilized as new initial weights for depth and odometry estimation tasks, bringing an additional performance gain. To sum up, our key contributions three-fold:
We propose a pre-training paradigm curated for various visuomotor driving tasks. To the best of our knowledge, this is the first attempt achieve a fully self-supervised framework without any need of pseudo-labels" leveraging the effect of pre-training by large-scale data to the full extent.
We devise a visual encoder capable of predicting ego-motion based on single visual input, being able extract feature representations closely related driving policy. Such design of visual encoder is flexible to extend to various downstream applications.
We demonstrate the superiority of our approach on a set of end-to-end driving scenarios, covering different types and difficulty levels. The performance in terms of various metries is improved from 2% to even over 100% in challenging cases with very limited data.
2
METHODOLOGY
2.1 OVERVIEW
The visuomotor policy learning for autonomous driving targets generating a policy æ, such that makes driving decisions, e.g., control actions planned trajectory, from visual observation Our goal is to pre-train a visual encoder Φ(x), which maps the raw image input to a compact repre- sentation containing important information for driving decision making. The representation is then utilized by the policy π (Ø(x)) ) to perform driving tasks. As shown in Fig. 2, our pre-training method pre-trains the visual encoder on unlabeled driving videos via two stages self-supervised manner.
Pseudo-labels here mean using another model trained additional labeled data create "artificial" labels for the unlabeled dataset.

---

Published as a conference paper at ICLR 2023
2.2 TWO-STAGE SELF-SUPERVISED TRAINING
Stage One: Self-supervised Geometric Modeling, During the first stage, given a target image 1 and source images f in sequence, we jointly estimate the depth of the target image, the intrinsics of the camera, and the 6-DoF ego-motion between these two frames. Given the estimations, we are able to model the 3D geometry of the scene, and reconstruct the target image by projecting pixels in the source images. Formally, the pixel-wise correspondence between I and By is calculated as:
P = KT (P. P.:,
where Pt and Py are the homogeneous coordinates of the pixel in I, and I respectively, K is the predicted camera intrinsic matrix, and (P;) represents the predicted depth value at pixel p; in 4. With this relationship, the target image - could be reconstructed with pixels in Ly, and be optimized by the photometric reconstruction error. Following Godard et al. (2019), we choose two images adjacent to the current frame the source images, ie.,, t' E (t (t- 1,
The DepthNet consists of a common mncoder-decoder structure (Godard et al., 2019) and estimates the depth map of the input image. Two images are stacked together and fed into the encoder of the PoseNet, whose bottleneck feature is then utilized to predict the camera intrinsics and the ego- motion via two separate MLP-based heads. For camera intrinsics estimation, optical center (Cz, cy) and focal lengths f2, fy are regressed similarly as in Gordon et al. (2019); Chanduri et al. (2021).
Stage Two:
Visuomotor Policy Pre-training. After the first stage of training, the DepthNet and PoseNet are well trained and fitted to the driving video data. Then, in the second stage, we replace the PoseNet for ego-motion estimation with the visual encoder ((x) prepared for downstream driv- ing policy learning tasks. Now the visual encoder only takes a single frame image as input and predicts ego-motion between the current frame and subsequent frame.
Specifically, the visual encoder estimates the ego-motion 7:-1441 based on It alone and T based on T followed by inverse operation, respectively. The visual encoder is optimized by the photometric reconstruction error similar to the first stage, aside from modification where the DepthNet and the intrinsics estimation are frozen and not backpropagated. This is empirically observed towards better performance. By doing so, the visual encoder is enforced to learn the actual driving policy, since the ego-motion between two consecutive frames is straightforwardly related to the driving decision or action taken at the current timestamp.
One might argue that the PoseNet trained the first stage could provide pseudo motion labels, with which the visual encoder could be directly supervised. However, the ego-motion predicted from the PoseNet is too sparse compared with the geometric projection approach. In our pipeline, every pixel provides supervision for the visual encoder that inaccurate depth estimation in some pixels could be mitigated by the accurate ones, i.e., it constructs a "global" optimization. In contrast, direct supervision from the PoseNet would be greatly affected by the undesirable prediction inaccuracy and noise results.
Thus far, the backbone of visual encoder d(x) has gained knowledge about the driving policy from the diverse driving videos. It can then be applied to downstream visuomotor autonomous driving tasks the initial weights. Besides, the DepthNet and PoseNet trained this large corpus of uncalibrated video data could also be utilized in depth and odometry estimation tasks.
2.3 Loss FUNCTION
Following Godard et al. (2019), the loss function is comprised of the photometric loss and the smoothness loss. The photometric error is comprised of an é term and an SSIM (structural similarity index measure) term (Wang et al., 2004):
fpe = a (1 SSIM (c + (1- (1 Ly Ie-)))
(2)
2
where we set 0.85 following the practice (Godard et al., 2017; 2019). The smooth loss is:
lg = 0. le 1. + (0,,d/e--100
(3)
where di is the mean- normalized inverse depth map. We also adopt the minimum reprojection loss and auto-masking scheme (Godard et al., 2019) improve self-supervised depth estimation.
4

---

Published as a conference paper at ICLR 2023
3 EXPERIMENTS
All pre-training experiments are conducted on the hours-long unlabeled YouTube driving videos (Zhang et al., 2022b). It covers different driving conditions e.g., geographical locations and weather. We sample 0.8 million frames in total at 1 Hz for training. For the first stage in PPGeo pipeline, we train the model for 30 epochs by Adam (Kingma & Ba, 2015) optimizer with learning rate 10-4 which drops to 10-5 after 25 epochs. For the second stage, the encoder is trained for 20 epochs using the Adam W (Loshchilov & Hutter, 2017) optimizer. A cyclic learning rate scheduler is applied with the learning rate ranging from 10 6 to 10 The batch size for both stages is 128. We use data augmentations including ColorJitter, RamdonGrayScale, and GaussianBlur.
3.1 DESCRIPTION ON COMPARED BASELINES
We use ResNet-34 (He et al., 2016) as the encoder and load different pre-trained weights for the initialization of downstream tasks. We compare PPGeo with pre-training methods including:
Random. We use the default Kaiming initialization (He et al., 2015) for convolution layers and constant initialization for batchnorms.
ImageNet. We use the model weight provided by Torchvision (Marcel & Rodriguez, 2010), which is pre-trained with the classification task on ImageNet (Deng et al., 2009).
MIM. The model pre-trained with the masked image modeling method on the YouTube driving video, which tries reconstruct images with random masked-out patches. SimMIM (Xie et al., 2022) is adopted as it is suitable for convolutional networks.
MoCo. We pre- mre-train the model using MoCo-v2 (Chen et al., 2020c) the YouTube driving videos. We exclude ReszedoReszzedCrop and RandomHorizontalFiii augmentations as they are not suitable for the driving task.
ACO. Following Zhang et al. (2022b), it is pre-trained using action-conditioned contrastive learning on the YouTube driving videos. ACO trains an inverse dynamic model to generate pseudo steer labels for driving videos, based on which steer-based discrimination is added on top of MoCo-v2.
SelfD. SelfD (Zhang et al., 2022a) is pre-training method strictly since it needs to train the whole policy model on the driving video for each task, while other pre-training methods aforemen- tioned provide a general pre-training visual model for all tasks. We still include it for comparison due to its close relationship to our target. Specifically, we follow Zhang et al. (2022a) to train the model for each task with the following pipeline: training on the task data - training on the YouTube data with pseudo-labe label > fine-tuning on the task data.
3.2 DESCRIPTION ON DOWNSTREAM AUTONOMOUS DRIVING TASKS
We carry out experiments under (1) three imitation learning based closed-loop driving tasks in CARLA (Dosovitskiy et al., 2017), (2) one reinforcement learning based driving task in CARLA. and (3) an open-loop planning task on real-world autonomous driving dataset nuScenes (Caesar et al., 2020), to fully validate the effectiveness of PPGeo. We briefly describe each task below.
Navigation. It corresponds to the goal-conditioned navigation task in the CoRL2017 bench- mark (Dosovitskiy et al., 2017). The agent is trained in Town0l and tested in Town02 with unseen weather, and there are no other traffic participants. We use different sizes of training data (from 4K to 40K) following Zhang et al. (2022b) to evaluate the generalization ability of pre-trained vi- sual encoders when labeled data is limited and conduct the closed-loop evaluation. The evaluation metric is success rate, denoting the portion of 50 pre-defined routes finished without any collision. And traffic lights are ignored here. CILRS (Codevilla et al., 2019), a classic image based end-to-end autonomous driving model, is adopted for training and evaluation.
Navigation Dynamic. This is the navigation dynamic task in the CoRL2017 benchmark (Dosovit- skiy et al., 2017). The setting differentiates from Navigation that there other dynamic objects such as randomly generated vehicles, which substantially increases the difficulty of driving safety.
Leaderboard Town05-long. This challenging and realistic benchmark corresponds to the Leader- Board benchmark (CARLA, 2022). We collect 40K training data in Town01, 1, 03, 04, 06 and evaluate on 10 routes in the unseen Town05 (Prakash al., 2021). Due the challenging scenarios in this
5

---

Published as a conference paper at ICLR 2023
Table 1: The Successful Rate of the closed-loop Navigation task (mean by 3 random trials).
Navigation of training samples
Pre-train Method
10% (4K)
20% (8K)
40% (16K)
100% (40K)
Random
0.0 0.0
9.6 5.2
15.3 4.5
73.3
2.3
ImageNet
24.7.1 2.0
42.0
69.3
6.4
87.3
1 4.6
MIM
4.7 1.2
8.0
31.3
1 2.3
57.3
+
MoCo
7.7
9.2
48.7
18.7
4.2
69.3
= 1.2
ACO
24.0 1: 2.0
1..0
71.3
1 1.2
92.0
1 3.5
SelID
12.0 0.0
32.0 0.0
50.7
2.3
62.7
1.2
PPGeo (ours)
42.0 + 2.0
73.3 + 6.1
91.3 1.2
96.7 + 1.2
task, we evaluate different pre-training approaches with the satte-of-the-ee image-based autonomous driving model TCP (Wu et al., 2022). The metrics of this task are Driving Score, Route Completion, and Infraction Score (all the higher the better), where the main metric Driving Score is the product of Route Completion and Infraction Score.
Reinforcement Learning. Proximal Policy Optimization (PPO) (Schulman et al., 2017) is used to train the CILRS (Codevilla et al., 2019) model initialized with different pre-trained weights in CARLA TownO1 environment. The reward shaping details follow Roach (Zhang al., 2021). We also conduct experiments to freeze the pre-trained visual encoder during training to further study the effectiveness of the pre-trained feature representations.
nuScenes Planning. This task involves trajectory planning in real-world dataset nuScenes (Caesar et al., 2020). Given the current visual input, the model plans 3-second trajectory (0.5 Hz), and the planned trajectory is compared with the ground truth log. We also calculate the collision rate, where a collision is defined as overlaps with future vehicles and pedestrians based on planned waypoints. The planning model used here is comprised of visual encoder and GRU-based planner to predict each waypoint auto-regressively. We use the official train-val split for training and evaluation.
3.3 NUMERIC COMPARISON ON DOWNSTREAM TASKS
For imitation learning based closed-loop driving tasks, the evaluation results shown in Table 1- 3. We present the plot between episode return and environment steps of each method in Fig. 3 for the reinforcement learning experiments. The open-loop nuScenes planning results are provided in Table 4. We could observe that PPGeo outperforms other baselines by a large margin in all tasks.
Note that the model is tested under different number of fine-tuning samples from 10% (4K) to full 40K in the Navigation and Navigation Dynamic tasks. In the case of the particularly small size of training samples, PPGeo still demonstrates competitive performance and has larger improvement gap of over 100%. This validates the generalization ability of the pre-trained visual encoder, which important when adapting to a new environment with very limited labeled data. the more chal- lenging and real-world style Leaderboard Town05-long task in Table 3, the model pre-trained with our method achieves the highest driving score and infraction score. PPGco well handles cases where the agent needs to stop, leading to much fewer vehicle collisions and red light infractions.
Since ACO considers steering angles only during pre-training, its performance degrades on more challenging scenarios where brake and throttles also important. SelfD performs slightly better than ACO in complex cases while it significantly degenerates when the task data is limited, as affected by the unsatisfying pseudo labeling model. ImageNet pre-training also shows competitive performance, which might credit to its ability of finding salient objects in the scene when the input contains little irrelevant information (see examples in Sec. 3.5).
3.4 DEPTH AND ODOMETRY ESTIMATION
In this part, we explore whether the large-scale training on uncalibrated data could benefit the depth and odometry estimation models as well and validate the effectiveness of first-stage training. Specif- ically, we employ the DepthNet and PoseNet trained after the first stage as initial weights for Mon- odepthv2 (Godard et al., 2019), and conduct experiments on KITTI (Geiger et al., 2012). Results in Table 5 indicate that pre- trainin on large-scale driving videos could bring performance improve- ment to both depth and odometry estimation tasks, which an additional harvest of our pre-training framework. We refer readers Godard et al. (2019) for details about the metrics of these tasks.

---

Published as a conference paper at ICLR 2023
Table 2: The Successful Rate of the closed-loop Navigation Dynamic (mean by 3 random trials).
Navigation Dynamic-
# of training samples
Pre-train Method
10% (4K)
20% (8K)
40% (16K)
100% (40K)
Random
0.0 + 0.0
2.0 0.0
10.0 + 0.0
32.0 + 8.0
ImageNet
10.7 1.2
28.7
64.7 2.3
72.7 1.2
MIM
7.3 + 1.2
10.3 + 2.5
14.7 3.1
58.7
MoCo
4.7 + 1.2
12.0
28.0 5.3
66.7
ACO
8.0 1.2
12.0 - 0.0
22.0 - 2.0
47.3 1..3:50 5.0
SelID
8,0 0.0
29.3 1.2
38.0 1.6
59.3 - 6.4
PPGeo (ours)
23.3 + 1.2
34.0 5.3
71.3 1.2
84.0 5.3
Table 3: Closed-loop Leaderboard Town05-long task results. Besides three main metrics, infraction details also reported (all the lower the better). Evaluation repeats 3 times with the mean reported.
Pre-train Method
Driving Score
Infraction Score
Route Completion
Collisions podestrian
Collisions vehiele
Collisions layout
Off-road violations
Agent blocked
Red light
violations
Random
33,50 1.67
00.02
60.49.12.93
0.0910.07
11610.40
0.00.10.00
0.44
0.97
0,53.10.12
ImageNet
41.29=322
0.77:10.3
57.5224.87
0.0010.00
0.7120.20
0.1540.01
1.0120116
0.2920.10
MIM
36.39-0.21
0.72.10.04
61.75.12.26
0.1410.11
0.9110.12
0.04.10.07
0.1810.17
0.8710.03
0.14.10.11
MoCo
32.10=2.04
64.0924.01
0.1320.11
0.7920.16
0.4910.07
0.8120.15
0.4520.13
ACO
33.05=3.00
0.67=20.06
$9.52=3.21
0.00+00000
0.69=0.28
0.05=20.07
0.54+0.05
0.9400.08
0.73=0.10 10
SelfD
38.76-3.02
0.65++0.03
68.7247.36
0.17+00.7
0.84-0.18
0.00-+0.00
0.32+0.03
0.75-0.15
0.12-00.8
PPGeo
47.44-5663
0.79:10.08
65.05=3.11
0.04:2005
0.54:0.29
0.00:20.00
0,16::0111
0.7620.10
0.0420.05
Visual Encoder Fine-tuning
Visual Encoder Frozen
ImageNet
ImageNNt
ACO
ACO
PPN
PPG.u
Eptoode Reun
500
Lptoode Meturn
300
200
100
410
700
400
800)
Steps
Steps
Figure 3: Learning curves of the RL agents using PPGeo and three other best pre-training baselines. Left: the pre-trained visual encoder is jointly fine-tuned during RL training; Right: the visual en- coder frozen during RL training. The episode return the mean with standard deviation in shade across three runs with different random seeds.
Table Open-loop nuScenes planning results. We evaluate the Eg distance between model predic- tions and the ground truth trajectory and collision rate in horizons from second to 3 seconds.
L2 (m)
Collision Rate (%)
Pre-train Method
1s
2s
3s
1s
3s
Random
1.621
2,722
3.851
0.550
1.779
3.375
ImagNet
1.331
2.202
3.086
0.315
0.550
1.366
MIM
1.412
2.357
3.331
0.297
0.622
1.507
MoCo
1.528
2.545
3.585
0.560
1.235
2.390
ACO
1.496
2.496
3.519
0.446
1.178
2.223
SelfD
1.419
2.359
3.316
0.353
0.923
2.044
PPGeo (ours)
1.302
2.154
3,018
0.270
0.425
0.941
3.5 VISUALIZATION RESULTS
Here we provide heatmaps the feature representations learned by different pre- vre-training methods using Eigen-Cam (Muhammad & Yeasin, 2020) to show the attended regions in Fig. 4. In many cases (Row 1&2), our model mainly concentrates on the lane in front the ego vehicle, which is highly related driving. And our model PPGeo well captures the specific cues causing the brake

---

Published as a conference paper at ICLR 2023
Table Improvement from our pre-training method on depth and odometry estimation tasks.
Pre-train Method
Depth Estimation
Odometry Estimation
abs 4
sq squrel 4
mmsc log +
a2
a3
Sequence 09
Sequence 10
ImageNet
0.118
0.902
4.873
0.196
0.958
0.981
0.017=00010
0.015-100100
PPGeo
0.114
0.805
4.599
0.186
0.874
0.962
0.984
0.016:0.009
0.013:10.099
Origin
Ours
ImageNet
MoCo
Figure 4: Eigen- Cam (Muhammad & Yeasin, 2020) activation maps of the learned representation from different pre-training methods on the driving video data.
Table 6: Ablative study on key designs of PPGeo on the Navigation task.
Navigation of training samples
Experiment
10% (4K)
20% (8K)
40% (16K)
100% (40K)
Single stage
2.0
53.3 1.2
79.3 + 4,2
92.7 + 2.3
No frozen 2nd stage
32.7
58.0
86.0
92.0
PoseNet direct supervision
18.0
62.0..00
76.7 1 1.2
90.0 1 0.0
PPGeo
42.0
73.3
91.3 1.2
96.7
action including front vehicles (Row 3&4) and traffic lights (Row 5). We also observe that the model pre-trained with ImageNet classification tends to capture salient objects the image. This helpful when the salient objects are straightforwardly related to the driving decision (Row 4); but it may focus on wrong objects when the input contains other irrelevant information (Row 2&3).
3.6 ABLATIVE STUDY
We conduct ablative study different designs of PPGeo the Navigation task in Table 6. Train- ing the visual encoder and DepthNet performance, indicating that it is quite challenging for the visual encoder to learn the correct ego- motion depth estimation is also trained from scratch. Moreover, jointly optimizing the DepthNet in the second stage (Row 2, frozen) degrades the depth estimation quality and harms the per- formance. In Row 3, we observe that utilizing the PoseNet obtained in the first stage to provide pseudo label supervision directly leads to inferior results, since an inaccurate pseudo label impairs the learning process to great extent.
single stage simultaneously (Row 1) leads to an inferior
4 RELATED WORK
Pre-training for NLP and General Vision. Pre-training or representation learning has proved to be an essential key to the success of artificial intelligence. In the field of Natural Language Processing (NLP), with the powerful capability of Transformer (Vaswani et al., 2017), pre-training on large- scale datasets with large models then fine-tuning downstream tasks has become the dominant paradigm (Kenton & Toutanova, 2019; Brown et al., 2020). As for the field of Computer Vision, training specific downstream tasks with the supervised pre-trained weights of visual encoder via
8

---

Published as a conference paper at ICLR 2023
ImageNet classification task is widely adopted. Recently, unsupervised and self-supervised learn- ing methods such as contrastive learning (He et al., 2020; Chen et al., 2020c;b) and masked im- age modeling (Bao et al., 2021; He et al., 2022; Xie et al., 2022: Peng et al., 2022; Xue et al., 2022) have gained impressive improvement over ImageNet pre-training on various vision bench- marks. Very recent vision-language co-training approaches (Radford et al., 2021; Wang al., 2022) demonstrate their extraordinary potential in the domain of multi-modal learning and applications. Yet, these generic representation learning methods adopt various data augmentation techniques to achieve translation and view invariance, while visuomotor driving sets in a highly dynamic environ- ment. In this work, we show that the ever-victorious pre-training methods may not be the optimal choice, and introduce curated paradigm for visuomotor driving policy learning.
Pre-training for Visuomotor Applications. Learning a control policy directly from raw visual input is challenging since the model needs to reason about visual pixels and dynamic behaviors simultaneously. Moreover, training visuomotor models from scratch usually requires tons of labeled data or environment interactions. To this end, recently, Shah & Kumar (2021) shows that feature representations from ResNet (He et al., 2016) pre-trained on ImageNet classification is helpful for RL-based dexterous manipulation tasks. Parisi et al. (2022) conducts extensive experiments on applying "off-the-shelle pre-trained vision models in diverse control domains and validates their benefits to train control policies. CLIP (Radford al., 2021) is also adopted in some embodied AI and robot navigation problems (Shah et al., 2022). Besides borrowing pre-trained weights for visuomotor tasks, researchers in robotics now desire paradigm learning policy representations from raw data directly. Xiao et al. (2022); Radosavovic et al. (2022); Seo et al. (2022); Gupta et al. (2022) inherit the MIM spirit realize visual pre-training for control tasks. Yang & Nachum (2021) investigates unsupervised representation learning objectives from D4RL environments (Fu et al., 2020), and Yamada et al. (2022) further adopts lask-induced approaches to learn from prior tasks. However, compared with visuomotor driving, the visual inputs of such control tasks are less diverse which usually concentrate on objects and are much more compact.
To our best knowledge, ACO (Zhang et al., 2022b) is the only pre-training method customized for autonomous driving. By first training an inverse dynamic model on nuScenes (Caesar et al., 2020), they get pseudo steer labels of the driving videos and then construct the steer-conditioned discrimination for contrastive learning following MoCo. However, ACO ignores other crucial driv- ing factors such as throttle and brakes, and its performance is largely limited by the inverse dynamic model. SelfD (Zhang et al., 2022a) is not strictly designed for pre-training while it also makes use of vast amounts of videos to learn driving policies via semi-supervised learning. It acquires the pseudo labeling knowledge from the target domain. These two methods both depend on the accuracy of pseudo labeling. In contrast, we realize fully self superviedd learning through dense geometric reconstruction, evading the possible adverse effect.
Policy Learning for Autonomous Driving. Visuomotor autonomous driving learns a driving policy directly from sensor inputs in an end-to-end manner (Codevilla et al., 2018; 2019; Liang et al., 2018: Chen al., 2020a; Prakash et al., 2021; Chen et al., 2021; Wu et al., 2022: Shao et al., 2022; Hu et al., 2022; 2023). In essence, the inherent difficulty of the urban-style autonomous driving tasks makes such methods data-hungry. Interfuser (Shao et al., 2022), the current top-1 method on the CARLA Leaderboard (CARLA, 2022), requires 3 million labeled data samples for imitation learning (behavior cloning specifically). RL-based model MaRLn (Toromanoff et al., 2020) needs 20 million environment steps of interaction. The sample efficiency problem greatly impedes the real-world application of such approaches. In this work, we propose a self-supervised pre-training pipeline to learn driving policy related representations unlabeled driving videos, and pave the way for these visuomotor autonomous driving models to further achieve satisfying performance.
5 CONCLUSION
In this work, we have proposed a fully self-supervised visuomotor driving policy pre-training paradigm PPGeo by modeling the 3D geometry of large-scale unlabeled driving videos. Taking a direct approach to infer the ego-motion and benefiting from the two-stage pre-training pipeline, enable the visual encoder to leam driving policies based on single visual input. Our method outperforms the peer pre- pre-trainig approaches by a large margin on a series of visuomotor driving tasks.

---

Published as a conference paper at ICLR 2023
REFERENCES
Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In ICLR, 2021. 9
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurlPS, 2020. 8
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020. 6,
CARLA. CARLA autonomous driving leaderboard. 2022. 5,9
Sai Shyam Chanduri, Zeeshan Khan Suri, Igor Vozniak, and Christian Müller. Camlessmonndeephh: Monocular depth estimation with unknown camera parameters. arXiv preprint arXiv:2110.14347, 2021. 4
Annie S Chen, Suraj Nair, and Chelsea Finn. Learning generalizable robotic reward functions from" in-the-wild" human videos. In RSS, 2021. 9
Dian Chen, Brady Zhou, Vladlen Koltun, and Philipp Krähenbühhl. Learning by cheating. In CoRL,, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020b. 9
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020c. 2, 5,
Felipe Codevilla, Matthias Müller, Antonio López, Vladlen Koltun, and Alexey Dosovitskiy. End-
to-end driving via conditional imitation learning. In ICRA, 2018. 9
Felipe Codevilla, Eder Santana, Antonio López, and Adrien Gaidon. Exploring the limitations of behavior cloning for autonomous driving. In ICCV, 2019. 5, 6,
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009. 2,
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA:
An open urban driving simulator. In CoRL, 2017. 5
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, al. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In JCML, 2018. 1
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012. 6
Clément Godard, Oisin Mac Aodha, and Gabriel J. Brostow. Unsupervised monocular depth esti- mation with left-right consistency. In CVPR, 2017. 3.
Clément Godard, Oisin Mac Aodha, Michael Firman, and Gabriel Brostow. Digging into self- supervised monocular depth prediction. In ICCV, 2019. 3,4,
Ariel Gordon, Hanhan Li, Rico Jonschkowski, and Anelia Angelova. Depth from videos in the wild: Unsupervised monocular depth learning from unknown cameras. In ICCV, 2019. 4
Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martín-Martin, and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206. 11894, 2022. 9
10

---

Published as a conference paper at ICLR 2023
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016. 5.9
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020. 2,9
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022. 2,
Matteo Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In AAAI, 2018.
Shengchao Hu, Li Chen, Penghao Wu, Hongyang Li, Junchi Yan, and Dacheng Tao. St-p3: End- to-end vision-based autonomous driving via spatial-temporal feature learning, In ECCV, 2022.
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du, Tian- wei Lin, Wenhai Wang, Lewei Lu, Xiaosong Jia, Qiang Liu, Jifeng Dai, Yu Qiao, and Hongyang Li. Goal-oriented autonomous driving. In CVPR, 2023. 9
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT, 2019. 8
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015. 5
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. In NeurlPS, 2020. 1
Sergey Levine, Chelsea Finn, Trevor Darrell, and Pieter Abbeel. End-to-end training of deep visuo- motor policies. JMLR, 2016. 1
Xiaodan Liang, Tairui Wang, Luona Yang, and Eric Xing. Cirl: Controllable imitative reinforcement learning for vision-based self-driving. In ECCV, 2018. 9
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv: 2017. 5
Sébastien Marcel and Yann Rodriguez. Torchvision the machine-vision package of torch. In ACMMM, 2010.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning, Nature, 2015.
Mohammed Bany Muhammad and Mohammed Yeasin. Figen-cam: Class activation map using principal components. In IJCNN, 2020. 7,8
Simone Parisi, Aravind Rajeswaran, Senthil Purushwalkam, and Abhinav Kumar Gupta. The unsur- prising effectiveness of pre-trained vision models for control. In ICMI., 2022. 1,
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with vector-quantized visual tokenizers. arXiv preprint arXiv:2208. 06366, 2022. 9
Aditya Prakash, Kashyap Chitta, and Andreas Geiger. Multi-modal fusion transformer for end-to-
end autonomous driving. In CVPR, 2021. 5.9
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICMI., 2021. 2,9
11

---

Published as a conference paper at ICLR 2023
Iijaa Radosavovic, Tete Xiao, Stephen James, Pieter Abbeel, Jitendra Malik, and Trevor Darrell. Real world robot learning with masked visual pre-training. In CoRL, 2022. 2, 9
Mamshad Nayeem Rizve, Kevin Duarte, Yogesh S Rawat, and Mubarak Shah. In defense of pseudo- labeling: An incertainty-awan pseudo-label selection framework for semi-supervised learning. In ICLR, 2020. 2
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. 6
Younggyo Seo, Danijar Hafner, Hao Liu, Fangchen Liu, Stephen James, Kimin Lee, and Pieter
Abbeel. Masked world models for visual control. arXiv preprint 11X4:22206.12244, 2022, 9
Dhruv Shah, Blazej Osinski, Brian Ichter, and Sergey Levine. Lm-nav: Robotic navigation with
large pre-trained models of language, vision, and action. In CoRL, 2022. 2, 9
Rutav Shah and Vikash Kumar. Rrl: Resnet as representation for reinforcement learning. In ICML, 2021. 1, 9
Hao Shao, Letian Wang, Ruobing Chen, Hongsheng Li, and Yu Liu. Safety-enhanced autonomous driving using interpretable sensor fusion transformer. CoRL. 2022. 9
Marin Toromanoff, Emilie Wirbel, and Fabien Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. In CVPR, 2020. 1,9
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017. 8
Wenhui Wang. Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image a foreign lan- guage: Beit pretraining for all vision and vision- liniuae tasks. arXiv preprint arXiv:2208. 10442, 2022. 9
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. 77P, 2004. 4
Erik Wijmans, Abhishek Kadian, Ari Morcos, Stefan Lee, Irfan Essa, Devi Parikh, Manolis Savva, and Dhruv Batra. Dd-ppo: Learning near-perfect pointgoal navigators from 2.5 billion frames. arXiv preprint arXiv: /9/ 100357, 2019.
Penghao Wu, Xiaosong Jia, Li Chen, Junchi Yan, Hongyang Li, and Yu Qiao. Trajectory-guided control prediction for end-to-end autonomous driving: simple yet strong baseline. In NeurIPS, 2022. 6,
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. CVPR, 2018.
Tete Xiao, Ilija Radosavovic, Trevor Darrell, and Jitendra Malik. Masked visual pre-training for motor control. arXiv preprint arXiv:2203.066 07673, 2022. 2,9
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022. 2, 5,
Hongwei Xue, Peng Gao, Hongyang Li, Yu Qiao, Hao Sun, Houqiang Li, and Jiebo Luo. Stare at what you see: Masked image modeling without reconstruction. urXiv preprint urXiv:2211.08882 2022. 2.9 9
Jun Yamada, Karl Pertsch, Anisha Gunjal, and Joseph J Lim. Task-induced representation learning. In ICLR, 2022. 2.9
Mengjiao Yang and Ofir Nachum. Representation matters: offline pretraining for sequential decision making. In /CML, 2021. 9
Denis Yarats, llya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In ICLR, 2020.
12

---

Published as a conference paper at ICLR 2023
Jimuyang Zhang, Ruizhao Zhu, and Eshed Ohn-Bar. Selfd: Self-learning large-scale driving policies from the web. In CVPR, 2022a. 5,
Qihang Zhang, Zhenghao Peng, and Bolei Zhou. Learning to drive by watching youtube videos: Action-conditioned contrastive policy pretraining. In ECCV, 2022b. 5,
Richard Zhang, Phillip Isola, and Alexei Efros. Colorful image colorization. ECCV, 2016.
Zhejun Zhang, Alexander Liniger, Dengxin Dai, Fisher Yu, and Lue Van Gool. End-to-end urban driving by imitating reinforcement learning coach. In ICCV, 2021. 6
13


Reviews:
[{"review_id": "zUxsMyFUNPC", "reviewer": "Reviewer_hNsW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a two state pre-training technique that uses\nunlabelled, uncalibrated driving data. This pre-training is used to improve\nperformance of policy training networks. \n\nThe two state consists of first, predicting the depth and pose\nbetween consecutive frames by doing odometry and pose change estimation.\nThen, as a second step, predict the motion based on a single frame directly.\n\nThe paper shows that those proxy tasks applied \nsequentially can be used as a pre-training  that benefits CARLA imitation and reinforcement\nlearning policies and can also be used for odometry prediction and nu-scenes offline motion prediction.", "summary_of_the_review": "The paper provides a useful and novel method for pre-training backbone networks\nto be used on different autonomous driving tasks.\nI would recommend clear acceptance, however I think there are reproduction and clarity issues that \nstill need to be addressed. With this I am recommending marginal acceptance.", "strength_and_weaknesses": "Strengths\n\n1. The paper proposes a novel and general pre-training method that can be used on different contexts\nand domains. This includes end-to-end driving in CARLA and odometry prediction on KITTI. It is capable of doing\nso by training the backbone network to predict the whole photometric transformation and that correlates\nmore with actions than previous methods used in literature. The presented results have superior performance\nthan ImageNet pre-training, largely used on this domain.\n\n2. The experimental section is very vast and clear. The experiments done in CARLA are complemented by\nnu-scenes and KITTI. \n\n3. Pre-trainings for the autonomous driving context are useful since the amount of data available is vast\nand methods like this helps to improve diverse applications in autonomous driving which can benefit from\nmore effective pre-trainings than ImageNet.\n\nWeaknesses \n\n1.  I had several issues understanding exactly what was being proposed on this paper. The technical session is constituted by around only a single page with no supplementary material for a 3-steps algorithm. I think being concise on explanations is key but I felt I missed some details during this explanation. I was able to get the general idea easily but I still don't think I am able to get all the details even after reading the methodological session a few times. \n     + The lack of a clear mathematical formulation for the whole system really hindered my understanding. For example how precisely, the phase one is used for phase 2 and then adapted for phase 3 (downstream tasks). \n     + As quoted on page for \"Then, in the second stage, we replace the PoseNet for ego-motion estimation with the visual encoder prepared for downstream driving policy learning tasks\". This description in my opinion is kind of vague, and generate some doubts. What is does \" prepared for downstream driving\" means exactly ? This is replacing which parts of the network ?\n    +  I understood that there are issues if we train the second phase directly but those were not clearly described. Why the second phase is so benefited by the first phase as it was shown on the ablation studies ? Could you provide more details on that ? \n    + I missed some extra justification on using the photometric loss function. Isn't predicting pixels harder than pseudo-actions ? Maybe the paper would benefit going a bit deeper on this discussion. Maybe the fact that there is 2 phase pre-training helps the photometric prediction and makes this method more beneficial.\n\n\n2. Following a bit the weakness 1 , the paper has, in my opinion, some reproducibility problems.\n    +  I couldn't find the description of the architectures used for the different networks. I know that the policy network uses CIRLS architecture but how does that fit exactly with the other steps and the pre-training networks. I guess you have to be pre-training only the resnet-34 for all stages, but that is not clearly specified.\n   + From the  metadriverse datatase that was was used.  How the 0.8 million images were chosen ? I feel the dataset has a big impact on performance for this case and there was little discussion on this.\n\n3.  I missed comparison with papers that use action labels like [1] which can also be obtained freely on many of the domains the authors analysed. \n\n\n\n[1] Xiao, Yi, et al. \"Action-Based Representation Learning for Autonomous Driving.\" Conference on Robot Learning. PMLR, 2021.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "q29JyXcddF", "reviewer": "Reviewer_DZe7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new pre-training task tailored for end-to-end self-driving. The idea is to take an unsupervised monocular depth model (for instance [1]) and distill the component that takes as input 2 images and predicts the pose between them into a single-image model. The resultant model therefore learns a representation that is useful for predicting short-term future motion of the ego vehicle. The authors get good results when using this network as an initialization for 3 imitation learning CARLA tasks, 1 RL CARLA task, and 1 imitation learning nuScenes task.\n\n[1] \"Digging Into Self-Supervised Monocular Depth Estimation\"", "summary_of_the_review": "I think this paper is a principled study in how one can leverage unsupervised depth prediction for pre-training driving policies. Ideally, the authors would demonstrate to what extent the model can be used for standard self-driving perception tasks such as object detection. For the current tasks, it seems mostly like a domain adaptation paper in which the authors train for motion prediction on YouTube videos then fine-tune on synthetic or nuScenes images, in which case I think it's quite intuitive that this pre-training strategy would outperform for instance image classification (Imagenet). If the authors can expand the set of tasks for which their method is a top-performing pre-training strategy, the impact of the paper will increase substantially.", "strength_and_weaknesses": "This paper presents an interesting and convincing application of unsupervised depth models for pre-training end-to-end self-driving policies. I found the empirical analysis very thorough; the authors include open-loop and closed-loop experiments in CARLA as well as an open-loop real-world task on nuScenes. I also like the paper at a conceptual level. Unsupervised depth models certainly learn valuable representations and I think it makes sense to probe the extent to which these representations can enable other tasks.\n\nThe main weakness that I see is primarily with respect to impact. I'm wondering if the authors have tested how well their model can be used as a pre-training stage for maybe object detection or segmentation? In terms of applicability to self-driving, demonstrating that the author's method boosts detection performance would be a huge win. In many ways, I think the results for motion prediction are very intuitive in that the pre-training stage is basically imitation learning on 1-second long future trajectories, so the main difference between the pre-training stage and testing stage is domain gap. To claim this method is useful for pre-training, I think it's important to show task generalization.\n\nI have a few other comments and edits below:\n\n- page 3 contribution 1 - authors should clarify the scope of this claim. I don't think the goal of this paper is significantly different enough from [1] to be able to claim their paper is the first to attempt this task. If the \"without pseudo-labels\" part is important, the authors should define pseudo-labels and explain why it's significant to avoid using them.\n\n- page 4 - \"casual\" should be \"causal\" I think?\n\n- I recommend visualizing Table 1 as a graph of performance vs. # training samples, with a curve for each pre-training method\n\n- Table 3 and Table 5 - bold the numbers that are better to improve interpretability\n\n- SelfD baseline - SelfD seems more general than the authors' approach in that it trains the encoder to predict the future pose for multiple future timesteps. Could the authors comment on why the authors choose to train their visual encoder to predict only the pose 1 second into the future?\n\n- Have the authors ablated the choice of sampling the videos at 1 Hz for pre-training? I'm wondering if it's important that the pre-training frequency roughly matches the CARLA frequency. If they match, is it correct to say that the pre-training task is equivalent to imitation learning? My sense though is that ACO and SelfD also pre-train with different variants of imitation learning.\n\n[1] \"Learning to drive by watching youtube videos\" Qihang Zhang, Zhenghao Peng, and Bolei Zhou.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W6vtIb9UtW", "reviewer": "Reviewer_nuYx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work targets the end-to-end autonomous driving task from monocular images.  \nIn particular, they propose **PPGeo**, and leverage large-scale unlabeled driving videos mined from the web in order to pre-train a visual encoder.  \nTo do this, they first train a DepthNet and PoseNet on their unlabeled dataset, using consecutive images $I_i, I_{i+1}$ to predicting camera intrinsics $K$, depth $D$, and ego-motion $T$ to minimize a photometric loss, following Monodepth2.\nIn the second stage of pre-training, they freeze the PoseNet and DepthNet, train a separate visual encoder to predict ego-motion from a *single image* and minimize the same loss.\nBy learning ego-motion, this visual encoder learns important features which are directly aligned with the downstream task (planning).  \nThey show improved results on the CARLA simulator and real-world nuScenes dataset (open-loop).", "summary_of_the_review": "Leveraging large amounts of unlabeled data as a self-supervised pre-training tasks for autonomous driving is a great unexplored direction and well motivated. The writing is clear and I am glad to see experiments across a variety of benchmarks. However, there are a few issues that I currently have that leans me against acceptance - (1) we need to see how their method performs against the current SOTA, not just the self-supervised baselines and (2) question some parts of the design of the pipeline (throwing away the ego-motion from the PoseNet after stage 1).", "strength_and_weaknesses": "\n## Strengths\n\n* Good high-level goal - using large amounts of unlabeled data as pre-training\n* Variety of experiments on real world datasets (nuScenes) in addition to the CARLA simulator\n* Clear writing + notation, particularly method and description of baselines\n\n## Weaknesses\n\n* Table 1,2, 3, 4 show competitive reslts against other pre-training methods, but are missing **all** baselines from literature. \n* Visual encoder seems redundant with PoseNet. In Sec 2.2, rather than using the PoseNet which operates on consecutive frames, the authors use a new visual encoder on single camera images since the authors claim \"single input setting aligns with downstream driving tasks\". I would argue the contrary that driving should be done using more than one image. Additionally, operating on image sequences allow for the pre-trained PoseNet to directly act as the visual encoder, simplifying Stage 2 of the pipeline.\n* Section 2.2 \"ego-motion predicted from the PoseNet is too spare\": This claim goes without evidence.\n\n## Minor Issues\n\n* Figure 1: Hard to see how this directly motivates the work.\n* Figure 2: I am confused why \"... Since a car is ahead, We need to STOP\" is there.\n* Page 7 is quite busy ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FvSm-sfmH6", "reviewer": "Reviewer_Wqac", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes PPGeo - Policy Pre-training via Geometric modeling,  a driving policy paradigm which uses a self-supervised framework for policy pretraining in visuomotor driving. Policy representations are learnt by modeling 3D geometric scenes (pose and depth) on public datasets. This is in turn done in two stages. In the first stage, the method generates pose and depth predictions with two input consecutive frames. In the second stage, driving policy representations are learnt from a single image by predicting the future ego motion. Experiments are conducted on lots of datasets under different conditions and applications. \n", "summary_of_the_review": "The proposed method in the paper appears to be of good value. With improved presentation, the paper can get better rating. ", "strength_and_weaknesses": "The proposed method will be an important contribution in the autonomous driving field. The use of self-supervised learning on large public datasets and utilizing them to learn driving policy is certainly useful. The usefulness of this general method to depth and odometry estimation is also a plus. \n\nThough the paper’s contribution is important, there are some issues in the presentation of the paper, which if fixed, will make this paper even better.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster