Model-Based Imitation Learning for Urban Driving
Authors: Anthony Hu
Gianluca Corrado
Nicolas Griffiths
Zachary Murez
Corina Gurau
Hudson Yeo
Alex Kendall
Roberto Cipolla
Jamie Shotton
Venue: NeurIPS 2022 Accept (2022)
Keywords: Autonomous Driving
Imitation Learning
World Models

Abstract: An accurate model of the environment and the dynamic agents acting in it offers great potential for improving motion planning. We present MILE: a Model-based Imitation LEarning approach to jointly learn a model of the world and a policy for autonomous driving. Our method leverages 3D geometry as an inductive bias and learns a highly compact latent space directly from high-resolution videos of expert demonstrations. Our model is trained on an offline corpus of urban driving data, without any online interaction with the environment. MILE improves upon prior state-of-the-art by 31% in driving score on the CARLA simulator when deployed in a completely new town and new weather conditions. Our model can predict diverse and plausible states and actions, that can be interpretably decoded to bird's-eye view semantic segmentation. Further, we demonstrate that it can execute complex driving manoeuvres from plans entirely predicted in imagination. Our approach is the first camera-only method that models static scene, dynamic scene, and ego-behaviour in an urban driving environment. The code and model weights are available at https://github.com/wayveai/mile.

Model-Based Imitation Learning for Urban Driving
Anthony Hu!
Gianluca Corrado!
Nicolas Griffiths'
Zak Murez!
Corina Gurau
Hudson Yeo!
Alex Kendall'
Roberto Cipolla?
Jamie Shotton'
Wayve, UK.
University of Cambridge, UK. research@wayve ai
Abstract
An accurate model of the environment and the dynamic agents acting in it offers great potential for improving motion planning. We present MILE: Model-based Imitation LEarning approach to jointly learn model of the world and a policy for autonomous driving. Our method leverages 3D geometry as an inductive bias and learns a highly compact latent space directly from high-resolution videos of expert demonstrations. Our model is trained on an offline corpus urban driving data, without any online interaction with the environment. MILE improves upon prior state-of-the-art by 31% in driving score on the CARLA simulator when deployed in completely new town and new weather conditions. Our model can predict diverse and plausible states and actions, that can be interpretably decoded to execute
bird's s-eye view semantic segmentation. Further, complex driving manoeuvres from plans entirely predicted in imagination. Our approach is the first camera-only method that models static scene, dynamic scene, and ego-behaviour in an urban driving environment. The code and model weights are available at https:/ thub. cww/ wooo//aavea/mmm.emm
demonstrate that
Introduction
From an early age we start building internal representations of the world through observation and interaction [1]. Our ability to estimate scene geometry and dynamics is paramount to generating complex and adaptable movements. This accumulated knowledge of the world, part of what we often refer to as common sense, allows to navigate effectively in unfamiliar situations [2].
In this work, we present MILE, a Model-based Imitation LEarning approach to jointly learn a model of the world and a driving policy. We demonstrate the effectiveness of our approach in the autonomous driving domain, operating on complex visual inputs labelled only with expert action and semantic segmentation. Unlike prior work on world models [3, 4, 5]. our method does not assume access to ground truth reward, nor does it need any online interaction with the environment. Further, previous environments in OpenAl Gym [3], MuJoCo [4], and Atari [5] were characterised by simplified visual inputs as small as 64 x 64 images. contrast, MILE operates on high-resolution camera observations of urban driving scenes.
Driving inherently requires geometric understanding of the environment, and MILE exploits 3D geometry an important inductive bias by first lifting image features to 3D and pooling them into sred's-eye view (BeV) representation. The evolution of the world is modelled by latent dynamics model that infers compact latent states from observations and expert actions. The learned latent state the input to driving policy that outputs vehicle control, and can additionally be decoded BeV segmentation for visualisation and as supervision signal.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Our method also relaxes the assumption made in some recent work [6, 7] that neither the agent nor its actions influence the environment. This assumption rarely holds in urban driving, and therefore MILE is action-conditioned, allowing us to model how other agents respond to ego-actions. We show that our model can predict plausible and diverse futures from latent states and actions over long time horizons. It can even predict entire driving plans in imagination to successfully execute complex driving manoeuvres, such as negotiating a roundabout, or swerving to avoid a motorcyclist (see videos in the supplementary material).
We showease the performance of our model on the driving simulator CARLA [8], and demonstrate a new state-ofi of--o-o-thhhhee art- MILE achieves 31% improvement in driving score with respect to previous methods [9, 10] when tested in new town and new weather conditions. Finally, during inference, because we model time with a recurrent neural network, we can maintain a single state that summarises all the past observations and then efficiently update the state when new observation available. We demonstrate that this design decision has important benefits for deployment in terms of latency, with negligible impact on the driving performance.
To summarise the main contributions of this paper:
We introduce a novel model-based imitation learning architecture that scales to the visual complexity of autonomous driving in urban environments by leveraging 3D geometry as an inductive bias. Our method trained solely using an offline corpus of expert driving data, and does not require any interaction with an online environment or access to a reward, offering strong potential for world application.
Our camera-oniy model sets a new state-of-the of-the-art on the CARLA simulator, surpassing other approaches, including those requiring LiDAR inputs.
Our model predicts distribution of diverse and plausible futures states and actions. We demonstrate that it can execute complex driving manoeuvres from plans entirely predicted in imagination.
2
Related Work
Our work is at the intersection of imitation learning, 3D scene representation, and world modelling.
Imitation learning. Despite that the first end-to-end method for autonomous driving was envisioned more than 30 years ago [11]. early autonomous driving approaches were dominated by modular frameworks, where each module solves a specific task [12, 13, 14]. Recent years have seen the development of several end-to-end self-driving systems that show strong potential to improve driving performance by predicting driving commands from high-dimensional observations alone. Conditional imitation learning has proven to be one successful method to learn end-to-end driving policies that can be deployed in simulation [15] and real-world urban driving scenarios [16]. Nevertheless, difficulties of learning end-to-ent policies from high- sine--imensional visual observations and expert trajectories alone have been highlighted [17].
Several works have attempted to overcome such difficulties by moving past pure imitation learning. DAgger [18] proposes iterative dataset aggregation to collect data from trajectories that are likely to be experienced by the policy during deployment. NEAT [19] additionally supervises the model with BeV semantic segmentation. ChauffeurNet [20] exposes the learner to synthesised perturbations of the expert data in order to produce more robust driving policies. Learning from All Vehicles (LAV) [10] boosts sample efficiency by learning behaviours from not only the ego vehicle, but from all the vehicles in the scene. Roach [9] presents an agent trained with supervision from a reinforcement learning coach that was trained on on-policy and with access to privileged information.
3D scene representation. Successful planning for autonomous driving requires being able to understand and reason about the 3D scene, and this can be challenging from monocular cameras. One common solution is to condense the information from multiple cameras to single bird's s-ye representation of the scene, This can be achieved by lifting each image in 3D (by learning a depth distribution of features) and then splatting all frustums into common rasterised BeV grid 121, 22, 23]. An alternative approach is to rely on transformers to learn the direct mapping from image to s-ee view [24, 25, 26], without explicitly modelling depth.

---

World models. Model based methods have mostly been explored in a reinforcement learning setting and have been shown to be extremely successful [3. 27, 5]. These methods assume access to reward, and online interaction with the environment, although progress has been made on fully offline reinforcement learning [28, 29]. Model-based imitation learning has emerged as an alternative to reinforcement learning in robotic manipulation [30] and OpenAl Gym [31]. Even though these methods do not require access to a reward, they still require online interaction with the environment to achieve good performance.
Learning the latent dynamics of a world model from image observations was first introduced in video prediction [32, 33, 34]. Most similar to our approach, [4. 5] additionally modelled the reward function and optimised a policy inside their world model. Contrarily to prior work, our method does not assume access to a reward function, and directly learns policy from an offline dataset. Additionally, previous methods operate on simple visual inputs, mostly of size 64 64, contrast, MILE is able to learn the latent dynamics of complex urban driving scenes from high resolution 600 x 960 input observations, which important to ensure small details such as traffic lights can be perceived reliably.
Trajectory forecasting. The goal of trajectory forecasting is estimate the future trajectories of dynamic agents using past physical states (e.g. position, velocity), and scene context (e.g. as an offline HD map) [35. 36, 37, 38]. World models build latent representation of the environment that explains the observations from the sensory imputs of the ego-agent (e.g. camera images) conditioned on their actions. While trajectory forecasting methods only model the dynamic scene, world models jointly reason on static and dynamic scenes. The future trajectories of moving agents is implicitly encoded the learned latent representation of the world model, and could be explicitly decoded given we have access to future trajectory labels.
[35, 37, 38] forecast the future trajectory of moving agents, but did not control the ego-agent. They focused on the prediction problem and not on learning expert behaviour from demonstrations, [39] inferred future trajectories of the ego-agent from expert demonstrations, and conditioned on some specified goal to perform new tasks. [36] extended their work to jointly model the future trajectories of moving agents as well as of the ego-agent.
Our proposed model jointly models the motion of other dynamics agents, the behaviour of the ego-agent, well the static scene. Contrary prior work, we do not assume access to ground truth physical states (position, velocity) or to an offline HD map for scene context. Our approach is the first camera-only method that models static scene, dynamic scene, and ego-behaviour in an urban driving environment.
3 MILE: Model-based Imitation LEarning
In this section, we present MILE: our method that learns to jointly control autonomous vehicle and model the world and its dynamics. An overview of the architecture is presented in Figure 1 and the full description of the network can be found in Appendix C. We begin by defining the generative model (Section 3. 1), and then derive the inference model (Section 3.2). Section 3.3 and Section 3.4 describe the neural networks that parametrise the inference and generative models respectively. Finally, in Section 3.5 we show how our model can predict future states and actions to drive in imagination.
3.1 Probabilistic Generative Model
Let 01.7 be a sequence of T' video frames with associated expert actions a1.7 and ground truth BeV semantic segmentation labels Vı.r. We model their evolution by introducing latent variables S1r7 that govern the temporal dynamics. The initial distribution is parameterised as ~ ~NO0, I). and we additionally introduce a variable h1 ~ 8(0) that serves as a deterministic history. The transition consists of (i) deterministic update h fo(h,, st) that depends on the past history h, and past state 8,, followed by (ii) stochastic update 8t+1 ~ N(uu ar), ) og (he+1, a)) where we parameterised s, as normal distribution with diagonal covariance. We model these transitions with neural networks: fo is a gated recurrent cell, and (No σo) are multi-layer perceptrons. The full probabilistic model is given by Equation (1).
3

---

Y1
ā1
62
Y2
ag
63
Y:
7
98
1
TT
98
bg
hy
h1
fo
fo
f6
PAp. di
Pe..
p.g. -g
Sampl
Poje (%
Sumple
Po %
a1
X2
O
O2
Observing
Imagining
Figure I: Architecture MILE.
(i) The goal infer the latent dynamics (hI:T; 81:T) that generated the observations 01.7, the expert actions a and the bird's-eye view labels y1.7. The latent dynamics contains deterministic history h, and stochastic state St-
(ii) The inference model, with parameters ø, estimates the posterior distribution the stochastic state g(s) act)) N (Ns [A,,h,, a xt), as (h, ar-15 x)) with x, co.)). e is the observation encoder that lifts image features to 3D, pools them to bird's- "rdd's-eye view, and compresses to ID vector.
(iii) The generative model, with parameters 0, estimates the prior distribution of the stochastic state p(s; |h, St-1) ~ N ((u à oe (h, å â 1)), with h, = (h: St-1) the deterministic transition, and â лe (h:-1, 1.--1.50 S;-1) the predicted action. It additionally estimates the distributions of the observation p(o, h, s:) ~ (go (h, s,), I), the bird' s-eye view segmentation p(y, h,, s:) Categorical (he s;)), and the action p(a. [h,, st) Laplace( Ty [h, st),
(iv) In the diagram, we represented our model observing inputs for timesteps, and then imagining future latent states and actions for step.
h
6(0)
NN(O I)
h
8o(
Se41
(( o (a
(1)
O:
~ ~N(go (h,, st), I)
CCategorica((1 (h, Laplace(ma no (h,, st),
a,
with 8 the Dirac delta function, go the image decoder, to the BeV decoder, and т. the policy, which will be described in Section 3.4.
3.2
Variational Inference
Following the generative model described Equation (1), we can factorise the joint probability as:
T
p(ouT, Yu., auT, NuT,SiTT)
p(h, Se ht- 11, Sr-1. a a (p(( B., St)
(2)
1=1

---

with
p(h, s, (1, 11-1: ax-1) p(h, ( a. a.-1) p(o,, Yt, ag h4, 8t) p(o, |h, S, se)p(y st)p(a)) St h4 8t)
(3)
(4)
Given that h, is deterministic according to Equation (I), we have 1o(h(-1,8--1)-- Therefore, in order to maximise the marginal likelihood of the observed data p(O1:T, Vr:r, ). we need to infer the latent variables $1.7- We do this through deep variational inference by introducing variational distribution s defined and factorised follows:
8(h,
T = II Sc-1) act)
9n.s
(5)
with q(h, 1, St-1) ( the Delta dirac function defined above, and g 8(0).
We parameterise this variational distribution with neural network with weights Φ. By applying Jensen's 's inequality, we can obtain variational lower bound on the log evidence:
log p(01:r; VI.r, a1r)) 2 L(o1.r. VI.T, a1:;; 0, Φ)
E Bo
(, st) +
log ply (
log b4 St
image reconstruction
bird's- s-yee segmentation
action
E ti
DKL (9(s: acd)
p(s; h s(-1))
posterior and prior matching
Please refer to Appendix B for the full derivation. We model (S; Ogr, aci) a Gaussian distribution so that the Kullback-Leible (KL) divergence can be computed closed-form. Given that the image observations are modelled as Gaussian distributions with unit variance, the resulting loss is the mean-squ mean-squared error. Similarly, the action being modelled as Laplace distribution and the BeV labels as a categorical distribution, the resulting losses are, respectively, L1 and cross-entropy. The expectations over the variational distribution can be efficiently approximated with single sequence sample from qn.s, and backpropagating gradients with the reparametrisation trick [40].
3.3 Inference Network Φ
The inference network, parameterised by é, models q(s, O agr), which approximates the true posterior p(Se/ogt, act). ). It is constituted of two elements: the observation encoder eg. that embeds input images, route map and vehicle control sensor data low-dimensional vector, and the posterior network (Mip Jx), that estimates the probability distribution of the Gaussian posterior.
3.3.1
Observation Encoder
The state of our model should be compact and diwddmmeniooall in order to effectively learn dynamics. Therefore, we need to embed the high resolution input images to a low-dimensional vector. Naively encoding this image as shown in Section 5.2. Instead, we explicitly encode 3D geometric inductive biases in the model.
1D vector similarly image classification task results in poor performance
Lifting image features to 3D. Since autonomous driving is geometric problem where it is necessary to reason on the static scene and dynamic agents 3D, we first lift the image features 3D. More precisely,
encode the image inputs O, E 3x HxxH XW with image encoder to extract features U2 RCC H. W., Then similarly to Philion and Fidler [21]. we predict a depth probability distribution for each image feature along predefined grid of depth bins d, IR Dx Ml.xW. Using the depth probability distribution, the camera intrinsics and extrinsics M, we can lift the image features 3D: Lift (u,, d,, K-1, M)) RR Dx D3
Pooling to BeV. The 3D feature voxels are then sum-pooled to BeV space using a predefined grid with spatial extent Ha x Wo and spatial resolution byes. The resulting feature is b, C.C, W.
5

---

Mapping to 1D vector. In traditional computer vision tasks (e.g. semantic segmentation [41]. depth prediction [42]), the bottleneck feature usually spatial tensor, the order of 10 1 10
features. Such high dimensionality is prohibitive for world model that has to match the distribution of the priors (what thinks will happen given the executed action) to the posteriors (what actually happened by observing the image input). Therefore, using convolutional backbone, we compress the BeV feature b, to single vector X E RC" As shown in Section 5.2, we found it critical to compress in BeV space rather than directly in image space.
Route map and speed. We provide the agent with goal in the form of route map [9], which is a small grayscale image indicating to the agent where navigate at intersections. The route map is encoded using a convolutional module resulting in 1D) feature The current speed is encoded with fully connected layers as m, each timestep i, the observation embedding x; is the concatenation of the image feature, route map feature and speed feature: x, = [x rt, m2] E RC, with C = 512
3.3.2 Posterior Network
The posterior network (p)) estimates the parameters of the variational distribution g(s//ocs aci) ~ N (pa(hy ar-1. (o()))), ob a (x())) with h, (.(h--1..,1)) Note that h, was inferred using fo because we have assumed that h, is deterministic, meaning that q( (hy h1,hh,,,,- St-1) p(h, bt-1) = ō(h, fo St-1). The dimension of the Gaus- sian distribution is equal to 512.
3.4 Generative Network 0
The generative network, parameterised by 0, models the latent dynamies (hı:7, $1.1) well as the generative process of (OL.T. VI.7, a1:r). ). comprises a gated recurrent cell fo. a prior network (Mo: og), an image decoder gø, BeV decoder lo, and policy To.
The prior network estimates the parameters of the Gaussian distribution ((ss(h;,1,8:-) ~ N (Ho(h,, á.-1). (h,,, I) with h, = fo 8;-1) and â;- To Th 1;-)). Since the prior does not have access to the ground truth action a;-1s the latter estimated with the learned policy â To (h, St-1)
The Kullback-eibler divergence loss between the prior and posterior distributions can be interpreted as follows. Given the past state () the objective is to predict the distribution of the next state St. As we model an active agent, this transition decomposed into (i) action prediction and (ii) next state prediction. This transition estimation is compared to the posterior distribution that has access to the ground truth action a,- and the image observation o,, The prior distribution tries to match the posterior distribution. This divergence matching framework ensures the model predicts actions and future states that explain the observed data. The divergence of the posterior from the prior measures how many nats of information were missing from the prior when observing the posterior. At training convergence, the prior distribution should be able to model all action-state transitions from the expert dataset.
The image and BeV decoders have an architecture similar StyleGAN [43]. The prediction starts as a learned constant tensor, and is progressively upsampled to the final resolution. At each resolution, the latent state is injected in the network with adaptive instance normalisation. This allows the latent states to modulate the predictions at different resolutions. The policy is a multi-layer perceptron. Please refer to Appendix C for full description of the neural networks.
3.5 Imagining Future States and Actions
Our model can imagine future latent states by using the learned policy to infer actions år+i To ST+)), predicting the next deterministic state hy = (p(hT+,,S7=:) and sampling from the prior distribution 87+141 "N(u (h 4r+1), oo (h T)), for i 2 0. This process can be iteratively applied to generate sequences of longer futures in latent space, and the predicted futures can be visualised through the decoders.

---

Table 1: Driving performance on a new town and new weather conditions in CARLA. Metrics are averaged across three runs. We include reward signals from past work where available.
Driving Score
Route
Infraction
Reward
Norm. Reward
CILRS [17]
7.8 0.3
10.3 + 0.0
76.2 = 0.5
LBC [47]
12.3 + 2.0
31.9 2.2
66.0 = 1.7
TransFuser [48]
31.0 3.6
47.5 5.3
76,8 = 3,9
Roach [9]
41.6 1.8
96.4 2.1
43.3 + 2.8
4236 + 468
0.34 + 0.05
LAV [10]
46.5 3.0
69.8 2.3
73.4 + 2.2
MILE
61.1 +: 3.2
97.4 0.8
63.0 + 3.0
7621 + 460
0.67 0.02
Expert
88.4 + 0.9
97.6 + 1.2
90.5 + 1.2
8694 88
0.70 + 0.01
Experimental Setting
Dataset.
The training data was collected in the CARLA simulator with an expert reinforcement learning (RL) agent [9] that was trained using privileged information input (BeV semantic segmentations and vehicle measurements). This RL agent generates more diverse runs and has greater driving performance than CARLA's in-built autopilot [9].
We collect data at 25Hz in four different training towns (Town01, Town03, Town04, Town06) and four weather conditions (ClearNoon, WetNoon, HardRainNoon, ClearSunset) for a total of 2.9M frames, or 32 hours of driving data. At each timestep. we save tuple (o, route,, speed, a, yt). with Or IR 3x000000x000 960 the forward camera RGB image, route; E TR x64x64 the route map (visualized as an inset on the top right of the RGB images in Figure 2), speed, E R the current velocity of the vehicle, E I the action executed by the expert (acceleration and steering), and E R x19 192 the BeV semantic segmentation. There are C = semantic classes: background, road, lane marking, vehicles, pedestrians, and traffic light states (red, yellow, green). In urban driving environments, the dynamics of the scene do not contain high frequency components, which allows us to subsample frames at 5Hz in our sequence model.
Training. Our model was trained for 50, 000 iterations on a batch size 64 on 8 V100 GPUs, with training sequence length T' 12. We used the AdamW optimiser [44] with learning rate 10-4 and weight decay 0.01.
Metrics. We report metrics from the CARLA challenge [45] to measure on-road performance: route completion, infraction penalty, and driving score. These metrics are however very coarse, as they only give sense of how well the agent performs with hard penalties (such as hitting virtual pedestrians). Core driving competencies such as lane keeping and driving an appropriate speed are obscured. Therefore we also report the cumulative reward of the agent. At each timestep the reward [46] penalises the agent for deviating from the lane center, for driving too slowly/fast, or for causing infractions. It measures how well the agent drives at the timestep level. order to account for the length of the simulation (due to various stochastic events, can be longer shorter), we also report the normalised cumulative reward. More details on the experimental setting is given in Appendix D.
5 Results
5.1 Driving Performance
We evaluate model inside the CARLA simulator town and weather conditions never during training. We picked Town05 as it is the most complex testing town, and use the 10 routes of Town05 specified in the CARLA challenge [45], in four different weather conditions. Table I shows the comparison against prior sfate-of-the--e methods: CILRS [17], LBC [47], TransFuser [48], Roach [9], and LAV [10]. We evaluate these methods using their publicly available pre-trained weights.
MILE outperforms previous works all metrics, with a 31 % relative improvement in driving score with respect LAV. Even though some methods have access to additional sensor information such as LiDAR (TransFuser [48], LAV [10]), approach demonstates superior performance while only
7

---

Table 2: Ablation studies. We report driving performance on new town and new weather conditions in CARLA. Results are averaged across three runs.
Driving Score
Route
Infraction
Reward
Norm, Reward
Single frame, no 3D Single frame
51.8 3.0
78.3 = 3.0
68.3 2.8
1878 296
0.20 0.04 0.60 + 0.01
59.6 - 3,6
94.5 = 0.6
64.7 - 3.3
6630 = 168
No
63.0 1.5
91.5 + 5.5
69.1 2.8
4564 1791
0.40 0.15
No prior/post. matching
63.3 2.2
91.5 = 5.0
68.7 - 1.8
6084 = 1429
0.55 0.07
No segmentation
55.0 + 3.3
92.5 = 2.4
60.9 + 3.9
7183 = 107
0.64 + 0.02
MILE
61.1 3.2
97.4 = 0.8
63.0 = 3.0
7621 460
0.67 + 0.02
Expert
88.4 +: 0.9
97.6 + 1.2
90.5 + 1.2
8694 + 88
0.70 + 0.01
using RGB images from the front camera. Moreover, we observe that our method almost doubles the cumulative reward of Roach (which was trained on the same dataset) and approaches the performance of the privileged expert.
5.2 Ablation Studies
We next examine the effect of various design decisions in our approach.
3D geometry. We compare our model to the following baselines. Single frame that predicts the action and BeV segmentation from a single image observation. Single frame, no 3D which is the same model but without the 3D lifting step. And finally, No 3D which is MILE without 3D lifting. As shown in Table 2, in both cases, there is significant drop in performance when not modelling 3D geometry. For the single frame model, the cumulative reward drops from 6084 1878. For MILE, the reward goes from 7621 to 4564. These results highlights the importance of the 3D geometry inductive bias.
Probabilistic modelling. At any given time while driving, there exist multiple possible valid behaviours. For example, the driver can slightly adjust its speed, decide to change lane, or decide what is a safe distance to follow behind vehicle. A deterministic driving policy cannot model possible, it will often learn the
these subtleties. In ambiguous situations where multiple choices mean behaviour, which is valid in certain situations (e.g. the mean safety distance and mean cruising speed reasonable choices), but unsafe in others (e.g. in lane changing: the expert early, or late; the mean behaviour is to drive on the lane marking). We compare MILE with a No prior/post. matching baseline that does not have Kullback-Leibbe divergence loss between the prior and posterior distributions, and observe this results in a drop in cumulative reward from 7621 to 6084.
change lane
5.3 Fully Recurrent Inference in Closed-Loop Driving
We compare the closed-loop performance of our model with two different strategies:
(i) Reset state: for every new observation, we re-initialise the latent state and recompute the new state [r], with T matching the training sequence length.
(ii) Fully recurrent: the latent state is initialised at the beginning of the evaluation, and is recursively updated with new observations. It never reset, and instead, the model must have learned a representation that generalises to integrating information for orders of magnitude more steps than the T used during training.
Table 3 shows that our model can be deployed with recurrent updates. matching the performance of the Reset state approach, while being much more computationally efficient (7x faster from 6.2Hz with T = 12 of fixed context to 43.0Hz with fully recurrent approach). A hypothesis that could explain why the Fully recurrent deployment method works well is because the world model has learned to always discard all past information and rely solely on the present input. To test this hypothesis, we add Gaussian noise to the past latent state during deployment. the recurrent network is simply discarding all past information, its performance should not be affected. However in Table 3, we see that the cumulative reward significantly decreases, showing our model does not simply discard all past context, but actively makes use of it.
8

---

Table 3: Comparison of two deployment methods. (i) Reset state: for each new observation fresh state is computed from zero-inin initialiseed latent state using the last T observations, and (ii) Fully recurrent: the latent state is recurrently updated with new observations. We report driving performance on an unseen town and unseen weather conditions in CARLA. Frequency is in Hertz.
Driving Score
Route
Infraction
Reward
Norm. Reward
Freq.
Reset state
61.1 3.2
97.4 0.8
63.0
7621
0.67 0.02
6.2
Fully recurrent
62.1 1 0.5
93.5 1 4.8
66.6 1 3.4
7532 1 1122
0.67 0.04
43.0
Recurrent+noise
48.8
81.1 7.0 61.5
3603
0.35 0.07
43.0
5.4
Long Horizon, Diverse Future Predictions
Our model can imagine diverse futures in the latent space, which can be decoded to BeV semantic segmentation for interpretability. Figure 2 shows examples of multi-modal futures predicted by MILE.
RGB iuput (ime 1)
GT (tine 13 Pred fime 1)
1+1..s
t:22.0s
1--1.0s
8.0%
Figure Qualitative example multi-modal predictions, for 8 seconds the future. BeV segmenta- tion legend: black = ego- vehicle white background, gray road, dark gray=lane marking, blue vehicles, cyan = pedestrians, green/yellow/red traffic lights. Ground truth labels (GT) outside the field-of-view of the front camera masked out. In this example, futures predicted by the model: 1) (top row) driving through the green light, 2) (bottom row) stopping because the model imagines the traffic light turning red. Note the light transition from green, to yellow, to red, and also at the last frame t 8.0s how the traffic light in the left lane turns green.
visualise two distinct
Insights from the World Model
6.1 Latent State Dimension
In our model, we have set the latent state to be a low-dimensional 1D vector of size 512. In dense image reconstruction however, the bottleneck feature is often a 3D spatial tensor of dimension (channel, height, width). We test whether it is possible to have a 3D tensor state instead of ID vector. We change latent state to have dimension 256 12 x 12 (40k distributions), 128 24 x 24 (80k distributions), and 64 x 48 48 (160k distributions, which is the typical bottleneck size in dense image prediction). Since the latent state is now a spatial tensor, we adapt the recurrent network to be convolutional by switching the fully-connected operations with convolutions. evaluate the model in the reset state and fully recurrent setting and report the results in Figure 3.
latent probabilistic
Bullyy
7.0221 7,6022
7,465 0:008
6,007
5,000
4.506
9.794
2,500
Latent state dimension
Figure 3: Analysis on the latent state dimension. We report closed-loop driving performance in a new town and new weather in CARLA.

---

In the reset state setting, performance decreases as the dimensionality of the latent state increases. Surprisingly, even though the latent space is larger and has more capacity, driving performance is negatively impacted. This seems to indicate that optimising the prior and posterior distributions in the latent space is difficult, and especially more so as dimensionality increases. The prior, which is a multivariate Gaussian distribution needs to match the posterior, another multivariate Gaussian distribution. What makes this optimisation tricky is that the two distributions are non-stationary and change over time during the course of training. The posterior needs to extract the relevant information from the high-resolution images and incorporate in the latent state in order to reconstruct BeV segmentation and regress the expert action. The prior has to predict the transition that matches the distribution of the posterior.
Even more intriguing is when we look at the results in the fully recurrent deployment setting. When deployed fully recurrent manner in the simulator, without resetting the latent state, the model needs to discard information that is no longer relevant and continuously update its internal state with new knowledge coming from image observations. 512, there is almost no different in driving performance between the two deployment modes. The picture is dramatically different when using a higher dimensional spatial latent state. For all the tested dimensions, there is a large gap between the two deployment settings. This result seems to indicate that the world model operating on high- nigh-dimensional spatial states has not optimally learned this behaviour, contrarily to the one operating on low-dimensional vector states.
our original latent state dimension of
6.2 Driving in Imagination
Humans are believed to build an internal model of the world in order to navigate in it [49, 50, 51]. Since the stream of information they perceive is often incomplete and noisy, their brains fill missing information through imagination. This explains why it is possible for them to continue driving when blinded by sunlight for example. Even if no visual observations are available for brief moment, they can still reliably predict their next states and actions to exhibit a safe driving behaviour. We demonstrate that similarly, MILE can execute accurate driving plans entirely predicted from imagination, without having access to image observations. We qualitatively show that it can perform complex driving maneuvers such as navigating roundabout, marking pause stop sign, or swerving to avoid a motorcyclist, using an imagined plan from the model (see supplementary material).
Quantitatively, we measure how accurate the predicted plans are by operating in the fully recurrent setting. We alternate between the observing mode where the model can see image observations, and the imagining mode where the model has to imagine the next states and actions, similarly to driver that temporarily loses sight due to sun glare. In Appendix A. we show that our model can retain the same driving performance with up to 30% of the drive in imagining mode. This demonstrates that the model can imagine driving plans that are accurate enough for closed loop driving. Further, it shows that the latent state of the world model can seamlessly switch between the observing and imagining modes. The evolution of the latent state is predicted from imagination when observations not available, and updated with image observations when they become accessible.
7 Conclusion
We presented MILE: a Model-based Imitation LEarning approach for urban driving, that jointly learns a driving policy and world model from offline expert demonstrations alone. Our approach exploits geometric inductive biases, operates on high-dimensional visual inputs, and sets a new state-of-the-art on the CARLA simulator. MILE can predict diverse and plausible future states and actions, allowing the model to drive from plan entirely predicted from imagination.
An open problem is how to infer the driving reward function from expert data, this would enable explicit planning in the world model. Another exciting avenue is self-supervision in order to relax the dependency on the bird's s-eye view segmentation labels. Self-supervision could fully unlock the potential of world models for real-world driving and other robotics tasks.
Acknowledgements. We would like to thank Vijay Badrinarayanan, Przemyslaw Mazur, and Oleg Sinavski for insightful research discussions. We are also grateful to Lorenzo Bertoni, Lloyd Russell, Juba Nait Saada, Thomas Uriot, and the anonymous reviewers for their helpful feedback and comments on the paper.
10

---

References
[1] H. B. Barlow. Unsupervised learning. Neural computation, 1(3):295-311, 1989.
[2] D. M. Wolpert and M. Kawato. Multiple paired forward and inverse models for motor control. Neural networks, 11(7-8):1317-1329) 1998.
[3] D. Ha and J. Schmidhuber. Recurrent world models facilitate policy evolution. Advances in Neural Information Processing Systems (NeurIPS), 2018.
[4] D. Hafner, T. Lillicrap, I. Fischer, R. Villegas, D, Ha, H. Lee, and J. Davidson. Learning Latent Dynamics for Planning from Pixels. In Proceedings of the International Conference on Machine Learning (ICML), 2019.
[5] D. Hafner, Lillicrap, M. Norouzi, and J. Ba. Mastering atari with discrete world models. Proceedings of the International Conference on Learning Representations (ICLR), 2021.
[6] D. Chen, V. Koltun, and P. Krähenbühl. Learning to drive from world rails. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 15590- 15599, 2021.
[7] V. Sobal, A. Canziani, N. Carion, K. Cho, and LeCun. Separating the world and ego models for self-driving. arXiv preprint arXiv:220..07/84 2022.
[8] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An Open Urban Driving Simulator. In Proceedings of the Conference on Robot Learning (CoRL.), pages 1-16, 2017.
[9] Z. Zhang, Liniger, D. Dai, Yu, and L. Van Gool. End-to-end urban driving by imitating a reinforcement learning coach. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 15222- 2021.
[10] D. Chen and P. Krähenbühl. Learning from all vehicles. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[11] D. A. Pomerleau. ALVINN: An autonomous land vehicle in neural network. Advances in Neural Information Processing Systems (NeurIPS), 1, 1988.
[12] A. Bacha, C. Bauman, R. Faruque, M. Fleming, C. Terwelp, C. Reinholtz, D. Hong, A. Wicks, Alberi, D. Anderson, et al. Odin: Team VictorTango's Entry in the DARPA Urban Challenge. Journal of Field Robotics, 25(8):467-4922 2008
[13] D. Dolgov, S. Thrun, M. Montemerlo, and Diebel. Practical search techniques in path planning for autonomous driving, Ann Arbor, 1001 105)) 2008.
[14] J. Leonard, J. How, S. Teller, M. Berger, S. Campbell, G. Fiore, L. Fletcher, E. Frazzoli. A. Huang, Karaman, et al. A perception-driven autonomous urban vehicle. Journal of Field Robotics, 25(10):727-774, 2008.
[15] F. Codevilla, M. Müller, A. López, V. Koltun, and A. Dosovitskiy. End-to-end driving via conditional imitation learning. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2018.
[16] Hawke, R. Shen, C. Gurau, S. Sharma, Reda, N. Nikolov, P. Mazur, S. Micklethwaite, N. Griffiths, A. Shah, et al. Urban Driving with Conditional Imitation Learning. In Proceedings of the International Conference on Robotics and Automation (ICRA), pages 251-257, 2020.
[17) F. Codevilla, F. Santana, A. M. López, and A. Gaidon. Exploring the Limitations of Behavior Cloning for Autonomous Driving. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR), pages 2329-9338, 2019.
[18] S. Ross, G. Gordon, and D. Bagnell. A Reduction of Imitation Learning and Structured Prediction to Re-Regret Online Learning. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS), pages 627-635, 2011.

---

[19] K. Chitta, A. Prakash, and A. Geiger. NEAT: Neural Attention Fields for End-to-End Au- tonomous Driving. In Proceedings of IEEE International Conference on Computer Vision (ICCV), 2021.
[20] M. Bansal, A. Krizhevsky, and A. Ogale. ChauffeurNet: Learning to drive by imitating the best and synthesizing the worst. In Proceedings of Roboties: Science and Systems (RSS), 2019.
[21] Philion and S. Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
[22] A. Saha, O. Mendez, C. Russell, and R. Bowden. Enabling Spatio-temporal aggregation in Birds-Fye-Vien Vehicle Estimation. In Proceedings of the International Conference on Robotics and Automation (ICRA), 2021.
[23] A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V. Badrinarayanan, R. Cipolla, and A. Kendall. FIERY: Future Instance Prediction in B-rd View From Surround Monocular Cameras. In Proceedings of the IEEE International Conference Computer Vision (ICCV), pages 15273-15282, 2021.
[24] L. Peng. Z. Chen, Z. Fu, P. Liang, and E. Cheng. Bevsegformer: Bird's eye view semantic segmentation from arbitrary camera rigs. arXiv preprint arXiv:2203.044 04050, 2022.
[25] N. Gosala and A. Valada. Bird''-eye-view panoptic segmentation using monocular frontal view images. IEEE Robotics and Automation Letters, 2022.
[26] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Q Yu, and J. Dai. BEVFormer: Learning bird's s-eye-view representation from multi-camera images via spatiotemporal transformers. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
[27] J. Schrittwieser, I. Antonoglou, T. Hubert, K. Simonyan, L. Sifre, S. Schmitt, A. Guez, E. Lock- hart, D. Hassabis, T. Graepel, T. Lillicrap, and D. Silver. Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. Nature, 2020.
[28] W. Zhou, S. Bajracharya, and D. Held. PLAS: Latent Action Space for Offline Reinforcement Learning. In Proceedings of the Conference on Robot Learning (CoRL), 2020.
[29] T. Yu, G. Thomas, L. Yu, S. Ermon, J. Zou, S. Levine, C. Finn, and T. Ma. MOPO: Model-based Offline Policy Optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[30] P. Englert, A. Paraschos, M. P. Deisenroth, and J. Peters. Probabilistic model-based imitation learning. Adaptive Behavior, 21(5):388 403, 2013.
[31] R. Kidambi, J. Chang, and W. Sun. Mobile: Model-based imitation learning from observation alone. Advances in Neural Information Processing Systems, 34, 2021.
[32] M. Babaeizadeh, C. Finn, D. Erhan, R. H. Camphell, and S. Levine. Stochastic variational video prediction. In Proceedings of the International Conference on Learning Representations (ICLR), 2018
[33] E. Denton and R. Fergus. Stochastic Video Generation with a Learned Prior. In Proceedings of the International Conference on Machine Learning (ICML). 2018.
[34] J.-Y. Franceschi, E. Delasalles, M. Chen, S. Lamprier, and P. Gallinari. Stochastic latent residual video prediction. In Proceedings of the International Conference on Machine Learning (ICML), 2020.
[35] N. Lee, W. Choi, P. Vernaza, C. B. Choy, P. H. S. Torr, and M. K. Chandraker. DESIRE: distant future prediction in dynamic scenes with interacting agents. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2017.
[36] N. Rhinehart, R. McAllister, K. M. Kitani, and S. Levine. PRECOG: prediction conditioned on goals in visual multi-agent settings. Proceedings of the IEFE International Conference on Computer Vision (ICCV), 2019.
12

---

[37] H. Zhao, J. Gao, T. Lan, C. Sun, B. Sapp. B. Varadarajan, Y. Shen, Y. Shen, Y. Chai, C. Schmid, C. Li, and D. Anguelov. TNT: Target-driven trajectory prediction. In Proceedings of the Conference Robot Learning (CoRI.), 2020.
[38] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone. Trajectron++: Dynamically-feasiblee trajectory forecasting with heterogeneous data. In Proceedings of the European Conference Computer Vision (ECCV), 2020.
[39] N. Rhinehart, R. McAllister, and S. Levine. Deep imitative models for flexible inference, plan- ning, and control. In Proceedings of the International Conference on Learning Representations (ICL.), 2020.
[40] D. P. Kingma and M. Welling. Auto-encoding variational bayes. Proceedings of the International Conference on Learning Representations (ICLR), 2014.
[41] L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam. Rethinking Atrous Convolution for Semantic Image Segmentation. arXiv preprint 1rXxv:1706.05588 2017.
[42] C. Godard, O. Mac Aodha, M. Firman, and G. J. Brostow. Digging into self-supervised
monocular depth prediction. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), October 2019.
[43] T. Karras, S. Laine, and T. Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[44] I. Loshchilov and F, Hutter. Decoupled Weight Decay Regularization. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[45] CARLA Team. CARLA Autonomous Driving Leaderboard. https: hatee///eeeeeebbarre carla. org/ see//et_staareee 2019.
[46] M. Toromanoff, E. Wirbel, and F. Moutarde. End-to-end model-free reinforcement learning for urban driving using implicit affordances. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[47] D. Chen, B. Zhou, V. Koltun, and P. Krähenbühl. Learning by cheating. In Conference Robot Learning, pages 66-75, 2020.
[48] A. Prakash, K. Chitta, and A. Geiger. Multi-modal fusion transformer for end-to-end au- tonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 7077-7087, 2021.
[49] T. Madl, K. Chen, D. Montaldi, and R. Trappl. Computational cognitive models of spatial memory in navigation space: A review. Neural Networks, 65:18-44 (1188-43,22 2015.
[50] R. A. Epstein, E. Z. Patai, J. B. Julian, and H. J. Spiers. The cognitive map in humans: spatial navigation and beyond. Nature Neuroscience, 20(11):1504-151: 2017.
[51] J. L.. Park, P. A. Dudchenko, and D. I. Donaldson. Navigation in real-world environments: New opportunities afforded by advances in mobile brain imaging. Frontiers in Human Neuroscience, 12, 2018.
[52] CARLA Team. CARLA Maps.
map/, 2022.
[53] M. Henaff, A. Canziani, and Y. LeCun. Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Traffic. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
[54] B. Cheng, M. D. Collins, Y. Zhu, T. Liu, S. Huang, H. Adam, and L. Chen. Panoptic-deeplab: A simple, strong, and fast baseline for bottom-up panoptic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[55] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2016.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section and Section 6.
(b) Did you describe the limitations of your work? [Yes] See Section 7.
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] See Section 3.
(b) Did you include complete proofs of all theoretical results? [Yes] See Appendix B.
3. If you experiments..
(a) Did you include the code. data, and instructions needed reproduce the main ex- perimental results (either the supplemental material or as a URL)? [Yes] See https://githee
coom/wayveai/min
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix C and Appendix D.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] See Section
(d) Did you include the total amount compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any assets either in the supplemental material or as URL? [Yes] See https: //t/tttttbb.t/////////// com/ wwwwwwavaa/////e meeiaiimiil... /mile.
(d) Did you discuss whether and how consent was obtained from people whose data you're 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "yJFla3eoT8D", "reviewer": "Reviewer_FfUk", "summary": "This work develops an model-based imitation learning pipeline for autonomous driving.\nTheir observation network encodes multi-view camera images and route information into a latent embedding which is then used in a variational generative model to predict future BEV semantic segmentation and expert actions.\nTo my knowledge, this is the first work to show the effectiveness of world models in autonomous driving simulators.", "questions": "* Does this work have a submission to https://leaderboard.carla.org/leaderboard/? This is important to my final rating.\n* Related to the trajectory forecasting point - is there any particular reason why rasterized BEV labels were used for future prediction rather than detections (i.e. boxes)?\n* BEV labels would require auto-labeling or human annotations - how much driving data is needed for a \"good enough\" world model?", "limitations": "yes", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* Clear writing, technical portion + model architecture easy to understand, since this work seems like FIERY but instead predicts driving actions.\n* Their model, which uses cameras only, shows significant improvements over even lidar-based methods.\n* Supplementary video shows fairly good short-term predictions.\n* I agree with and am glad to see the discussion of driving score - the score is definitely biased towards certain types of routes and adding the reward/normalized reward as another source of quantitative evaluation is useful.\n\n### Weaknesses\n* The architecture figure looks more complicated than the architecture actually is. I would suggest adding a simpler toy figure in place of Figure 1, moving the current figure to the technical section.\n* I would like to see a bit of discussion or related works on trajectory forecasting. It would be beneficial to discuss some of the difference between world models and trajectory forecasting since there are many recurring themes between these two areas.\n* Adding dataset size to Table 1 would be helpful for comparison.\n* It seems like a few important experiments refer solely to the supplementary - I'd suggest some spacing edits to try to fit these into the main paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "4NmaW94cdwi", "reviewer": "Reviewer_W3CA", "summary": "This paper proposes a predictive model of visual data for the task of imitation learning in CARLA. The model is trained to forecast RGB images, semantic segmentation images, and expert actions in the variational inference framework. The proposed method incorporates several new components, including a 3D geometry inductive bias that leverages camera intrinsics and extrinsics, as well as a variation on training called “observation dropout”. Experiments illustrate the effectiveness of the proposed approach on several towns in CARLA, relative to other recent approaches. The proposed approach is overall the most performant. Ablations also illustrate the importance of the 3D inductive bias in the single frame prediction setting, the importance of multi-frame prediction, the importance of the semantic segmentation modeling, and the importance of stochasticity.", "questions": "Aside from addressing the main points in the weaknesses (the two regarding the 'main claims' are the most important)\n\n- The “deterministic temporal” baseline needs more explanation. Is it using the same architecture? What is the precise training objective?", "limitations": "No, there is no clear \"limitations\" section. The paper requires some discussion of limitations relative to (a) scalability w.r.t. availability of training data (b) feasibility w.r.t. existing model-free imitation learning methods, among others.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n- The presents the first evaluation of a visual world model on the task of offline imitation learning for urban driving in simulation\n- The paper demonstrates compelling performance results of the main method\n- The paper presents an interesting approach for “lifting” the representation into 3D using the camera intrinsics and extrinsics was incorporated into the model. \n\n# Weaknesses\nOverall, there are many proposed components and claims of the method that lack quantitative evidence.\n\n- In the main claims of the introduction, L35 claims that the method “scales … by leveraging 3D geometry as an inductive bias”. However, the 3D inductive bias was not ablated from the main method (only from the single-frame prediction). Therefore, the claim has no evidence. Quantitative proof of this claim is needed in order me to agree with the inclusion of this claim, and thus I must weaken my evaluation of the paper's soundness and contributions.\n- In the main claims of the introduction, L59 claims that “Our model predicts a distribution of diverse and plausible futures that can be decoded”. However, there is no quantitative evaluation on the diversity or the plausibility of the forecasted futures. This claim also appears in the conclusion. Quantitative proof of this claim is needed in order me to agree with the inclusion of this claim, and thus I must weaken my evaluation of the paper's soundness and contributions.\n- A missing point of comparison is an existing model-based imitation learning approach that has been applied to CARLA [A]. This approach models only the future positional trajectory of the ego-vehicle. Although it would require a bit of experimentation, it would be informative to compare against it, or something like it, that purely models the future motion of the ego-vehicle, as opposed to the entire image. I suspect the current approach might perform quite well just by modeling the future positions, as opposed to the future semantic segmentation and RGB observations. I would at least like to see the proposed approach evaluated against itself when just trained to forecast the future positions of the vehicle.\n- L100 The high resolution input is quite interesting, but it was not ablated (relative to the standard Dreamer model, which uses 64x64x3 images). The evaluation would be improved if a comparison to simply resizing the larger dimension of the input image to be 64 and running that as a comparison. This would enable us to understand the effect of using a higher resolution image. It is possible there is no effect, or that there is a significant effect. The results are important and relevant in either case.\n- “Observation dropout” in S3.5 was a new method that was proposed but not ablated. The proposed approach should ablate this proposed component in order to highlight its efficacy.\n- The ablation of the semantic segmentation targets for the main method is missing. The conclusion mentions “another exciting avenue is self-supervision in order to relax the dependency on the bird’s-eye view segmentation labels”. However, this should have been already investigated by simply removing this supervision from the main model and investigating its effect.\n\n## Post-rebuttal update\nThe weaknesses I described were more-or-less addressed in the revised version and the authors' responses. \n\n[A] Deep Imitative Models for Flexible Inference, Planning, and Control, ICLR 2020. https://openreview.net/forum?id=Skl4mRNYDr\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1-cSaYl7JhN", "reviewer": "Reviewer_s7YQ", "summary": "The paper presents an offline imitation learning approach that simultaneously predicts the evolution of the environment and imitates the expert driver. The proposed approach uses camera-based observations and a semantic segmentation map to build a world representation. The camera images are lifted to 3D and combined into a BeV representation. The environment evolution is performed in a compact latent space. The proposed method achieves better performance and generalization than prior work on a CARLA benchmark.", "questions": "In addition to addressing the above listed weaknesses, I have the following questions for the authors. Assuming these questions and concerns are addressed, I am currently inclined to raise the score to an accept.\n\n1. How would the BeV semantic segmentation labels be achieved without LiDAR in a non-simulated setting?\n2. What is the explicit definition of the lift operation in line 157?\n3. In lines 209 and 210, it is stated that the RL expert was trained using privileged information (e.g., BeV semantic segmentation). However, my understanding was that the proposed model is also trained on BeV semantic segmentation. What is the privileged information available to the RL expert, but not the proposed model?\n4. Are any of the considered baselines RL agents outside the expert? One of the motivations behind the proposed method is the ability to avoid constructing a reward. However, the cumulative reward is a key metric of evaluation for the proposed method, thus a comparison to an RL agent trained with this reward while having access to the same input information as MILE would be worthwhile.\n5. Is higher better for the infraction penalty?\n6. What makes the recurrent approach faster in Sec. 5.4? Were the models for the two evaluation methods trained differently?", "limitations": "The authors have listed some limitations as part of future work. Some potential additional limitations to include are: imitation learning methods can be subject to adversarial attacks and the model does not have safety guarantees.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well written and the ideas are easy to follow.\n* The problem is well-motivated and the literature review does a good job at contextualizing the paper in prior work.\n* The idea behind end-to-end feature learning, while incorporating geometric bias is intuitive.\n* Overall, the empirical evaluation of the method is extensive and convincing. The analysis and discussion is thoughtfully constructed, and the different components of the architecture were thoroughly ablated.\n* The figures are informative and effectively illustrate the proposed approach and results.\n\nWeaknesses:\n* The sentence in the abstract is a bit strong 'So far, such world models have been shown to be highly effective at solving games, but only in simple visual environments with little interaction among agents.'. There has been lots of work in the trajectory prediction setting that builds world models in dynamic environments (e.g., [1,2]).\n* The related works section is missing a discussion of trajectory prediction methods which are closely related to imitation learning as presented in the paper.\n* One of the motivations for the imitation learning approach over RL is the difficulty in constructing rewards. However, in the evaluation, the most effective metric is determined to be cumulative reward. The considered baselines do not seem to include an RL approach with access to the same input information as MILE.\n\n[1] Zhao, Hang, et al. \"TNT: Target-driven Trajectory Prediction.\" Conference on Robot Learning. PMLR, 2021.\n[2] Salzmann, Tim, et al. \"Trajectron++: Dynamically-feasible trajectory forecasting with heterogeneous data.\" European Conference on Computer Vision. Springer, Cham, 2020.\n\nThe paper needs to be proofread for typos. The following is a non-exhaustive list of the typos I found:\n\n1. The BeV acronym is defined in line 30, but not consistently used after that.\n2. Line 106: 'a the full description' should read 'the full description'.\n3. Line 168: 'indicating the agent where to navigate' should read 'indicating to the agent where to navigate'.\n4. Line 234: 'as it obtained' should read 'as it is obtained'.\n5. The references should be proofread (e.g., to ensure the year is not entered twice in a citation, the conference venue is listed instead of ArXiv when available, etc.).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept