Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion
Authors: Lunjun Zhang
Yuwen Xiong
Ze Yang
Sergio Casas
Rui Hu
Raquel Urtasun
Venue: ICLR 2024 poster (2024)
Keywords: discrete diffusion; world model; autonomous driving
Primary area: applications to robotics, autonomy, planning

Abstract: Learning world models can teach an agent how the world works in an unsupervised manner. Even though it can be viewed as a special case of sequence modeling, progress for scaling world models on robotic applications such as autonomous driving has been somewhat less rapid than scaling language models with Generative Pre-trained Transformers (GPT). We identify two reasons as major bottlenecks: dealing with complex and unstructured observation space, and having a scalable generative model. Consequently, we propose Copilot4D, a novel world modeling approach that first tokenizes sensor observations with VQVAE, then predicts the future via discrete diffusion. To efficiently decode and denoise tokens in parallel, we recast Masked Generative Image Transformer as discrete diffusion and enhance it with a few simple changes, resulting in notable improvement. When applied to learning world models on point cloud observations, Copilot4D reduces prior SOTA Chamfer distance by more than 65% for 1s prediction, and more than 50% for 3s prediction, across NuScenes, KITTI Odometry, and Argoverse2 datasets. Our results demonstrate that discrete diffusion on tokenized agent experience can unlock the power of GPT-like unsupervised learning for robotics.

Published as a conference paper at ICLR 2024
COPILOT4D: LEARNING UNSUPERVISED WORLD MODELS FOR AUTONOMOUS DRIVING VIA DISCRETE DIFFUSION
Lunjun Zhang
Yuwen Xiong
Ze Yang
Sergio Casas
Rui Hu
Raquel Urtasun
Waabi
University of Toronto
(1zhang, yxiong, zyang, sergio, rhu, urtasun]ewaabil
ABSTRACT
Learning world models can teach an agent how the world works in an unsuper- vised manner. Even though it can viewed as a special case of sequence model- ing, progress for scaling world models on robotic applications such as autonomous driving has been somewhat less rapid than scaling language models with Gener- ative Pre-trained Transformers (GPT). We identify two reasons major bottle- necks: dealing with complex and unstructured observation space, and having a scalable generative model. Consequently, we propose Copilot4D, novel world modeling approach that first tokenizes sensor observations with VQVAE, then predicts the future via discrete diffusion. To efficiently decode and denoise tokens in parallel, we recast Masked Generative Image Transformer as discrete diffu- sion and enhance it with few simple changes, resulting notable improvement. When applied to learning world models on point cloud observations, Copilot4D reduces prior SOTA Chamfer distance by more than 65% for 1s prediction, and more than 50% for 3s prediction, across NuScenes, KITTI Odometry, and Argov- erse2 datasets. Our results demonstrate that discrete diffusion on tokenized agent experience can unlock the power of GPT-like unsupervised learning for robotics.
Current Observation
Future GT
World Model Prediction
Prediction vs GT
Accurate Near-Tem 1s Prediction
Trush
Prodiction
Model
Diverse Multi-Future 3s Prediction
Traffic the other side of road
Figure 1: Our unsupervised world model Copilot4D can produce accurate near-term 1s predic- tions and diverse multi-future 3s predictions directly the level of point cloud observations.
INTRODUCTION
World models explicitly represent the knowledge of an autonomous agent about its environment. They are defined generative model that predicts the next observation in environment given past observations and the current action. Such a generative model can learn from any unlabeled agent experience, and be used for both learning and planning in the model-based reinforcement learning framework (Sutton 1991p. This approach has excelled in domains such as Atari (Kaiser al. 2019), robotic manipulation (Nagabandi et al. 2020), and Minecraft (Hafner et al. 2023
Learning world models can be viewed as a special case of sequence modeling on agent experience. While Generative Pre-trained Transformers (GPT) (Brown al. 2020 have enabled rapid progress

---

Published as a conference paper at ICLR 2024
in natural language processing (NLP) via sequence modeling of unlabeled text corpus, progress for scaling world models has been less rapid in robotic applications such as autonomous driving. Predic- tion systems in autonomous driving still require supervised learning, either on the level of bounding boxes (Luo et al. 2018), semantic segmentation (Sadat et al., 2020), or instance segmentation (Hu et al. 2021). However, just as GPT learns to understand language via next token prediction, if world model can predict unlabeled future observations really well, it must have developed general understanding of the scene including geometry and dynamics. We ask: what makes it difficult to learn an unsupervised world model that directly predicts future observations?
(i) The observation space can be complex and unstructured. Whether it is autonomous driving or robotic hands solving Rubik's 's cubes (Akkaya et al. (201)), selecting loss on the observation space and building a generative model that captures meaningful likelihoods can be highly non-trivial. By contrast, in natural language processing, language models like GPT (Brown et al. (020) first tokenize a text corpus, then predict discrete indices like classifier, leading to impressive success. Fortunately, this gap can addressed by training VQVAE-like (Van Den Oord et al. 2017) model to tokenize any inputs, from images (Ramesh et al. 202T) to point clouds (Xiong et al. 2023).
(ii) The generative model needs to be scalable. In particular, language models are known to scale well (Kaplan et al. 2020), but they only decode one token at a time. In domains such as autonomous driving, a single observation has tens of thousands of tokens, so parallel decoding of tokens becomes a must. On the other hand, decoding all the tokens of an observation in parallel, which is sufficient for achieving success in Minecraft (Hafner al. (023), would incorrectly assume that all those tokens are conditionally independent given past observations. Thanks to Masked Generative Image Transformer (MaskGIT) (Chang et al. 20222, we can train model to iteratively decode an arbitrary number of tokens in parallel. In this work, we recast MaskGIT into the discrete diffusion framework (Austin et al. 2021 resulting in a few simple changes that notably improve upon MaskGIT.
The analysis above sheds light on a scalable approach to building world models: tokenize each observation frame with VQVAE, apply discrete diffusion on each frame, and autoregressively predict the future. We apply this approach to the task of point cloud forecasting in autonomous driving (Weng et al.] 2021 Mersch al.) 2022 Khurana et al. (023). which aims to predict future point cloud observations given past observations and future ego vehicle poses. This task is essentially about building an unsupervised world model on Lidar sensor observations. We design our neural architectures to leverage prior knowledge for this task: our tokenizer uses an implicit representation for volume rendering (Mildenhall et al., (021)) of ray depth in the VQVAE decoder; the world model uses Transformer that interleaves spatial (Liu al. 2021) and temporal blocks in Bird-Eye View (BEV). After tokenization, our world model operates entirely on discrete token indices.
Our approach, named Copilot4D, significantly outperforms prior state-of-the-ar for point cloud forecasting in autonomous driving. On NuScenes (Caesar et al., 2020). KITTI Odometry (Geiger al. 2012), and Argoverse2 (Wilson et al. (202)) datasets, Copilot4D reduces prior SOTA Cham- for distance by 65% - 75 % for Is prediction, and more than 50 % for 3s prediction. In Figure we showease that not only is our world model able make accurate predictions on a 1s time hon- zon, it has also managed to learn the multi-modalii of future observations on 3s time horizon. The results validate our analysis that the combination of tokenization and discrete diffusion can unlock the possibility of learning world models at scale on real-world data.
2 RELATED WORK
World Models predict the next observation in an environment given the current action and the past observations. The idea of learning a world model from data dates back to adaptive control (Slotine al. 199I). which applies parameter estimation to fixed structure of the dynamics. Under model- based reinforcement learning frameworks such as Dyna (Sutton) 199T). many attempts have been made to use deep generative models as world models. Ha Schmidhuber (2088) trained VAE (Kingma & Welling) (013) to encode observations, and recurrent neural net (RNN) on the latent codes to model the dynamics. Dreamer-v2 (Hafner et al. 2020) finds that replacing Gaussian latents with discrete latents significantly improves world modeling for Atari. IRIS (Micheli et al.) 2022) shows that using a Transformer (Vaswani et al. 2017) rather than an RNN for dynamics modeling further improves Atari results. Those prior works provide valuable guide for building world models for autonomous driving; we tackle the point-cloud forecasting task (Weng et al.) 2021 Mersch et al.) 2022 Weng et al. 2022 Khurana et al. 2023) using lessons learned from those other domains.
2

---

Published as a conference paper at ICLR 2024
Diffusion Models are class of generative models that define forward process from data distribu- tion to noise distribution in closed-form form, and then learn the reverse process from noise distribution to data distribution (Sohl-Dickstein et al., 2015). Diffusion for continuous random variables typi- cally uses Gaussian noise, and utilizes properties of Gaussian distributions to simplify the training objectives (Ho et al., 2020 Kingma et al. (202T) and speed up inference (Song al., (020). In com- parison, diffusion for discrete data (Austin et al., 2021) has received less attention from the commu- nity. Recently, Masked Generative Image Transformer (MaskGIT) (Chang al. 2022) shows that training a BERT (Devlin et al. 2018)) on image tokens with an aggressive masking schedule can out- perform Gaussian diffusion. MaskGIT has been successfully applied to many applications such as text-to-image generation (Chang et al.] (023)) video prediction (Gupta et al., 2022 Yu et al. 2023), and point cloud generation (Xiong et al. 2023). Our work sheds light on the connection between MaskGIT and discrete diffusion, and how MaskGIT can be further improved based on diffusion.
3D Representation for Point Clouds has long been studied in both robot perception and 3D scene generation. For self-driving perception, point cloud data from Lidar sensor typically plays an im- portant role. Modern approaches such as VoxelNet (Zhou & Tuzel 2018) and PointPillars (Lang et al. 2019) first apply PointNet (Qi et al. 2017 7) each voxel or pillar, typically followed by 2D convolution in Bird-Eye View (BEV) Yang et al. 2018) for tasks such as 3D object detection. For 3D generation, implicit neural scene representation has been gaining popularity since Neural Radi- ance Fields (NeRF) (Mildenhall et al. 2021); with an implicit function for occupancy, point clouds can be obtained through differentaible depth rendering (Rematas et al. 2022 Yang et al. 2023). This representation is also used in prior work on point cloud forecasting (Khurana et al. (023). Our tokenizer draws ideas from 3D detection to design the encoder, and from implicit neural scene rep- resentation to design the decoder. After tokenization, however, the specific 3D representations are abstracted away from the world model, which operates on discrete tokens.
BACKGROUND: DIFFUSION MODELS
We review diffusion for a single random variable zo (which can be trivially extended to multi-variate xo). Given xo and the forward process 1( diffusion typically learns reverse process
po ( (ark) by maximizing an evidence-lower bound (ELBO) log Pa (2r0 2 - Latho (No, ()
Eg de (mo)
Σ D
(xx)) + log pe (1o| (Kz(g(=k(zo) ll p((x))
For discrete random variables, since we can easily sum over its probabilities, po (1 (2x) is often parameterized to directly infer un as done in D3PM (Austin et al. 202T):
po xR) = Σ g( IA. Po mx)
(1)
Calculating the posterior g [x 2o) = ((===11=00gg (= (1/(1))/(22)/)) is necessary in the original diffusion loss, which means that the cumulative forward transition matrix defined in g(r (r)) often requires a closed-form solution. In D3PM, absorbing diffusion recasts BERT (Devlin al. 2018) as a one-step diffusion model, where the forward diffusion process gradually masks out ground-truth tokens. VQ-Diffusion (Gu et al.) 2022) points out that, when Iy o Fp. the posterior in absorbing diffusion is not well-defined since in that case (0) = O, which motivated adding uni- form diffusion in their model. By contrast, MaskGIT (Chang al. 2022) has significantly simpler training and sampling procedures based on BERT alone: for training, it masks a part of the input tokens (with aggressive masking schedule) and then predicts the masked tokens from the rest; for sampling, it iteratively decodes tokens in parallel based on predicted confidence.
Classifier-fe free diffusion guidance (Ho & Salimans 2022) has become a standard tool for diffusion- based conditional generation. Given context c, it has been shown that sampling from fo (Ta (z, c) x po(zo Irk c) (po aak, c)/po()oo) (20))x>))) rather than directly from po Po(2ol=2s (2 c) performs signifi-
cantly better. Chang et al. 2023) has proposed directly modifying the logits of MaskGIT:
logits,ig (ãol 2x+1, c) logits(ãol (1*+1, c) + (logits (Zo [ c) logits @o (xx)))
(2)
in order to perform classifier-fres guidance analogous its counterpart in diffusion.
4 METHOD: COPILOT4D
Given a sequence of agent experience (, a
a,,,.. ... (T-1), (T) (.TT!).,""7 (!T) where o is an observa- tion and is action, we aim learn world model po that predicts the next observation given
3

---

Published as a conference paper at ICLR 2024
Observation
Encoder
Decoder
Reconstruction
Codebook Lookup
BEV Tokens
Render
Tokenizer
Time
Diffuslon steps
Autoregressively predict future fromes
Mask Token
Predict frame t+1
Spatio-Temporal Transformer
Prodict
frama b+2
t-2
t-1
Post abservation takens and actions
World Model
Action of t+1
Figure 2: An overview of our method for Copilot4D, which first tokenizes sensor observations with a VQVAE-like tokenizer, then predicts the future via discrete diffusion. The tokenizer encodes point clouds into discrete latents in Bird-Eye View (BEV), and does reconstruction via differentiable depth rendering. The world model a discrete diffusion model that operates on BEV tokens.
the past observations and actions. the autonomous driving setting that we tackle, the observations (o J. are point clouds from the Lidar sensor, and the actions {a(*), are SF(3) poses of the ego vehicle. We first tokenize each observation (*)) into x(() {o..... [V| 1} where N is the number of tokens in each observation, and V is the vocabulary defined by the learned codebook in VQVAE. We denote x(1) as the tokenized observation t. The learning objective is:
arg max Σ log pa (+
(1) a (2),
(1-1) (1-1))
(3)
Our world model is discrete diffusion model (Austin et al. 2021 Lezama et al. (023) that is able to perform conditional generation given past observations and actions. We denote x; as the tokenized observation É under forward diffusion step k. = is the original data distribution, and the total number of steps K can be arbitrary inference. We outline the inference process follows: to predict an observation at timestep 1+1, the world model first tokenizes past observations o(!) (()) into (!) .....! x(),,, applies discrete diffusion for next frame prediction to decode the initially fully masked x k
(++1) into fully decoded x (++1) and then passes X (t+1) into the decoder of the tokenizer
render the next observation of (++1), For visual overview of our method, see Figure 2
4.1 TOKENIZE THE 3D WORLD
We propose a novel VQVAE-like Van Den Oord et al. 2017) model to tokenize the 3D world represented by point clouds (Xiong 2023).. The model learns latent codes in Bird-Eye View (BEV) and is trained to reconstruct point clouds via differentiable depth rendering.
The encoder uses standard components from point-cloud based object detection literature: first, aggregate point-wise features each voxel with a PointNet (Qi et al.) 2017); second, aggregate voxel-wise features into BEV pillars (Lang et al. 2019); finally, apply S Swin Transformer backbone (Liu et al. (222)) to obtain a feature map that is 8x downsampled from the initial voxel size in BEV,
The output of the encoder, z E(o), goes through a vector quantization layer produce 2.
The novelty of our tokenizer lies in the decoder, which produce two branches of outputs after a few Swin Transformer blocks. The first branch uses an implicit representation (Mildenhall et al. 2021) so that we can query occupancy values at continuous coordinates. To query (z, y. 2). we apply bilinear interpolation on 3D neural feature grid (NFG) (Yang et al., (023) outputted by the decoder to obtain feature descriptor, which then goes through a multi-layer perceptron (MLP) and sigmoid to arrive at an occupancy value a [0, 1]. Given ray r(h) p + hdd starting point p and traveling in direction d, the expected depth D can be calculated via differentiable depth rendering on N, sampled points { U4, 24) N. along the ray:
i-1
Nr
a; = (MMM (I,, Vi. (.)))) w: = a; I (1 - (()=))) D(r, 2) = wehy i (4)
j=1
4

---

Published as a conference paper at ICLR 2024
Algorithm Training
Algorithm 2 Sampling
1: repeat
1: Xx all mask tokens
2: Xo {1...... IVV N 7(x0)
2: for k K 1,...,0. do
3: uo Uniform((, 1)
3:
X po(: xx+1)
4: Randomly mask ["(ua)[N] tokens in Xo
l = log po (Ro xx+1)+ Gumbell 10
5: U1 ~ Uniform((. (0, 1)
5:
On non-mask indices of x241: l. f- +.00
6: Randomly noise (u1 7)% of remaining tokens
M [k]
7: X +
masked-and-noissd Xo
7:
X f X. on top-M M indices 1x
8: arg max, log px xx) with cross entropy
end for 9: return X0
9:
until converged
Figure 3: Our improved discrete diffusion algorithm. Differences with MaskGIT (Chang et al, 2022 are highlighted in blue 7(u) = cos(uT/) /2) is the mask schedule. We set = 20 by default.
The second branch learns a coarse reconstruction of the point clouds by predicting whether a voxel has points in its inputs. We denote this binary probability v. During inference, this branch is used for spatial skipping (Li et al. 2023) to speed up point sampling in rendering.
The loss function for the tokenizer is combination of the vector quantization loss Lug and the rendering loss Creader. The vector quantization loss learns the codebook and regularizes the latents: Lig A1 [s)] 2 + /2||sg 2] E(o)]l In the rendering loss, supervision is applied on both branches: the depth rendering branch has an L1 loss on depth with an additional term that encourages W; to concentrate within of the surface (Yang et al., (023); the spatial skipping branch optimizes binary cross entropy. The tokenizer is trained end-to-end to reconstruct the observation:
Crender
Er DD(r 2) Dgella + Σ E1(h - Dyel c)][w:]122 + BCE(V, Vgt)
With pretrained tokenizer, both the inputs and the outputs of the world model are discrete tokens.
4.2 MASKGIT AS A DISCRETE DIFFUSION MODEL
Masked Generative Image Transformer (MaskGIT) (Chang et al. 2022) has been shown to scale for a variety of applications. Interestingly, discrete diffusion models such D3PM (Austin al., 2021] have not yet seen similar success, despite having a much more first-principled framework and toolbox. We observe that the key to recasting MaskGIT as discrete diffusion model is the following proposition in Campbell et al. (2022). It turns out that the parameterization introduced in Equation I allows further lower bound on ELBO under data distribution (xx) (also see A.4).
K
By(2 [log po (2o)] 2 By(=o)l Cetbb (zo, 0) 2 En(zo)((aalaa (o Dog 2x)] +
Σ Jrm
(6)
Which is almost the same loss MaskGIT loss, except that: for the diffusion posterior g(rx (rr)) to be well-defined when Fy * zo. uniform diffusion in non-masked locations is needed; and the loss is applied not just to masked locations. This implies that few simple changes can turn MaskGIT into an absorbing absorbing-uniform uniform discrete diffusion model. During training, after masking a random proportion of tokens in No. we inject up to v7% of uniform noise into the remaining tokens, and apply a cross entropy loss to reconstruct Tg. 17 is fixed hyper-parameter. During sampling, besides parallel decoding, we allow the model to iteratively denoise earlier sampled tokens. The differences with MaskGIT are highlighted in [ and For conditional generation, classifier-free diffusion guidance can be applied by modifying the logits pe for sampling Xe and calculating 4 according to Equation (2). While resampling tokens has been known to help MaskGIT (Lezama et al., 2022 2023). our method only requires training a single model rather than two separate ones.
We now use our discrete diffusion algorithm to build world model on top of observation tokens.
4.3 LEARNING A WORLD MODEL
For an autonomous agent, the environment can be viewed as black box that receives an ac- tion and outputs the next observation. A world model is learned generative model that can be
5

---

Published as a conference paper at ICLR 2024
Condition on the past, predict the future
Joint modeling of the past and the future
Model each frame individually
Training objectives
World Model inputs
Attention mask
Quary
Query
Query
between frames
Kdy
Kay
Ko
Training targets
Ground Truth
Mask
Token crmm sompled from codebook)
Figure 4: The training objectives of our world model in Copilot4D. Other than future prediction, also train the model to jointly model the past and the future, and individually model each frame. Joint modeling ensures that the model can accurately predict the future even with imperfect past conditioning, Individual frame modeling is necessary for glassifier-free diffusion guidance. During training, which objective to optimize is randomly sampled at each iteration.
used in place of the environment. Let (x('),
Given the context (tt-1) (xD ,, .... ,xx 1 a(t-1)) as past agent history, a discrete diffusion world model pe learns to (t)
predict the next observation x starting from fully masked XK going through K intermediate
steps K XK-1 ... X during the reverse process of diffusion. Using Equation (
(g(T)) logpo(x!") (t) | c"(-")) By(r))
ΣΣΣ E x(x) (6
[log (xx
|x c cl"
|x (1))
k=
Autorogressive future prediction
Discrete dilllusion on each observation
However, in the GPT-like formulation of autoregressive modeling, the model is always able to see all past ground-truth tokens for next frame prediction during training. In robotics, depending on the discretization of time, the world might only change incrementally within the immediate next frame; learning predict only the immediate next observation will not necessarily lead long-horizon reasoning abilities even the loss is optimized well. Therefore, training the world model should go beyond next observation prediction and instead predict an entire segment of future observations given the past. Accordingly. we design the world model to be similar to a spatio-temporal version of BERT, with causal masking in the temporal dimension. Future prediction done via masking, infilling, and further denoising. The model is trained with a mixture of objectives (see Figure 4
1. 50% of the time, condition on the past, denoise the future.
2. 40% of the time, denoise the past and the future jointly.
3. 10% of the time, denoise each frame individually, regardless of past or future.
The first objective is about future prediction. The second objective also has a future prediction component, but jointly models the future and the past, resulting in harder pretraining task. The third objective aims to learn an unconditional generative model, which is necessary for applying classifier- free diffusion guidance during inference. By the word denoise, we are referring to Algorithm I where parts of the inptus are first masked and noised, and the model learns to reconstruct the original inputs with cross coos--yn loss. All three objectives can be viewed as maximizing the following:
Eg(r),kr... .. Sample [logPo( x _ (t-1) o(x (10 (1) > (x!"))
(T) k,
[T-))
x
Ignored for Objective type
During inference, we still autoregressively predict one frame at time. Each frame is sampled using Algorithm 2 with classifier-fre free diffiusion guidance (CFG) in Equation (2). At each timestep t, the context in diffusion guidance is c(t-1), the past observation and action history of the agent. See Figure 10 in the Appendix for an illustration of how CFG is used in our world model.
Next, we outline how both training (with our mixture of objectives) and inference (with classifier- free diffusion guidance) can be implemented with spatio-temporal Transformer.

---

Published as a conference paper at ICLR 2024
Current Oeservation
GT
Ground Truth
Curmnt
Previous SOTA
(4D-Dcc)
kenteer Recorstructin
Be Predictin
To Prediction
de Predictoo
Ours
Current Obsenvtion
0.5s Future
So
Ground Truth
Prediction
Prediction
Previous SOTA (4D-Occ)
Reconernuction
0c
Prediction
Prediction
Figure 5: Qualitative comparison against prior shate-of-the-atee method 4D Occupancy (4D-Occ) on Argoverse2 Lidar dataset. Copilot4D achieves significantly better results, demonstrating greater capabilities on novel-view synthesis of environment as the ego vehicle moves, understanding the motion of other vehicles in the scene, and modeling the Lidar pattern of ground points.
4.4 A SPATIO-TEMPORAL TRANSFORMER FOR WORLD MODELING
The architecture our world model a spatio-tempori Transformer that simply interleaves spatial attention and temporal attention. For spatial attention, we Swin Transformer (Liu al. 2021) on each individual frame. For temporal attention, we use GPT2 blocks (Radford et al. 2019 to attend
over the same feature location across time. We use U-Net (Ronneberger et al. 2015 structure that combine three levels of feature with residual connections, and make predictions at the same resolution as the initial inputs. Actions, which in our case are the poses of the ego vehicle, are added to the beginning of each feature level corresponding to their observations, after being flattened and going through two linear layers with LayerNorm (Ba al. 2016p in between.
Temporal attention mask plays a crucial role in both training and inference. During training, when optimizing first two types objective, causal masking applied to all temporal Transformer blocks; when optimizing the third type of objective to learn unconditional generative model, the temporal attention mask becomes an identity matrix such that each frame can only attend to itself. During inference, the model decodes and denoises guidance can be efficiently implemented by increasing temporal sequence length by 1, and setting the attention mask to be causal mask within the previous sequence length, and an identity mask for the last frame, so that this added frame becomes unconditional generation (see Figure TO).
frame at time; classifier-free diffusion
5 EXPERIMENTS
In this section, we aim to answer the following questions: (1) Can our proposed world modeling approach outperform previous state-of-the-art point cloud forecasting methods on large-scale self-

---

Published as a conference paper at ICLR 2024
Table 1: Results on NuScenes and KITTI Odometry datasets. Comparison against previous state-
of-the-art methods: SPFNet (Weng et al. 2021), S2Net (Weng al. 2022). ST3DCNN (Mersch) et al. 2022), and 4D Occupancy (4D-Occ) (Khurana et al. The color magenta means that metric computed within the Region Interest defined in Khurana et (2023); -70m to +70m in both z-axis and y-axis, -4.5m to +4.5m in 2-axis. L1 Med means median LI error within ROI;
AbsRel Med means the median absolute relative L1 error percentage within ROI.
NuScenes 1s
Chamfer
L1 Medl
AbsRel Medl
L1 Mean
AbsRel
Chamfer,
SPFNet
2.24
4.58
34.87
4.17
S2Net
1.70
3,49
28.38
2.75
4D-Oce
1,41
0.26
4.02
1.40
10.37
2.81
Copilot4D
0.36
0.10
1.30
1.30
8.58
2.01
NuScenes 3s
SPFNet
2.50
5.11
32.74
4.14
S2Net
2.06
4.78
30.15
3,47
4D-Occ
1.40
0.43
6.88
1.71
13.48
4.31
Copilot4D
0.58
0.14
1.86
1.51
10.38
2.47
KITTI 1s
ST3DCNN
4.11
3.13
26.94
4.51
4D-Occ
0.51
0.20
2.52
1.12
9.09
0.61
Copilot4D
0.18
0.11
1.32
0.95
8.59
0.21
KITTI
ST3DCNN 4.19
3,25
28.58
4.83
4D-Occ
0.96
0.32
3.99
1.45
12.23
1.50
Copilot4D 0.45
0.17
2.18
1.27
11.50
0.67
Table 2: Results on Argoverse 2 Lidar Dataset. We evaluate on evenly subsampled 4000 frames on the Argoverse 2 Lidar validation set. All metrics computed within the ROI.
Is Prediction
Chamfer
LI Med|
AbsRel Medj
LI Mean
AbsRel Mean
4D-Occ
1.42
0.24
1.67
2.04
11.02
Copilot4D 0.26
0.15
0.94
1.61
8.75
38 Prediction
4D-Occ
1.99
0.42
2.88
2.62
15.66
Copilot4D 0.55
0.19
1.26
1.99
11.86
driving datasets? (2) How important is classifier-free diffusion guidance in a discrete diffusion world model? (3) Does our improved discrete diffusion algorithm achieve better performance compared to MaskGIT, in terms of learning a world model?
Datasets and Experiment Setting: We use NuScenes (Caesar et al., 2020). KITTI Odometry Geiger et al. 2012), and Argoverse2 Lidar (Wilson et al. 2023). three commonly used large-scale datasets for autonomous driving. Our evaluation protocol follows Khurana et al. (2023): on each dataset, we evaluate Is prediction and 3s prediction by training two models. Each model is given past point cloud observations and future poses (which are the actions) of the ego vehicle. For KITTI and Argoverse2, each model receives 5 past frames and outputs 5 future frames (spanning either 1s or 3s). For NuScenes, the 2Hz dataset is used; outputs two future frames; 3s prediction takes in 6 past frames and outputs 6 future frames. While our world modeling approach is not limited to this experimental setting, we follow the same protocol to be able to directly compare against prior methods.
result, is prediction takes two past frames and
we follow the common metrics for point cloud forecasting (Khurana et al. 2023 which include Chamfer distance, L1 depth for raycasting (L1 Mean), and relative LI error ratio (AbsRel). However, we notice an issue with the previously proposed metrics: while model predictions are made only within the region of interest (ROI), the ground-truth point clouds are not cropped accord-
Metrics:
8

---

Published as a conference paper at ICLR 2024
Table 3: Our method for classifier-free diffusion guidance (CFG) significantly improves prediction results and especially the Chamfer Distance metric.
NuScenes 3s
Chamfer
LI Medl
AbsRel Med
LI Meant
AbsRel Mean
w 0.0 (no CFG)
1.40
0.13
1.81
1.23
8.34
w 1.0
0.56 (60% J)
0.13
1.78
1.22
9.32
w = 2.0
0.58
0.14
1.86
1.27
9.90
Table 4: Our proposed discrete diffusion algorithm significantly improves upon previous masked modeling method MaskGIT (Chang et al. 2022). Both models use only 10 sampling steps per frame of prediction (128 128 tokens), applied with classifier-free diffusion guidance 2.0.
NuScenes 3s
Chamfer
LI Medj
AbsRel Med
LI Mean)
AbsRel Mcan
MaskGIT
0.82
0.16
2.09
1.41
11.66
Ours
0.58 (29% !)
0.14
1.86
1.27
9.90
ing to the ROI, resulting in artifically high error metrics simply because the ROI might not cover the full point cloud range. Consequently, we report metrics computed within the ROI in magenta, which better reflects the performance of a model. We also report the median of LI depth error (L1 Med), since the median is more robust to outliers than the mean. Following previous evaluation protocols, the ROI defined to be - 70m to + +70m in both r-axis and y-axis, -4.5m to +4.55 in z-axis around the ego vehicle.
Benchmark against state-of-the-art: Table and 2 show quantitative comparisons with state- of-the-art unsupervised point cloud forecasting methods on three datasets. Our baselines include SPFNet (Weng et al. (021), S2Net (Weng et al. 2022). ST3DCNN (Mersch et al. 2022), and 4D Occupancy (Khurana et al. 2023 Our method Copilot4D is able outperform prior methods by significant margin across all three datasets. In particular, for Is prediction, we are able to see 65% 75% reduction in Chamfer Distance compared prior SOTA across all three datasets; for 3s prediction, we are able to see more than 50% reduction in Chamfer. We also present qualitative comparison with previous SOTA (4D Occupancy) in Figure Copilot4D learns qualitatively better future predictions, demonstrating impressive ability synthesize novel views on the background as the ego vehicle moves and forecast the motion of other vehicles.
Classifier-free diffusion guidance (CFG) is important: in Table 3 we show that CFG reduces Chamfer by as much as 60% in an ablation on NuScenes 3s prediction. The result not surprising considering that CFG has become a standard tool in text-to-image generation; here we show that using past agent history (c(t-1) in Section 4.3 as CFG conditioning improves world modeling. Intuitively, CFG amplifies the contribution of the conditioned information making predictions.
Our proposed simple changes to MaskGIT train the model to do a harder denoising task Algorithm and thus allow the inference procedure Algorithm 2 to iteratively revise chosen tokens. Those changes notably improve MaskGIT on ablation run of NuScenes 3s prediction task. shown in Table 4 In our case, each frame has 128 x 128 tokens with 10 diffusion steps; average each diffusion step decodes 1600 tokens in parallel. Being able to denoise and resample already decoded tokens reduces Chamfer distance 3s prediction by 29%.
Improvement upon MaskGIT:
CONCLUSION
Learning unsupervised world models is promising paradigm where GPT-like pretraining for robotics can potentially scale. In this work, we first identify two practical bottlenecks of this paradigm: simplifying the complex observation space, and building scalable generative model for spatio-temporal data. We then propose Copilot4D, which combines observation tokenization, discrete diffusion, and the Transformer architecture as a new approach for building unsupervised world models, achieving state-of-the-an art results for the point cloud forecasting task in autonomous driving. One particularly exciting aspect of Copilot4D is that is broadly applicable to many do- mains. We hope that future work will combine our world modeling approach with model-based reinforcement learning to improve the decision making capabilities of autonomous agents.

---

Published as a conference paper at ICLR 2024
7 ACKNOWLEDGEMENT
We thank Chris Zhang, Anqi Joyce Yang, Thomas Gilles, and many others on the Waabi team for helpful discussions and valuable support throughout the project.
REFERENCES
Ilge Akkaya, Marcin Andrychowicz, Maciek Chociej, Mateusz Litwin, Bob McGrew, Arthur Petron, Alex Paino, Matthias Plappert, Glenn Powell, Raphael Ribas, et al. Solving rubik's cube with a robot hand. arXiv preprint arXiv:1910.07113. 2019.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne Van Den Berg. Structured denoising diffusion models in discrete state-spaces. Advances in Neural Information Processing Systems, 34:17981-179999 2021.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 1rXiv:1607.06455 2016.
arXiv preprint
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint 338. 2013.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamners Advances in neural information processing systems, 33:1877-19011 2020.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan. Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Andrew Campbell, Joe Benton, Valentin De Bortoli, Thomas Rainforth, George Deligiannidis, and Arnaud Doucet. A continuous time framework for discrete denoising models. Advances in Neural Information Processing Systems, 35:28266-28279, 2022.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11322 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. arXiv preprint arXiv:2301 00000000044., 2023.
Jacob Devlin, Ming-Wes Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04800. 2018.
Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. Jukebox: generative model for music. arXiv preprint wxXv:20055.00311, 2020.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint 1rXiv:2010. 1929, 2020.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp.
JEEE, 2012.
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to- text-to-image synthesis. Proceedings of the (EEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
10

---

Published as a conference paper at ICLR 2024
Agrim Gupta, Stephen Tian, Yunzhi Zhang, Jiajun Wu, Roberto Martin-Marthn and Li Fei-Fei. Maskvit: Masked visual pre-training for video prediction. arXiv preprint arXiv:2206. 11894, 2022.
David Ha and Jürgen Schmidhuber. Recurrent world models facilitate policy evolution. Advances in neural information processing systems, 31, 2018.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193. 2020.
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains through world models. arXiv preprint arXiv:2301.04100 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. arXiv: 08415, 2016.
Gaussian error linear units (gelus). arXiv preprint
Jonathan Ho and Tim Salimans. arXiv:2207.12590. 2022.
Classifier-free diffusion guidance.
arXiv preprint
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. Fiery: Future instance prediction bird's-cye view from surround monocular cameras. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 15273-15282, 2021.
Minyoung Huh, Brian Cheung, Pulkit Agrawal, and Phillip Isola. Straightening out the straight- through estimator: Overcoming optimization challenges vector quantized networks. arXiv preprint arXiv:2305.088-22 2023.
Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos, Blazej Osinski, Roy H Campbell, Konrad Czechowski, Dumitru Erhan, Chelsea Finn, Piotr Kozakowski, Sergey Levine, et al. Model-based reinforcement learning for atari. arXiv preprint arXiv: 00tv://0000 19199990303774 2019.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08366 2020.
Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1124-1124, 2023.
Diederik Kingma, Tim Salimans, Ben Poole, and Jonathan Ho. Variational diffusion models. Ad- vances in neural information processing systems, 34:21696-21707, 2021.
Diederik P Kingma and Max Welling, arXiv:1312.6114 2013.
Auto-encoding variational bayes.
arXiv preprint
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 12697-12705, 2019.
José Lezama, Huiwen Chang, Lu Jiang, and Irfan Essa. Improved masked image generation with token-critic. In European Conference on Computer Vision, pp. 70-86. Springer, 2022.
Jose Lezama, Tim Salimans, Lu Jiang, Huiwen Chang, Jonathan Ho, and Irfan Essa Discrete predictor-corrector diffusion models for image synthesis. In International Conference on Leam- ing Representations (ICIR), 2023.
11

---

Published as a conference paper at ICLR 2024
Ruilong Li, Hang Gao, Matthew Tancik, and Angjoo Kanazawa. Nerface: Efficient sampling accel- erates nerfs. arXiv preprint arXiv:2305.0496,, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision. pp. 10012-10022, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.083 2017.
Wenjie Luo, Bin Yang, and Raquel Urtasun. Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with single convolutional net. In Proceedings of the IEEE conference on Computer Vision and Puttern Recognition, pp. 3569-3577, 2018.
Chris J Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. arXiv preprint arXiv:161 1.0712111.00011, 2016.
Benedikt Mersch, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks. In Conference on Robot Learning, pp. 1444-1454. PMLR, 2022.
Vincent Micheli, Eloi Alonso, and François Fleuret. Transformers are sample efficient world models. arXiv preprint (rXXi:2209.00888 2022.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Anusha Nagabandi, Kurt Konolige, Sergey Levine, and Vikash Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pp. 1101-1112. PMLR, 2020.
Ofir Press and Lior Wolf. Using the output embedding to improve language models. arXiv preprint arXiv: 1608. 05859, 2016.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training, OpenAl Technical Report, 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831 PMLR, 2021.
Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi, Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12932-12942, 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention- MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceed- ings, Part III 18, pp. 234-241 Springer, 2015.
Abbas Sadat, Sergio Casas, Mengye Ren, Xinyu Wu, Pranaab Dhawan, and Raquel Urtasun. Per- ceive, predict, and plan: Safe motion planning through interpretable semantic representations. In Computer Vision ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pp. 430. Springer, 2020.
12

---

Published as a conference paper at ICLR 2024
Jean-Jacques E Slotine, Weiping Li, et al. Applied nonlinear control, volume 199. Prentice hall Englewood Cliffs, NJ, 1991.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, al. Using deep- speed and megatron to train megatron-turing nlg 530b, large-scale generative language model. arXiv preprint arXiv:2201.11998 2022.
Jascha Sohh-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 2(4):160-163, 1991.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée
Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint 23X2: 2023,
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Xinshuo Weng, Jianren Wang, Sergey Levine, Kris Kitani, and Nicholas Rhinehart. Inverting the pose forecasting pipeline with sp12: Sequential pointcloud forecasting for sequential pose fore- casting. In Conference robot learning, pp. 11 11-20. PMLR, 2021.
Xinshuo Weng, Junyu Nan, Kuan-Hui Lee, Rowan McAllister, Adrien Gaidon, Nicholas Rhinehart, and Kris M Kitani. S2net: Stochastic sequential pointcloud forecasting. In European Conference on Computer Vision, pp. 549-564. Springer, 2022.
Benjamin Wilson William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khan- delwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Ar- goverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00495 2023.
Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, and Raquel Urtasun. Learning compact represen- tations for lidar completion and generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1074-1083, 2023.
Bin Yang. Wenjie Luo, and Raquel Urtasun. Pixor: Real-time object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7652- 7660, 2018
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1389-1399, 2023.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang. Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 10459-10469, 2023.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4490- 4499, 2018
13

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 ADDITIONAL QUANTITATIVE RESULTS
Table 5: Zero-shot transfer performance across datasets: training on Argoverse 2 (AV2) Lidar and evaluating on KITTI Odometry. Compared to 4d Occupancy (4d-Oec) Khurrana et al. (023). our method Copilot4D achieves significantly better dataset transfer results.
[AV2 KITTI] is Prediction
Chamfer
LI Mcan
AbsRel|
Chamfer,
4D-Occ
2.52
1.71
14.85
3.18
Copilot4D
0.36
1.16
11.1
0.44
[AV2 KITTI] 3s Prediction
4D-Occ
4.83
2.52
23.87
5.79
Copilot4D
1.12
1.91
19.0
1.65
Zero-shot transfer performance across datasets: Table follows prior protocols of training on Argoverse2 and testing KITTI Odometry, and shows that: Copilot4D achieves more than 4x smaller Chamfer distance compared to 4D Occupancy, on both Is and prediction. Our results indicate that tokenization, discrete diffusion, and spatio-temporal Transformer form a powerful combination that can achieve greater degree cross-dataset transfer.
Table 6: Results the test of Argoverse Lidar dataset. Besides the results validation presented in the main paper, we also evaluate on evenly subsampled 4000 frames on the Argoverse 2 Lidar test set. The results are very similar; we present the test-set results here for completeness. All metrics are computed within the ROI.
Prediction
Chamfer
LI Medl
AbsRel Med
LI Mcan
AbsRel Mcan
4D-Occ
1.51
0.25
1.66
2.07
11.21
Copilot4D
0.25
0.15
0.96
1.64
9.02
Prediction
4D-Occ
2.
0.45
3.05
2.69
16.48
Copilot4D
0.61
0.20
1.29
2.08
12.49
Ablation studies on the tokenizer:
Table 7 provides ablation the effect of spatial skipping (Li ct al. 2023), where train another tokenizer from scratch without the coarse reconstruction branch or the spatial skipping process; this tokenizer is only supervised with the depth rendering loss. The table shows that spatial skipping improves point cloud reconstructions. We illustrate the details of the spatial skipping process in Figure
We also show that differentiable depth rendering using implicit representation is crucial. In Figure 9 we provide a qualitative comparison between the VQVAE UltraLiDAR (Xiong et al. 2023) and our proposed tokenizer. The UltraLiARR model only predicts whether voxel has points present, and similar to our coarse reconstruction branch. The results show that UltraLiDAR is unable to reconstruct fine-grained geometry due to the limited resolution of voxel predictions, whereas our tokenizer is able to produce high-fidelity point clouds that recover the details in the input point clouds.
A.2 MODEL DETAILS
Both the tokenizer and the world model are quite lightweight. The tokenizer is 13-Million parame- ter model; the world model is a 39-Million parameter model. To put the parameter counts in context, the tokenizer has fewer parameters than ResNet-34 (He et al., 2016) (21.8 M parameters), and the current world model has fewer parameter count than a ResNet- results such small model scale is another highlight of our algorithm.
(44.5M parameters). Achieving
14

---

Published as a conference paper at ICLR 2024
Observation
PointNet-ike Voxel Encoder
3D Voxel Feature Volume
BEV Pillor Pooling
3D Points in voves
Shapp (1924, 64.
[
dirrat
1024
1024
2D BEV Features
4x downsompled
8X downsampled
Vector Quantization
SIT
lavin Mixals
6-lain Airaks
2
ambied
Marging
Polddtty
Lineor
WincDA
Codebook Lookup
Shapes
(1924, 1024, 64)
(256, 256, 128)
(128, 128, 256)
[128, 128, 1024)
Ex clownsampped
4x downsompled
3D Neural Feature Grid (512, 512, 54, 16)
Shapos
(128 120, 2583
(256, 268, 128)
Query contrnuuus coordinchas clong therry
6 Blocks
Imapppttion
64
512
Rey
(1024, 1024, 84)
Aigensi
Reconstruction
Rendering
Linear
(128, 123,
64
1024
Towenol
Sporial
Rander
1024
m
I
(-)) Dir.
Ray r(h)
3D Binary Voxal Logits
Sensor orign hn
Figure 6:
Detailed architecture of our point cloud tokenizer, which combines PointNet, Bird- Eye View (BEV) representation, Neural Feature Grid (NFG), and differentiable depth rendering.
Spatial Resolutions
(32, 32)
Dim=512
Spatio-Temparra Block
Merging
Merging
(64, 64)
Dim=384
2 Spptto-Tempprra Blocks
Spatio-Temparal Block
Patch Merging
Level Merging
(128, 128)
Dim=256
Teproo-Temppr Blocks
-pptiioTermprr Blocks
Spatio-Temporal Block
Iamporoi
spatic
Temporal Black
Loghts
Logits
Loghs
Actions
Logit Shopes:
(128,
128, 1024)
2 Swin
Blockss
BEV Tokens
Figure 7: Detailed architecture of our U-Net based Transformer world model, which interleaves spatial and temporal Transformer blocks and applies multiple levels of spatial feature resolutions.
15

---

Published as a conference paper at ICLR 2024
Spatial Skipping in Point Cloud Tokenizer
Training:
Find nameemply vcsele
Mizz Einary non remplyy
Somp points maniooo
notrvmmmity stowlc olligg themry
truft capii
honvemptyy cowns
Plx pooted ron emply
Somplle paints dlong soy quary
enyyy
Inference:
Maauppsd the binory moskk to increase recoll)
Stotrple par rite inaile musi paoepppeeee non-omphyy vokats olong the ray
Ground bulht poin
Depth rer denclaning
Cthers pohite
Sompleedyoon theroy/tocuury
Figure 8: Illustration the spatial skipping process. We find that spatial skipping (Li et al, 2023) not only speeds up tokenizer inference but also results in slightly better results. Rather than uniformly sampling points along the ray for depth rendering, we sample points only within the estimated non-empty regions along the ray. Specifically, voxels are non-empty, apply max pooling on this binary mask to increase recall of non-empty voxels, and sample (ray) points only within the max-poote max-pooled non-empty voxels. During training, the ground- truth binary mask of non non-empinn cam--memtess can be computed from original point clouds; this mask is also used to supervise the coarse reconstruction branch. During inference, we threshold the prediction from the coarse reconstruction branch to produce the mask.
predict binary mask indicating which
NuScenes [Reconstruction]
Chamfer (within ROIJ
L1 Median,
L1 Mean,
Chamfer
No spatial skipping With spatial skipping
0.148
0.044
1.42
1.66
0.082
0.044
0.82
1.64
Table 7: Ablation study on spatial skipping in the tokenizer. Here we train another tokenizer from scratch without the coarse reconstruction branch or the spatial skipping process. Our results show that spatial skipping leads to improved reconstructions, which is likely due to the fact that it allows the sampling of points along the rays to focus on the non-empty regions.
A.2.1 TOKENIZER
The tokenizer follows VQVAE Van Den Oord et al. 2017) structure to encode point clouds into Bird-Eye View (BEV) tokens and reconstruct input point clouds via differentiable depth rendering.
Tokenizer encoder The initial layer in the encoder a voxel-wise PointNet (Qi et al., 2017) similar to VoxelNet (Zhou & Tuzel] 2018) that encodes the distance of each point its corre- sponding voxel center, with one modification to PointNet: while the initial PointNet uses max pooling as the permutation-invariant aggregation function, use sum operation Layer- Norm Ba et al. 2016). which is also a 14.0625cm the x, y. z dimensions, following an input voxel size similar to the one used in Xiong et al. (2023), since it was noted that voxel sizes could matter lot. We model the 3D world the 80m, 80m] x [-80m, 80m] [-4.5m, 4.5m) region around the ego vehicle; after initial PointNet (with feature dimension 64), we obtain 3D feature volume of tensor shape 1024 x 1024 x 64 x 64. Following the 3D object detection literature, we pool the 3D feature volume into 2D Bird-Eye View (BEV) representation, using aggregation function (sum oper- ation + LayerNorm) on the z-axis, after going through another Linear layer and adding a learnable embedding based on the z-axis of voxel.
invariant function. We use voxel size of
15.625cm x 15.625em
The encoder backbone Swin Transformer (Liu et al. 2021). We add ViT-style (Dosovitskiy al. 2020) absolute positional encodings of spatial coordinates to the beginning of the backbone.
16

---

Published as a conference paper at ICLR 2024
Input Point Clouds
Our Tokenizer
UltraLiDAR Reconstruction
Reconstruction
NuScenes
KITTI
Argoverse
Qualitative comparison between Copilot4D tokenizer and UltraLiDAR (Xiong al.) 2023). The UltraLiDAR model also uses a VQ-VAE for point clouds, but only predicts whether voxel has points present not, and is unable to reconstruct more fine-grained geometry. Our tokenizer overcomes this challenge via a combination of implicit representation and differentiable depth rendering, leading significantly improved reconstructions and qualitatively different results.
Figure 9:
The initial patch size is 4 for the first two Swin layers (with the feature dimension being 128, and the number of attention heads being 8), leading to a feature map shape of 256 x 256 x 128. We then downsample the feature map and increase the patch size from 4 to 8 via Patch Merging layer, and apply Swin Transformer layers (with the feature dimension being 256, and the number of attention heads being 16), leading to feature map shape of 128 128 x 256. The resolution of the encoder output is 128 x 128, which is 8x downsampled from the initial voxel size, meaning that each feature map grid is in charge of a 1.25m 1.25m region in BEV. We find important place a LayerNorm (followed by GELU activation (Hendrycks & Gimpel 2016) and Linear layer) on the encoder output before vector quantization, which has also been observed in Huh et al.) (2023). Following Xiong et al. (2023), we increase the feature dimension to 1024 with linear layer before vector quantization; we also find this to important for good reconstruction.
Vector quantization layer Codebook collapse, meaning that only few codes are used, is known to be a common issue in VQVAE training. We empirically find that the random restart strategy proposed in Dhariwal et al. (2000) is insufficient for avoiding codebook collapse in case. We instead resort to the K-Means clustering strategy in Xiong et al. (2023). More specifically, we set up memory bank to store the most recent encoder outputs before vector quantization; the size of the memory bank is 10 times the size of the codebook. We define a code to be a dead code if has not been used for 256 iterations. If more than 3% of the entire codebook have become dead codes, then we run K-Means clustering on the memory bank to re-intialize the entire codebook. However, each codebook must go through at least 200 iterations before it can be re-initialized.
We use the straight- througg gradient estimator (Bengio et al. 2013). 3) as done in the original VQVAE. In the vector quantization loss Lvg A Asllsg [(o) 2 + Az [||s ž E(o) 1|2, we set X 0.25 and A2- 1.0; the intuition is that the codebook should change more slowly than the features.
Tokenizer decoder The decoder backbone also a Swin Transformer, symmetrical to the en- coder backbone. The Paich Merging layer for downsampling in the encoder corresponds to Patch Upsample layer for upsampling in the decoder: first, we use linear layer to upsample the feature map (similar to a deconvolution layer); then, apply LayerNorm use another Linear layer to reduce the feature dimension accordingly. The output of the decoder backbone is 256 256 128 tensor in 2D BEV, which is then fed into separate branches.
each upsampled feature; finally,
17

---

Published as a conference paper at ICLR 2024
As mentioned in the main paper, the first branch is 3D neural feature grid (NFG) that supports bilinear interpolation for querying continuous coordinates. The NFG branch uses a LayerNorm and Linear( (128, 2 x 64 X 16) layer to get a 3D feature volume of tensor shape 512 x 512 64 16, with each voxel represented by a 16-dimensional vector. Once this 3D feature volume has been computed, each query (Ira, Vi, 2,) along different rays only requires bilinear interpolation this feature volume to obtain an initial 16-dimensional vector, which can then be passed into lightweight two-layer produce an occupancy value with a sigmoid gate at the end. This occupancy value can then be used for differentiable depth rendering in Equation (4). An L1 loss is applied supervise the rendered depth, with an additional term that encourages the learned sample weight distribution w; to concentrate within meters of the surface, as outlined Equation (
MLP (we set its hidden dimension be 32, with ReLU activation)
We set the margin e = 0.4.
The second branch is a coarse reconstruction branch that is used for spatial skipping during infer- ence. Given the 2D BEV feature map of tensor shape 256 256 x 128 from the decoder backbone, we apply LayerNorm on each feature, followed by Linear( [28,, 4 x 64) layer, to produce a 3D volume of binary classification logits 1024 x 1024 x 64, which is trained to estimate whether each voxel has points present in the input observation. The bias of the final logit is initialized to be -5.0, since most voxels are empty in point cloud observations.
At inference, given discrete BEV tokens and query rays, the tokenizer decoder takes in the BEV tokens to produce the 3D) NEG and the coarse binary voxel predictions from the two branches. At first, the spatial skipping branch can provide coarse estimate of where sample the points along the rays. This is achieved by adding Logistic noise to the binary logits (Maddison et al. (016) and then thresholding the binary probabilities. We then apply max pooling on this binary 3D volume to produce a much coarser estimate of binary voxel predictions and consequently increase recall (in our case, we chose use a max pooling factor of in Bird-Eye View). For query ray, we search among all intersecting coarse voxels and find all of the coarse voxels estimated to have points present. Subsequently, following standard spatial skipping (Li et al., 2023), for each ray, we only sample points within those coarse voxel grids to query the 3D NFG, leading to much more efficient sampling. The final point clouds are obtained via depth rendering on the NFG branch.
A.2.2 WORLD MODEL
The world model follows a U-Net (Ronneberger et al. 2015) structure with three feature levels corresponding to 1 1x, 2x, and 4x downsampled resolutions. The inputs are 128 128 tokens; the outputs are 128 x 128 x 1024-dimensional logits where 1024 is the vocabulary size of our VQVAE codebook. Adopting the common practice of language models, we use weight tying (Press & Wolf, 2016) between the embedding layer and the final softmax layer. We use Swin Transformer (Liu et al. 2 for all spatial attention layers, and GPT-2 blocks (Radford et al.) 2019) for all temporal attention layers. Spatial attention and temporal attention are interleaved in the following manner: every 2 spatial blocks are followed by temporal block. The temporal blocks apply attention on same feature map location across frames. The feature dimensions for the three feature levels (256, 184, 512). with the number of attention heads being (8, 12, 16), meaning that we use a fixed 32-dimension per head across all levels.
We design our network to be similar to the GPT-2 Transformer, in the sense that the overall structure of every feature level should be: sum of all residuals in Transformer blocks + one final LayerNorm - one final Linear layer. The benefit of such a design is that, since the structure of our neural net becomes similar to GPT-2, we can directly apply the initialization and optimization strategies found to work well for language models. To make this design compatible with the U- Net structure (which first transitions down and then transition up to make dense predictions), how to merge information when transitioning up across feature levels becomes important. We use the following architecture:
The first feature level transitioning down has (2 spatial + 1 temporal 2 spatial + temporal) blocks, with a window size 8 in spatial blocks. Following Swin Transformer, the Patch Merging layer is used for downsampling.
The second feature level transitioning down also has (2 spatial + 1 temporal + 2 spatial 1 temporal) blocks, with a window size 8 in spatial blocks.
18

---

Published as a conference paper at ICLR 2024
The third feature level transitioning down has (2 spatial + temporal) blocks, with window size 16 in spatial blocks.
The network then transitions up the second feature level using an upsampling layer before adding residual connection. The goal is to upsample the higher-level feature map and merge it with the lower-level feature map. We name such layer Level Merging, which borrows designs from the Patch Merging layer: first we use a linear layer to output the 2x upsampled feature map (similar to deconvolution layer), concatenate with the lower-level feature map, applies LayerNorm every feature, and uses linear projection to reduce the feature dimension. A residual connection is then applied.
Back to the second feature level transitioning up, the feature map goes through (2 spatial -> 1 temporal) blocks, followed by another Level Merging layer to return to the first feature level. Predictions are made at the first feature level.
Back to the first feature level, the feature map goes through additional (2 spatial + temporal + 2 spatial + temporal) blocks.
We use a final LayerNorm followed by a weight matrix (under weight tying with the initial embedding layer) to output the logits.
All our Transformer layers are pre-norm layers, meaning that LayerNorm (Ba et al. 2016 is moved the input of each sub-block, as done in both GPT-2 and Swin Transformer. In a pre-norm Trans- former, the main branch of the network becomes just sum of the residuals since LayerNorm is only placed inside each residual. Therefore, any information we want to condition the network on can be simply added at the very beginning of the network, and will be processed by all following residual modules. The following inputs are added to the beginning of the Transformer world model:
The embeddings of given observation tokens (which discrete indices). The weight of this embedding layer also used for the final softmax layer. On the input side, masking in done via an additional learnable token, as done in BERT, After the embedding layer, we additonally apply Linear + LayerNorm + Linear.
The ego vehicle poses, which are the actions of the ego vehicle. Those x 4 matrices are flattened into 16-dimensional vector, which then goes through Linear LayerNorm Linear, and added to all feature map locations of corresponding temporal frames;
ViT-style (Dosovisskiy et al. 2020) absolute positional encodings of spatial coordinates, which are the same across all temporal frames;
Learnable temporal positional encodings, broadcast to all feature map locations of corre- sponding temporal frames;
Other than the spatio-temporal positional encodings, any information we want to condition the neu- ral net on should first go through Linear + LayerNorm - Linear; we have found that applying LayerNorm the inputs before any Transformer layers is important for learning stability.
We remove the bias parameter in all Linear layers (Tourron et al. (202)), except for the Linear layers outputting query, key, and value during attention in Swin Transformer blocks.
A.3 TRAINING TRANSFORMERS
Initialization Transformer initialization has long been known to be important for successful op- timization. We adopt the initialization and optimization strategies found to work well for language models. Following MT-NLG (Smith et al. 2022). all weights are initialized using fan-in initializa- tion with a normal distribution of O mean and standard deviation of VI/(3H) where H is the input feature dimension of each layer. We note that this initialization strategy is the closest to the GPT-1 (Radford et al.) 2018) initialization scheme. In addition, we use residual scaling at initializa- tion as done in GPT-2: on each feature level, we count the number of residual connections L (which is the number of Transformer blocks times 2), and rescale the weight matrix of each linear layer before the residual connection by VI/L.
The learning rate schedule has a linear warmup followed by cosine decay (with the minimum of the cosine decay set to be 10% of the peak learning rate), and the B2 of AdamW
Optimization
19

---

Published as a conference paper at ICLR 2024
Classifier-Free Guidance (CFG) logits
W.
Conditional Logits
Unconditional
Logits
World Model
World Model
Actions lega poses)
4
3
BEV
Tokens
Shared
Timesteps
Mask
t+1
Figure 10: Illustration of classifier-free diffusion guidance (CFG) during inference for our world model. The conditional logits are conditioned upon the past agent history (past observations and actions). The unconditional logits are not conditioned on the past. Classifier-fres diffusion guidance amplifies the difference between the two to produce the CFG logits used for sampling, Note that CFG can be efficiently implemented with single forward pass at each diffusion step by increasing temporal sequence length by 1, and setting the attention mask to be causal mask within the previous sequence length and an identity mask for the last frame. In addition, thanks to causal masking and factorized spatio-temporal Transformer, for all past timesteps, we only need cached keys and values from the temporal blocks.
Diffusion step
CFG Logits
Sampled Takens
World
Sampling per location
Output tokens
Model
Add tokens
Log likelihoods sampled tokens
Binary mask
Set the log likelihoods at mask locations inf
Select the top locations
Mask Token
M [=(+/))]]
Add Gurbbel noise
Sort
Illutration discrete diffusion sampling at each diffusion step. This procedure corresponds to Algorithm 2 A is the current diffusion step, K is the total number of diffusion steps, N is the number of tokens in each observation, r(u) = cos(ur /2) is the mask schedule.
Figure 11:
Loshchilov & Hutter 2017) is lowered to be 0.95. A weight decay of 0.0001 is applied, but any bias parameters, embedding or un-embedding layers, and LayerNorm parameters are excluded from weight decay. For training tokenizers, 4000 iterations, a max gradient norm clipping of 0, a batch size 16, and a cosine decay length of 0.4 million iterations. For training the world model, we use learning rate of 0.001, linear warmup length of 2000 iterations, max gradient norm clipping of 5.0, batch size of 8, and a cosine decay length of 0.75 million iterations. The cross entropy loss uses 0.1 label smoothing. The same set of hyperparameters are used across all datasets.
learning rate of 0.001. linear warmup length of
20

---

Published as a conference paper at ICLR 2024
A.4 ALTERNATIVE DERIVATION FOR DISCRETE DIFFUSION TRAINING OBJECTIVE
In discrete diffusion, the forward diffusion process is (Austin et al. 2021
q ax - = Cat( = x2-1
where Cat refers to a categorical distribution, and Qk is the forward transition matrix, [Qxlay = (xr j| (meaning that each row of Q sums to but not necessarily the columns).
The cumulative transition matrix
Q ..Q.. Qx
And as a result, if 2 can be written in closed-form, then (( mn) can be written in terms of Q2-
(x> mo) Cat(ax;; 2
We aim to learn a neural net parameterized by to reverse the forward diffusion process:
po(
xkk) Σ
g(
Iu, 5o sx)
An
We present an alternative derivation for the lower bound that we optimize in this paper:
(g( llog pe (zo)]
(yo flog / po(20,2111.. ...IK) xxx))) r.
[po(Foo | [x))
=Eq(no)
log Ex(zri (mo)
p(xx))
( 20)
log Po -1 [xk)
2 (o(ro))(o (mo)
+ -log logp(xx))
g(æ1.K xx0)
K
Pa xR)
log
+ log logp(xx))
q( xb-1)
12
K
K
log q( x2-1))
=Eg(Fox))
log Po (xx-1 xx) + logg(<x))
k21
x21
Rg((o=))
log Σ (
Ik. 5o)oo xxk)
+ By(oukk)
log logp(*k)
log (ok t (k--1)
20
121
K
g( 50 xK)
Bg((ook)
log
po xx)) + Ci
(00 xzk)
121
Zu
g(00 xk-1)
=Ey(sro
log
q( xkk)
po xx))
C1
g(āg
xxk)
Zo
K
Do (To ass)) C1
(g((ro:))
2 log
q IIn)
#21
K
K
(( x
= (g((ro.k)
Σ
g(
1o xxx)
+ +C + + Ey(rox))
Σ
Σ
q(ão Zt log
q(āo xx)
121 Zg
k>1 50
C2
21

---

Published as a conference paper at ICLR 2024
By(
Σ Eo
q(50 xk-1) loggo xx) + C1 + C2
(1(20-0-1--00000 fog po (ão mx)e + +1+
Pe(col=e-)aatauba sa(aulm v)(( 1)((Fol .) Ilog Po (Øo xx)] C2
Bx ((taalln---(an-aa 1 Po xx)] + C1 +2
121
=
logio xx)] + C1 + C2
1
Analyzing the constants C1 and C2:
C1 By((ooi))
log ( [xx-1)+
log p(gp(x))
k=1
that p(arx) (((a))
=B((xa)))
log (2 Ib- +
log g((x))
K
k
C2 = Ex((x.))
log ( (T 12xx)
Bx(x
Σ
Σ ((ā 1x= ((ā xx)
k=1 no
-Eg(xark)
log (( xk-1)
Σ log g((xa)
Eg (o) [x))]
K
C + C2 = Ex( (oog
log g( xx)]
Therefore, this inequality eventually becomes:
K
K
E Be( po(c))) (oo
B (lg + Be(xn ((x)
log a [m)]
(x(eo)a(ee log xR)] C
Which arrives at Equation (6). This loss function avoids the computation of g(zk-1
I'k 2o) as
defined in Equation g and po Po(xx- [x xx)) as defined in Equation
22

---

Published as a conference paper at ICLR 2024
A.5 ABSORBING-UNIFOND DISCRETE DIFFUSION
In this section, we review absorbing-unifont uniform discrete diffusion, and illustrate its connection to our improved version of MaskGIT in detail. First, we aim express all transition matrices in terms of matrix exponential, which provides a convenient expression for the cumulative transtion matrix:
DO
Qx exp(A:
R
n!
Q exp (E R
In absorbing diffusion, each token has a og probability of turning into a mask token at step & (using slightly overloaded notation, be confused with the occupancy values in the depth rendering equation). Let em be a one-hot vector where the index of the mask token is 1.
Q = Rx,()xRR) = =xp(7> (Le, -1)
-xp(-71k + (1 exp( ())(-n))_ee,
= (1 - ax) 0 le
Which utilizes the fact that (1e (1e - 1) = (-1) - 7) and therefore (1e 1) (-1)a+) (1e), I). Breaking down cumulative absorbing transition matrix Q.
Qx =xxp exp ((E
7ss((1e)) 7/
(lem -n)
$5
expp
(-
+ (1-exp) -expp (-E
$
85k
(1 az)l + (1
II (1 a.) Ie sSk
In uniform diffusion, each token has 8% probability of turning into a random non-mask token at step k. With M non-mask categories (M [VD, the transition matrix can be expressed as:
3R M
Q = I - -Ba(1 - em en +
(1 - em) - _mm)
- Bb) + B& eme.
+
1
(1 em)(1 - -em)")
M
=exp (npeR. Ru exp (1/= M
(I em)(1 -em) (I em em)
=xxp(-ma)1 I + (1 - -exp(-1x)) eme T + M (1 - -mm)(1 cm)
Breaking down cumulative uniform transition matrix Φ
Qh exp
((Σn)
(1 em em) (1 em enn
175
sSk
Σ 1/5) + (1- exp (-I Enaa) emme
(eme) + (1 em)(1 em
1
exp
M
o5k
Ba)1 + (1 II (1- BB)
Cm eT + // (1 (1- em) - eem)
1
M
$5k
23

---

Published as a conference paper at ICLR 2024
In absorbing-uniffrm diffusion, with em being the (M 1)-th row, the transition matrix is:
Qk QEQR
Wk +
VR
Vk
1/k
W/ + 1/k
V/
Vk
1/
1/k
Wk
Vk
Vk
1/k
1/k
1/1
Wk: +
1/k 0
0
0
1
3k
Wk
B%
OA
M
Note that O% and Q are commuting matrices; R., and R. inside matrix exponentials also commute
(so that exp(m: R.) exp(7s R.) exp(1ı Ru + 7nh R.)). This property allows us to write the cumulative transition matrix
Q QuQ
Wk + VV
VK
V
Vk
XB
Wh + V
V
VR
XA
F
V
W +
V
Vk
X
V
T
7/k
Wh
+
PA
TA
0
0
K II a5k
B, - 025)
X
(1 a.) $
II
1/
(1
Wk Xx)
M
Intuitively, Wig is the probability that a ground-truth token neither flips into a random token nor gets absorbed into the mask token after forward diffusion steps. Xa is the probability of having mask token after & forward diffusion steps.
Given that the posterior can be expressed
g(ak Fk- 1, (0011 xo)
q( xo)
q Th. mo)
g(ox xx-11
(8)
g(0x oо)
q(rx x0)
We can express it in matrix form for discrete diffusion:
q( 2% j, 30 10-1lu- [Qulij
z0Qu- 2 QT
q( wk,
[Quly
x 20
Allowing us to efficiently compute the closed-fore form posterior from Qx- 1. Qu. and Qx defined earlier.
A.6 TRAINING AND INFERENCE OF ABSORBING-UNIFON DISCRETE DIFFUSION
During training, we first apply masking, which corresponds to absorbing diffusion with Q and then add uniform noise, which corresponds to uniform diffusion with Q" We add at maximum r/ of uniform noise under linear schedule, where we set y) = 20 by default. 7) controls the minimum signal-to-noise ratio among the non-mask tokens.
Despite its simplicity, Algorithm 1 is training an sesorbing-uniform discrete diffusion model. To see this, we simply need define the corresponding noise schedules for uni- form diffusion and absorbing diffusion. The uniform discrete diffusion noise schedule is set to be 3x 1/(K/
1) and consequently (1 II (1 3a)) = n k/K. Since [I- - 3.)) has now become a linear function with respect to k. we can simply uniformly sample noise ratio without computing any transition matrix, as done Algo- rithm As for the absorbing diffusion noise schedule, we define aa II.5 a,) as cosine schedule, and ag be solved accordingly with ag -a//ag11-
24

---

Published as a conference paper at ICLR 2024
During inference, we find that when decoding zy, marginalizing Ez, ((2x/ #1++1 #1 Po) Ø*+1) is not necessary; instead, we directly decode the predicted 20 from po (0(0 as done in MaskGIT. During sampling, once a token has transitioned from mask token non-mask token, even though it can be further revised and resampled, it cannot go back to become masked again, as one would expect from the reverse process of absorbing-unifone diffusion. This is achieved by setting 4 f +o0 on non-mask indices of zx41 Algorithm 2
done in pe (x
We find that, during the sampling process Tn po(- 1, the sampling quality can be improved by using top-K sampling rather than vanilla categorical sampling (similar what would expect from a language model). In our experiments, we sample from the top 3 logits per feature location.
A.7 ADDITIONAL QUALITATIVE RESULTS
We present additional qualitative results: Figure and Figure 14 outline our results on NuScenes; Figure [3] results on KITTI, and Figure 16 and Figure 17 adds to the results Argoverse
2 Lidar. Overall, our models achieve considerably better qualitative results on all three datascts.
Predictions under counterfactual actions We visualize how well the world model can predict the future under counterfactual actions in Figure 12 Here we modify the future trajectories of the ego vehicle (which are action inputs to the world model), and we demonstrate that the world model is able to imagine alternative futures given different actions. The imagined futures are consistent with the counterfactual action inputs. Moreover, the world model has learned that other vehicles in the scene are reactive agents.
Failure cases of our current models We present some failure modes our current models us- ing red arrows in Figure [4 Figure I6 and Figure [7 Modeling vehicle behaviours a 3s time horizon needs further improvement, as the accuracy for 3s prediction seems lower than 1s predic- tion; however, we note that this inaccuracy could be partly due the multi-modality of 3s future prediction (multiple futures could be equally valid). Additionally, on a 3s time horizon, sometimes new vehicles (not present in the past present frames) enter the scene. Currently, the world model does not seem to have learned how to hallucinate new vehicles coming into the region of interest. Nevertheless, those failure cases do not necessarily reflect fundamental limitations of our proposed algorithm; we believe that the issues listed above can be addressed to a large extent by increasing data, compute, and model size under our framework.
25

---

Published as a conference paper at ICLR 2024
Ego Vehicle Action
Scene
Current
Future 1.2s
Future 2.4s
Future 3.0s
GT
Original
Actions
Counter
factual Actions
Scene
Ego Vehicle Action
Current
Future 1.2s
Future 2.4s
Future 3.0s
GT
Original Actions
Counter factual Actions
Visualization of predicted future under counterfactual actions. Here modify the future trajectories of the ego vehicle (which are action inputs to the world model), and demonstrate that the world model is able imagine alternative futures given different actions. The orange trajectory represents the actions taken by the ego vehicle regarding where to drive. The imagined futures are consistent with the counterfactual action inputs. addition, our world model is able to learn that other vehicles in the scene are reactive agents. Notably, in Scene 1, the counterfactual action is for the ego vehicle brake, and our world model has learned that the vehicle behind will also react by braking to avoid a collision with the ego vehicle.
Figure 12:
26

---

Published as a conference paper at ICLR 2024
Gurtent Ocservaton
0.5a Fulure GT
Ta Futune GT
0.6s & Future ST
Ground Truth
Cumeen Obsenvato
0.5s Prediction
Prediction
0.5x To Prediction
Previous SOTA
(DD-Dcc)
Tokurizer rncerstructonn
0.5
Prediction
Curs
Current Observation
Future
Future
Ground Truth
Prediction
Previous SOTA (4D-Ooc)
Reccstataccion
Prediction
Preditioon
Cumment Obeenatioo
Futum GT
Future
Ground Truthh
Current Obsanvation
Pred
Presiction
Previous SOTA (4D-Occ)
Oura
Figure 13: Qualitative comparison NuScenes is prediction. The last column overlaps the point clouds from 0.5s prediction and is prediction to make clear how vehicles are moving.
27

---

Published as a conference paper at ICLR 2024
Oureett Ocservtion
To Futu
Futuro
Bo Future
Ground Trulh
Gumert Oroemoaiio
Prediction
2o Prodiction
So Prediction
Previous SOTA
(4D-Ooc)
Tokenizer
Prediction
Prediction
Curs
Cumeet Observation
Future
Future
Future
Ground Trulh
Ourent Oeservatio
Prediction
Prediction
Previous SOTA (4D-Ooo)
Reconstructin
Ours
Cummnt Okoervatio
Fulure OT
Za Future GT
3e Fulure OT
Ground Truth
876
Cument Ubseevvtion
Prediction
Pred chto
Prediction
Previous SOTA (4D-Oco)
Reconstruction
Ours
Figure 14: Qualitative comparison on NuScenes 3s prediction. The orange circles highlight where our model does well; the red arrows point out some failure cases of our model.
28

---

Published as a conference paper at ICLR 2024
Cureen Obsenvatoo
GT
Ground Trulh
Previous SOTA (4D-Ooc)
Tokenizer Reco rsoruction
1.8a Fulure GT
2.4 Ruture GT
35 Preciction
Ours
Observation
2.4 Future GT
Future
Ground Truth
Currnnt Obser usion
1.8s Future GT
2.4 Futre or
Prediction
Previous SOTA
(4D-Ooc)
Tokennze zer Reconstructio
1.8s Future GT
2.4 Future GT
To Prediction
Ours
Ground Truth
Cumrent Ubscnvation
1,8s Puture GT
2:45 Future GT
Prediction
Previcus SOTA (4D-Ooo)
Tokentner Reconstracionn
1.8s Puture GIT
2.4 Future GT
Th Prediction
Ours
Figure 15: Qualitative comparison on KITTI Odometry. Note that the color of point merely dependent its height (z-axis value); therefore. if the colors of the predicted points different from the colors of the ground-truth points, means that the predicted point heights are off. Our method clearly achieves significantly better qualitative results compared to prior SOTA.
29

---

Published as a conference paper at ICLR 2024
Oursent Ocseraaioo
Oh Futurn GT
to Futuro
Bo Future
Ground Truth
Gument Ocoomaation
de Predicion
Prodiction
So Prediction
Previous SOTA (4D-Ooc)
Prediction
Prediction
Curs
Current Observation
Future
Ground Truth
Oeservatio
Prediction
Prediction
Previous SOTA
(4D-Ooo)
Ours
Gummnt Oksorvvtion
de Future CT
Future OT
3e Fulure OT
Ground Truth
Dbseenation
Precictin
Phed
Prediction
Previous SOTA (4D-Oco)
Roconstruction
Ours
Figure 16: Additional qualitative comparison Argoverse2 Lidar dataset. The orange circles highlight where our model does well; the red arrows point out some failure cases our model.
30

---

Published as a conference paper at ICLR 2024
Ossenatiio
0.5a Future GT
to Futuro
3o Future
Ground Truth
Gumnet Ocownotion
0 Prediction
Predictio
So Prediction
Previous SOTA
(4D-Ooc)
Reco conructnttion
Preciction
Prediction
Prediction
Curs
HoRe 6 COm
Current Observation
Futur
Ground Trulh
Oeservatio
Prediction
Prediction
Previous SOTA
(4D-Ooo)
Reconstructon
Ours
Gummet Oboervttion
Be Futur CT
Fulure OT
3e Fulure OT
Ground Truth
Cumeen Ubseevation
Preciation
Phed ctton
Prodiction
Previous SOTA (4D-Oco)
Prediction
Ours
Figure 17: Additional qualitative comparison Argoverse2 Lidar dataset. The orange circles highlight where our model does well; the red arrows point out some failure cases our model.
31


Reviews:
[{"review_id": "rArc0WbxO5", "reviewer": "Reviewer_2hr7", "summary": "The work presents a groundbreaking technique for learning world models in an unsupervised manner, with a particular application to autonomous driving. It addresses the complexity of interpreting unstructured sensor data by implementing a Vector Quantized Variational AutoEncoder (VQ-VAE) to tokenize this data, followed by the prediction of future states through a discrete diffusion process. This technique modifies the Masked Generative Image Transformer (MaskGIT) into a discrete diffusion model, which leads to a substantial increase in prediction accuracy. The proposed approach stands out for its ability to tokenize sensor inputs and utilize a spatio-temporal Transformer for the efficient decoding of future states, which has demonstrated an improvement in prediction accuracy over existing methods on autonomous driving datasets. The model achieves a significant reduction in prediction errors and also shows competence in generating both precise short-term forecasts and diverse long-term predictions, thereby holding great promise for the application of GPT-like learning paradigms in robotics.", "questions": "How is the result if we predict much further, like 9s? In other dataset, like WOMD, the prediction horizon tends to be slightly longer so it would be great to know if the performance would drop significantly if we predict much further away states. Secondly, could you also share some insights on how this model could exactly be integrated within modern self-driving systems and work with other modules such as planning? How would noise in perception, like Lidar affect the prediction accuracy?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper introduces a novel approach by combining VQ-VAE tokenization with a discrete diffusion process, which is kind of innovating. The idea of simplifying the observation space and tokenizing the observation space makes it much easier to model the complex observation space that are usually the case for self-driving.The proposed method's improvement is demonstrated through rigorous experimental validation, showing significant improvements in prediction accuracy over existing methods.The reduction in Chamfer distance for both short-term and long-term predictions indicates a high-quality advancement in the field of point cloud predictions. The paper is also well-structured, with a clear exposition of the methodology, which includes tokenization of sensor data and the subsequent prediction process.", "weaknesses": "The paper mostly addresses the prediction of near term future states but it is not clear if we go much further, how would the accuracy be? With a diffusion model, the inference could be slow, so this model may not be suitable for use on board but mostly would be useful for simulations and other tasks that don't require real time feedback or predictions. This may limit the application of this approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g71pJI9SrH", "reviewer": "Reviewer_9NMX", "summary": "The paper presents a state-of-the-art world model for driving data. Among several major contributions, the paper describes a new way to tokenize point clouds using a VQVAE combined with a PointNet; the paper also proposes a combination of generative masked modeling and discrete diffusion for learning a world model. The proposed method is tested on three commonly used lidar datasets and is shown to achieve state-of-the-art on 1s and 3s time horizon prediction.", "questions": "1. “We hope that future work will combine our world modeling approach with model-based reinforcement learning to improve the decision making capabilities of autonomous agents.” – Are you planning to release your code?\n2. What hardware is required to train your model?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper proposes a tokenizer for point clouds, which could have major applications across robotics.\n2. The combination of MaskGIT with discrete diffusion and classifier-free guidance is novel. The idea of both decoding and denoising tokens is very interesting.\n3. The proposed model outperforms prior state-of-the-art by a large margin.\n4. The methods section is clear even though it proposes several novel models and losses.", "weaknesses": "The unnumbered first equation in Section 3 should be explained better.\n\nMinor:\n* It is not fully clear to me what “SE(3) ego poses” mean.\n* Figure 1 and 5 might be easier to read if you zoom in on the circled areas.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1TgRv4Osxm", "reviewer": "Reviewer_2kSt", "summary": "The paper proposes to use a VQ-VAE+diffusion approach (similar to certain image diffusion pipelines) for the task of learning point-cloud world models for autonomous driving. They design task-specific encoder and decoder architectures to encode point-cloud observations as a sequence of discrete tokens, apply an interleaved spatial-temporal transformer and a discrete diffusion model to predict discrete codes for future frames, and decode with a model based on neural occupancy representations. The proposed model compares favorably to SOTA baselines for the task on standard metrics.", "questions": "- It would be interesting to conduct a more thorough analysis of the modifications to MaskGIT, perhaps on a simpler discrete diffusion problem.\n - Why is L1 median reported inside the ROI but L1 mean reported for the full scene?\n - How important are the novel task-specific encoder and decoder (including the rendering and reconstruction losses on the encoder) to the final performance?\n - It's mentioned that the model is relatively small; how much do the results change when scaling the model (up/down)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed approach gives strong empirical performance. The architecture takes advantage of structure in the problem at several points in useful and interesting ways (in particular the combination of localized neural occupancy and BEV tokenization is quite interesting, and seems novel). I think the backbone (transformer+discrete diffusion) is comparatively less novel, but this is the first time I've seen it applied to the autonomous driving setting and it is interesting to see that it still gives strong performance.\n\nSeparately the authors propose several improvements to MaskGIT. These modifications seem to be crucial for the performance of their algorithm, but it would be interesting to see if these improvements generalize to the original image-based setting or to other discrete diffusion settings (though doing this on anything other than a toy problem would probably be outside of the scope of this paper).\n\nThe authors also identify an issue with standard point-cloud prediction metrics and propose a simple modification. Though this part may be less relevant to the ICLR community, it is an important observation for the self-driving community and should not be overlooked.", "weaknesses": "The proposed architecture is highly specific to point-cloud occupancy prediction (the novelty lies largely in the encoder and decoder, which are task-specific architectures).\n\nThe introduction/related work are somewhat intermixed, which is fine, but leads to confusing presentation. It's not entirely clear from the introduction/related work how the proposed method relates to MaskGIT, and although MaskGIT is heavily referenced it is never clearly described. Background of discrete diffusion could also be better described.\n\nAblations of differences to MaskGIT are good, but it would be useful to also present ablations of the other task-specific model components (encoder/decoder, and maybe also the BEV token grouping?) as compared to general-purpose versions.\n\nMinor note: point cloud prediction visualizations are a little difficult to parse - I'm not sure how they could be improved but it's quite difficult to analyze results/see what's changed between two different images, both in Fig. 1 and Fig. 5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4PwVINe278", "reviewer": "Reviewer_amws", "summary": "The study focuses on the development of unsupervised world models to enhance an autonomous agent's understanding of its environment. Though world models are a form of sequence modeling, their adoption in robotic applications like autonomous driving hasn't scaled as rapidly as language models such as GPT. Two primary challenges identified are the complex nature of observation spaces and the need for a scalable generative model. To address these, the research proposes a novel approach: (1) Tokenization of Sensor Observations. (2) Prediction using Discrete Diffusion. Applying this method to point cloud observations (which play a crucial role in autonomous driving) showed a significant improvement. The proposed model reduced the Chamfer distance by more than 65% for a 1-second prediction and over 50% for a 3-second prediction on major datasets like NuScenes, KITTI Odometry, and Argoverse2.", "questions": "1. Could the authors elaborate more on “using past agent history as CFG conditioning improves world modeling”. What agent history is used here and how is it used? Does it introduce additional knowledge and make the comparison unfair?\n\n2. Since the proposed method introduces a world model, could the authors demonstrate some qualitative examples of different future predictions with different actions? A longer horizon could be helpful to check the consistency of generated frames. I wonder how diverse the future prediction is and how accurate the prediction matches the given action.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The contribution is clear and important to the autonomous driving society. Developing a driving world model is recognized as a critical step for scene understanding and decision-making.\n\n2. This paper is well-written and easy to follow. The figures are intuitive and informative. \n\n3. The experimental results are surprisingly good, which improves a lot over existing SOTA methods.", "weaknesses": "1. The metrics for evaluating the performance of the world model may not be reasonable enough. For point cloud, most of the points describe the background, which is usually static and irrelevant to the downstream task. The prediction of the motion of dynamic objects is more important. Maybe the author can also report the comparison results for dynamic objects or show the advantage of using such a model for some downstream tasks.\n\n2. The conclusion says “One particularly exciting aspect of our approach is that it is broadly applicable to many domains. We hope that future work will combine our world modeling approach with model-based reinforcement learning to improve the decision-making capabilities of autonomous agents.”  It is unclear to me what the advantage of using such a world model is for MBRL, for example, compared to object segmentation and tracking pipelines.\n\n3. There are many complex modules in the pipeline, including VQ-VAE, diffusion model, neural feature grid, and transformer. Not sure if it is easy to reproduce the results and extend it to other datasets or tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F899EMDr1f", "reviewer": "Reviewer_4daR", "summary": "This paper proposes a point cloud forecasting-based world model for autonomous driving. The model first tokenizes point clouds into discrete BEV tokens (codebook/vocabulary) following VQVAE and UltraLiDAR (Xiong et al., 2023). Then tokens are decoded to reconstruct the point clouds with an implicit representation depth rendering branch and a classical coarse voxel reconstruction branch. MaskGIT is further leveraged to a discrete diffusion model, with different masking conditions and history information (the classifier-free diffuision) guidance applied, to realize the future prediction ability. The point cloud forecasting method is evaluated on three datasets and achieves state-of-the-art results.", "questions": "- In the introduction, the task definition of point cloud forecasting is 'to predict future point cloud observations given past observations and future ego vehicle poses'. In which paper, the future ego vehicle pose is provided?\n- Why not report full results under all metrics in ablation study tables?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The presentation of the paper is good, especially in the methodology part. Symbols and figures are clear and helpful for understanding.\n- The model architecture is detailed in the appendix. The authors also provide details besides the model structure, such as the K-means clustering strategy to solve codebook collapse and LayerNorm to stabilize training, which are valuable empirical findings for future research.\n- MaskGIT with diffusion is interesting. It could probably be applied to other tasks as well.", "weaknesses": "- The reviewer is confused about the motivation of discrete tokenization and masked image modeling.\n  -  The proposed method adopts a VQVAE-like model to capture the complex 3D world, as mentioned in the introduction challenge (i). Classic BEV (the method in BEVFusion, BEVFormer, etc.,) can also realize this ability IMO. This undermines the motivation to use discrete tokenization and the necessity to use a discrete diffusion model in the after.\n  -  Table 4 presents the ablation of the discrete diffusion algorithm. The motivation to use MaskGIT seems its parallel decoding strategy. How about the masked image modeling? What will the results or inference time be like if no masked image modeling (or even MaskGIT) is applied?\n  -  The intuition of using different masking strategies for world model training comes from the robotics field. It would be valuable if ablations on this could be presented as well.\n  -  In light of the above points, a naive baseline should be simple diffusion modeling with simple BEV features.\n- The gain of point cloud forecasting mostly comes from CFG, which involves past poses and actions. Without this, the results are close to 4D-Occ. \n  - Taking the current action as input is reasonable for a world model, but much information about history could make the prediction rely heavily on the past. If the prediction horizon is longer, such a long history is also weird as poses at the very beginning are intuitively unhelpful. \n  - As this task implicitly involves ego-planning, this could lead to causal confusion though impressive results are obtained under the open-loop scenario. The authors have stated that combining the world modeling approach with model-based RL is a future direction, yet, it is important to demonstrate its effectiveness for the decision-making task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)