Title: Towards foundational LiDAR world models with efficient latent flow matching
Venue: NeurIPS 2025
Accepted: yes
Keywords: LiDAR World Models

Generative models

Domain Adaption
Abstract: LiDAR-based world models offer more structured and geometry-aware representations than their image-based counterparts. However, existing LiDAR world models are narrowly trained; each model excels only in the domain for which it was built. This raises a critical question: can we develop LiDAR world models that exhibit strong transferability across multiple domains? To answer this, we conduct the first systematic domain transfer study across three demanding scenarios: (i) outdoor to indoor generalization, (ii) sparse- to dense-beam adaptation, and (iii) non-semantic to semantic transfer. Given different amounts of fine-tuning data, our experiments show that a single pretrained model can achieve up to 11\% absolute improvement (83\% relative) over training from scratch and outperforms training from scratch in 30/36 of our comparisons. This transferability significantly reduces the reliance on manually annotated data for semantic occupancy forecasting: our method exceeds previous baselines with only 5\% of the labeled training data of prior work. We also observed inefficiencies of current generative-model-based LiDAR world models, mainly through their under-compression of LiDAR data and inefficient training objectives. To address these issues, we propose a latent conditional flow matching (CFM)-based framework that achieves state-of-the-art reconstruction accuracy using only half the training data and a compression ratio 6 times higher than that of prior methods. Our model also achieves SOTA performance on semantic occupancy forecasting while being 1.98x-23x more computationally efficient (a 1.1x-3.9x FPS speedup) than previous methods.

Towards foundational LiDAR world models with efficient latent flow matching

Abstract

LiDAR-based world models offer more structured and geometry-aware representations than their image-based counterparts. However, existing LiDAR world models are narrowly trained; each model excels only in the domain for which it was built. This raises a critical question: can we develop LiDAR world models that exhibit strong transferability across multiple domains? To answer this, we conduct the first systematic domain transfer study across three demanding scenarios: (i) outdoor to indoor generalization, (ii) sparse- to dense-beam adaptation, and (iii) non-semantic to semantic transfer. Given different amounts of fine-tuning data, our experiments show that a single pretrained model can achieve up to 11% absolute improvement (83% relative) over training from scratch and outperforms training from scratch in 30/36 of our comparisons. This transferability significantly reduces the reliance on manually annotated data for semantic occupancy forecasting: our method exceeds previous baselines with only 5% of the labeled training data of prior work. We also observed inefficiencies of current generative-model-based LiDAR world models, mainly through their under-compression of LiDAR data and inefficient training objectives. To address these issues, we propose a latent conditional flow matching (CFM)-based framework that achieves state-of-the-art reconstruction accuracy using only half the training data and a compression ratio 6 times higher than that of prior methods. Our model also achieves SOTA performance on semantic occupancy forecasting while being 1.98x-23x more computationally efficient (a 1.1x-3.9x FPS speedup) than previous methods. Our project page contains additional visualizations and released code.

1 Introduction

World models enable agents to implicitly learn the dynamics of the environment by predicting future sensory observations, typically through generative models operating in a latent space [40, 33]. Starting from classic motion prediction tasks like optical flow [29, 31], recent advances have extended motion prediction to RGB-video-based forecasting, i.e., RGB world models, which have demonstrated impressive performance in applications such as autonomous driving [41, 16, 47], robotic navigation [4], and other embodied tasks [7, 54, 28, 55, 21, 1, 3]. These models can generate video sequences conditioned on historical frames and sometimes natural language input. While an image-based world model may suffice for repetitive tasks (e.g., learning robotic arm movements), its utility is limited in complex scenarios that demand geometrically structured information, such as autonomous driving. In these scenarios, the lack of explicit semantic and geometric representations restricts practical application, as it necessitates additional steps to extract spatial information.

In contrast to RGB images, LiDAR provides rich geometric structure and implicitly offers a sparse representation for semantic cues about the environment. Unlike dense, pixel-based observations, a

Different beam LiDAR

sensor adaptation

Other conditions

Mean IOU

Forecast

42.00

World model

Performance

Adaptation

Evaluation

Sparse

Dense

33.60

Historical latent

Future latent

Indoor LiDAR data

adaptation

1

1

1 Mean mlOU

25.20

16.80

parameters(M)

Inference

compression

LiDAR

35.00

28.00

21,00

14.00

7.00

8.40

1000

200

50

40

5

Future

Frames

10000

10

1500

Historical

Frames

Semantic Occupancy

World model

15

750

600

video sequences

25

200

FPS

GFLOPs/ frame

autonomous vehicles

LiDAR points from

DynamicCity

OccWorld

Ours: Sem occ w/ future traj

Ours: Sem occ w/ hist traj

DOME* (future traj)

Down-stream tasks fine-turning

Occupancy forecasting pretrain

3D object in a LiDAR point cloud is represented as a cluster of 3D points, encoding essential spatial and semantic information in a compact form. Despite these advantages, foundational world models of LiDAR data remain relatively underexplored: prior LiDAR world models were built for specific domains [56, 20, 57]. In comparison, recent advances in 'foundation' models, e.g. pretraining with RGB image forecasting [3] and trajectory forecasting [60] have demonstrated the effectiveness of pretraining in improving performance on downstream tasks. This motivates our central question: Can we develop a foundational LiDAR world model for ground vehicles that yields downstream performance gains on diverse forecasting tasks, particularly those suffering from data limitations, after fine-tuning? This goal is partly motivated by the fact that many of the causal factors that govern the motion of objects are shared across different domains and environments-many aspects of the dynamics of the world should, in principle, be transferable and can be observed in unlabeled LiDAR.

We investigate pretraining and fine-tuning a LiDAR world model across three diverse transfer tasks (Fig 1): varying-beam occupancy forecasting, indoor occupancy forecasting, and semantic occupancy forecasting. First, LiDAR hardware varies in beam count and scan patterns, often degrading model generalization [56, 39]. We consider this cross-sensor setting to assess robustness to hardware variations. Second, robots equipped with LiDAR sensors often operate in vastly different environments [50, 59, 15], from outdoor to indoor settings. While cross-domain adaptation has been extensively studied in perception tasks [24, 34, 30], it remains underexplored in the context of dynamic learning. Given that LiDAR data can be scarce in certain environments (e.g., indoor), it is promising to investigate whether dynamic knowledge learned in data-rich, long-range domains can be transferred to those with limited data and different operational ranges. Lastly, tasks like semantic occupancy forecasting [6, 14, 23, 58]) rely heavily on costly semantic labels [46, 44]. Our goal is to leverage large-scale unlabeled data to learn a universal 3D dynamics prior for ground vehicles, enabling strong semantic forecasting via fine-tuning on minimal labeled data.

Extensive experiments demonstrate that our LiDAR world model significantly improves the convergence speed of downstream tasks. For all mentioned tasks, we observed relative performance gains with varying amounts of fine-tuning data. In particular, for semantic occupancy forecasting, this scheme of learning dynamics prior to semantics patterns allows us to achieve superior performance to OccWorld [58] with only 5% of its required labeled data. Furthermore, our results highlight the importance of representation alignment during fine-tuning. Although the data compression structure we propose later is data-efficient, we found that either using the pretrained data compressor

directly or retraining the V AE from scratch with the fine-tuning data leads to suboptimal performance. This phenomenon is attributable to a feature space mismatch: in both scenarios, the encoder learns a feature space mapping that diverges from the pretrained one, which in turn degrades the performance of the flow model pretrained on the original feature space. We identify two superior strategies: fine-tuning the V AE, and for tasks where the V AE cannot be fine-tuned directly, applying a cosine-similarity-based alignment loss.

Wealso find that the current architectural paradigms used in LiDAR world models [38, 12] suffer from 2 issues: redundant model parameters and excessive training time . Regarding the former, the latent representation tends to retain a large number of channels, which significantly increases the parameter count of the dynamics model. Regarding the latter, state-of-the-art models often require thousands of epochs to converge. This inefficiency stems not only from the model scale but also from the inherently slow and compute-intensive nature of denoising diffusion paradigms-particularly those combining DDPM-based training with DDIM-style sampling. These challenges have significantly hindered our exploration of LiDAR world model transferability.

To address these issues, we first propose a Swin Transformer-based VAE architecture for LiDAR data compression. This architecture achieves a compression ratio of 192x-over 6x higher than previous state-of-the-art methods-while matching or exceeding their reconstruction quality. We also propose an efficient flow-matching-based generative model. Compared to previous latent diffusionbased or transformer-based deterministic schemes, our approach requires only 4.38% and 28.91% of the FLOPs that they require, respectively. This efficiency significantly accelerates our analysis of transferability in downstream tasks. In summary, our contributions are:

The first study on building transferable LiDAR world models: world models of LiDAR videos that exhibit substantial transferability to downstream forecasting tasks . We show the efficacy of LiDAR world models to 3 diverse fine-tuning tasks: semantic occupancy forecasting, indoor occupancy forecasting, and high-beams occupancy forecasting, and confirm that it outperforms the baseline of training from scratch on the fine-tuning data, and that the relative performance gain is more pronounced with less fine-tuning data.

An approach to substantially reduce reliance on human labels for semantic occupancy forecasting: our method exceeds the performance of previous methods with only 5% of the human labels.

Efficient V AE-baesd architectures for data compression and voxel-based LiDAR world modeling. The former achieves the SOTA reconstruction accuracy at a 6x improvement in compression rate over prior work. Using these encodings, our world model achieves SOTA performance with only 4.47% to 50.23% of the FLOPs and 1.1 to 3.9 times higher FPS.

2 Related Work

We categorize previous work by LiDAR-based world modeling for geometric future prediction, semantic 4D occupancy forecasting for semantic-aware scene understanding, and foundational world models (FWMs) that aim to generalize dynamic knowledge across domains and tasks.

LiDAR-based World Models. Distinct from general LiDAR generation tasks, LiDAR-based world modeling (also known as LiDAR/Occupancy Forecasting) aims to forecast future sensor observations based on past observation. Occ4D [20] and Occ4cast [27] propose forecasting future LiDAR points or occupancy grids via a differentiable occupancy-to-points module, yet without explicitly modeling latent transition dynamics. UNO [2] further introduces occupancy fields within a NeRF-like [35] framework to enhance forecast fidelity. S2Net [49] adopts a pyramid-LSTM architecture to predict future latents extracted by a variational RNN, while PCPNet [32] leverages range-view semantic maps and a transformer backbone to improve real-time inference performance. Although numerous works [37, 39, 51, 36, 19] have introduced increasingly powerful diffusion-based models, for general data generation, progress in LiDAR forecasting remains comparatively limited compare to the RGBbase ones: Copilot4D [56] achieved state-of-the-art performance in LiDAR forecasting by adopting a MaskGiT-based latent diffusion model with a carefully-designed temporal modeling objective. BEVWorld [57] extended this approach by incorporating multi-modal sensor inputs, enabling future LiDAR prediction even in the absence of current LiDAR frames. However, few works have addressed the transferability of dynamics learning across domains-an appealing path to enhance the widespread deployment of world models in real-world autonomous systems.

Semantic 4D Occupancy Forecasting : The RGB video-based world models tried to forecast physical consistent video from RGB input. However, these representations lack of geometric and explicit semantic annotation, the usage of the generated forecasts observations in control task still need extra component to recover depth and predict semantics. Semantic 4D occupancy forecasting aims to address this gap by predicting future semantic (LiDAR) occupancy maps based on past observations-either ground-truth annotations or model-generated results. OccWorld [58] employed an auto-regressive transformer to jointly forecast future latent states and corresponding axis offsets. OccSora [45] further advanced the field by being the first to generate 25 seconds semantic videos conditioned on 512x compressed inputs. Later, DynamicCity [6] improved upon this by decomposing the 3D representation into a more compact HexPlane[9] structure, enabling faster inference. Building upon future trajectories/bev layout and extended datasets, DOME [14] and uniScenes [23] continue to push the forecasting accuracy to new state-of-the-art levels. Importantly, these methods rely on labeled semantic ground-truth data for training, making them dependent on expensive human annotations and thus challenging to scale.

Foundational world model (FWM). We call a world model 'foundational' if it leads to performance gains over learning from scratch on multiple downstream forecasting tasks. In autonomous driving, GAIA-1 [16] and GAIA-2 [41] introduced image-based FWMs that can generate controllable driving scenarios. In the field of indoor robotics, like navigation, NWM [4] proposed an RGB image-based navigation method with a conditional DiT structure. Cosmos [1] takes this ambition further, aiming to generalize across both indoor and outdoor environments. However, due to limitations in the RGB modality, existing FWMs do not provide explicit depth information, which makes it harder to define prediction and planning modules to utilize their output.

3 Efficient latent conditional flow matching

In this section, we first introduce our novel point cloud data compressor in Sec.3.1, which achieves state-of-the-art performance under high compression ratio. Based on the compact representation from it, Sec.3.2 presents the flow matching-based generative forecasting model that serves as the testbed for all subsequent fine-tuning approaches. In Sec.3.3, we introduce the V AE fine-tuning for better representation alignment, which will benefit the final forecast performance.

3.1 Data compression

LiDAR

[H, W, 128]

Positional

sweeps1

embedding

Height&

Semantic

ResConv

embedding

Arbitrary lidar representation

Channel dim

→

2D BEV Features

Patchify

Voxel

Map: 1

0y

Conv&Swish

Activation

Layer Norm

Conv

ResConv

X N

DownSample

ResConv

Transformer

Channel-

Block

wise split

Align&

[H/2N, H/2N, 512]

Densification

x N

x M

Channel reduce neck

Embedding

Reparameterized

N(0, 1)

Parameters

Dense

Occupancy!

sampling

Map:0d

[H, W, 288]

β

ResConv

ResConv

bilinear

interpolation

Similarity map

Semantic annotation

Transformer

Block

Conv&

MLP

Differentiable ray rendering

× N

x M

Occupancy!

Semantic

Channel recover

× N

Map:0s

Recovered

LiDAR points

While SD3 [12] provides a strong baseline for compressing image-based RGB signals, LiDARspecific data compression remains underexplored. Existing LiDAR world models typically adopt the SD3 encoder-decoder architecture with minimal modifications, retraining it directly on LiDAR BEV representations. As an alternative, we propose a Swin Transformer-based architecture and demonstrate that it achieves state-of-the-art performance across all forms of LiDAR compression, as detailed in Table 2. As noted in previous works [23], discrete coding-based compression is not only prone to problems such as codebook collapse but is also generally inefficient in compression. Here we have also used continuous coding to achieve a higher compression ratio as shown in Figure 2.

Data processing and encoding : In this work, bold notation indicates a multidimensional array (e.g., a vector or a matrix). Given a raw LiDAR sensor scan o l ∈ R L × 3 , the voxelized map o v , dense occupancy map o d , and semantic occupancy map o s can be generated through successive steps of voxelization, densification (i.e., accumulating points from the entire sequence into each frame), and semantic annotation. Our structure is capable of processing any of these representations derived from the raw LiDAR input. Depending on the final reconstruction requirements, voxelization can be either a learning-based approach or a direct binarization process. Taking the compression of o d ∈ R H × W × D × C (a H,W,D 3D grid with C-dim features at each voxel) as an example, a Gaussian encoding distribution q ( z d | o d )= N ( z d ; µ q , σ q ) is constructed from three main stages: embedding, feature learning and downsampling, and channel reduction. Specifically, after applying height and class embeddings, a 2D BEV feature map is obtained, which is subsequently processed by a standard 2D Swin Transformer encoder. Unlike the original Swin Transformer design, which utilizes patch merging for downsampling, we replace this operation with conventional convolutional layers-a modification we empirically find to outperform the original approach. Channel reduction is performed by a lightweight network neck, compressing the feature representation into a 16-dimensional latent space. Samples are drawn using the reparameterization trick z = µ q + σ q ⊙ ϵ , ϵ ∼ N (0 , I ) , and ⊙ denotes the Hadamard product.

Decoding and o l representation recovery. The decoder q (ˆ o d | z d ) is designed to mirror the encoder by starting with the same symmetric 2D block structure. Different from previous methods that used 3D blocks in the decoder to increase temporal feature consistency, we found doing so to have a negative effect in our structure, for both reconstruction and final forecasting result, as shown in the ablation study. Depending on the source representation, the reconstructed ˆ o d can be used to render the points with a differentiable ray rendering module or get the occupancy map by calculating a similarity score with the class embedding.

3.2 Forecasting with Conditional flow matching

With the proposed VAE, we are able to mitigate the parameter redundancy: most of the parameters of the existing methods come from the high dimensionality of the latents. To further make model training more efficient, we present a new structure based on flow matching [26], which we show leads to SOTA performance in both forecasting accuracy and computational efficiency, as shown in Table 1. For a semantic occupancy forecast task, given o t 0 : t 2 s , our VAE encodes continuous frames to z t 0 : t 2 s ∈ R ( t 2 -t 0 ) × H × W × 16 . As t 1 is the middle index of these frames, our objective is to obtain the future latent z t 1 : t 2 s from the historical latent z t 0 : t 1 s using a flow-matching model G θ .

As illustrated in Figure 3, the training objective is to regress a velocity field u θ t . This regression is conditioned on several inputs: a time step t ∈ [0 , 1] representing the progress along a probability path from an initial distribution (standard Gaussian) to the target distribution; the interpolated future latent state x t ; the historical trajectory J t 0 : t 1 ∈ R ( t 1 -t 0 ) × 3 ; and historical observations o t 0 : t 1 s .

Specifically, the noising added follows a linear interpolation method as shown in equation 1, where σ used to balance the scale of noise and latents and ϵ ∼ N (0 , 1) . The z t 0 : t 1 s will be used as a part of condition in training: we concatenate the historical latents and noised future latents along time dimension to get the input of G θ , denoted by z joint ∈ R ( t 2 -t 0 ) × H × W × 32 .

Based on the spatial-temporal DiT structure proposed in previous methods [33, 23, 14], we observe that the convergence speed remains suboptimal. Specifically, given z joint, the initial 3D convolutional layer increases the channel dimension to C ′ , resulting in z ′ joint ∈ R ( t 2 -t 0 ) × H × W × C ′ . The width and height dimensions are then flattened before being fed into the spatial DiT, where multi-head attention

to:t1

DiT-Based Block

DiT-Based Block

Frame-wise VAE

Encoder

DiT-Based

DiT-Based

↑

Embedding

Positional

Block

Flash attention

Block

[2:0

Historical

Neck

framelatentas

↑

condition

Init 3DConv

↑

ResConv

Down/Up-sample

dimension

Time

Rescale

↑

Temporal DiT

Spatial DiT block

XN

XN

t1:t2

zto:t2

Noised future

latent

Fused extra

condition

MLP

ut(xt,t,

Fourier

+

1:t2

Past Trajectory

Embedding

MLP

Jto:t1

(MHA) is applied and normalized using AdaLN. However, applying the same strategy in the temporal DiT would only fuse pixels along the temporal axis, limiting the temporal receptive field to just a single latent pixel(in a block). To compensate, we observed that most of the previous methods like uniScenes [23] and DOME [14] use 14-18 stacked blocks to ensure the temporal consistency, which is also one of the reasons of redundant parameters. While this design remains feasible in Latte [33], which operates on 3D video patches, our per-frame latent representation, making the learning of temporal dependencies more challenging. We found that this issue can be effectively mitigated by simply inserting a 3D convolutional layer with a larger receptive field after the spatial DiT. Furthermore, organizing the network in a UNet-style architecture, as opposed to a single-stride DiT backbone, is shown through later experiments to further enhance the forecast performance and reduce computational load. The training objective was designed following the Rectified Flow [26]:

3.3 Improved fine-tuning with representation alignment

The pretrain of world model can now be start with framework proposed in 3.2 & 3.1 and large amounts of unlabeled data o d in a self-supervised way to learn the unified prior knowledge for environment dynamics. For non-semantic to semantic transfer specifically, we now already have encoder/decoder pair denoted by q d ( z d | o d ) / q d ( ˆ o d | z d ) and forecasting model G d θ . Although we can train the other data compressor q s ( z s | o s ) / q s ( ˆ o s | z s ) and use the corresponding latent z s to fine-tuning pretrained G d θ , we have discovered that the latent spaces of z s and z d are not aligned, which is affects the performance of fine-tuning.

For subtask 1 and 2 in Fig 1, this suboptimality can be tackled by fine-tuning the data compressor first, while in case of subtask 3 (semantic occupancy forecasting), due to the difference in network embedding layer dimensions, it is non-trivial to fine-tune the semantic data based on a V AE pretrained on non-semantic data. Instead, we use the latents of the corresponding dense occupancy o d to guide the formulation of the subspace of the semantic V AE.

Specifically, we add a cosine similarity term in the loss when fine-tuning the V AE, as shown in the last term of Equation 2. z s and d s stand for the latent from a pair o s and o d via training from scratch and pretrained VAE, respectively. L lovasz [5] is another reconstruction term used to optimize IoU.

Therefore, our full pipeline is 3-fold: we first train the data compressor and flow matching model via the easily accessible unlabeled data. Then we use the pretrained data compressor to guide the structure of latent space for subtask's data. Finally, with the aligned representation, we fine-turn the pretrained weights to obtain the final results.

4 Experiments

We designed our experiments to investigate several questions: Q1 : Can we improve compression and performance over existing designs? Q2 : Can we develop a LiDAR world model that exhibits substantial superiority to models trained from scratch on three downstream forecasting tasks (high beam occupancy forecasting, indoor occupancy forecasting, and semantic occupancy forecasting)? Q3 : How well does each fine-tuning variant perform and why?

4.1 Experimental design

We use nuScenes[8] (2Hz) as the pretraining data for the flow matching model. We trained 2 different types of foundational models on it, one was trained based on the original LiDAR sweep ( o v ), totaling 27,000 frames, denoted by G v θ . The other type of model was trained using densified LiDAR frames o d ), totaling 19,000 frames, denoted by G d θ . The latter training set is a subset of the former part and the number of o d equal to o s .

For the beam adaptation subtask, we down-sampled 11 sequences from the KITTI360 raw dataset [25] from 10Hz to 2Hz to match the foundational model setting. Also, we collected an indoor navigation dataset using a Clearpath Jackal equipped with an OSO-128 LiDAR sensor (training set with 23,504 frames and validation set with 9,720 frames). Finally, for the semantic occupancy forecasting, we present twofold experiments: in section 4.2, we follow the official splitting [44] and train the model from scratch. Then in Section 4.3 and Appendix H.1, we only use first half of the training data to pretrain the G d θ as the foundation model, which then fine-tuning on the other half o s ( o s and o d are 1v1 correspondence) to avoid the dynamic knowledge of fine-tuning data to be already seen during pretraining stage. From the next section, we use o ′ s and o ′ d to denote the partial training data. For more details on the model setup, please refer to our appendix.

4.2 Encoder structure exploration and semantic occupancy forecasting evaluation

For model evaluation, Table 2 compare reconstruction results of our Swin-Transformer VAE with previous methods compress data from 8× to 512×. At 32×, we achieve 99.2% mIoU and 97.9% IoU-far above UniScenes's 92.1%/87.0% at the same rate. Even at 192×, we maintain 93.9%/85.8%, surpassing OccWorld and DOME by over 11% mIoU, and at an extreme 768× our model still delivers a 9.7% relative gain despite 1.5× higher compression.

Based on the strong VAE performance, our forecasting model advances the state-ofthe-art of semantic occupancy forecasting while maintaining real-time throughput. As shown in Table 1, our model achieves a one-second mIoU of 33.17%, surpassing the previous SOTA model's 28.69% and OccLLama's 25.05% by 4.48% and 8.12%

respectively and maintaining 21.09% and 15.64% for two and three second prediction which outpaces prior works by at least 2.5% relative performance improvement.

For future-trajectory-conditioned forecasting, our mIoU surpasses DOME by more than 5.5% across all frames. Our semantic occupancy forecasting model runs at 22.22 FPS, requiring only 389.46 GFlops per frame and 30.37 million parameters. Compared to DynamicCity which generates 16 frames altogether, our model uses half of GFlops per frame, 66% of parameters, and 1.1 times higher FPS. In the futuretrajectory-conditioned regime, our model sustains 21.4 FPS without increasing in computational cost and parameter count, whereas DOME utilizes 23 times more flops per frame (our model is 4.38% of this amount), 15 times more parameters, 4 times

slower FPS, and at least 5.5% less absolute performance. Due to the stochastic nature of our proposed CFM model and the limitation of mIoU and IoU in evaluating model's ability to generate diverse but plausible future predictions, we also report the negative log likelihood (NLL) in bits-per-dimension. Our model also present better performance compare with previous SOTA stochastic models.

In order to further measure sample quality, we computed the 3D Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) in Table 4. Our model establishes a new state of the art on both FID and KID. On average, for our model without future trajectories, it reduces FID to just 28.3 % / 48.5 % and KID to 22.7 % / 44.4 % of the scores reported by the deterministic (OccWorld) and stochastic (DOME) models, respectively. See Appendix A.2.2 for more details of these metrics.

Then, benefiting from the low information loss of the data compressor we proposed, it is in fact highly suitable to be used as feature extractor for 4D inception-based metrics (temporal consistency measurement). After we revise the structure with more temporal (4D) module, we retrain the proposed data compressor to eval the FVD score of generated results. In Table 3, to give context for temporal consistency, we include an approach that randomly shuffles the ground truth future ordering and compare the FVD with the correctly ordered GT. This tells us the FVD when the predictions are individually correct but temporally inconsistent. Our model scores of 7.68 and 7.80 are also significantly lower than previous methods.

4.3 LiDAR world model transferability

First, In Table 5, we compare the pretrained model with previous 4D forecasting (occupancy forecasting) methods [53, 20]. Ours exhibits better performance in non-semantic occupancy forecasting, particularly at the 3-second horizon. Then, using the architecture in Section 4.2, we evaluate the performance gains from fine-tuning our world model with varying strategies and data scales across individual tasks and present the results in Figure 4.

For the high-beam LiDAR adaptation task, all configurations based on the pretrained world model outperform from-scratch VAE and CFM training (blue line) across all frames and data fractions. Pretraining CFM alone yields 17.11% and 6.68% gains for 1s and 3s predictions at 10% data. With both components pretrained and fine-tuned, the gains increase to 14.48% and 26%, respectively. This demonstrates that our Swin Transformer-based VAE effectively captures transferable geometric priors for downstream tasks with varying LiDAR characteristics and the effectiveness of the feature alignment. For the indoor LiDAR adaptation task, all pretrained model variants outperform from-

Both from scratch

Bothpretrained(full-finetuning)

---OccWorld full nuSc data

Only CFM pretrained

Both pretrained (LoRAfinetuning r=128,alpha=128)

1s forecasting horizon

2s forecasting horizon

3s forecasting horizon

IOU of Cross Beam Adaptation

18

15

口

14

口

16

15

口

13

18

14

12

13

12

11

32

口

30

口

29

30

■

28

27

28

26

26

25

24

R

32.5

20

14

18

25.95

16

14.94

12

11.09

25.75

口

20.0

200%data

12 -

14

200%data

15.14

10 -

200%data

10.51

181

8

10

a

8

6

10

25

50

100

10

25

50

100

10

25

50

100

Fraction of fine-tuning data (%)

Fraction of fine-tuning data(%)

Fraction of fine-tuning data (%)

scratch training when fine-tuning data is limited (less than 25%), with full-parameter tuning of both VAE and CFM achieving the best results. At 10% data usage, full fine-tuning yields at least a 4.47% gain across all prediction horizons.

However, as data availability increases (e.g., beyond 25% for 1s prediction), training from scratch eventually surpasses all fine-tuning methods. This is likely because the indoor dataset represents a task sufficiently distinct from our pre-training domain, making it easier to learn from scratch without the influence of pre-trained priors. We hypothesize that this result would no longer hold if substantial indoor data were included in the pre-training set. Similar results are also observed in the semantic occupancy forecasting task, where all V AE/CFM combinations based on the pre-trained model outperform training from scratch in mIoU. Notably, with just 10% of the semantic fine-tuning data-which is only 5% of the total data used in prior methods-full-parameter fine-tuning yields

an 82.6%/80.72%/69.70% relative mIoU gain for 1s/2s/3s prediction, surpassing OccWorld's performance [58]. After 50% data usage, the performance of from-scratch training exceeds that of the CFM-only and LoRA CFM fine-tuning methods. Nevertheless, full-parameter VAE+CFM fine-tuning consistently achieves the best performance across all data fractions and prediction horizons.

These results confirm that foundational models can capture transferable, LiDAR-based dynamic priors from unlabeled data to support downstream tasks requiring semantic interpretation. The improvements are especially notable in low-data regimes, demonstrating a reduced dependence on human-labeled samples. Overall, across all tasks, our pre-trained model achieves up to an 11.17% absolute performance gain and outperforms training from scratch in 30 out of 36 comparison points. See the Appendix for detailed numerical values corresponding to Figure 4.

However, a key question remains: why do models that fine-tune both the VAE and CFM modules consistently outperform other settings? In other words, how does fine-tuning the VAE impact the CFM's forecasting accuracy?

To further analyze, in Table 6, we compute the similarity of latent spaces under different data scales using CKA [22] and CKNNA [18]. The similarities are measured between pairwise samples w/w.o semantic information, i.e., o ′ d with o ′ s obtained from fine-tuned VAE or VAE trained only on fine-tuning data. We observe that the effectiveness of V AE fine-tuning is primarily attributed to aligning the latent space of the new domain with that of the original domain, rather than improving reconstruction accuracy. For example, when using 100% of the fine-tuning data o ′ s , the non-pretrained VAE achieves even higher mIoU/IoU than the pretrained one, yet the final performance shows a clear gap-suggesting that preserving latent structure plays a more crucial role. Although paired data is not available for the first two adaptation scenarios, estimating latent similarity in other domains could further support our hypothesis, details are shown in Appendix.

4.4 Ablation study

As mentioned in Section 3.2, we now present a breakdown of the performance improvements contributed by each individual module. As shown in Table 7, we observe consistent gains from CFM, 3D convolution, and CFG components, while using classifier-free guidance gives us the most significant improvement in both IoU and mIoU. We also examine the impact of NFE on FLOPS, FPS, and accuracy.

As shown in Figure 5, taking NFE=10, we get the best forecasting accuracy while maintaining a considerable level of efficiency in terms of run time and GFlops for the semantic occupancy forecasting task. For the experiment of VAE data-efficient and other details, please see the Appendix.

5 Conclusion

In this work, by designing a model that can be efficiently pretrained on large-scale outdoor LiDAR data, we show it can be effectively fine-tuned on diverse downstream tasks. Our approach consistently outperforms training from scratch, especially in low-data regimes. To address inefficiencies in existing models, we introduce a new VAE structure for high-ratio LiDAR compression and a conditional flow matching approach for forecasting. These components enable strong reconstruction performance and improve training and inference efficiency without sacrificing forecasting accuracy. In addition, our VAE fine-tuning strategies further boost performance. While results are encouraging, future work is needed to extend generalization to additional environments, incorporate multi-modal inputs, and integrate with planning and control systems. Overall, our findings suggest that scalable and transferable LiDAR world models are feasible and can significantly reduce reliance on annotated data in practical applications.

Acknowledgments

This research was enabled in part by the Digital Research Alliance of Canada ( alliancecan.ca ).

References

Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575 , 2025.

Ben Agro, Quinlan Sykora, Sergio Casas, Thomas Gilles, and Raquel Urtasun. Uno: Unsupervised occupancy fields for perception and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14487-14496, 2024.

Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, et al. Cosmos-transfer1: Conditional world generation with adaptive multimodal control. arXiv preprint arXiv:2503.14492 , 2025.

Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models. arXiv preprint arXiv:2412.03572 , 2024.

Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-union measure in neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 4413-4421, 2018.

Hengwei Bian, Lingdong Kong, Haozhe Xie, Liang Pan, Yu Qiao, and Ziwei Liu. Dynamiccity: Large-scale lidar generation from dynamic scenes. arXiv preprint arXiv:2410.18084 , 2024.

Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian Ichter, et al. pi 0 : A vision-language-action flow model for general robot control. arXiv preprint arXiv:2410.24164 , 2024.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11621-11631, 2020.

Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 130-141, 2023.

Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations, 2019.

Christopher Choy, JunYoung Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 3075-3084, 2019.

Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning , 2024.

Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In International Conference on Learning Representations (ICLR) , 2019.

Songen Gu, Wei Yin, Bu Jin, Xiaoyang Guo, Junming Wang, Haodong Li, Qian Zhang, and Xiaoxiao Long. Dome: Taming diffusion model into high-fidelity controllable occupancy world model. arXiv preprint arXiv:2410.10429 , 2024.

James R Han, Hugues Thomas, Jian Zhang, Nicholas Rhinehart, and Timothy D Barfoot. Drmpc: Deep residual model predictive control for real-world social navigation. IEEE Robotics and Automation Letters (RA-L) , 2025.

Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprint arXiv:2309.17080 , 2023.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.

Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint arXiv:2405.07987 , 2024.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems , 35:26565-26577, 2022.

Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecasting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model, 2024.

Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited, 2019.

Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified occupancy-centric driving scene generation. arXiv preprint arXiv:2412.05435 , 2024.

Zhuoling Li, Xiaogang Xu, SerNam Lim, and Hengshuang Zhao. Towards unified 3d object detection via algorithm and data unification. arXiv preprint arXiv:2402.18573 , 2024.

Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(3):3292-3310, 2022.

Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.03003 , 2022.

Xinhao Liu, Moonjun Gong, Qi Fang, Haoyu Xie, Yiming Li, Hang Zhao, and Chen Feng. Lidarbased 4d occupancy completion and forecasting. In 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , pages 11102-11109. IEEE, 2024.

Yuecheng Liu, Dafeng Chi, Shiguang Wu, Zhanguang Zhang, Yuzheng Zhuang, Bowen Yang, He Zhu, Lingfeng Zhang, Pengwei Xie, David Gamaliel Arcos Bravo, et al. Omnieva: Embodied versatile planner via task-adaptive 3d-grounded and embodiment-aware reasoning. arXiv preprint arXiv:2509.09332 , 2025.

Libo Long, Xiao Hu, and Jochen Lang. Learning to handle large obstructions in video frame interpolation. In Proceedings of the 32nd ACM International Conference on Multimedia , pages 5221-5229, 2024.

Libo Long, Xiao Hu, and Jochen Lang. Shape and texture: What influences reliable optical flow estimation? In Proceedings of the Computer Vision and Pattern Recognition Conference , pages 27894-27903, 2025.

Libo Long and Jochen Lang. Detail preserving residual feature pyramid modules for optical flow. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pages 2100-2108, 2022.

Zhen Luo, Junyi Ma, Zijie Zhou, and Guangming Xiong. Pcpnet: An efficient and semanticenhanced transformer network for point cloud prediction. IEEE Robotics and Automation Letters , 8(7):4267-4274, 2023.

Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048 , 2024.

Björn Michele, Alexandre Boulch, Tuan-Hung Vu, Gilles Puy, Renaud Marlet, and Nicolas Courty. Train till you drop: Towards stable and robust source-free unsupervised 3d domain adaptation. In European Conference on Computer Vision , pages 1-19. Springer, 2024.

Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

Kazuto Nakashima, Xiaowen Liu, Tomoya Miyawaki, Yumi Iwashita, and Ryo Kurazume. Fast lidar data generation with rectified flows. arXiv preprint arXiv:2412.02241 , 2024.

Lucas Nunes, Rodrigo Marcuzzi, Benedikt Mersch, Jens Behley, and Cyrill Stachniss. Scaling diffusion models to real-world 3d lidar scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14770-14780, 2024.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4195-4205, 2023.

Haoxi Ran, Vitor Guizilini, and Yue Wang. Towards realistic scene generation with lidar diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14738-14748, 2024.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 10684-10695, 2022.

Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. Gaia-2: A controllable multi-view generative world model for autonomous driving. arXiv preprint arXiv:2503.20523 , 2025.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.

Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR) , 2021.

Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems , 36:64318-64330, 2023.

Lening Wang, Wenzhao Zheng, Yilong Ren, Han Jiang, Zhiyong Cui, Haiyang Yu, and Jiwen Lu. Occsora: 4d occupancy generation models as world simulators for autonomous driving. arXiv preprint arXiv:2405.20337 , 2024.

Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yunpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du, Jiwen Lu, and Xingang Wang. Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 17850-17859, 2023.

Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14749-14759, 2024.

Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. Occllama: An occupancy-language-action generative world model for autonomous driving. arXiv preprint arXiv:2409.03272 , 2024.

Xinshuo Weng, Junyu Nan, Kuan-Hui Lee, Rowan McAllister, Adrien Gaidon, Nicholas Rhinehart, and Kris M Kitani. S2net: Stochastic sequential pointcloud forecasting. In European Conference on Computer Vision , pages 549-564. Springer, 2022.

Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00493 , 2023.

Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang, and Raquel Urtasun. Ultralidar: Learning compact representations for lidar completion and generation. arXiv preprint arXiv:2311.01448 , 2023.

Ziyang Yan, Wenzhen Dong, Yihua Shao, Yuhang Lu, Liu Haiyang, Jingwen Liu, Haozhe Wang, Zhe Wang, Yan Wang, Fabio Remondino, et al. Renderworld: World model with self-supervised 3d label. arXiv preprint arXiv:2409.11356 , 2024.

Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14673-14684, 2024.

Lingfeng Zhang, Yuecheng Liu, Zhanguang Zhang, Matin Aghaei, Yaochen Hu, Hongjian Gu, Mohammad Ali Alomrani, David Gamaliel Arcos Bravo, Raika Karimi, Atia Hamidizadeh, et al. Mem2ego: Empowering vision-language models with global-to-ego memory for long-horizon embodied navigation. arXiv preprint arXiv:2502.14254 , 2025.

Lingfeng Zhang, Yuening Wang, Hongjian Gu, Atia Hamidizadeh, Zhanguang Zhang, Yuecheng Liu, Yutong Wang, David Gamaliel Arcos Bravo, Junyi Dong, Shunbo Zhou, et al. Et-planbench: Embodied task-level planning benchmark towards spatial-temporal cognition with foundation models. arXiv preprint arXiv:2410.14682 , 2024.

Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. arXiv preprint arXiv:2311.01017 , 2023.

Yumeng Zhang, Shi Gong, Kaixin Xiong, Xiaoqing Ye, Xiao Tan, Fan Wang, Jizhou Huang, Hua Wu, and Haifeng Wang. Bevworld: A multimodal world model for autonomous driving via unified bev latent space. arXiv preprint arXiv:2407.05679 , 2024.

Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Occworld: Learning a 3d occupancy world model for autonomous driving. In European conference on computer vision , pages 55-72. Springer, 2024.

Lipu Zhou, Daniel Koppel, and Michael Kaess. Lidar slam with plane adjustment for indoor environment. IEEE Robotics and Automation Letters , 6(4):7073-7080, 2021.

Yang Zhou, Hao Shao, Letian Wang, Steven L Waslander, Hongsheng Li, and Yu Liu. Smartpretrain: Model-agnostic and dataset-agnostic representation learning for motion prediction. arXiv preprint arXiv:2410.08669 , 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: Our core innovation: the idea of a 'more efficient and portable world model' has been corroborated by subsequent experiments.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We address our model's limitation when we discuss that sometimes when the amount of fine-tuning data is close to that of the pretraining data, from-scratch training could actually yield slightly better performance then fine-tuning model. We discuss the model's computational effiency of the model when performing Gflops, parameters, and FPS analysis for semantic occupancy forecasting model.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our model is designed based on proved theoretical concepts (i.e. conditional flow matching) and applying them into designing foundational model architecture.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In the paper, we discuss our experiment and model setup including some important selection of hyperparameters and model dimensions. Also, we will release our code and pretrained weights to the foundational model for reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: Yes, we will organize our repo and share the code, data, and weights for the model to be tested and verified. We won't release code during submission time for anonymity.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: we include all the data configurations, model configurations, and hyperparameters such as LoRA rank in the paper. We also provide justification to most important hyperparameter selections in the appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We conducted a sensitivity analysis under sampling randomization.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We mention the computation power required to train the model and run the inference, as well as the total number of parameters as a reference.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All the authors of the paper have read the NeurIPS Code of Ethics and ensure that every protocol/regulations is followed through the entire writing and submission process. Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: in the conclusion, we mention how this world model can be implemented into navigation or planning models or autonomous agents, which ultimately will benefit the society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our model is conditional generative/forecasting model which does not have high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: we have included all the paper that either has been working on similar fields or lays theoretical framework for our research. For the repo, we cite/license all the codes which we got our inspiration from/utilized during the research.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: in this project, we collect an indoor navigation dataset which we include necessary information on the collection and pre-processing process. This should count as a new asset for our project.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our project deals with publicly-accessible LiDAR datasets and personally collected datasets. So we don't deal with human subjects in this research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our model deals with online dataset which does not involve subject studies or anything similar.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLM is not used for any important, original, or non-standard component of the core of this paper.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Appendix

Contents

A Model Setup and Further Evaluation

A.1 Training Setup

For all flow matching based generative (foundational) model training, different from previous methods [14, 23] that used several thousand training epochs, we used 4x RTX 4090 to train models for 200 epochs with a batch size of 8. We trained the V AE part for 100 epochs with a batch size of 16 if not otherwise specified.

Following prior work[56], we also adopt the AdamW as the optimizer with β 1 and β 2 set to 0.9 and 0.99 for flow matching training, 0.99 and 0.999 for VAE training. We set the weight decay of all normalization layers to 0 and all of the other layers' to 0.001. The learning rate schedule has a linear warmup followed by cosine decay (with the minimum of the cosine decay set to be 20% of the peak learning rate). We also use EMA with a 0.9999 decay rate to ensure the updates of parameters are stable. During the training, we found that the classifier free guidance is also important in a flow-matching based framework. We randomly set 25% of historical latents in a batch to 0 during training to let the model learn to generate future latents without any conditions. In the sampling phase, we use a typical fuse method as shown in Eq. 3 to get the final output when it's the t -th step, where s is set to 2.

Finally, we also noticed that the matching of noise scale and latent scale is important. Different from SD3's VAE, the value of latent obtained from our VAE is actually smaller, with a standard deviation of about 0.02, which makes it easy to drown the signal in noise if we use standard Gaussian noise in the training phase. Therefore, we scaled the compressed latent up by a factor of 10 in the implementation.

A.2 Evaluation metrics

In this section, we will provide the details of all metrics used to further evaluate the performance of our proposed pipeline. All results will be presented in the last part of this section.

A.2.1 Latent space similarity metrics

Given o d ∈ R H × W × D × C and its compressed latent z ∈ R h × w × c , the compression ratio γ can be calculated as in Eq. 4.

In Section 4.3, we introduce CKA (Centered Kernel Alignment) [22] and CKNNA(Centered Kernel Nearest-Neighbor Alignment )[18] to evaluate the similarity of latent spaces as both metrics are insensitive to linear transformations. Specifically, we treat each latent pixel (location) as an individual sample (i.e., every sample in CKA/CKNNA evaluation can be represented as z i ∈ R 1 × c ). For the non-semantic to semantic adaptation subtask, given k samples from e latent, the representation metrics from o d and o s can be denoted by X ∈ R u × c and Y ∈ R u × c respectively. We first construct the kernel matrix:

Then the CKA can be calculated as:

where K c and L c are centered K and L , using K c = HKH , H = I n -1 n 1 n 1 T n . 1 n is an all-ones column vector. CKNNA is a revised version of CKA that focuses on the local manifold similarity.

For every sample i in K c and L c , we use cosine similarity to find out the nearest k neighbor in original samples, let knn K c and knn L c be the indices of its k nearest neighbors, respectively. Then we can define a mutual-KNN mask α ( i, j ) by:

̸

Local alignment is Align KNN ( K c , L c ) = ∑ u i =1 ∑ u j =1 α ( i, j ) K c ( i, j ) L c ( i, j ) . CKNNA is:

Use k to represent the number of neighbors we selected, when k → u, α ij = 1 for all off-diagonal pairs and CKNNA reduces to the standard CKA.

A.2.2 Inception-based metrics

Fidelity measurement Following [6], we report both the 3-D Fréchet Inception Distance (FID) and the Kernel Inception Distance (KID, i.e., the squared Maximum Mean Discrepancy in feature space). Unlike FID-which assumes the Inception features form a single multivariate Gaussian and thus compares only the first two moments-KID employs a characteristic RBF kernel and provides an unbiased estimate that is sensitive to discrepancies in all higher-order statistics.

To obtain latent features from generated and ground-truth samples under comparable conditions, we retrain an autoencoder on o s . The autoencoder is based on the MinkowskiUNet32 architecture (sparse)[11], but we adapt it to our 200 × 200 × 16 occupancy inputs by reducing the feature channels from {64, 128, 256, 512} with four down-sampling stages to {32, 64, 128} with three down-sampling stages. The training objective is to recover the compressed latent as much as possible under a cross-entropy loss.

For the output of 6 frames of future semantic occupancy, we evaluated frame-wise FID and KID. Specifically, we first extracted all non-empty voxels from the down-sampled semantic occupancy (size 25, 25, 2) and took the average, rather than performing global pooling as in previous work, to avoid the influence of empty voxels on the metrics. Given the flattened feature z g i and z e j from individual samples, we use µ g , µ e and Σ g , Σ e to represent the mean and channel covariance matrix over M ground truth samples and N estimated samples. Then we calculate FID with the following equations.

For KID, we use a RBF-kernel-based unbiased-statistic estimation version as shown in equation 10.

̸

Temporal Consistency Assessment FID and KID provide assessments of single-frame fidelity, whilst for multi-frame continuity we adhere to previous methodology by employing FVD for evaluation. Given that no existing RGB video encoder can capture this 4D temporal information, we modified the proposed VAE to enable it to measure the continuity of 4D occupancy. Within the encoder and decoder shown in Figure 2, we respectively incorporated 3D attention/conv layers to capture temporal information.

A.2.3 Negative log likelihood evaluation

Computation of exact log probability density Due to the stochastic nature of our proposed conditional flow matching model, pairwise-sample-comparison metrics such as IoU and mIoU

̸

provide a limited measure of model quality, since these "deterministic" metrics penalize models for generating diverse but plausible predictions of the future that are substantially different from the recorded future. In other words, the model's ability to capture the uncertainty in future prediction is not measured well by the IoU and mIoU metrics. Therefore, we also evaluate the exact log probability of our CFM model that doesn't penalize it substantially for assigning probability density to other modes.

With our CFM model, G θ , we can compute the log probability of any generated future prediction or future ground-truth samples with the following methods. First, as described by previous works [10], the generative process of a continuous normalizing flow works as described below: starting from a sample from a base distribution z 0 ∈ p z 0 ( z 0 ) and a parametrized ODE G θ = G ( z ( t ) , t ; θ ) which is the flow function, we can obtain z ( t 1 ) from the target distribution by solving for the initial value problem z ( t 0 ) = z 0 , ∂z ( t ) ∂t = G . In this process, the rate of change of log-probability density follows the instantaneous change of variables formula [10]:

With this equation, the total change in log probability density from t 0 to t 1 is calculated by integrating with respect to time:

Given any sample x in the target distribution, we can compute z 0 that generates x and the log likelihood of x by solving for the following IVP [10]:

with z ( t 1 ) = x and log p ( x ) -log p ( t 1 ) = 0 .

After obtaining the change in log probability density, we can add this change to the log probability of the prior distribution to determine the exact log likelihood of x.

Practically, as what is done for FFJORD [13], the trace of the Jacobian of the flow function can be approximated with the Hutchinson's trace estimator which takes o ( n ) . Moreover, we use a standard Gaussian as the base distribution which gives:

where D is the number of scalar dimensions of the sample. This enables us to determine the exact log likelihood of any generated sample or sample from the future occupancy latent space (any sample from the future occupancy distribution) numerically.

Details on NLL evaluation For all log likelihood values computed, we use an Euler ODE solver with a step size of 0.02, relative and absolute tolerance of 1 × 10 -5 . The trace of the Jacobian is obtained by utilizing Hutchinson's trace estimator with the probe vector sampled from a Rademacher distribution.

After obtaining the exact log probability, we evaluate the negative log likelihood in the form of bits-per-dimension (BPD) which is obtained by:

where D is the number of scalar dimensions of the latent sample x and the division by ln 2 converts the unit to bits.

DDPM log likelihood computation For discrete DDPM-based model, we need to construct and solve the IVP introduced previously with a mathematically equivalent representation for the flow function G .

For a model (i.e. DOME [14] which is constructed based on a 1000-step DDPM) whose noise variances { β k } N -1 k =0 are linearly spaced, to reuse the same parameters for likelihood evaluation, we can embed this chain in the piece-wise-constant variance-preserving SDE

We can write its deterministic form [43]

with score s θ ( x t , t ) = -ˆ ε θ ( x t , t ) / σ ( t ) . With this approximation of the flow function, the IVP can be constructed and solved following the same procedure as the CFM model.

A.3 More results on semantic occupancy forecasting task

The NLL values in terms of BPD and the corresponding standard deviation (across the entire validation set; we use ground-truth samples not the generated predictions) obtained from the semantic occupancy forecasting task are shown in table 8. Our semantic occupancy forecasting model conditioned on historical/past trajectory achieves comparable NLL values with DOME [14]: only 0.25% absolute difference (4.1% relative) while DOME utilizes future trajectory in the condition. For our model conditioned on future trajectory, the NLL value is lower than that of DOME by 25% relative difference (1.49% absolute). This shows that our CFM model better fits the true future-occupancy distribution than the previous (stochastic) SOTA model.

In terms of standard deviation in NLL, for our models (conditioned on either history or future trajectory), the standard deviation is at least 86% (relative) less than that of DOME. This statistical metric further supports the conclusion drawn above by demonstrating the high consistency of our model's performance in terms of assigning high probability to correct futures.

Different from image, LiDAR occupancy from a BEV perspective has a clear depth correspondence between grid cells and environments. However, the global pooling operation will take the average of features in all grid cells, agnostic to geometric location. Here, inspired by [39], we also measured the FID and MMD of features from different depth bins. Specifically, in the obtained z e i ∈ R 25 × 25 × 2 , every cell stands for 3 . 2 m × 3 . 2 m area in the environments. Based on the distance from the center point, we divide the area into three zones: within ±8 m, ±8 to ±24 m, and ±24 to ±40 m. We then take the average of the non-empty voxels in each zone and concatenate them to get the final feature for evaluation, denoted as FID r and KID r in Table 9.

Our model establishes a new state of the art on both FID and KID. On average, for our model without future trajectories, it reduces FID to just 28.3%/48.5% and KID to 22.7%/44.4% of the

scores reported by the deterministic (OccWorld) and stochastic (DOME) models, respectively. The advantage remains under the stricter metrics FID r and KID r : our method attains only 23.36%/48.79% (FID r ) and 19.30%/44.25% (KID r ) of the corresponding values.

Interestingly, we observed that the FID r /KID r of OccWorld decreases slightly (around 2%) when we forecast longer-term future results. We attribute this to the auto-regressive displacement forecasting design in OccWorld, which preserves more diversity at longer horizon, although the overall performance is lower. Specifically, we calculated the category variance at different time horizons (all numbers follow in × 10 -3 ): for the ground truth, the variance fluctuated between 3.055 and 3.021, while in Occworld, it rose from 2.865 at t=0.5 seconds to 3.038, and our model fell from 3.086 to 2.821.

B Dataset introduction

As mentioned earlier, we used nuScenes Semantic Occupancy ( o s ), KITTI360 raw sweeps[25], and our own collection of indoor LiDAR data to test the three subtasks. In the semantic occupancy forecasting task, we further divide nuScenes semantic occupancy data into pretrain set ( o ′ s ) and fine-tuning set. For pretraining, we use voxelized LiDAR sweeps in the nuScenes dataset. In Table 10, we list the specific number of frames of all these datasets after down-sampling to 2Hz.

Specifically, for the indoor LiDAR dataset, it consists of 25 sequences that we collected in 2 different scenarios, using an OSO-128 LiDAR sensor with a 90-degrees vertical field of view. A robot, modeled with unicycle kinematics, was manually driven through a university building over the course of five days. An expert human operator controlled the robot, skillfully avoiding pedestrians while navigating toward designated goals. As shown in Figure 6, compared to the data in the autonomous driving scenario, where the vast majority of vehicles move parallel to ego vehicles, the data we collected includes more complex trajectories with more objects (people).

As we mentioned in the main text, the amount of data after downsampling is still large considering that we only collected this data in 2 scenes. This makes it easier for the model to learn domain-specific knowledge and explains why the performance of our pretrained model drops after using more than 50% of the data.

C Detailed pretraining and fine-tuning results

In this section, we explain more details about the results from our experiments on how different pretrained/from-scratch VAE and CFM combinations perform on each of the three subtasks, which, again, are high-beam LiDAR adaptation, indoor occupancy forecasting, and semantic occupancy forecasting. For LoRA[17], across all our downstream tasks, we choose a rank of 128 and α of 128. Please check section E for details on this selection. All models present in this section are trained for 40 epochs.

C.1 High-beam adaptation and indoor occupancy forecasting

In Table 11 and Table 12, we organize the performance of each of the pretrain/from-scratch combinations (VAE pretrained and CFM pretrained and fine-tuned with LoRA[17], both pretrained and full-parameter fine-tuned, only CFM pretrained, and both from-scratch) for high-beam adaptation and indoor occupancy forecasting in terms of IoU. IoU is computed through inferring the trained CFM

model with NFE=10. For the high-beam adaptation task, the pretrained CFM model is trained on original LiDAR sweeps, denoted by G v θ . From the result, it is evident that full-parameter pretrained (both or only-CFM, with or without LoRA) yields exceeding performance over from-scratch trainings. Specifically, at 10% of fine-tuning data usage, our pretrained model with full parameter fine-tuning achieves more than 14.48% relative performance improvement (more than 1.5% absolute performance improvement) across the three-second forecasting horizon and even at 100% data usage, our model still offers an absolute performance growth of about 1%. This demonstrates the effectiveness of our pretrained foundational models in transferring geometric knowledge learned from 32 beam LiDAR data to tasks that demand different geometric properties (64 beams in this case) at different availability of fine-tuning data.

For the indoor occupancy forecasting task, we use the same pretrained CFM model as the high-beam adaptation task. From the result, it is shown that at low data availability for fine-tuning (10% and 25%), our pretrained models achieve exceeding performance over the from-scratch training. But when the amount of fine-tuning data increases to more than 50%, the from-scratch training obtains comparable or even better performance. Again, we explain this performance threshold to be resulting from the fine-tuning data amount which overrides the pretraining data and the lack of variance in the geometry of the indoor occupancy dataset collected.

C.2 Semantic Occupancy Forecasting

For semantic occupancy forecasting task, the pretrained CFM model is trained on densified LiDAR frames, denoted by G d θ . The results with numerical value of mIoU and IoU for different pretrained/from-scratch VAE and CFM combinations (NFE=10) at different fine-tuning data fractions are presented in Table 13. It is shown that at 10% of fine-tuning data utilization, our pretrained VAE and CFM with full-parameter fine-tuning provides a 11.17% absolute forecasting performance improvement in mIoU. In general, even the performance gain decays as the amount of fine-tuning data usage increases (which makes sense as it is getting closer and closer to the amount of pretraining data), the effectiveness of our foundational model prevails across all fine-tuning data fractions (at 100%, we still have about 1% performance improvement over the from-scratch model). One key thing to note is that we only fine-tune/train-from-scratch our semantic occupancy forecasting model with half of the available semantic labels for nuScenes (50% of the data used for the same task for other baseline models). This additional splitting of the training set is to prevent the overlap between pretrain and fine-tuning data in terms of geometric structure, as both densified occupancy forecasting

and semantic occupancy forecasting models share the same set of scenes and the difference is that semantic occupancy forecasting data is labeled by category. This means that at 10% of fine-tuning data usage, having both VAE and CFM full-parameter fine-tuned based on our foundational model is able to achieve comparable performance with training-from-scratch and OccWorld (< ± 1 . 5% absolute performance difference in both IoU and mIoU across the three-second forecasting horizon) with only 5% of the data it consumes. Please refer to section H for visualizations of the forecasting results.

D More details on representation alignment

As we mentioned before, during the pretraining phase, we found that it is more appropriate to use new data in fine-tuning the original V AE as opposed to training a V AE from scratch, even though the performance of our vae trained from scratch was excellent. We attribute this to the fact that the pretrained generative model is in fact more adapted to the data distribution of the original latent, whereas the subspace structure of a latent learned from scratch with fine-tuned data would in fact be very different.

Specifically, for sub-task 3 (nonsemantic-to-semantic transfer), we cannot directly fine-tune the V AE considering the inconsistent size of the embedding layers. Instead, we use the approach in Fig. 7: considering that o s and o d match one-to-one, and the original VAE is fine-tuned on o d , we can directly constrain the direct outputs of the encoder: mu and sigma. Previous work has tried to use bidirectional KL or EMD distance as a loss, but this does not work in our case: the data for the two modalities are already very close, differing only by semantic information.

Therefore we compared the latent from pair-wise o d and o s in line 1 of Table 14. We can see that both Dual-KL and EMD are very close, only cosine similarity is very different, which actually show us that the most significant difference in the latent learned from these two V AEs is in fact the direction of the high-dimensional tensors. Then after using cosine similarity to constrain the o s VAE learning, the result is shown in line 2. With the tensor orientation restriction, we can successfully align the latent of the two V AEs, and the subsequent generative model training demonstrates that our alignment scheme can further improve the results.

q(zaloa)

q(oalza)→ Loss

Pair-wise sample w/w.o

semanticfeatures

Representation

Align

q(zsl0s)

q(oalsa)→Loss

Fine-turningData

PretrainData

VAEfrom

scratch

+Fine-turned

Pretrained

1

1

Pretrain

VAE

Pretrained

q(zalod)

q(zalod)

q(zalod)

VAE

Za

For different beam LiDAR adaptation and outdoor-indoor generalization, while we can directly finetune the V AE, we cannot directly compare the similarity of the subspaces (CKA[22] and CKNNA[18] require the same or pair-wise samples as inputs to different networks). Our solution is shown in Figure 8. First, we use the original pretrained V AE, denoted by q ( z d | o d ) to get the latent of fine-tuning data without any training (adaptation), denoted by z old . Then we use the fine-tuned and training from scratch VAE to obtain the z b and z a . If our hypothesis is correct, the distance between ( z a , z old ) should be smaller than ( z b , z old ) .

As data shown in Table 15 and 16, we listed the ( z a , z old ) and ( z b , z old ) in every 2 continuous lines for different fractions of fine-tuning data. We observe that with the increase of the data amount used to fine-tuning the V AE, we can have better results in terms of IoU, even better (when using 100% high beam KITTI data) or very close to the pretrained one. However, what really determines the forecast performance is actually the similarity between the latent space after fine-tuning the V AE and the latent space after only pretraining. Compared to training from scratch, the latent space after

fine-tuning the V AE is more similar to the latent produced by using pretrained weight directly, which allows for better use of the knowledge from the foundational model. In other words, fine-tuning the VAE is in fact about leaving the original latent space as unaltered as possible while allowing the weights of the original V AE to be adapted (and accurately compressed) to the new sample.

E Analysis on LoRA fine-tuning with different ranks

To determine the optimal LoRA[17] configuration for our downstream tasks, we tested the following ranks: 32, 64, and 128 with α = rank and a dropout rate of 0.05 on the semantic occupancy forecasting subtask with 10% of fine-tuning data and both V AE and CFM components pretrained as explained earlier. We apply LoRA only to the linear layers, embedding layers, 2D and 3D convolution layers in the CFM architecture. The training is done for 40 epochs for LoRA and full-parameter fine-tuning baseline. The goal of this mini-scale experiment is to find the balance between model performance and parameter efficiency. The result is shown in table 17. First, after sampling the CFM, the model yields suboptimal performance under the rank of 32 and 64, with 3.51% and 2.48% absolute performance gaps with the full-parameter fine-tuning in terms of mean mIoU across threesecond forecasting horizon, respectively. This indicates that the model does not acquire a satisfactory level of semantic information from fine-tuning data within the 40 epochs training process which might be caused by lack of depth in the LoRA layers. On the other hand, rank of 128 demonstrates performance that is closest to the full-parameter fine-tuning: 0.25% less in mean IoU and 0.91% less in mean mIoU. This shows that a rank of 128 offers enough depth for information to be extracted from the semantic occupancy data and demonstrates comparable performance against the baseline. This rank and α combination is what we presented earlier for all downstream tasks under the "LoRA" fine-tuning category.

We did not test any higher or lower rank because: since both ranks of 32 and 64 have a considerable performance gap compared to the full-parameter fine-tuning, lower rank means even less parameters (depth) for LoRA layers which makes the model less likely to learn useful information from the finetuning data and consequently, is unlikely to achieve comparable performance with the full-parameter method; on the other hand, a rank of 128 already has 10.21 million trainable parameters in total where our CFM model alone only has 11.51 million parameters. This means that under such rank, we are fine-tuning about the same number of parameters as the full-parameter fine-tuning approach.

F Additional ablation studies

F.1 The data efficiency of proposed VAE structure

Follow the official split of train/validation set in nuScenes, we test the proposed V AE performance under different fractions of training data. As shown in Table 18, our method only needs half of the training data to exceed all of the previous data compressors, as we mentioned in the abstract.

F.2 VAE and forecasting result break down

In Table 19, we present the compression performance of the proposed VAE across different semantic categories. With a 3× compression ratio, our method consistently outperforms prior approaches in all categories. Notably, for rare classes such as Construction Vehicle, Trailer, and Vegetation, our model achieves improvements of 22.2%, 18.8%, and 25.4% over DOME's VAE, respectively. These results indicate that our V AE effectively mitigates the impact of data imbalance across categories in the compression process.

Now, we present a per-category performance analysis. In Table 20, we compare the average IoU for 3s under each category. Compared to OccWorld[58] , our improvement in forecasting performance for medium-sized objects is particularly significant: we improve the forecasts of bicycle, Const. Veh. and Motorcycle by 85.36%, 118.12% and 67.17%, respectively. A broad performance gain is also observed when including future trajectory as a condition, the top 2 improvement categories are also small objects, relatively 65.18% and 51.2% for Motorcycle and Pedestrian respectively.

F.3 Forecasting performance as a function of the VAE's compression ratio

We present the reconstruction rates of the V AE architecture at various compression ratios in Table 2. Then a further question arises: which category of information is more susceptible to loss? In Table

21, we show the performance drop for 4 listed reconstruction rates. Interestingly, we notice that the

voxel missing in the small and rare foreground objects is much less severe than in the background (vegetation, manmade) and large objects (construction vehicle) during compression ratio increase, while these small targets are more important for safe driving.

Next we show the forecast performance as a function of VAE for different compression rates. First, the results of CFM based on latent with different compression ratios are shown below. For time considerations, all CFMs here are trained for 200 epochs. At the highest reconstruction rate of x32,

we find that the network is not fully fitted at 200 epochs, and a longer training time may allow the network to fit better. However, since this is clearly contrary to the 'sample/training efficiency' of this paper, we keep the training time the same for fair evaluation. For latents with higher compression multiples, the forecast performance continues to degrade. We believe that this is not only due to the fact that it retains less information on large objects, but also because at larger compression multiples, latents become very sensitive to noise, i.e., the latent space loses smoothness. We find that at multiples of x384 and x768, noise of the order of 1e-4 can crash the network (the magnitude of the latent quantity is approximately between 1e-2 and 1e-3).

F.4 Model performance with different fractions of pretraining data

In the previous section, we showed that based on the foundational model, the performance boost when we increase the fine-tuning data amount and also the result w/w.o the foundational model. But another question is whether the performance of the model improves with the inclusion of more pretraining data? We designed a simple experiment to answer this question.

30.00

25.00

3s Mean loU

20.00

15.00

10.00

High-beamadaptation

Indooroccupancyforecasting

Semanticoccupancyforecasting

50

09

70

80

06

100

PercentageofPretrainDataUsed(%)

We choose 25% of fine-tuning data for 3 different subtasks to simplify the fine-tuning. In Fig. 9, 100% data on the x-axis represents the full pretraining data O v . We observe a clear performance

improvement (relative improvements of 21.9% , 22.2%, and 12.9% for indoor occupancy forecasting, semantic occupancy prediction, and high-beam adaptation tasks) as the amount of pretraining data varies from 50% of O v to 100%.

F.5 CFMperformance sensitivity analysis with sampling randomization

The inference process of flow matching is essentially solving the Probability Flow ODE (PF-ODE) as mentioned by previous works[42], which is actually deterministic given the sampled Gaussian noise is fixed. Here we designed an experiment which explores the performance change when using different i.i.d. Gaussian noise as input. Specifically, using the Euler solver, we evaluated the semantic occupancy forecasting model with 5 random seeds and recorded the average and standard deviation for IoU and mIoU across the three-second forecasting horizon (on the validation data).

We present the results in Table 23, and note that the standard deviation of the performance metrics for all three models is less than 1% of the mean across the entire forecasting horizon. Therefore, we conclude that our CFM model has low performance sensitivity to randomization in the sampling process and the uncertainty around the performance metrics evaluated is minimal.

G Limitation and future work

Physical consistency. Although our method achieves state-of-the-art performance across different time horizons, the issue of continuous consistency in generated videos remains unresolved. For foreground objects, our current solution only achieves smooth output by performing timing processing within the network structure, but in the generated scenes, foreground objects are still prone to discontinuities. As shown in Table 20, although our work has greatly improved the accuracy of the forecasted foreground objects, there is still no theoretical or design guarantee that the foreground object will be continuous in the output (i.e., it will not suddenly appear or disappear in a few frames).

Multi-modality forecasting. For a safe end-to-end autonomous driving system, it is still very difficult for the current occupancy world model to not only decode the occupancy map after obtaining the future representation, but also forecast the future multi-modal trajectories of surrounding agents. As we shown in Table 23, the randomness in flow matching or diffusion (originates from Gaussian noise), under a given condition, the future forecast outputs are in fact quite fixed. This may be solved via more detailed control for foreground and background generation. We hope that this work will provide an efficient implementation framework for solving this issue in the future and accelerate the progress of subsequent work.

H Visualization

H.1 Samples from the pretrained model

Here we visualize the forecasting results for two of the foundational models that are pretrained. Specifically, visualization for sparse occupancy forecasting is shown in figure 10 and visualization for dense occupancy forecasting is shown in figure 11. Please check more visualizations on our project page.

t=1s

2s

Ours

G

t=2s

t=3s

H.2 Samples from the semantic occupancy forecasting model

3s

s.no

OccWorld

G

manmade

vegetation

terrain

sidewalk

driveablesurface

trafficcone

pedestrian

car

other flat

t=1s

t=2s

t=3s

OccWorld

manmade

vegetation

terrain

sidewalk

driveablesurface

trafficcone

pedestrian

car

other flat

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a foundational LiDAR world model that can be  fine-tuned for various downstream tasks such as semantic occupancy forecasting and adapting to different LiDAR sensor configurations. The authors introduce a Swin Transformer-based VAE architecture to achieve state-of-the-art reconstruction accuracy with higher compression ratios and reduced computational costs. The model demonstrates transferability and reduced reliance on labeled data, achieving good performance with only a small fraction of the training data required by previous methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tThe paper is well-written, with a clear methodology and extensive experimental results that validate its effectiveness.\n\n2.\tThe motivation behind the module design is clearly articulated.\n\n3.\tThe experiments are comprehensive. The authors demonstrate the effectiveness of their design across three challenging scenarios, achieving strong performance.\n\nWeaknesses:\n\n1.\tThe design primarily aims to address issues such as redundant model parameters and prolonged training time. However, it remains unclear why and how the proposed Swin Transformer-based VAE architecture effectively tackles these problems.\n\n2.\tAlthough the paper claims to be the first to introduce a foundational LiDAR world model, its future forecasting component appears to be quite similar to some existing self-supervised point cloud forecasting methods [1][2].\n\n3.\tSome descriptions and illustrations are not clear, and in some cases may be confusing. For instance, z_s^{t_1:t_2} is defined as the prediction target of the model in line 188 page 5, yet it is also shown as an input in Figure 3, which creates ambiguity.\n\n[1] Yang, Z., Chen, L., Sun, Y., & Li, H. (2024). Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14673–14684.\n[2] Khurana, T., Hu, P., Held, D., & Ramanan, D. (2023). Point cloud forecasting as a proxy for 4D occupancy forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1116–1124.\n\nQUESTIONS:\nSee weakness.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the first general-purpose LiDAR world model with key contributions on 1) A Swin-Transformer VAE compresses voxelised LiDAR grids by 192 × while preserving ≥ 93 % mIoU. 2) A Conditional Flow-Matching UNet predicts future latent frames in real time (~22 FPS), needing far fewer FLOPs than diffusion baselines. 3) Pre-training on nuScenes lets the model fine-tune with only 5 % labelled data (for sparse-to-dense completion, outdoor-to-indoor forecasting, and semantic occupancy), beating task-specific SOTA. 4) A cosine-similarity regulariser keeps latent codes aligned when the encoder is replaced, safeguarding transfer.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Originality: The authors are the first to package foundation-model style pre-training, compression, and cross-task transfer into a single LiDAR-only world model. No previous work nails the whole pipeline end-to-end.\n2. Quality: Experiments are thorough: 192× compression still keeps ≥ 93 % mIoU, ablations cleanly isolate the VAE, flow-matching, and latent-alignment tricks, and real-time numbers (≈ 22 FPS) are reported on the same hardware as baselines.\n3. Significance: Needing only 5 % labelled data to beat task-specific SOTA on three diverse tasks is a big win for anyone strapped for annotations.\n4. Clarity: Figures are self-explanatory, training details live in the appendix, and the writing flows—you rarely re-read a sentence to “get it.” \n\n\n\nWeaknesses\n \nI am not persuaded by the motivation of the proposed downstream task \"outdoor to indoor generalisation\", since intuitively, the outdoor is a much more challenging scenario and suffers more from data annotation scarcity compared to indoor; it would be helpful if the author could provide more empirical evidence to strengthen the motivation. Additionally, the introduction appears to be overly long, and some typos, such as those on line 7, should be carefully checked.\n\nQUESTIONS:\nQuestion: Why is outdoor-to-indoor the right showcase for “cross-domain” generalisation when indoor scenes are typically easier and better labelled?\nSuggestion: Please provide quantitative evidence—e.g., annotation cost comparison, data-scarcity statistics, or failure cases—showing that indoor fine-tuning really benefits from the outdoor pre-training, and clarify why the reverse direction is less meaningful.\nImpact on score: A convincing justification (or an alternate, harder transfer task) would remove my main conceptual doubt and could raise the score to 6/6.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a foundational LiDAR world model trained via latent flow matching on large-scale LiDAR sequences. It combines a Swin-VAE for efficient BEV occupancy compression and a conditional flow matching decoder for future occupancy prediction. The model is pretrained in a self-supervised manner and evaluated on multiple downstream tasks, showing strong generalization under low-data regimes and significant efficiency gains.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The paper introduces a pretraining architecture that combines a Swin-VAE for BEV compression with a latent flow matching module for temporal modeling. This framework enables scalable self-supervised learning over large-scale LiDAR sequences and supports transfer across multiple forecasting tasks.\n2. The pretrained model significantly reduces reliance on human labels. For example, on semantic occupancy forecasting, it outperforms supervised baselines using only 5% of labeled data, indicating strong inductive priors learned from geometric dynamics.\n3. The Swin-VAE achieves up to 6× higher compression than prior work while maintaining high reconstruction fidelity. Combined with latent-space forecasting, the method runs up to 28.2× faster and requires substantially fewer FLOPs than baselines.\n\n**Weaknesses:**\n1. The paper does not compare to recent works like Visual Point Cloud Forecasting (CVPR'24), which also use point cloud forecasting as a pretraining objective. Without conceptual or empirical comparison, the novelty claim of being the “first foundational LiDAR world model” lacks proper grounding.\n2. Figures 2 and 3 are not referenced in the main text, making it harder to follow how temporal modeling is implemented in practice.\n3. Although the model aims to predict future LiDAR scenes, it does not evaluate the geometric or structural quality of the predicted outputs. There is no analysis of consistency, plausibility, or failure modes, leaving the realism and utility of the predictions unassessed.\n\nQUESTIONS:\n1. Why is Visual Point Cloud Forecasting (CVPR 2024) not discussed or compared, given that it also uses point cloud forecasting as a self-supervised pretraining objective?\n2. Figures 2 and 3 illustrate key components of your method but are never referenced in the main text. Can you clarify why they were omitted from the narrative, and how readers are expected to connect them with Sections 3.2 and 3.3?\n3. As a LiDAR world model, does your method support explicit reconstruction of future point clouds? If so, how do you evaluate the geometric quality of these outputs beyond voxel mIoU—for example, in terms of fidelity, completeness, or temporal consistency?\n\nLIMITATIONS:\nno", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores building foundational LiDAR world models, transferability to downstream forecasting tasks and efficient data compression via CFM.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n- the \"foundational\" model for LiDAR-based world modeling could be of interest to relevant community\n\n**Weaknesses**:\n- The \"Foundational\" Claim seems Overstated. The paper uses transfer learning success as evidence of a \"foundational\" model that has learned a \"universal 3D motion prior.\" However, the pre-training (outdoor driving) and all fine-tuning tasks (different LiDAR beams, indoor navigation, semantic driving scenes) fall within the domain of ground-level navigation. The current results of a large model pre-trained on a relevant, large dataset performs well on similar downstream tasks, appears not necessarily evidence of a foundational model.\n- Trade-offs in compression efficiency and forecasting. The paper's focus on high-ratio compression (up to 768x) is a key part of its efficiency claim. However, there is a lack of analysis on what information is lost during this compression and how that impacts the model's ability to learn nuanced dynamics. High compression might filter out subtle cues (e.g., a slight change in a pedestrian's posture) that are critical for predicting complex, long-term events. The model may be efficient at forecasting general motion but could be brittle in safety-critical edge cases, a trade-off that is never explored.\n\nQUESTIONS:\n1. What are the challenges of buiding a foundational LiDAR world model and how do you address them? The reasoning is missing in the introduction. Alternatively, can augmenting a powerful, existing image-based foundational model (like GAIA-2 or Cosmos) with a LiDAR-based perception module also achieve your goal? Could you provide evidence or a stronger argument for why a monolithic LiDAR world model is fundamentally superior to a hybrid system that combines existing visual world model with a LiDAR perception module for geometric accuracy? What critical capabilities are lost in such a modular approach?\n\n2. Is it proper to state the learned representation as \"universal 3D motion prior\"? What specific properties of your learned representation suggest it is universal rather than just a highly effective prior for ground vehicle navigation?\n\n3. Could you provide an analysis/discussion of forecasting performance as a function of the VAE's compression ratio? At what point does the information loss from compression begin to significantly degrade the model's ability to predict complex or subtle dynamic events? This is crucial for understanding the model's limitations in safety-critical scenarios.\n\nLIMITATIONS:\n1. Performance in Non-Ideal Conditions: The model is trained and evaluated on well-curated datasets. Its robustness to real-world sensor noise, artifacts, and adverse weather conditions (e.g., rain, fog, snow), which significantly degrade LiDAR quality, is unknown and unaddressed.\n\n2. Safety considerations: For a model intended for safety-critical applications like autonomous driving, an analysis of its failure modes is essential. The paper lacks any discussion of when the model's forecasts are likely to be inaccurate or unreliable. Understanding these failure cases is critical for safe deployment.\n\n3. Potential for Malicious Use: A powerful generative model capable of creating realistic 4D representations of real-world scenes could potentially be used to mislead or attack autonomous driving systems.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}