Towards foundational LiDAR world models with efficient latent flow matching
Authors: Tianran Liu
Shengwen Zhao
Nicholas Rhinehart
Venue: NeurIPS 2025 poster ()
Keywords: LiDAR World Models
Generative models
Domain Adaption
Primary area: deep_learning
TL;DR: First systematic domain-transfer study for LiDAR world models and a SOTA pipeline: a model pretrained on unlabeled autonomous-driving LiDAR generalizes to downstream tasks using only 5–10% labeled data.

Abstract: LiDAR-based world models offer more structured and geometry-aware representations than their image-based counterparts. However, existing LiDAR world models are narrowly trained; each model excels only in the domain for which it was built. This raises a critical question: can we develop LiDAR world models that exhibit strong transferability across multiple domains? To answer this, we conduct the first systematic domain transfer study across three demanding scenarios: (i) outdoor to indoor generalization, (ii) sparse- to dense-beam adaptation, and (iii) non-semantic to semantic transfer. Given different amounts of fine-tuning data, our experiments show that a single pretrained model can achieve up to 11\% absolute improvement (83\% relative) over training from scratch and outperforms training from scratch in 30/36 of our comparisons. This transferability significantly reduces the reliance on manually annotated data for semantic occupancy forecasting: our method exceeds previous baselines with only 5\% of the labeled training data of prior work. We also observed inefficiencies of current generative-model-based LiDAR world models, mainly through their under-compression of LiDAR data and inefficient training objectives. To address these issues, we propose a latent conditional flow matching (CFM)-based framework that achieves state-of-the-art reconstruction accuracy using only half the training data and a compression ratio 6 times higher than that of prior methods. Our model also achieves SOTA performance on semantic occupancy forecasting while being 1.98x-23x more computationally efficient (a 1.1x-3.9x FPS speedup) than previous methods.

Towards foundational LiDAR world models with efficient latent flow matching
Tianran Liu
Shengwen Zhao
Nicholas Rhinehart
(tianran.1is jm. zhao>amaill utoronto.ca ca
nick. wucckhhhhehhrtt@ttrrtttt.ccc ca
University of Toronto
Abstract
LiDAR- based world models offer more structured and geometry-awane representa- tions than their image-based counterparts. However, existing LiDAR world models are narrowly trained; each model excels only in the domain for which it built. This raises critical question: can we develop LiDAR world models that exhibit strong transferability across multiple domains? To answer this, conduct the first systematic domain transfer study across three demanding scenarios: (i) outdoor to indoor generalization, (ii) sparse- dense-beam adaptation, and (iii) non-semantic to semantic transfer. Given different amounts of fine-tuning data, our experiments show that a single pretrained model achieve up % absolute improvement (83% relative) over training from scratch and outperforms training from scratch in 30/36 of our comparisons. This transferability significantly reduces the reliance on manually annotated data for semantic occupancy forecasting: our method ex- ceeds previous baselines with only 5% of the labeled training data of prior work. LiDAR world
We also observed inefficiencies of current generative- models, mainly through their compeesompnession of LiDAR data and inefficient training objectives. To address these issues, we propose a latent conditional flow matching (CFM)-based framework that achieves state-of-the-art reconstruction accuracy using only half the training data and compression ratio 6 times higher than that of prior methods. Our model also achieves SOTA performance on seman- tic occupancy forecasting while being 1.98x-23x more computationally efficient (a 1.1x-3.9x FPS speedup) than previous methods. Our project page contains additional visualizations and released code.
Introduction
World models enable agents to implicitly learn the dynamics of the environment by predicting future sensory observations, typically through generative models operating a latent space [40, 33]. Starting from classic motion prediction tasks like optical flow [29, 31J, recent advances have extended motion forecasting, i.e., RGB world models, which have demonstrated
prediction to RGB-video-basee impressive performance in applications such as autonomous driving [41, 16, 47], robotic navigation [4], and other embodied tasks [7, 54, 28, 55, 21, 1, 3]. These models can generate video sequences conditioned on historical frames and sometimes natural language input. While an image- based world model may suffice for repetitive tasks (c.g.. learning robotic movements). its utility is limited in complex scenarios that demand geometrically structured information, such as autonomous driving. In these scenarios, the lack of explicit semantic and geometric representations restricts practical application, as it necessitates additional steps to extract spatial information.
In contrast to RGB images, LiDAR provides rich geometric structure and implicitly offers sparse representation for semantic cues about the environment. Unlike dense, pixel-based observations,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Haun rou
42.00
World mode
Evelection
sparse
Densa
Adas
Hotokbattsrt
staparion
35.00
7.00
8.40
1008
200-
10090
1536
750
Forme
Works made.
600)
Ocoiaria
Ourr Sere Hot Ower Seme Pettree
foreaating
Doxm-atream turiig
Figure 1: Left: The overall pipeline of our method: LiDAR dataset from autonomous driving scenarios to train the proposed LiDAR world model. This well-trained world model is able to generalize well on the listed downstream tasks after fine-tuning, although scene and signal properties quite different. Right: Comparison of our proposed world model with previous methods on nuScenes 4D semantic occupancy forecasting metrics of mloU, loU, and inference efficiency. Our approach achieves the best results in terms of both efficiency and performance.
we used the most readily publicly available
3D object in a LiDAR point cloud is represented as a cluster of 3D points, encoding essential spatial and semantic information in compact form. Despite these advantages, foundational world models of LiDAR data remain relatively underexplored: prior LiDAR world models were built for specific domains [56, 20, 57]. In comparison, recent advances "foundation" models, e.g. pretraining with RGB image forecasting [3] and trajectory forecasting [60] have demonstrated the effectiveness of pretraining in improving performance on downstream tasks. This motivates our central question: Can we develop foundational LiDAR world model for ground vehicles that yields downstream performance gains on diverse forecasting tasks, particularly those suffering from data limitations, after fine-tume tuning? This goal is partly motivated by the fact that many of the causal factors that govern the motion of objects are shared across different domains and environments-mman aspects of the dynamics of the world should, in principle, be transferable and can be observed in unlabeled LiDAR.
We investigate pretraining and fine-tuning a LiDAR world model across three diverse transfer tasks (Fig 1): varying-beam occupancy forecasting, indoor occupancy forecasting, and semantic occupancy forecasting. First, LiDAR hardware varies in beam count and scan patterns, often degrading model generalization [56, 39J. We consider this cross-sensor setting to assess robustness to hardware variations. Second, robots equipped with LiDAR sensors often operate in vastly different environments [50, 59, 15], from outdoor to indoor settings. While cross-doman
adaptation has been extensively studied in perception tasks [24, 34, 30], remains underexplored in the context of dynamic learning. Given that LiDAR data can be scarce in certain environments (e.g.. indoor), it is promising to investigate whether dynamic knowledge learned in data-rich, long-range domains can be transferred to those with limited data and different operational ranges. Lastly, tasks like semantic occupancy forecasting [6, 14, 23, 58]) rely heavily on costly semantic labels [46, 44]. Our goal is to leverage large-scale unlabeled data to learn universal 3D dynamics prior for ground vehicles, enabling strong semantic forecasting via fine-tuning on minimal labeled data.
Extensive experiments demonstrate that our LiDAR world model significantly improves the conver- gence speed of downstream tasks. For all mentioned tasks, we observed relative performance gains with varying amounts of fine-tuning data. In particular, for semantic occupancy forecasting, this scheme of learning dynamics prior to semantics patterns allows us to achieve superior performance to OceWorld [58] with only 5% of its required labeled data. Furthermore, our results highlight the importance representation alignment during fine-tuning. Although the data compression struc- ture propose later is data-efficient, we found that either using the pretrained data compressor

---

directly or retraining the VAE from scratch with the fine-tuning data leads to suboptimal perfor- mance. This phenomenon attributable to a feature space mismatch: in both scenarios, the encoder learns a feature space mapping that diverges from the pretrained one, which in turn degrades the performance of the flow model pretrained on the original feature space. We identify two superior strategies: fine-tuning the VAE, and for tasks where the VAE cannot be fine-tuned directly, applying a biaine-siillarty-baase -baed alignment loss.
We also find that the current architectural paradigms used in LiDAR world models [38, 12] suffer from 2 issues: redundant model paramefers and excessive training time. Regarding the former, the latent representation tends to retain large number of channels, which significantly increases the parameter count of the dynamics model. Regarding the latter, state-of-the-arr models often require thousands of epochs to converge. This inefficiency stems not only from the model scale but also from the inherently slow and combining DDPM-basek training with DDIM-style sampling. These challenges have significantly hindered our exploration of LiDAR world model transferability.
nature of denoising diffusion paradigms-- particularly those
To address these issues, we first propose a Swin Transformer-based VAE architecture for LiDAR data compression. This architecture achieves compression ratio of 192x-over 6x higher than previous state-of-the-art methods-while matching or exceeding their reconstruction quality. We also propose an efficient flow-mate www-matching--ars based generative model. Compared to previous latent diffusion- based or transformer- the FLOPs that they require, respectively. This efficiency significantly accelerates our analysis of transferability in downstream tasks. In summary, our contributions are:
deterministic schemes, our approach requires only 4.38% and 28.91 % of
The first study on building transferable LiDAR world models: world models of LiDAR videos that exhibit substantial transferability to downstream forecasting tasks. We show the efficacy of LiDAR world models to 3 diverse fine-tuning tasks: semantic occupancy forecasting, indoor occupancy forecasting, and high- high-beams occupancy forecasting, and confirm that it outperforms the baseline of training from scratch the fine-tuning data, and that the relative performance gain is more pronounced with less fine-tuning data.
An approach to substantially reduce reliance on human labels for semantic occupancy forecasting: our method exceeds the performance of previous methods with only 5% of the human labels.
Efficient VAE-baesd architectures for data compression and voxel-based LiDAR world modeling. The former achieves the SOTA reconstruction accuracy at 6x improvement in compression rate over prior work. Using these encodings, our world model achieves SOTA performance with only 4.47% to 50.23% of the FLOPs and 1.1 to 3.9 times higher FPS.
2 Related Work
We categorize previous work by LiDAR- based world modeling for geometric future prediction, semantic 4D occupancy forecasting for semantic-aware scene understanding, and foundational world models (FWMs) that aim to generalize dynamic knowledge across domains and tasks.
LiDeR-based World Models. Distinct from general LiDAR generation tasks, LiDAR-based world modeling (also known as LiDAR/Occupancyy Forecasting) aims to forecast future sensor observations based on past observation. Occ4D [20] and Occ4cast [27] propose forecasting future LiDAR points or occupancy grids via a differentiable occupancy-to-point module, yet without explicitly modeling latent transition dynamics. UNO [2] further introduces occupancy fields within NeRF-like [35] framework to enhance forecast fidelity. S2Net [49] adopts a pyramid-I STMM architecture to predict future latents extracted by a variational RNN, while PCPNet [32] leverages range-view semantic maps and a transformer backbone to improve real-time inference performance. Although numerous works [37, 39, 51, 36, 19] have introduced increasingly powerful diffusion-based models, for general data generation, progress in LiDAR forecasting remains comparatively limited compare to the RGB- base ones: Copilot4D [56] achieved otate-of-thhear performance in LiDAR forecasting by adopting a MaskGiT-based latent diffusion model with a designed temporal modeling objective. BEVWorld [57] extended this approach by incorporating multi-modal sensor inputs, enabling future LiDAR prediction even in the absence of current LiDAR frames. However, few works have addressed the transferability of dynamies learning across domains an appealing path to enhance the widespread deployment of world models in real-world autonomous systems.
3

---

Semantic 4D Occupancy Forecasting: The RGB video-based world models tried to forecast physical consistent video from RGB input. However, these representations lack of geometric and explicit semantic annotation, the usage of the generated forecasts observations in control task still need extra component to recover depth and predict semanties. Semantic 4D occupancy forecasting aims to address this gap by predicting future semantic (LiDAR) occupancy maps based on past observations- either ground-truth annotations or model-generated results. Occ World [58] employed an auto-regressive transformer to jointly forecast future latent states and corresponding axis off'stts OceSora [45] further advanced the field by being the first to generate 25 seconds semantic videos conditioned on 512x compressed inputs. Later, DynamicCity [6] improved upon this by decomposing the 3D representation into a more compact HexPlane[9] structure, enabling faster inference. Building upon future trajectories/bev layout and extended datasets, DOME [14] and uniScenes [23] continue to push the forecasting accuracy to new state-of-the-arr levels. Importantly, these methods rely on labeled semantic ground-truth data for training, making them dependent expensive human annotations and thus challenging to scale.
Foundational world model (FWM). We call world model "foundational" if leads performance gains over learning from scratch on multiple downstream forecasting tasks. In autonomous driving, GAIA-1 [16] and GAIA-2 [41] introduced image-based FWMs that can generate controllable driving scenarios. the field indoor robotics, like navigation, NWM [4] proposed an RGB image- based navigation method with conditional DiT structure. Cosmos [1] takes this ambition further, aiming to generalize across both indoor and outdoor environments. However, due to limitations in the RGB modality, existing FWMs do not provide explicit depth information, which makes it harder define prediction and planning modules to utilize their output.
3 Efficient latent conditional flow matching
In this section, we first introduce our novel point cloud data compressor Sec.3.1, which achieves state-of-the-att performance under high compression ratio. Based on the compact representation from it, Sec.3.2 presents the flow matching- for all subsequent fine-tuning approaches. In Sec.3.3, we introduce the VAE tne-tuning for better representation alignment, which will benefit the final forecast performance.
generative forecasting model that serves as the testhed
3.1 Data compression
W 120)
&ight
Semeric
FancCow
Aikoayy (dler reprosoraaiion
Paidelyy
Vore
Corn Swis
Layer form
Charmil-bee flanhs dctn
FerCorn
Seitt Book
Wis
PV2 1:22 3T21
Entonding
Dense
Repora neterteed
KCD.))
Douusency
0% MJ 2001
Simileity mep
Greadd ceutates
Tandoors
Caw my renderig
Serardic Mape B,
XM
XN
Resovered
LOWI pointa
Figure 2: Architecture of our VAE for LiDAR compression. The model enables high compression ciclos-exceeddin those of previous methods methods-alongside alongside high-fidelity reconstructions, Here we use O, to represent the raw voxelized LiDAR points (Occupancy) and Od, O, stand for densified or semantic labeled occupancy respectively.

---

While SD3 [12] provides a strong baseline for compressing image-based RGB signals, LiDAR- specific data compression remains underexplored. Existing LiDAR world models typically adopt the SD3 encoder-decoder architecture with minimal modifications, retraining it directly on LiDAR BEV representations. As an alternative, we propose a Swin Transformer-base demonstrate that it achieves state-of-the-ar performance across all forms of LiDAR compression, as detailed in Table 2. As noted in previous works [23], discrete coding-bauee based compression is not only prone to problems such as codebook collapse but is also generally inefficient in compression. Here we have also used continuous coding to achieve a higher compression ratio as shown in Figure 2.
architecture and
Data processing and encoding: In this work, bold notation indicates multidimensional array (e.g.. a vector or matrix). Given raw LiDAR sensor scan Oj E R the voxelized map o,, dense occupancy map On, and semantic occupancy map o, can be generated through successive steps of voxelization, densification (i.e., accumulating points from the entire sequence into each frame), and semantic annotation. Our structure is capable of processing any of these representations derived from the raw LiDAR input. Depending on the final reconstruction requirements, voxelization can approach
direct binarization process. Taking the compression of Oa E I x DxC (a H, W, D 3D grid with C-dim features at each voxel) as an example, Gaussian encoding distribution (== =N(za Mg (a) is constructed from three main stages: embedding, feature learning and downsampling, and channel reduction. Specifically, after applying height and class embeddings, a 2D BEV feature map is obtained, which is subsequently processed by standard 2D Swin Transformer encoder. Unlike the original Swin Transformer design, which utilizes patch merging for downsampling, we replace this operation with conventional convolutional layers modification we empirically find to outperform the original approach. Channel reduction is performed by lightweight network neck, compressing the feature representation into 16-dimensional latent space. Samples are drawn using the reparameterization trick 2 = pa + ag 3 6 ~ (O,, I), and @ denotes the Hadamard product.
be either a learning-basee
Decoding and o representation recovery. The decoder q(ó.) za) is designed to mirror the encoder by starting with the same symmetric 2D block structure. Different from previous methods that used 3D blocks in the decoder to increase temporal feature consistency, we found doing so to have a negative effect in our structure, for both reconstruction and final forecasting result, shown in the ablation study. Depending on the source representation, the reconstructed O₂ can be used to render the points with a differentiable ray rendering module get the occupancy map by calculating similarity score with the class embedding.
3.2 Forecasting with Conditional flow matching
With the proposed VAE, we are able to mitigate the parameter redundancy: most of the parameters of the existing methods come from the high dimensionality of the latents. To further make model training more efficient, we present a new structure based on flow matching [26], which we show leads to SOTA performance in both forecasting accuracy and computational efficiency, as shown in Table 1. For semantic occupancy forecast task, given of0: our VAE encodes continuous frames to 2 z E
H XW x16 As 4 is the middle index of these frames, our objective is to obtain the future latent z/: from the historical latent z/0:1 using a flow-matching model Gg.
As illustrated in Figure 3, the training objective is to regress velocity field uf This regression is conditioned on several inputs: a time step 1 E [0, 1] representing the progress along a probability path from an initial distribution (standard Gaussian) to the target distribution; the interpolated future latent state z;; the historical trajectory Juash E R (f: xt, and historical observations ot
Specifically, the noising added follows a linear interpolation method as shown in equation where used to balance the scale of noise and latents and e ~ (0 1). The 200: will be used as part of condition training: we concatenate the historical latents and noised future latents along time dimension to get the input of Ge. denoted by Zjoint E (t(t-1o)xx1 Wx32
x (1 t)e tazttitz
(1)
Based on the spatial-temporal DiT structure proposed in previous methods [33, 23, 14], we observe that the convergence speed remains suboptimal. Specifically, given Zjoint. the initial 3D convolutional layer increases the channel dimension to C", resulting in 2/oint E (R(2-fo)) XXC" The width and height dimensions are then flattened before being fed into the spatial DIT, where multi-head attention
5

---

Cafty
DTF Roo--Boed Block
Frame wise VAE Encoder
Prefioral Srewkling
DI -i--aaddd Block
DrT Saseed
Hatorical frome usent-os 26 condition
ResConv
DownrUp sample
Time dimmrssion
Roscalo
Tempocal
Spatial
Pulty
Fused actra ocncition
x
"c (t foft (aat
Fourler Embedding
PautTiojoctoyy fel
Figure 3: Architecture at the training stage of our proposed conditional velocity field predictor for time L. Historical frame latents are extracted via frame-wise VAE encoder, and noised future (target) latents are formed by injecting noise at timestep f. Latents are concatenated along the time dimension and passed through DiT blocks.
(MHA) is applied and normalized using AdaLN. However, applying the same strategy in the temporal DiT would only fuse pixels along the temporal axis, limiting the temporal receptive field to just single latent pix block). To compensate,
observed that most of the previous methods like uniScenes [23] and DOME [14) use 14-18 stacked blocks to ensure the temporal consistency, which also of the reasons of redundant parameters. While this design remains feasible in Latte [33], which operates on 3D video patches, our per-frame latent representation, making the learning of temporal dependencies more challenging. We found that this issue can be effectively mitigated by simply inserting a 3D convolutional layer with a larger receptive field after the spatial DIT. Furthermore, organizing the network in a UNet-style architecture, opposed a single-stride DiT backbone, is shown through later experiments to further enhance the forecast performance and reduce computational load. The training objective was designed following the Rectified Flow [26]:
(()) E.. (z) (z, - t ~ sigmoid(N((0, 1)), æo ~ N(O, I), 2, 9z.-
=o)ll"
3.3 Improved fine-tuning with representation alignment
The pretrain of world model can now be start with framework proposed in 3.2 & 3.1 and large amounts of unlabeled data Oa in a self-supervised way to learn the unified prior knowledge for environment dynamics. For non-semantic to semantic transfer specifically, encoder/decoder pair denoted by qa( (==a)oo) / ga(Oa) zd) and forecasting model G Although we can train the other data compressor g, (z. o.) qu ( (2.) and use the corresponding latent 2, to aligned,
now already have
fine-tuning pretrained G have discovered that the latent spaces of 2, and za which is affects the performance fine-tuning.
For subtask 1 and 2 in Fig 1, this suboptimality be tackled by fine-tuning the data compressor first, while case of subtask 3 (semantic occupancy forecasting), due to the difference network embedding layer dimensions, it is non-trivial to fine-tune the semantic data based VAE pretrained on non-semantic data. Instead, we use the latents of the corresponding dense occupancy oa to guide the formulation of the subspace of the semantic VAE.
Specifically, we add a cosine similarity term in the loss when fine-tuning the VAE, as shown in the last term of Equation 2. z, and d, stand for the latent from pair o, and 0₂ via training from scratch and pretrained VAE, respectively. Liovasz [5] is another reconstruction term used to optimize loU.
Lsem-VAE Lce(os, 0z) + 3DL (gs (zs) (o) p(zz)) + XL (0,, Os) x.Coos d.) (2)

---

Therefore, our full pipeline is 3-fold: we first train the data compressor and flow matching model via the easily accessible unlabeled data. Then we use the pretrained data compressor to guide the structure of latent space for subtask's data. Finally, with the aligned representation, we fine-turn the pretrained weights to obtain the final results.
Experiments
We designed our experiments to investigate several questions: Q1: Can we improve compression and performance existing designs? Q2: Can we develop LiDAR world model that exhibits substantial superiority to models trained from scratch on three downstream forecasting tasks (high beam occupancy forecasting, indoor occupancy forecasting, and semantic occupancy forecasting)? Q3: How well does each fine-tuning variant perform and why?
4.1 Experimental design
We nuScenes[8] (2Hz) as the pretraining data for the flow matching model. We trained 2 different types of foundational models on it, one was trained based on the original LiDAR sweep (o,), totaling 27, 7,000 frames, denoted by Gğ- The other type of model was trained using densified LiDAR frames oa), totaling 19,000 frames, denoted by GØ- The latter training set is subset of the former part and the number of Oa equal to O,,
For the beam adaptation subtask, we down-sampled 11 sequences from the KITTI360 raw dataset [25] from 10Hz to 2Hz to match the foundational model setting. Also, we collected an indoor navigation dataset using Clearpath Jackal equipped with OSO-128 LiDAR sensor (training set with 23,504 frames and validation set with 9,720 frames). Finally, for the semantic occupancy forecasting, we present twofold experiments: in section 4.2, follow the official splitting [44] and train the model from scratch. Then in Section 4.3 and Appendix H.1, we only use first half of the training data to pretrain the G as the foundation model, which then fine-tuning Iv1 correspondence) to avoid the dynamic knowledge of fine-tuning data to be already seen during pretraining stage. From the next section, more details on the model setup, please refer to our appendix.
the other half o, (o, and од
o', and o' denote the partial training data. For
4.2 Encoder structure exploration and semantic occupancy forecasting evaluation
For model evaluation, Table 2 compare reconstruction results of our previous methods compress data from 8x to 512x. At 32x, we achieve 99.2% mloU and 97.9% loU-far above UniScenes's 92.1 %/87.09 700 the same rate. Even 192x, maintain 93. 03.9%/85.88% surpassing WorWorld and DOME by over 11% mloU, and at an extreme 768x our model still delivers 9.7% relative gain despite 1.5x higher compression.
VAE with
Table 1: Comparison of future occupancy forecasting performance on full nuS validation set. t: Methods use future trajectory information. DynamicCity generates 16 frames jointly with 1000 steps DDPM-style sampling, we report 50 steps DDIM FPS for faster eval but still keep the original accurancy. All FPS tested based on 1x RTX4090 without kernel fusion other CUDA acceleration methods. -: Unreported or unable to be computed due to code unavailability.
mloUt
lol't
Mean
GFLOPs
Method
2s
3s
Avg
2s
3s
Avg
(bits/dim)
Params.
Frame]
FPS-
OceWorld [58]
25.75
15.14
10.51
17.13
34,63
25.07
20.19
26.63
72.39
1347.00
16.97
RenderWorld [52]
28.69
18.89
14.83
20.80
37.74
28.41
24.08
30.08
416
OceLLama [48]
25.05
19.49
15.26
19.93
34.56
25.83
24.41
29.17
DynamieCity"
34.12
25.82
45.43
774.44
19.30
Ours
33.17
21.09
15.64
23.33
40.53
30.37
24.44
31.78
6.29
30.37
389.46
22.22
DOME [14]
29.39
20.98
16.17
22. 18
38.84
31.25
26.30
32.13
444.07
8891.98
5.48
Ours!
36.42
27.39
21.66
28.49
43.68
36.89
31.98
37.52
4.55
30.37
389.46
21.43
Based on the strong VAE performance, our forecasting model advances the state-of- the-art of semantic occupancy forecasting while maintaining real-time throughput. As shown in Table 1, our model achieves a one-second mloU of 33.17%, surpassing
the previous SOTA model's 28.69% and OccLLama's 25.05% by 4.48% and 8.12%
7

---

respectively and maintaining 21.09% and 15.64% for two and three second predic- tion which outpaces prior works by least 2.5% relative performance improvement. For
forecasting, our mloU surpasses DOME by more than 5.5% across all frames. Our semantic occupancy forecasting model runs at 22.22 FPS, requiring only 389.46 GFlops per frame and 30.37 million parameters. Compared to DynamicCity which generates frames altogether, model uses half of GFlops per frame, 66% of parameters, and 1.1 times higher FPS. In the future- model sustains 21.4 FPS without increasing in computational cost and parameter count, whereas DOME utilizes 23 times more flops per frame (our model is 4.38% of this amount), 15 times more parameters, 4 times slower FPS, and at least 5.5% less absolute performance. Due to the stochastic nature of our proposed CFM model and the limitation of mloU and loU in evaluating model's ability to generate diverse but plausible future predictions, we also report the negative log likelihood (NLL) in bits-per-dimension. Our model also present better performance compare with previous SOTA stochastic models.
Table Occupancy reconstruction perfor-
mance at various compression ratios.
Method
Cont.
Comp. Ratiot
mloUt
lol.
Ooel Jama [48]
K
8
75.2
63.8
OecWorld
65.7
62.2
OceSora [45]
512
27.4
37.0
DOME [14]
64
83.1
77.3
UniScenes (23)
32
92.1
87.0
rajjectory-conditione regime,
UniScenes (23)
512
72.9
64,1
Ours
99.2
97.9
Ours
192
92.8
85.8
Ours
384
88.3
76,9
69.3
In order to further measure sample quality. we computed the 3D Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) in Table 4. Our model establishes FID and KID. On average, for our model without future trajectories, it reduces FID to just 28.3 % / 48.5 % and KID to 22.7 % / 44.4 % of the scores reported by the deterministic (OccWorld) and stochastic (DOME) models, respectively. See Appendix A.2.2 for more details of these metrics.
state of the art both
Then, benefiting from the low information loss of the data compressor we proposed, it is in fact highly suitable to be used as feature extractor for 4D inception-based metries (temporal consistency measurement). After we revise the structure with more temporal (4D) module, we retrain the proposed data compressor to eval the FVD score of generated results. In Table 3, to give context for temporal consistency, include approach that randomly shuffles the ground truth future ordering and compare the FVD with the correctly ordered GT. This tells us the FVD when the predictions individually correct but temporally inconsistent. Our model scores of 7.68 and 7.80 are also significantly lower than previous methods.
Table FVD Score Comparison,
Table FID and KID(x 10 3) comparison for our method and other generative-base based and deterministic-ber based models.
3s (6 frames) videos used as input
Method
FVD J (x10-3)
2s
3s
AVG
Method
OceWorld DOME
18.68
FID
KID
FID
KID
FID
KID
KID
9.79
OccWorld
9.54
11.7
8.56
10.46
7.78
9.80
8.62
10.60
Ours (hist, traj.)
7.80
DOME
4.37
4.39
5.09
5.44
5.63
6.53
5.03
5.42
Ours (fut. traj.)
7.68
Ours (Hist. traj)
1.67
1.49
2.81
2.85
4.02
4.74
2.83
3.02
Reorder GT
12.07
Ours (Fut. traj)
1.61
1.47
2.90
3.05
3.91
4.55
2.80
3.02
4.3 LiDAR world model transferability
First, Table 5, we compare the pretrained model with previous 4D forecasting (occupancy fore- casting) methods [53, 20]. Ours exhibits better performance in non-semantic occupancy forecasting, particularly at the 3-second horizon. Then, using the architecture in Section 4.2, we evaluate the performance gains from fine-tuning world model with varying strategies and data scales across individual tasks and present the results in Figure 4.
For the high-beam LiDAR adaptation task, all configurations based on the pretrained world model VAE and CFM training (blue line) across all frames and data fractions. Pretraining CFM alone yields 17.1 1 % and 6.68% gains for 1s and 3s predictions at 10% data. With both components pretrained and fine-tuned, the gains increase 14.48% and 26%, respectively. This demonstrates that our Swin Transformer-basee VAE effectively captures transferable geometric priors for downstream tasks with varying LiDAR characteristics and the effectiveness of the feature alignment. For the indoor LiDAR adaptation task, all pretrained model variants outperform from-
outperform
8

---

Reth from sorech -
Soth pretmited (nll-tnttninggi
GrrWhrrd full nutoo data
Only prstevinnl
Soith pretntial (FaRA Enstaning 128, abyra- 1280
forecseting bortson
2x foreractng tomtom
In ferecasting hortron
100 singlesten
12
11
010 S1 Maggoete 1290 Teatea Aineteilen
29
28
22.5
30.0
27.5
200% data
200% data
10.51
30.0
200% date
17.
10.0
6
10
100
10
25
50
100
100
Proction Tho traing data 0%
Praction Enn tuning date
Prection Riso tuning Cata 196
Figure IoU/mIoU comparison across forecasting horizon the presence different fraction of fine-tuning data used from total data available between the various training procedures. In row order, each row refers to the results of (i) different beam adaptation (ii) outdoor-indoor adaptation, and (iii) semantic occupancy forecasting, respectively.
scratch training when fine-tuning data is limited (less than 25%), with full-parameter tuning of both VAE and CFM achieving the best results. At 10% data usage, full fine-tuning yields least 4.47% gain across all prediction horizons.
However, data availability increases (e.g., beyond 25% for Is prediction), training from scratch eventually sur- passes all fine-tuning methods. This is likely because the indoor dataset represents a task sufficiently distinet from our pre-training domain, making it easier learn from scratch without the influence of pre-trained priors. We hypothesize that this result would no longer hold if sub- stantial indoor data were included in the pre-training set. Similar results are also observed in the semantic occu- pancy forecasting task, where all VAE/CFM combinations based on the pre-trained model outperform training from scratch in mloU. Notably, with just 10% of the seman- tic fine-tuning data-which is only 5% of the total data used in prior secchoossfull-praaaae an 82.6 92.6%/80..7 700.772%%66600 formance [58]. After 50% data usage, the performance of from-scratch training exceeds that of the CFM-only and LoRA CFM fine-tuning methods. Nevertheless, full pallmerreeeter VAE+CFM fine-tuning consistently achieves the best performance across all data fractions and prediction horizons.
Table 5: Pretraining performance on dif- ferent LiDAR representations, without future trajectory. ViDAR only re- leased weights trained on dataset.
loUt
Methods
1s
Avg
VIDAR (12L)*
13.11
12.48
11.78
12.46
Oco4D (L2L)
26.96
19.51
16.81
21.09
Ours-o,
26.98
21.56
18.26
22.27
Ours-oa
39.64
29.35
23.73
30.91
fine-tuning yields 62. 90% relative mloU gain for 1s/2s/3s prediction, surpassing OccWorld's per-
These results confirm that foundational models can capture transferable, LiDAR- based dynamic priors from unlabeled data to support downstream tasks requiring semantic interpretation. The improvements are especially notable in low-data regimes, demonstrating reduced dependence on human-labeled samples. Overall, across all tasks, our pre-trained model achieves up an 11.17% absolute performance gain and outperforms training from scratch in 30 out 36 comparison points. See the Appendix for detailed numerical values corresponding to Figure 4.

---

However, key question remains: why do models that fine-tune both the VAE and CFM modules consistently outperform other settings? In other words, how does fine-tuning the VAE impact the CFM's forecasting accuracy?
To further analyze, in Table 6, we compute the similarity of latent spaces under different data scales using CKA [22] and CKNNA [18]. The similarities are measured between pairwise samples w/w.o semantic information, i.e., 022 with o', obtained from fine-tuned VAE or VAE trained only on fine-tuning data. We observe that the effectiveness of VAE fine-tuning is primarily attributed to aligning the latent space of the new domain with that of the original domain, rather than improving reconstruction accuracy. For example, when using 100% of the fine-tuning data o',. the non con-pretadined VAE achieves even higher mIoU/IoU than the pretrained one, yet the final performance shows a clear gap suggesting that preserving latent structure plays more crucial role. Although paired data is not available for the first two adaptation scenarios, estimating latent similarity in other domains could further support our hypothesis, details are shown in Appendix.
4.4
Ablation study
Table 6: VAE fine-tuning evaluation in semantic to semantic adaptation.
Table 7: A Ablation study of different designs in dynamic learning. We use Mean mloU here, all experiments trained for 100 epochs.
Pretrain
Fine-tusing
mloU
IoU
CKNNA
Cosine
%
10% oz
87.97
75.22
0.739
0.27%
0.907
10% o,
75.81
72.27
0.653
0.221
0.183
Mohhod
U-Net
CFM
after Temporal DIT
3D Conr
CRG
Seale
mioU
76.92
0.273
0,900
25% o
83.09
75.92
0.619
0.205
0.236
UNet
Baeeline
x
K
x
X
1.0
1.0
17.77
17:42
50% of
89.72
80.09
0.744
0,275
0.901
CFM
X
X
1.0
20.
50% o4
90.26
82,03
0.600
0.191
0.237
3D Conv
/
X
1,0
20.67
CPG
1.0
21.05
100% of
82.56
0.750
0:275
0.910
Rescale
/
10
23.33
100% 0
92.14
83.41
0.628
0.199
0.219
As mentioned in Section 3.2, we now present a breakdown of the performance im- provements contributed by each individual module. As shown in Table 7, we ob- serve consistent gains from CFM, 3D convolution, and CFG components, while us- ing classifier-free guidance gives us the most significant improvement in both IoU FLOPS, FPS, and accuracy.
and mloU. We also examine the impact of NFE As shown in Figure 5, taking NFE=1O, we get the best forecasting accuracy while maintaining considerable level of efficiency in terms run time and GFlops for the semantic occupancy forecasting task. For the experiment of VAE data-efficient and other details, please see the Appendix.
Cruchs ITS, Mear Ie Meen recU
Mea kel
35
2520
J0c5
5 Conclusion
-22
In this work, by designing model that can be efficiently pretrained on large-scale outdoor LiDAR data, we show it can be effectively fine-tuned on diverse downstream tasks. Our approach consistently outperforms training from scratch, especially in low-data regimes. existing models, we introduce a new VAE structure for high-ratio LiDAR compression and a conditional flow matching approach for forecasting. These components enable strong reconstruction performance and improve training and infer- ence efficiency without sacrificing forecasting accuracy. In addition, VAE fine-tuning strategies further boost performance. While results are encouraging, future work is needed to extend general- ization to additional environments, incorporate multi-modal inputs, and integrate with planning and control systems. Overall, our find- ings suggest that scalable and transferable LiDAR world models are feasible and can significantly reduce reliance on annotated data in practical applications.
2005
32
500
To address inefficiencies
Figure 5: The performance and efficiency of proposed VAE + CFM architecture un- der different NFE value dur- ing the sampling process. We observe that NFE=10 corre- sponds to the best model per- formance, with reasonable ef- ficiency in terms of FPS and Gflops per frame.
Acknowledgments
This research was enabled in part by the Digital Research Alliance of Canada (alliancecan.co ca).
10

---

References
[1] Niket Agarwal, Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, et al. Cosmos world foundation model platform for physical ai. arXiv preprint arXiv:2501.03575, 2025.
[2] Ben Agro, Quinlan Sykora, Sergio Casas, Thomas Gilles, and Raquel Urtasun. Uno: Unsu- pervised occupancy fields for perception and forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14487-14496, 2024.
[3] Hassan Abu Alhaija, Jose Alvarez, Maciej Bala, Tiffany Cai, Tianshi Cao, Liz Cha, Joshua Chen, Mike Chen, Francesco Ferroni, Sanja Fidler, et al. Cosmos-transserl: Conditional world generation with adaptive multimodal control. arXiv preprint arXiv:2503./4499 2025.
[4] Amir Bar, Gaoyue Zhou, Danny Tran, Trevor Darrell, and Yann LeCun. Navigation world models, arXiv preprint arXiv:2412.03577 2024.
[5] Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: tractable surrogate for the optimization of the intersection-over-une measure in neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4413-4421, 2018.
[6] Hengwei Bian, Lingdong Kong, Haozhe Xie, Liang Pan, Yu Qiao, and Ziwei Liu. Dynamiccity: Large-scale lidar generation from dynamic scenes. arXiv preprint arXiv:2410. 2024.
[7] Kevin Black, Noah Brown, Danny Driess, Adnan Esmail, Michael Equi, Chelsea Finn, Niccolo Fusai, Lachy Groom, Karol Hausman, Brian lchter, et al. pig: A vision-language-actio flow model for general robot control. arXiv preprint arXiv:2410.2414 164, 2024.
[8] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 11621-11631, 2020.
[9] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 30-141, 2023.
[10] Ricky T. Q. Chen, Yulia Rubanova, Jesse Bettencourt, and David Duvenaud. Neural ordinary differential equations, 2019.
[11] Christopher Choy, Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3075-3084, 2019.
[12] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow trans- formers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
[13] Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. Ffjord: Free-form continuous dynamics for scalable reversible generative models. In Interna- tional Conference on Learning Representations (ICLR), 2019.
[14] Songen Gu, Wei Yin, Bu Jin, Xiaoyang Guo, Junming Wang, Haodong Li, Qian Zhang, and Xiaoxiao Long. Dome: Taming diffusion model into high-fidelity controllable occupancy world model. arXiv preprint 2xXi:24400.14222 2024.
[15] James R Han, Hugues Thomas. Jian Zhang, Nicholas Rhinehart, and Timothy D Barfoot. Dr- mpe: Deep residual model predictive control for real-world social navigation. IEEE Robotics and Automation Letters (RA-L), 2025.

---

[16] Anthony Hu, Lloyd Russell, Hudson Yeo, Zak Murez, George Fedoseev, Alex Kendall, Jamie Shotton, and Gianluca Corrado. Gaia-1: A generative world model for autonomous driving. arXiv preprini arXiv:2309. 17080, 2023.
[17] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[18] Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hypothesis. arXiv preprint 20Xx:2205500788,, 2024.
[19] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in neural information processing systems, 35:26565-265777 2022.
[20] Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting proxy for 4d occupancy forecasting. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[21] Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, and Chelsea Finn. Openvla: An open-source vision-language-action model, 2024.
[22] Simon Korbllith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited, 2019.
[23] Bohan Li, Jiazhe Guo, Hongsi Liu, Yingshuang Zou, Yikang Ding, Xiwu Chen, Hu Zhu, Feiyang Tan, Chi Zhang, Tiancai Wang, et al. Uniscene: Unified occupancy-centre driving scene generation. arXiv preprint arX1v:2412.05435, 2024.
[24] Zhuoling Li, Xiaogang Xu, SerNam Lim, and Hengshuang Zhao. Towards unified 3d object detection via algorithm and data unification. arXiv preprint arXiv:2402. 2857//3,3 2024.
[25] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292-3310, 2022.
[26] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. arXiv preprint arXiv:2209.0300 03003, 2022.
[27] Xinhao Liu, Moonjun Gong, Qi Fang, Haoyu Xie, Yiming Li, Hang Zhao, and Chen Feng. Lidar- based 4d occupancy completion and forecasting. 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 11102-11 11099 IEEE, 2024.
[28] Yuecheng Liu, Dafeng Chi, Shiguang Wu, Zhanguang Zhang, Yuzheng Zhuang, Bowen Yang, He Zhu, Lingfeng Zhang, Pengwei Xie, David Gamaliel Arcos Bravo, et al. Omnieva: Embodied versatile planner via task-adaptive 3d dd-grounded and embodiment- reasoning. arXiv preprint arXiv:2509.09332. 2025.
[29] Libo Long, Xiao Hu, and Jochen Lang. Learning to handle large obstructions in video frame interpolation. In Proceedings of the 32nd ACM International Conference Multimedia, pages 5221-5229, 2024.
[30] Libo Long, Xiao Hu, and Jochen Lang. Shape and texture: What influences reliable optical flow estimation? In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 27894-27903, 2025.
[31] Libo Long and Jochen Lang. Detail preserving residual feature pyramid modules for optical flow. In Proceedings of the IEEE/CVE Winter Conference on Applications of Computer Vision, pages 2100-2108, 2022.
[32] Zhen Luo, Junyi Ma, Zijie Zhou, and Guangming Xiong. Pepnet: An efficient and semantic- enhanced transformer network for point cloud prediction. IEEE Robotics and Automation Letters, 8(7):427744274, 2023.
12

---

[33] Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Ziwei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401 24010000444 2024.
[34] Björn Michele, Alexandre Boulch, Tuan-Hung Vu, Gilles Puy, Renaud Marlet, and Nicolas Courty. Train till you drop: Towards stable and robust source-free unsupervised 3d domain adaptation. In European Conference on Computer Vision, pages 1-19. Springer, 2024.
[35] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106, 2021.
[36] Kazuto Nakashima, Xiaowen Liu, Tomoya Miyawaki, Yumi Iwashita, and Ryo Kurazume. Fast lidar data generation with rectified flows. arXiv preprint arXih:2412.02241, 2024.
[37] Lucas Nunes, Rodrigo Mareuzzi, Benedikt Mersch, Jens Behley, and Cyrill Stachniss. Scaling diffusion models to real-world 3d lidar scene completion. In Proceedings of the IEEE/CVP Conference Computer Vision and Patlern Recognition, pages 14770-14780, 2024.
[38] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4195-4205, 2023.
[39] Haoxi Ran, Vitor Guizilini, and Yue Wang. Towards realistic scene generation with lidar diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14738 14748, 2024.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[41] Lloyd Russell, Anthony Hu, Lorenzo Bertoni, George Fedoseev, Jamie Shotton, Elahe Arani, and Gianluca Corrado. Gaia-2: A controllable multi -ietw generative world model for autonomous driving, arXiv preprint arXiv:2503.20522, 2025.
[42] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
[43] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score- generative modeling through stochastic differential equations. In International Conference on Learning Representations (ICLR), 2021.
[44] Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, and Hang Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36:64318 14-438343330, 2023.
[45] Lening Wang, Wenzhao Zheng, Yilong Ren, Han Jiang, Zhiyong Cui, Haiyang Yu, and Jiwen Lu. Ocesora: 4d occupancy generation models as world simulators for autonomous driving. arXiv preprint arXiv:2405.203333 2024.
[46] Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yunpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du, Jiwen Lu, and Xingang Wang. Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 17850-17859, 2023.
[47] Yuqi Wang, Jiawei He, Lue Fan, Hongxin Li, Yuntao Chen, and Zhaoxiang Zhang. Driving into the future: Multiview visual forecasting and planning with world model for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14749-14759, 2024.
[48] Julong Wei. Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, and Wenchao Ding. Ocellama: An sccupancy-Language-en arXiv preprint arXiv:2409.03272 2024.
generative world model for autonomous driving.
[49] Xinshuo Weng, Junyu Nan, Kuan-Hui Lee, Rowan McAllister, Adrien Gaidon, Nicholas Rhinehart, and Kris M Kitani. S2net: Stochastic sequential pointcloud forecasting. In European Conference Computer Vision, pages 549-564. Springer, 2022.
13

---

[50] Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting, arXiv preprint 000Vv:2301.00493, 2023.
[51] Yuwen Xiong, Wei-Chiu Ma, Jingkang Wang. and Raquel Urtasun. Ultralidar: Learning compact representations for lidar completion and generation. arXiv preprint arXtv:2311.0144,, 2023.
[52] Ziyang Yan, Wenzhen Dong, Yihua Shao, Yuhang Lu, Liu Haiyang, Jingwen Liu, Haozhe Wang. Zhe Wang. Yan Wang, Fabio Remondino, al. Renderworld: World model with self-supervised 3d label. arXiv preprint arXiv:2409. 1356, 2024.
[53] Zetong Yang, Li Chen, Yanan Sun, and Hongyang Li. Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14673-14684, 2024.
[54] Lingfeng Zhang, Yuecheng Liu, Zhanguang Zhang, Matin Aghaei, Yaochen Hu, Hongjian Gu, Mohammad Ali Alomrani, David Gamaliel Arcos Bravo, Raika Karimi, Atia Hamidizadeh, et al. Mem2ego: Empowering vision-language models with global-to-ego memory for long-horizon embodied navigation. arXiv preprint 2rXiv:2502.142555 2025.
[55] Lingfeng Zhang, Yuening Wang, Hongjian Gu, Atia Hamidizadeh, Zhanguang Zhang, Yuecheng Liu, Yutong Wang, David Gamaliel Arcos Bravo, Junyi Dong, Shunbo Zhou, al. Et-plan- bench: Embodied task-level planning benchmark towards spatial-temprral cognition with foundation models. arXiv preprint arXiv:2410. 14682, 2024.
[56] Lunjun Zhang, Yuwen Xiong, Ze Yang, Sergio Casas, Rui Hu, and Raquel Urtasun. Copilot4d: Learning unsupervised world models for autonomous driving via discrete diffusion. arXiv preprint arXiv:2311.0111/ 2023.
[57] Yumeng Zhang, Shi Gong, Kaixin Xiong, Xiaoqing Ye, Xiao Tan, Fan Wang. Jizhou Huang, Hua Wu, and Haifeng Wang. Bevworld: A multimodal world model for autonomous driving via unified bev latent space. arXiv preprint arXiv:2407.05677 2024.
[58] Wenzhao Zheng, Weiliang Chen, Yuanhui Huang, Borui Zhang, Yueqi Duan, and Jiwen Lu. Oceworld: Learning 3d occupancy world model for autonomous driving. In European conference on computer vision, pages 55-72. Springer, 2024.
[59] Lipu Zhou, Daniel Koppel, and Michael Kaess. Lidar slam with plane adjustment for indoor environment. IEEE Robotics and Automation Letters, 6(4):7073-7080, 2021.
[60] Yang Zhou, Hao Shao, Letian Wang. Steven Waslander, Hongsheng Li, and Yu Liu. Smart- pretrain: Model-agnostic and dataset-agnostic representation learning for motion prediction. arXiv preprint arXiv:2410.08669 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our core innovation: the idea of "more efficient and portable world model" has been corroborated by subsequent experiments.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We address our model's limitation when we discuss that sometimes when the amount fine-tuning data is close to that of the pretraining data, from-scratch training could actually yield slightly better performance then fine-tuning model. We discuss the model's computational effiency of the model when performing Gflops, parameters, and FPS analysis for semantic occupancy forecasting model.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex- system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: Our model is designed based on proved theoretical concepts (i.e. conditional flow matching) and applying them into designing foundational model architecture.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: In the paper, we discuss our experiment and model setup including some important selection hyperparameters and model dimensions. Also, will release our code and pretrained weights to the foundational model for reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Yes, will organize our repo and share the code, data, and weights for the model to be tested and verified. We won't release code during submission time for anonymity.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.ce cc/
httgg/dd//oodeuuumitii ctaaumiissiooooicyyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nips. https/ccpcpcbgggddeeeeeeeeeeeeee geopod/iiggudi//eeeeeeeooe werdeesuumisiioooooiiioo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we include all the data configurations, model configurations, and hyperpa- rameters such as LoRA rank in the paper. We also provide justification to most important hyperparameter selections in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We conducted sensitivity analysis under sampling randomization.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention the computation power required to train the model and run the inference, as well as the total number of parameters reference.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported didn't ' make it into the paper).
paper (e.g., preliminary or failed experiments that
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: ptttps://neurippi
Answer: [Yes]
Justification: All the authors the paper have read the NeurIPS Code of Ethics and ensure that every protocol/regulations is followed through the entire writing and submission process.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
18

---

Answer: [Yes]
Justification: in the conclusion, we mention how this world model can be implemented into navigation planning models autonomous agents, which ultimately will benefit the society.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model conditional generative/foreccsting model which does not have high risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we have included all the paper that either has been working on similar fields lays theoretical framework for our research. For the repo, we cite/license all the codes which got our inspiration from/utilized during the research.
19

---

Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswitheee theode. contattsetss has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: this project, we collect an indoor navigation dataset which we include necessary information the collection and pre-processing process. This should count new asset for our project.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our project deals with aublicly-accessible LiDAR datasets and personally collected datasets. So we don't deal with human subjects this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
20

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our model deals with online dataset which does not involve subject studies anything similar.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is not used for any important, original, or non-standard component of the of this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to our LLM policy (https : //tp/r/ee..p//////ee Ccc/onfferenees/22222 20MM/LLM) for what should or should not be described.
21

---

Appendix
Contents
1
Introduction
1
2 Related Work
3 Efficient latent conditional flow matching
3.1
Data compression
3.2
Forecasting with Conditional flow matching
3.3
Improved fine-tuning with representation alignment
Experiments
4.1
Experimental design
7
4.2
Encoder structure exploration and semantic occupancy forecasting evaluation
4.3
LiDAR world model transferability
4.4
Ablation study
10
Conclusion
10
Model Setup and Further Evaluation
23
A.I
Training Setup...
23
A.2
Evaluation metrics
................................
23
A.2.1
Latent space similarity metrics
23
A.2.2
Inception-based metrics
24
A.2.3 Negative log likelihood evaluation
24
A.3 More results on semantic occupancy forecasting task
26
B Dataset introduction
27
C Detailed pretraining and fine-tuning results
27
C.1 High-beam adaptation and indoor occupancy forecasting
27
C.2 Semantic Occupancy Forecasting.
28
More details on representation alignment
30
Analysis on LoRA fine-tuning with different ranks
31
F
Additional ablation studies
32
E.
The data efficiency of proposed VAE structure
32
F.2
VAE and forecasting result break down
32
F33
Forecasting performance as a function of the VAE's compression ratio
32
F.4
Model performance with different fractions of pretraining data
33
F5
CFM performance sensitivity analysis with sampling randomization
34
Limitation and future work
H Visualization
34
H.I
Samples from the pretrained model
34
H.2
Samples from the semantic occupancy forecasting model
36
22

---

A Model Setup and Further Evaluation
A.1 Training Setup
For all flow matching based generative (foundational) model training, different from previous methods [14, 23] that used several thousand training epochs, used 4x RTX 4090 train models for 200 epochs with batch size of 8. We trained the VAE part for 100 epochs with a batch size of 16 if not otherwise specified.
Following prior work[56], we also adopt the Adam as the optimizer with 31 and f3 set to 0.9 and 0.99 for flow matching training, 0.99 and 0.999 for VAE training. We set the weight decay of all normalization layers to 0 and all of the other layers' to 0.001. The learning rate schedule has linear warmup followed by cosine decay (with the minimum of the cosine decay of the peak learning rate). We also use EMA with a 0.9999 decay rate to ensure the updates of parameters are stable. During the training, found that the classifier free guidance is also important in flow-matching based framework. We randomly set 25% of historical latents in batch to during training to let the model learn generate future latents without any conditions. In the sampling phase, we use a typical fuse method as shown in Eq. 3 to get the final output when it's the t-th step, where is set 2.
be 20%
A (o = (1+ 8) po - 8 (21,
33)
Finally, we also noticed that the matching of noise scale and latent scale is important. Different from SD3''s VAE, the value of latent obtained from VAE is actually smaller, with standard deviation of about 0.02, which makes easy to drown the signal in noise if we use standard Gaussian noise in the training phase. Therefore, scaled the compressed latent up by factor of 10 in the implementation.
A.2
Evaluation metries
In this section, we will provide the details all metrics used to further evaluate the performance of proposed pipeline. All results will be presented in the last part of this section.
A.2.1
Latent space similarity metrics
Given Og E RBxWxDxC and its compressed latent z E hxuxx the compression ratio y can be calculated in Eq. 4.
hxwxc
44)
HXW DxC
In Section 4.3, we introduce CKA (Centered Kernel Alignment) [22] and CKNNA Kernel Nearest-Neighbor Alignment [18] to evaluate the similarity of latent spaces both metrics are insensitive to linear transformations. Specifically, we treat each latent pixel (location) as an individual sample (i.e., every sample in CKA/CKNNA evaluation can be represented as E RRxx).. For the non-semantic semantic adaptation subtask, given samples from latent, the representation metrics from Oa and o, can be denoted by X uxxe and Y paxe respectively. We first construct the kernel matrix:
X x2 20
| 1
K₁ exp
Lij = exp
(5)
20
Then the CKA can be calculated as:
tr(K.
CKAARF X, Y)
(6)
Vtr(K K.K,) ir L.)
where K. and L. are centered K and L, using K, HKH, H 11, 1, an all-ones column vector. CKNNA is revised version of CKA that focuses the local manifold similarity.
23

---

For every sample i in K. and Le. we use cosine similarity to find out the nearest k neighbor in original samples, let knnk, and knn,, be the indices of its nearest neighbors, respectively. Then we can define mutual-KNN mask a(i;j) by:
1, j E knnk. (i) oi
A j E knn (1),
i i j-
0, otherwise,
Local alignment is Align KNN (Key Lo) = Ei Ej= ali, Ka(t,j) Lc( CKNNA is:
Ali Ko Lc)
CKNNA(K,L)
(8)
Align Ke Ke) Aligm (Le Le)
Use k to represent the number of neighbors we selected, when h: + ais for all off-diagonal pairs and CKNNA reduces to the standard CKA.
A.2.2 Inception-based metries
Fidelity measurement Following [6], report both the 3-D Fréchet Inception Distance (FID) and the Kernel Inception Distance (KID, i.e., the squared Maximum Mean Discrepancy in feature space). Unlike FID-which assumes the Inception features form single multivariate Gaussian and thus compares only the first two moments KII) employs a characteristic RBF kernel and provides an unbiased estimate that is sensitive to discrepancies in all higher-order statistics.
To obtain latent features from generated and ground-trutt samples under comparable conditions, Og. The autoencoder is based on the MinkowskiUNer322 architecture (sparse)[1 I], but we adapt it to our 200 x 200 x 16 occupancy inputs by reducing the feature channels from (64, 128, 256, 512] with four down-sampling stages (32, 64, 128) with three down-sampling stages. The training objective is to recover the compressed latent as much as possible under a cross-entropy loss.
we retrain autoencoder
For the output frames of future semantic occupancy, we evaluated frame-wise FID and KID. Specifically, first extracted all non-empty voxels from the down-sampled semantic occupancy (size 25, 25, 2) and took the average, rather than performing global pooling as in previous work, to avoid the influence of empty voxels on the metrics. Given the flattened feature z and 25 from individual samples, we use μ², c and E9, Ee to represent the mean and channel covariance matrix over M ground truth samples and N estimated samples. Then we calculate FID with the following equations.
|| - 1 Th > 2(> 1/3
FID
For KID, we use a RBF-kernel-base bobiassd-ssttistee estimation version as shown in equation 10.
KID(P,Q) = MMD (P.Q)
1
2
Σ Eighs
to (2) 20)
Σ 37'"
ho (o(2f
Σ Σ
N(N
M 1)
NM
(10)
where hi (x. y) = exp -1|x y1l2/ v1l2/(20z)).
Temporal Consistency Assessment FID and KID provide assessments of single-frame fidelity, whilst for multi-frame continuity we adhere previous methodology by employing FVD for eval- uation. Given that no existing RGB video encoder can capture this 4D temporal information, we modified the proposed VAE to enable it to measure the continuity of 4D occupancy. Within the encoder and decoder shown in Figure 2, we respectively incorporated 3D attention/cony layers to capture temporal information.
A.2.3 Negative log likelihood evaluation
Computation of exact log probability density Due to the stochastic nature of our proposed conditional flow matching model, pairvise--amplee-cmpaiiss metrics such as IoU and mloU
24

---

provide a limited measure of model quality, since these "deterministic" metrics penalize models for generating diverse but plausible predictions of the future that are substantially different from the recorded future. In other words, the model's ability to capture the uncertainty in future prediction is not measured well by the loU and mloU metrics. Therefore, we also evaluate the exact log probability of our CFM model that doesn't tt penalize it substantially for assigning probability density to other modes.
With our CFM model, Ga. we can compute the log probability of any generated future prediction or future ground-trutt ttuuh samples with the following methods. First, as described by previous works [10], the generative process of a continuous normalizing flow works as described below: starting from sample from a base distribution 2 E P=., (20) and a parametrized ODE Ge G( t: f) which is
the flow function, we can obtain (t)) from the target distribution by solving for the initial value
problem =(4)) 20. 8t
@z(t) G. In this process, the rate of change of fog-probability density follows the instantaneous change of variables formula [10]:
i log p(z)) -Tr( dz(()
0G
(11)
Ōt
With this equation, the total change in log probability density from to to 1 is calculated by integrating with respect to time:
log p(2(()) log logr(=(to)) - S Tr(
8G Oz(()
) dt
(12)
Given any sample x in the target distribution, we can compute zn that generates x and the log likelihood of by solving for the following IVP [10]:
[
20 J G(z(t),t;0)d dt
(13)
logp(x) - log P:o (2o) = 1r -Tr( 8G dt az())
with z(41) = x and log p(z) - log 0.
After obtaining the change in log probability density, we can add this change the log probability of the prior distribution to determine the exact log likelihood of x.
Practically, what is done for FFJORD [13], the trace of the Jacobian of the flow function can be approximated with the Hutchinson's 's trace estimator which takes o(n). Moreover, we use a standard Gaussian the base distribution which gives:
log Pzo (20)
211 D (log(2r)
(14)
where D is the number of scalar dimensions of the sample. This enables to determine the exact log likelihood of any generated sample or sample from the future occupancy latent space (any sample from the future occupancy distribution) numerically.
Details on NLL evaluation For all log likelihood values computed, we use an Euler ODE solver with step size of 0.02, relative and absolute tolerance of 1 x 10-5. The trace of the Jacobian is obtained by utilizing Hutchinson's trace estimator with the probe vector sampled from a Rademacher distribution.
After obtaining the exact log probability, we evaluate the negative log likelihood in the form of bits-per-dimension (BPD) which is obtained by:
BPD(z) log p(x)
(15)
Dln
where D is the number of scalar dimensions of the latent sample x and the division by In 2 converts the unit to bits.
DDPM log likelihood computation For discrete 3DPM-based model, we need to construct and solve the IVP introduced previously with mathematically equivalent representation for the flow function G.
25

---

For model (i.e. DOME [14] which is constructed based on 1000-step DDPM) whose noise variances {33k 1 -N are linearly spaced, to reuse the same parameters for likelihood evaluation, we can embed this chain the picce-wise-constant variance-preserving SDE
1 ((t) dt + BB(() dwy, B(t) BR for tE (N- dx;
k+1
(16)
2
We can write its deterministic form [43]
1 +. G t: B(t) 2; 3(t) ((t) (x,,)))
1
(17)
with score (2 1) = - (æt, (0,,)//((0 (1). With this approximation of the flow function, the IVP can be constructed and solved following the same procedure as the CFM model.
A.3 More results on semantic occupancy forecasting task
The NLL values terms of BPD and the corresponding standard deviation (across the entire validation set; use ground-truth samples not the generated predictions) obtained from the semantic occupancy forecasting task are shown in table 8. Our semantic occupancy forecasting model conditioned on historical/past trajectory achieves comparable NLL values with DOME [14]: only 0.25% absolute difference (4.1% 1% relative) while DOME utilizes future trajectory in the condition. For our model conditioned future trajectory, the NLL value is lower than that of DOME by 25% relative difference (1.49% absolute). This shows that our CFM model better fits the true occure-oy distribution than the previous (stochastic) SOTA model.
In terms of standard deviation NLL., for our models (conditioned on either history or future trajectory), the standard deviation is at least 86% (relative) less than that of DOME. This statistical metric further supports the conclusion drawn above by demonstrating the high consistency of our model's performance in terms of assigning high probability to correct futures.
Table 8: Mean negative likeliidl for different semantic occupancy forecasting models.
Model
Mean NLL (bits/dim)
Std. NLL
Ours (Hist. traj.)
6.29
0.04
DOME (Fut. traj.)
6.04
0.28
Ours (Fut. traj.)
4.55
0.02
Different from image, LiDAR occupancy from a BEV perspective has a clear depth correspondence between grid cells and environments. However, the global pooling operation will take the average of features in all grid cells, agnostic to geometric location. Here, inspired by [39], we also measured the FID and MMD features from different depth bins. Specifically, in the obtained 2 E 25x2522 every cell stands for 3.2m x 3.2m area in the environments. Based on the distance from the center point, we divide the area into three zones: within +8 m, #8 to +24 m, and +24 to +40 m. We then take the average of the non-empty voxels in each zone and concatenate them to get the final feature for evaluation, denoted as FID, and KID, Table
Table 9: FID, and KID, (x10-2) comparison for our method and other generative- and deterministic-baseed models.
1s
2s
3s
AVG
Method
FID,-
KID,
FID,-
KID,
FID,,
KID,
FID,
KID,
OccWorld DOME
46.20
6.98
44.91
6.51
44.00
6.23
45.03
6.57
18.09
2.28
21.56
2.87
25.03
3.46
21.56
2.87
Ours (Hist. traj)
7.20
0.78
12.39
1.47
17.80
2.49
12.46
1.58
Ours (Fut. traj)
6.71
0.74
12.29
1.54
16.82
2.32
1 11.94
1.53
Our model establishes a new state of the art on both FID and KID. On average, for our model without future traeectories, it reduces FID to just 28.3% and KID to 22.7% of the
26

---

scores reported by the deterministic (OceWorld) and stochastic (DOME) models, respectively. The advantage remains under the stricter metries FID, and KID,: our method attains only 23.36% (FID,) and 19.30% (KID,) of the corresponding values.
Interestingly, we observed that the FID,/KID, of OccWorld decreases slightly (around 2%) when we forecast longer-term future results. We attribute this to the auto-regressive displacement forecasting design in OccWorld, which preserves more diversity at longer horizon, although the overall performance is lower. Specifically, we calculated the category variance at different time horizons (all numbers follow in x x):->;; : for the ground truth, the variance fluctuated between 3.055 and 3.021, while in Occworld, it rose from 2.865 at t=0.5 seconds to 3.038, and our model fell from 3.086 2.821.
B Dataset introduction
As mentioned earlier, we used nuScenes Semantic Occupancy (o.), KITTI360 raw sweeps[25]. and our own collection of indoor LiDAR data test the three subtasks. In the semantic occupancy forecasting task, we further divide nuScenes semantic occupancy data into pretrain set (o') and fine-tuning set. For pretraining, we use voxelized LiDAR sweeps in the nuScenes dataset. In Table 10, we list the specific number of frames of all these datasets after down-sampling to 2Hz
Table 10: Training-validation set splitting for each dataset
Dataset
training set
validation set
O2 d'. KITTI-360 Indoor LiDAR Dataset
19728
4219
11208
4219
12093
4912
23504
9720
nuScenes LiDAR sweep
22632
4368
Specifically, for the indoor LiDAR dataset, it consists of 25 sequences that we collected in 2 different scenarios, using an OSO- LiDAR sensor with 90-degrees vertical field of view. robot, modeled with unicycle kinematics, was manually driven through a university building over the course of five days. An expert human operator controlled the robot, skillfully avoiding pedestrians while navigating toward designated goals. As shown in Figure 6, compared to the data in the autonomous driving scenario, where the vast majority of vehicles move parallel to ego vehicles, the data we collected includes more complex trajectories with more objects (people).
As we mentioned in the main text, the amount of data after downsampling is still large considering that we only collected this data in 2 scenes. This makes it easier for the model to learn domain-specific knowledge and explains why the performance our pretrained model drops after using more than 50% of the data.
C Detailed pretraining and fine-tuning results
In this section, we explain more details about the results from experiments on how different pretrained/from-seraae VAE and CFM combinations perform on each of the three subtasks, which, again, high-beam LiDAR adaptation, indoor occupancy forecasting, and semantic occupancy forecasting. For LoRA[17], across all our downstream tasks, we choose a rank of 128 and a of 128. Please check section E for details this selection. All models present in this section are trained for 40 epochs.
C.1 High-beam adaptation and indoor occupancy forecasting
In Table and Table 12, we organize the performance of each the pretrain/from-scrate combi- nations (VAE pretrained and CFM pretrained and fine-tuned with LoRA[17], both pretrained and full-parameter fine-tuned, only CFM pretrained, and both from-scratch) for high-beam adaptation and indoor occupancy forecasting in terms of loU. IoU is computed through inferring the trained CFM
27

---

(a) An enclosed indoor corridor with pedestrians walking parallel to the rohot's direction of motion.
MOUGH
(b) indoor intersection with complex pedestrian trajectories. Figure 6: Indoor dataset: We collected over 200K raw point cloud frames (was down- samn-sd to around 23000 frames in training) at two indoor locations, incorporating complex movements of the crowd and changes in the indoor scenery.
model with NFE=10. For the high-beam adaptation task, the pretrained CFM model is trained on original LiDAR sweeps, denoted by Gjj- From the result, it is evident that full-parameter pretrained (both or only-CFM, with without LoRA) yields exceeding performance over from-scratch trainings. Specifically, at 10% of fine-tuning data usage, our pretrained model with full parameter fine-tuning achieves more than 14.48% relative performance improvement (more than 1.5% absolute performance improvement) across the three-second forecasting horizon and even at 100% data usage, our model still offers an absolute performance growth of about 1%. This demonstrates the effectiveness of our pretrained foundational models in transferring geometric knowledge learned from 32 beam LiDAR data tasks that demand different geometric properties (64 beams in this case) at different availability of fine-tuning data.
For the indoor occupancy forecasting task, we use the same pretrained CFM model as the high-beam adaptation task. From the result, it is shown that at low data availability for fine-tuning (10% and 25%), our pretrained models achieve exceeding performance over the sernsc training. But when the amount of fine-tuning data increases to more than 50%, the scrmtsccrachh training obtains comparable or even better performance. Again, we explain this performance threshold be resulting from the fine-tuning data amount which overrides the pretraining data and the lack of variance in the geometry of the indoor occupancy dataset collected.
C.2 Semantic Occupancy Forecasting
For semantic occupancy forecasting task, the pretrained CFM model is trained on densified Li- DAR frames, denoted by G The results with numerical value of mloU and IoU for different prerrainedffrom-scraadd VAE and CFM combinations (NFF=10) at different fine-tuning data fractions are presented in Table 13. is shown that 10% of fine-tuning data utilization, our pretrained VAE and CFM with full-parameter fine-tuning provides 1 1.17%% absolute forecasting performance improvement in mloU. In general, even the performance gain decays as the amount of fine-tuning data usage increases (which makes sense as it is getting closer and closer to the amount of pretraining data), the effectiveness of our foundational model prevails across all tne-ttnnng data fractions (at 100%, we still have about 1% performance improvement over the from- from-scratch model), One key thing to note that we only fin-ture/rrain-from-ssee scratch our semantic occupancy forecasting model with half of the available semantic labels for nuScenes (50% of the data used for the same task for other baseline models). This additional splitting of the training set is to prevent the overlap between pretrain and fine-tuning data in terms of geometric structure, both densified occupancy forecasting
28

---

Table 11: High- adaptation fine-tuning results.
Data Fraction
Horizon
LoRA
Full-parameter
Only CFM-pretrained
Train from scratch
10%
1s
18.57
20.69
19.23
16.42
2s
14.03
15.11
14.09
12.27
3s
11.77
12.33
11 11.49
10.77
25%
Is
22.70
21.74
20.45
18.53
2s
16.68
16.23
14.97
13.92
13.77
13.03
12.00
11.69
50%
Is
23.70
22.78
21.95
21,43
17.70
17.17
16.21
15.84
14.60
14.02
13.17
13.01
100%
24.16
24.50
24.00
23.40
18.50
18.28
17.66
17.32
38
15.39
14.93
14.16
14.08
Table 12: Indoor occupancy forecasting fine-tuning results.
Data Fraction
Horizon
LoRA
Full-parameter
Only CFM-pretrained
Train from scratch
10%
Is
29.40
30.40
29.73
27.28
2s
26.60
26,45
25.82
24.96
3s
25.41
24.99
24.23
23.92
25%
Is
33.06
33.38
32.90
32.34
2s
28.70
29.36
28.70
29.10
26.93
27.55
26.70
27.44
50%
1s 33.58
34.27
34.08
35.13
28.97
30.19
29.73
31.24
3s
27.11
28.21
27.71
29.17
100%
34.49
35,64
35.32
36.52
2s
30.11
31.71
31.07
32,50
3s
28.18
29.60
28.98
30.48
and semantic occupancy forecasting models share the same set of scenes and the difference that semantic occupancy forecasting data is labeled by category. This means that 10% of fine-tuning data usage, having both VAE and CFM full-parameter fine-tuned based on our foundational model is and Oce World 1.5%% absolute performance difference in both IoU and mloU across the three-second forecasting horizon) with only 5% of the data it consumes. Please refer section H for visualizations of the forecasting results.
able achieve comparable performance with
Table 13: Breakdown fine-tun finettnning results on the semantic occupancy task.
Data Fraction
Forecasting Horizon
LoRA
Full-parameter
Only CFM Pretrain
Scratch
IoU
mloUi
IoU
mloU
IoU
mloU
ToU:
mloU
35,92
23.29
36.24
25.95
24.09
27.00
13.52
10%
2s
26.01
14.19
26.17
14.91
25.61
14.63
19.41
8.25
3s
20.60
9.80
20.87
11.09
20.24
10.21
15.81
6.14
1s
37,21
27.42
37.85
28.88
36.64
27.80
33.84
22.73
25%
27.46
17.23
27.09
17.58
26.95
17.03
24.62
13.65
22.04
12.31
21.36
12.30
21.61
11.84
19.29
9.38
37.17
28.47
38.40
30.84
37.09
28.23
37.02
28.10
50%
2s
27.59
17.48
27.81
19.00
27.53
17.64
27.30
17,50
38
22.31
12.44
22.14
13.62
22.29
12.58
21.80
12.34
1s
37.13
28.83
38.91
31.64
36.59
28.39
38.21
30.54
100%
2s
27.80
17.56
28.58
19.59
27.46
18.01
28.22
19.06
38
22.65
12.52
23.13
14.22
22.50
13.02
22.31
13.45
29

---

D More details on representation alignment
As we mentioned before, during the pretraining phase, we found that it is more appropriate to use new data in fine-tuning the original VAE as opposed training VAE from scratch, even though the performance of our vae trained from scratch was excellent. We attribute this to the fact that the pretrained generative model in fact more adapted to the data distribution of the original latent, whereas the subspace structure of a latent learned from scratch with fine-tuned data would in fact be very different.
Specifically, for sub-task considering the inconsistent size of the embedding layers. Instead, we use the approach in Fig. considering that o, and match one-(o-one,, and the original VAE is fine-tuned on Op, we can directly constrain the direct outputs of the encoder: mu and sigma. Previous work has tried to use bidirectional KL or EMD distance as a loss, but this does not work in our case: the data for the two modalities are already very close, differing only by semantic information.
mansemaatic-t--semaale transfer), we cannot directly fine-tune the VAE
Table 14: Comparison of feature alignment metrics before and after alignment.
Dual KL
EMD
CKA
CKNNA
CosSim
Dense-Sem feature Dense-Sem (Aligned) feature
0.004
0.008
0.739
0.278
0.183
0.002
0.004
0.944
0.410
0.907
Therefore we compared the latent from pair-wise Ou and o, in line of Table 14. We can see that both DL. and EMD are very close, only cosine similarity is very different, which actually show us that the most significant difference in the latent learned from these two VAEs is in fact the direction of the high-dimensional tensors. Then after using cosine similarity to constrain the o, VAE learning, the result shown in line 2. With the tensor orientation restriction, we can successfully align the latent of the two VAEs, and the subsequent generative model training demonstrates that our alignment scheme can further improve the results.
pizglos)
((GilRg)
Fine Fine-turning Data
Pretrain Data
Pairette samplewwwa
Representation Align
VAE from
Pretrsined Fine-turned VAE
Pretrain
Pretrained
((rglog)
@Gka log)
(ipyloc)
VAE
Zp
Za
Zold
Figure Use pretrained VAE for o to implicitly guide the structure of o, latent space
Figure The comparison of latent space struc- ture for non-paired data (in subtask 1 and 2)
For different beam LiDAR adaptation and outdoor-indoor generalization, while we can directly fine- tune the VAE, we cannot directly compare the similarity of the subspaces (CKA[22) and CKNNA[18) require the same or pair-wise wise samples inputs to different networks). Our solution is shown in Figure First, we use the original pretrained VAE, denoted by q(Zaloa) to get the latent of fine-tuning data without any training (adaptation), denoted by Zold- Then we use the fine-tuned and training from scratch VAE to obtain the 26 and 2,. If our hypothesis is correct, the distance between (Za, Zold) should be smaller than (z,, Roid).
As data shown in Table 15 and 16, we listed the (Za, 2old) and (2g, Zola) in every continuous lines for different fractions of fine-tuning data. We observe that with the increase of the data amount used to fine-tuning the VAE, we can have better results in terms of IoU, even better (when using 100% high beam KITTI data) very close to the pretrained one. However, what really determines the forecast performance actually the similarity between the latent space after fine-tuning the VAE and the latent space after only pretraining. Compared to training from scratch, the latent space after
30

---

fine-tuning the VAE is more similar to the latent produced by using pretrained weight directly, which allows for better use of the knowledge from the foundational model. other words, fine-tuning the VAE is in fact about leaving the original latent space as unaltered as possible while allowing the weights of the original VAE be adapted (and accurately compressed) to the new sample.
Table 15: VAE fine-tuning evaluation in differ- ent beam adaptation
Table 16: VAE fine-tuning evaluation in out- door to indoor adaptation.
Pretrain
Fine-tuning
IoU
CKA
CKNNA
Pretrain
Fine-tuning
IoU
CKA
CKNNA
Oy
10% KITTI
97.97
0.661
0.231
10% Indoor
95.82
0.739
0.245
Ø
10% KITTI
85.08
0.477
0.171
2
10% Indoor
88.77
0.592
0.186
25% KITTI
98,16
0.625
0.214
Od
25%
Indoor
96.65
0.720
0,233
Ø
25% KITTI
88.66
0.489
0.188
25% Indoor
92.14
0.606
0.199
0y
50% KITTI
98.31
0.610
0.211
0d
50%
Indoor
97.15
0.691
0.221
Ø
50% KITTI
90.79
0.495
0.188
50%
Indoor
94.09
0.611
0,203
0g
100% KITTI
98.40
0.611
0.208
Od
100%
Indoor
98.18
0.688
0.220
Ø
100% KITTI
98.46
0.494
0.189
Ø
100% Indnor
95.97
0.609
0,200
E
Analysis on LoRA fine-tuning with different ranks
To determine the optimal LoRA[17] configuration for our downstream tasks, tested the following ranks: 32, 64, and 128 with a = rank and a dropout rate of 0.05 on the semantic occupancy forecasting subtask with 10% of fine-tuning data and both VAE and CFM components pretrained as explained earlier. We apply LoRA only to the linear layers, embedding layers, 2D and 3D convolution layers in the CFM architecture. The training is done for 40 epochs for LoRA and full-parameter fine-tuning baseline. The goal of this mini-scale experiment is to find the balance between model performance and parameter efficiency. The result is shown in table 17. First, after sampling the CFM, the model yields suboptimal performance under the rank of 32 and 64, with 3.51% % and 2.48% absolute performance gaps with the full-paree aalammrretttrr fine-tuning in terms of mean mIoU across three- second forecasting horizon, respectively. This indicates that the model does not acquire satisfactory level of semantic information from fine-tuning data within the 40 epochs training process which might be caused by lack of depth in the LoRA layers. On the other hand, rank of 128 demonstrates performance that is closest the full-panc pal-merreettr fine-ture mun-nining: 0.25% less in mean IoU and 0.91% less in mean mloU. This shows that rank of 128 offers enough depth for information to be extracted from the semantic occupancy data and demonstrates comparable performance against the baseline. This rank and combination is what we presented earlier for all downstream tasks under the "LoRA" fine-tuning category.
We did not test any higher or lower rank because: since both ranks of 32 and 64 have a considerable performance gap compared the full-parameter fine-tuning, lower rank means even less parameters (depth) for LoRA layers which makes the model less likely to learn useful information from the fine- tuning data and consequently, is unlikely achieve comparable performance with the -al-mearreete method: on the other hand, rank of 128 already has 10.21 million trainable parameters in total where our CFM model alone only has 11.51 million parameters. This means that under such rank, we are fine-tuning about the same number of parameters as the full-parameter fine-tuning approach.
Table 17:
Comparison of different LoRA configurations (rank and values) and full parameter fine-tuning for semantic occupancy forecasting task. We report the method type, number of trainable parameters, mean IoU, and mean mloU. Note that our parammer CFM models only have 1 11.51 M parameters, excluding VAE which has 18,86 M parameters.
Method
Rank
a
Parameters (M)
mean IoU t
mean mloU 1
Full-parameter
11.51
27.76
16.67
LoRA
32
32
2.55
26.27
13.16
LoRA
64
64
5.10
27.14
14.19
LoRA
128
128
10.21
27.51
15.76
31

---

F Additional ablation studies
F1
The data efficiency of proposed VAE structure
Follow the official split of train/validation under different fractions of training data. As shown in Table 18, our method only needs half of the training data to exceed all of the previous data compressors, as we mentioned in the abstract.
in nuScenes, we test the proposed VAE performance
Table 18: Performance of proposed VAE under different training data fraction, under 192x compres-
sion rate, 100 epochs training
Data fraction
IoU
mloU
100%
85.8
93.8
50%
83.4
92.1
25%
82.0
90.2
10%
78.9
85.8
F.2 VAE and forecasting result break down
Table 19: Per-class loU and overall metrics across different semantic occupancy compression models.
Coost. Veh.
Motorcyete
Pedestrian
Traffic Cone
Drive. Surf.
Other Flat
Man mo---adde
Vegetation
Sidewalk
Others
Barrier
Bicycle
Traller
Truck
Termin
Bos
Car
Method
Comp.
mIoU
IoU
OccWorld([88]
16
65.7
62.2
45.0
72.2
69.6 68.2 69.4 44.4 70.7 74.8 67.6 56.1 65.4 82.7 78.4 69.7 66.4 52.8 43.7
OccSor[[45]
512
27.4
37.0
11.7
22.6 0.0 34.6
29.0 16.6 8.7
11.5 3.5
20.1 29.0 61.3 38.7 36.5 31,1 12.0 18.4
OccLLAMA[48]
16
63.8
93.5
77.3 75.1 60.8 90.7
88.6 91.6 67.3 73.3 81.1 88.9 74.7 71.9 48.8 42.4
DOME]14
64
83.1
77.3
36.6
90.9
95.9
85.8 92.0 69.1 95.3 96.8 92.5 77.5 85.6 93.6 94.2 89.0 85.5 72.2 58.7
Ours
192
92.8
85.8
88.5 97.8 97.5 93.7 96.0 86.2 98.4 97.6 97.6 92.1 94.7 97.2 98.5 95.8 94.8 83.6 68.3
In Table 19, we present the compression performance of the proposed VAE across different semantic categories. With 3x compression ratio, our method consistently outperforms prior approaches in all categories. Notably, for rare classes such as Construction Vehicle, Trailer, and Vegetation, our model achieves improvements of 22.2%, 18.8%, and 25.4% over DOME's VAE, respectively. These results indicate that our VAE effectively mitigates the impact of data imbalance across categories in the compression process.
Table 20: 3 seconds average per- class IoU for semantic occupancy forecasting
Const. Voh.
Motorcycle
Podestrion
Traffic Cone
Dovve Surf.
Other Flat
Mon-nade
Vegetation
Others
Barieer
Bicycle
Trailer
Truck
Sidewalk
Terrain
Bus
Car
Method
mioU
IoU
OccWorld[58]
17.13
26.63
12.23
20.77
8.27 20.50 19.86
12.58 7.89 8.95 8.45 13.04 17.73 35.09 23,65 23.97 20.66 17.01 20.17
Ours (Hist. Traj.)
23.33
31.78
21.41
23.70 15.33 26.00 23.05
27.44 13.19 10.29
13.10 21.89 24.85 40.68 30.99 29.29 26.64 21.52
26.57
DOMB[14)
22.18
32.13
19.84
25.66
15.36
21.03
21.98
23.96
11.36
7,99. 14.79
18.02
21.58 39.84 30.46 28,
25.35
23.01 27.22
Ours (Fut. Trajo)
28.49
37.52
28.38
33.77
19.99 29.81 28.17 32.46 18.76 12.08 20.78 25.59 30.85 43.27 34,01 32.65 29.34 30.10 34.26
Now, we present a per-category performance analysis. In Table 20, we compare the average IoU for 3s under each category. Compared Oce Dcc for medium-sized objects is particularly significant: we improve the forecasts of bicycle, Const. Veh. and Motorcycle by 85.36%, 118.1 118.128 12% and 67. 17%, respectively. A broad performance gain is also observed when including future trajectory as condition, the top 2 improvement categories are also small objects, relatively 68.18% and 51.2% for Motorcycle and Pedestrian respectively.
improvement in forecasting performance
E33 Forecasting performance function of the VAE's compression ratio
We present the reconstruction rates of the VAE architecture at various compression ratios in Table Then further question arises: which category of information is more susceptible to loss? In Table
32

---

21, we show the performance drop for 4 listed reconstruction rates. Interestingly, we notice that the
Table 21: Proposed VAE per category reconstruction rate break down for different compression ratio
Coms,. Vola
Motorcylle
Polesttio
Traffic Cone
Drive. Surf
Sidewalk
Man ande
Vegetution
Others
Bayyloo
Trailer
Car
98 61
99.00 09.68
99.07
97.87
x192
88.51
97 80
97.45
93.63
96.00
84.19
98
97.65
97. 68
89.12
92.71
97.27
98. 56
94.40
93.86
83.66
68.27
x384
79.56
95.14
96.03
87.75
88.94
75.26
96.34
95.28
95.33
81,89
85.67
93.01
96.61
86.94
86.61
72,52
58.32
90.77
79.57
78.89
90.49
87.73
76.63
90.74
77.29
76.03
62.53
Drop (x32->x192)
10.05
1.58
1.16
5.91
3.68
13.50
1.17
L81
1.39
9.72
6.50
2.44
1.03
4.81
4.97
14,21
25.39
Drop (x32-ox3344)
19.00
4.24
2.58
11.79
10.74
22:43
3.25
4.18
3.74
16.95
13.54
6.70
2.98
12.27
12.22
25.35
35.34
Drop (x32.xx768)
32.42
10.64
7.84
19,97
20.79
35,32
9.10
11.39
11.34
28.87
22.58
8.85
21.92
22.80
38.34
voxel missing in the small and rare foreground objects much less severe than in the background (vegetation, manmade) and large objects (construction vehicle) during compression ratio increase, while these small targets are more important for safe driving.
Next we show the forecast performance as a function of VAE for different compression rates. First, the results of CFM based latent with different compression ratios shown below. For time considerations, all CFMs here are trained for 200 epochs. At the highest reconstruction rate of x32,
Table 22:
Forecasting result with different compression ratio VAE
Compression
1 loU
1 mloC
2 IoU
mloU
3 IoU
3 mloU
Avg IoU
Avg mIoU
x32
38.40
28.96
28.54
18.52
23.14
13.48
30.02
20.31
x192
40.53
33.17
30.37
21.09
24.44
15.64
31.78
23.33
x384
39.68
32,96
29.95
20.91
24.25
14.98
31.29
22.95
x768
34.66
27.10
24.51
17.30
22.61
12.95
27.26
19.11
we find that the network is not fully fitted at 200 epochs, and longer training time may allow the network to fit better. However, since this is clearly contrary the 'sample/training efficiency' of this paper, we keep the training time the same for fair evaluation. For latents with higher compression multiples, the forecast performance continues to degrade. We believe that this not only due to fact that it retains less information on large objects, but also because at larger compression multiples, latents become very sensitive to noise, i.c., the latent space loses smoothness. We find that at multiples of x384 and x768, noise of the order of lc-4 can crash the network (the magnitude of the latent quantity is approximately between le-2 and le-3).
F.4 Model performance with different fractions of pretraining data
In the previous section, we showed that based on the foundational model, the performance boost when we increase the fine-tuning data amount and also the result w/w.o the foundational model. But another question is whether the performance of the model improves with the inclusion of more pretraining data? We designed a simple experiment answer this question.
30.00
25.00
36 Maantaa
20.00
25.00
10.00
H gisccamm adeptation
Intuor aecupansy fonwassing
Semsanti accupansy frewasiing
50
60
70
80
90
100
Percentap of Pretrain Lata Used tot
Figure 9: Performance improvement from different ratios of pretraining data
We choose 25% of fine-tuning data for different subtasks to simplify the fine-tuning. In Fig. 9, 100% data on the x-axis represents the full pretraining data O,, We observe clear performance
33

---

improvement (relative improvements of 21.9%, 22.2%, and 12.9% for indoor occupancy forecasting. semantic occupancy prediction, and high-beam adaptation tasks) as the amount of pretraining data varies from 50% of O, to 100%.
F.5
CFM performance sensitivity analysis with sampling randomization
The inference process of flow matching is essentially solving the Probability Flow ODE (PF-ODE) as mentioned by previous works[42], which actually deterministic given the sampled Gaussian noise is fixed. Here we designed an experiment which explores the performance change when using different i..d. Gaussian noise as input. Specifically, using the Euler solver, we evaluated the semantic occupancy forecasting model with 5 random seeds and recorded the average and standard deviation for IoU and mIoU across the three-second forecasting horizon (on the validation data).
We present the results in Table 23, and note that the standard deviation of the performance metrics for all three models is less than 1 % of the mean across the entire forecasting horizon. Therefore, we conclude that our CFM model has low performance sensitivity to randomization in the sampling process and the uncertainty around the performance metrics evaluated minimal.
Table 23; Different CFM models and their average IoU/mIoU (#std) over a threc-second horizon, tested with 5 different random seeds.
Model trained on
IoU
mloUt
28
3s
Avg
3s
Avg
Og O,
26.6710.02
21.5610001
18.26:1002
22.26:10.02
39.82.10.13
29.34:1004
23.73:10.77
30.93:0.001
40.6440.10
30.31+006
24.21+40:77
31.7240004
33.57++028
21.15+0.05
14.99+0.66
23.25+0..6
G Limitation and future work
Physical consistency. Although our method achieves state-of-the-arg performance across different time horizons, the issue of continuous consistency in generated videos remains unresolved. For foreground objects, our current solution only achieves smooth output by performing timing processing within the network structure, but in the generated scenes, foreground objects are still prone to discontinuities. As shown in Table 20, although our work has greatly improved the accuracy of the forecasted foreground objects, there is still no theoretical or design guarantee that the foreground object will be continuous the output (i.e., will not suddenly appear or disappear in few frames).
Modtityy forecasting. For a safe end-to-end autonomous driving system, it is still very difficult for the current occupancy world model to not only decode the occupancy map after obtaining the future representation. but also forecast the future multi-modal trajectories of surrounding agents. As we shown in Table 23, the randomness in flow matching or diffusion (originates from Gaussian noise), under given condition, the future forecast outputs are in fact quite fixed. This may be sol solved via more detailed control for foreground and background generation. We hope that this work will provide an efficient implementation framework for solving this issue in the future and accelerate the progress of subsequent work.
H Visualization
H.1 Samples from the pretrained model
Here we visualize the forecasting results for two of the foundational models that are pretrained. Specifically, visualization for sparse occupancy forecasting is shown in figure 10 and visualization for dense occupancy forecasting shown in figure 11. Please check more visualizations on our project page.
34

---

t= 15
t= 2s
t= 38
Ours
GT
Ours
GT
Figure 10: Visualization for the sparse occupancy(O.)) forecasting over the three-second forecasting horizon.
t= 1s
t= 2s
t= 3s
Ours
GT
Ours
GT
Figure 11: Visualization for the dense occupancy(o.) forecasting over the three-second forecasting horizon.
35

---

H.2
Samples from the semantic occupancy forecasting model
t= 1s
t=
2s
3s
Ours
OccWorld
GT
manmade
vegetation
lerrain
sidewalk
driveable surface
traffic pedestrian
other flat
Visualization for the semantic occupancy(O.) forecasting over the three-second forecasting horizon. Compared to the previous method, our method retains more details this example (as shown in the red circle)
Figure 12:
t=
t= 2s
t= 3s
Ours
OcoWorld
GT
manmade
vegetation
terrain
sidewalk
driveable surface
traffic pedestrian
other flat
Figure 13: Visualization for the semantic occupancy(O_,) forecasting over the three-second forecasting horizon for intersection. Compared to the previous method, as shown by the yellow circle, our methods better forecasts foreground (close-ranged) objects such as cars and barriers.
36


Reviews:
[{"review_id": "HpBJszduqa", "reviewer": "Reviewer_N8qa", "summary": "The paper explores building foundational LiDAR world models, transferability to downstream forecasting tasks and efficient data compression via CFM.", "questions": "1. What are the challenges of buiding a foundational LiDAR world model and how do you address them? The reasoning is missing in the introduction. Alternatively, can augmenting a powerful, existing image-based foundational model (like GAIA-2 or Cosmos) with a LiDAR-based perception module also achieve your goal? Could you provide evidence or a stronger argument for why a monolithic LiDAR world model is fundamentally superior to a hybrid system that combines existing visual world model with a LiDAR perception module for geometric accuracy? What critical capabilities are lost in such a modular approach?\n\n2. Is it proper to state the learned representation as \"universal 3D motion prior\"? What specific properties of your learned representation suggest it is universal rather than just a highly effective prior for ground vehicle navigation?\n\n3. Could you provide an analysis/discussion of forecasting performance as a function of the VAE's compression ratio? At what point does the information loss from compression begin to significantly degrade the model's ability to predict complex or subtle dynamic events? This is crucial for understanding the model's limitations in safety-critical scenarios.", "limitations": "1. Performance in Non-Ideal Conditions: The model is trained and evaluated on well-curated datasets. Its robustness to real-world sensor noise, artifacts, and adverse weather conditions (e.g., rain, fog, snow), which significantly degrade LiDAR quality, is unknown and unaddressed.\n\n2. Safety considerations: For a model intended for safety-critical applications like autonomous driving, an analysis of its failure modes is essential. The paper lacks any discussion of when the model's forecasts are likely to be inaccurate or unreliable. Understanding these failure cases is critical for safe deployment.\n\n3. Potential for Malicious Use: A powerful generative model capable of creating realistic 4D representations of real-world scenes could potentially be used to mislead or attack autonomous driving systems.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n- the \"foundational\" model for LiDAR-based world modeling could be of interest to relevant community\n\n**Weaknesses**:\n- The \"Foundational\" Claim seems Overstated. The paper uses transfer learning success as evidence of a \"foundational\" model that has learned a \"universal 3D motion prior.\" However, the pre-training (outdoor driving) and all fine-tuning tasks (different LiDAR beams, indoor navigation, semantic driving scenes) fall within the domain of ground-level navigation. The current results of a large model pre-trained on a relevant, large dataset performs well on similar downstream tasks, appears not necessarily evidence of a foundational model.\n- Trade-offs in compression efficiency and forecasting. The paper's focus on high-ratio compression (up to 768x) is a key part of its efficiency claim. However, there is a lack of analysis on what information is lost during this compression and how that impacts the model's ability to learn nuanced dynamics. High compression might filter out subtle cues (e.g., a slight change in a pedestrian's posture) that are critical for predicting complex, long-term events. The model may be efficient at forecasting general motion but could be brittle in safety-critical edge cases, a trade-off that is never explored.", "was_revised": true, "final_justification": "I am satisfied with the rebuttal, so I choose to maintain the positive rating.", "rebuttal": "Thank you for your thoughtful review and helpful feedback, we appreciate it. Please see our per-question responses below.\n\n> Re: Q1.1: What are the challenges of building a foundational LiDAR world model? The reasoning is missing in the introduction.\n\nThank you for your comment, we will include a discussion of the challenges and our approach to them in the introduction near lines 36. Here is specific respond:\n\nThe main challenges are the 1. **availability of the data** and 2. the **costs of training**. We will include these contents in the final version to better demonstrate our motivation.\n1. As you mentioned, Cosmos and GAIA-2 often require millions of GPU hours to train and the results of the training are built on sufficient data: Cosmos used 100 million RGB video clips with diverse content for pre-training; assuming 10s per clip, this equates to about 277k hours of pre-training data (see page 7 of the Cosmos technical report). However, this diversity and scale is not reproducible with existing public dataset of LiDAR: Argoverse2, as the largest open LiDAR dataset, only provides around 166 hours of data of exclusively autonomous driving scenes. Our approach involves two aspects to address this issue. First, we simplified the task itself from building a ‘foundational model’ that is pre-trained on an all-encompassing dataset to one which is trained on data that is easy to be obtained and used in a data-poor domain, which is one of the reasons why we use ‘Towards’ in the title. On the other hand, considering the lack of lidar data diversity, we collected an indoor lidar dataset ourselves (about 4 hour length) to explore the transferability on large gap domains.\n2. For dynamic learning, most of the previous LiDAR world models require a few thousands epochs of training, which may take a few thousand GPU hours.  We designed an efficient approach that required fewer than 200 epochs (about 14 hours on 2x 5090) for the network to converge and achieve the SOTA performance on Voxel-based LiDAR forecasting, as we shown in Table 2.\n\n> Re: Q1.2 “Can augmenting a powerful, existing image-based foundational model (like GAIA-2 or Cosmos) with a LiDAR-based perception module also achieve your goal?”\n\nThank you for the thoughtful question, this is definitely an interesting direction for future exploration and comparison to our proposed approach. Intuitively, augmenting Cosmos or GAIA with LiDAR could help the model as RGB images provide a lot of semantic information. However, there are some practical considerations: Cosmos and GAIA can generate high-fidelity samples but the model parameters are up to 7B/14B (ours is 30.3M) and the inference time per video sequence is in minute-scale, which makes real-time forecasting difficult.  We tested with the smallest version of cosmos (2B, video2world, bf16 precision) on the L40s, requires 32.6GB of video memory and about 4.5 minutes to produce a 5-second video, while our model achieves about 24FPS with only 2.35GB memory in the same environment. Obtaining future observation is only a precursor to generating control signals, and the latency at this stage is already high.  \n\nFurthermore, fine-tuning a world model such as Cosmos using LiDAR could be  challenging. The essential reason is that we lack accurate LiDAR depth for self-supervision in arbitrary scenarios (although it may be possible to provide pseudo labelling through models like Depth Anything,).\n\n> Re Q1.3 and Q1.4: “Could you provide evidence or a stronger argument for why a monolithic LiDAR world model is fundamentally superior to a hybrid system” and “What critical capabilities are lost in such a modular approach?”\n\nWhile we agree it would be interesting to approach LiDAR forecasting with a hybrid world model, our paper does not claim that the LiDAR world model would be fundamentally superior to this hypothetical hybrid system. An interesting fact is that on some mature perception tasks (e.g. 3D detection/segmentation), SOTA's approach almost invariably uses a LiDAR/RGB fusion scheme. As mentioned before, one of the main problems with doing this in the world model is that not any of the scenarios are accurately depth-referenced, but we still believe this will be one of the possible directions for future development.\n\n\n> Re Q2: “Is it proper to state the learned representation as \"universal 3D motion prior\"? What specific properties of your learned representation suggest it is universal rather than just a highly effective prior for ground vehicle navigation?”\n\nOne piece of evidence for this is that in the indoor data, although the platform of LiDAR sensor is still a ground vehicle, the scope of the scene and the movement of foreground objects is completely different from the outdoor scene. The movement of the crowd is more complex, and the environment changes from an open background to a closed scene. Our framework still has performance gain by finetuning the foundational model that is trained on unlabeled outdoor data.\n\nNevertheless, we agree that the qualifier \"for ground vehicles\" is needed. We did not add this point to the explicit contributions statement originally, because it is a broader goal of the line of work, as opposed to an explicit contribution of this paper. We will make this clear in the paper.\n\n> Re Q3: “Could you provide an analysis/discussion of forecasting performance as a function of the VAE's compression ratio? At what point does the information loss from compression begin to significantly degrade the model's ability to predict complex or subtle dynamic events? This is crucial for understanding the model's limitations in safety-critical scenarios.”\n\nHere is the case of using VAEs with different compression ratios as data compressor for the model: we first analysed which part of the scene would be more likely to be lost in the compression process, as shown in the table below. Note that here we use bfp16 acceleration for FPS measurement, which will have about 0.5% to 1.4% negative impact on performance compared with the one reported in the paper.  \n\n(Drop over 30% are bolded)\n| Category              | x32  | x192  | x384  | x768  | Drop&nbsp;(x32→x192) | Drop&nbsp;(x32→x384) | Drop&nbsp;(x32→x768) |\n|-----------------------|------:|------:|------:|------:|----------------:|----------------:|----------------:|\n| others                | 98.56 | 88.51 | 79.56 | 66.14 | 10.05 | 19.00 | **32.42** |\n| barrier               | 99.38 | 97.80 | 95.14 | 88.74 |  1.58 |  4.24 | 10.64 |\n| bicycle               | 98.61 | 97.45 | 96.03 | 90.77 |  1.16 |  2.58 |  7.84 |\n| bus                   | 99.54 | 93.63 | 87.75 | 79.57 |  5.91 | 11.79 | 19.97 |\n| car                   | 99.68 | 96.00 | 88.94 | 78.89 |  3.68 | 10.74 | 20.79 |\n| construction_vehicle  | 97.69 | 84.19 | 75.26 | 62.37 | 13.50 | 22.43 | **35.32** |\n| motorcycle            | 99.59 | 98.42 | 96.34 | 90.49 |  1.17 |  3.25 |  9.10 |\n| pedestrian            | 99.46 | 97.65 | 95.28 | 88.07 |  1.81 |  4.18 | 11.39 |\n| traffic_cone          | 99.07 | 97.68 | 95.33 | 87.73 |  1.39 |  3.74 | 11.34 |\n| trailer               | 98.84 | 89.12 | 81.89 | 69.97 |  9.72 | 16.95 | 28.87 |\n| truck                 | 99.21 | 92.71 | 85.67 | 76.63 |  6.50 | 13.54 | 22.58 |\n| driveable_surface     | 99.71 | 97.27 | 93.01 | 87.55 |  2.44 |  6.70 | 12.16 |\n| other_flat            | 99.59 | 98.56 | 96.61 | 90.74 |  1.03 |  2.98 |  8.85 |\n| sidewalk              | 99.21 | 94.40 | 86.94 | 77.29 |  4.81 | 12.27 | 21.92 |\n| terrain               | 98.83 | 93.86 | 86.61 | 76.03 |  4.97 | 12.22 | 22.80 |\n| manmade               | 97.87 | 83.66 | 72.52 | 62.53 | 14.21 | 25.35 | **35.34** |\n| vegetation            | 93.66 | 68.27 | 58.32 | 50.56 | 25.39 | **35.34** | **43.10** |\n\nWe notice that the voxel missing in the small & rare foreground objects is much less severe than in the background (vegetation, manmade) and large objects (construction vehicle) during compression ratio increase, while these small targets are more important for safe driving. This should help address some of your safety concerns. \n\nNext we show the forecast performance as a function of VAE for different compression rates. First, the results of CFM based on latent with different compression ratios are shown below. For time considerations, all CFMs here are trained as 200 epochs.\n\n| Compression | 1&nbsp;s&nbsp;IoU | 1&nbsp;s&nbsp;mIoU| 2&nbsp;s&nbsp;IoU | 2&nbsp;s&nbsp;mIoU | 3&nbsp;s&nbsp;IoU | 3&nbsp;s&nbsp;mIoU | **Avg IoU** | **Avg mIoU** |\n|-------------|----------------:|-----------------:|----------------:|-----------------:|----------------:|-----------------:|-----------:|-------------:|\n| ×32  | 38.40 | 28.96 | 28.54 | 18.52 | 23.14 | 13.48 | 30.02 | 20.31 |\n| ×192 | **40.53** | **33.17** | **30.37** | **21.09** | **24.44** | **15.64** | **31.78** | **23.33** |\n| ×384 | 39.68 | 32.96 | 29.95 | 20.91 | 24.25 | 14.98 | 31.29 | 22.95 |\n| ×768 | 34.66| 27.10 | 24.51 | 17.30 | 22.61 | 12.95 | 27.26 | 19.11 |\n\nAt the highest reconstruction rate of x32, we find that the network is not fully fitted at 200 epochs, and a longer training time may allow the network to fit better. However since this is clearly contrary to the ‘sample/training efficiency’ of this paper, we keep the training time the same for fair evaluation. For latent with higher compression multiples, the forecast performance continues to degrade. We believe that this is not only due to the fact that it retains less information on large objects, it is also because at larger compression multiples, latent becomes very sensitive to noise: we find that at multiples of x384 and x768, noise of the order of 1e-4 can crash the network.\n\nFinally, we again appreciate reviewer's comments, which makes the analysis of our article more comprehensive. We hope that our response will help to solve your problem and provides sufficient reason to raise the score."}, {"review_id": "8p88IifIcv", "reviewer": "Reviewer_qmHa", "summary": "This paper proposes a foundational LiDAR world model trained via latent flow matching on large-scale LiDAR sequences. It combines a Swin-VAE for efficient BEV occupancy compression and a conditional flow matching decoder for future occupancy prediction. The model is pretrained in a self-supervised manner and evaluated on multiple downstream tasks, showing strong generalization under low-data regimes and significant efficiency gains.", "questions": "1. Why is Visual Point Cloud Forecasting (CVPR 2024) not discussed or compared, given that it also uses point cloud forecasting as a self-supervised pretraining objective?\n2. Figures 2 and 3 illustrate key components of your method but are never referenced in the main text. Can you clarify why they were omitted from the narrative, and how readers are expected to connect them with Sections 3.2 and 3.3?\n3. As a LiDAR world model, does your method support explicit reconstruction of future point clouds? If so, how do you evaluate the geometric quality of these outputs beyond voxel mIoU—for example, in terms of fidelity, completeness, or temporal consistency?", "limitations": "no", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. The paper introduces a pretraining architecture that combines a Swin-VAE for BEV compression with a latent flow matching module for temporal modeling. This framework enables scalable self-supervised learning over large-scale LiDAR sequences and supports transfer across multiple forecasting tasks.\n2. The pretrained model significantly reduces reliance on human labels. For example, on semantic occupancy forecasting, it outperforms supervised baselines using only 5% of labeled data, indicating strong inductive priors learned from geometric dynamics.\n3. The Swin-VAE achieves up to 6× higher compression than prior work while maintaining high reconstruction fidelity. Combined with latent-space forecasting, the method runs up to 28.2× faster and requires substantially fewer FLOPs than baselines.\n\n**Weaknesses:**\n1. The paper does not compare to recent works like Visual Point Cloud Forecasting (CVPR'24), which also use point cloud forecasting as a pretraining objective. Without conceptual or empirical comparison, the novelty claim of being the “first foundational LiDAR world model” lacks proper grounding.\n2. Figures 2 and 3 are not referenced in the main text, making it harder to follow how temporal modeling is implemented in practice.\n3. Although the model aims to predict future LiDAR scenes, it does not evaluate the geometric or structural quality of the predicted outputs. There is no analysis of consistency, plausibility, or failure modes, leaving the realism and utility of the predictions unassessed.", "was_revised": true, "final_justification": "The rebuttal has addressed my main concerns regarding the comparison with ViDAR, clarification of the methodology, and evaluation metrics. The additional results and explanations resolve my doubts, and I therefore maintain my positive recommendation.", "rebuttal": "Thank you for pointing out our shortcomings in comparison and experiment design! We provide a more detailed comparison and analysis here.\n\n> Re Q1:  “Why is Visual Point Cloud Forecasting (CVPR 2024) not discussed or compared, given that it also uses point cloud forecasting as a self-supervised pretraining objective?”\n\nWe’re sorry if our statement caused some misunderstanding. We would like to make two points here to show the difference between our method and ViDAR, and why we did not compare the results of point cloud forecasts in the paper.\n\n1. We would like to emphasise that by utilising 4D prior knowledge from large-scale pre-training, we can reduce the amount of data used in the downstream world model task(dynamic learning), especially for non-semantic to semantic transferability. By contrast, the contribution of ViDAR here is to improve the performance of perceptual(3D)/end-to-end(4D) autonomous driving that requires labels, with the backbone pretrained via self-supervised visual point cloud forecasting. The 4D tasks benefit from ViDAR, like the future occupancy forecasting (non-semantic, Table 8 in ViDAR) results they reported, is based on fine-tuning ViDAR pre-trained UniAD, while the fine-tuning of UniAD needs labels. Therefore it is inappropriate to directly compare the IOUs of the point cloud forecast by our method (Table 2) with Table 8 in ViDAR, since our training does not require future labels. We admit that theoretically ViDAR can also forecast semantic occupancy, but this is not implemented in the article or in the publicly available codes.\n\n2. Rendering the point cloud in real-time and calculating the gradient in forward propagation is very memory-intensive and time-consuming(in Occ4d[1] implementation, it needs over 20G memory for the PC rendering in a 800x800 voxel space), and given our real-world situation we use voxel- rather than point-based forecasting. This reduction in granularity allows our model to be faster than the previous baseline model like OccWorld(Deterministic Transformer based) or DOME (DDPM based).\n\nWhile it is not practical to directly compare results after pre-training, we note that we can indeed compare results from pre-training. Again, we thank the reviewers for pointing out this point, which was previously overlooked. For simplicity, we voxelise all of the ViDAR and Occ4d forecasts to calculate the IOUs and compare them with our approach. Since ViDAR only releases the weight pretrained on ⅛ nusc for 3s future forecasting(the line we marked with star in following tables), but not the one they mentioned in the paper, we use it as an instance here. From the chamfer distance perspective, the one pretrained on ⅛ of nusc is close to the one on full dataset.\n(ViDAR results)\n|    | 1s     | 2s     | 3s     |\n|---------------|--------|--------|-------|\n| L1 error*       | 2.541  | 2.740  | 2.990  |\n| Abs rel error*  | 0.188  | 0.209  | 0.244  |\n| Chamfer*       | 1.251  | 1.489  | 1.790  |\n| Chamfer in paper, full dataset      | 1.120  | 1.380  | 1.730  |\n| IOU*           | 13.11  | 12.48  | 11.78  |\n\n(Occ4d results)\n| | 1s     | 2s     | 3s     |\n|---------------|--------|--------|-------|\n| L1 error      | 1.44 | 1.80 | 2.16 |\n| Abs rel error | 0.10 | 0.14 | 0.18 |\n| Chamfer       | 1.11   | 1.45   | 1.96   |\n| IOU           | 26.96  | 19.51  | 16.81  |\n\nCompared to these methods, our performance remains ahead of the curve, especially at 2s and 3s forecast horizon, as shown in the following table.\n|     | 1s     | 2s     | 3s     |\n|-----------------|--------|--------|-------|\n| ViDAR* (Img2LiDAR)     | 13.11  | 12.48  | 11.78  |\n| Occ4d (LiDAR2LiDAR)     | 26.96  | 19.51  | 16.81  |\n| Ours            | **26.98** | **21.56** | **18.26** |\n\nOur method exceeds the Occ4d and also ViDAR in the pretraining stage, this can also be another aspect of the effective learning of the dynamic prior during our pre-training (without semantics). Another noteworthy point is that for both ViDAR and Occ4d, the prediction of the point cloud requires every ray angle from the future point cloud (which can be obtained from the vehicle's future pose and sensor's internal parameters), and the network is only responsible for forecasting the depth, whereas our approach here does not require any information from the future. \n\n> Re: “Figures 2 and 3 illustrate key components of your method but are never referenced in the main text. Can you clarify why they were omitted from the narrative, and how readers are expected to connect them with Sections 3.2 and 3.3?”; \"... harder to follow how temporal modeling is implemented in practice.\"\n\nWe thank the reviewer for pointing out this inconsistency. We have added references and more explanations of them in the main text. The two figures were indeed intended to complement the technical exposition in Section 3.1 (Data compression) and 3.2 (Conditional Flow Matching, CFM). We clarify the connection below.\n\nFigure 2 is an overview of the Swin-VAE encoder/decoder architecture which is described in Section 3.1 and Figure 3 depicts specific architectural design for our conditional velocity field predictor that comes immediately after the encoder. We expect the reader to connect these two figures with the description of our methodology. Basically we need to first train a VAE to compress the data, then use the compressed latent to forecast future latent (via CFM), finally decode the future observation. In the training of CFM, the VAE is frozen.\n\nFor Section 3.3 (Representation alignment), we explained how we align the latent space of VAE (described in Figure 2 and Section 3.1) when performing cross-domain finetuning (e.g. dense occupancy → semantic occupancy) with more details on related visualization and experimental results in Appendix D. We will add references for clarity. Specifically, we found that when the non-semantic pre-trained CFM weights are directly used to fine-tune the latent with semantic information, the results are not optimal. After comparing CKA and CKNNA we found that the hidden space of both (latent w/w.o. semantic information) is not aligned. To solve this problem, we introduced a cosine similarity term (Eq. 2) to encourage the alignment of latent w/w.o. semantic information in the space when fine-turning the data compressor for semantic occupancy (as shown in Figure 6). The other reason for this is that in practice, because of the difference in embedding, we cannot simply fine-tune the VAE for non-semantic data with semantic data. This alignment proves to be effective, and we can obtain further improvements of 2 to 6 per cent after alignment (as shown in Table 8). \n\nFurther, when the pre-training data and the fine-tuning data were not ‘paired’, we simply fine-tuned the representation of the data compressor on the fine-tuning data instead of directly using the weights obtained on the pre-training data. Experiments have also demonstrated the effectiveness of this fine-tuning(in Table 6 and 7).\n\nTherefore, the overall experimental flow is: 1. Use the pre-trained data to train the VAE as a data compressor. 2. Use this VAE to obtain the latents of the pre-trained data. 3. Use these latents to pre-train the CFM. 4. Fine-tune this VAE on the downstream data (either directly or by using the cos term to align). 5. Fine-tune the CFM on the downstream.\n\n> Re: As a LiDAR world model, does your method support explicit reconstruction of future point clouds? If so, how do you evaluate the geometric quality of these outputs beyond voxel mIoU—for example, in terms of fidelity, completeness, or temporal consistency?\n\nOur approach supports explicit point cloud rendering, but given the computational cost and practical usefulness we don't do it, so that's why our last rendering step in the decoder section of Figure 2 is dashed.\n\nRe: evaluating geometric quality: for fidelity, completeness, and temporal consistency, following DynamicCity [2] and LiDM [3], we use latents from Minkovski-Unet for fidelity measurement (FID/KID) and trained a 4D semantic occupancy network based on our VAE for temporal consistency(FVD) measurement. Specifically, for FID/KID:\n\n|        | 1s FID | 1s KID | 2s FID | 2s KID | 3s FID | 3s KID | AVG FID | AVG KID |\n|--------------------|--------|--------|--------|--------|--------|--------|---------|---------|\n| OccWorld           | 9.54   | 11.7   | 8.56   | 10.46  | 7.78   | 9.80   | 8.62    | 10.60   |\n| DOME               | 4.37   | 4.39   | 5.09   | 5.44   | 5.63   | 6.53   | 5.03    | 5.42    |\n| Ours (Hist. traj)  | 1.67   | 1.49   | 2.81   | 2.85   | 4.02   | 4.74   | 2.83    | 3.02    |\n| Ours (Fut. traj)   | 1.61   | 1.47   | 2.90   | 3.05   | 3.91   | 4.55   | **2.80**    | **3.02**    |\n\nFor FVD:\nBased on our VAE, we include 3D conv and time attention in both the encoder and decoder to ensure that the network can capture the temporal relation. The reconstruction rate of the network used as occupancy video feature extractor is 81.39 and 87.75 in terms of IOU and mIOU. The FVDs are shown below (3s, 6 frames). To give context for temporal consistency, we include an approach that randomly shuffles the ground truth future ordering and compare the FVD with the correctly ordered GT. This tells us the FVD when the predictions are individually correct but temporally inconsistent. \n\n|         | FVD (×10⁻³) |\n|-------------------|---------|\n| OccWorld          | 18.68       |\n| DOME              | 9.79        |\n| Ours (hist. traj.)| 7.80        |\n| Ours (fut. traj.)| **7.68**        |\n| Reorder GT        | 12.07        |\n\nFinally we would like to thank the reviewers again for help and hope that our answers will solve your problems and provide sufficient reason to raise the score.\n\n[1] Point Cloud Forecasting as a Proxy for 4D Occupancy Forecasting, CVPR2023\n\n[2] DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes, ICLR2025\n\n[3] LiDAR Diffusion Models, CVPR2024"}, {"review_id": "9TgBKpgSMF", "reviewer": "Reviewer_br6R", "summary": "The paper introduces the first general-purpose LiDAR world model with key contributions on 1) A Swin-Transformer VAE compresses voxelised LiDAR grids by 192 × while preserving ≥ 93 % mIoU. 2) A Conditional Flow-Matching UNet predicts future latent frames in real time (~22 FPS), needing far fewer FLOPs than diffusion baselines. 3) Pre-training on nuScenes lets the model fine-tune with only 5 % labelled data (for sparse-to-dense completion, outdoor-to-indoor forecasting, and semantic occupancy), beating task-specific SOTA. 4) A cosine-similarity regulariser keeps latent codes aligned when the encoder is replaced, safeguarding transfer.", "questions": "Question: Why is outdoor-to-indoor the right showcase for “cross-domain” generalisation when indoor scenes are typically easier and better labelled?\nSuggestion: Please provide quantitative evidence—e.g., annotation cost comparison, data-scarcity statistics, or failure cases—showing that indoor fine-tuning really benefits from the outdoor pre-training, and clarify why the reverse direction is less meaningful.\nImpact on score: A convincing justification (or an alternate, harder transfer task) would remove my main conceptual doubt and could raise the score to 6/6.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\n\n1. Originality: The authors are the first to package foundation-model style pre-training, compression, and cross-task transfer into a single LiDAR-only world model. No previous work nails the whole pipeline end-to-end.\n2. Quality: Experiments are thorough: 192× compression still keeps ≥ 93 % mIoU, ablations cleanly isolate the VAE, flow-matching, and latent-alignment tricks, and real-time numbers (≈ 22 FPS) are reported on the same hardware as baselines.\n3. Significance: Needing only 5 % labelled data to beat task-specific SOTA on three diverse tasks is a big win for anyone strapped for annotations.\n4. Clarity: Figures are self-explanatory, training details live in the appendix, and the writing flows—you rarely re-read a sentence to “get it.” \n\n\n\nWeaknesses\n \nI am not persuaded by the motivation of the proposed downstream task \"outdoor to indoor generalisation\", since intuitively, the outdoor is a much more challenging scenario and suffers more from data annotation scarcity compared to indoor; it would be helpful if the author could provide more empirical evidence to strengthen the motivation. Additionally, the introduction appears to be overly long, and some typos, such as those on line 7, should be carefully checked.", "was_revised": true, "final_justification": "After carefully reviewing the other reviewers’ discussions, I will maintain my original rating.", "rebuttal": "We would first like to thank the reviewer for their feedback. The insights and questions you kindly offered regarding the nature and motivation for the outdoor-to-indoor navigation task are helpful to improve the quality of our paper. Below we give a detailed explanation of why we choose outdoor-to-indoor navigation as one of our cross-domain experiments, why it is practically relevant, and why the reverse direction is less meaningful.\n\n> Re: “Why is outdoor-to-indoor the right showcase for “cross-domain” generalisation when indoor scenes are typically easier and better labelled?” & “showing that indoor fine-tuning really benefits from the outdoor pre-training.”\n\nWe would like to clarify why we frame outdoor-to-indoor as the transfer direction. Althought both directions are interesting, outdoor-to-indoor is the more practical direction due to the diversity and scale of existing LiDAR data. Outdoor LiDAR datasets are a. Plentiful, freely available, and well-calibrated: many are already aligned with GPS/IMU and camera sensors. b. Rich in dynamic behavior: diverse scenes, agents, and motion profiles.\n\nOutdoor environments exhibit a wider range of dynamic behaviors that are essential for training a transferable world model in many aspects: High-momentum ego motion: a vehicle may travel tens of meters and rotate several degrees between 0.5s sweeps, producing strong range-rate cues and frequent self-occlusion. Articulated actors: buses articulate, cyclists lean, trailers pivot, capturing realistic temporal dynamics. Long-range occlusions: objects can disappear behind others and reappear several frames later. Mixed mobility: static poles, parked cars, and fully dynamic agents co-exist, and must be disambiguated from geometry alone. Training on such diverse outdoor motion helps establish strong dynamics priors that generalize across domains. \n\nOur goal is to test whether such motion priors learned from highly dynamic outdoor environments can transfer to indoor scenes, which feature: \n- Short-range occlusions (2–4 m instead of 40 m),\n- Dense LiDAR returns and cluttered geometry\n- Slow, fine-grained human and robot motion,\n- Rapid local changes in confined space.\n\nIn summary, we believe the dynamics prior in outdoor environments is more general and could be more suitable in a more complex scenarios.\n\n> Re: “ clarify why the reverse direction is less meaningful.”\n\nWe believe the reverse direction would be interesting to test, and not necessarily less meaningful; we will make this clear in the paper. However, it is simply less practical to attempt this transfer direction for several reasons. First, indoor datasets are scarcer. Second, motion information embedded in indoor environments is, in some sense, not as diverse as that in outdoor scenes. Motion typically happens in small ranges and the variation in objects is limited."}, {"review_id": "7uHsHs0woy", "reviewer": "Reviewer_Tnf1", "summary": "This paper proposes a foundational LiDAR world model that can be  fine-tuned for various downstream tasks such as semantic occupancy forecasting and adapting to different LiDAR sensor configurations. The authors introduce a Swin Transformer-based VAE architecture to achieve state-of-the-art reconstruction accuracy with higher compression ratios and reduced computational costs. The model demonstrates transferability and reduced reliance on labeled data, achieving good performance with only a small fraction of the training data required by previous methods.", "questions": "See weakness.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.\tThe paper is well-written, with a clear methodology and extensive experimental results that validate its effectiveness.\n\n2.\tThe motivation behind the module design is clearly articulated.\n\n3.\tThe experiments are comprehensive. The authors demonstrate the effectiveness of their design across three challenging scenarios, achieving strong performance.\n\nWeaknesses:\n\n1.\tThe design primarily aims to address issues such as redundant model parameters and prolonged training time. However, it remains unclear why and how the proposed Swin Transformer-based VAE architecture effectively tackles these problems.\n\n2.\tAlthough the paper claims to be the first to introduce a foundational LiDAR world model, its future forecasting component appears to be quite similar to some existing self-supervised point cloud forecasting methods [1][2].\n\n3.\tSome descriptions and illustrations are not clear, and in some cases may be confusing. For instance, z_s^{t_1:t_2} is defined as the prediction target of the model in line 188 page 5, yet it is also shown as an input in Figure 3, which creates ambiguity.\n\n[1] Yang, Z., Chen, L., Sun, Y., & Li, H. (2024). Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 14673–14684.\n[2] Khurana, T., Hu, P., Held, D., & Ramanan, D. (2023). Point cloud forecasting as a proxy for 4D occupancy forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 1116–1124.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for their thoughtful and detailed feedback, which helped us identify and address several important missing elements in our submission.\n\n> Re Q1:  \"The design primarily aims to address issues such as redundant model parameters and prolonged training time. However, it remains unclear why and how the proposed Swin Transformer-based VAE architecture effectively tackles these problems.\"\n\nWe are sorry for the misunderstandings that may have arisen from the narrative in our paper. Here we would like to recount the frame to help you better understand our thinking.\n\nIn order to explore the adaption of the LiDAR world model on different subtasks and different data volumes, a challenge is prolonged training time: normally over a thousand GPU hours for SOTA model training. It is important to note that these training hours focus primarily on the training of the transition model(Figure 3), assuming a 2-stage training that follows the traditional Latent diffusion model.  \n\nExcessive training time for transition models can be decomposed into 1. impropriety of the transition model training objective and 2. parameter redundancy of the transition model. For the former, we address it by replacing the original objective with a new CFM-based one via a redesigned DiT-based network(Figure 3). The related ablation study is shown in Table 4. For the latter, we found that it’s **actually correlated with the channel of latents**(128 in OccWorld and 64 in DOME), this motivates us to have a better data compressor for a higher compression ratio as we present in Figure 2. Therefore, the data compressor in fact **only solves the previous problem of low latent compression rates**. Its own training won't take less time than the data compressor in previous work, but **the low-dim latent it generates will help us reduce the parameters in the transition model**.\n\nFurther, for the question “Why does your VAE work better in terms of information preserving?” Apart from the advantages of the structure itself (Swin Transformer v.s. All conv data compressors before), we attribute this to height embedding, removal of all 3D structures and further compression of the dimension at the bottleneck.\n\nSpecifically, we first attribute this to the removal of all temporal processing modules: experimentally, our aggregation for spatial dimensional information at the transition model is sufficient, and adding 3D conv in decoder like DOME instead affects the reconstruction rate, as shown in the following table.\n\n| ×192 compression ratio          | IoU  | mIoU |\n|--------------------------------|-----:|-----:|\n| With 3-D conv + 3-D attention   | 82.33 | 88.75 |\n| Without (2-D only)              | 85.12 | 92.81 |\n\nWith these 2 different kinds of latents, we retrain the model for comparsion in forecasting performance:\n\n| Variant | 1 s&nbsp;IoU | 1 s&nbsp;mIoU | 2 s&nbsp;IoU | 2 s&nbsp;mIoU | 3 s&nbsp;IoU | 3 s&nbsp;mIoU | **Avg IoU** | **Avg mIoU** |\n|---------|------------:|--------------:|------------:|--------------:|------------:|--------------:|------------:|-------------:|\n| With ~  | 39.72 | 31.68 | 28.77 | 20.00 | 21.14 | 13.91 | **29.87** | **21.86** |\n| W.o. ~  | 40.53 | 33.17 | 30.37 | 21.09 | 24.44 | 15.64 | **31.78** | **23.33** |\n\nAt the same time, we note the importance of the Layer-norm block (LN+Conv+ Swish Activation) before sampling and height embedding, as shown in the Table below. The former contributes to scale stabilisation of latent whereas the latter has been overlooked from time to time by previous work (they borrowed directly from the data compressor for stable diffusion, whereas the height dimension does not exist in video data). \n\n| Component              | Variant  | IoU   | mIoU |\n|------------------------|----------|------:|-----:|\n| LN block               | With     | 83.62 | 89.80 |\n|                        | W.o.     | 85.12 | 92.81 |\n| Height embedding       | With     | 81.92 | 88.63 |\n|                        | W.o.     | 85.12 | 92.81 |\n\nThese ablation study in VAE work mechanism will be updated in the appendix. We hope that these tricks will help later work on designing networks more efficiently, and these answers will answer your questions.\n\n> Re Q2: Although the paper claims to be the first to introduce a foundational LiDAR world model, its future forecasting component appears to be quite similar to some existing self-supervised point cloud forecasting methods [1][2].\n\nWe would like to make two points here to show the difference between our method and ViDAR, and why we did not compare the results of point cloud forecasts in the paper.\n\n1. We would like to emphasise that by utilising 4D prior knowledge from large-scale pre-training, we can reduce the amount of data used in the downstream world model task(dynamic learning), especially for non-semantic to semantic transferability. By contrast, the contribution of ViDAR here is to improve the performance of perceptual(3D)/end-to-end(4D) autonomous driving that requires labels, with the backbone pretrained via self-supervised visual point cloud forecasting. The 4D tasks benefit from ViDAR, like the future occupancy forecasting (non-semantic, Table 8 in ViDAR) results they reported, is based on fine-tuning ViDAR pre-trained UniAD, while the fine-tuning of UniAD needs labels. Therefore it is inappropriate to directly compare the IOUs of the point cloud forecast by our method (Table 2) with Table 8 in ViDAR, since our training does not require future labels. We admit that theoretically ViDAR can also forecast semantic occupancy, but this is not implemented in the article or in the publicly available codes.\n\n2. Rendering the point cloud in real-time and calculating the gradient in forward propagation is very memory-intensive and time-consuming(in Occ4d[1] implementation, it needs over 20G memory for the PC rendering in a 800x800 voxel space), and given our real-world situation we use voxel- rather than point-based forecasting. This reduction in granularity allows our model to be faster than the previous baseline model like OccWorld(Deterministic Transformer based) or DOME (DDPM based).\n\nWhile it is not practical to directly compare results after pre-training, we note that we can indeed compare results from pre-training. Again, we thank the reviewers for pointing out this point, which was previously overlooked. For simplicity, we voxelise all of the ViDAR and Occ4d forecasts to calculate the IOUs and compare them with our approach. Since ViDAR only releases the weight pretrained on ⅛ nusc for 3s future forecasting(the line we marked with star in following tables), but not the one they mentioned in the paper, we use it as an instance here. From the chamfer distance perspective, the one pretrained on ⅛ of nusc is close to the one on full dataset.\n\n(ViDAR results)\n|    | 1s     | 2s     | 3s     |\n|---------------|--------|--------|--------|\n| L1 error*       | 2.541  | 2.740  | 2.990  |\n| Abs rel error*  | 0.188  | 0.209  | 0.244  |\n| Chamfer*       | 1.251  | 1.489  | 1.790  |\n| Chamfer in paper, full dataset      | 1.120  | 1.380  | 1.730  |\n| IOU*           | 13.11  | 12.48  | 11.78   |\n\n(Occ4d results)\n| | 1s     | 2s     | 3s     |\n|---------------|--------|--------|--------|\n| L1 error      | 1.44 | 1.80 | 2.16 |\n| Abs rel error | 0.10 | 0.14 | 0.18 |\n| Chamfer       | 1.11   | 1.45   | 1.96   |\n| IOU           | 26.96  | 19.51  | 16.81  |\n\nCompared to these methods, our performance remains ahead of the curve, especially at 2s and 3s forecast horizon, as shown in the following table.\n|     | 1s     | 2s     | 3s     |\n|-----------------|--------|--------|-------|\n| ViDAR* (Img2LiDAR)     | 13.11  | 12.48  | 11.78  |\n| Occ4d (LiDAR2LiDAR)     | 26.96  | 19.51  | 16.81  |\n| Ours            | **26.98** | **21.56** | **18.26** |\n\nOur method exceeds the Occ4d and also ViDAR in the pretraining stage, this can also be another aspect of the effective learning of the dynamic prior during our pre-training (without semantics). Another noteworthy point is that for both ViDAR and Occ4d, the prediction of the point cloud requires every ray angle from the future point cloud (which can be obtained from the vehicle's future pose and sensor's internal parameters), and the network is only responsible for forecasting the depth, whereas our approach here does not require any information from the future. \n\n\n> Re Q3: \"Some descriptions and illustrations are not clear, and in some cases may be confusing. For instance, $z_s^{t_1:t_2}$ is defined as the prediction target of the model in line 188 page 5, yet it is also shown as an input in Figure 3, which creates ambiguity.\"\n\nThank you for pointing out the ambiguity. We use time point $t_1$ as the demarcation point when the prediction started, illustrated on line 188. The historical latent $z_s^{t_0:t_1}$ stand for encoded past frames and is fed into the network unchanged, as the condition. The training of flow matching need us to construct a noised future ground truth, act as a training sample along the probability path between prior distribution to the data distribution. Therefore, we have to use $z_s^{t1:t2}$ in Figure 3. However, in sampling process(test stage), the $x_t$ here will be pure Gaussian noise, so the future infos won't be leaked. We understand it seems ambiguious that we use the \"objective\" in training stage, we will update a more detailed description here."}]

Author rebuttal:

Meta-review: This paper presents a LiDAR world modeling framework combining a Swin-VAE for efficient compression and conditional flow matching for temporal prediction. Pretraining on large-scale outdoor LiDAR data allows effective transfer to downstream tasks with minimal labeled data, achieving state-of-the-art results in semantic occupancy forecasting and domain adaptation. The model also demonstrates high efficiency, operating in real time with significantly fewer FLOPs than diffusion-based baselines. Reviewers appreciated the technical depth, comprehensive experiments, and originality of unifying pretraining, compression, and transfer into a single LiDAR-only world model. Strengths include clear ablations, reduced annotation needs, and systematic evaluation across tasks. The rebuttal provided valuable additions: detailed compression–forecasting trade-offs, new comparisons against Occ4D and ViDAR, and evaluations with fidelity and temporal consistency metrics (FID, KID, FVD). These addressed most concerns and strengthened the work. Remaining weaknesses include that the “foundational” claim may be overstated, the motivation for outdoor-to-indoor transfer could be clearer, and broader robustness or failure-mode analysis would further enhance the study. Nonetheless, the reviewers converged on positive scores, agreeing that the paper is technically solid, impactful, and relevant to the NeurIPS community. Overall, the submission is recommended for acceptance.
Decision: Accept (poster)