CSV-Occ: Fusing Multi-frame Alignment for Occupancy Prediction with Temporal Cross State Space Model and Central Voting Mechanism
Authors: Ziming Zhu
Yu Zhu
Jiahao Chen
LingXiaofeng
Huanlei Chen
Lihua Sun
Venue: ICML 2025 poster (2025)
Keywords: 3D Semantic Occupancy Prediction
Semantic Segmentation
Autonomous Driving
Temporal Information
State Space Model
Voting Mechanism
Primary area: applications->computer_vision

Abstract: Recently, image-based 3D semantic occupancy prediction has become a hot topic in 3D scene understanding for autonomous driving. 
Compared with the bounding box form of 3D object detection, the ability to describe the fine-grained contours of any obstacles in the scene is the key insight of voxel occupancy representation, which facilitates subsequent tasks of autonomous driving. In this work, we propose CSV-Occ to address the following two challenges: (1) Existing methods fuse temporal information based on the attention mechanism, but are limited by high complexity. We extend the state space model to support multi-input sequence interaction and conduct temporal modeling in a cascaded architecture, thereby reducing the computational complexity from quadratic to linear. (2) Existing methods are limited by semantic ambiguity, resulting in the centers of foreground objects often being predicted as empty voxels. We enable the model to explicitly vote for the instance center to which the voxels belong and spontaneously learn to utilize the other voxel features of the same instance to update the semantics of the internal vacancies of the objects from coarse to fine. Experiments on the Occ3D-nuScenes dataset show that our method achieves state-of-the-art in camera-based 3D semantic occupancy prediction and also performs well on lidar point cloud semantic segmentation on the nuScenes dataset. Therefore, we believe that CSV-Occ is beneficial to the community and industry of autonomous vehicles.

CSV-Oce:
Fusing Multi-frame Alignment for Occupancy Prediction with State Space Model and Central Voting Mechanism
Temporal Cross
Ziming Zhu'
Yu Zhu Jiahao Chen
Xiaofeng Ling' 12 Huanlei Chen
Lihua Sun
Abstract
Recently, image-based 3D semantic occupancy prediction has become hot topic 3D scene un- derstanding for autonomous driving. Compared with the bounding box form of 3D object detec- tion, the ability to describe the fine-grained con- tours of any obstacles in the scene is the key in- sight of voxel occupancy representation, which facilitates subsequent tasks of autonomous driv- ing. In this work, we propose CSV-Oee ad- dress the following two challenges: (1) Exist- ing methods fuse temporal information based on the attention mechanism, but are limited by high complexity. We extend the state space model to support multi-input sequence interaction and conduct temporal modeling in a cascaded archi- tecture, thereby reducing the computational com- plexity from quadratic to linear. (2) Existing methods are limited by semantic ambiguity, re- sulting in the centers of foreground objects often being predicted empty voxels. We enable the model to explicitly vote for the instance center to which the voxels belong and spontaneously learn to utilize the other voxel features of the
Figure I. Examples of Internal Occupancy Vacancy (IOV). The yellow grids are the occupancy predictions of the model for "bus", purple for "truck", cyan for "construction vehicle", red for "pedes- trian", orange for "barrier", light yellow for "traffic cone", and green for "vegetation". these four examples, IOV occurred for "bus", "truck", and "construction vehicle".
1. Introduction
Image-based based 3D semantic occupancy prediction (Mescheder et al., 2019; Peng al., 2020) represents the structuring of 3D scenes into grids and determining whether the grids are occupied and the specific semantic categories of the occu- pied grids only from visual images. the environmental understanding of autonomous driving, it provides more ge- ometric details than 3D object detection and complements LiDAR-based perception.
same instance update the semantics of the in- ternal vacancies of the objects from coarse fine. Experiments the Occ3D-nuSceme dataset show that our method achieves state-of-the- -frt in camera-based 3D semantic occupancy prediction and also performs well on lidar point cloud seman- tic segmentation on the nuScenes dataset. Code will be available at https: //ggttttuubb//// com/ ZeaZoM/CSV CSV-Occ.
Temporal information is extremely crucial for the visual system to understand the surrounding environment. For example, detecting the multi-angle geometry of highly oc- cluded objects or inferring the relative distance objects very difficult for 2D images without depth information. And temporal cues can make up for the inherent deficiencies of static 2D images in these aspects. Taking the inspiration of Vision Transformers (ViTs) (Dosovitskiy et al., 2021; Liu et al., 2021; Touvron et al., 2021; Zhang et al., 2023a) as the starting point, some existing studies, such as (Koh et al., 2023; Tong et al., 2023; Lu et al., 2023; Liu et al., 2024a), will align historical BEV features (Li et al., 2022; Liu et al., 2023a) based on the attention mechanism (Vaswani, 2017)
'School of Information Seience and Engineering, Fast China University of Science and Technology, Shanghai, China Shanghai Key Laboratory of Intelligent Sensing and Detection Technology, East China University of Science and Technology. Shanghai, China Shanghai Motor Vehicle Inspection Certification Tech Innova- tion Center Co., Ltd., Shanghai, China. Correspondence to: Zhu <zhuyu@ ccust..du.en">.
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CSV-Oce: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
effectively represent the current environment. Although this brings strong learning ability, with the increase in the number of tokens, its quadratic complexity will introduce a large amount of computational overhead in downstream tasks involving large spatial resolutions. To address this challenge, inspired by VMamba (Liu et al., 2024b), Vision Mamba (Zhu et al., 2024), and Cross Attention mechanisms (Gheini al., 2021; Lin et al., 2022) in computer vision, we propose the Cross State Space Module, feature fusion method that extends the State Space Model (SSM) (Gu & Dao, 2023; Mehta et al., 2023; Wang et al., 2023a) to sup- port the interaction of multiple input sequences, thereby reducing the computational complexity from quadratic to linear.
hume
hames
frames
Figure 2. The influence of multi-frame prediction on the IOV problem. The blue grids are the horizontal sections of the model's s occupancy predictions for two "cars", and the gray grids are "man- made". When the number of frames increases from 4 to 8, the IOV may still become serious.
instance to which the voxel belongs and enhance the associative ability of the internal semanties of the ob- ject, enabling it output more accurate 3D semantic occupancy representations.
In addition, after observing the data, as shown in Fig. we found that the centers of foreground objects (such car, bus, and truck, etc.) are often predicted as empty voxels, especially for large objects. We call this problem "Internal Occupancy Vacancy (IOV)". Different from the common back occlusion problem, as shown in Fig. 2, when we in- crease the number of input frames, although the model suc- cessfully predicts the occupancy of the occluded surface of the target through the increase of the viewing angle, still cannot solve the IOV. The center feature (Yin et al., 2021; Chen et al., 2023; Bai et al., 2022) is the best representation of the entire instance for 3D object detection (Lang et al., 2019; Shi et al., 2020; Wang et al., 2022; Li et al., 2022: Yang et al., 2020). However, 3D semantic occupancy pre- diction is different from object detection. Existing advanced 3D semantic occupancy prediction methods (Wang al.,
Experiments on the OesennsuSeenes dataset show that our method achieves the state-of-the-an performance of camera- based 3D semantic occupancy prediction. After further projecting the 3D semantic occupancy results to the lidar point cloud semantic segmentation, it also performs well on the nuScenes dataset.
2. Related Work
2.1. Temporal Modeling in 3D Perception
Camera-based 3D perception has attracted much attention in the field of autonomous driving due to its cost-effectiveness and rich visual attributes. Utilizing temporal information to compensate for the lack of depth information in single-frame 2D planar images can significantly improve the performance of camera-based 3D perception. Recently, 3D perception methods based on multi-frane images have unified this issue into the problem of voxel feature map transformation from historical frames to the current frame. One type of method (Park et al., 2022; Li et al., 2023a; Ma et al., 2024; Wang et al., 2024: Zhou et al., 2024; Liu et al., 2023b) inherits the inspiration from BEVDet4D (Huang & Huang, 2022), inde- pendently extracts and constructs the BEV feature map of each frame, and predicts the relative motion of the target be- tween adjacent frames, thereby aligning the turget-by-targer features of the historical frame to the current frame position, and then concatenating with the BEV feature map of the cur- rent frame. Another type of method (Koh al., 2023; Tong et al., 2023; Lu al., 2023; Liu et al., 2024a) avoids ex- plicitly predicting the relative displacement between frames. They follow the TSA paradigm proposed by BEVFormer (Li et al., 2022), predefine a learnable BEV query in the cur- rent frame, and extract information from the historical BEV feature map through the deformable attention mechanism (Zhu et al., 2020). Although this method can effectively
2023b; Li et al., 2023b; Huang al., 2023; Li et al., 2023c; Pan et al., 2024; Wang et al., 2024; Liu et al., 2024a; Lu et al., 2023; An et al., 2024; Shi et al., 2024; Jang et al., 2024; Ye et al., 2024) perform voxel-wise classification on the 3D spatial volume feature map, while ignoring the fact that foreground voxels actually have instance aggregativity (multiple voxels belonging to the same instance have po- tential complementary features). To address this issue, we propose the Voting- Enhancement Mechanism, which spontancously learns to utilize the other voxel features of the same instance to update the semantics of the internal vacancies of the object from coarse to fine, thereby reducing the IOV.
Our contributions are summarized follows:
In CSV-Oce, we propose the Cross State Space Module (Cross SSM). It extends the SSM to a feature fusion method that supports multi-sequence input and uses cascading approach to achieve the interaction of histor- ical temporal volume features, which can enhance the model's ' ability to integrate temporal information.
We also propose the Voting-based Enhancement Mech- anism. This enables the model to adaptively infer the
2

---

CSV-Oce: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
TVT: Time-based View Transformation
VEM: Voding Enhanceneent Mechanism
Frun
Imoegee Festore Extredoo
SSW
Vosel Peobire Entocdo
ficaliioaion
Luce
Fraz
Vrut
SCW
Semantic Occupancy
Sunound camora Images
ROR
AlL
fuce
Mull-acale Surround decoerrraaaaaaamme Features
Mull-acale Vosal Features
Viow Qawyy
Occupancy
Coarse Semandic Ocoupancy
Figure 3. The overall architecture of the proposed method. First, we extract multi-scale features of multi-camera images using the image backbone. Next, we employ time-based view transformation to integrate information from multiple cameras frames, and use the voxel backbone to upsample and concatenate 3D volume features in multi-scale manner. Finally, the voting- based enhancement mechanism applied refine the coarse semantic occupancy prediction.
fuse historical information into the current frame, with the increase in the number of tokens, its quadratic complexity will introduce large amount of computational overhead in downstream tasks involving large spatial resolutions. To address this limitation, we extend SSM to multi-sequence input for temporal information encoding, thereby achieving linear computational complexity.
3. Method
3.1. Problem Definition
The camera-based 3D semantic occupancy prediction task is designed to predict the dense semantic voxel volume surrounding the ego-vehicle through the analysis of multi- view images. Specifically, we leverage the multi-view image set the current frame, denoted as /mg = { 12..... Jn), as the model input, where represents the index of the camera image viewpoint. The camera-based 3D semantic occupancy prediction task is formulated follows:
2.2. 3D Semantic Occupancy Prediction
Recently, camera-based 3D semantic occupancy predic- tion has received considerable attention. This prediction aims to reconstruct the 3D scene structure from images and predict the occupancy and semantic attributes of all voxels. MonoScene (Cao De Charette, 2022) adopts a camera-based method and 3D UNet (Ronneberger et al., 2015: Çiçek et al., 2016) architecture; TPVFormer (Huang et al., 2023) uses three perspective views to represent the 3D scene for predicting 3D occupancy; OccFormer (Zhang et al.. 2023b) decomposes 3D processing into local and global transformer paths; SurroundOce (Wei et al., 2023) achieves mne-grained results through multi-scale supervi- sion; FB-OCC (Li et al., 2023c) uses forward-backward projection aggregate multi-image information; PanoOce (Wang et al., 2024) unifies 3D semantic occupancy predic- tion and 3D panoramic segmentation tasks; SparseOec (Liu et al., 2024a) employs mask-guided sparse sampling to pre- dict semantic occupancy from 3D sparse representations; OctreeOce (Lu et al., 2023) utilizes variable granularity oc- tree representations to adapt to object shapes and semantic regions of different sizes and complexities. To promote occupancy representation learning, we propose new frame- work named CSV-Occ, which utilizes the temporal state space model and voting mechanism to optimize multi-frame fusion and update the internal features of objects.
= F=F((gg)) = F (F.
(I)
Where represents the model integrating multi-view cam- era image information for 3D semantic occupancy predic- tion. The model then generates the semantic voxel volume Ô E (Pe.Pi...... PC xxW xL as the output. In this context, the parameter ranges from O to 1, presenting the proba- bility of voxel grid occupancy. The variable C' denotes the total count of semantic classes within the scene, with Pe indicating the probability that the grid contains an empty occupied voxel. Additionally, H. W, and L represent the height, width, and length of the voxel volume, respectively.
3.2. Overall Architecture
Fig. 3 illustrates the overall architecture of our method. Given
of surround-camera images, we first use an image feature extractor (such as 10s (Dai al., 2017)) to extract multi-scale surround-camera, features F H 1 j= at m scales from N cameras individually. In the Time-based View Transformation (TVT) module, for the surround cameera features at each level, we utilize voxel queries to learn volume features and use the cascaded Cross
1am
3

---

CSV-Oce: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
State Space Module (Cross SSM) to fuse the volume fea- tures of historical frames into the current frame enrich the feature information. Then, the Voxel Feature Extractor is used to upsample and concatenate the multi-scale volume features to form high-resolution volume features. Next, the occupancy classification head predicts the coarse semantic occupancy based on the high- high-resolution volume features. In the Voting-based based Enhancement Mechanism (VEM), the model explicitly predicts the relative displacement of each foreground voxel to the instance center it belongs to through Relative Central Regression (RCR), and then updates the high-resolution voxel features through the Parallel Interac- tion Strategy (PIS). Finally, the occupancy classification head predicts the refined semantic occupancy.
Cross SSM: Cross State Space Module
Embodding
A,D
Patch
Pantbn
Patch Merger
MLP
Ay mime & hus (4 0.,,
3: - G.fy + D
Pazsh Portion
MLP
y Br.Yo....1h
Figure 4. Illustration of cross state space module. We take the 3D feature maps of the current frame and the past frames simul- taneously the input. Where @ represents the element-wise multiplication of tensors.
3.3. Time-based View Transformation
We project the reference voxels to the camera views and use deformable attention (Zhu et al., 2020) to query pixels and aggregate information (More details are in Sec. A of the Appendix).
we believe that the C variable in the state space model can also be obtained by mapping another input feature sequence tensor similarly.
Thus, shown in Fig. 4, we take the 3D voxel features of the current frame T and the 3D voxel features of the histor- ical frame T - i simultaneously as the input of the Cross SSM. Vr- is taken as x E 1 B E RS 8xdaa is ob- tained by the MLP mapping of x, A E R Sxxraa is obtained by the MIP mapping of B. C E SR Xdem is obtained by the MLP mapping of Vy.. Both A E RRdem xdeaa and D E 12 are learnable tensors. Among them, datae represents the SSM state dimension. Where S represents the length of the input sequence. Specifically, we expand Vr-; and Vy into sequences of the same size IR Sx through patch partition respectively, and then use them as the input of the state space model. Finally, the output of the state space model is reconstructed as Vr E xxxxxxxxdm through patch merger. In the patch partition, we use simple pooling downsam- pling and tensor flattening without introducing any complex processes, and the patch merger is implemented by simple reshape and interpolation upsampling.
To effectively obtain better multi-frame voxel features, we design cascading structure after the deformable attention mechanism to gradually fuse the information in the multi- frame voxel features, as shown in Fig. 3. Based on the obser- vation that directly adding or concatenating voxel features from different frames leads to performance degradation (as shown in the ablation experiments in Table 3), we decom- pose this interaction from the current frame voxel feature to the fused voxel feature Vi E R dlxd into k cascading steps. Between the input current frame voxel feature and the fused cascaded voxel feature, we refer to the intermediate voxel feature containing information from different frames as V4.+ E h xxxxxxxddnn Step by step, the model gradually fuses the information of historical frames to effectively and efficiently learn the final occupancy descriptor Vr.
3.3. 1. CROSS STATE SPACE MODULE
Inspired by Mamba (Gu& & Dao, 2023), we achieve the fu- sion and interaction of framer- 3D voxel features through SSM (Gu& & Dao, 2023; Mehta et al., 2023; Wang et al., 2023a). To achieve this goal, we need to expand the vanilla Mamba into multiple inputs.
3.4. Voting-based Enhancement Mechanism
Based on the observation that the IOV problem is prevalent in semantic occupancy prediction (as shown in Fig. 1 and Fig. 2), we developed the RCR and PIS mechanisms, which can learn to spontancously utilize the coarse semantic oc- cupancy predictions and high-resolution volume features to fill the vacant semantic features inside the objects.
The original Attention formula (Dosovitskiy et al., 2021; Vaswani, 2017) can be approximated as a SSM through simplification, addition of new terms, and transformation (derivations are in Sec. B of the Appendix). This makes possible for SSM to achieve the interaction of different input sequences. And we found in the derivation that the role of the C variable in the state space model is similar to the query tensor in Attention (in Cross Attention (Lin et al., 2022; Gheini et al., 2021), query tensor can be obtained by mapping another input feature sequence tensor). Therefore,
1.4.1. RELATIVE CENTRAL REGRESSION
The foreground class objects often have the IOV problem (there are a total of 10 types of foreground objects such as cars, trucks, bicycles, etc. in the nuScenes dataset). In the nuScenes dataset, we only focus on the refinement of the internal voxels of these 10 types of objects and do not need
4

---

CSV-Oce: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
As shown the right side of Fig. 6, taking example, first, seen is used as the key and value, and fual is used as the query to perform cross attention calculation respectively and is added with the residual of fool to obtain the updated feature foal:
as an
RCR: Reletive Central Regression
Conrse
Saraatt tee Ocoupancy
XUXIC +1)
Class Minsc
MLP
FL CrossAttn (Fiet Sden) + foal for j 1, 1........ G1
NwW xLxxd..oo
(4)
For each fvod E R we perform global pooling to obtain a feature descriptor E R and then sort these feature descriptors. Use this order as the sequence of each Mia to obtain ft..
Figure 5. Illustration of relative central regression. We use the BEV plane for a simple 2D illustration, while it is actually a 3D process. Different colors of the blocks represent different cate- gories of voxels. Gray indicates other non non-foreee foreground categories, blank indicates non-oce ooc-occd areas, and the dashed arrows indicate the relative direction and distance of the central voxels voted by the voting voxels.
Sort (f
G1 |== ) GP
G1 j=1
21, vot
j=1
to refine the entire occupancy space, otherwise, it will bring a huge amount of computational redundancy. As shown in Fig. 5, we first filter out the voxel coordinates occupied by the above 10 types of objects through coarse semantic occupancy predictions, and use them the class mask to filter the high-resolution volume features, that is, only the voxel features Vr, E R Niga dood of the above 10 types of objects are retained:
Where, GP represents the global pooling function. After determining the order of f'ul use it the key and value, and f as the query, and perform cross attention calculation in sequence to obtain the updated feature flm:
fu. = CrossAttn (eear fvval
for j = 1, 1,...,Gr G1 (6)
Vrigga Mask (Vr, Ocuars))
3.5. Supervision Signal
Where Ôccarse E {pe: Pi.......... PC)
| XXXXL represents the coarse semantic occupancy prediction. For these features, we predict the offsets 8een E NR of these voxels from the central voxels of their respective objects (including the voxel distances Az; Ay,, Az in the H, W, and L direc- tions) through the MLP:
Occupancy Classification. We adopt simple two-layer MI. for voxel-wise multi-classification (including occu- pancy status and semantic category) of high-resolution voxel features without other complex designs. Then, the occu- pancy prediction Ögeo E { HxW XL and the semantic prediction Ôsem E {(11 .P..P2..... PC
}c) H W x are output.
Loss Function. The total loss consists of two parts:
een = MLP (Vr,
L= L= + LRCR
Thus, obtain the voting relationship (that is, the voxel- relative central voxel pair) of each voxel to the central voxel of its respective object.
The occupancy classification head is supervised by Loce- Specifically, Love consists of the focal loss focal) (Ross & Dollár, 2017), the Lovasz loss (Liow) (Berman et al., 2018), and the scene-class affinity loss (Lufí) (Cao & De Charette, 2022). Among them, Laff calculates the losses Lafs(Osem: Osem) and Lafs( ('gen Ogen) for the semantic labels (including voxel semantic categories) and the geometric labels (including only the voxel occupancy status), respectively. Loce is calculated as follows:
3.4.2. PARALLEI INTERACTION STRATEGY
The voxel-relative central voxel pairs can be one-to-one or many-to-one (that is, is possible that multiple voxels vote to the same center voxel of the belonging object), as shown on the left side of Fig. 6. For each voted central voxel feature f there are G; voting voxels features { j== that have a voting relationship with it, where represents the index of the vexel-relative central voxel group. We hope that they can update their own features through sufficient interaction.
Coc = >ffeca + Ntov + Naff
(8)
Lass = Lats(O (0mm: Osem) ) + Laf O Ogeo)
5

---

CSV-Oce: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
PIS: Parallel Intoraction Strategy
Valing
Reletionsip
CA
f
Vating Retationa in
Foding
Soriing
Grobal Faalurrs
Local Festures
KY
CA
CA
Figure 6. Illustration of parallel interaction strategy. The voxel-relative central voxel pairs conduct feature interaction through cross- attention calculation, thereby updating all foreground voxel features.
RCR is supervised by CRCR. Specifically, we utilize the bounding box labels, project them onto the volumetric coor- dinate system, obtain the voxels to which the center of each object instance belongs, and calculate the distance deviation of all voxels within the bounding box of each instance to the central voxel to obtain deen. We predict a three-dimensional vector for each voxel belonging to the foreground classes to regress the relative center offset and use the MSE loss for supervision:
We train our models for all experiments over 24 epochs. We apply common data augmentation strategies, including color transformations, flips, rotations, and scaling in both image and 3D space.
4.2. Main Results
3D Semantic Occupancy Prediction. In Table 1, we present a comparative experiment with other state-of-the- art 3D semantic occupancy prediction methods on Oce3D- nuScenes (Tian et al., 2024) validation set. Due to the default image backhone of FB-OCC (Li et al., 2023c) being R50 (ResNet50) (He et al., 2016) and the resolution of the input images differing from that of other methods, we modi- fied the image backbone and resolution in the FB-OCC (Li et al., 2023c) open-source code to ensure a fair evaluation. Compared to previous methods, our approach demonstrates superior performance in terms of mloU, particularly ex- celling in medium and small foreground classes (such as barrier, pedestrian, bicycle, and car). This highlights that using voting- enhancement mechanism for handling foreground classes aligns better with scene characteristies and enhances the ability for semantic filling within objects. Additionally, our method performs exceptionally well in medium and large scene structure classes (such as driveable surface and sidewalk), emphasizing that utilizing a cross state space module for the fusion and interaction of inter- frame 3ID voxel features is more in line with the temporal variations of the scene, thereby enhancing the ability to leverage temporal information.
LRCH ARCRMSE (Beem dcen)
4. Experiments
4.1. Implementation Details
Network. For CSV-Occ-B. B. the input image resolution is 1600 x 896. We employ ResNet101-DCN (Dai et al., 2017) as the image backbone. The Feature Pyramid Network (FPN) (Lin et al., 2017) extracts multi-scale features with downsampling strides of 8, 16, 32, and 64. The time-based view transformation module fuses frames with a frame interval of 2. The voxel feature extractor adopts 3 layers of 3D deconvolution to upsample the BEV size by 2x and the height by 4x The occupancy classification head consists of two layers of 128-dimensional MLP and softmax activation, generating an occupancy space resolution of 200 x 200 x 16. For CSV-Occ-S, the input image size is reduced to 800 800x448 448, and we employ ResNet50 (He et al., 2016) as the image backbone, with the remaining network details the same as CSV-Occ-B.
LiDAR Semantic Segmentation. To better assess the effec- tiveness of our method, we project the 3D semantic occu- pancy prediction results onto the point cloud and conduct comparative experiments on the LiDAR semantic segmen- tation task. As shown in Table 2, compare our results with those of other 3D semantic occupancy prediction meth-
Training. For CSV-Occ-B and CSV-Occ-S, we train the models on 4 NVIDIA A40 48G GPUs with batch size set to 4. During training, we utilize the AdamW (Loshchilov, 2017) optimizer with an initial learning rate of 2 x 10 -4 weight decay of 0.05, and apply cosine annealing strategy.
6

---

CSV-Oce: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
Table 3D semantic occupancy prediction performance on Oec3D-nuSeenes dataset. In all methods, only a single-mode camera image is used as input during testing, without any additional augmentations model ensembles incorporated. "4f" and refer to the model's imput containing data from 4 frames or 8 frames, respectively incorporating temporal information. The symbol "e" marks that these models use the visible mask during training.
SONAE VIT
MUTORCYCLA
VESETTATION
METIOOSS
IAAG
PUBLICATION
MIOUU
Mosnteess &Cho&& The CHARETTS, 2003)
R161
CNP4113
6.06
1.75
7.33
4.76
6.93
0:38
560
3.98
3.01
5.90
1.65
1.17
16.91
6.33
7.93
7.43
1.01
7.65
BENDEE ORGANG sl., 20211
B101
11:23
3,89
13:29
4.18
13.97
0.00
5.56
53.03
19.04
26.43
21:74
14.31
(ZHAWOTT TPVPeessI (ESANO ET Al. 3023)
30036)
0.300
13.33
1.40
R161
CXPR-13
28:34
0.67
19.20
14:24
41.54
46.09
19.21
22.64
13.87
14.34
39.20
33.31
56.18
35.63
35.00
31.01
19.97
16:12
CIF-Oce (RAM 63 al., 2014)
M161
NIPS 21
25.33
3.89
19.33
30.35
38.29
42.54
16.99
24.35
22.72
21.05
22.95
31.13
53:31
35.84
37.94
35.23
20.79
18.00
REV (rr (L)GT Ml., 2023
R181
B2CV122
50.24
10.13
47.01
24.5
47.57
54.52
20.33
21.85
23.62
25.33
33.03
58.35
$1.95
40.65
50.93
55,02
43.06
35.15
BE (h)
SwIndB
axX 73
03:03
12:19
1966
$3.00
37.7
17.99
22.35
56.13
82.3
05.61
FS-OCC dan (La 3023cy"
R50
ICCVV22
40.67
10.c3
47.54
20.02
45.76
50.15
25.45
28.56
23.00
29.51
54.31
57.44
50.00
46.51
54.3
57.14
44.95
35.02
FI-OCE (en (1a
M111
03:34
10.90
19
17.53
15.00
08
86
16
19.89
56.3
16.43
LXV-0ce cass
Quiz
62.31
11:75
20.65
29.20
70.05
15:13 12
20.14
36.9%
23:70
30.38
38.80
62.04
82.53
62.35
56.00
36.92
45.97
40.05
(4ry*
14.13
33.14
Table LiDAR semantic segmentation results on nuScenes validation set. "4f" refer to the model's input containing data from 4 frames, incorporating temporal information. In Modality, "L" and "C" represent LiDAR input and camera input respectively.
MOLL
STTER PLANT
TRICK
METHODS
IMAOR
MONACITY
POREICATION
RANGENET- (MILaro POLARNEY (ZAANN ET AL.. 2020)
2019)
1905/19
65.5
66.0
21.3
77.2
50.9
30.2
66.8
69.6
52.1
54.2
72.3
94.1
66.6
10.1
83.1
79.8
CVPR'20
71.0
74.7
28.2
35.3
90.9
35.1
77.5
71.3
58.8
57.4
76.1
96.3
71.1
74.7
14.0
87.3
85.7
SALSANEXT (CORTIOAAA ET Ac., 2020)
ISVC 20
72.2
26.8
54.1
85.9
38.4
42.2
72.8
72.3
53.1
61.3
76.5
96.0
70.8
21.2
71.5
86.7
84-4
CYLINEE3DD++ (2a0 AL., 2021)
CVPR
76.1
76.4
40.3
91.2
51.5
78.0
54.9
62.1
96.8
71.6
76.2
75.4
98.5
(X0
43.4
66.9
73.5
75.5
76.0
REVFono (LFRT at.,, 2022]
K161
BOC 22
36.2
54.0
22.8
26.7
74.0
45.8
53.1
46.5
14.7
54.7
65.5
$5.5
59.1
50.5
52.8
71.0
63.0
IPVFORMER (HOANN ar at., 20233
R101
CV19 25
68.9
70.0
60.9
93.7
85.6
49.8
68.4
59.3
16.2
65.3
83.0
93.3
64.4
64.3
54.5
8i.6
78.3
Pasodee (4a) (Wasor in Al.,, 2024)
830
CVPR124
08.1
70.7
57.9
03.3
85.0
50.7
64.3
59.4
35.3
63.8
81.6
54.3
66.4
04.8
68.0
79.1
75.6
PosoOce (49) (WANG ET Al., 2014)
K101
21.6
74.3
43.7
05.4
$7.0
56.1
64.6
66.2
41.4
71.5
85.9
95.1
70.1
67.0
58.1
90.9
71.4
CSV (dr)
R50
Oues OUKS
41.7
51.3
02.5
41.1
05.1
95.5
07.4
69.2
56.5
85.8
75.3
CSv (40)
R161
73.4
74.4
44.1
95.6
89.5
53.9
65.8
67.5
45.2
72.6
88.5
97.3
71.9
72.4
69.8
97.3
76.9
ods and LiDAR semantic segmentation methods based on point cloud inputs on the nuScenes validation set. Our method achieves the best mIoU performance among the image-based 3D semantic occupancy prediction methods, demonstrating excellent results for specific semantic classes (such as pedestrian, bicycle, and car). This success can be attributed to the feature correction and interaction capabil- ities of the voting- based enhancement mechanism and the cross-state space module. In comparison with the LiDAR semantic segmentation methods based on point cloud inputs, we have approached the performance of current state-of-the art lidar-based methods using purely image input.
affects the mloU, with the best performance achieved when the state dimension is set to By comparing #B, #E, #F, and #G, it is discovered that Cross SSM outperforms the utilization of temporal self-attention (Li et al., 2022), MLP mixer (Tolstikhin et al., 2021), direct concatenation. This indicates that Cross SSM is more capable of concentrating on the differences and similarities between frames, making it more suitable for the integration and interaction of inter- frame voxel features.
Time-based View Transformation and Voting-based En- hancement Mechanism. Table 4 verifies that TVT and VEM can have positive effects on the model. Compared with the single-frame model, the multi-frame training and inference model using TVT performs better in the 3D seman- tic occupancy prediction and LiDAR semantic segmentation tasks. VEM further enhances the interaction between object surface and internal feature learning, and also improves the performance of both tasks. The effect of VEM is shown in Fig. 7, and it can eliminate greatly alleviate IOV.
4.3. Ablation Study
We perform ablation experiments to evaluate the design choices of CSV-Occ Occ3D-nuScenes (Tian et al., 2024) validation set. By default, we utilize the CSV-Occ-B config- uration.
Different Multi-frame Feature Aggregation Methods. Table 3 compares the performance of different voxel feature aggregation methods used in the TVT module. #b is the default setting. By observing #A, #B, #C, and #D, we find that adjusting the state dimension in the Cross SSM
The Weights of the Loss Function. Table 5 shows the com- parison of various combinations of different loss function weights. Comparing Anea #A and #C, #B and #D, it can be found that L LECR can bring significient gains to the
7

---

CSV-Oce: Fusing Multi-frame Alignment for Occ. Pred. with Temporal Cross SSM and Central Voting Mechanism
Weo. VEM
VEN
Figure 7. Visualizations the occupancy predictions IOV with without VEM.
Table Ablation study different multi-frame voxel feature aggregation methods. TSA stands for the temporal self- att-nnttntion
structure.
STATE DIMENSION
MIOU (Occ.)
MIOU, (SEG.)
TRAINABLE PARAMS (M)*
#
FUSION MODULE
A
44.77
73.05
68.3
44.93
73.42
68.5
CROSS SSM (OuRS)
C D
44.89
73.41
68.9
16
43.58
72.70
69.6
E
TSA (LI ET AL., 2022)
43.17
72.62
69.3
F
MLP-MIXER (TOLSTIKHIN ET AL., 2021)
43.22
72.67
77.5
G
CONCAT. AND CONV.
42.31
72.53
68.8
Table 4. Effects of time-based view transformation and voting- based enhancement mechanism. TVT not used, we replace it with SCA and TSA of BEVFormer (Li et al., 2022).
Table 5. Ablation for the weights of the loss function.
MloU (Occ.) 1
MIoU
MIOU (Occ.)
MIOU, (SEG.)
ARCE
focul
Aivo
Aufs
(SEG.)
TVT
VEM
A
2.0
69.79
2.0
2.0
43.98
72.80
A
40.37
69.81
B
43.71
72.77
C
2.0
42.83
71.91
V
42.08
71.33
D
2.0
2.0
1.5
44.93
73.42
1.0
44.93
73.42
5.5
2.0
1.5
44.78
73.30
F
2.0
2.0
4.0
44.04
72.95
performance, which also means that explicitly supervising the voxel offsets can enable the model to more clearly learn the physical meaning of the center voting. We also con- various combinations of different loss function weights and found that the performance is the best when ARCR = Afoca 2, Nov 2, and Raff 1.5.
4, adjusted the BEV side length from 25 to 200 (max 200 200, not exceeding occupancy ground truth). The left figure shows samples inferred per second; the right shows inference memory consumption. As voxel query size grows, especially past 100, TSA and MLP mixer's inference speed and memory consumption worsen. Cross SSM's efficiency decline is more stable due its linear computational com- plexity. A larger query size means longer flattened token sequence, and Cross SSM needs only one scan for multi frame fusion.
ducted experiments
4.4. Temporal Computation Complexity
The following efficiency experiments were all measured with the batch size set to 1.
Fig. shows the impact changes in BEV size on infer- ence speed and memory. We explored how initialized voxel query size impacts model efficiency. Keeping its height at
Fig. 9 shows how the number inference frames affects model efficiency. The initialized voxel query size set at 100 100 4. Cross SSM outperforms in both inference
8

---

CSV-Oce: Fusing Multi-frame Alignment for Occ. Pred. with Temporal Cross SSM and Central Voting Mechanism
21000
16.
19619
Samples per Second (SPS)
Inference Memory (MB)
17000
13.3
124
13549
1233
8.7
13000
Cross SSM (Ours)
9000
-SA
MLP mixer
5000
2840
2040
3274
5588
0712
2344
3078
25
100
200
100
200
(Full Length)
(Full Longth)
BEV Side
BEV Side Leggth
Figure 8. The impact of changes in BEV size on inference speed and memory.
7250
Samples per Second (SPS)
10
16.1
Infercoce Memnryy (MB)
7159
93
Cross SSM (Ours)
8.
84
62.50
8515
8.4
6195
6224
5071
MLP mixer
5750
7.5
5473
74
7
5250
Inference Framess
Inference Frames
Figure 9. The impact of changes in the number of inference frames on inference speed and memory.
speed and memory consumption. Still, as the number of inference frames rises, Cross SSM's efficiency change trend is like that of TSA and MLP mixer. In CSV-Oce, the num- ber of inference frames equals the number of multi-frame fusion module calls, and the feature sequence length per call depends only on the voxel query size. So, more inference frames don't 't give Cross SSM a trend advantage.
future 3D scene perception.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences our work, none which we feel must be specifically highlighted here.
5. Conclusion
Acknowledgements
In this paper, propose CSV-Occ, novel based 3D semantic occupancy prediction method. CSV-Oce inte- grates temporal information through an extended state space model and enhances feature representation in a coarse-to- -torse--- fine manner by explicitly predicting the instance to which voxels belong, which achieves comprehensive understand- ing of the scene. Extensive experimental results on the nuScenes dataset and Dee3D-nuScenc confirm the effec- tiveness of CSV-Occ and its potential in advancing 3D se- mantic occupancy prediction. In our view. 3D semantic occupancy representation is promising new paradigm for
This work was supported in part National Natural Science Foundation of China under Grant 62476088, in part by the Shanghai Automotive Industry Science and Technology Development Foundation under Grant 2304, and in part by the Science and Technology Commission of Shanghai Municipality under Grant 20DZ2254400.
References
An, P., Zhu, D., Quan, S., Ding, J., Ma, J., Yang, Y.. and Liu, Q. Esc-net: Alleviating triple sparsity 3d lidar

---

CSV-Oce: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
point clouds for extreme sparse scene completion. IEEE Transactions on Multimedia, 26:6799-6810, 2024. doi: 10.1199/TMM.220433355:.
Gheini, M., Ren, X., and May, J. Cross-attention is all you need: Adapting pretrained transformers for machine translation. arXiv preprint arXiv:2104.08777: 2021.
Bai, X., Hu, Z., Zhu, X., Huang, Q. Chen, Y., Fu, H., and Tai, C.-L. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1090-1099, 2022.
Gu, A. and Dao, T. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXi::23/2.0077 2023.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leam- ing for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Berman, M., Triki, A. R., and Blaschko, M. B. The lovász- softmax loss: A tractable surrogate for the optimization of the mitersettion-ove--uiioo measure in neural networks. In Proceedings of the IEEE conference on computer vi- sion and pattern recognition, pp. 4413-4421, 2018.
Huang, J. and Huang, G. Bevdet4d: Exploit temporal cues in multi-camera object detection. arXiv preprint arXiv:2203. /7054, 2022.
Caesar, H., Bankiti, V., Lang, A. H., Vora, S., Liong, V. E.,, Xu, Q.. Krishnan, A., Pan, Y., Baldan, G., and Beijbom, O. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp. 1 11621-11631, 2020.
Huang, J., Huang, G., Zhu, Z.. Ye, Y., and Du, D. Bevdet: multi-camera 3d object detection in bird-eye-viee arXiv preprint 1rXiv:2112.11 17900 2021.
Huang, Y., Zheng, W., Zhang, Y., Zhou, J., and Lu, J. Tri- perspective view for vision-based 3d semantic occupancy prediction. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 9223 9232, 2023.
Cao, A.-Q. and De Charette, R. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1991-4001, 2022.
Chen, Y., Liu, J., Zhang, X., Qi, X., and Jia, J. Voxelnext: Fully sparse voxeinet for 3d object detection and tracking. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 21674-21683, 2023.
Hurtado, J. V., Mohan, R., Burgard, W., and Valada, A. Mopt: Multi-object panoptic tracking. arXiv preprint arXiv:2004.088 0...0004.///0 089, 2020.
Jang, H.-K., Kim, J., Kweon, H.. and Yoon, K.-J. Talos: En- hancing semantic scene completion via test-time adapta- tion on the line of sight. arXiv preprint arXiv:2410.15674 2024.
Çiçek, Ô., Abdulkadir, A., Lienkamp, S. S., Brox, T., and Ronneberger, O. 3d u-net: learning dense volumetric segmentation from sparse annotation. In Medical Image Computing and Computer-Assisted Intervention-MICCAl 2016: 19th International Conference, Athens, Greece, October 17-21, 2016, Proceedings, Part 11 19, pp. 424- 432. Springer, 2016.
Koh, J., Lee, J., Lee, Y., Kim, J., and Choi, J. W. Mgtanet: Encoding sequential lidar points using long short-term ootion-guided temporal attention for 3d object detec- tion. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1179-1187, 2023.
Cortinhal, T., Tzelepis, G., and Erdal Aksoy, E. Salsanext: Fast, uncertainty-awant semantic segmentation of lidar point clouds. In Advances Visual Computing: 15th International Symposium, ISVC 2020, San Diego CA, USA, October 5-7, 2020, Proceedings, Part I1 15, pp- 207-222. Springer, 2020.
Lang, A. H., Vora, S., Caesar, H., Zhou, L., Yang, J., and Beijbom, O. Pointpillars: Fast encoders for object detec- tion from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12697- 2019.
Dai, J., Qi, H., Xiong, Y., Li, Y., Zhang, G., Hu, H., and Wei, Y. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision,
Li, Y., Ge, 7.,, Yu, G., Yang, J., Wang, 7., Shi, Y., Sun, J., and Li, Z. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1477-1485, 2023a.
pp. 764-773, 2017.
Dosovitskiy,
A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. ICIRR 2021.
Li, Y., Yu, Z. Choy, C., Xiao, C., Alvarez, J. M., Fidler, S., Feng. C., and Anandkumar, A. Voxformer: Sparse voxel transformer for camera- based 3d semantic scene completion. Proceedings of the IEEE/CVF conference
10

---

CSV-Oce: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
on computer vision and paltern recognition, pp. 9087- 9098, 2023b.
Ma, J., Chen, X., Huang, J., Xu, J., Luo, Z., Xu, J., Gu, W., Ai, R., and Wang, H. Cam4doce: Benchmark for camera-only 4d occupancy forecasting in autonomous driving applications. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21486-21495, 2024.
Li, Z., Wang, Li, Xie, E., Sima, C., Lu, T., Qiao, Y., and Dai, Bevformer: Learning bird's e-yee-vew repre- sentation from multi-cament images via spatiotemporal transformers. In European conference on computer vision, pp. 1-18. Springer, 2022.
Mehta, H., Gupta, A., Cutkosky, A., and Neyshabur, B. Long range language modeling via gated state spaces. ICLR, 2023.
Li, Z., Yu, Z., Wang, W., Anandkumar, A., Lu, T., and Alvarez, J. M. Fb-bev: Bev representation from forward- backward view transformations. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
Mescheder, L., Oechsle, M., Niemeyer, M., Nowozin, S., and Geiger, A. Occupaney networks: Learning 3d re- construction in function space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14600-4700 2019.
2023c.
Lin, H., Cheng, X., Wu, X., and Shen, D. Cat: Cross atten- tion in vision transformer. In 2022 IEEE international conference multimedia and expo (ICME), pp. 1-6. IEEE, 2022.
Milioto, A., Vizzo, I. Behley, J., and Stachniss, C. Rangenet++: Fast and accurate lidar semantic segmen- tation. 2019 IEEE/RSJ international conference on intelligent robots and systems (IROS), pp. 4213-4220 IEEE, 2019.
Lin, T.-Y., Dollár, P., Girshick, R., He, K., Hariharan, B., and Belongie, S. Feature pyramid networks for object detection. In Proceedings of the IEEE conference computer vision and pultern recognition, pp. 1p.21172 2117-2125, 2017.
Pan, M., Liu, J., Zhang, R., Huang, P., Li, X., Xie, H., Wang, B., Liu, L., and Zhang, S. Renderoce: Vision-centric 3d occupancy prediction with 2d rendering supervision. In 2024 IEEE International Conference on Robotics and Automation (ICRA), pp. 12404-12411. IEEE, 2024.
Liu, H., Teng, Y., Lu, T., Wang. H., and Wang. L. Sparse- bev: perforreffo sparse 3d object detection from multi-camera videos. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 18580- 18590, 2023a.
Park, J., Xu, C., Yang, S., Keutzer, K., Kitani, K., Tomizuka,
M., and Zhan, W. Time will tell: New outlooks and a baseline for temporal multi-view 3d object detection. arXiv preprint arXiv:2210.02443,, 2022.
Liu, H., Chen, Y., Wang, H., Yang, Z., Li, T., Zeng, J., Chen, L., Li, H., and Wang, L. Fully sparse 3d occupancy prediction. arXiv preprint arXiv:2312.
2024a.
Peng, S., Niemeyer, M., Mescheder, L., Pollefeys, M., and Geiger, A. Convolutional occupancy networks. In Com- puter Vision-ECCh 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pp. 523-540. Springer, 2020.
Liu, Y., Yan, J., Jia, F., Li, S., Gao, A., Wang, T., and Zhang, X. Petrv2: A unified framework for 3d perception from multi-camera images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3262- 3272, 2023b.
Ronneberger, O., Fischer, P., and Brox, T. U-net: Con- volutional networks for biomedical image segmenta- tion. In Medical image computing and computer-assister intervention-MICCA. 2015: 18th international confer- ence, Munich, Germany, October 5-9, 2015, proceedings. part III 18, pp. 234-241. Springer, 2015.
Liu, Y., Tian, Y., Zhao, Y., Yu, H., Xie, L., Wang, Y., Ye, Q., and Liu, Y. Vmamba: Visual state space model. arXiv preprint 4rXiv:2401.10166, 2024b.
Liu, Z.., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., and Guo, B. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Ross, T.-Y. and Dollár, G. Focal loss for dense object detection. In proceedings of the IEEE conference computer vision and pattern recognition, pp. 2980-2988, 2017.
Loshchilov, Decoupled weight decay regularization. arXiv preprint 1rXiv:1711.05100 2017.
Shi, H., Wang, S., Zhang, J., Yin, X., Wang, Z., Wang, G., Zhu, Yang, K., and Wang, K. Offboard occupancy re- finement with hybrid propagation for autonomous driving. arXiv preprint arXiv:2403. 08504, 2024.
Lu, Y., Zhu, X., Wang, T., and Ma, Octreeace: Efficient and multi granularttuu occupancy prediction using octree queries. arXiv preprint 132.7.2312.03774, 2023.

---

CSV-Oce: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., and Li, H. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10529-10538, 2020.
Wei, Y., Zhao, L.,, Zheng, W., Zhu, Z.., Zhou, J., and Lu, J. Surroundoce: Multi-camera 3d occupancy prediction for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 21729- 21740, 2023.
Sirohi, K., Mohan, R.,. Büscher, D., Burgard, W., and Valada, A. Efficientlps: Efficient lidar panoptic segmentation. IEEE Transactions on Robotics, (83):1894-1114 2021.
Xu, J., Zhang, R., Dou, J., Zhu, Y., Sun, J., and Pu, S. Rpvnet: A deep and efficient togge-ooit--oxel fusion network for lidar point cloud segmentation. In Proceed- ings of the IEEE/CVF international conference on com- puter vision, pp. 16024-16033, 2021.
Tian, X., Jiang, T., Yun, L., Mao, Y., Yang. H., Wang, Y., Wang, Y., and Zhao, H. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. Advances in Neural Information Processing Systems, 36, 2024.
Yang, Z., Sun, Y., Liu, S., and Jia, J. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11040-11048, 2020.
Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, L., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. Mlp-mixer: An all-mlp architec- ture for vision. Advances in neural information process- ing systems, 14:24261-24272 2021.
Ye, D., Zhou, Z., Chen, W., Xie, Y., Wang, Y., Wang. P., and Foroosh, H. Lidarmultinet: Towards a unified multi- task network for lidar perception. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 3231-3240, 2023.
Tong, W., Sima, C.,, Wang, T., Chen, L.., Wu, S., Deng, H., Gu, Y., Lu, L., Luo, P., Lin, D., et al. Scene as occu- pancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8406-8415, 2023.
Ye, Z., Jiang, T., Xu, C., Li, Y., and Zhao, H. Cvt-occ: Cost volume temporal fusion for 3d occupancy prediction. In European Conference on Computer Vision, pp. pp. 381-397. Springer, 2024.
Touvron, H., Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transform- ers& distillation through attention. In International con- ference on machine learning. pp. 10347-10357. PMLR, 2021.
Yin, T., Zhou, X.,, and Krahenbuhl, P. Center- 3d object detection and tracking. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 11784- 11793, 2021.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Wang, J., Zhu, W., Wang, P., Yu, X., Liu, L., Omar, M., and Hamid, R. Selective structured state-spaces for long-form video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6387-6397, 2023a.
Zhang, X.,, Tian, Y., Xie, L.. Huang, W., Dai, Q., Ye, Q.. and Tian, Q. Hivit: A simpler and more efficient de- sign of hierarchical vision transformer. In The Eleventh International Conference on Learning Representations, 2023a.
Wang, X., Zhu, Z., Xu, W., Zhang, Y., Wei, Y., Chi, X., Ye, Y., Du, D., Lu, J., and Wang, X. Openoccupancy: A large scale benchmark for surrounding semantic occupancy perception. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 17850-17859, 2023b.
Zhang, Y., Zhou, Z., David, P., Yue, X., Xi, Z., Gong, B., and Foroosh, H. Polarnet: An improved grid representation for online lidar point clouds semantic segmentation. In Proceedings of the IEEE/CVF conference on computer 2020.
vision and pattern recognition, pp.
Zhang, Y., Zhu, Z., and Du, D. Oceformer: Dual-path transformer for vision-based 3d semantic occupancy pre- diction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9433-9443, 2023b.
Wang, Y., Guizilini, V. C., Zhang, T., Wang, Y., Zhao, H., and Solomon, Detr3d: 3d object detection from multi- view images via 3d-to-2d queries. In Conference on Robot Learning, pp. 180-191. PMLR, 2022.
Wang, Y., Chen, Y., Liao, X., Fan, L., and Zhang, Z. Panooce: Unified occupancy representation for camera- based 3d panoptic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1755--1168,,. 2024.
Zhou, Q. Cao, J., Leng, H., Yin, Y., Kun, Y., and Zimmer- mann, R. Sogdet: Semantic-occupancy guided multi-view 3d object detection. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 38, pp. 7668-766777 7676, 2024.
12

---

CSV-Oce: Fusing Multi-frame Alignment for Occ. Pred. with Temporal Cross SSM and Central Voting Mechanism
Zhu, L.,, Liao, B., Zhang, Q.. Wang, X., Liu, W., and Wang, X. Vision mamba: Efficient visual representation learn- ing with bidirectional state space model. arXiv preprint axX1v:240.004417, 2024.
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., and Dai, J. De- formable detr: Deformable transformers for end-to-end object detection. arXiv preprint 2rXiv:2010.04159, 2020.
7hu, X., Zhou, H., Wang, T., Hong, F., Ma, Y., Li, W., Li, H., and Lin, D. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9939-9948, 2021.
13

---

CSV-Oce: Fusing Multi-frame Alignment for Occ. Pred. with Temporal Cross SSM and Central Voting Mechanism
A. Details of Time-based View Transformation
Specifically, as shown in Fig. 3, we initialize a learnable 3D voxel query tensor Q Raxuxixdma as the basis for 3D spatial information aggregation. According to the given intrinsic and extrinsic parameters of the N cameras, we attempt to project the center coordinate of each voxel in the 3D voxel query onto the two-dimensional coordinate system of the N camera images (if there is view overlap in the surround-camera images, then some 3D voxels may be projected onto multiple views simultaneously) to obtain voxel-pixel pairs. Then, we sample the features around these projected 2D coordinates and achieve the aggregation of 2D features to 3D voxels through the deformable attention (Zhu et al., 2020) mechanism:
FP
DefAttn (QP,P(pP,)) X.))
(10)
Vhit
VEVA
Where FP and Q" denote the p-th element of the output feature and the 3D volume query, respectively. g" is the corresponding 3D coordinate of Q?, P is the 3D-to-2D projection function including intrinsic and extrinsic parameters, X, is the feature map of the i-th camera. Via represents the set of view indices projected by q". DefAttn represents deformable attention computation (Zhu al., 2020).
B. Derivations of Cross State Space Module
The length and feature dimension of the input sequence represented by S and d respectively. The original Attention formula is as follows:
QK T V
Attention(() (qaation(Q,A, Q, V) softmax
(11)
Vd
Where Q E Rxd, K E Sxd, and V = ffe1....,. us) E S are obtained by mapping the same input feature sequence tensor through three independent MIP mappings (in Cross Attention, Q can be obtained by mapping another input feature sequence tensor). For the convenience of subsequent derivation, we abandon the tensor value scaling and softmax, and we can obtain a new calculation:
Y =
(12)
Where Y E Sxd represents the output of the new calculation. Particularly, the j-th slice of along the feature channel dimension d, denoted as l E Sx1, can written as:
ybl QK V
(13)
We introduce the intermediate variable tensors W = (Wi,..., wis E Sxdxd and E E RSSSS, where the role of E is
similar to the Attention mask, and E is full matrix ensure the equality still holds. Meanwhile, let Q .g......sh and K K KO will = (ki....,ks. and we can obtain:
Q
b (eowin)
K WbT
E
vll
(14)
Where O represents the element-wist wise multiplication of tensors, and the division sign also represents the element-wise division of tensors. Introduce variable tensor H = (hı,..., hs} E Rxdxd construct a new term and add it to the original term. To ensure the equality holds, multiply the new term by the all-zero tensor O Sxd,
- = [ wli) 200 ht1++
(
K Wbi
② E
vil
(15)
14

---

CSV-Oce: Fusing Multi-frame Alignment for Occ. Pred. with Temporal Cross SSM and Central Voting Mechanism
Maintaining the same equality structure, replace O and E with E and M respectively while keeping the shape unchanged, where the element values of the lower triangular part of M are 1 and the rest are obtain ytl Y ty )s
RRSx1
where
=
ا = [ wir)
mp ((oo
o оD) ap + (oo
+ M
vbl
(16)
Wbi
The f-th slice of yb along the feature sequence dimension S, denoted as v! E R 1x
ا = a o rof
+
qt O
we
ki
(17)
Wi
Similarly, the 1-th slice of Y along the feature sequence dimension denoted as yr E IXxx and extract ge:
V1 qi (w,, 3 /11) Q1
Wr +
hig
vq)
Wt
k
qt u4 ④ h + We
Wi
(18)
=qhht
k Us
h. wi ④ hi + Wi
W
We define w4 Ii e and substitute for calculation, then transform he into the recursive form:
k vi Timer
h4 II بحم h1 + II
1=1
k Us c AAA
(19)
AlAit...
3 h1 + Aldit.nna
= Ade c 11-1 k Uz
(x1,..., Øs} E RSxd is the input feature sequence tensor, A RSxd and add skip connection Y/ to obtain the following formula in the form of state space model:
Substitute V = X B, C, where X
h4 = AA: O h + BT (x, O Ai)
(20)
1/1 C,h,, DOx
To sum up, we believe that the state space model can be regarded as a special subset of the Attention model, which makes it possible for the state space model to achieve feature interaction of two different input sequence tensors. And observing the above derivation, is not difficult to find that Q (in Cross Attention, Q can be obtained by mapping another input feature sequence tensor) gradually replaced by C' in the process, which means that we believe that the C variable in the state space model can also be similarly obtained by mapping another input feature sequence tensor. Thus, we constructed Cross SSM (Sec. 3 of the main text).
C. Datasets and Evaluation Metrics
The nuScenes Dataset (Caesar et al., 2020) comprises 1000 scene sequences, distributed as 700 in the training set, 150 in the validation set, and 150 in the test set. Each scene sequence is recorded at 20Hz frequency and lasts for 20 seconds.
15

---

CSV-Oce: Fusing Multi-frame Alignment for Occ. Pred. with Temporal Cross SSM and Central Voting Mechanism
Each sample contains RGB images from surround cameras and point cloud data from LiDAR sensor with 32 beams. For object detection task labels, key samples annotated rate of 2Hz with ground truth labels for 10 foreground object classes, including 3D bounding boxes and categories. In addition to the 10 foreground classes, semantic segmentation task labels also encompass background classes represented in per- per-point labels within the point cloud.
The Occ3D-nuScenes Dataset (Tian et al., 2024) is an extension the nuScenes dataset that expands the semantic occupancy prediction task label set. the egocentric coordinate system, the semantic occupancy space ranges from -40m 40m on the X and Y axes, and from -Im to 5.4m on the Z axis. The voxel size of the occupancy label is 0.4m 0.4m 0.4m (resulting include 17 categories, comprising 16 categories specifically designated for semantic segmentation tasks, well as "other" category. Additionally, visible masks are supplied for both the LiDAR and camera modalities.
spatial resolution 200 200 16 for occupancy labels). The semantic labels
Evaluation Metrics. We use the mean Intersection over Union (mloU) as the evaluation metric for assessing both LiDAR semantic segmentation and 3D semantic occupancy prediction tasks:
TP. mIoU = C IR++FR+FN
C
(21)
In this equation, TP,, FP;, and FN, represent the counts of true positives, false positives, and false negatives for class i, respectively, with C denoting the total number semantic classes.
D. Experimental results
FB
PanoCro
CeV
Ground
Figure 10. Qualitative comparison of 3D semantic occupancy prediction.
Visualization. Fig. 10 shows the qualitative results of our method and some other state-of-ther the-art methods on the Occ3D- nuScenes dataset, demonstrating that our method can understand the overall structure of the scene more completely and capture finer-grained geometric details.
Design of Time-based View Transformation. In the TVT module, initialize a learnalee voxel query convert image features into voxel space. Table 6 compares the impact of different voxel query sizes on the experimental results. #H is the default setting for our method. In settings #C, #D, #E, and #F, we changed the height dimension while fixing the BEV size. was observed that the coding density of height information significantly affects the performance of semantic occupancy prediction and segmentation tasks. Additionally, under a fixed number of voxels, we designed experiments where (1) #A and #C have 5k voxels, (2) #B and #D have 10k voxels, (3) #E and #G have 20k voxels, and (4) #F and #H have 40k voxels.
16

---

CSV-Oce: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
Table Ablation study for different voxel query sizes. The improvement BEV information density very conducive improving the fineness of realizing 3D scene understanding.
VoXEL QUERY SIZE
MIoU (Occ.)
MIOU, (SEG.)
VOXEL NUMBER
BEV SIZE
HEIGHT
A
8
5K
40.88
68.97
25 25
B
16
10K
42.38
71.91
C
2
5K
40.37
68.83
D
4
10K
42.51
71.93
50 50
E
8
20K
44.08
72.35
F
16
40K
44.91
73.06
G
2
20K
44.16
72.63
x 100
H
4
40K
44.93
73.42
Table 7. Ablation study for varying configurations for the number of aggregated frames and frame intervals. Key sample images
are sampled 2Hz (with frame interval 0.5s) in the OccnD-nuScenes dataset. The time span can calculated by the number fused frames and the number of frame intervals.
FRAME
TIME SPAN
MIoU (Occ.)
MIoU, (SEG.)
FRAMES
INTERVALS
1.5s
44.18
72.98
4
3.0s
44.93
73.42
6.0s
44.61
73.11
0.5s
43.59
72.42
E
2
1.0s
43.74
72.49
4
2.0s
43.97
72.56
6
3.0s
44.02
72.68
H
42.08
71.33
The results indicate that only the former experiment (1) outperforms the latter, suggesting that the improvement in mloU from enhancing the encoding density of height information is much smaller than that from increasing the encoding density of BEV information. This finding highlights the importance of fine-grained encoding of BEV information for achieving comprehensive scene understanding.
The Number of Frames and Frame Intervals. As mentioned above, one of the functions of the TVT module is to aggregate multi-frame voxel features. Table shows the impact of different settings for the number of aggregated frames and frame intervals on model performance in the TVT module. #B the default setting of our method. By comparing #A, #B, and #C, we find that #B (which selects a frame interval of length 2 while keeping the number of aggregated frames 4) can enhance the performance of the TVT module. However, observations from #D, #E, #F, and #G yield different results, with #G (which selects frame interval of length 6 while keeping the number of aggregated frames at 2) exhibiting the highest performance. From this, observe that the time spans of #B and #G are closer, indicating that the time span's impact on model performance is more significant. Comparing time spans that are relatively close, (1) #B, #F, and #G have time spans between 28 and 3s, while (2) #A and #E have time spans between 1s and 1.5s. We find that under similar time spans, an increase in the number of aggregated frames often leads to a better mloU, which underscores the importance of temporal information density. At the same time, we also discover that extending the time span does not infinitely improve model performance; in fact, when the time span is too long, the mloU of dynamic objects tends decrease. This due the dynamic ambiguity that arises from the information and features of dynamic objects extended time spans.
Design of Parallel Interaction Strategy. In Table 8, we conducted an ablation study on the effectiveness of the parallel interaction strategy (PIS). #A is a simple per-voxel baseline that processes features directly through a series of MLPs. #B introduces the parallel interaction strategy without feature descriptor sorting, which improves performance by 1.5 occupancy mloU and 0.56 segmentation mloU. To enhance the fusion efficiency of highly correlated features, we implemented feature descriptor sorting mechanism in #C and #D, which further increased inference accuracy, ultimately achieving the highest
17

---

CSV-Oce: Fusing Multi-frame Alignment for Occ. Pred. with Temporal Cross SSM and Central Voting Mechanism
Table Ablation study for parallel interaction strategy. The symbol represents the use paralle! interaction strategy.
MIOU (Occ.)
MIoU (SEG.)
#
PIS
SORTING
A
42.18
72.15
43.68
72.71
C
FORWARD
44.93
73.42
D
REVERSE
44.86
73.37
44.93 occupancy mIoU and 73.42 segmentation mIoU with forward sorting (compared to 44.86 occupancy mloU and 73.37 segmentation mloU with reverse sorting).
E. Limitation
Owithout VEM O VEM
90
80
66-1 18
68.24
20.18
mioU
50
48.73 27
68.09
40
40.71
44.93
41-11 13
441
45.07
$5.57
30
32.76
31:01
55.00 $3.04
35.07
20.97
33.94
20. 11
$3.40
36.40
$7.05
20
12:00
12:88
Iotal
others
barrier
bicycle
hus
car
const. veh.
motoreyele
pedestrian
traffic come
trailor
truck
drive, suf.
other flat
sidewalk
torainn
manmade
vegetation
Figure II. Performance comparison of each category with and without VEM.
Lower Performance on Certain Background Classes. Through statistics, we've found that the low performance of our method in certain background categories is primarily attributed to the VEM module. As depicted Fig. 11, upon enabling the voting mechanism, the performance all foreground categories has increased. Conversely, decline in performance observed in four background categories: driveable surface, other flat, manmade, and vegetation.
This occurs because VEM relies the coarse semantic occupancy prediction to divide foreground category voxels. When the coarse semantic occupancy prediction is inaccurate, VEM may predict relative centers for some background voxels as well. The occupancy of background categories usually has a large volume and lacks distinct instance centers. This leads to training confusion VEM. During inference, wrongly updates the features of background voxels, causing the occupancy classification head give incorrect category predictions. As result, the number of background voxels decreases, which directly impacts the prediction performance of background categories.
We also calculated the proportion of the number of foreground and background voxels predicted by the model with and without VEM. As shown in Fig. 12, it can be seen that VEM can significantly increase the number of foreground voxels. This partly indicates that VEM can improve the Internal Occupancy Vacancy (IOV) situation and successfully predict free voxels as foreground voxels. However, we unexpectedly found that the number of background voxels decreased, which we think is related to the training confusion caused by VEM.
Weak Instance-Leeve Occupancy Prediction. CSV-Occ can generate instance-level results, but requires additional
18

---

CSV-Occ: Fusing Multi-frame Alignment for Oce. Pred. with Temporal Cross SSM and Central Voting Mechanism
1.1
29.6
29.4
Foreground Voxels
Background Voxcls
Voxels
Without VEM
With VEM
Figure 12. The proportion of the number of foreground and background voxels predicted by the model with and without VEM.
Table 9. Comparison of instance-level occupancy prediction.
METHOD
MODALITY
PUBLICATION
PQ
SQ
RQ
PANOPTICTRACCKNTT (HURTADO ET AL., 2020)
L
ARXIV'20
51.4
80.2
63.3
EFFICIENTLPS (StROHI ET AL.,, 2021)
L
I-TR'21
62.0
83.4
73.9
LIDARMULITINET (YE ET AL., 2023)
L.
AAAT'23
81.8
89.7
90.8
PANOOCC (WANG ET AL., 2024)
C
CVPR'24
62.1
82.1
75.1
CSV-OCC-INSTANCE
C
OURS
48.3
79.6
60.5
post-processing. Our implementation involves three key steps: (1) Applying Relative Central Regression to the final semantic occupancy output for voxel-level center prediction (2) Clustering potentially scattered center predictions (Due to error in the center voting predicted by the model, multiple discrete center points often appear) to form coherent instances (3) The clustered central voxels then determine instance ID assignments through voting relationships. Since Oce3D-nuSeenes lacks instance-level occupancy ground truth, project our results onto LiDAR points for panoptic segmentation evaluation as reference metrics.
Table 9 shows comparison of instance-level occupancy prediction for multiple methods. PanoOce (Wang et al., 2024) remains the first purely camera- cunera-based approach for point cloud panoptic segmentation, achieving LiDAR-comparral performance through joint semantic occupancy prediction and 3D detection with bounding box supervision. However, our CSV-Occ differs fundamentally by excluding 3D bounding box size supervision (crucial for explicit instance boundary prediction).
19


Reviews:
[{"review_id": "nfKG2QcYWM", "reviewer": "Reviewer_a93Q", "summary": "This paper presents CSV-Occ, a method for camera-based 3D semantic occupancy prediction, aimed at improving scene understanding. It considers two key issues: reducing the high computational complexity of temporal information fusion and addressing the semantic ambiguity in predicting object centers. To overcome these challenges, CSV-Occ extends the state space model to support multi-input sequence interactions and explicitly predicts the instance to which each voxel belongs, refining feature representation from coarse to fine. Experiments on the Occ3D-nuScenes and nuScenes datasets demonstrate that CSV-Occ performs better than some of the existing methods in both 3D semantic occupancy prediction and lidar point cloud semantic segmentation.\n\n## update after rebuttal\nRead through the authors response and I'll keep my current rating.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the valuable time and effort you've dedicated during the review process. In light of the constructive feedback and suggestions you provided, we've meticulously examined our paper and work. **We've also summarized your comments and are replying to you one by one in a Question and Answer format as follows.** If you have any further questions or concerns, we're more than willing to cooperate with you to address them.\n\n---\n\n### **Q1**: Was there any consideration of integrating additional data sources, such as lidar or radar, to enhance the accuracy of scene understanding?\n\n**A1**: We haven't integrated multiple sensors (such as LiDAR or radar) into CSV-Occ yet. In the future, we'll seriously consider your suggestion of incorporating multi-sensor multi-modal inputs into our method and further applying it to more scenarios (like indoor scene datasets or more vision tasks) to evaluate its generality.\n\n### **Q2**: Could you elaborate on the reasons for the lower performance on certain background classes, such as trailers and vegetation?\n\n**A2**: Through statistics, we've found that the low performance of our method in certain background categories is primarily attributed to the VEM module. As depicted in the following figure, upon enabling the voting mechanism, the performance of all foreground categories has increased. Conversely, a decline in performance is observed in four background categories: driveable surface, other flat, man-made, and vegetation. (If the following images don't work, try this link: https://github.com/ZeaZoM/re/blob/main/figs/VEM%20cat.png)\n\n![](https://github.com/ZeaZoM/re/blob/main/figs/VEM%20cat.png)\n\nThis occurs because VEM relies on the coarse semantic occupancy prediction to divide foreground category voxels. When the coarse semantic occupancy prediction is inaccurate, VEM may predict relative centers for some background voxels as well. The occupancy of background categories usually has a large volume and lacks distinct instance centers. This leads to training confusion in VEM. During inference, it wrongly updates the features of background voxels, causing the occupancy classification head to give incorrect category predictions. As a result, the number of background voxels decreases, which directly impacts the prediction performance of background categories.\n\nWe also calculated the proportion of the number of foreground and background voxels predicted by the model with and without VEM. As shown in the figure below, it can be seen that VEM can significantly increase the number of foreground voxels. This partly indicates that VEM can improve the Internal Occupancy Vacancy (IOV) situation and successfully predict free voxels as foreground voxels. However, we unexpectedly found that the number of background voxels decreased, which we think is related to the training confusion caused by VEM. (If the following images don't work, please try this link: https://github.com/ZeaZoM/re/blob/main/figs/VEM%20ratio.png)\n\n![](https://github.com/ZeaZoM/re/blob/main/figs/VEM%20ratio.png)\n\n### **Q3**: The section that qualitatively evaluates the proposed approach against other SOTA methods is very helpful. It would be beneficial to move this content to the main paper, as qualitative analysis provides more insight into the impact of the approach than numbers alone. \n\n**A3**: Since the page limit for the main text when submitting the article is no more than 8 pages, we previously had to move some content to the appendix. However, in line with your suggestion, we will consider moving the qualitative analysis content from the appendix to the main text within the scope permitted by the article layout.", "claims_and_evidence": "The claims are generally well-supported by thorough experimentation, with near-perfect results. However, for some classes, the results still do not reach state-of-the-art (SOTA) levels, and the lack of explanation for these cases is a noticeable gap. Nevertheless, the claims are largely substantiated through the use of ablation studies, which provide strong support for the findings.", "methods_and_evaluation_criteria": "The evaluation criteria are well-founded, and the authors compare their proposed method with several other state-of-the-art (SOTA) approaches. To ensure a fair comparison, they even modified the FB-OCC code. The metric used for evaluation is mean Intersection over Union (mIoU), which is also reported for each class label.", "theoretical_claims": "The theoretical claims presented in the main paper appear promising. However, a significant portion of the theoretical details is covered in the supplementary section, such as the \"Derivations of the Cross State Space Module\" and evaluation metric formulas. As a result, the authors rely on the supplementary material to fully convey the theoretical claims.", "experimental_designs_or_analyses": "The experimental designs are well-structured, with a thorough comparison to multiple approaches. However, the proposed method does not perform as well on certain classes, such as trailers, vegetation, and manmade objects. While this is understandable given the focus on foreground classes, it would be beneficial if the authors provided some discussion on why the performance on background classes is slightly worse compared to state-of-the-art (SOTA) methods.", "relation_to_broader_scientific_literature": "The proposed approach is highly relevant to the field of scene understanding, particularly in the context of autonomous driving. By improving 3D semantic occupancy prediction, it provides valuable insights into how vehicles can better perceive and interpret their surroundings. Furthermore, the approach contributes to the broader literature by offering a more efficient and effective way to handle complex environmental scenarios, which has potential applications beyond just autonomous driving, such as robotics and urban planning.", "essential_references_not_discussed": "I am not aware of any essential references that were not covered in the paper.", "other_strengths_and_weaknesses": "As mentioned above, providing an explanation for the lack of performance on certain classes would add more value to the paper. Additionally, a qualitative analysis comparing the proposed approach with SOTA methods, highlighting where it excels and where it falls short, would also be helpful for better understanding its strengths and limitations.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. Could you elaborate on the reasons for the lower performance on certain background classes, such as trailers and vegetation?\n\n2. Was there any consideration of integrating additional data sources, such as lidar or radar, to enhance the accuracy of scene understanding?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CtaZm5e5vF", "reviewer": "Reviewer_9BHX", "summary": "The paper proposes CSV-Occ, a method for image-based 3D semantic occupancy prediction in autonomous driving. It introduces two key ideas: 1. Temporal fusion applied on voxel query results instead of BEV features, which is considered new. 2. A center voting mechanism to improve occupancy prediction inside object boundaries. Despite these contributions, the paper lacks an efficiency analysis to justify the use of state-space modeling (SSM) for computational benefits. Additionally, while the voting feature is fused into the network, no panoptic results are provided.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in the review process. **In response to your constructive feedback, we have thoroughly reviewed our paper and categorized your comments. Attached is Q&A response.** If you have any further questions, we will address them promptly.\n\n---\n\n### **Q1**: The projection of vision-only 3D semantic occupancy predictions onto point clouds for evaluation against LiDAR semantic segmentation is questionable.\n\n**A1**: When the volume of a single voxel shrinks to near zero, the semantic occupancy prediction task can approximate the point cloud semantic segmentation task. Semantic occupancy is a voxelized 3D point representation. Our evaluation method, using LiDAR semantic segmentation as a quantitative indicator for 3D semantic occupancy prediction, follows these relevant works: PanoOcc (CVPR'24), TPVFormer (CVPR'23), OccFormer (ICCV'23), Scene as Occupancy (ICCV'23)\n\n### **Q2**: Since center voting is already implemented, why not directly generate instance-level occupancy predictions?\n\n**A2**: As you said, we can generate instance-level results, and CSV-Occ requires additional post-processing. Our implementation involves three key steps:\n1. Applying Relative Central Regression to the final semantic occupancy output for voxel-level center prediction\n2. Clustering potentially scattered center predictions (Due to the error in the center voting predicted by the model, multiple discrete center points often appear) to form coherent instances\n3. The clustered central voxels then determine instance ID assignments through voting relationships. Since Occ3D-nuScenes lacks instance-level occupancy ground truth, we project our results onto LiDAR points for panoptic segmentation evaluation as reference metrics.\n\n|Method |Modality |PQ |SQ |RQ |\n|---|---|---|---|---|\n|PanopticTrackNet (arXiv 2020) |L |51.4 |80.2 |63.3 |\n|EfficientLPS (I-TR 2021) |L |62.0 |83.4 |73.9 |\n|LidarMulitiNet (AAAI 2023) |L |81.8 |89.7 |90.8 |\n|PanoOcc (CVPR 2024) |C |62.1 |82.1 |75.1 |\n|CSV-Occ-Instance (Ours) |C |48.3 |79.6 |60.5 |\n\nPanoOcc remains the first purely camera-based approach for point cloud panoptic segmentation, achieving LiDAR-comparable performance through joint semantic occupancy prediction and 3D detection with bounding box supervision. However, our CSV-Occ differs fundamentally by excluding 3D bounding box size supervision (crucial for explicit instance boundary prediction). This leads to center-voting derived instance results underperforming 2020 LiDAR-based PanopticTrackNet in evaluations. If you think it is necessary, we can consider adding the above results to the appendix of the article for analysis.\n\n### **Q3**: The claim of VEM is only supported by Figure 7, but lacks statistical analysis to reveal the mechanism.\n\n**A3**: Regarding the support for VEM, apart from Figure 7, we also did a quantitative ablation test in Table 4. Yet, as you suggested, a more detailed category-specific statistical analysis of VEM is needed. We found that the VEM module can cause a decrease in the performance of certain background categories. **For detailed charts and analysis (constrained by 5k-character rebuttal limits), please see Section A2 (Reviewer a93Q). Hope it eases your concerns.**\n\n### **Q4**: No direct evaluation or quantitative analysis is provided to compare efficiency with self-attention-based methods.\n\n**A4**: Per your feedback, we conducted efficiency analyses on Cross SSM, evaluating voxel query size and frame count impacts. Results show that Cross SSM has the optimal trainable parameters, inference speed, and memory. **For detail, please see Section A1 (Reviewer BeaF).**\n\n### **Q5**: CVT-Occ (ECCV'24), which is highly relevant and should be discussed.\n\n**A5**: We will cite it. **For our specific discussion on CVT-Occ, please refer to A2 (Reviewer QyW7).**\n\n### **Q6**: No SemanticKITTI experiments.\n\n**A6**: Since nuScenes has a larger sample size and each sample consists of six camera images forming a 360 degree surround view field of view. We evaluated semantic occupancy on the Occ3D-nuScenes dataset and point cloud semantic segmentation on the nuScenes dataset. In recent related works, there are also many that only evaluate on the nuScenes dataset, such as OPUS (NeurIPS'24), COTR (CVPR'24), Fully Sparse (ECCV'24), FB-OCC (ICCV'23).\n\nTherefore, we believe two experiments suffice to comprehensively demonstrate the effectiveness of our method. **We will still seriously consider incorporating SemanticKITTI in our future work.**\n\n### **Q7**: Time-based view transformation-terminology clarification is needed.\n\n**A7**: To resolve ambiguity in Section 3.3’s original title \"Time-based View Transformation\":\n1. Retitle to \"Time-based Feature Fusion\" (TFF)\n2. Add Subsection 3.3.1 \"View Transformation\" to clarify content scope\n3. Renumber subsequent subsections (3.3.1→3.3.2) and update Figure 3’s \"TVT\" labels to \"TFF\"\n4. Systematically replace all \"TVT\" abbreviations with \"TFF\" throughout the paper", "claims_and_evidence": "* No direct evaluation or quantitative analysis is provided to compare efficiency with self-attention-based methods. While reducing complexity from quadratic to linear is theoretically appealing, empirical proof (e.g., runtime comparison, FLOPs, inference speed) is missing.\n* The claim is only supported by Figure 7, but lacks statistical analylsis to reveal the mechanism.", "methods_and_evaluation_criteria": "* The paper introduces center voting but does not provide panoptic or instance understanding results, which would be the natural evaluation metric. Evaluating thing-stuff separation and instance-wise completeness would better support the effectiveness of the voting feature.\n* Temporal fusion can be applied beyond occupancy prediction, e.g., mapping, object detection and end-to-end planning. A broader evaluation would show whether CSV is specific to occupancy prediction or applicable to other self-driving tasks.", "theoretical_claims": "* The mathematical formulation for SSM-based temporal fusion is included.", "experimental_designs_or_analyses": "+ Ablation studies are provided at an architectural level, following standard experimental protocols. The model is evaluated on the Occ3D-nuScenes dataset, which is an appropriate benchmark. \n- The projection of vision-only 3D semantic occupancy predictions onto point clouds for evaluation against LiDAR semantic segmentation is questionable.", "relation_to_broader_scientific_literature": "* The paper could have cited more occupancy prediction literature, including indoor scene methods, which share methodological similarities.", "essential_references_not_discussed": "\"CVT-Occ: Cost Volume Temporal Fusion for 3D Occupancy Prediction\" (ECCV 2024), which is highly relevant and should be discussed.", "other_strengths_and_weaknesses": "- No SemanticKITTI experiments, which would help assess generalization beyond nuScenes.\n- Figure 2 caption might be incorrect?", "other_comments_or_suggestions": "- Time-based view transformation is essentially just view transformation—terminology clarification is needed.", "questions_for_authors": "* Since center voting is already implemented, why not directly generate instance-level occupancy predictions?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "utiC0aTkfj", "reviewer": "Reviewer_QyW7", "summary": "The paper introduces CSV-Occ, a method for camera-based 3D semantic occupancy prediction.\nCSV-Occ focuses on two key challenges.\n\nFirstly, the prior methods have usually exploited attention mechanisms for temporal modeling that have high computational complexity. This paper propose the Cross State Space Module (Cross SSM), a variant of Mamba architecture, to handle multi-sequence inputs with linear complexity.\n\nIn addition, the existing methods struggle with Internal Occupancy Vacancy (IOV), where the centers of large foreground objects are often predicted as empty. To address this, the authors propose the Voting-based Enhancement Mechanism (VEM), which refines semantic occupancy predictions by inferring the instance centers of objects.\n\nSCV-Occ achieves SoTA performance on Occ3D-nuScenes dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely express our profound gratitude for the precious time and dedicated effort you have invested in the review process. Taking into account the highly constructive feedback and suggestions you proffered, we have meticulously re-examined our paper and work. **We have comprehensively collated your comments and will respond to each of them individually in a Question-and-Answer format as detailed below.** Should you have any further questions or concerns, we are wholeheartedly committed to collaborating with you to resolve them. \n\n---\n\n### **Q1**: Have you explored using indoor point cloud data in conjunction with your method? The targeted problems are also important issues in that field.\n\n**A1**: CSV-Occ focuses on the sub-task of pure image-based semantic occupancy prediction in outdoor autonomous driving. Thus, we haven't explored applying it to indoor scene datasets or integrating indoor point cloud data for validation. However, in future work, we will strongly consider your suggestion to transfer our method to indoor scene datasets and extend it to multi-modal inputs that combine images and point clouds, in order to evaluate the generality of CSV-Occ.\n\n### **Q2**: It would be much better if the authors refer more recent works, such as OccFiner or TALoS.\n\n**A2**: We fully agree. \n1. OccFiner focuses on achieving a data closed loop and automatic annotation for pure vision SSC. In the first stage, it compensates for the prediction errors of the in-vehicle model through a multi-to-multi local propagation network, and fuses the relative spatial coordinates and semantic features. In the second stage, it conducts global propagation of the regional centers, converts the refined voxel labels into semantic point clouds, adjusts the coordinates, and conducts voxel voting. **We will cite it in our article: 'Shi H, Wang S, Zhang J, et al. Offboard Occupancy Refinement with Hybrid Propagation for Autonomous Driving[J]. arXiv preprint arXiv:2403.08504, 2024.'**\n\n2. TALoS mines the information in the driving environment, uses the point cloud observations at different moments as supervision. Through coordinate transformation, it obtains binary self-supervision for geometric completion based on the characteristics of LiDAR's line of sight, and constructs a loss function to guide the model training. **We will also cite it in our article: 'Jang H K, Kim J, Kweon H, et al. TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight[J]. arXiv preprint arXiv:2410.15674, 2024.'**\n\n3. The key insight of CVT-Occ is that as the camera moves with the vehicle, parallax is generated. By leveraging this multi-frame parallax information, the depth information that cannot be directly obtained from a single pixel can be compensated for, thus eliminating the uncertainty that occurs when converting image features into the 3D space. Methodologically, CVT-Occ projects rays from the center of the current-frame BEV feature volume towards the image pixels. Then, it samples multiple virtual points and their corresponding voxel features at a certain step along the rays. These virtual points are transformed into the historical BEV feature volume through coordinate system transformation to sample the corresponding voxel features. All the sampled voxel features are combined to form Cost Volume Features. After convolution and Sigmoid calculation, attention weights are generated, which are multiplied voxel-by-voxel with the current-frame BEV feature volume to update the features. **We will discuss and cite it in our paper: 'Ye Z, Jiang T, Xu C, et al. Cvt-occ: Cost volume temporal fusion for 3d occupancy prediction[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2024: 381-397.'**", "claims_and_evidence": "In my opinion, the claims in the paper are generally well-supported.\nExperimental results on the Occ3D-nuScenes dataset confirm that CSV-Occ outperforms existing methods on 3D semantic occupancy prediction and LiDAR semantic segmentation.", "methods_and_evaluation_criteria": "The proposed methods are sounding for the defined problems. \nThe Cross SSM is a nice solution for efficient temporal feature fusion, which is crucial for occupancy prediction in autonomous driving. \nThe VEM is slightly less novel in terms of academia, but still a meaningful technical solution for improving occupancy predictions, particularly for large objects. \nAnd also the evaluation criterial seems appropriate.\nThe authors evaluate CSV-Occ against existing baselines using the Occ3D-nuScenes dataset and achieve good results.", "theoretical_claims": "It seems that there is no special theoretical claims that needs validation.\nThe proposed method is verified with experimental results.", "experimental_designs_or_analyses": "The experimental designs are solid. The paper uses well-established datasets for evaluation, and the ablation studies are extensive enough to explore the impact of different components like the Cross SSM and VEM. \nBut more extensive validation might be helpful (more dataset including indoor cases).", "relation_to_broader_scientific_literature": "Literature review of this paper is pretty good, particularly in the area of 3D semantic occupancy prediction and temporal feature fusion.", "essential_references_not_discussed": "It would be much better if the authors refer more recent works, such as OccFiner [1] or TALoS [2].\n\n[1] OccFiner: Offboard Occupancy Refinement with Hybrid Propagation for Autonomous Driving\n\n[2] TALoS: Enhancing Semantic Scene Completion via Test-time Adaptation on the Line of Sight", "other_strengths_and_weaknesses": "The approach combines cutting-edge methods (Mamba, especially) for temporal feature fusion and semantic occupancy prediction.\nThe empirical results demonstrate clear improvements over previous methods.\n\nFrom my perspective, the academic novelty might be somewhat lacking, but the contribution to the field is substantial.", "other_comments_or_suggestions": "Please see the reviews above.", "questions_for_authors": "Have you explored using indoor point cloud data in conjunction with your method?\nThe targeted problems are also important issues in that field.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jTQo4oMUaO", "reviewer": "Reviewer_BeaF", "summary": "This paper focuses on image-based 3D semantic occupancy prediction. To address the challenges posed by the computational complexity of temporal methods and the semantic ambiguity leading to vacancy issues, the Cross-State Space Module (Cross SSM) and a Voting-based Enhancement Mechanism are proposed as targeted solutions. The approach achieves state-of-the-art performance on the OCC3D-nuScenes dataset.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Deeply grateful for your dedication and expertise throughout the review.\nBuilding on your insightful suggestions, we have systematically:\n\n1. **Revised the paper,**\n2. **Organized your comments by theme, and**\n3. **Provided detailed responses in an annotated Q&A below.**\n\nWe welcome any additional feedback to further refine this work.\n\n---\n\n### **Q1**: Temporal computation complexity – the paper lacks efficiency-related experiments to substantiate this claim it does not reflect the efficiency advantages in long-term sequences.\n\n**A1**: As shown in the following table, we have supplemented the efficiency experiments, including the number of trainable parameters, inference latency, and inference memory. The following efficiency experiments were all measured on a platform equipped with an Intel Xeon Gold 5318Y CPU and an NVIDIA A40 48G GPU, with the batch size set to 1.\n\n|Model |Fusion Module |mIoU (Occ.) |mIoU (Seg.) |Trainable Params (M) |Inference Latency (ms) |Inference Memory (MB) |\n|---|---|---|---|---|---|---|\n|CSV-Occ |Cross SSM (Ours) |**44.93** |**73.42** |**68.5** |**102** |**5588** |\n| |TSA |43.17 |72.62 |69.3 |119 |5712 |\n| |MLP-Mixer |43.22 |72.67 |77.5 |116 |6224 |\n\nWe explored how initialized voxel query size impacts model efficiency (If the following images don't work, please try this link: https://github.com/ZeaZoM/re/blob/main/figs/efficiency%20experiment%20BEV.png). Keeping its height at 4, we adjusted the BEV side length from 25 to 200 (max 200×200, not exceeding occupancy ground truth). The left figure shows samples inferred per second; the right shows inference memory consumption. As voxel query size grows, especially past 100, TSA and MLP mixer's inference speed and memory consumption worsen. Cross SSM's efficiency decline is more stable due to its linear computational complexity. A larger query size means a longer flattened token sequence, and Cross SSM needs only one scan for multi frame fusion. \n\n![](https://github.com/ZeaZoM/re/blob/main/figs/efficiency%20experiment%20BEV.png)\n\nThe figure below shows how the number of inference frames affects model efficiency (If the following images don't work, please try this link: https://github.com/ZeaZoM/re/blob/main/figs/efficiency%20experiment%20Frames.png). The initialized voxel query size is set at 100×100×4. Cross SSM outperforms in both inference speed and memory consumption. Still, as the number of inference frames rises, Cross SSM's efficiency change trend is like that of TSA and MLP mixer. In CSV - Occ, the number of inference frames equals the number of multi - frame fusion module calls, and the feature sequence length per call depends only on the voxel query size. So, more inference frames don't give Cross SSM a trend advantage.\n\n![](https://github.com/ZeaZoM/re/blob/main/figs/efficiency%20experiment%20Frames.png)\n\n### **Q2**: The novelty of the method seems somewhat weak. What are the technical challenges of applying Mamba's SSM to the 3D occupancy prediction task?\n\n**A2**: Our core challenge lies in extending Mamba's SSM to process dual-sequence inputs - a novel direction beyond existing vision-focused Mamba variants that primarily optimize feature map scanning for single-sequence SSM receptive fields. While standard \"self SSM\" (analogous to self-attention) updates features through intra-sequence correlations, we pioneer \"cross SSM\" to enable inter-sequence interaction akin to cross-attention.\n\nThe validation complexity stems from SSM's six parametric elements (x, A, B, C, D, Δ). Unlike attention where k/v derive from one sequence and q from another, SSM's A/D originate from learnable embeddings while B/C/Δ are mapped from the input sequence. Our key hurdle was determining which sequences should supply x/B/C/Δ for cross-SSM implementation, as the original SSM formulation provides no guidance. Through systematic permutation experiments, we empirically identified optimal configurations.\n\n|# |x |C |B |Δ |mIoU (Occ.) |mIoU (Seg.) |\n|---|---|---|---|---|---|---|\n|A |$V_{T-i}$ |$V_{T}^{'}$ |$V_{T-i}$ |$V_{T}^{'}$ |39.35 |67.19 |\n|B | | |$V_{T}^{'}$ |$V_{T-i}$ |38.60 |67.01 |\n|C | | |$V_{T-i}$ |$V_{T-i}$ |**44.93** |**73.42** |\n|D |$V_{T}^{'}$ |$V_{T-i}$ |$V_{T-i}$ |$V_{T}^{'}$ |33.25 |61.07 |\n|E | | |$V_{T}^{'}$ |$V_{T-i}$ |35.84 |64.50 |\n|F | | |$V_{T}^{'}$ |$V_{T}^{'}$ |32.53 |59.28 |\n\nEmpirical analysis revealed Approach #C's superior metrics prompted theoretical investigation into SSM-attention parallels. Through mathematical derivation (Appendix B), we demonstrated SSM constitutes a specialized attention subset, cross-attention's Q→C substitution implies cross-SSM implementation requires mapping C from secondary sequences - a pivotal insight enabling cross-sequence feature interaction.\n\nMethodologically, we preserved original SSM's feature-map scanning architecture (patch partition via pooling/flattening; merger via reshape/interpolation - cf. main text line 199) to isolate cross-SSM effects.", "claims_and_evidence": "This paper identifies two challenges in 3D semantic occupancy prediction:\n\n1. **Temporal computation complexity** – While this issue does exist, the paper lacks efficiency-related experiments to substantiate this claim.\n2. **Missing foreground object instance centers** – The authors demonstrate the effectiveness of their method through visualizations. However, this issue seems to stem from the fact that the ground truth data itself is based on LiDAR sequences, leading to hollow centers. Consequently, the proposed voting mechanism appears more like a post-processing optimization.", "methods_and_evaluation_criteria": "This paper proposes the Cross-State Space Module, and Table 3 demonstrates the performance advantages of the proposed fusion method. However, it does not reflect the efficiency advantages in long-term sequences.\n\nThe proposed Voting-based Enhancement Mechanism is also validated in Table 4.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "The experiments are conducted on occ3D-nuScenes and nuScenes, demonstrating the performance advantages of the proposed method. The ablation studies further highlight the effectiveness of the proposed modules.", "relation_to_broader_scientific_literature": "Occupancy plays a crucial role in the detection of general objects. Many previous methods, such as PanoOcc and FBOcc, have explored temporal approaches. Building on these works, this paper further investigates improvements in both performance and efficiency.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "1. Lacks experimental comparisons on with other methods in efficiency perspective.", "other_comments_or_suggestions": "No", "questions_for_authors": "1. The novelty of the method seems somewhat weak. What are the technical challenges of applying Mamba's SSM to the 3D occupancy prediction task?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper initially received ratings of (4, 2, 3, 2), which were somewhat varied. After the rebuttal, the ratings increased to (4, 3, 3, 3).

Although the final ratings are positive, reviewers raised concerns regarding the statistical analysis and efficiency evaluations, which do not adequately support the claimed advantages. The lack of SemanticKITTI experiments also limits the generalizability claims of the method. Reviewers also noted that the overall contribution of the paper seems somewhat modest, leaving them uncertain about its suitability for inclusion at ICML. Further revisions are suggested to clarify terminology related to "Time-based Feature Fusion" and "Time-based View Transformation," and to evaluate the method's effectiveness under higher-resolution settings, such as on Occ-Waymo.

Despite the concerns, the area chair found no clear justification to overrule the reviewers' recommendations. Consequently, the area chair suggests a "weak accept" if there is room in the program
Decision: Accept (poster)