OccProphet: Pushing the Efficiency Frontier of Camera-Only 4D Occupancy Forecasting with an Observer-Forecaster-Refiner Framework
Authors: Junliang Chen
Huaiyuan Xu
Yi Wang
Lap-Pui Chau
Venue: ICLR 2025 Poster (2025)
Keywords: camera-only occupancy forecasting
efficiency
effectiveness
autonomous driving
Primary area: applications to robotics, autonomy, planning
TL;DR: This paper proposes OccProphet, a camera-only framework for efficient and effective occupancy forecasting, in a lightweight Observer-Forecaster-Refiner pipeline, performing better and 2.6 times faster than Cam4DOcc reducing computational costs.

Abstract: Predicting variations in complex traffic environments is crucial for the safety of autonomous driving. Recent advancements in occupancy forecasting have enabled forecasting future 3D occupied status in driving environments by observing historical 2D images. However, high computational demands make occupancy forecasting less efficient during training and inference stages, hindering its feasibility for deployment on edge agents. In this paper, we propose a novel framework, \textit{i.e.}, OccProphet, to efficiently and effectively learn occupancy forecasting with significantly lower computational requirements while improving forecasting accuracy. OccProphet comprises three lightweight components: Observer, Forecaster, and Refiner. The Observer extracts spatio-temporal features from 3D multi-frame voxels using the proposed Efficient 4D Aggregation with Tripling-Attention Fusion, while the Forecaster and Refiner conditionally predict and refine future occupancy inferences. Experimental results on nuScenes, Lyft-Level5, and nuScenes-Occupancy datasets demonstrate that OccProphet is both training- and inference-friendly. OccProphet reduces 58\%$\sim$78\% of the computational cost with a 2.6$\times$ speedup compared with the state-of-the-art Cam4DOcc. Moreover, it achieves 4\%$\sim$18\% relatively higher forecasting accuracy. Code and models are publicly available at https://github.com/JLChen-C/OccProphet.

Published as a conference paper at ICLR 2025
OCCPROPHET: PUSHING EFFICIENCY FRONTIER OF CAMERA-ONLY 4D OCCUPANCY FORECASTING WITH
FRAMEWORK
Junliang Chen', Huaiyuan Xu", Yi Wang', Lap-Pui Chau
The Hong Kong Polytechnic University
Jun-liang liang. chen @connect polyu.hk (hua yuan.xu, xu, yi-eie. gi-eie.wang, lap-pui pee-puu.chauueeeee (ahaau@@poluu
ABSTRACT
Predicting variations in complex traffic environments is crucial for the safety autonomous driving. Recent advancements in occupancy forecasting have abled forecasting future 3D occupied status in driving environments by observing historical 2D images. However, high computational demands make occupancy forecasting less efficient during training and inference stages, hindering its feasi- bility for deployment edge agents. this paper, we propose novel frame- work, i.e., OccProphet, to efficiently and effectively learn occupancy forecasting with significantly lower computational requirements while improving forecast- ing accuracy. OccProphet comprises three lightweight components: Observer, Forecaster, and Refiner. The Observer extracts spatio-temporal features from 3D multi-frame voxels using the proposed Efficient 4D Aggregation with Tripling- Attention Fusion, while the Forecaster and Refiner conditionally predict and re- fine future occupancy inferences. Experimental results nuScenes, Lyft-Level5, and nuScenes- -cceupany datasets demonstrate that OccProphet is both training- and inference-friente with 2.6x speedup compared with the state-of-the-ar Cam4DOcc. Moreover, it achieves 4%~ 18% relatively higher forecasting accuracy. Code and models are publicly available at https:: http//itttt /ithhb..... com/ Jccheeecoo/occrrpppeett
OccProphet reduces 58%~78% of the computational cost
INTRODUCTION
29:2 (%)
Oct Prophot
29.0
OccProphet
2-3 2%% Bottor or
28.0
Faster Better
Molion Trend
28.6
28.4
28.2
28.0
5.0
Observed Video
Future Occupancy
Figure 2: Comparison of perfor- Figure 1: Illustationn of OccProphet. OccProphet only receives mance between Cam4DOcc and OccProphet.
multi-camera video input and produces futuree occupancies.
Autonomous driving holds significant promise for reshaping transportation and urban mobility. Per- ceiving 3D surroundings is critical for autonomous vehicles. There are typically two 3D percep- tion paradigms: detection-based perception and occupancy-based perception. The detection- basedd paradigm, such as monocular or multi-view 3D object detection (Chen et al.) [2066) Wang et al.) 2021 Huang al. 2021 Wang et al. 2022). equips autonomous vehicles with 3D perception ca- pabilities by detecting traffic participants and assigning them 3D bounding boxes. However, due to
Equal contribution
Corresponding author

---

Published as a conference paper at ICLR 2025
limitations in pre-defined object categories and rigid detection boxes, the detection-basee paradigm struggles to generalize to unknown objects and irregular structures, which are likely to appear in real-world traffic scenarios. To relieve these constraints, the occupancy- perception paradigm
(Huang et al. 2023a Wei et al. 2023b; Wang et al.) 2023b Tian et al., 2023: Xu et al., (025) offers more general and hne-ggaiine representation of the environment through learning occupied states in 3D space. This paradigm provides a stronger perception ability to autonomous vehicles, enabling them to better comprehend complex traffic environments.
Despite advancements in occupancy perception, most existing methods can only perceive the past and present states of the environment. They lack the ability capture and understand environmental dynamics and subsequently forecast the future scene. Forecasting is essential for safe planning, which assists autonomous vehicles to avoid potential collisions (Ding et al., 2019 [ong et al. 2020 Ding et al., 2021 Huang et al., 2023b; Pan et al.] 2024). Although some bird's s-eye view (BEV) approaches have achieved object motion forecasting the environment(Hu et al. 2021 Zhang al.)
2022 Fang et al. 2023 Ferenczi et al.. 2024), their forecasts are restricted
2D plane. This 2D
forecasting limits the comprehensive understanding of the entire 3D dynamic scene. In light this
limitation, occupancy forecasting methods (Weng et al. 2021: 2022; Mersch et al., 2022. Khurana ct al., [023] Agro et al., (2024) shift to predict future 3D occupancy for the whole environment. However, these methods rely on point cloud inputs from expensive LiDAR kits. To explore more cost-effective solutions, Cam4DOcc (Ma et al. 2024a) introduces a camera-only benchmark and baseline for occupancy forecasting.
While Cam4DOcc achieves remarkable performance in occupancy forecasting compared to its coun- terparts, the high computational cost makes it less efficient during training and inference. It ham- pers feasibility of deployment on edge agents, such as autonomous vehicles that operate under re- stricted computational budgets. In this paper, we propose a novel framework, dubbed as OceProphet (shown in Figure D]. to efficiently and effectively perform camera-oni onlyy occupancy forecasting. In OceProphet, we design three lightweight components to forecast future states: the Observer, Fore- caster, and Refiner. The Observer adopts 4D) feature aggregation and tripling-attention fusion strategy on the scduced-reoolutin Forecaster then infers future states according the scene condition and the Observer's outputs. Fi- nally, the Refiner enhances the quality of the forecast results through spatio-temporal interactions. The main contributions of this paper are summarized as follows:
features to extract spatio-tempore information efficiently. The
We propose OccProphet, novel camera-only occupancy forecasting framework, which is both efficient and effective during training and inference, towards on vehicle deployment.
We design a lightweight tracts spatio-temporal features from historical observations; the Forecaster conditionally predicts coarse future states; the Refiner promotes forecasting accuracy.
pipeline for OccProphet. The Observer ex-
Experimental results demonstrate that OccProphet achieves higher forecasting accuracy with less than half the computational cost of Cam4DOce. These improvements are consistently observed across the nuScenes (Caesar al. 2020). Lyft-Leve15 (Houston et al. [02T], and nuScenes- Occupancy (Wang et al.) 2023a) datasets, highlighting the superior efficiency and effectiveness of OccProphet (shown in Figure
2 RELATED WORK
2.1 OCCUPANCY PREDICTION
Occupancy prediction aims at modeling the current 3D occupancy layout in space, by observing his- torical and current environments. Occupancy prediction is adept at providing 3D dense descriptions for complex traffic scenarios, thereby gamering increasing attention from academia and industry. SSCNet (Song et al. (017)) was the first semantic occupancy prediction work, which simultaneously predicted occupied voxels and their semantics for an indoor scene using a depth image. MonoScene (Cao & De Charette 2022) extends SSCNet to outdoor scenarios by using an RGB image and in- corporating stronger supervisions. Training and evaluating occupancy prediction networks require benchmarks with ground truth occupancy labels, which are challenging due to the complexity of densely annotating 3D outdoor driving scenes. Wang al. (Wang et al. 2023b) propose OpenOc- cupancy, the first large-scale benchmark for semantic occupancy prediction, which covers multiple sensing modalities and provides high-resolution dense occupancy annotations. Occ3d, developed by
2

---

Published as a conference paper at ICLR 2025
Tian al. (Tian et al. 2023), is another widely used benchmark for occupancy prediction, whose hugh-quality labels are from the proposed technique of image-guin gegeeeggieedddde occupancy label refinement.
Considering the inherently dense nature of depicting 3D environments, ideal occupancy percep- tion emphasizes balancing efficiency and accuracy. Although some studies explore sparse queries (Li et al. 2023b) or tri-perspective view (TPV) representation (Huang et al. (023a) to elevate the efficiency of occupancy prediction, they inevitably sacrifice fine-grained details of 3D space. In contrast, many other methods (Zhang et al. 2023) Wei al.] 2023b; Ma et al., (024b) utilize 3D feature volumes to preserve 3D details of the scene, leading to higher occupancy prediction accu- racy, Recently, COTR (Ma et al., 2024b) proposes compact occupancy representation that pre- serves geometric details while reducing computational costs. Despite the significant advancements in occupancy prediction, including comprehensive benchmarks and powerful algorithms, all exist- ing methods focus exclusively on current occupancy and overlook future occupancy, which reflects potential variations in the 3D environment.
2.2 OCCUPANCY FORECASTING
Occupancy forecasting targets to predict future occupancy, starting from the current timestamp. Previous dominant works mainly adopt the BEV perspective for occupancy forecasting, reasoning about 2D occupancy changes on a BEV plane. For example, FIERY (Hu et al. 202I) extracts BEV features from multi-view image inputs and utilizes temporal model with 3D convolution to capture spatio-temporal states, which are then used to recursively forecast future instances states. BEVerse (Zhang et al.. 2022) introduces unified BEV representation framework that jointly achieves object perception and occupancy forecasting using multi-task supervision. To better align spatio-temporal information, TBP-Former (Fang et al. 2023) designs a pose-synchronized BEV encoder syn- chronize multi-frame BEV features during occupancy forecasting. While these BEV-based methods deliver impressive performance for forecasting pre-defined semantic categories (e.g., vehicles), they struggle to (I) forecast the motion of out-ofddistriuuton objects and (2) capture height information in the environment. In contrast, our method forecasts aansosa occupancy from a 3D perspec- tive, rather than BEV, thus enabling autonomous vehicles to monitor, comprehend, and reason about 3D dynamics the physical world.
To address the limitations of BEV occupancy forecasting, researchers have recently shifted their focus toward forecasting 3D occupancy without considering semanties. Specifically, Khurana et proxy task for the occupancy forecasting task, where point cloud rendering is used to bridge the two tasks. UnO (Agro et al. 2024) takes LiDAR point clouds input and performs occupancy forecasting using the proposed unsupervised learning paradigm, in which the forecasted occupancy should align with pseudo occu- pancy labels generated from future LiDAR data. However, these methods rely on point clouds from expensive LiDAR kits, leading to increased costs when implemented in autonomous vehicles.
al. (Khurana et al., 2023) treat LiDAR point cloud forecasting
Compared to LiDAR-based approaches, camera-only occupancy forecasting offers promising al- ternative with significantly lower costs. Cam4DOcc (Ma et al. (024a) introduces a comprehensive benchmark and dataset to evaluate camera-only occupancy forecasting algorithms on both movable and static objects beyond pre-defined categories. Additionally, it proposes a strong camera-only baseline for occupancy forecasting. However, this approach is still far from real-world application due to its high computational demands. Drive-OceWordd (Yang et al., 2024) introduces extra ac- tion condition inputs and planning supervision to enhance performance. In this paper, we propose a novel end-to-end framework with faster speed, and higher accuracy, enabling efficient and effective occupancy forecasting in a pure camera-only setting.
3
OCCPROPHET
3.1 OVERVIEW
The overview of the proposed OccProphet is illustrated in Figure 3 Given multi-frame surround- view RGB images as input, 2D features are extracted using a shared image encoder. These 2D features are subsequently lifted into 3D space and aggregated into multi-frame 3D voxel features through depth estimation and voxel pooling. We design the following unshared pipeline for each of the occupancy and occupancy flow branches. Specifically, the pipeline consists of four compo- nents: the Observer, Forecaster, Refiner, and Predictor. The Observer module efficiently and ef-
3

---

Published as a conference paper at ICLR 2025
Image E Encoder
bbserver
Forecaster
Refiner
P redictor Mulbi-ramm Future Occupancyl Flow
F
20 3D
+CG
20000 mnn
Mult-frame Mutt-Camera
Camera Parameters
Occupancy) Flow Head
Mult-frame
2D 3D
Voxels
Image
Reshape
Encoder
Efficent 4D Aggregation
Condition Generator
Conditonal Forecaste
Figure 3: Overview of OccProphet. It receives multi-frame images from surround-view cameras as input and outputs future occupancy or occupancy flow. It consists four key components: the Observer, Forecaster, Refiner, and Predictor. The Observer module aggregates spatio-temporal in- formation. The Forecaster module conditionally generates preliminary representations of future scenarios. These preliminary representations are refined by the Refiner module. Finally, the Predic- tor module produces the final predictions future occupancy or occupancy flow.
fectively aggregates spatio-temporal information within multi-frame observations (i.e., multi-frame 3D voxel features). The Observer's output then undergoes Forecaster, which adaptively predicts future states, ensuring flexibility across diverse traffic conditions. The Refiner module further hances the quality of these predictions by enabling cross-frame interactions. Finally, the Predictor module decodes the refined future states into either occupancy occupancy flow.
3.2 OBSERVER
The Observer takes the 4D motion-awan feature Fmotion as input, and generates spacetime- aware representation. Let I, {1") Neam denote the multi-camera RGB images timestamp t E (T). where Neam refers to the total number of surround-view cameras, and T is the total number input frames. applied to {I,]t = 1, I.....Ty T} to extract 2D features. For each frame, these 2D features are projected to 3D space and then aggregated into voxelized 3D features (Philion & Fidler 2020).
shared image encoder (i.e., ResNet (He al., (016)) is
The 3D features from multiple frames are aligned into the current-frame coordinate system us- ing degrees of freedom (6-DoF) ceo-vehicle poses. We then concatenate the aligned 3D fea- tures into 4D feature F' xgxxxXXxxx2 where C is the number of channels, and (X, Y, 2) represents the size of voxelized 3D feature volume. Subsequently, the motion-awar 4D feature F Footion E (C+))x)) x2 is generated by concatenating 6-DoF ego ego-vehicle poses.
To generate the spacetime-aware representation, directly processing the 4D motion-aware feature using convolutional operations is intuitive. However, the direct processing imposes a substantial computational burden, and ignores the fact that a large portion of the 3D space is unoccupied, which leads to the inherent sparsity of the motion-aware feature. To address this issue, we efficiently and effectively generate the spacetime-aware feature from Fmotion using the Observer module. The Observer comprises an Efficient 4D Aggregation module and a Tripling- Fusion module.
3.2.1 EFFICIENT 4D AGGREGATION
Directly aggregating the original 4D feature Fmotion will incur a high computational cost. For efficiency, we design the Efffcient 4D Aggregation (B4A) module to first produce compact features through downsampling, and then exploit spatio-temporal interactions on the compact features achieve aggregation, fol- lowed by the upsampling process to compen- sate for the information loss. The architec- ture of the E4A module is shown in Fig- ure 4 We first reduce the channel number of Fmotion from (C 6) to C through 3D convolution, thereby forming a feature E XTxXXXxXXx We gradually downsample O
Mull- Valoo
Arberton
Downsampling
f Upsampling
Residual Connection
Figure 4: Efficient 4D Aggregation (E4A).
4

---

Published as a conference paper at ICLR 2025
to reduce computation during aggregation. However, such downsampling inevitably results in non- negligible information loss, especially for small objects. To compensate for this loss, we conduct two operations. On one hand, perform apatio-temporal interactions the downsampled features, that is, the Pripling-Attentonn Fusion module (to be illustrated in Section (.2.2). On the other hand, the post-interaction features upsampled, and further summed with the features at the same reso- lution prior to downsampling. These two operations continue until the resolution of the upsampled feature matches the resolution of the initial motion-aware feature.
The output 4D representation of the E4A module possesses spatio-temporal awareness. effectively preserves the 3D ge- ometric details of the
Table I: Comparisons with differ- ent representation styles. Np: Total number of parameters.
that the BEV and TPV representations are unable to capture. As shown in Table D using the E4A representation achieves higher performance than the BEV and TPV representations with fewer parameters and slight increase in computational costs.
Style
Np (M)
FLOPs (G)
IoU, (%)
BEV
102
1370
TPV
181
1574
26.96
E4A
67
1737
27.50
3.2.2 TRIPLING-ATTER ATTENTION FUSION
4D feature can considered as a combination multiple
3D voxel-wise features along the temporal dimension. The Tripling-Attenninn Attention Fusion (TAF, shown in Figure module is specifically designed to facilitate apatio-tempone interactions across multiple 3D features, depicted the left of Figure 5 Notably, downsampling reduces computational cost by lowering feature resolution, while the Tripling-Attention Fusion module makes a further step through the proposed tripling operation, as illustrated on the right of Figure 5
The tripling operation is de- signed to understand the 3D space from three complemen- tary and compact perspectives, which can retain 3D scene in- formation with fewer compu- tational costs. Specifically, tripling operation decomposes 3D feature into three dis- tinct branches: scene, height, and BEV. This decomposition compresses 3D features into ID and 2D features, significantly re- ducing the subsequent computa- tional overhead. The scene branch can extract the global context of the corresponding frame, pro- viding an overall understanding of the scenario. The height branch retains vertical details, serving as complementary clues to the 2D BEV branch to enhance the representation capability of the 3D geometric information. The three branches can be computed as follows:
Multi-frame Voxels
Um
Linear
Conv
Temporal Attention
Temporal Attention
Temporal Attention
indowed Attention
S
S,
3D GAP
2D GAP
GAP
Tripling
Tripling
Scene
Height
BEV
Multi- frame
U
TAF
Voxels
Tripling
Figure 5: Tripling-Atteniion Fusion (left) and Tripling (right).
(Gaaaa (U))));
H = Act((Nomt(Coanm) (GAM (U)))),
(2)
B W-MSA(GAAPID(UJ))
where U E IR x denotes the i-th downsampled feature input to the Tripling-Attention Fusion module. GAPıD. GAPæ. and GAP are 3D, 2D, and ID global average pooling. Linear, ConVID. Norm, and Act refer to single linear layer. ID convolution, normalization, and activa- tion. respectively, W-MSA is the window-based multi-head self-attention block (Liu et al. 2021). S X2xxxxxxxxx H. R'xxxxxxx 3' and B E R Txxxx 2 x denote the outputs of scene, height, and BEV branches. After the tripling operation, we interact and fuse scene, height, and BEV features. Specifically, we first independently apply the temporal attention along the time axis to different branches, then sum over the three branches using the broadeast technique. The process formulated as follows:
UHF TAwene (S) TAneigh (H) TA (B),
where TA TAscemes Thnnight and TABEV denote temporal attention for scene, height, and BEV branches; is broadcast-style plus; Unr TT. TxXx x
is the output feature of the TAF module.
5

---

Published as a conference paper at ICLR 2025
3.3 FORECASTER
Given a spatio-temporal representation Oaba RTxxxXxX x2 output from the Observer, the Fore- caster (shown in Figure (6) is supposed to generate future states. We first reshape Oobs by collapsing the time axis into the channel axis, resulting in the reshaped features Obss RCCXXxXx2,
Farare
Multi-frame uture Vxxels
straightforward approach to forecasting is using single linear layer to predict features for future frames. This approach encourages the network to learn static weights to fit different traffic scenarios. How- ever, traffic situations vary significantly across different traffic scenarios, presenting distinct challenges in predict- ing future changes using a single linear layer. For instance, forecasting environmental changes is more difficult in crowded intersection with many moving objects than on an empty highway with few vehicles. The spatio-temporal complexity in the former is far greater than in the latter.
F future
F cond
C Conditional
Condition
cond
F Forecaster
G Generator
Linear
shared
O'
Linar
Linear
3D GAP
3D GAP
Considering these challenges, we propose forecasting oc- cupancy with flexibility to adapt to various traffic scenarios featuring diverse spatio-temporal complexities. To achieve this, we design novel Forecaster module that predicts fu- ture states based on the overall environmental condition. The Forecaster comprises Condition Generator and Conditional Forecaster. We first use Con- dition Generator comprising a 3D GAP and a shared linear layer across different frames to extract the condition from the observation O
Reshape
Multi-frame
Condition
Past Voxels
Weights
Figure 6: Architecture of Forecaster.
Foodd =
(5)
where Foood E RTXXXxxIxxI denotes the overall environmental condition, which is then reshaped into Found TC by collapsing the time axis into the channel axis. Foond is subsequently passed Conditional Forecaster to predict future states. Specifically, linear layer is applied Found to produce adaptive weights for specific traffic scenarios. Another linear layer is then used to predict future states conditioned by these adaptive weights.
Woond = Linear( CalinearFFconi Feoma), Fl = Linear ohs [Wond))
where W Woddd TCXTCC denotes the adaptive weights, F' E CXX'XXXXXX is the future states of T' future frames. F Flunur is reshaped into Frucure RT xxxxxxxxxxxx to recover the time axis. F Frouree as the preliminary feature for future environments, is then refined by the Refiner module.
3.4 REFINER
Since the Forecaster module predicts Fiunure using linear projection, it inevitably lacks cross-frame interactions. The Refiner is designed to enhance the forecasted results via further interactions be- tween future frames, as well incorporating historical frames as supplementary information. The E4A module, described in Section Section 3 is spatio-temporal interaction module. Further- more, taking a review the E4A, for any input feature Q E xxxxxxxXXxX the function of the E4A module can be formulated as:
Q' = -4A(Q) = QQQ((),
(7)
where Q' E #TXxxX*V*Z is the output feature of E4A, and F denotes the transformation func- tion. Considering residual networks can aid in refinement and network optimization (He et al., 2016), it reasonable to regard E4A refinement transformation for features, which also reduces the learning complexity of earlier modules. Building on this insight, we further introduce a Refiner reusing the E4A architecture to refine the forecasted future states. The Refiner is applied to the forecasted feature Fiuture from the Forecaster and the feature output Oots from the Observer, produc- ing an enhanced feature V = E4A([[0sss (ruuee))+11:4++ wine E RT XXX Z for subsequent forecasting of occupancy and flow.
For fair comparisons, the Predictor for occupancy and occupancy flow prediction, and the overall training loss function follow those in Cam4DOcc (Ma et al. 2024a).

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUPS
Datasets. Following Cam4DOcc, 700 out of 850 scenes nuScenes (Caesar al. 2020) and nuScenes- -cccppenc (Wang et al., 2023a) datasets, and 130 out of 180 scenes in Lyft-Levell (Hous-) ton al. 202T) with occupancy labels are used for training, while the remaining scenes are used for evaluation. For nuScenes and nuScenes-Occupange the numbers of sequences for training and eval- uation are 23930 and 5119, respectively. For Lyft-Level5, 15720 and 5880 sequences are used for training and evaluation, respectively. The total length of the sequence is set as 7, including 3 frames as observations (2 past frames and 1 present frame) and future frames for forecasting. The range of the occupancy labels is [-51.2 2 m, 51.2 ml] for and y axes, and [-5 3 m] for & axis. The voxel resolution is 0.2 m, and the grid size is (512, 512, 40) for occupancy labels. The forecasting performances are reported with different time intervals due to the different annotated frequencies, i... 2 Hz for nuScenes and nuScenes- muScenes-Occupaney Occupancy, and 5 Hz for Lyft-Level5.
Evaluation Protocol and Metries. To fully evaluate the forecasting performance, we follow Cam4DOce to adopt the following three-level camera-only occupancy forecasting tasks: (1) Fore- casting inflated general movable objects (GMO): the categories for occupancy grids within bounding boxes from nuScenes and Lyft-Level5 datasets are defined as GMO, while the remaining categories defined as others. (2) Forecasting fine-grained GMO: the same category definition as (1), while the bounding box labels of GMO are replaced with fine-graines voxel-wise annota- tions from the nuScenes-Oee Occuupncy dataset. (3) Forecasting fine-grained GMO and fine-grained general static objects (GSO): the labels of GMO and GSQ are from fine-grained annotations. For Lyft-Level5,, the evaluation is only conducted on task (1) due the lack of fine- nee-grained occupancy annotations. For all tasks, intersection over union (IoU (%)) is adopted as the evaluation metric to evaluate occupancy estimation performance for each category of the current frame (IoU,), and any of future frames (IoU,), and the whole span (IoU,). More details can be found in Cam4DOce.
Implementation Details. The proposed OccProphet takes 6 images with 448 x 800 pixels from different surround views. Following Cam4DOcc, we use ResNet (He et al. 2016) pre-trained on ImageNet (Deng et al. 2009) with FPN (Lin et al. 2017) as the image encoder. For ablation studies, we use ResNet18 for efficiency, while ResNet34 with deformable convolution (Dai et al.. 2017) is adopted for main results. All models are trained with a batch size of 4 on 4 RTX 4090 GPUs and tested on a single RTX 4090 with 24G memory. AdamW (Loshchilov & Hutter 2019) optimizer with an initial learning rate of 3e-4 and weight decay of 0.01 is adopted to train the models.
4.2 MAIN RESULTS
Evaluation forecasting inflated GMO. The results of forecasting inflated GMO are listed in Table 2 with five comparison methods in total: OpenOccupancy-CC SPC, PowerBEV-3D, BEVDet4D, and OCFNet (Cam4DOcc). OccProphet achieves higher performance than all the com- pared approaches. Specifically, on the nuScenes dataset, OceProphet surpasses the BEV-based ap- proaches PowerBEV-3D and BEVDet4D by 2.76-1 1.. 1.28% 2.07-5.699% 2.28-7.29% in IoU,, IoU,, IoU, respectively. OccProphet also surpasses the voxel-based method Cam4DOce all the metrics, especially IoU, by relatively 4.18% and 15.92% on nuScenes and Lyft-Level5 datasets respec- tively. Qualitative results in Figure demonstrate the superiority of OceProphet. The first group indicates OccProphet's 's adaptability in low-light conditions.
Table 2: Performance on forecasting inflated GMO. SPC: SurroundDepth (eei et al.
2023a
PCPNet (Luo et al. 2023 Cylinder3D (Zhu et al. 2021).
nuScenes
Lyft-Level5
Method
IoU,
loU, (2
IoU, IoU, IoU, (0.8 s) IoU,
OpenOccupancy- C Wang et al.
2023b
12.17
11.45
11.74
14.01
13.53
13.71
SPC [Luo et al. 2023 Wei 2023a
Zhu et al. 2021
1.27
1.42
PowerBEV-3D (Li et al.
2023a
23.08
21.25
21.86
26.19
24.47
25.06
BEVDet4D (Huang& Huang. 2022
31.60
24.87
26.87
OCFNet (Cam4100cc)
Ma et al. 2024a
31.30
26.82
27.98
36.41
33.56
34.60
OccProphet (ours)
34.36
26.94
29.15
43.38
37.92
40.11

---

Published as a conference paper at ICLR 2025
PRONT
FROh
LEFT
RIGHT
FAINT
BACK
BACK
BACK
RIGHT
Images
Cam4DOcc
OccP rophet
Ground Truth
Figure 7: Qualitative results of Cam4DOcc and OccProphet in the future 2 seconds. Black arrows denote the motion trends of moving objects. Red dashed rectangles represent that the results of OccProphet are more consistent with the ground truth than those of Cam4DOcc.
Evaluation on forecasting fine-grained GMO. The second task evaluates the performance of forecasting fine-grained GMO. As shown in Table 3 when changing the GMO labels from inflated format using bounding boxes to fine-grained voxel- labels, the performances of approaches decrease significantly except the point cloud prediction method SPC achieving slightly better per- formance compared to Table 2 As pointed out in Cam4DOcc, the reason is that the fine-graing grained 3D structures of the moving objects are difficullt to estimate using past and current continuous RGB images, while the labels for training SPC are also fine-graine mne-grained and sparse. However, SPC still per- forms than other counterparts due to the lack of shape consistency. Cam4DOce keeps the lead over other competitors. Despite the challenging task, OccProphet surprisingly precedes all other ap- proaches by a large margin. Specifically, the performances of OceProphet in IoU,, IoU,, loU, are relatively 34.3%, 10.43%, 18.61% % higher than those of Cam4DOce. The above results demonstrate the effectiveness of OccProphet in capturing the intricate 3D details of moving objects.
Table 3: Performance forecasting fine-grained GMO.
nuScenes muspenes-Occupancc
Method
IoU.
IoU, (2 s)
IoU,
OpenOccupancy- Wang Luo et al. 2023 Wei el al.
2023b
10.82
8.02
8.53
2023a
Zhu al. 2021
5.85
1.12
PowerBEV-3ID al. 2023a
5.91
5.25
5.49
OCFNet (Cam4DOce) (aa et al.
2024a
11.45
9.68
10.10
OccProphet (ours)
15.38
10.69
11.98
Table 4: Performance on forecasting fine-grained GMO and fine-grained GSO.
IoU,
IoU, (2 s)
IoU,
Method
GMO
GSO
mean
GMO
GSO
mean
GMO
OpenOccupancy-CC Wang et al.
2023b
9.62
17.21
13.42
7.41
17.30
12.36
7.86
SPC (Luo et al. 2023 Wei et al.
2023a:
Zhu et al. 2021
5.85
3.29
4.57
1.08
1.40
1.24
1.12
PowerBEV-3D (Li
2023a
5.91
5.25
5.49
OCFNet (Cam4DOcc)
Ma al.
2024a
11.02
17.79
14.41
9.20
17.83
13.52
9.66
OccProphet (ours)
13.71
24.42
19.06
9.34
24.56
16.95
10.33
8

---

Published as a conference paper at ICLR 2025
Evaluation forecasting fine-grained GMO and fine-grained GSO. The performances of forecasting fine-gres ane-ggained GMO and fine-grained GSO, are listed in Table passes Cam4DOcc in all metrics.
OccProphet still sur-
4.3 ABLATION STUDIES
Effectiveness of each component. Table demonstrates the ablation studies on the effectiveness of each component. Removing the Observer (Row 3) leads to around 1% drop in IoU,, showing the importance of the Observer extracting spatio-temporal information from observations. If the Forecaster is removed (Row 4), the performance decreases by 0.74%, in- dicating the advantage of predicting future states adaptively based on the traffic environment compared direct prediction. Row shows that re- moving the Refiner brings 0.8% drop. If all the components are removed (the last row), where only a single linear layer is used predict future states, the performances are sharply reduced to 26.07%, further reveal- ing the effectiveness of the components. Qualitative results of using the Observer not are in Figure g The integration of Observer promotes spatio-temporal representa- tiveness, enhancing the forecasting consistency with the ground truth. Other qualitative results are shown in the appendix.
Table Effectiveness of each component.
Method
IoUy
OccProphet
28.24
w/o Observer
27.25
w/o Forecaster
27.50
w/o Refiner
27.44
w/o All
26.07
FRONT LEFT
FROTT
RIGHT
BACk
BACK
BACK
EEFT
IIGHT
ERONT
FRON
FROST
RIGHT
MARK
RACE
JACK
Images
w/o Observer
w/ Observer
Ground Truth
Figure 8: Qualitative results of using the Observer or not. Black arrows denote the motion trends of moving objects. Red dashed rectangles represent that the results with the Observer are more consistent with the ground truth than those without the Observer.
Effectiveness of E4A representation. To further justify the effectiveness of E4A representation, we experiment with different representation styles of the Observer. As shown in Table [ the UNet like E4A balances computational cost and performance well.
4.4 COMPARISON OF MODEL COMPLEXITY
In this section, we compare the model complexity be-
Table 6: Comparison of model complexity. Np.
Total number of parameters. Mem.: Occupied GPU memory during training.
tween Cam4DOcc and Oc- cProphet. As shown in Table 6, the number of parameters, memory usage, and FLOPs of OccProphet are decreased by 58-78% compared to Cam4DOcc, while OceProphet achieves 4% relative increase IoU, y. and has 2.6x the FPS speed of Cam4DOcc, justifying the efficiency and effectivvness of OccProphet.
Method N, (M) Mem. (G) FLOPs (G)
FPS
IoU,
Cam4DOoc
370
57
6434
1.7
27.98
OccProphet 82(7882())))) 24(58%.) 1985(69% (1.661551) 29.15 (4%
CONCLUSION
This paper proposes OccProphet, novel camera-only framework for occupancy forecasting. The framework employs effective training and inference. Such efficiency and effectiveness are achieved through 4D aggrega- tion and attention fusion OceProphet's 's superiority in both forecasting accuracy and efficiency. It outperforms the tfate-of-the- art Cam4DOce in occupancy forecasting by relatively 4%, 18% three datasets, while operating
Forecree-rooeeate---Reeeee pipeline, specifically designed for efficient and
features. Experimental results demonstrate

---

Monoscene: Monocular 3d semantic scene completion. In Published as a conference paper at ICLR 2025
2.6x faster and reducing computational costs by 78%. We hope OccProphet motivate fu- ture research in efficient occupancy forecasting and its applications in on-vehicle deployment.
ACKNOWLEDGEMENTS
The research work was conducted in the JC STEM Lab of Machine Learning and Computer Vision funded by The Hong Kong Jockey Club Charities Trust and was partially supported by the Research Grants Council of the Hong Kong SAR, China (Project No. PolyU 15215824).
REFERENCES
Ben Agro, Quinlan Sykora, Sergio Casas, Thomas Gilles, and Raquel Urtasun. Uno: Unsupervised occupancy fields for perception and forecasting. In CVPR, 2024.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020.
Anh-Quan Cao and Raoul De Charette. CVPR, 2022.
Xiaozhi Chen, Kaustav Kundu, Ziyu Zhang, Huimin Ma, Sanja Fidler, and Raquel Urtasun. Monoc- ular 3d object detection for autonomous driving. In CVPR, 2016.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Wenchao Ding, Lu Zhang, Jing Chen, and Shaojie Shen. Safe trajectory generation for complex urban environments using spatio-temporal semantic corridor. RA-L, 2019.
Wenchao Ding, Lu Zhang, Jing Chen, and Shaojie Shen. Epsilon: An efficient planning system for automated vehicles in highly interactive environments. T-RO, 2021.
Shaoheng Fang. Zi Wang, Yiqi Zhong, Junhao Ge. and Siheng Chen. Tbp-former: Learning tem- poral bird's s-eye-view pyramid for joint perception and prediction in vision-centric autonomous driving. In CVPR, 2023.
Bryce Ferenezi, Michael Burke, and Tom Drummond. Motionperceiive:: Real-time occupancy fore- casting for embedded systems. RA-L,, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2016.
John Houston, Guido Zuidhof, Luca Bergamini, Yawei Ye, Long Chen, Ashesh Jain, Sammy Omari, Vladimir Iglovikov, and Peter Ondruska. One thousand and one hours: Self-driving motion pre- diction dataset. In CoRL, 2021.
Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. Fiery: Future instance prediction in bird's bird's-eye view from surround monocular cameras. ICCV, 2021.
Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues in multi-camera 3d object detec- tion. arXiv preprint arXiv:2033.1705. 2022.
Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: performerfommnce multi- camera 3d object detection in bird-eye-view. arXiv preprint 12. 2021.
10

---

Published as a conference paper at ICLR 2025
Yuanhui Huang, Wenzhao Zheng, Yunpeng Zhang, Jie Zhou, and Jiwen Lu. Tri- Tripperspective view for vision-based 3d semantic occupancy prediction. CVPR, 2023a.
Zhiyu Huang, Haochen Liu, and Chen Lv. Gameformer: Game-theoretic modeling and learning of transformer-based interactive prediction and planning for autonomous driving. In ICCV, 2023b.
Tarasha Khurana, Peiyun Hu, David Held, and Deva Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecasting. In CVPR, 2023.
Peizheng Li, Shuxiao Ding, Xicyuanli Chen, Niklas Hanselmann, Marius Cordts, and Juergen Gall. Powerbev: a powerful yet lightweight framework for instance prediction in bird's-eye view. arXiv preprint arXiv:2306. 10761, 2023a.
Yiming Li, Zhiding Yu, Christopher Choy, Chaowei Xiao, Jose M Alvarez, Sanja Fidler, Chen Feng, and Anima Anandkumar. Voxformer: Sparse voxel transformer for camera-based 3d semantic scene completion. In CVPR, 2023b.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer. Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, ICLR, 2019.
Zhen Luo, Junyi Ma, Zijie Zhou, and Guangming Xiong. Pepnet: An efficient and semantic- enhanced transformer network for point cloud prediction. IEEE Robotics and Automation Letters, 8(7):4267-4274, 2023.
Junyi Ma, Xieyuanli Chen, Jiawei Huang. Jingyi Xu, Zhen Luo, Jintao Xu, Weihao Gu, Rui Ai, and Hesheng Wang. Cam4doce: Benchmark for camera-only 4d occupancy forecasting in au- tonomous driving applications. In CVPR, 2024a.
Qihang Ma, Xin Tan, Yanyun Qu, Lizhuang Ma, Zhizhong Zhang, and Yuan Xie. Cotr: Compact occupancy transformer for vision-based 3d occupancy prediction. In CVPR, 2024b.
Benedikt Mersch, Xieyuanli Chen, Jens Behley, and Cyrill Stachniss. Self-supervised point cloud prediction using 3d spatio-temporal convolutional networks. In CoRL, 2022.
Chenbin Pan, Burhaneddin Yaman, Tommaso Nesti, Abhirup Mallik, Alessandro G Allievi, Senem Velipasalar, and Liu Ren. Vlp: Vision language planning autonomous driving. CVPR, 2024.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 194-210. Springer, 2020.
Haoran Song, Wenchao Ding, Yuxuan Chen, Shaojie Shen, Michael Yu Wang, and Qifeng Chen. Pip: Planning-informe trajectory prediction for autonomous driving. ECCV, 2020.
Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from single depth image. In Proceedings of the IEEE conference computer vision and pattern recognition, 2017.
Xiaoyu Tian, Tao Jiang, Longfei Yun, Yucheng Mao, Huitong Yang, Yue Wang, Yilun Wang, and Hang Zhao. Oce3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. NeurIPS, 2023.
Wenwen Tong, Chonghao Sima, Tai Wang, Li Chen, Silei Wu, Hanming Deng, Yi Gu, Lewei Lu, Ping Luo, Dahua Lin, et al. Scene as occupancy. ICCV, 2023.
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Feos3d: Fully convolutional one-stage monocular 3d object detection. In ICCV, 2021.

---

Published as a conference paper at ICLR 2025
Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yumpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du, Ji- wen Lu, and Xingang Wang. Openoceupancy. A large scale benchmark for surrounding semantic occupancy perception. In ICCV, 2023a.
Xiaofeng Wang, Zheng Zhu, Wenbo Xu, Yunpeng Zhang, Yi Wei, Xu Chi, Yun Ye, Dalong Du, Ji- wen Lu, and Xingang Wang. Openoceupancy: A large scale benchmark for surrounding semantic occupancy perception. In ICCV, 2023b.
Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon, Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In CORL, 2022.
Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Yongming Rao, Guan Huang, Jiwen Lu, and Jie Zhou. Surrounddepth: Entangling surrounding views for self-supervised multi-camera depth estimation. In Conference on robot learning, pp. 539-549. PMLR, 2023a.
Yi Wei, Linqing Zhao, Wenzhao Zheng, Zheng Zhu, Jie Zhou, and Jiwen Lu. Surroundoce: Multi- camera 3d occupancy prediction for autonomous driving. In ICCV, 2023b.
Xinshuo Weng, Jianren Wang. Sergey Levine, Kris Kitani, and Nicholas Rhinehart. Inverting the pose forecasting pipeline with sp12: Sequential pointcloud forecasting for sequential pose fore- casting. In CoRL, 2021.
Xinshuo Weng, Junyu Nan, Kuan-Hui Lee, Rowan McAllister, Adrien Gaidon, Nicholas Rhinehart, and Kris M Kitani. S2net: Stochastic sequential pointeloud forecasting. ECCV, 2022.
Huaiyuan Xu, Junliang Chen, Shiyu Meng, Yi Wang, and Lap-Pui Chau. A survey on occupancy perception for autonomous driving: The information fusion perspective. Information Fusion, 14: 102671, 2025.
Yu Yang. Jianbiao Mei, Yukai Ma, Siliang Du, Wenqing Chen, Yijie Qian, Yuxiang Feng, and Yong Liu. Driving in the occupancy world: Vision-centric 4d occupancy forecasting and planning via world models for autonomous driving. arXiv preprint arXiv:2408. 14197, 2024.
Yunpeng Zhang, Zheng Zhu, Wenzhao Zheng, Junjie Huang, Guan Huang. Jie Zhou, and Jiwen Lu. Beverse: Unified perception and prediction in birds-eye-view for vision-centric autonomous driving. arXiv preprint urXiv:2205.097433 2022.
Yunpeng Zhang, Zheng Zhu, and Dalong Du. Oceformer: Dual-path transformer for vision-based 3d semantic occupancy prediction. In ICCV, 2023.
Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9939-9948, 2021.
A APPENDIX
A.1 ADDITIONAL QUANTITATIVE RESULTS
Forecasting inflated GMO and fine-grained GSO: the category definitions of GMO and GSO are the same those in the task of Forecasting fine-grained GMO. The labels of inflated GMO generated from bounding boxes, while the labels of fine-grained GSO are occupancy annotations. The performances of forecasting inflated GMO and fine-grained GSO are listed in Table 7 SPC is still the worst, where the IoU of inflated GMO remains consistent in Table 2 cally outperforms other approaches including OpenOccupancy- and Cam4DOce by a large margin. Regarding GMO, OccProphet is 3.77% and 0.92% higher than Cam4DOcc in the current and future moments, respectively. For GSO, OccProphet surpasses Cam4DOce by 6.46% and 6.38% in IoU, and loU,, y, respectively. When taking multiple future frames for evaluation, OccProphet remains the best with an impressive superiority of 2.21% over the second best Cam4DOce in IoU,, These sults demonstrate the superiority of our method of extracting aattiotemmpoal features and predicting future states.
OccProphet dramati-
12

---

Published as a conference paper at ICLR 2025
Table 7: Performance forecasting inflated GMO and fine-grained GSO.
IoU,
1oUy (2
IoU,
Method
GMO
GSO
mean
GMQ
GSO
mean
GMO
OpenOccupancy-CC C Wang et al.
2023b
13.53
16.86
15.20
12.67
17.09
14.88
12.97
SPC (Luo et al. 2023 Wei et al.
2023a
Zhu et al. 2021
1.27
3.29
2.28
1.40
PowerBEV-3D (Li
2023a
23.08
21.25
21.86
OCFNet (Cam4DOcc) Ma al.
2024a
29.84
17.72
23.78
25.53
17.81
21.67
26.53
OccProphet (ours)
33.61
24.18
28.89
26.45
24.19
25.32
28.74
A.2
QUALITATIVE RESULTS OF ABLATION STUDIES
Effectiveness of the Forecaster. Figure shows the qualitative results with or without using the Forecaster module. The Forecaster is adept at perceiving moving objects, while forecasting with only a single linear layer (w/o Forecaster) tends to miss some moving objects.
FRONT
RACK
RIGAT
RIGHT
Ground Truth Figure 9: Qualitative results of using the Forecaster or not. Black arrows denote the motion trends of moving objects. Red dashed rectangles represent that the results with the Forecaster are more consistent with the ground truth than those without the Forecaster.
Images
w/o Forecaster
w/ Forecaster
Effectiveness of the Refiner. Figure 10 shows the qualitative results using the Refiner or not. It evident that using the Refiner yields better forecast results for both moving and static objects, indicating its forecast refinement capability.
KACK
616
HIG
G Ground Truth Figure 10: Qualitative results of using the Refiner or not. Black arrows denote the motion trends of moving objects. Red dashed rectangles represent that the results with the Refiner are more consistent with the ground truth than those without the Refiner.
Images
w/o Refiner
w/ Refiner
A.3 DISTINCTIONS FROM THE TRADITIONAL ENCODER-DECODER DECODER ARCHITECTURE
The traditional encoder-decoder architecture comprises encoder for representation extraction and a decoder for occupancy prediction, as adopted by OceNet (Tong et al. 2023) and Cam4DOcc. How- the traditional architecture either loses 3D geometry details-e.g. OccNet adopts BEV-based encoder. or introduces high computational cost-e.g., Cam4DOce utilizes vanilla 3D convolutional networks the encoder and decoder.
13

---

Published as a conference paper at ICLR 2025
In OceProphet, the Observer works similarly to encoder, while the combination of Forecaster and Refiner works similarly to a decoder. Unlike the traditional encoder-decoder architecture, Oc- eProphet pushes the efficiency frontier of 4D occupancy forecasting. To achieve this, the Observer and Refiner enable spatio-temporal interaction using the Efficient 4D Aggregation module, and the Forecaster adaptively predicts future states using lightweight condition mechanism. Overall, Sheerver-Forccastee-RReeee framework emphasizes 4D spatio-temporal interaction and conditional forecasting, meanwhile maintaining efficiency.
A.4 OCCUPANCY FORECASTING OVER VARYING TIME HORIZONS
The performance of occupancy forecasting over varying time horizons is critical in autonomous driv- ing scenarios. We compare OccProphet with OpenOccupancy-CC (Wang al. 2023b), PowerBEV- 3D (Li et al. 2023a), and Cam4DOcc (Ma et al. 2024a) in terms of occupancy forecasting accuracy over varying time horizons, as shown in Table 8 We can see that (1) OccProphet consistently out- performs other approaches across all time horizons on three datasets. (2) The longer the forecasting period, the lower the accuracy of methods, indicating that forecasting becomes increasingly dif- ficult.
Table 8: Performance on occupancy forecasting over varying time horizons.
nuScenes
Lyft-LevelS
muScenes-Occupen
Method
0.5s
1.0s
1.5s
2.0s
0.2s
0.4s
0.6s
0.8s
0.5s
1.0s
1.5s
2.0s
OpenOccupancy-CC
12.07
11.80
11.63
11.45
13.87
13.77
13.65
13.53
9.17
8.64
8.29
8.02
PowerBEV-3D
22.48
22.07
21.65
21.25
25.70
25.25
24.82
24.47
5.74
5.56
5.41
5.25
OCTNet (Cam4DOcc)
29.36 28.30
27.44
26.82
35.58
34.96
34.28
33.56
10.64
10.20
9.89
9.68
OccProphet (ours)
32.17 29.60
27.95
26.94
42.34
40.87
39.38
37.92
13.64
12.10
11.27
10.69
A.S
FAILURE CASES
A.5.1
UNSEEN SCENARIOS
To test forecasting performance unseen scenarios, we conduct cross-domain experiment. Specif- ically, we train the model on the Lyfe-Level5 dataset and test it the nuScenes dataset. The visu- alization results are shown in Figure TT. We can observe that cross-domain occupancy forecasting performs worse than intra-domain forecasting within single dataset. We consider that this perfor- mance gap may be due to the difference in sensor settings between the two datasets. In the future, research generalized occupancy forecasting will be valuable direction worth exploring.
EIGHT
Images
nuScenes nuSce
Figure 11: Failure cases in unseen scenarios. Red dashed rectangles indicate that the intra-domain results (e.g., nuScenes to nuScenes) are more consistent with the ground truth than cross-domain occupancy forecasting (e.g., Lyft-Level5 to nuScenes).
A.5.2 OCCLUDED SCENARIOS
We investigate the forecasting performance in occluded scenarios, which frequently occur traffic scenarios. Ground truth occupancy and forecasted future occupancy by OccProphet, over varying time horizons, are qualitatively visualized in Figure 12] It can be observed that the occupancy labels of the occluded object over different time horizons complete, ensuring the training reliability. The forecasted results of OceProphet demonstrate its capability of handling occluded scenarios to certain degree, which can still be further improved in future research.
14

---

Published as a conference paper at ICLR 2025
FRONT
FRONT RIGH1
EOOTT
FROM
FRONT
RONN
ERONT
AIGH
LIGHT
FIGHT
AACK
BACK
BACK
Past Current Input Images
1.99
RACK
EA6K
BACK
Past and Current Ground Truth
Future Forecast
0.s
1.0c
1.56
2.0
Future Ground Truth
05s
1.0s
1.5s
2.05
Figure 12: Failure cases in occluded scenarios. In input images, yellow rectangles indicate the object of interest, while dashed rectangles denote that the object is nearly entirely occluded. Red rectangles indicate that the occupancy ground truths of the occluded object complete past, current, and future frames. Blue rectangles reveal that OccProphet is able to forecast the future occupancy of the occluded object, which can mitigate the occlusion issue to certain extent.
FRURT
BACK
BCKK
Images
CamADOce
OccProphet
Ground Trutti
Figure 13: Failure cases in dense scenarios. Red dashed rectangles indicate that the results of OccProphet are more consistent with the ground truth than those of Cam4DOcc.
15

---

Published as a conference paper at ICLR 2025
A.5.3 DENSE SCENARIOS
To evaluate model precision fine granularity, we qualitatively visualize the occupancy forecasting of Cam4DOcc and OccProphet in dense scenarios, as shown in Figure [3] The visualization reveals that both Cam4DOcc and OceProphet encounter challenges in fine- tneegraindd forecasting, indicating huge difficulty. In comparison, OccProphet's 's results are closer to the ground truth. We believe that this is attributed to our proposed Observer-Forees Seoerver-Foreeestee--e Refiner framework. We regard fine-grained occupancy forecasting as valuable direction for future research.
16


Reviews:
[{"review_id": "FCfHw5giHs", "reviewer": "Reviewer_gfQ3", "summary": "This paper introduces OccProphet, a novel camera-only framework for occupancy forecasting. It features an Observer-Forecaster-Refiner pipeline optimized for efficient training and inference, utilizing 4D aggregation and tripling-attention fusion on reduced-resolution features. Experimental results show that OccProphet outperforms existing methods in both accuracy and efficiency.", "questions": "1. What part of the Cam4DOcc model contributes to its high computational cost? How much memory does Cam4DOcc use during training and testing? Approximately how many GPU days are needed for full training?\n\n2. In multi-frame 3D occupancy results, how can environmental dynamics be effectively captured? The proposed Observer-Forecaster-Refiner pipeline seems to learn the dynamics of objects in the latent space without strict physical constraints. If the real world presents scenarios that are rare or unseen in the dataset, could 4D predictions fail? Are there any examples of such failures?\n\n3. 4D prediction is challenging, especially when occlusions are frequent, leading to potential voxel loss. Have the labels in the 4D dataset been completed to account for these occlusions? If they have, can the authors' method handle situations where any frame from the historical RGB images loses an object?\n\n\n4. Are the 6-DoF camera parameters in the Observer used to align historical 3D features with the current frame? In lines 188-191, why does F change to F_{motion}, resulting in an extra matrix dimension of 6×X×Y×Z? Is this converting the 6-DoF pose into a matrix?\n\n5. In lines 206-207, how does C+6 become C? Is this done through a 1×1×1 3D convolution?\n\n6. What is the difference between the E4A module shown in Figure 4 and the UNet structure? It looks like a 4D version of UNet. Why does the FLOPs increase significantly while the number of parameters in E4A decreases? Is this due to the upsampling process?\n\n7. In TAF, after applying 3D, 2D, and 1D global average pooling, the same scale features perform temporal attention. Would cross-attention between different scales yield better results?\n\n8. In the Forecaster, is the condition for prediction learned from past voxels?\n\n9. What does the colon symbol in line 313 mean?\n\n10. The comparisons could be more thorough. For example, it would be helpful to follow the output of the 4D BEV method, like [1], with a 2D to 3D detection head for comparison.\n[1] Bevdet4d: Exploit temporal cues in multi-camera 3D object detection", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The article is well-written and easy-to-follow.\n2. Figure 2 is helpful to understand the effect of OccProphet.\n3. The ablation study is helpful in demonstrating the benefits of each module in OccProphet.\n4. Extensive experimental results on several benchmarks back up the effectiveness of OccProphet.", "weaknesses": "1. Some symbols and design details of the model are unclear.\n2. The analysis of failure scenarios is lacking, but this is not a major concern.\n3. See Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UVRQr82UEN", "reviewer": "Reviewer_uB9U", "summary": "OccProphet is an efficient occupancy forecasting framework for autonomous driving, reducing computational costs by 58%–78% and achieving a 2.6× speedup over existing methods while maintaining high accuracy. Through its Observer, Forecaster, and Refiner modules, OccProphet predicts future 3D occupancy from 2D images, making it suitable for real-time edge deployment.", "questions": "The current baselines lack sufficient comparison methods and evaluation metrics, which are inadequate to demonstrate the superiority of the proposed approach.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "a). OccProphet addresses the computational limitations of previous methods, a crucial improvement for deploying autonomous vehicles on edge devices.\n\nb). The authors’ writing style is clear and concise, effectively conveying the ideas and concepts of 4D occupancy prediction.", "weaknesses": "a) Although the authors have distinctively named the three components as Observer, Forecaster, and Refiner to differentiate them from previous methods, the paper should more clearly highlight the distinctions from the traditional encoder-decoder architecture to better emphasize its contribution.\n\nb) The tripling-attention and reduced-resolution feature aggregation may sacrifice some granularity or detail in the forecasts, possibly affecting the precision of the model in dense scenarios.\n\nc) The paper lacks details on how well the model performs over varying forecasting time horizons, especially under extended timeframes, which are critical in autonomous driving scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6Y4Db5cyoQ", "reviewer": "Reviewer_ZdEq", "summary": "This paper propose a method called OCCPROPHET for efficient occupancy forecasting with camera-only inputs. The proposed framework is consisted of Observer, Forecaster, and Refiner. OCCPROPHET first embeds the sequence of camera images into 3D voxel features. Then Observer applies 4D feature aggregation to gradually reduce the spatial resolution of the 3D features and a tripling-attention fusion strategy on the lower-resolution 3D features to reduce information loss. The Forecaster component forecast the state of the environment based on the feature output from Observer. Finally, Refiner utilize temporal relationship to enhance the quality of the 3D voxel features to generate the final prediction of future occupancy. The key idea is reducing spatial resolution at the very beginning of the network to reduce computational cost during embedding and forecasting. The tripling-attention part take spatiotemporal interactions into account to reduce the information loss and the Refiner part makes use of temporal correspondence to increase the granularity of the prediction. The proposed method makes a good trade-off between efficiency and effectiveness. The extensive experiment results show that OCCPROPHET largely reduce computational cost while achieving slightly better performance.", "questions": "1) Could the authors show experiment results on training for longer with both OCFNet (Cam4DOcc) and OCCPROPHET to see whether the performance improvement stems from early convergence.\n\n2) If not, is there any analysis or explanation about the performance improvement with respect to the information loss?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper propose OCCPROPHET to improve the efficiency of occupancy forecasting algorithm while maintaining and even improving the performance. The key idea is to embed a coarse-level 3D voxel features at the very beginning of the network to reduce computational cost and meanwhile utilize the temporal-spatial relationship to reduce information loss during embedding and forecasting. Finally, in order to guarantee prediction quality, the Refiner part enhance the feature quality with temporal-spatial correspondence and increase the granularity of the prediction. OCCPROPHET provides a better balance between efficiency and effectiveness in occupancy forecasting problem. The experiments are extensive and the results demonstrate both the efficiency and effectiveness of OCCPROPHET.", "weaknesses": "The reviewer is concerned about the effectiveness part of OCCPROPHET. Since OCCPROPHET reduce spatial resolution at the beginning of the network, there should be information loss no matter what strategy OCCPROPHET uses to compensate that. At the same time, temporal-spatial interaction should also be utilized in OCFNet (Cam4DOcc). The reviewer is a bit confused about the better performance OCCPROPHET shows than that of OCFNet (Cam4DOcc). The reviewer is concerned about whether the performance improvement stems from the fact that OCFNet (Cam4DOcc) is underfit and OCCPROPHET converges faster.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4miALyvJhX", "reviewer": "Reviewer_t8d8", "summary": "The paper introduces a novel framework named OccProphet, designed to predict the 3D occupied status of driving environments based on historical 2D images, with a focus on improving efficiency and reducing computational demands during training and inference stages. This is particularly aimed at enhancing the feasibility of deploying such technology on edge agents like autonomous vehicles.The OccProphet framework consists of three lightweight components: the Observer, Forecaster, and Refiner. The Observer extracts spatio-temporal features from 3D data using an Efficient 4D Aggregation with Tripling-Attention Fusion method. The Forecaster and Refiner work together to conditionally predict and refine future occupancy inferences. The paper claims that OccProphet is both training- and inference-friendly, reducing computational costs by 58% to 78% with a 2.6x speedup compared to the state-of-the-art Cam4DOcc method, while achieving 4% to 18% higher forecasting accuracy. The experimental results are demonstrated using the nuScenes, Lyft-Level5, and nuScenes-Occupancy datasets.", "questions": "What is the actual frames-per-second of the model. And is it good enough for deployment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a new framework, OccProphet, which is designed to be efficient and effective for camera-only 4D occupancy forecasting, a critical capability for autonomous driving.\n2. The framework significantly lowers computational requirements by 58% to 78% compared to the state-of-the-art Cam4DOcc, making it more feasible for deployment on edge agents like autonomous vehicles.\n3. OccProphet achieves a relative increase in forecasting accuracy of 4% to 18% over existing methods, which is a substantial improvement in the field of autonomous driving perception.", "weaknesses": "The paper does not give detailed explanations for the design of the three proposed modules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)