Let Occ Flow: Self-Supervised 3D Occupancy Flow Prediction
Authors: Yili Liu
Linzhan Mou
Xuan Yu
Chenrui Han
Sitong Mao
Rong Xiong
Yue Wang
Venue: CoRL 2024 (2024)
Keywords: 3D occupancy prediction
occupancy flow
Neural Radiance Field
TL;DR: the first self-supervised work to joint predict the 3D occupancy and occupancy flow

Abstract: Accurate perception of the dynamic environment is a fundamental task for autonomous driving and robot systems. This paper introduces Let Occ Flow, the first self-supervised work for joint 3D occupancy and occupancy flow prediction using only camera inputs, eliminating the need for 3D annotations. Utilizing TPV for unified scene representation and deformable attention layers for feature aggregation, our approach incorporates a novel attention-based temporal fusion module to capture dynamic object dependencies, followed by a 3D refine module for fine-gained volumetric representation. Besides, our method extends differentiable rendering to 3D volumetric flow fields, leveraging zero-shot 2D segmentation and optical flow cues for dynamic decomposition and motion optimization. Extensive experiments on nuScenes and KITTI datasets demonstrate the competitive performance of our approach over prior state-of-the-art methods.

Let Oce Flow: Self-Supervised 3D Occupancy Flow Prediction
Yili Liu
Linzhan Mou!
Xuan Yu²
Chengrui Han
Sitong Mao
Rong Xiong' Yue Wang
"Zhejiang University
Huawei Cloud Computing Technologies Co., Ltd. {ylliu01, @zju. .nu.cn
Abstract: Accurate perception of the dynamic environment is fundamental task for autonomous driving and robot systems. This paper introduces Let Oce Flow, the first self-supervised work for joint 3D occupancy and occupancy flow predic- tion using only camera inputs, eliminating the need for 3D annotations, Utilizing TPV for unified scene representation and deformable attention layers for feature aggregation, our approach incorporates novel attention -based temporal fusion module to capture dynamic object dependencies, followed by 3D refine module for fine-gained volumetric representation. Besides, method extends differen- tiable rendering to 3D volumetric flow fields, leveraging sero-shot 2D segmenta- tion and optical flow cues for dynamic decomposition and motion optimization. Extensive experiments on nuScenes and KITTI datasets demonstrate the competi- tive performance of our approach over prior state-of-the-art methods. Our project page is available https://eliliu2233. github.io/etooccloo..
Keywords: 3D occupancy prediction, occupancy flow, Neural Radiance Field
Introduction
Accurate perception of the surrounding environment crucial autonomous driving and robotics for downstream planning and action decisions. Recently, vision-based 3D object detection [1, 2] and occupancy prediction [3, 4, 6, 7] have garnered significant attention due to their cost-effectiveness and robust capabilities. Building these foundations, the concept of occupancy flow prediction has emerged to offer comprehensive understanding of the dynamic environment by estimating the scene geometry and object motion within a unified framework [8, 9].
Existing occupancy flow prediction methods [8, 9] rely heavily on detailed 3D occupancy flow an- notations, which complicates their scalability extensive training datasets. Previous research has attempted to address the dependency 3D annotations for occupancy prediction tasks. Some strate- gies utilized LiDAR data to generate occupancy labels [3, 10, 11], but struggle to balance between accuracy and completeness due to LiDAR's inherent sparsity and dynamic object movements. Re- cently, some methods have emerged to enable 3D occupancy predictions a self-supervised manner by integrating the NeRF-style differentiable rendering with either weak 2D depth cues [7] or photo- metric supervision [5, 6). However, existing supe-supe techniques take static world assumption a premise and never consider the motion of foreground objects. This raises question: Can we bridge 2D perception cues enable self supervisd occupancy flow prediction?
To address the aforementioned issues, we introduce Let Occ Flow, a novel method for simulta- neously predicting 3D occupancy and occupancy flow using only camera input, without requiring any 3D annotations for supervision, as shown Figure Our approach employs Tri-perspective VieW(TPV) [11] as unified scene representation and incorporates deformable attention layers to aggregate multi-view image features. For temporal sequence inputs, we utilize an attention-based
'Equal contribution. |Corresponding author.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Previous supervised works 3b Orcupanty Armotation
Occupency Flow Prediction
Occ Flow (Ours)
Imagei
3D Arnotation Supervision
2b Rendering Supervision
Bounding Box Arrotction
zb Opn flowr
Figure I: Let Oec Flow introduces a novel supervised training paradigm for 3D occupancy and occupancy flow prediction. Unlike earlier studies that relied on expensive annotations for 3D occu- pancy flow, our method employs readily available 2D labels train the occupancy flow network.
temporal fusion module to effectively capture sequently, we use 3D refine module for spatial volume feature aggregation and transform the scene representation into fine-grained volumetric fields of SDF and flow.
dependencies of dynamic objects. Sub-
Differentiable rendering techniques are then applied to jointly optimize scene geometry and object motion. Specifically, for geometry optimization, we employ a photometric consistency supervi- sion [6] and optional range rendering supervision [12, 13]. For motion optimization, we leverage semantic and optical flow cues for dynamic decomposition and supervision. These cues are con- venient to obtain from pre-trained 2D models, which have shown superior zero-shot generalization capability in comparison to 3D models. Furthermore, previous studies have demonstrated that self- supervised photometric consistency [14, 15] enables the pre-trained flow estimator to effectively narrow the generalization gap across different data domains. Specifically, we employ preitrained 2D open-vocabulary segmentation model GroundedSAM [16] to derive a segmentation mask of movable objects. In addition, we use optical flow cues from an off-the-shelf 2D estimator [17, 18] offer effective supervision of pixel correspondence for joint geometry and motion optimization.
In summary, main contributions
follows:
1. We proposed Let Oce Flow, the first self-supervised method for jointly predicting 3D occupancy and occupancy flow, by integrating 2D optical flow cues into geometry and motion optimization.
2. We designed novel attention-based temporal fusion module for efficient temporal interaction. Furthermore, we proposed flow-oriented optimization strategy to mitigate the training instability and sample imbalance problem.
3. We conducted extensive experiments on various datasets with qualitative and quantitative analyses to show the competitive performance of our approach.
2 Related Work
3D Occupancy Prediction. Vision-based 3D occupancy prediction has emerged promising method to perceive the surrounding environment. TPVFormer [11] introduces tri terpperseeeeive view representation from multi-camera images. SurroundOce [3] utilizes sparse LiDAR data to generate dense occupancy ground truth as supervision. Rather than using 3D labels, RenderOce [7] employs volume rendering to enable occupancy prediction from weak depth labels. Recent works [4, 5, 6] propose LiDAR-free self- supervised manner by photometric consistency. Despite the progress in static occupancy prediction, there has been comparatively less focus occupancy flow predic- tion-a critical component for dynamic planning and navigation in autonomous driving.
Neural Radiance Field. Reconstructing 3D scenes from 2D images remains a significant challenge in computer vision. Among the various approaches, Neural Radiance Field (NeRF)[19] has gained increasing popularity due to its strong representational ability. To enhance training efficiency and improve rendering quality, several methods/20, 21, 22] have adopted explicit voxel-based represen- tations. Additionally, some approaches [23, 24] introduced NeRF into surface reconstruction by developing Signed Distance Function(SDFF) representation. Further advancements [25, 26, 27, 28]

---

Temporal Fusion Madule
Volumetnic 5bF Field
Occupancy
futi-rome Trages How-toast e. feel
High Resclation Volume
Volumetric Field
2D-to-3D Encoder
3b Occupancy Flaw Prediction
autinike Eooper
Tripprrpar
6y
3D Refine
Modiye
Flow
Terporal- Fused
Volume
Rendering Self Supervision
UDAR Rays (Optene))
Rendered Depth
Vo lakume Randernig
Frome
Fvesesi
Gytied Tlw Price
Revdered Oprica Plan
Frome
kypw
Figure 2: The overall architecture of Let Oce Flow. We employ deformable-attentinn layers to integrate multi-view image input into TPV representation. The temporal fusion module utilizes BEV-based backward-foo forwwrd attention to fuse temporal feature volumes. The 3D Refine Module further aggregates spatial features and upsample the fused volume into high-solution representa- tion. Then we apply two separate MLP decoders to construct volumetric SDF and flow fields, and finally perform self-supervised occupancy flow learning utilizing reprojection consistency, optical flow cues, and optional LiDAR ray supervision via differentiable rendering.
have been made to facilitate scene-level sparse view reconstruction by conditioning NeRF on image features, laying the groundwork for generalizable NeRF architectures.
Dynamic Driving Scene Reconstruction. Recent approaches proposed to combine flow predic- tion with vision-based occupancy network [8, 9] for dynamic scene reconstruction, while they rely heavily on complete 3D occupancy and flow annotations. Previous approach [13] has successfully applied NeRF into self-supervised learning of dynamic driving scene reconstruction, while typi- cally requires expensive per-scene optimization. To the best of our knowledge, we are the very first work to achieve generalizable dynamic driving scene reconstruction in supe-supe manner.
3 Method
The overall framework is shown in Figure 2. In Sec.3.2, we first extract 3D volume features V from multi-view temporal sequences with 2D-to-3D feature encoder. To capture the geometry and motion information, we employ an attention-basse temporal fusion module and a 3D refine module to ensure effective spatial-temporss interaction. In Sec.3.3, we construct the volumetric SDF and flow field using two MLP decoders and perform differentiable rendering to optimize the network in self-supervised manner. Finally in Sec.3.4, we illustrate a flow-oriented two-stage training strategy and dynamic disentanglement scheme for joint geometry and motion optimization.
3.1 Problem Formulation
We aim to predict 3D occupancy On and occupancy flow F, of the surrounding scenes at timestep using a temporal sequence of multi-view camera inputs. Formally, task represented as:
O,, F. = G k. I4
where 9 a neural network, I represents N-view images {1} I?,... ...,, INN] at timestamp t. O, e xxx denotes the 3D occupancy at timestamp f, representing the occupied probability of each Ig 3D volume grid with the value between and And # W x2 denotes the occupancy flow at timestamp t, representing the object motion vector excluding ego- moo-ooooon of each grid in the horizontal direction of 3D space, as described in OceNet [8].

---

Alighment
BFAM
BEV-Volume Fusion
Vi-2011
Volume-fo-BEl Be-BEV
Sy-4)
Aligned Voluma
Sieci
Cose
Skites
Up-esil
Tenpers Prrke
Bp-1)
Peeleg
Torpers handBEy
Paied Volane
Figure 3: Architecture of temporal fusion module. The temporal fusion module consists ego- motion alignment and temporal feature aggregation. While ego ago-motion alignment is employed in voxel space to align temporal feature volumes, BEV-based temporal feature aggregation leverages deformable attention with backward-forward process to achieve temporal interaction.
3.2 2D-to-3D Encoder
2D-to-3D Feature Encoder. Following existing 3D occupancy methods [6, 11], we utilize TPV to effectively construct a unified representation from multi-view image input, which pre-define a set of queries in the three axis-aligned orthogonal planes:
T "r T Tw22, 3], THH "xxxC T E ER Tw2 R Wx2xC
where H, W, Z denote the resolution of the three planes and C represents the feature dimension. We
integrate multi view multi-scale image features F extracted from the 2D encoder backbone [29]
into triplane using deformable cross-attention [30] (DAttn) and feed forward network (FFN):
T
{TP) (psPP-(p))) F)), E P (HW, ZH, WZ
(3)
Then we construct a full-scale 3D feature volume V by broadcasting each TPV plane along the corresponding orthogonal direction and aggregating them by summation up [11].
Temporal Fusion. To effectively capture temporal geometry and object motion, we design tem- poral fusion module to enhance the scene representation further. Specifically, module integrates historical feature volume into the current frame. As depicted in Figure 3, this module comprises two main components: ego-motion alignment in voxel space and Backward- Forward Attention Module (BFAM) in BEV space. To compensate for ego-motion, we first apply feature warping by transforming historical feature volumes to the current frame, according to the relative pose in ego coordinate:
temporal fusion
VE T. E k-
1}
where 1/ E RRHxWxxC is the aligned feature volume in the current local ego coordinate. Pt E x x2 denotes the predefined reference points of current feature volume, Th is the ego pose transformation matrix from t t,, and ( indicates the linear interpolation operator.
Inspired by prior works [31, 32]. compress the aligned feature volume V/2 into series of BEV slices B, E TR. XXCC along the z-axis and employ deformable attention [30] (DAttn) between these BEV slices in the horizontal direction. To effectively capture overall information in the vertical dimension, we derive a global BEV feature B, using an average pooling operator and integrate into the BEV slices. Furthermore, we employ a Backward-Forwee Attention Module (BFAM) to enhance the interaction between temporal BEV features:
B1, Bt, + B DAttn(Br,, (Br,, B,,1)
(5)
where B1,, B,, represents the BEV features adjacent frames t;, t,. and (3 learnable scale factor. In the backward process, 1, 4 while in the forward process, 1n +1
After BFAM, we recover the fused feature volume V by merging fused BEV slices, and utilize the Volume Fusion module to achieve interaction between fused global BEV features B, and fused feature volume V for the final feature volume V':
V' V + a (conv(VV) unsqueeze(B,,
(6)

---

where conv is a x 1 convolution module and a is a sigmoid function.
3D Refine Module. Following the acquisition of the final fused feature volume V we utilize a residual 3D convolution network to further aggregate spatial features and subsequently upsample to refined feature volume Vrefine with high resolution using 3D deconvolutions.
3.3 Rendering-Baser Optimization
Rendering Strategy. Given refined feature volume Vrefine, we employ two separate MLP (O,, θ5} to construct volumetric SDF field S and volumetric flow field F::
S O (Vrsfine)
F = (Vrafine)))
During training, we employ a rendering bastin strategy to concurrentyy optimize the 3D occupancy and occupancy flow without 3D annotations. Following [4, 6], we utilize SDF for geometric repre- sentation, with occlusion-awand rendering as NeuS[23). Details are shown in the supplementary.
Rendering- Based Supervision. Inspired by [4, 5, 6, 25], we utilize a reprojection photometric loss for effective optimization of the volumetric SDF field:
Crepros (x)
Σ wi wrCphoto((4,, KT (TTTITTTTKK---- [x, +. fi)))
(8)
where N denotes the number of points sampled along the ray, x is the sampled pixel on the current image, 7,+1 represents the transformation matrix from timestamp to in ego coordinate, K and T are the intrinsic and extrinsic parameters of cameras, respectively. Additionally, w/, d,, and fa denote the weight, depth, and flow of 7th sampled point. For the photometric loss Lepott, we utilize the combination of LI and D-SSIM loss, following monodepth [33].
Though photometric loss is sufficient for geometric supervision, optimizing object motion jointly re- mains significant challenge. Therefore, we incorporate optical flow cues from an off-the-shed flow estimator [17, 18] offer an additional source of supervision for geometry and motion estimation.
flow (x)
Σ will
" (TK (x, di] + f() x111
where Oc- represents the estimated forward flow map from frame f to +
When LiDAR is available, could combine LiDAR supervision to enhance the geometry. Specif- ically, first independently generate rays from the LiDAR measurement to ensure effective sampling rate in regions with sufficient geometric observations, thereby avoiding occlusion prob- lems from camera extrinsic. Furthermore, we implement a Scale-Invariant loss Lrange [34] between rendered ranges and LiDAR measurements to mitigate the global scene scale influence.
3.4 Floe-tniented Optimization
Due the training instability caused by the inherent ambiguities in jointly optimizing geometry and motion without ground truth labels, and the sample imbalance problem of moving objects and the static background, it becomes imperative to employ an effective flow-oriented optimization strategy.
Two Stage Optimization. To tackle the training instability challenge, we introduce a two-stage optimization strategy. Initially, we optimize the SDF field with the static assumption following the method proposed by SelfOce [6] generate a coarse geometric prediction. In the second stage, we jointly optimize the volumetric SDF and flow field with extra 2D optical flow supervision.
Dynamic Disentanglement. To effectively regularize the motion in static regions and solve the sample imbalance problem, we propose to conduct dynamic disentanglement using dynamic optical flow fd. which subtracted by flow cues fopt and statie flow f computed by rendered depth
1 -==10 popt f = (O. - (KT 'T, TTK [25, d] x)
(10)
5

---

To mitigate incorrect decomposition caused by inaccurate depth estimation in background re-
Oreun Shage
gions (eg. sky and thin structure), we lever- age the zero-shot open-vocabulary 2D seman- tic segmentation model, Grounded-SAr [16]. to extract binary masks for movable foreground objects. With flow error mask thresholded by dynamic flow Sa we could get an accurate dy- namic mask for scene decomposition. Then we can regularize the volumetric flow field by min- imizing the rendered flow map static regions: Lareg 1 Σ welfll..
Graaded
fidernioo
Pynanic Natc
Renebred Dupth
Opticl
Optical Ploy Entinstr
Pln Emror Bick
Figure 4: Our effective dynamic disentanglement scheme provides accurate dynamic mask for joint geometry and motion optimization.
Compared to the static background, dynamic objects such as cars or pedestrians account for a small proportion in driving scenes but hold signif- icant importance for driving perception and planning. To address the imbalance caused by moving objects, we employ a scale factor to adjust the flow loss of dynamic rays based on their occurrence.
flow flow + +-oftow y mim Matl Ma, thre)
(11)
where foo and flow denote the static and dynamic parts of the flow loss Liow calculated by Equation 9, while MaUl and Ma represent the number of all sampled rays and dynamic rays, and thre a threshold hyperparameter.
Static Flow Supervision. Since optical flow provide the pixel correspondence between adja- cent frames, they could be used to supervise the scene geometry effectively. Specifically, utilize the dynamic mask mentioned above to decouple the static regions and then leverage the forward static flow for geometry supervision. To effectively use the abundant training data in self-supervised training, we sample an auxiliary previous frame based on the Gaussian distribution of the distance between the ego coordinates to offer additional backward flow for geometric supervision.
Finally, with the SDF regularization loss Leeiks LH Togmooth, overall loss function is follows:
Lreproj + Nowwo - + Axange Advag futreg + Aest bask + Au L + Asmooth A smooth
(12)
4 Experimental Results
4.1 Self-supervised 3D Occupancy Prediction
We conduct experiments on 3D occupancy prediction using the SemanticKITTI [35] dataset without LiDAR supervision and employ RayloU [36] as evaluation metrics. We also evaluate the results of monocular depth estimation using Abs Rel, Sq Rel, RMSE, RMSE log metrics.
As reported in Table 1, our method sets the new state-of-the-art for 3D occupancy prediction and depth estimation tasks without any form of 3D supervision compared with other supervised and self-supervised approaches. Thanks to our effective spatial-temporal feature aggregation and the integration of optical flow cues for supervision, our method greatly enhances the geometric repre- sentation capabilities, compared to other rendering-based methods [5, 6, 25).
Table 1: 3D occupancy prediction and depth estimation on SemanticKITTI [35] dataset. MonoScene [38] is trained with 3D supervision and cannot render monocular depth. We adapt the OccNeRF [5] to the SemanticKITT [35] dataset based on the official implementation.
Occupancy
Depth
Method
RayloU 2n 4m
RayloU?
Abs Roll
Sq RelL
RMSEL
RMSE log.
MonoScene [38]
26.00
36.50
49.09
37.19
SceneRF [25]
20.65
35.74
56.35
37.58
0.146
1.048
5.240
0.241
OccNeRF [5]
23.22
40.17
62.57
41.99
0.132
1.011
5.497
0.233
SelFOce [6]
23.13
35.38
50.31
36.27
0.152
1.306
5.386
0.245
Ours
28.62
45.60
66.95
47.06
0.117
0.817
4.816
0.215

---

Deptis
Occupancy
Occuponcy
Figure 5: Visualization results for depth estimation, 3D occupancy and occupancy flow pre- diction on the KITTI [37] dataset. Our method can predict visually appealing depth maps, fine- grained occupancy, and accurate dynamic decomposition and motion estimation.
4.2 Self-supervised Occupancy and Occupancy Flow Prediction
We conduct experiments 3D occupancy and occupancy flow prediction the KITTI-MOT [37] and nuScenes [39] datasets. We use OccNeRF [5] and RenderOcc [7] as rendering based baselines. To compare with OccNeRF, we employ concatenation&ccom temporal fusion module as in Rende- rOcc. We also add flow head to these two methods with extra 2D optical flow labels as supervision for fair comparison. To compare with OceNet [8], we use their official implementation reproduce the results with GT 3D occupancy flow labels from OpenOce [8] as supervision.
Due to the absence of occupancy flow annotations, we report the occupancy flow evaluation metrics on the KITTI-MOT dataset [37] using LiDAR-projecto depths and generated flow labels. More details are presented in the supplementary materials. On the nuScenes [39] dataset, with annotated occupancy flow ground truth, we report the RayloU [36] and mAVE metrics as in OccNet [8].
Table 2: Self-supervised occupancy and occupancy flow prediction on KITTI-MOT [37].
3D Occupancy & Occupancy Flow
Method
Supervision
DEL
EPE]
DG.
EPE,FG]
D_5%.
FL10%
SF 10900
OccNeRF-C*15]
3,547
6.986
7,976
10.445
0.204
0.272
0.306
Ours-C:
3.431
3.529
6.429
7.635
0.215
0.084
0.118
OccNeRF-L-
11,284
0.309
Ours-L
C+L
2.610
3.488
3.106
7,510
0.154
0.083
0.105
Table 3:
3D occupancy and occupancy flow prediction on nuScenes [39].
3D Occupancy & Occupancy Flow
Method
Supervision
RayloU 2n tu
RayloU
mAVEL
OccNet [8]
3D
29.28
39.68
50.02
39.66
1.61
OccNoRF-C* [5]
C
9.93
19.06
35.84
21.61
1.53
Ours-C
C
17.49
28.52
44.33
30.12
1.42
RenderOcc*
T. 20.27
32.68
49.92
36.67
1.63
OccNcRF-1.
C+I.
16.62
29.25
49.17
31.68
1.59
We report the results in Table and Table 3, C and denote Camera and Lidar supervision. Compared to rendering- baased methods [5, 7], our approach performs much better both KITTI- MOT [37] and nuScenes [39], owing to our effective temporal fusion module and flow-oriented optimization strategy. Compared to 3D supervised OccNet [8], our approach achieves comparable performance on nuScenes [39], validating the effectiveness of our self-supervised training paradigm.
4.3 Ablation Study
We systematically conduct comprehensive ablation analyses the temporal fusion module, opti- mization strategy, and static flow supervision to validate the effectiveness of every component of our method. All experiments are trained for 16 epochs, and without LiDAR supervision by default.

---

Table 4: Temporal fusion module ablation on KITTI-MOT [37] dataset.
Temporal Fusion Module
DEL
EPEL
DELFGL
EPE_FGL
D
FL10%
SFL10%
Params.
Concatenation&:1 3D Cony.
3,515
3,638
6.423
7.743
0.223
0.090
0.126
11.39M
BEV Pooling & Forward Attn.
3,600
3,763
6.339
7.971
0.228
0.091
0,124
5.21M
BEV Pooling BFAM
3.462
3,583
6.409
7.673
0.211
0.089
0.122
6.62M
Ours
3.431
3,529
6.429
7.635
0.215
0.084
0.118
6.62M
Table Optimization strategy ablation on KITTI-MOT [37] dataset. Since the absence of two- stage optimization brings instability during training, we use extra LiDAR ray supervision here.
Two Stage
Dynamic
DEL
EPFL
DRUGG
EPR_EG
D_58.L
FLJ0SL
SF.10%
Optimization
Disentanglement
3.760
4,403
4.194
8.853
0.258
0.121
0.152
3.064
3.633
3,600
7,966
0.191
0.089
0.114
2,610
3,488
3,106
7,510
0.154
0.083
0.105
Table 6: Static flow supervision ablation on KITTI-MOT [37] dataset.
Backward Static Flow Supervision
Forward Static
DEL
EPEL
DELFGI
EPE.FG]
D.5%
FL.10%
SF.10%
Flow Supervision
0,099
0.149
0.123
3.431
3.529
6,429
7.635
0.215
0.084
0.118
Temporal Fusion. We report the results of different temporal fusion settings in Table 4. Compared to the 3D Convolution module in [7], our method achieves better performance with lower memory cost due to our efficient BEV-based attention mechanism and the effective long-distance temporal teraction. Moreover, our BFAM module outperforms the forward-only attention method introduced in [8] due its enhanced temporal interaction capabilities. Additionally, Our method effectively incorporates local and global information in the height dimension compared to BEV-only attention.
Optimization Strategy. We investigate the effect of our optimization strategy Table It is shown that both two-stage optimization and dynamic disentanglement scheme benefit mitigating the impact of the ambiguities for joint optimization of scene geometry and object motion.
Static Flow Supervision. We verify the efffcctiveness of static flow supervision in Table 6. As reported, extra pixel correspondence priors from forward and backward static flow can effectively improve the quality of scene geometry, thus making occupancy flow better.
5 Discussion
Limitation and Future work. Although we use temporal sequence input to better exploit the his- torical information, our model cannot completely handle the occlusion problem due to the inherent rendering-basee limitation. Subsequent research could investigate long-term occupancy flow mod- cling and solutions leverage the temporal sequence supervision to scale up the visible range of perspective. addition, the accuracy of occupancy flow prediction relies on the quality of optical flow cues. Future work can pay attention the improvement of the flow supervision quality. Finally, our occupancy flow prediction does not explicitly enforce consistency within instances, and future work may explore to integrate instance perception into occupancy flow prediction.
Conclusion. We introduced Let Oce Flow, the first self-supervised method for vision-based 3D occupancy and occupancy flow prediction. Utilizing the effective temporal fusion module and flow- oriented optimization,
approach effectively captures dynamic object dependencies, thus enhanc- ing both scene geometry and object motion. Furthermore, our method extends the differentiable rendering to the volumetric flow field for self-supervised learning. Consequently, our work paves the way for future research into more efficient and accurate self-supervised learning frameworks in dynamic environment perception.

---

Acknowledgments
We sincerely thank the anonymous reviewers for their helpful comments in revising the paper. This work was supported in part by the National Nature Science Foundation of China under Grant 62373322, and in part by Zhejiang Provincial Natural Science Foundation of China under Grant No. LD24F030001.
References
[1] Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and J. Dai. Bevformer: Learning bird's- eye-view representation from multi-camera images via spatiotemporal transformers. arXiv preprint arXiv:2203.17270, 2022.
[2] Y. Li, Z. Ge, G. Yu, J. Yang, Z. Wang, Shi, J. Sun, and Z. Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv preprint (xXi::2206.109)) 2022.
[3] Y. Wei, L. Zhao, W. Zheng, Z. Zhu, J. Zhou, and J. Lu. Surroundoce: Multi-camera 3d occupancy prediction for autonomous driving. arXiv preprint arXiv:2303.0953: 2023.
[4] W. Gan, N. Mo, H. Xu, and N. Yokoya. simple attempt for 3d occupancy estimation in autonomous driving. arXiv preprint 2rXiv:2303.10077 2023.
[5] C. Zhang, J. Yan, Y. Wei, Li, L.. Liu, Tang, Y. Duan, and J. Lu. Occnerf: Advancing 3d occupancy prediction in lidar-free environments. arXiv preprint arXiv:2312.09243, 2023.
[6] Y. Huang, W. Zheng, B. Zhang, J. Zhou, and Lu. Selfoce: Self-supervised iision-base 3d occupancy prediction. arXiv preprint arXiv:2311.1275: 2023.
[7] M. Pan, J. Liu, R. Zhang, Huang, X. Li, L. Liu, and S. Zhang. Renderoce: Vision-centric 3d occupancy prediction with 2d rendering supervision. arXiv preprint arXiv:2309.09502, 2023.
[8] W. Tong, C. Sima, T. Wang, L. Chen, S. Wu, H. Deng, Y. Gu, L. Lu, P. Luo, D. Lin, et al. Scene as occupancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 8406-8415, 2023.
[9] J. Li, X. He, C. Zhou, X. Cheng, Y. Wen, and D. Zhang. Viewformer: Exploring spatiotem- poral modeling for multi-view 3d occupancy perception via view-guided transformers. arXiv preprint arXiv:2405.04299 2024.
[10] Chen Minl Dawei Ziao,Liang Xiao,Yiming Nie and Bin Dai. Oce-bev: Multi-camera unified pre-training via 3d scene reconstruction. arXiv preprint, 2023.
[11] Y. Huang, W. Zheng, Y. Zhang, J. Zhou, and J. Lu. Tri-perspective view for vision-based 3d semantic occupancy prediction. arXiv preprint arXiv:2302.07817. 2023.
[12] Z. Yang, Y. Chen, Wang, S. Manivasagam, W.-C. Ma, J. Yang. and R. Urtasun. Unisim:
A neural closed-loop sensor simulator. In CVPR, 2023.
[13] J. Yang, B. Ivanovic, O. Litany, X. Weng, S. W. Kim, B. Li, T. Che, D. Xu, S. Fidler, M. Pavone, and Y. Wang. Emernerf: Emergent spatial-temporal scene decomposition via supervupervi arXiv preprint 1.XXi:2331100000 2023.
[14] R. Jonschkowski, A. Stone, J. T. Barron, A. Gordon, K. Konolige, and A. Angelova. What matters in unsupervised optical flow. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16, pages 557-572. Springer, 2020.
[15] Y. Mao, B. Shen, Y. Yang, K. Wang, R. Xiong, Y. Liao, and Y. Wang. ı-dba: Neural implicit dense bundle adjustment enables image-only driving reconstruction, 2024.

---

[16] T. Ren, S. Liu, A. Zeng, J. Lin, K. Li, H. Cao, J. Chen, X. Huang, Y. Chen, F. Yan, et al. Grounded sam: Assembling open-world models for diverse visual tasks. arXiv preprint WiXi:2401.141522 2024.
[17] H. Xu, J. Zhang, J. Cai, H. Rezatofighi, F. Yu, D. Tao, and A. Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[18) N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker: It is better to track together. arXiv:2307.07633 2023.
[19] B. Mildenhall, P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[20] Fridovich-Kar Keil, A. Yu, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radi- ance fields without neural networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5501-5510, 2022.
[21] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a mul- tiresolution hash encoding. ACM transactions on graphics (TOG), -154):1-15, 2022.
[22] C. Sun, M. Sun, and H. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In CVPR, 2022.
[23] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv preprint arXiv: 106. 10689, 2021.
[24] R. Liang, J. Zhang, H. Li, C. Yang, Y. Guan, and N. Vijaykumar. Spidr: Sdf-based neural point fields for illumination and deformation. arXiv preprint arXiv:2210.08399, 2022.
[25] A.-Q. Cao and R. de Charette. Scenerf: Self-supervised monocular 3d scene reconstruction with radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9387-9398, 2023.
[26] Wimbauer, N. Yang, C. Rupprecht, and D. Cremers. Behind the scenes: Density fields for single view reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9076-9086, 2023.
[27] D. Charatan, Li, A. Tagliasacchi, and V. Sitzmann. pixelsplat: 3d gaussian splats from image pairs for scalable generalizable 3d reconstruction. arXiv preprint arXiv:23/2./2337 2023.
[28] Y. Chen, H. Xu, C. Zheng, B. Zhuang, M. Pollefeys, A. Geiger, T.-J. Cham, and J. Cai. Mvsplat: Efficient 3d gaussian splatting from sparse multi-view images. arXiv preprint arXiv:2403.1 14627, 2024.
[29] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie. A convnet for the 2020s. In Proceedings of the JEEE/CVF conference on computer vision and patlern recognition, pages 11976- 2022.
[30] 7. Xia, X. Pan, Song, L. E. Li, and G. Huang. Vision transformer with deformable attention. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4794 4803, 2022.
[31] Y. Zhang, Z. Zhu, and D. Du. Occformer: Dual-path transformer for vision-based 3d semantic occupancy prediction. arXiv preprint arXiv:2304.053116 053166, 2023.
[32] Z. Xia, Z. Lin, X. Wang, Y. Wang. Y. Xing, S. Qi. N. Dong, and M.-H. Yang. Henet: Hybrid encoding for end-to-end multi-task 3d perception from multi-vien view cameras. arXiv preprint axXiv2404.02557, 2024.
10

---

[33] C. Godard, O. Mac Aodha, M. Firman, and G. Brostow. Digging into self-supervised monoc- ular depth prediction. October 2019.
[34] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from single image using multi-scale deep network. Advances in neural information processing systems, 27, 2014.
[35] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall. Se- mantickiti: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9297-9307, 2019.
[36] H. Liu, Y. Chen, H. Wang, Z. Yang, 'T. Li, J. Zeng. L. Chen, H. Li, and L.. Wang. Fully sparse 3d occupancy prediction. arXiv preprint arXiv:2312.17116 2024.
[37] A. Geiger, P. Lenz, and R. Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference computer vision and pattern recognition, pages 3354-3361. IEFE, 2012.
[38] A. Cao and R. De Charette. Monoscene: Monocular 3d semantic scene completion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3991-4001, 2022.
[39] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[40] K. Tian, Y. Jiang Q. Diao, C. Lin, L. Wang, and Z. Yuan. Designing bert for convolutional networks: Sparse and hierarchical masked modeling. arXiv preprint arXiv:2301.03880, 2023.
[41] T.-Y. Lin, Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. Feature pyramid networks for object detection, 2017.
[42] Z. Li, Z. Yu, D. Austin, M. Fang, Lan, J. Kautz, and M. Alvarez. FB-OCC: 3D occupancy prediction based on forward-backwart view transformation. arXiv:2307.01992 2023.
[43] A. Dosovitskiy, P. Fischer, E. Ilg. P. Hausser, C. Hazirbas, V. Golkov, Van Der Smagt, D. Cremers, and Brox. Flownet: Learning optical flow with convolutional networks. In Proceedings of the IEEE international conference on computer vision, pages 2758-2766, 2015.
[44] N. Mayer, E. Ilg. P. Hausser, P. Fischer, D. Cremers, A. Dosovitskiy, and T. Brox. A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4040 4048, 2016.
[45] R. Jonschkowski, A. Stone, T. Barron, A. Gordon, K. Konolige, and Angelova. What matters in unsupervised optical flow. In Computer Vision-ECCV 2020: 16th European Con- ference, Glasgow, UK, August 28-28, 2020, Proceedings, Part I1 16, pages 557-572. Springer, 2020.
[46] C. Godard, O. Mac Aodha, M. Firman, and G. Brostow. Digging into self-supervised monoc- ular depth estimation. In Proceedings of the IEEF/CVF international conference on computer vision, pages 3828-3838, 2019.
[47] Y. Wei, L. Zhao, W. Zheng, Z. Zhu, Y. Rao, G. Huang. J. Lu, and J. Zhou. Surrounddepth: En- tangling surrounding views for self-supervised multi-camera depth estimation. In Conference on Robot Learning, pages 539-549. PMLR, 2023.
[48] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe, Unsupervised learning of depth and ego- motion from video. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1851-1854 2017.

---

Supplementary Material
Our supplementary material includes dataset details, implementation details, evaluation metrics, and additional visualization results.
A Dataset Details
The SemanticKITTI [35] dataset is a large-scale dataset based on the KITTI Vision Bench- mark [37], which includes automotive LiDAR scans voxelized into 256 x 256 x 32 grids with 0.2m resolution for 22 sequences. The dataset contains 28 classes with non-moving and moving objects. In our experiments, we only use RGB images captured by cam2 as input from the KITTI odometry benchmark and follow the official split of the dataset, with sequence 08 for validation and other sequences from O to 10 for training.
The KITTI-MOT [37] dataset is based on the KITTI Multi-Object Tracking Evaluation 2012, which consists of 21 training sequences and 29 test sequences. It holds stereo images from two gewarddfacing cameras and LiDAR point clouds. We selected all the training sequences except se- quences where the ego-car is stationary or contains only a small number of moving objects(sequence 0012, 0013, and (017) for training, and select testing sequences 0000-0014 for validation. We also apply LiDAR data and pseudo-optical flow coarsely select dynamic frames for higher sampling rate in training process.
The nuScenes [39] dataset consists of 1000 sequences of various driving scenes of 20 seconds duration each and the keyframes are annotated at 2Hz. Each sample includes RGB images collected by surround cameras with 360 horizontal FOV and LiDAR point clouds from 32-beam LiDAR sensors. The 1000 scenes are officially divided into training, validation, and test splits with 700, 150, and 150 scenes, respectively.
B Implementation Details
B.1 Model Architecture
We adopted ConvNeXt-Base with SparK [40] pretraining the 2D image backbone and four-level FPN [41] to extract image features. We utilize TPV [11] uniformly divided to represent cuboid area for multi-view feature integration, i.e. [80, 80, 6.4] meters around the ego car for nuScenes [39] and [51.2, 51.2, 6.4] meters in front of the ego car for SemanticKITTI [35] and KITTI-MOT [37]. Considering the computational cost of the subsequent temporal fusion module, we use half of the output occupancy resolution for TPV grid cell, i.e. 0.8 meters for nuScenes and 0.4 meters for SemanticKITTI and KITTI-MOT, respectively.
For the input temporal sequence, utilize 2 previous frames for temporal fusion module. Our 3D refine module includes three-layer residual 3D convolution block and 3D FPN block for spatial feature integration. To upsample the volume into the output occupancy resolution, we use deconvolution module as FBOCC [42]. We adopt separate two-layer MLPs (O,, }5} as decoders to construct volumetric fields for SDF and flow.
B.2 2D Flow Estimation
For occupancy flow prediction in KITTI-MOT [37] dataset, we leverage the flow estimation model of Unimatch [17] trained on FlyingChairs [43], FlyingThings3D [44] and fine-tuned on KITTI [37] to predict optical flow maps for supervision directly. Note that the Unimatch model is trained a su-
pervised manner, we provide an unsupervised flow fine-tuning strategy following [45] when adapted different driving scenes. Specifically, we fine-tune the Unimatch model utilizing unsupervised flow techniques including stopping the gradient at the occlusion mask, encouraging smoothness before upsampling the flow field, and continual self-supervision with image resizing.
12

---

As for the nuScenes [39] dataset, the 3D occupancy flow ground truth data is only available at 2Hz keyframes so it is challenging for optical flow estimation. Thus we employed tracking-based flow estimation strategy using CoTracckrVV2 [18]. We aim to obtain accurate optical flow between two keyframes by including the non-keyframe sequences to capture the long-term track. Specifically, we first use open-vocabulare 2D segmentor GroundedSAM[ to predict dynamic foreground semantic segmentation. Then take the adjacent keyframes and the non-keyframe sequences between the two keyframes video input and conduct dense track to capture the long-term motion dependency in the masked regions. Once we have the initial pixel coordinates in the first keyframe and the corresponding pixel coordinates in the next, we subtract the two and get the final optical flow map in the masked regions. We take this optical flow map with the dynamic foreground mask as pseudo- flow labels for occupancy flow prediction.
B.3 Training Settings
The resolution of the input image is 512x1408 for nuScenes [39], 352x for SemanticKITTI [35], and 352x1216 for KITTI-MOT [37]. For loss weight, we set À flow = x 10-3 Aesk = 0.1, Adreg = 0.1, if present, and weights for the edge Aedge and the LiDAR Atdar losses are 0.02 and 0.2 re- spectively, During training, we adopt the AdamW optimizer with an initial learning rate le-4 and weight decay of 0.01. We use the multi-step learning rate scheduler with linear warming up in the first 1k iterations. We train our models with total batch size of 8 on 8 NVIDIA A100 80GB GPUs with 8 epochs for the first stage, and 16 epochs for the second stage. Experiments on Se- manticKITTI [35] and KITTI-MOT [37] take less than one day, while experiments on nuScenes [39] finish within two days.
B.4 Temporal Fusion Module
Algorithm provides the pseudo-code of our proposed temporal fusion module. In the BEV fusion process, we employ a Backward Forward Attention Module (BFAM) with deformable attention (DAttn) to fuse the BEV features from two adjacent frames. To illustrate our algorithmic process clearly, we visually demonstrate our temporal fusion module in the demo video.
Algorithm Pseudo-code for Temporal Fusion Module
1: Input: A temporal sequence of Voxel features (V- (V-(n-1):-- V-1, Vo). Vo represents Voxel feature of the current frame and V-; corresponds to the i-th frame before V.. sequence of transformation matrix of ego coordinates (T' (I-(m-1)):-.. T,,,,, To).
2: for i = 0 to n do
3: V_ EggoMotionAAlgnmeent(
4: B < MeanPoolingg(V.))
5: B_, + Concatenaae(VolumeTTTiiisss
> temporal volumes to BEV slices
6: end for
7: for i 0 to - do
8: B_(++1) t B-(i+1) + 3 DAttn( (1+1); (B B((t)))))
> backward process
9: end for
10: for i= n to
11: B_, B-; + 8- DAttDAt B_ (B_ (i+1)) B-i)) (,,))
D forward process
12: end for
V, Bg SlicesTo Volume (B)
V' +- Volumeeumoo Bg))
15: return 5"
B.5 Differentiable Rendering
In the rendering stage, conduct uniform sampling of N points P {pilé 1,..,N) along the ray and apply a tri-linear interpolation operation efficiently compute the SDF values for each point
13

---

from the volumetric SDF field [22]. Furthermore, the unbiased rendering weights can be calculated by wy = [T 0g), with og denoting the opacity value proposed by NeuS [23]:
($(s:) (((++1) 0)
a; = max
(13)
((s))
where (xx) is sigmoid function ((x) (1 + e ( with temperature coefficient Ç. Let d, and fi denote the depth and flow of the i-th point, we can calculate the rendered depth d and flow f of the ray by:
N
N widi, f Σ wifi
(14)
i=1
B.6 Regularization.
Due to the explicit physical interpretation of SDF representation, regularization can be effectively applied to the volumetric SDF field to prevent over-fitting with limited supervision. Initially, an eikonal term employed for each sample point along camera and LiDAR rays to promote the establishment of valid SDF field:
Cas (1-1
(15)
where VS denotes the pre-computed gradient field of the volumetric SDF grid. Additionally, also utilize Hessian loss Lu on the second-order derivatives of the SDF grid, following the ap- proach of SelfOcc [6]. Furthermore, an edge-aware smoothness loss Lemooth is employed following monodepth2 [33] to enforce neighborhood consistency of rendered depths and flows.
C Evaluation Metrics
C.1 Depth Estimation Evaluation Metrics
Following [46, 47,48) 48], we use evaluation metrics for self-supervised depth estimation as follows:
1
Abs Rel:
d d'1 ld',
Sq Rel:
Σ
d d'1 /d"
(16)
T
T
DET
dET
RMSE:
d d
RMSE log:
Σ der
logg - log d
(17)
dET
where d and d' indicate predicted and ground truth depths respectively, and T indicates all pixels on the depth map. We calculate metrics for depth values in the range of [0.1, 80] meters using 1:2 resolution against the raw image.
C.2 3D Occupancy Prediction Evaluation Metrics
Previous approaches [6, 25, 38] utilize the intersection over union (IoU) as the evaluation metrics of 3D occupancy prediction on SemanticKITTI [35] dataset. However, according to our experiment results, the penalty overly strict in evaluating the reconstruction quality effectively. As illustrated in Figure 6, we observed that rendering-
methods [6. 25] often generate true positive (TP) predictions(marked in green the figure) distributed on the ground or in invisible regions below the ground. This prevents effective evaluation of reconstruction details, deviation of one voxel will lead to an IoU of zero. Furthermore, rendering- based methods tend to predict a thick surface
14

---

due to the absence of supervision in invisible regions, causing a large number of false positive (FP)
meaked red in the figure) and a significant reduction of the precision metric.
Input Image
SelfOce (loU)
Ours(IoU)
Occupancy(Ours)
SeltOce (RayIoU)
Ours (RayIoU)
Figure 6: The comparison of IoU and RayloU measurement results in 3D occupancy prediction.
To demonstrate the advanced occupancy prediction quality of our approach, we employ Ray-based IoU (RayloU) [36] as our evaluation metric and re-evaluate other comparative methods using their open-source checkpoint models. Initially, we generated LiDAR rays with origins sampled from the sequence, and then compute the travel distances of each LiDAR ray before intersecting with the occupied voxels. A query ray is classified predicted depth and ground-truth depth is less than pre-defined threshold (Im, 2m, and 4m). We finally average over distance thresholds and compute the mean RayloU metric.
positive(TP)) when the I.1 emror between the
As is shown Figure 6, the RayloU metric reflected the superiority our method on the detailed prediction of the tree and bicycles compared SelfOcc.
C.3 Occupancy Flow Prediction Evaluation Metries
On the KITTI-MOT [37] data, due to the absence of ground-truth occupancy flow labels, we conduct the evaluation following the scene flow benchmark of KITTI dataset [37]. Specifically, we use LiDAR-projected depth and pseudo 2D optical flow cues to generate disparity and optical flow labels. And we computed end-point error of rendered and ground-truth disparity (DE) and optical flow (EPE). D_5% represent the percentage of disparity outliers with end-point error smaller than 4 pixels or 5% of the ground-trufi disparity. FI.10% is the percentage of optical flow outliers with end-point error smaller than 8 pixels or 10% of the ground-truth flow labels. And SF_10% is the percentage of scene flow outliers (outliers in either D 10% or Fl 10%). To better evaluate the scene flow in foreground regions, we also reported the foreground disparity error (DE_FG) and optical flow error (EPE_FG) by leveraging the semantic mask obtained from GroundedS [16].
On the nuScenes [39] dataset, since have the ground-truth occupancy flow labels, use Ray- based loU (RayIoU), and absolute velocity error (mAVE) for occupancy flow, following the evalu- ation metrics of Occupancy and Flow track for CVPR 2024 Autonomous Grand Challenge.
For RayloU measurement, we perform evaluation on the OpenOce benchmark [8] as introduced in subsection C.2. We use mAVE to indicate the velocity errors for a set of true positives (TP) with threshold of 2m distance. The absolute velocity error (AVE) defined for classes ('car', 'truck', 'trailer', 'bus", "construction vehicle', 'bicycle", 'motorcycle', 'pedestrian') in m/s.
D Additional Ablative Studies
D.1 Flow Supervision
To demonstrate the effectiveness to introduce 2D optical flow supervision into approach, conducted supplementary ablation by dropping the flow supervision.
15

---

Dropping flow supervision would result in zero flow predictions in dynamic regions. The ambiguity between camera and object motion makes the photometric loss insufficient for jointly predicting ac- curate geometry and motion. This challenge is even more pronounced occupancy-level geometry and motion prediction.
As shown in Table 7, while depth-related metrics (DE, DE FG, D 5% cxhibit minimal variance, there is notable increase in the End-Point Error of foreground objects (EPE_ _G), the optical flow outlier rate(Fl)10%) and the scene flow outlier rate underscoring the importance of flow supervision in our model.
Table 7: Static flow supervision ablation on KITTI-MOT [37] dataset.
Optical Flow Supervision
EPE4
PGJ
EPE RG1
5%
10%
2,447
3.649
2,936
9,663
0.149
0.114
0.139
2.610
3,488
3.109
7,510
0.154
0.083
0.105
D.2 Optimization Strategy on nuScenes
To demonstrate the results our ablations on different datasets, we conducted a supplementary ab- lation by employing nuScenes with 256 x 704 image resolution and a half-epoch setting (8 epochs). This additional analysis serves to underscore the effectiveness of our method.
As demonstrated in Table 8, the ablation results on nuScenes align well with those on KITTI-MOT. The joint optimization of geometry and motion without two-stage optimization results in poor geo- metric performance due to the inherent ambiguities of ego and object motion. The average velocity error (mAVE) only accounts for true positive predictions, thus underestimating our full model by disregarding incorrect occupancy predictions in dynamic regions. Additionally, Our proposed dy- namic disentanglement strategy enhances the performance by introducing geometric regularization of static regions and addresses the imbalance problem.
Table 8: Optimization strategy ablation on nuScenes [39] dataset. Since the absence of two-stage optimization brings instability during training, we use extra LiDAR ray supervision here.
Two Stage
Dynamic
Occupancy Occupancy Flow
Optimization
Disentanglement
RayloUtn, an 4uf
RayloUt
MAVEL
10.57
21.05
41.15
24.26
1.36
16.44
30.53
50.51
32.49
1.59
22.95
36.82
54.00
37.92
1.56
E Visualization Results
E.1
Depth Estimation
Figure shows the qualitive comparison of depth estimation on the SemanticKITTI [35] validation set. The visualization results illustrate the successful prediction of detailed and accurate depth by method compared to other rendering-based approaches due to our effective temporal fusion module and optical flow supervision.
E.2 3D Occupancy Prediction
Figure shows the qualitive comparison of 3D occupancy prediction on the SemanticKITTI [35] validation set. occupancy are color-coded according to the height of the voxels. Our method achieves precise occupancy prediction for thin structures such as poles, trees, and cyclists. Also, it provides smooth predictions for cars and road surfaces compared to other supervised [38] and self-supervised [5, 6, 25] methods.
16

---

We provided more visualization results of depth estimation, novel view depth synthesis, and 3D occupancy prediction to illustrate the superiority of our method, As is shown in Figure 9, our method exhibited strong performance across these three tasks, which is due to the accurate prediction of visible surfaces. In addition, our method with effective temporal fusion can provide occlusion perception beyond the visible surface, thus generating high-quality novel-view depth estimations and global-view occupancy predictions.
E.3 Oceupancy Flow Prediction
Figure and Figure 11 shows the visualization results of the depth estimation and occupancy flow prediction tasks on the KITTI-MOT [37] and nuScenes [39] validation set. For flow visualization, the color and brightness represent the motion direction and magnitude respectively. Our proposed method can simultaneously provide accurate 3D occupancy and occupancy flow prediction.
Input Image
SelfOce
OccNeRF
Figure 7: Qualitative comparison for self-supervised depth estimation with other baselines on the SemanticKITTI [35] validation set.
Input Image
MonoScene
SceneRF
OccNeRF
SelfOcc
Ours
Input Image
MonoScene
SceneRF
OccNeRF
SelfOcc
Ours
Figure 8: Qualitative comparison for 3D occupancy prediction on the SemanticKITTI [35]
validation set. The red bounding box shows the most noticeable part.
17

---

Imput Emage
Depth (laput View)
Depth (K-2m, Yam-10 View)
Occupancy (Input View)
Occupancy (Global View)
Figure 9: Visualizations of depth estimation, novel view depth synthesis, 3D occupancy pre- diction on the SemanticKITTI [35] validation set. (X+2m) means moving +2 meters along the x-axis the LiDAR coordinate, and (Yaw+109) means turning left for 10%.
Inpet Image
Rendered Depti
3D Occupancy
3D Occupancy (UV)
Occupancy Flow
Input Image
Rendered Depti
3D Occupancy
Occupancy How
Uccupancy How (GV)
Figure 10: Visualizations of depth estimation, 3D occupancy, and occupancy flow prediction on the KITTI-MOT [37] validation set. GV indicates the global view.
FRONT LEFT
FRONT
FRONT RIGHT
BACK LEFT
BACK
BACK RIGHT
Figure II: Visualizations of depth estimation, 3D occupancy, and occupancy flow prediction on the nuScenes [39] validation set. We show the six input surrounding images in the first row and the estimated depth from the corresponding views in the second row. The third and fourth rows demonstrate the predicted 3D occupancy and occupancy flow results separately.
18


Reviews:
[{"review_id": "27rRqAlr95", "reviewer": "Reviewer_hWju", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Proposed a model that jointly predicts 3D occupancy and occupancy flow from 2D optical flow supervision.", "summary_of_recommendation": "Recommended a weak reject due to its inconsistency in evaluation and lack of analysis to dissect the contribution. First, it is hard to determine which level of performance it has based on the provided evaluation.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "In addition to the review, I have some questions:\n- I wonder if there is a reason not applying BFAM to other methods in Table 3. It can be a good ablation with different modules and fair comparison.\n- It was hard to understand what static flow supervision does. Does it provide 3D correspondences between adjacent frames? First, it does not sound like self-supervision; this is lifted supervision from externally computed 2D optical flow as a pseudo-ground-truth. Second, if the ego motion alignment is possible, I think a transformation can recover the static motion.\n- Is it possible to drop the flow supervision? It provides good understanding of how good the true self-supervised model is or how effective 2D flow supervision is.\n- Is there any reason that reprojection loss does not have the auto-encoding part? Adding a color prediction will enable auto-encoding.\n- Have you tried to predict the static transformation to match the static part of the occupancy flow and the transformed occupancy? It may simplify the model as the flow prediction becomes residual flow prediction. Also, it can be used as an extra training signal to enforce the consistency between the occupancy and the flow.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "93rTZKoFAK", "reviewer": "Reviewer_Dnni", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a novel 3d occupancy flow prediction network, leveraging zero-shot 2d segmentation from pre-trained foundation models. SOTA results are demonstrated on multiple datasets.", "summary_of_recommendation": "Potentially good paper, but i am concerned about experimental settings that makes it hard to compare with previously reported results.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 3, "questions_for_rebuttal": "I would be open to increasing my scores if my concerns around experimental results above were adequately addressed.\n\n----\nAfter rebuttal I increased my score from \"weak reject\" to \"weak accept\"", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bcf1wtL6k6", "reviewer": "Reviewer_PYze", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a novel self-supervised 3d occupancy and flow prediction architecture. The model creates a “Temporal-Fused Volume” by geometrically aligning previous frames, processing with a Backward-Forward Attention Module on BEV compressed representations of the feature volumes and recombining with the original aligned volume. This is refined with a 3d convolution and then rendered with an MLP as a signed-distance-function or flow volume. A combination of loss functions is used including photometric reconstruction, l1-flow (from pretrained model), scale-invariant-depth loss from lidar (when available). To improve stability, a two-stage optimization process is employed to first optimize the SDF field with a static flow assumption, before learning flow. Further attention is paid to the imbalance between static and dynamic regions with “Dynamic Disentanglement” to apply a weighting factor to the loss in dynamic areas. Dynamic areas are found when geometric inconsistencies are found and combined with a pretrained segmentation model that has identified a dynamic object. The paper demonstrates SOTA performance in most fields and comparable settings, they also provide ablations showing the improvement each innovation contributes.", "summary_of_recommendation": "Learning 3D Occupancy Flow with self- or weak- supervision is an important area of study to reduce annotation intensity. The authors demonstrate SOTA on popular benchmark suites with their method. However, this method’s immediate applicability is not demonstrated on a live platform and there is key missing inference cost/latency information needed to gauge whether it can run on at the edge.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 2, "quality_of_limitations": 3, "questions_for_rebuttal": "How does the number of adjacent frames used for the temporal fusion module trade off accuracy and runtime?\nWhat is the proportional split in the two-stage optimization method between the first and second stage? How do you determine when to switch between them?\nWhat impact does the 3d refine module have compared to interpolation for upsampling?\nCould the authors clarify whether the A100 used for training were the 40GB or 80GB variants in Section B3.\nWhat is the runtime inference cost of this system, would it be achievable to run on the typical compute budget of an edge robot or autonomous vehicle in real-time?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach for joint 3D occupancy and occupancy flow prediction. The reviewers acknowledge the importance of the problem being addressed. They find the paper well written and appreciate the combination of recent advances and good experiments. However, the reviewers have also raised important concerns regarding inconsistent baseline evaluations, concerns that it is not entirely self-supervised as claimed, novelty is not entirely clear as it combines many components from existing work, inconsistencies in the results on different datasets and ablations. Limitations should be also mentioned in the main paper. Each of the reviewers have listed several questions to be addressed in the rebuttal. 
Post-rebuttal: Most of the reviewers' concerns have been sufficiently addressed. Please address the remaining suggestions in the final version.
Decision: Accept