LiDARGrid: Self-supervised 3D Opacity Grid from LiDAR for Scene Forecasting
Authors: Chuanyu Pan
Aolin Xu
Venue: CoRL 2024 (2024)
Keywords: 3D perception
lidar
opacity grid
occupancy grid
neural rendering
self-supervised learning
mobile robot
autonomous driving

Abstract: Timely capturing the dense geometry of the surrounding scene with unlabeled LiDAR data is valuable but under-explored for mobile robotic applications. Its value lies in the huge amount of such unlabeled data, enabling self-supervised learning for various downstream tasks. Current dynamic 3D scene reconstruction approaches however heavily rely on data annotations to tackle the moving objects in the scene. In response, we present LiDARGrid, a 3D opacity grid representation instantly derived from LiDAR points, which captures the dense 3D scene and facilitates scene forecasting. Our method features a novel self-supervised neural volume densification procedure based on an autoencoder and differentiable volume rendering. Leveraging this representation, self-supervised scene forecasting can be performed. Our method is trained on NuScenes dataset for autonomous driving, and is evaluated by predicting future point clouds using the scene forecasting. It notably outperforms state-of-the-art methods in point cloud forecasting in all performance metrics. Beyond scene forecasting, our representation excels in supporting additional tasks such as moving region detection and depth completion, as shown by experiments.

LiDARGrid: Self-supervised 3D Opacity Grid from LiDAR for Scene Forecasting
Chuanyu Pan
Aolin Xu
Abstract: Timely capturing the dense geometry of the surrounding scene with unlabeled LiDAR data is valuable but under-cxplored for mobile robotic applications. Its value comes from the huge amount of such unlabeled data. Current 3D scene perception methods however heavily rely on data annotations during training, to be able to tackle the moving objects in the scene during inference. In response, we present LiDARGrid, 3D opacity grid represenlation that be instantly generated from LiDAR points through novel neural volume densification network. As the generated opacity grid can accurately capture the dense 3D geometry of the scene in real time, we also propose a scene forecasting network to predict the future opacity grids. Both the densification and forecasting networks are trained by self-supervision on NuScenes autonomous driving LiDAR dataset. The overall pipeline is evaluated by forecasting future point clouds from the forecast. scenes. Tt notably outperforms state-of-the-art methods in point cloud forecasting in all performance metrics. Beyond scene forccasting, opacity grid representation also excels in supporting tasks such as moving region detection and depth completion, as shown by experiments.
Introduction
Contemporary mobile robotics, e-g. aulonomous driving systems, demand accurate and timely perception of the surrounding scene. Traditional solutions perform object detection and pose estimation from camera and/or LiDAR data, which heavily rely on the amount and quality of data annotation well sophisticated sensor fusion. As huge amounts of unlabeled camera and LiDAR data are accumulated each year, there is growing interest in both industry and research communities to develop perception solutions that can be based on unlabeled data, while at the same time overcome the inherent sensor limitations, such as the depth ambiguity camera and the sparse and unstructured nature LiDAR data.
One direction of technology advancement that addresscs the above challenge is the 3D occupancy grid representation of the scene [1, 2, 3, 4, 5, 7, 8). This method has exhibited promise in various aspects of mobile robots, encompassing 3D object detection, segmentation, and scene reconstruction, primarily due to their accuracy and computational efficiency in modeling the 3D environment. Compared with camera, LiDAR has unique advantages in accurate 3D perception and robustness to different lighting conditions. However, most of these works derive their 3D grid representations from cameras, lcaving the exploration of grid representation from LiDAR data largely uncharted. Furthermore, we notice that very few of these works delve into scene forecasting, which takes few historical frames and aims at predicting future frames using certain representations, except for c.g. [6]. However, in [6]. the occupancy grid is constructed using supervised learning where the annotation of dynamic objects are needed. More literature review and discussions about related works on 3D occupancy grid construction and forecasting are relegated to Appendix B.
'Honda Research Institute, USA. Email: panchuanyu450gmail. com, aolin ww@honda-ri.com Work done during an internship at Honda Research Institute, USA, now at Meshy.
8th Conference Robot Learning (CoRL 2024), Munich, Germany.

---

To fill in these research gaps, we propose LiDARGrid, a dense 3D opacity grid representation from LiDAR points. Motivated by optical models of volume rendering [9] and neural volume rendering, c.g. NeRF [10], we conceptualize our representation as a 3D grid, where cach voxel represents the LiDAR opacily, or more precisely, the incremental likelihood of a ray to stop marching by hitting some particle in that voxel. This representation allows for distance rendering of LiDAR points, such that one compute the distance from any position to object's surface within the grid, thus reconstructing the scene geometry. The proposed 3D opacity grid is generated from LiDAR points by an autoencoder-basse densification network, trained by matching the rendered LiDAR distance with the ground truth distance. Though the densified grid may be converted to a binary-valued occupancy grid by simple thresholding, we keep as a continuous- valued grid, which can be used as a universal representation of the 3D scene for diverse perception and forecasting tasks. For this reason, we name our representation as the 3D opecity grid.
We explore the application of this representation in scene forecasting, where a forecasting network based on 3D convolution and UNET architecture takes historical grids as input and predicts future grids. Notably, both the densification network and the forecasting network are trained on unlabeled LiDAR points, adopting a self-supervised approach that can readily access vast pool of training data. The performance of scene forecasting is evaluated by the point cloud forecasting metrics. Khurana et al. [11] is the most recent work that predicts future point clouds with grid-based representation and known LiDAR pose. It shows much crisper results than previous point cloud forecasting works [12, 13) with model-free prediction and unknown LiDAR pose, partly due to the fact that the 3D grid better preserves scene geometry. Advancing along the direction of [11] by incorporating the densification network and the improved forecasting network, we achieve notable improvements the point cloud forecasting metrics, thus pushing forward the seate-of-thheane performance of this task. We also provide comprehensive ablation studies evaluate each module and justify the key design choices toward the improved scene forecasting. Besides scene forecasting, we extend the utility of our 3D grid representation to diverse applications, including moving region detection and depth completion. In summary, our contributions include:
1. We propose a 3D opacity grid representation for mobile robot perception. The representation accurately capture the 3D scene geometry and can be generated in real time, which are crucial for various perception and forecasting tasks. The representation is learned solely based on unlabeled LiDAR data.
2. We propose 3D convolutional forecasting network for our opacity grid. It outperforms the state-of-the-art approaches point cloud forecasting tasks in all performance metrics.
We provide a comprehensive ablation st stu to elucidate the efficacy of our method, along with demonstrations of its potential in extended applications, such as moving region detection and depth completion.
2 Method
2.1 3D Opacity Grid Representation
Representation Formulation Our representation of the scene is inspired by the optical model for volume rendering used in computer graphics [9, 10). It is assumed that the 3D space is permeated by particles that can scatter light, and the density of the particles varies across the space. We define the representation as a 3D grid of dimension H x W x L with H, W, I. being positive integers, scaled by a factor of S E to align with the real-world metric in meters. In other words, S the voxel size in meters. The origin of the real-world coordinate system is centered within this grid. The continuous value of a voxel at position (h, w. I) {l....,If} HI} (l....,W W} {l..... L} in the grid, denoted as oh, w, (), represents
2

---

(3) Generate ID Opacity Grid Representation
(G) Applications
Pount Forceasting
LADAR Points
Spane Grid
Densiftcttion Network
Dosse Grid
Ray-distance Los
Fature Opacity Orid Forcesstnng
Movement Detection
Histonial Grids E (-Tpee......
Forecasting Network
Future Gridkt e(), Tuuuel
Depth Complesion
Figure 1: Overview of our approach. (a) To generate 3D opacity grid representations (LiDARGrid), our approach first initializes a sparse grid with LiDAR points and then transforms it into dense 3D opacity grid through a densification network. Note that there are no skip connections in the densification network, unlike the forecasting network below. The training of the network is supervised by the ray-distance loss defined in section 2.1. (b) We illustrate the efficacy of our representation through scene forecasting, that is to predict a sequence of future grids from historical grids through 3D convolutional forecasting network, which is also supervised by the ray-distance loss. (c) Utilizing our represenlation, we perform multiple (asks, such point cloud forecasting, movement detection, and depth completion.
the density of particles at the real-world point Zhwi (S(w S(1 S(h h ))) which is approximately proportional to the number of particles within that voxel.
This representation is deterministic, but it gives rise to a simple probabilistic model LiDAR point rendering: suppose the particles are uniformly placed random within each voxel, and independently across voxels, then as the voxel size S decreases to infinitesimal, ah, w, I) becomes the rate of change of the probability of a LiDAR ray hitting any partiele in that voxel [9, 14]. With normal sized grid, given ray o + d with origin and direction d, if the ray intersects with a sequence of voxels (V1...., vn) at distance (T,,).,,T the probability of the ray hitting partiele when traveling from 14-1 to 02 can be derived as [14]
ag 1- 1x exp (ra ri-1)).
(1)
We see that a essentially represents the opacity of each voxel: the larger the value (v), the more likely that a ray stops at voxel v. Hence we name the proposed representation as 3D opacity grid. Applying the above probabilistic point rendering model, the distance R that ray travels before stopping at voxel is random variable, taking value r with probability
Ps
a.). s=1
(2)
The expected distance of ray is then
E[R Σ
pi
(3)
The process of tracing these rays can be likened simulating LiDAR beams as they traverse through open space and terminate upon encountering an object's 's surface. Utilizing Eq. 3, given the 3D opacity grid, can compute the expected distance between the starting point and the end point of a LiDAR ray.
Initialize Grid with LIDAR Points To acquire the 3D opacity grid from LiDAR points, we first. initialize a grid by mapping the LiDAR points to the grid. For a LiDAR point (z, represented in the real-world coordinate, we map it to voxel grid at position W (1 11, 2
1) and set the voxel value to constant dg. Applying this process to all LiDAR points, can initialize sparse grid of spatial occupancy.
3

---

Volume Densification The major challenge we address is generating a dense and contimuous-valued opacity grid from the sparse initial grid. naive way would be applying a low-pass filter to the sparse grid, howover its rigidity hinders the geometric consistency of the filtered grid. With this in mind, we design a learnable autoencoder densilier:
dense- = (Vg((Vparse)) V Vaparse: V Vense E "R
(4)
The encoder E-) maps the sparsely initialized grid into a low-dimensional representation with a series of convolution layers, extracting low-frequency information from the input. The decoder (-) upsamples the intermediate features with transposed convolution to reconstruet grid with the same dimension as the input. Importantly, unlike the popular UNET architecture, we do not include skip connections between the encoder and decoder layers in order to retain more low-froquency information when passing through the network. Experiments show that this design choice is crucial for achieving a good densification performance. The network is trained in a self-supervised manner with the ray-distance loss:
((P dense)
Σ 1r-- |
o+d-rEP
where is a set of LiDAR points, r is the ground-truth distance of point the LiDAR origin o. E[R] can be calculated by Eq. and is differentiable for backpropagation. We randomly rotate and translate the point cloud as data augmentation to acquire robust densification. The network eventually learns pattern to map V Vsparse to dense representation V deese- Furthermore, the small-size intermediate feature vector can be stored as compressed version of the 3D grid, which largely improves the memory efficiency of this representation, and can potentially be even more memory-effiiient than storing the LiDAR points of a scene.
2.2 Scene Forecasting with 3D opacity grid
We apply our 3D opacity grid representation to scene forecasting and show that with accurate 3D geometry reconstructed, we are able lo reach better performance on such tasks.
Problem Definition The goal of scene forecasting to predict how the surrounding scene evolves in the future. Encoding the scene with our grid representation, the problem can be defined as finding a forecasting model F(-), so that
(Vet1,..., VI+Tua = piV V.)
where V, E RRHWx Tprev and Tiuture the numbers of history frames and future frames respectively. Following the setting of [11], for each frame, we first transform the LiDAR points from their local sensor coordinate to the coordinate at frame based on the knowledge of LiDAR pose in each frame. Then we use the transformed points to initialize and densify the grid as stated in Section 2.1. Note that this coordinate transformation is optional, but would be a reasonable way using the proposed method practice.
Scene Forecasting Network Fnlightened by the success of 2D convolution on image natural extension of the 2D occupancy
perception, regard our 3D grid representation grid and apply 3D convolution network to predict future grids. It can preserve local spatial information while extracting important features due to its 3D shift equivariance. Our scene forecasting network F(.) is a UNET-style [15] 3D convolutional encoder-decoder network. Each pair of corresponding layers in the encoder and decoder with the same feature size are connected by a skip layer. Following Eq. 6, the input Dprev E To x XxxXX and output Ditture E Thuuur x xxW.xx XL are series of temporal-continnous frames. Note that it is general framework that allows various designs on the intermediate fcature layer. For example, one can add LSTM modules [16] to process the latent features. As baseline model, we are using identity mapping for simplicity.
Loss Function An intuitive way train is to use Dinture (V++ Ve+Tnenr)) as supervision, where V, generated from the future ground-truth LiDAR points:
Lvoxel =
Σ D Diturr D Luturel
(7)

---

However, due to high dimensionality, we observe that the network tends to learn an average result with this loss function because occupancy change on an object's surface barely affects the overall loss. Therefore, we propose to use the sparse ground-truth LiDAR points directly as a weaker supervision:
Tratura
Cry
C(Peti Veti)
(8)
where L(Pt+i, Ve+s) shares the same definition in Eq. 5, Pots is the set of LiDAR points at frame t t+i. We compare these two loss functions in the experiment section.
Point Cloud Forecasting Our method can be easily applied to point cloud forecasting tasks. Given future 3D grids (Ve+1...., Ve+Thaure and LiDAR rays (o, (d:)), one simulate future LiDAR points with volune rendering (Eq. to Eq. 3). We follow [11] and set (o, (d:))
from ground-truth future LiDAR rays.
2.3 Applications beyond Scene Forecasting
In addition to scene or point cloud forecasting, our 3D opacity grid representation can be applied to other tasks. We describe two additional tasks: moving region detection and depth completion in the experiment section, and show promising results on both of them.
Experiments
3.1 Sctup
Dataset Our experiments were mainly performed on the NuScenes dataset [17], which is a public autonomous driving dataset containing 1000 driving sequences collected by 6 cameras, LiDAR, and 5 RaDAR sensors. We only use LiDAR data, which has a frame rate of 2Hz with around 20000 points per frame. We follow the NuScenes' setting and split the dataset into 850 training scenes and 150 testing scenes. We train our densification network and forecasting network on the training and show evaluation results on the testing set.
Grid and Networks Details Following [11], the grid covers a 4.5m x 70m x 70m 3D space. Specifically, we set the grid dimension and size to H = 45, W = 700, L = 700, and = 0.1. Each voxel in the opacity grid is initialized with σn = 1 if contains a LiDAR point and otherwise. Our densification network contains an encoder with 4 downsampling layers and decoder with upsampling layers. Each downsampling layer shrinks the grid size by 2 and doubles the channel size. Upsampling layers reverse this process. Our forecasting network shares similar structure with 3D convolution and skip connections between each pair of encoder-decoder layers.
To ensure a rigorous comparison, we adopt the identical evaluation metrics as presented in [11]. Specifically, we report the in-grid error (L1) and relative in-grid error (AbsRel), measuring the accuracy of distance predictions along LiDAR rays. Additionally, we employ the vanilla Chamfer distance (Vanilla CD) and in-grid Chamfer distance (In-grid CD) [11, 12] to gauge the spatial distribution error of the predicted point cloud, with measurements expressed in square meters (m?).
Evaluation Metrics
3.2 Evaluate Scene Forecasting with Point Cloud
Basclincs We benchmarked our mothod against the most recent stato-of-tho-art approaches in point cloud forecasting, including S2Net[12), SPFNet[13]] and 4DOce[11]. Notably, both S2Net and SPFNet utilize the range map[18] as the scene representation, translating the 3D point cloud into 2D and subsequently employing 2D convolution techniques. 4DOcc employs similar grid representation but distinguishes itself by using sparse grids acquired from LiDAR points input, and further processing them through 2D convolution.
5

---

Method
Horizon
L1(m)
In-grid CD₁
Vanilla CDJ
Is
3.49
28.38
1.70
2.75
S2NN[[12]
de
4.78
30.15
2.06
3.47
SPFNNe(13)
1s
4.58
34.87
2.24
4.17
ds
5.11
32.74
2.50
4.14
18
1.40
10.37
1.41
2.81
4DOce[11]
3s
1.71
13:48
1.40
4.31
Ours
1s
1.12
9.04
0.68
1.74
3s
1.50
12.04
1.04
2.13
Table Comparison with the state-of-the-art point cloud lorecasting methods on NuScenes Dataset[17]. We follow the evaluation metrics in 4DOcc[11]. The 1s and 3s horizons refer to 2 frames and frames respectively, for both input and output.
Method
Horizon
[.(m))
Abs)
In-grid CD,,
Vanilla CD)
w/o VD
1.23
0.76
1.88
1.52
12.53
1.07
2.18
1s
1.12
9.04
0.68
1.74
w/
VD
1.50
12.04
1.04
2.13
Table 2: Evaluation on point cloud forecasting without and with Volume Densification (VD).
Quantitative Results Table presents comprehensive comparative analysis between our method and the bascline models. Notably, our approach excels in reducing the average L1 distance error, showcasing an improvement of 0.28m and 0.21m for the is and 3s settings, respectively, when contrasted with the current statc-of-the-art. Most notably, our method demonstrates a substantial reduction in the in-grid Chamfer distance error, achieving significant improvement of 50% and 29% for the is and 3s settings, and simultaneously lowering the vanilla Chamfer distance error by 38% and 50% for the same settings. This robust performance not only attests to method's proficiency in predicting the scene's underlying geometry (Ll and AbsRel) but also underscores its capacity to predict uncorrelated samples (Chamfer Distance), such as point clouds. Our method achieves this remarkable improvement in Chamfer distance, despite not having employed as direct training supervision, underscoring the accuracy and robustness our approach in predicting future scene geometry.
Run time
The proposed pipeline GPU, being able to support real-time applications. Further improvements may be gained from network optimization and CUDA acceleration.
at 5 frames/second Quadro RTX 6000
3.3 Ablation Study
Several key designs contribute to the improved scene forecasting, including volume densification, 3D convolutional forecasting network, and its loss function. To thoroughly evaluate the effect of these modules, we conduct a comprehensive ablation study on them.
Opacity Grid Densification We argue that inputting dense and geometry-present 3D reprosentation the forocasting network is essential to the forccasting performance. In this section, we train and test. our forecasting network with or without volume densification. For the 'without' setting, simply initialize the grid with LiDAR points introduced in Section 2.1 and input it into the forccasting network, the same as what is done in [11].
Table shows the forecasting results with and without volume densification. Upon introducing volume densification, we observed consistent improvements across all evaluation metrics for both Is and 3s settings. These improvements underscore the consistent effectiveness of the volume densification module in enhancing the forecasting task. It is noteworthy that volume densification offers benefits beyond scene forecasting, We illustrate this through visualizations

---

Figure 2: Visualizations of volume densification. Top-left: RGB image of the current view. Top-right: input sparse grid derived from LiDAR points. Bottom-left: densification result from the network with skip connection. Bottom-right: a densification result from the network without skip connection, which is used in our method.
of the opacity grid, both before (top-right) and after (bottom-right) densification in Figure 2, revealing a densified 3D grid with fewer holes and a more continuous spatial distribution. This not only bolsters forecasting but also enhances its applicability broader tasks, as discussed in Section 3.4, such as moving object detection.
Densification Network We state in section 2.1 that the encoder of the densification network should be low-pass filter. Since the supervision is sparse, letting high-frequency signals pass through the network will cause overfitting. We will show this by comparing the nelwork with or without skip connections. Specifically, for the network with skip connection:
V dense (E (Vsaaee)))) + +M(V sparse)
(9)
where M(.)) is a linear mapping, others share the same definition with Eq.(
Figure 2 shows that the network with skip connection generates grids with holes, where the occupied voxels are distributed discretely since the network is overfitting the sparse input. The network without a skip connection maps the input to a low-dimensional manifold and generates a continuous and dense grid after decoding.
Loss Function In section 2.2, introduced two distinct loss functions, namely Lyozel and Cray, employed in training the forecasting network. In this section, delve into comparative analysis of the performance outcomes associated with these two loss functions. In both scenarios, training is carried out until convergence is achieved. For the Lvoxel setting, we apply the L1 loss to all voxels within the grid, utilizing the 'sum' reduction. The learning rate is set to 1o-6, with other conditions held constant.
Table 3 shows the results of applying these two loss functions. We show that compared to L using Lray will drastically reduce the prediction error, on both LI metric and chamfer distance. It is because, in areas obstructed from LiDAR rays, small noises are introduced due to the absence of information the LiDAR-invisible regions. Employing L compels the model to accommodate these inaccuracies, leading to confusion and a tendency to neglect regions with accurate LiDAR supervision. Conversely, the use of Lray mitigates this issuc, resulting in more accurate and robust. forecasting model.
3.4 Extended Applications
Moving Region Detection Through the alignment of two temporally adjacent 3D opacity grids within the same coordinate system, our method enables the detection of moving regions

---

Method
Horizon
L1(m)
AbsR
In-grid CD}
Vanilla CDL
1s
2.48
23.27
1.53
2.45
Lyosel
3s
3.92
41.70
3.22
4.27
9.04
0.68
1.74
Cray
3s
12.04
1.04
2.13
Table Evaluation on point cloud lorecasting by training with L1 Voxel Loss Lvoxel and Ray-distance Loss Lray.
Figure 3: Moving region detection. Top-left: RGB image of current view. Top-right: rendered depth current view within 3D opacity grid; darker colors indicating closer distances. Bottom-left: point cloud of entire scene; detected moving points highlighted in red. Bottom-right: detailed view of detected moving points, which are pedestrians.
in the scene without relying on any object detection module. Traditional methods for this purpose e.g. [19] face challenges in ellectively aligning LiDAR rays across consecutive frames due the stochastic nature of ray directions over time. Our representation, as it can reconstruct dense scene geometry, overcomes this limitation. Figure 3 shows an example, where points on moving pedestrians are clearly detected by our method.
Depth Completion Existing camera-LiDDAR-besse depth completion require sophisticated fusion pipelines. Meanwhile, LiDAR alone offers only sparse depth maps. With our method, one can simply use the dense opacity grid generated from LiDAR to render depth values for any virtual camera pixel, following the principles outlined in Eq. 1 through 3. The promising performance of our method can be visualized in Figure 4.
RGB Image
(b) LIDAR Depth Map
Complete Depth Map
Figure 4: Depth completion: dense depth map are solely from sparse LiDAR input.
8

---

Acknowledgment
The authors would like to thank Seth Zhihao Zhao and Behzad Dariush for helpful discussions.
References
[1] A look Tesla's occupancy networks. occupancy-netoork// 2022. Thinkautonomous blog.
ai/blog/
22 Y. Huang, W. Zheng, Y. Zhang, J. Zhou, and Lu. pripperspective view for vision- basion- 3d semantic occupancy prediction. arXiv preprint arXiv:2302 07817, 2023.
[3] Y. Wei, Zhao, W. Zheng, Z. Zhu, J. Zhou, and J. Lu. Surrounocc:: Multi-camera 3d occupancy prediction for autonomous driving. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 21729-21740, 2023.
[4] Q. Zhou, J. Cao, H. Leng, Y. Yin, Y. Kun, and R. Zimmermann. Sogdet: Semantic-occupancy guided multi-view 3d object detection. orXiv preprint orXiv:2308. 18794, 2023.
[5] W. Tong, C. Sima; T. Wang, L., Chen, S. Wu, H. Deng, Y. Gu, L. Tu,, P. Luo, D. Lin, et al. Scene as occupancy. In Proceedings of the IREE/CVF International Conference on Computer Vision, pages
2023.
[6] X. Liu, M. Gong, Q. Fang, H. Xie, Y. Li, H. Zhao, and C. Feng. Lidar-based 4d occupancy completion and forecasting. arXaw:2300. 11299, 2023.
[7] Z. Ming, J. S. Berrio, M. Shan, and S. Worrall. Occfusion: Multi-sensor fusion framework for 3d semantic occupancy prediction. arXiv:2403.0100 2024.
[8] H. Xu, J. Chen, S. Meng. Y. Wang, and L.-P. Chau. A survey on occupancy perception for autonomous driving: The information fusion perspective. arXiv:2405. 05173, 2024.
[9] N. Max. Optical models for direct volume rendering. IEEE Transactions on Visualization and Computer Graphics, 1(2):99-108, 1995.
[10] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106 2021.
[11] T. Khurana, P. Hu, D. Held, and D. Ramanan. Point cloud forecasting as a proxy for 4d occupancy forecasting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pallern Recogrition, pages 1116 1124, 2023.
[12] X. Weng, J. Nan, K.-H. Lee, R. McAllister, A. Gaidon, N. Rhinehart, and K. M. Kitani. S2net: Stochastic sequential pointcloud forecasting. In European Conference Computer Vision, pages 549-564, Springer, 2022.
[13] X. Weng, J. Wang, S. Levine, K. Kitani, and N. Rhinehart. Inverting the pose forecasting pipeline with spf2: Sequential pointeloud forecasting for sequential pose forecasting. In Conference on robot learning, pages 11-20. PMLR, 2021.
[14] A. Tagliasacchi and B. Mildenhall. Volume rendering digest (for nerf). arXie preprint ar Xix::22000 024 17, 2022.
[15] O. Ronneberger, P. Fischer, and Brox. U-net: Convolutional networks for biomedical image segmentation, 2015.
[16] Y. Wang, M. Long, Wang, Z. Gao, and P. S. Yu. Predrnn: Recurrent neural networks for predictive learning using spatiotemporal Istans. Advances neural information processing systems, 30, 2017.
[17] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, Baldan, and O. Beijbom. muscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pollern recognition, pages 11621-11631 11631, 2020.
[18] L. Caccia, Н. Van Hoof, A. Courville, and J. Pineau. Deep generative modeling of lidar data. In 2019 IEEE/RSJ International Conference on Intelligent Hobots and Systems (IROS), pages 0034-5040 LEEE, 2019.
9

---

[19] X. Chen, S. Li, B. Mersch, L. Wiesmann, J. Gall, J. Behley, and C. Stachuiss, Moving object segmentation in 3d lidar data: A learning- approach exploiling sequential data. IEEE Robolics and Aulomation Letlers, 6(4):6529 6536, 2021.
[20] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang. S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 9660-8669, 2019.
[21] C. Sima, W. Tong, T. Wang, L. Chen, S. Wu, H. Deng, Y. Gu, L. Lu,, P. Luo, D. Liu,, and H. Li. Scene us occupancy. 2023.
[22] z. Li, Z. Yu, Austin, M. Fang. S. Lan, Kautz, and J. M. Álvarez. Fb-occ: 3d occupancy prediction based on forward-backwer view transformation. ArXiv preprint, 2023.
[23] Y. Zhong and H. Peng. Real-time semantic 3d dense occupancy mapping with efficient free space representations. In 2022 IEFF 25th International Conference on Intelligent Transportation Systems (ITSC), pages 230 236, 2022.
[24] S. Thrun, W. Burgard, and D. Fox. Probubilistic robofics. Intelligent robotics and autonomous agents. MIT Press, 2005.
[25] X. Tian, T. Jiang, L. Yun, Y. Wang, Y. Wang, and H. Zhao. Occ3d: A large-scale 3d occupancy prediction benchmark for autonomous driving. arXiv preprint or Niv::0304 14365, 2023.
[26] B. Miller, H. Chen, A. Lai, and I. Gkioulekas. Objects as volumes: A stochastic geometry view of opaque solids. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogrition (CVPR), 2024.
[27] Y. Hong, K. Zhang, J. Gu, S. Bi, Y. Zhou, D. Liu, F. Liu, K. Sunkavalli, T. Bui, and H. Tan. LRM: Large reconstruction model for single image to 3d. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
[28] J. Tang, Z. Chen, X. Chen, T. Wang, G. Zeng, and Z. Liu. LGM: Large multi-view Gaussian model for high-resolution 3d content creation. In Proceedings of the Eurapean Conference on Computer Vision (ECCV), 2024.
[29] W. Jang al. Nvist: In the wild new view synthesis from a single image with transformers. In Proceedings of the SEE//CVF Conference on Computer Vision and Pattern Recognition, 2024.
[30] C. Feldmann, N. Siegenheim, N. Hars, Rabuzin, M. Ertugrul, L. Wolfart, M. Pollefeys, and M. R. Oswald. Nerfmentation: NeRF-based angmentation for monocular depth estimation. Fff::201.0377:1 2024.
Xiv preprint
[31] H. Wu, X. Zuo, S. Leutenegger, O. Litany, K. Schindler, and S. Huang. Dynamic lidar re- simulation using compositional neural fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pullern Recognition (CVPR), 2024.
[32] D. Nuss, S. Reuter, M. Thom, T. Yuan, G. Krehl, M. Maile, A. Gern, and K. Dietmayer. A random finite set approach for dynamic occupancy grid maps with real-time application. The International Journal of Robotics Research, 37(8):841-88) 866- 2018.
[33] S. Hoermann, M. Bach, and K. Dietmayer. Dynamic occupancy grid prediction for urban autonomous driving: A deep learning approach with fully automatic labeling. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2056 2063, 2018
[34] N. Mohajerin and M. Rohani. Multi-step prediction of occupancy grid maps with recurrent neural networks. In 2019 IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pages 10592-10600, 2019. doi:10. /CV...22P..........
[35] M. Schreiber, V. Belagiannis, C. Glaser, and K. C. J. Dietmayer. Dynamic occupancy grid mapping with recurrent neural networks. 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6717-6724, 2020. URL https://api semanticscholar org/ CorpusID: : 226975761.
[36] M. Schreiber, V. Belagiannis, C. Gläser, and K. Dietmayer. A multi-task recurrent neural network for end-to-end dynamic occupancy grid mapping. In 2022 IEEE Intelligent Vehicles Symposium (IV), page 315-322, 2022.
10

---

[37] L. Wang; H. Ye, Q. Wang, Y. Gao, C. Xu, and F. Gao. Learning-based 3d occupancy prediction for autonomouus navigalion in occluded enviromments. In 2021 IEEE/RSJ International Conference on Intellient Robols and Systems (IROS), pages 4509 4516, 2021. Ree:10.119/RRO551II 2111111668.2.1.6663333.
[38] B. Agro, Q. Sykora, S. Casas, T. Gilles, and R. Urtasun. Uno: Unsupervised occupancy fields for perception and forecasting. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pallern Recogriition (CVPR), 2024.
[39] Z. Yang, L. Chen, Y. Sun, and H. Li. Visual point cloud forecasting enables scalable autonomous driving. In Proceedings of the IEEE/CVF Conference Compuler Vision and Pallern Recognition, 2024.
[40] X. Lin, C. R. Qi, and L. J. Guihas. Flownetãd: Learning scene flow in 3d point clouds. In 2019 TREE/CVF Conference on Computer Vision and Pattern Recognátion (CVPR), pages 529-531 2019. doi:10. 1ww.11/999/.........
A Appendix: additional ablation studies
A.1 2D Convolution vs. 3D Convolution
Some previous works [12, 16, 20] use 2D convolution in their forecasting network to process intermediate representation. 4DOcc [11] follows these works and applies 2D convolution to their 3D representation. We argue that 3D convolution will be more suitable operation for our 3D grid representation than 2D convolution. Therefore, we conduct experiments to compare these two oporations. For 2D convolution, we follow [11] and reshape the input from size Tprev x x W to (Tprev H) x W x L.
Table 4 illuminates the forecasting results achieved by employing 2D convolution versus 3D convolution within the network. Notably, observed substantial improvement when transitioning from 2D to 3D convolution, particularly evident in the L1 and AbsRel metrics. This marked enhancement can be attributed to the intrinsic ability of the 3D convolution network to effectively extract 3D spatial information, courtesy of its equivariance property and synergy with the 3D grid representation. Furthermore, the 3D convolution framework affords the capability to process spatial and temporal information separately, thus providing more versatile platform for the incorporation of modules designed to address temporal consistency testament to its broader utility and effectiveness in the forecasting task.
Method
Horizon
Ll(m)4
AbsRel((6))
In-grid CDL
Vanilla CD4
2D Conv
1s
1.30
10.42
0.79
1.91
38
1.81
15.00
1.39
2.53
Is
1.12
9.04
0.68
1.74
3D Conv
3s
1.50
12.04
1.04
2.13
Table 4: Evaluation results on point cloud forecasting metrics with the forecasting network involving 2D and 3D convolution.
B Appendix: additional literature review
B.1 3D occupancy grid scene representation
The 3D occupancy grid representation has gained increasing attention in recent years. Tesla introduced a real-time camcra-basod ncural network solution for the 3D occupancy grid in 2022, named occupancy network [1]. The major advantage of such scene representation that its accuracy need not be limited by the object-level annotation, thus enabling 3D
perception of scenes containing unseen and irregular objects, either static or moving, Up- to-date souresourced studies occupancy network alternatives include Surroundoce [3], TPVfrreer [2], OccNet [21] and FBOcc [22] to name a few, all of which are camera- based.
11

---

Conventional LiDAR-based 3D occupancy representations mainly appear in the 3D mapping and segmentation literature, as reviewed in [23], and are far less studied compared to a large volume of literature on 2D occupancy from range sensors [24]. Most L-DAR- based 3D occupancy relies on approaches such as kernel methods and graphical models or other Bayesian methods densify LiDAR point clouds, mostly resulting in a continuous volume representation. LiDAR-basod 3D occupancy grid first appears in [11], however, the representation is solely constructed by placing point cloud in the grid, hence sparse. Our 3D opacity grid representation is directly tied to the volume rendering model in computer graphics [9, 10], where it is assumed that the space is permeated by particles, and the value of each voxel in the grid represents the density, proportional to the number, of particles in that voxel. The representation is constructed by densifying LiDAR point cloud using carefully tailored neural network, hence dense, and it can run in real time.
Methods of obtaining ground-truth 3D occupancy grid from camera and LiDAR are discussed in [1, 25J. In these works, the moving objects are first detected and then separated from static backgrounds, so that the LiDAR point clouds on different types of objects can be identified and averaged out differently. In contrast, the training of the neural networks we use for opacity grid generation and forecasting does not rely on object annolation all, and the trained densification network produces a dense opacity grid solely based on the LiDAR point cloud in the current frame.
Rigorously speaking, our 3D opacity grid is not equivalent to a 3D occupancy grid; instead, it is more of an intermediate and universal representation of the 3D scene. Nevertheless, as the density of particles in a voxel reflects the extent to which that voxel occupied, an opacity grid may be converted to an occupancy grid by thresholding the value of each voxel into a binary value. The quantitative comparison between the 3D opacity grid of a scene generated by our method and its ground truth 3D occupancy grid is left a future study. Some relevant theoretical and experimental investigations to this problem can be found in [26] and the reference therein.
Some recent works propose to generate 3D scene representations c.g. triplane, NeRF or Gaussian splatter, by directly feeding images to trained neural networks, including (27, 28, 29, 30, 31]. Conceptually our method is closely related to these works, in that we directly generate 3D opacity grid by feeding LiDAR points to densification network. An important feature of our method is that we explicitly use the NeRF-like volume rendering model to derive the ray-distance loss for training, which distinguishes our method from the pure model-free methods.
B.2 Scene or occupancy grid forecasting
The problem of occupancy grid forecasting is to predict how the occupancy grid evolves in time. It arises when the scene contains moving objects. The majority of existing works toward this problem consider the prediction of 2D dynamic occupancy grid [32, 33, 34, 35, 36). The prediction of a scene represented by 3D occupancy grid appears to be a new problem, and is initiated by [11]. Different from conventional abjoct-based scene prediction, the occupancy grid prediction under consideration is self-supervised, meaning that there is no need to perform object detection and tracking, and unannotated sensor data is directly used for prediction.
We have to distinguish the prediction problem studied this work, which is about estimating future scene, from many existing works titled terms of occupancy grid prediction, which are about scene completion or estimating occupancy of occluded areas. Works in the latter category include (37, 3, 22, 25). To this end, we term the problem under study opacity grid forecasting.
12

---

B.3 Point cloud forecasting, 3D flow, and moving region detection
Point cloud forecasting has also arisen as a new problem in computer vision for robotic perception in recent years [12, 13, 11], as has the potential benefit downstream tasks with large volume of unannotated LiDAR data. Earlier works on this problem [12, 13] do not separale sensor movement from the scene and predict the point cloud in fully model-free manner. The recent work [11] reformulated this problem with fixed reference frame, but leveraging known sensor pose, and improved the forecasting accuracy by leveraging the 3D occupancy grid representation. We follow the direction proposed in [11], and propose new methods for 3D opacity grid construction and forecasting to further improve the performance. Another closely related work [38], where an implicit continuous occupancy field is learned without supervision for point cloud forecasting; representation and explicitly forecast the grid. Some more recent works on point cloud forecasting also rely on camera-based object detection, e.g. [39].
contrast, we utilize the discrete grid
Another advantage of working with LiDAR data is that the time evolution of the point cloud can provide information on object movement in 3D space. This problem has been studied [40], where an end-to-end framework on point cloud flow is proposed. Some recent works on 3D occupancy completion and annotation also take the flow into consideration, by estimating the velocity in 3D of each voxel in the grid [1, 25). There are also plenty of works motion estimation from 2D dynamic occupancy grid, e.g. [33] for end-to-end learning-basee based approach and the references therein. In those works, challenging issue is the extreme imbalance between static and dynamic cells, and pixel-wise balancing is usually applied in the loss function counteracting. We also propose a solution for moving region detection based on our densified 3D opacity grid, where we synthesize the point cloud the previous scene but the ray directions of the current scene, and detect the distance change of the point cloud which gives hints the moving region.
13


Reviews:
[{"review_id": "K5pB9izG2b", "reviewer": "Reviewer_8n74", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "A self-supervised lidar pointcloud forecasting method with a learned 3d opacity voxel representation of lidar point clouds, and a convolutional forward prediction network. The method achieves SOTA on nuScenes lidar forecasting task.", "summary_of_recommendation": "Good paper on point cloud forecasting, i recommend to accept given that the current weaknesses can be addressed.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "See weaknessess listed above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kewQA455HD", "reviewer": "Reviewer_V6p6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "Authors propose a combination of Unet-style 3D network and learnable autoencoder for various operations on voxel-grid scenes.", "summary_of_recommendation": "Method does not appear novel, nor are experiments extensive enough to accept as an in-depth analysis/ablation of applying 3D convolution Unet to various 3D scene problems.", "technical_quality": 2, "clarity_of_presentation": 3, "potential_impact": 2, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "There are many concerns, but the main issues from my perspective are that:\n\n1) An opacity grid as described is just a voxel grid with each point having a density/opacity value. This is not a novel concept, with many existing works operating on voxel grids where the voxels are not simply binary \"occupied or unoccupied\".\n2) The main contribution is scene forecasting, however the method to achieve this is simply a Unet style autoencoder with no notable technical novelty. While demonstrating that a voxel based representation outperforms e.g. 2D ones is interesting, that alone does not feel sufficient as a contribution.\n3) For other proposed applications qualitative data is provided without either quantitative results or comparison to existing methods. While sufficient to demonstrate broad application potential, it does not demonstrate whether the proposed method is competitive for those applications.\n\nThere are also various minor areas to address, from occasional grammar mistakes, to the confusion created by figure 2 being coloured based on distance, despite being an \"opacity grid\". At first I was expecting it to be coloured based on opacity. Also unsure why the \"sky\" in figure 2 is dark when in theory it should be both completely opaque and have a distance of infinity.\n\nBut those minor areas are just areas to consider updating for any final edits, it's the main three which prevent me from being able to accept this paper for submission.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "M9D0s6MFB9", "reviewer": "Reviewer_rLH6", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a new 3D opacity grid representation derived from LiDAR points with neural volume densification and differentiable volume rendering to support tasks like scene forecasting.", "summary_of_recommendation": "Generally acceptable but need further clarification and information.", "technical_quality": 3, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "* Is it possible to move the `related works` section to the main paper?\n* In the self-supervised training of the densification network, does all the test points in `ray-distance loss` also inputed to the encoder? If so, will this harm the generalization ability? More analysis is appreciated.\n* In the line `155`, what exactly is the `identity mapping`? need more details.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper proposes a 3D opacity grid representation learning approach for LiDAR points for scene forecasting. The reviewers find the paper generally well written and to be addressing an important problem. They appreciate the simplicity of the approach and the good results. However, the reviewers have also raised several concerns including lacking comparisons with state-of-the-art methods, concerns on the effectiveness of the dense voxel representation, unclear relation to certain existing work which raises the question of novelty, experiments lacking in-depth analysis, lacking quantitative comparisons, among others. Importantly, related work. conclusions, and limitations sections are missing. Each of the reviewers has listed several questions to be addressed in the rebuttal.
Post-rebuttal: Most of the reviewers' concerns have been sufficiently addressed. Please address the remaining suggestions in the final version.
Decision: Accept