LiDARGrid: Self-supervised 3D Opacity Grid from LiDAR for Scene Forecasting

Authors: chuanyu  pan
aolin  xu
Venue: robot-learning_org_CoRL_2024_Conference
Published: 2024-06-07 12:44:49

Abstract: Timely capturing the dense geometry of the surrounding scene with unlabeled LiDAR data is valuable but under-explored for mobile robotic applications. Its value lies in the huge amount of such unlabeled data, enabling self-supervised learning for various downstream tasks. Current dynamic 3D scene reconstruction approaches however heavily rely on data annotations to tackle the moving objects in the scene. In response, we present LiDARGrid, a 3D opacity grid representation instantly derived from LiDAR points, which captures the dense 3D scene and facilitates scene forecasting. Our method features a novel self-supervised neural volume densification procedure based on an autoencoder and differentiable volume rendering. Leveraging this representation, self-supervised scene forecasting can be performed. Our method is trained on NuScenes dataset for autonomous driving, and is evaluated by predicting future point clouds using the scene forecasting. It notably outperforms state-of-the-art methods in point cloud forecasting in all performance metrics. Beyond scene forecasting, our representation excels in supporting additional tasks such as moving region detection and depth completion, as shown by experiments.

Main paper: 
 introduction: 
 Contemporary mobile robotics, e.g. autonomous driving systems, demand accurate and timely perception of the surrounding scene. Traditional solutions perform object detection and pose estimation from camera and/or LiDAR data, which heavily rely on the amount and quality of data annotation as well as sophisticated sensor fusion. As huge amounts of unlabeled camera and LiDAR data are accumulated each year, there is a growing interest in both industry and research communities to develop perception solutions that can be based on unlabeled data, while at the same time overcome the inherent sensor limitations, such as the depth ambiguity of camera and the sparse and unstructured nature of LiDAR data.One direction of technology advancement that addresses the above challenge is the 3D occupancy grid representation of the scene [1, 2, 3, 4, 5, 6, 7, 8] . This method has exhibited promise in various aspects of mobile robots, encompassing 3D object detection, segmentation, and scene reconstruction, primarily due to their accuracy and computational efficiency in modeling the 3D environment. Compared with camera, LiDAR has unique advantages in accurate 3D perception and robustness to different lighting conditions. However, most of these works derive their 3D grid representations from cameras, leaving the exploration of grid representation from LiDAR data largely uncharted. Furthermore, we notice that very few of these works delve into scene forecasting, which takes a few historical frames and aims at predicting future frames using certain representations, except for e.g. [6] . However, in [6] , the occupancy grid is constructed using supervised learning where the annotation of dynamic objects are needed. More literature review and discussions about related works on 3D occupancy grid construction and forecasting are relegated to Appendix B.To fill in these research gaps, we propose LiDARGrid, a dense 3D opacity grid representation from LiDAR points. Motivated by optical models of volume rendering [9] and neural volume rendering, e.g. NeRF [10] , we conceptualize our representation as a 3D grid, where each voxel represents the LiDAR opacity, or more precisely, the incremental likelihood of a ray to stop marching by hitting some particle in that voxel. This representation allows for distance rendering of LiDAR points, such that one can compute the distance from any position to an object's surface within the grid, thus reconstructing the scene geometry. The proposed 3D opacity grid is generated from LiDAR points by an autoencoder-based densification network, trained by matching the rendered LiDAR distance with the ground truth distance. Though the densified grid may be converted to a binary-valued occupancy grid by simple thresholding, we keep it as a continuous-valued grid, which can be used as a universal representation of the 3D scene for diverse perception and forecasting tasks. For this reason, we name our representation as the 3D opacity grid.We explore the application of this representation in scene forecasting, where a forecasting network based on 3D convolution and UNET architecture takes historical grids as input and predicts future grids. Notably, both the densification network and the forecasting network are trained on unlabeled LiDAR points, adopting a self-supervised approach that can readily access to a vast pool of training data. The performance of scene forecasting is evaluated by the point cloud forecasting metrics. Khurana et al. [11] is the most recent work that predicts future point clouds with grid-based representation and known LiDAR pose. It shows much crisper results than previous point cloud forecasting works [12, 13] with model-free prediction and unknown LiDAR pose, partly due to the fact that the 3D grid better preserves scene geometry. Advancing along the direction of [11] by incorporating the densification network and the improved forecasting network, we achieve notable improvements in the point cloud forecasting metrics, thus pushing forward the state-of-the-art performance of this task. We also provide comprehensive ablation studies to evaluate each module and justify the key design choices toward the improved scene forecasting. Besides scene forecasting, we extend the utility of our 3D grid representation to diverse applications, including moving region detection and depth completion. In summary, our contributions include:1. We propose a 3D opacity grid representation for mobile robot perception. The representation can accurately capture the 3D scene geometry and can be generated in real time, which are crucial for various perception and forecasting tasks. The representation is learned solely based on unlabeled LiDAR data.2. We propose a 3D convolutional forecasting network for our opacity grid. It outperforms the state-of-the-art approaches on point cloud forecasting tasks in all performance metrics.3. We provide a comprehensive ablation study to elucidate the efficacy of our method, along with demonstrations of its potential in extended applications, such as moving region detection and depth completion. 
3d opacity grid representation: 
 Representation Formulation Our representation of the scene is inspired by the optical model for volume rendering used in computer graphics [9, 10] . It is assumed that the 3D space is permeated by particles that can scatter light, and the density of the particles varies across the space. We define the representation as a 3D grid of dimension H × W × L with H, W , L being positive integers, scaled by a factor of S ∈ R to align with the real-world metric in meters. In other words, S is the voxel size in meters. The origin of the real-world coordinate system is centered within this grid. The continuous value of a voxel at position (h, w, l) ∈ {1, . . . , H} × {1, . . . , W } × {1, . . . , L} in the grid, denoted as σ(h, w, l), represents We illustrate the efficacy of our representation through scene forecasting, that is to predict a sequence of future grids from historical grids through a 3D convolutional forecasting network, which is also supervised by the ray-distance loss. (c) Utilizing our representation, we perform multiple tasks, such as point cloud forecasting, movement detection, and depth completion.the density of particles at the real-world pointx hwl = S(w -W 2 ), S(l -L 2 ), S(h -H 2 ), which is approximately proportional to the number of particles within that voxel. This representation is deterministic, but it gives rise to a simple probabilistic model of LiDAR point rendering: suppose the particles are uniformly placed at random within each voxel, and independently across voxels, then as the voxel size S decreases to infinitesimal, σ(h, w, l) becomes the rate of change of the probability of a LiDAR ray hitting any particle in that voxel [9, 14] . With a normal sized grid, given a ray o + d • r with origin o and direction d, if the ray intersects with a sequence of voxels (v 1 , ..., v n ) at distance (r 1 , ..., r n ), the probability of the ray hitting a particle when traveling from v i-1 to v i can be derived as [14] EQUATIONWe see that σ essentially represents the opacity of each voxel: the larger the value σ(v), the more likely that a ray stops at voxel v. Hence we name the proposed representation as 3D opacity grid. Applying the above probabilistic point rendering model, the distance R that a ray travels before stopping at a voxel is a random variable, taking value r i with probabilityEQUATIONThe expected distance of a ray is thenEQUATION)The process of tracing these rays can be likened to simulating LiDAR beams as they traverse through open space and terminate upon encountering an object's surface. Utilizing Eq. 3, given the 3D opacity grid, we can compute the expected distance between the starting point and the end point of a LiDAR ray.To acquire the 3D opacity grid from LiDAR points, we first initialize a grid by mapping the LiDAR points to the grid. For a LiDAR point (x, y, z) represented in the real-world coordinate, we map it to a voxel grid at position⌊ z S + H 2 ⌋, ⌊ x S + W 2 ⌋, ⌊ y S + L 2 ⌋and set the voxel value to a constant σ 0 . Applying this process to all LiDAR points, we can initialize a sparse grid of spatial occupancy.The major challenge we address is generating a dense and continuous-valued opacity grid from the sparse initial grid. A naive way would be applying a low-pass filter to the sparse grid, however its rigidity hinders the geometric consistency of the filtered grid. With this in mind, we design a learnable autoencoder densifier:V dense = D(E(V sparse )) V sparse , V dense ∈ R H×W ×L (4)The encoder E(•) maps the sparsely initialized grid into a low-dimensional representation with a series of convolution layers, extracting low-frequency information from the input. The decoder D(•) upsamples the intermediate features with transposed convolution to reconstruct a grid with the same dimension as the input. Importantly, unlike the popular UNET architecture, we do not include skip connections between the encoder and decoder layers in order to retain more low-frequency information when passing through the network.Experiments show that this design choice is crucial for achieving a good densification performance. The network is trained in a self-supervised manner with the ray-distance loss:EQUATION)where P is a set of LiDAR points, r is the ground-truth distance of a point to the LiDAR origin o. E[R] can be calculated by Eq. 3 and is differentiable for backpropagation. We randomly rotate and translate the point cloud as data augmentation to acquire robust densification.The network eventually learns a pattern to map V sparse to a dense representation V dense . Furthermore, the small-size intermediate feature vector can be stored as a compressed version of the 3D grid, which largely improves the memory efficiency of this representation, and can potentially be even more memory-efficient than storing the LiDAR points of a scene.We apply our 3D opacity grid representation to scene forecasting and show that with accurate 3D geometry reconstructed, we are able to reach better performance on such tasks.The goal of scene forecasting is to predict how the surrounding scene evolves in the future. Encoding the scene with our grid representation, the problem can be defined as finding a forecasting model F(•), so thatEQUATIONwhere V i ∈ R H×W ×L ; T prev and T future are the numbers of history frames and future frames respectively. Following the setting of [11] , for each frame, we first transform the LiDAR points from their local sensor coordinate to the coordinate at frame t based on the knowledge of LiDAR pose in each frame. Then we use the transformed points to initialize and densify the grid as stated in Section 2.1. Note that this coordinate transformation is optional, but would be a reasonable way of using the proposed method in practice.Enlightened by the success of 2D convolution on image perception, we regard our 3D grid representation as a natural extension of the 2D occupancy grid and apply a 3D convolution network to predict future grids. It can preserve local spatial information while extracting important features due to its 3D shift equivariance. Our scene forecasting network F(•) is a UNET-style [15] 3D convolutional encoder-decoder network. Each pair of corresponding layers in the encoder and decoder with the same feature size are connected by a skip layer. Following Eq. 6, the input D prev ∈ R Tprev×H×W ×L and output D future ∈ R T future ×H×W ×L are series of temporal-continuous frames. Note that it is a general framework that allows various designs on the intermediate feature layer. For example, one can add LSTM modules [16] to process the latent features. As a baseline model, we are using identity mapping for simplicity.Loss Function An intuitive way to train is to use D future = (V t+1 , ..., V t+T future ) as supervision, where V i is generated from the future ground-truth LiDAR points:EQUATIONHowever, due to high dimensionality, we observe that the network tends to learn an average result with this loss function because occupancy change on an object's surface barely affects the overall loss. Therefore, we propose to use the sparse ground-truth LiDAR points directly as a weaker supervision:EQUATIONwhere L(P t+i , Vt+i ) shares the same definition in Eq. 5, P t+i is the set of LiDAR points at frame t + i. We compare these two loss functions in the experiment section.Point Cloud Forecasting Our method can be easily applied to point cloud forecasting tasks. Given future 3D grids (V t+1 , ..., V t+T future ) and LiDAR rays (o, {d i }), one can simulate future LiDAR points with volume rendering (Eq. 1 to Eq. 3). We follow [11] and set (o, {d i }) from ground-truth future LiDAR rays.In addition to scene or point cloud forecasting, our 3D opacity grid representation can be applied to other tasks. We describe two additional tasks: moving region detection and depth completion in the experiment section, and show promising results on both of them. 
setup: 
 Dataset Our experiments were mainly performed on the NuScenes dataset [17] , which is a public autonomous driving dataset containing 1000 driving sequences collected by 6 cameras, 1 LiDAR, and 5 RaDAR sensors. We only use LiDAR data, which has a frame rate of 2Hz with around 20000 points per frame. We follow the NuScenes' setting and split the dataset into 850 training scenes and 150 testing scenes. We train our densification network and forecasting network on the training set and show evaluation results on the testing set.Grid and Networks Details Following [11] , the grid covers a 4.5m × 70m × 70m 3D space. Specifically, we set the grid dimension and size to H = 45, W = 700, L = 700, and S = 0.1. Each voxel in the opacity grid is initialized with σ 0 = 1 if it contains a LiDAR point and 0 otherwise. Our densification network contains an encoder with 4 downsampling layers and a decoder with 4 upsampling layers. Each downsampling layer shrinks the grid size by 2 and doubles the channel size. Upsampling layers reverse this process. Our forecasting network shares a similar structure with 3D convolution and skip connections between each pair of encoder-decoder layers.To ensure a rigorous comparison, we adopt the identical evaluation metrics as presented in [11] . Specifically, we report the in-grid error (L1) and relative in-grid error (AbsRel), measuring the accuracy of distance predictions along LiDAR rays. Additionally, we employ the vanilla Chamfer distance (Vanilla CD) and in-grid Chamfer distance (In-grid CD) [11, 12] to gauge the spatial distribution error of the predicted point cloud, with measurements expressed in square meters (m 2 ).Baselines We benchmarked our method against the most recent state-of-the-art approaches in point cloud forecasting, including S2Net [12] , SPFNet [13] , and 4DOcc [11] . Notably, both S2Net and SPFNet utilize the range map [18] as the scene representation, translating the 3D point cloud into 2D and subsequently employing 2D convolution techniques. 4DOcc employs a similar grid representation but distinguishes itself by using sparse grids acquired from LiDAR points as input, and further processing them through 2D convolution. Quantitative Results Table 1 presents a comprehensive comparative analysis between our method and the baseline models. Notably, our approach excels in reducing the average L1 distance error, showcasing an improvement of 0.28m and 0.21m for the 1s and 3s settings, respectively, when contrasted with the current state-of-the-art. Most notably, our method demonstrates a substantial reduction in the in-grid Chamfer distance error, achieving a significant improvement of 50% and 29% for the 1s and 3s settings, and simultaneously lowering the vanilla Chamfer distance error by 38% and 50% for the same settings. This robust performance not only attests to our method's proficiency in predicting the scene's underlying geometry (L1 and AbsRel) but also underscores its capacity to predict uncorrelated samples (Chamfer Distance), such as point clouds. Our method achieves this remarkable improvement in Chamfer distance, despite not having employed it as direct training supervision, underscoring the accuracy and robustness of our approach in predicting future scene geometry.The proposed pipeline can run at 5 frames/second on a Quadro RTX 6000 GPU, being able to support real-time applications. Further improvements may be gained from network optimization and CUDA acceleration.Several key designs contribute to the improved scene forecasting, including volume densification, 3D convolutional forecasting network, and its loss function. To thoroughly evaluate the effect of these modules, we conduct a comprehensive ablation study on them.We argue that inputting dense and geometry-preserved 3D representation to the forecasting network is essential to the forecasting performance. In this section, we train and test our forecasting network with or without volume densification. For the 'without' setting, we simply initialize the grid with LiDAR points introduced in Section 2.1 and input it into the forecasting network, the same as what is done in [11] .Table 2 shows the forecasting results with and without volume densification. Upon introducing volume densification, we observed consistent improvements across all evaluation metrics for both 1s and 3s settings. These improvements underscore the consistent effectiveness of the volume densification module in enhancing the forecasting task. It is noteworthy that volume densification offers benefits beyond scene forecasting. We illustrate this through visualizations of the opacity grid, both before (top-right) and after (bottom-right) densification in Figure 2 , revealing a densified 3D grid with fewer holes and a more continuous spatial distribution. This not only bolsters forecasting but also enhances its applicability to broader tasks, as discussed in Section 3.4, such as moving object detection.We state in section 2.1 that the encoder of the densification network should be a low-pass filter. Since the supervision is sparse, letting high-frequency signals pass through the network will cause overfitting. We will show this by comparing the network with or without skip connections. Specifically, for the network with skip connection:EQUATIONwhere M(•) is a linear mapping, others share the same definition with Eq.(4).Figure 2 shows that the network with a skip connection generates grids with holes, where the occupied voxels are distributed discretely since the network is overfitting the sparse input. The network without a skip connection maps the input to a low-dimensional manifold and generates a continuous and dense grid after decoding.In section 2.2, we introduced two distinct loss functions, namely L voxel and L ray , employed in training the forecasting network. In this section, we delve into a comparative analysis of the performance outcomes associated with these two loss functions. In both scenarios, training is carried out until convergence is achieved. For the L voxel setting, we apply the L1 loss to all voxels within the grid, utilizing the 'sum' reduction. The learning rate is set to 1e-6, with other conditions held constant.Table 3 shows the results of applying these two loss functions. We show that compared to L voxel , using L ray will drastically reduce the prediction error, on both L1 metric and chamfer distance. It is because, in areas obstructed from LiDAR rays, small noises are introduced due to the absence of information in the LiDAR-invisible regions. Employing L voxel compels the model to accommodate these inaccuracies, leading to confusion and a tendency to neglect regions with accurate LiDAR supervision. Conversely, the use of L ray mitigates this issue, resulting in a more accurate and robust forecasting model.Moving Region Detection Through the alignment of two temporally adjacent 3D opacity grids within the same coordinate system, our method enables the detection of moving regions in the scene without relying on any object detection module. Traditional methods for this purpose e.g. [19] face challenges in effectively aligning LiDAR rays across consecutive frames due to the stochastic nature of ray directions over time. Our representation, as it can reconstruct dense scene geometry, overcomes this limitation. Figure 3 shows an example, where points on moving pedestrians are clearly detected by our method.Depth Completion Existing camera-LiDAR-based depth completion require sophisticated fusion pipelines. Meanwhile, LiDAR alone offers only sparse depth maps. With our method, one can simply use the dense opacity grid generated from LiDAR to render depth values for any virtual camera pixel, following the principles outlined in Eq. 1 through 3. The promising performance of our method can be visualized in Figure 4 . Conventional LiDAR-based 3D occupancy representations mainly appear in the 3D mapping and segmentation literature, as reviewed in [23] , and are far less studied compared to a large volume of literature on 2D occupancy from range sensors [24] . Most LiDARbased 3D occupancy relies on approaches such as kernel methods and graphical models or other Bayesian methods to densify LiDAR point clouds, mostly resulting in a continuous volume representation. LiDAR-based 3D occupancy grid first appears in [11] , however, the representation is solely constructed by placing point cloud in the grid, hence sparse. Our 3D opacity grid representation is directly tied to the volume rendering model in computer graphics [9, 10] , where it is assumed that the space is permeated by particles, and the value of each voxel in the grid represents the density, proportional to the number, of particles in that voxel. The representation is constructed by densifying LiDAR point cloud using a carefully tailored neural network, hence dense, and it can run in real time.Methods of obtaining ground-truth 3D occupancy grid from camera and LiDAR are discussed in [1, 25] . In these works, the moving objects are first detected and then separated from static backgrounds, so that the LiDAR point clouds on different types of objects can be identified and averaged out differently. In contrast, the training of the neural networks we use for opacity grid generation and forecasting does not rely on object annotation at all, and the trained densification network produces a dense opacity grid solely based on the LiDAR point cloud in the current frame.Rigorously speaking, our 3D opacity grid is not equivalent to a 3D occupancy grid; instead, it is more of an intermediate and universal representation of the 3D scene. Nevertheless, as the density of particles in a voxel reflects the extent to which that voxel is occupied, an opacity grid may be converted to an occupancy grid by thresholding the value of each voxel into a binary value. The quantitative comparison between the 3D opacity grid of a scene generated by our method and its ground truth 3D occupancy grid is left as a future study. Some relevant theoretical and experimental investigations to this problem can be found in [26] and the reference therein.Some recent works propose to generate 3D scene representations e.g. triplane, NeRF or Gaussian splatter, by directly feeding images to trained neural networks, including [27, 28, 29, 30, 31] . Conceptually our method is closely related to these works, in that we directly generate a 3D opacity grid by feeding LiDAR points to a densification network. An important feature of our method is that we explicitly use the NeRF-like volume rendering model to derive the ray-distance loss for training, which distinguishes our method from the pure model-free methods.The problem of occupancy grid forecasting is to predict how the occupancy grid evolves in time. It arises when the scene contains moving objects. The majority of existing works toward this problem consider the prediction of 2D dynamic occupancy grid [32, 33, 34, 35, 36] . The prediction of a scene represented by a 3D occupancy grid appears to be a new problem, and is initiated by [11] . Different from conventional object-based scene prediction, the occupancy grid prediction under consideration is self-supervised, meaning that there is no need to perform object detection and tracking, and unannotated sensor data is directly used for prediction.We have to distinguish the prediction problem studied in this work, which is about estimating future scene, from many existing works titled in terms of occupancy grid prediction, which are about scene completion or estimating occupancy of occluded areas. Works in the latter category include [37, 3, 22, 25] . To this end, we term the problem under study as opacity grid forecasting.Point cloud forecasting has also arisen as a new problem in computer vision for robotic perception in recent years [12, 13, 11] , as it has the potential to benefit downstream tasks with a large volume of unannotated LiDAR data. Earlier works on this problem [12, 13] do not separate sensor movement from the scene and predict the point cloud in a fully model-free manner. The recent work [11] reformulated this problem with a fixed reference frame, but leveraging known sensor pose, and improved the forecasting accuracy by leveraging the 3D occupancy grid representation. We follow the direction proposed in [11] , and propose new methods for 3D opacity grid construction and forecasting to further improve the performance. Another closely related work is [38] , where an implicit continuous occupancy field is learned without supervision for point cloud forecasting; in contrast, we utilize the discrete grid representation and explicitly forecast the grid. Some more recent works on point cloud forecasting also rely on camera-based object detection, e.g. [39] .Another advantage of working with LiDAR data is that the time evolution of the point cloud can provide information on object movement in 3D space. This problem has been studied in [40] , where an end-to-end framework on point cloud flow is proposed. Some recent works on 3D occupancy completion and annotation also take the flow into consideration, by estimating the velocity in 3D of each voxel in the grid [1, 25] . There are also plenty of works on motion estimation from 2D dynamic occupancy grid, e.g. [33] for an end-to-end learning-based approach and the references therein. In those works, a challenging issue is the extreme imbalance between static and dynamic cells, and pixel-wise balancing is usually applied in the loss function counteracting. We also propose a solution for moving region detection based on our densified 3D opacity grid, where we synthesize the point cloud in the previous scene but in the ray directions of the current scene, and detect the distance change of the point cloud which gives hints on the a moving region. 
 
 Appendix:  


Decision: This paper proposes a 3D opacity grid representation learning approach for LiDAR points for scene forecasting. The reviewers find the paper generally well written and to be addressing an important problem. They appreciate the simplicity of the approach and the good results. However, the reviewers have also raised several concerns including lacking comparisons with state-of-the-art methods, concerns on the effectiveness of the dense voxel representation, unclear relation to certain existing work which raises the question of novelty, experiments lacking in-depth analysis, lacking quantitative comparisons, among others. Importantly, related work. conclusions, and limitations sections are missing. Each of the reviewers has listed several questions to be addressed in the rebuttal.
Post-rebuttal: Most of the reviewers' concerns have been sufficiently addressed. Please address the remaining suggestions in the final version.

Reviews: review: title: title: Good paper on point cloud forecasting
paper_summary: paper_summary: A self-supervised lidar pointcloud forecasting method with a learned 3d opacity voxel representation of lidar point clouds, and a convolutional forward prediction network. The method achieves SOTA on nuScenes lidar forecasting task.
main_review: main_review: Strengths:
- The paper addresses an important open problem
- The proposed method is conceptually simple yet appears to be effective
- Strong results that improve SOTA of the nuScenes lidar forecasting task 
- The paper is well written and easy to follow, with a few gaps discussed below

Weaknesses:
- There is no related work section. Some related work is mentioned in the introduction but not to a sufficient depth. In particular it would be important to better understand how exactly the proposed method differs from the baselines (SPFNet, 4dOcc); what other pointcloud representations have been proposed before; and discuss pointcloud forecasting with different input modalities.
- There is no conclusion section, and limitations are not discussed.
- It is unclear to me if the baselines in Table 1 represent the SOTA, it would be useful to justify the choice of baselines, and discuss if other better performing methods exist (potentially with different inputs / non-grid-based representation). For example, VIDAR [1] has demonstrated better results than 4dOcc with visual input. 
- Ablation results in Table 2 reveal that the contribution of learned dense voxel representation is not substantial (SOTA is achieved even without volume densification). I wonder where does the performance gain come from then? 

Minor:
- In line 131, is the learned representation indeed smaller to store than to original point cloud? This would depend on the size of reduced voxel, and the number of feature dimensions.
- it would be good to elaborate on what makes the learned voxel "dense" if it is only supervised with the same sparse point cloud? I wonder if randomly masking out parts of the input pointcloud (i.e. using a masked autoencoder) could further help the network to learn a densified representation
- it would be helpful to explain the metrics in more details

[1] Yang, Zetong, et al. "Visual point cloud forecasting enables scalable autonomous driving." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
questions: questions: See weaknessess listed above.
limitations: limitations: 1
review_summary: review_summary: Good paper on point cloud forecasting, i recommend to accept given that the current weaknesses can be addressed.
title: title: LiDARGrid Review: Novelty and Experiment concerns.
paper_summary: paper_summary: Authors propose a combination of Unet-style 3D network and learnable autoencoder for various operations on voxel-grid scenes.
main_review: main_review: The paper's strength is the novel viewpoint with respect to what voxel grids with density values actually are.

Unfortunately, while viewing a density value as proportional to opacity in interesting, it is unclear how this redefinition of the representation leads to a novel method / improved performance. The authors make the difference to an occupancy grid clear, however the difference with respect to papers that provide values for each voxel - whether density, features created by a small pointnet applied to the points in the voxel, etc, is unclear.

As a direct example, 3D Semantic Segmentation with Submanifold Sparse Convolutional Networks (by Benjamin Graham; Martin Engelcke; Laurens van der Maaten) was a CVPR paper in 2018 which also converts point clouds to a voxel grid, and creates a density-based value for each voxel before applying a 3D Unet. However there are countless others which also fit this (broad) description.

Specifically, their proposed method is:

1) Authors first take a standard voxel density grid and redefine it as an "opacity grid".
2) A learnable autoencoder is used to densify the grid, as a single LiDAR Scan results in a sparse scene.
3) Either a UNET-style 3D convolutional network predicts future grids, or predicts moving regions, or completes a sparse depth map.

None of these steps appear particularly novel, other than being applied to scene forecasting instead of an alternative 3D scene task.
questions: questions: There are many concerns, but the main issues from my perspective are that:

1) An opacity grid as described is just a voxel grid with each point having a density/opacity value. This is not a novel concept, with many existing works operating on voxel grids where the voxels are not simply binary "occupied or unoccupied".
2) The main contribution is scene forecasting, however the method to achieve this is simply a Unet style autoencoder with no notable technical novelty. While demonstrating that a voxel based representation outperforms e.g. 2D ones is interesting, that alone does not feel sufficient as a contribution.
3) For other proposed applications qualitative data is provided without either quantitative results or comparison to existing methods. While sufficient to demonstrate broad application potential, it does not demonstrate whether the proposed method is competitive for those applications.

There are also various minor areas to address, from occasional grammar mistakes, to the confusion created by figure 2 being coloured based on distance, despite being an "opacity grid". At first I was expecting it to be coloured based on opacity. Also unsure why the "sky" in figure 2 is dark when in theory it should be both completely opaque and have a distance of infinity.

But those minor areas are just areas to consider updating for any final edits, it's the main three which prevent me from being able to accept this paper for submission.
limitations: limitations: 1
review_summary: review_summary: Method does not appear novel, nor are experiments extensive enough to accept as an in-depth analysis/ablation of applying 3D convolution Unet to various 3D scene problems.
title: title: Weak accept with further questions.
paper_summary: paper_summary: This paper presents a new 3D opacity grid representation derived from LiDAR points with neural volume densification and differentiable volume rendering to support tasks like scene forecasting.
main_review: main_review: This paper presents a new 3D opacity grid representation derived from LiDAR points with neural volume densification and differentiable volume rendering to support tasks like scene forecasting, moving region detection, and depth completion. This innovative approach merges the concepts of neural radiance fields and explicit LiDAR point clouds to provide a novel form of 3D neural representation.

### Strengths:
* The 3D opacity grid combines the idea of neural radiance fields and explicit LiDAR point clouds, brings a new paradigm of 3D neural representation.
* The new representation is tested on 3D scene forecasting task and achieved superior performance.
* The presentation of this paper is generally clear and easy to follow.

### Weaknesses:
* Authors provided qualitative showcases of additional tasks like moving region detection and depth completion. It would be better to also have quantitative comparisons.
* Explicit `Limitations` section would be great.
* Some additional clarification and questions need to be answered.
questions: questions: * Is it possible to move the `related works` section to the main paper?
* In the self-supervised training of the densification network, does all the test points in `ray-distance loss` also inputed to the encoder? If so, will this harm the generalization ability? More analysis is appreciated.
* In the line `155`, what exactly is the `identity mapping`? need more details.
limitations: limitations: 1
review_summary: review_summary: Generally acceptable but need further clarification and information.