Towards Flexible 3D Perception: Object-Centric Occupancy Completion Augments 3D Object Detection

Authors: chaoda  zheng
feng  wang
naiyan  wang
shuguang  cui
zhen  li
  sse
  cuhk-shenzhen
xiaomi  ev
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-12 12:29:44

Abstract: While 3D object bounding box (bbox) representation has been widely used in autonomous driving perception, it lacks the ability to capture the precise details of an object's intrinsic geometry. Recently, occupancy has emerged as a promising alternative for 3D scene perception. However, constructing a high-resolution occupancy map remains infeasible for large scenes due to computational constraints. Recognizing that foreground objects only occupy a small portion of the scene, we introduce object-centric occupancy as a supplement to object bboxes. This representation not only provides intricate details for detected objects but also enables higher voxel resolution in practical applications. 
We advance the development of object-centric occupancy perception from both data and algorithm perspectives. On the data side, we construct the first object-centric occupancy dataset from scratch using an automated pipeline. From the algorithmic standpoint, we introduce a novel object-centric occupancy completion network equipped with an implicit shape decoder that manages dynamic-size occupancy generation. This network accurately predicts the complete object-centric occupancy volume for inaccurate object proposals by leveraging temporal information from long sequences. Our method demonstrates robust performance in completing object shapes under noisy detection and tracking conditions. Additionally, we show that our occupancy features significantly enhance the detection results of state-of-the-art 3D object detectors, especially for incomplete or distant objects in the Waymo Open Dataset.

Main paper: 
 claims: 
 Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: Our main contributions include the presentation of an object-centric occupancy dataset and a shape-completion method, which are accurately reflected in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
3d occupancy prediction and shape completion: 
 3D semantic occupancy prediction (SOP) [15, 27, 32, 28, 11] has become a critical task in visioncentric autonomous driving, where algorithms primarily perceive the environment using RGB cameras. These vision-centric models typically discretize the surrounding environment into a volumetric grid and predict the occupancy status of each voxel by properly aggregating information from single-/multi-view RGB image(s). For occupied voxels, the models additionally predict the corresponding semantic class. Another similar task is 3D semantic scene completion (SSC) [24] . Unlike SOP, which only needs to predict the occupancy for visible regions, SSC additionally requires the model to determine the occupancy status at unseen regions. It is worth noting that although SOP and SSC are predominantly associated with vision-centric approaches, they are also applicable to sparse LiDAR or multi-modal inputs [33, 34] . Existing SOP and SSC methods primarily focus on scene-level occupancy, while our work concentrates on object-centric occupancy for better shape representation. Besides, semantics for occupied voxels are not necessary for our setup, as our primary concern is the geometric structure within an object bbox, whose class label is given. Unlike our occupancy-based method, a majority of shape completion approaches focus on surface reconstruction of objects [1, 21] . However, surface-based representations are less suitable for autonomous driving perception, as they do not directly support tasks like collision avoidance.As demonstrated in [36, 39, 22] , a single-frame detector can directly benefit from temporal information by taking the concatenation of several history frames as inputs. Although such a simple multi-frame strategy shows noticeable improvements, the performance becomes easily saturated as the number of input frames increases (e.g.,2 ∼ 4 frames). Besides, the computational cost grows significantly as the number of input frames increases, which is not ideal for real-time applications.To remedy this issue, [9] employs a residual point probing strategy to remove redundant points in the multi-frame inputs. Besides, [3] opts for an object-centric approach that conducts the temporal aggregation at the level of tracklet proposals, which allows for longer sequences (i.e.,16 frames) to be processed with lower computational costs. Furthermore, [19, 8] demonstrate human-level detection performance by leveraging past and future information of entire object tracks. However, they are limited to offline applications since they require access to future frames. More recently, MoDAR [13] improves detection by augmenting LiDAR point clouds using motion forecasting outputs, which consist of future trajectory points predicted from long history subsequences (i.e., 90 frames). Compared to MoDAR [13] , our method is able to aggregate all the historical information via the compact implicit latent embeddings. Besides, our method goes beyond detection by predicting the complete object-centric occupancy for each proposal.Implicit shape representation [12] represents 3D shapes with a continuous function. Compared to traditional explicit representations (e.g., point clouds, meshes, volumetric grids), implicit representations can describe shape structure in continuous space, and are more memory-efficient. Rather than manually designing the implicit function, recent works [16, 17, 37, 20] propose to learn the implicit function from data. Specifically, they employ neural networks to approximate the implicit function, which can be trained in a data-driven manner. These neural functions typically take continuous 3D coordinates as inputs and output the related shape attributes at the queried positions (e.g., color, density, signed distance, etc.) For example, [17] learns a signed distance function (SDF) from high-quality 3D meshes for better shape representation. While [16] learns a neural radiance field from multi-view images to achieve better view synthesis. Our implicit shape decoder shares similarities with DeepSDF introduced in [17] . However, instead of predicting the signed distance at a queried position, we predict its occupancy probability.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Discussed in the "Limitations" section in the appendix. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
object-centric occupancy dataset: 
 High-quality datasets are critical for learning-based methods. However, existing datasets do not satisfy our requirements for object-centric occupancy perception due to unaligned coordinate systems and reduced resolutions. We discuss these limitations and introduce our automated annotation pipeline in the following subsections.Occupancy representation discretizes the 3D space into a volumetric grid, wherein each voxel is classified as occupied or free. Given our objective is to more accurately represent complex object structures, background elements -despite their extensive coverage -are not our primary focus. Therefore, we define object-centric occupancy as a 3D grid centered on the object's coordinate. As different object instances vary in size, their corresponding occupancy resolutions also vary, if given a predefined voxel size. In contrast, existing scene-level occupancy datasets [27, 32, 31] use an occupancy volume to represent an entire scene centered at the ego vehicle's coordinate system. Since all scenes are bounded by a fixed range, the occupancy resolution remains constant when the voxel size is given.One convenient way to construct our object-centric occupancy dataset is to extract object occupancy from existing ego-centric datasets using object detection annotations. However, this approach has two significant limitations. Firstly, as scene-level occupancy is centered at the ego vehicle's coordinate system, the extracted object voxels may appear jagged due to coordinate misalignments, as shown in Fig. 3 . Transforming these jagged object voxels to the object's coordinate system inevitably leads to information loss. Secondly, existing scene-level datasets have adopted a large voxel size (e.g., (0.4m) 3 ) to save computational costs for large scenes. However, this voxel size is inadequate for capturing the fine-grained details of objects, especially for smaller objects. For this reason, we introduce an automated pipeline to annotate the object-centric occupancy dataset from scratch. Similar to previous scene-level approaches [27, 31] , we can construct object-centric occupancy annotations based on any existing 3D detection datasets [25, 2] . However, instead of generating an occupancy volume for the entire scene, we create it for each annotated object instance under its local coordinate system.For each designated object, we gather points within its annotated bounding boxes over time, transform these points from sensor coordinates to the bounding box coordinates and aggregate them into a dense point cloud. After that,we directly voxelize it under the local object coordinate system, yielding the object-centric occupancy grid.Additionally, we perform occlusion reasoning to classify unoccupied voxels as either free or unobserved by comparing each voxel center's range value to raw range images from LiDAR scans. This strategy is significantly faster than traditional ray-casting [27] . After finishing the annotation, every tracked object within the detection dataset is associated with an object-centric occupancy grid. This grid is centered at the local coordinate and has a size determined by the object's size and the desired resolution. Please refer to Appendix A.1 for more details about the dataset generation pipeline.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: No theoretical results are presented in the paper. Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
sequence-based occupancy completion network: 
 Fig. 4 illustrates the architecture of our object-centric occupancy completion network. Our method utilizes an object sequence as input, formulated as a {(P t , B t )} T t=0 , where P i ∈ R N ×3 is the point cloud at timestamp t and B t ∈ R 7 is the corresponding noisy 3D object bbox. The input sequence can be generated using off-the-shelf 3D detection [36, 6] and tracking [30] systems. Our main objective is to predict the complete object-centric occupancy grid for each proposal in the trajectory. Additionally, we use the occupancy features to further refine the detection results of the 3D detector. Our network primarily focuses on Regions of Interest (RoIs) defined by object proposals. Given that different objects have varying sizes and proposals for the same object may also vary due to inaccurate detection, efficiently decoding the occupancy volume from feature space for each dynamic-sized proposal poses a significant challenge. Conventional scene-level occupancy perception approaches [28, 32] typically apply dense convolution layers to decode the occupancy volume. However, this strategy encounters several limitations in the context of dynamic-size object-centric occupancy. First, since we require feature interaction across timestamps, the features for different proposals are better if in the same size. However, decoding a dynamic-sized volume from a fixed-size feature map is non-trivial for convolution. Secondly, the dense convolution operation becomes computationally expensive for high occupancy resolution. One alternative is sparse convolution [5, 10] , however, it cannot fill the unoccupied voxels with the correct occupancy status.Drawing inspiration from the recent success of implicit shape representations [16, 17] , we tackle the aforementioned challenge through an implicit shape decoder D. This decoder is capable of predicting the occupancy status of any position within the RoI based on its corresponding latent embedding. Specifically, the decoder takes in the latent embedding z along with a query position q ∈ R 3 at the RoI coordinate, and subsequently outputs the occupancy probability at q:EQUATIONwhere D : R e × R 3 → R [0,1] is implemented as an MLP. The latent z ∈ R e is a fixed-length embedding depicting the geometrics within the RoI. The latent z and query position q are concatenated before being sent to D. Besides enabling flexible feature interaction and efficient computation, the implicit shape decoder also allows for easier occupancy interpolation or extrapolation with continuous query positions.Having the implicit shape decoder in place, the next step is to obtain a latent embedding z that accurately represents the complete object shape within the RoI. To achieve accurate shape completion and detection, two information sources are essential: 1) the partial geometric structure of each RoI, and 2) the motion information of the object over time. To make different RoIs share the same embedding space, we encode each RoI under a canonical local coordinate system. However, transforming the RoI to the local coordinate system inevitably loses the global motion dynamics of the object, reducing the network's ability to handle detection drifts. Therefore, we encode each RoI using two separate encoders: E local that encodes the RoI in the local coordinate system and E global in the global coordinate system.Specifically, we employ the sparse instance recognition (SIR) module in FSD [6] as our RoI encoder. SIR is a PointNet-based network [18] characterized by multiple per-point MLPs and max-pooling layers. Drawing inspiration from LiDAR R-CNN [14] , we additionally enhance the point cloud with size information of the RoI. This augmentation involves decorating each point within the RoI with its offset relative to the boundary of the RoI, enabling it to be box-awared. All points are transformed to the local coordinate system defined by the detected bounding box before being sent to E local . Conversely, E global directly encodes the RoI in the global coordinate system. For a given object sequence {(P t , B t )} T t=0 , we separately encode each RoI using E local and E global , yielding two sets of latent embeddings Z l and Z g ∈ R T ×e .After RoI encoding, we use the motion information from Z g to enrich the local shape latent embeddings Z l . First, we employ a transformer mechanism [29] to Z g to enable feature interaction across timestamps. To ensure online applications, we restrict each RoI feature in Z g to only attend to its historical features, thereby preventing information leakage from future timestamps:EQUATIONwhere CausualAttn is a causal transformer that restricts the attention to the past timestamps. γ(•) is a sinusoidal positional encoding [29] that encodes the temporal timestamp T ∈ R T ×1 . ϕ(•) is a learnable MLP that encodes the bbox information B ∈ R T ×7 in the global coordinate system.Next, we fuse the enriched global latents Z ′ g with the local latents Z l to obtain the final latent embeddings Z ∈ R T ×e :EQUATIONwhere 'Concat' denotes the concatenation operation, and 'MLP' is a multi-layer perceptron that projects the concatenated features to the desired dimension c.Given the final latent embeddings Z, we can predict the complete object-centric occupancy volume for each proposal by querying the implicit shape decoder D at different positions. During training, we randomly sample a fixed number of query positions within each RoI to compute the loss. During inference, we query the decoder at all voxel centers within the RoI to obtain the complete occupancy volume. Since Z now encodes information of complete object shapes, it provides more geometric information for better detection. To retain motion information, we additionally fuse Z with the global RoI feature Z g :EQUATIONThe fused feature Z det is then fed into a detection head for bbox and score refinement (Fig. 4 ).The overall training loss consists of three components: the occupancy completion loss L occ , the bbox loss L det , and the objectness loss L score :EQUATIONwhere λ det = 2 and λ score = 1 are hyperparameters that balance the three losses. We use the binary cross-entropy loss for L occ and L score , and the L1 loss for L det .5 ExperimentsQuestion: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? Answer: [Yes] Justification: Please see the "Implementation Details" section for more information. Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
implementation details: 
 Position Query Sampling. During training, we randomly sample 1024 voxel centers and corresponding occupancy statuses from each annotated occupancy as the position queries. To ensure the occupancy prediction is not biased, we adopt a balanced sampling strategy, where 512 points are sampled from the occupied voxels and 512 from the free voxels. For an RoI that matches a ground-truth (GT) bbox, we transform the corresponding query set to its coordinate system using the relative pose between the RoI and the bbox. These position queries are then sent to the implicit decoder D to compute the occupancy loss. During the inference, we generate the dense occupancy volume for each RoI by querying the decoder at all voxel centers within the RoI under the local coordinate system.Network Training. In order to generate inputs for our network, we first use FSD [6] and Center-Point [36] as our base detectors to generate object proposals. Then we leverage ImmortalTracker [30] to associate the detection results into object tracklet proposals. We use the generated object tracklet proposals in addition to GT tracklets as our training sequences. To facilitate parallel training, we regularize each tracklet to a fixed length of 32 frames via padding or cutting during training. To achieve faster convergence, we compute the loss at all timestamps within each tracklet instead of only at the last one. During the inference, the model outputs the refined box at timestamp t by looking at all the history boxes.Dataset. Our method is evaluated on the Waymo Open Dataset (WOD) [25] . We use the official training set, comprising 798 sequences for training, and 202 sequences for evaluation. We apply our automatic pipeline on WOD to construct the object-centric occupancy annotations with the voxel size set to 0.2m. All experiments are conducted on rigid objects (i.e., vehicles) to ensure accurate evaluation of shape completion using our annotated ground-truths. Evaluation Metrics. For shape completion, we adopt the widely-used intersection-over-union (IoU) to evaluate the quality of the predicted occupancy volumes. Due to the object-centric nature of our method, we cannot calculate the IoU directly between the predicted and the ground-truth occupancy volumes because they are in different coordinate systems and may have different sizes (noisy RoI vs. GT box). To overcome this issue, we employ a two-step process as illustrated in Fig. 5 . Firstly, we transform the ground-truth (GT) box to the coordinate system of the RoI using the relative pose. This transformation aligns the GT box with the RoI, enabling a consistent comparison. Subsequently, we determine the predicted occupancy status of each voxel center within the transformed GT box. For voxels falling inside the RoI (hit), their occupancies are determined by the corresponding predicted occupancies within the RoI. On the other hand, voxels located outside the RoI (missed) are considered as free. By applying this process, we construct a predicted occupancy volume within the GT box. Finally, we compute the IoU by comparing the predicted occupancy volume in the GT box with the ground-truth occupancy volumes. During the IoU calculation, we ignore unobserved voxels in the GT volume for a fair assessment. Besides, RoIs that do not intersect with any GT boxes are excluded from the evaluation. We also report mean IoU that is respectively averaged at track and box levels to provide a more detailed evaluation. "-E" denotes using GT bbox which may outside the predicted RoIs.For object detection, we adopt the official 3D detection metrics in WOD [25] , including Average Precision (AP) and Average Precision Weighted by Heading (APH) at IoU thresholds of 0.7 for vehicles. Meanwhile, based on the number of points contained within each object, the data is divided into two difficulty levels: LEVEL 1, where the number of points is greater than 5, and LEVEL 2, where the number of points is between 1 and 5.Comparison against Baseline. Since object-centric occupancy is a novel task, no learning-based methods can be used for comparison as far as we are acknowledged. We compare our method with the baseline that directly accumulates and voxelizes the history point clouds within the noisy tracklet proposals. We evaluate the shape completion performance on three types of tracklet inputs: ground-truth (GT) tracklets, tracklets generated by CenterPoint (CP) [36] , and tracklets generated by FSD [6] . As shown in Tab. 1, the shape completion performance is strongly correlated with the quality of the input tracklets, where better tracklets lead to better shape completion. In all cases, our method outperforms the baseline, even when the input tracklets are noise-free GTs. This is because our method can effectively complete the object shape even at early timestamps by leveraging learned knowledge from training data, whereas the baseline only becomes effective at later timestamps when more views are visible.Robustness. To simulate unsatisfied detection and tracking results, we add some slight noise to GT box proposals. From Tab. 1 we can find that the baseline performance drops significantly (>10% IoU), while our method maintains a stable performance in this case (<5% IoU), demonstrating the robustness of our method to noisy inputs. Compared to noisy GT tracklets, tracklets generated by CP and FSD may additionally contain mismatched or missed targets, leading to a more significant performance drop from the baseline. In contrast, our method demonstrates its strong robustness to these noisy and inaccurate tracklets. These results indicates that our method can effectively complete the object shape even when the input tracklets are noisy or inaccurate.Results with GT bbox. Thanks to the implicit shape decoder, our method has the potential to predict the occupancy status at any position even "outside" the RoI, which is non-trivial for the baseline or CNN-based methods. To demonstrate this ability, we conduct an experiment by querying the implicit decoder at all voxel centers within the GT box (even for those outside the RoI). Unlike our standard evaluation, where we simply treat the missed positions as free (see Fig. 5 ), we query the implicit decoder at these positions to obtain the predicted occupancy status. As shown in Tab. 1, the shape completion performance is further improved when considering the extrapolated results outside the RoIs (Ours-E), demonstrating the flexibility of our implicit shape representation.Generalization. The last row in Tab. 1 presents occupancy completion results obtained by directly applying our trained model to the tracklet proposals generated by FSDv2 [7] . Due to better detection, our method with FSDv2 still outperforms the version with CenterPoint even without retraining. However, it performs slightly worse compared to using FSD tracklets, despite FSDv2 having better detection results than FSD. This indicates that significant detection improvements generally lead to better shape completion (FSDv2 vs. CenterPoint). However, for detectors with similar performance (e.g., FSD vs. FSDv2), improved detections do not necessarily guarantee better shape completion without retraining..Tab. 2 presents the 3D detection results on the WOD validation set. Significant improvements are observed when applying our methods to the tracklet proposals generated by CenterPoint [36] and FSD [6] . Compared to the previous state-of-the-art MoDAR [13] , our method achieves notably greater enhancements on 1-frame CenterPoint (e.g., 8.6% vs. 3.2% improvement in L1 AP). Moreover, our method can be seamlessly integrated with other stateof-the-art detectors without requiring retraining on their respective tracklets in the training data. For example, applying our method (trained on CP and FSD tracklets) to FSDv2 [7] yields significant improvements, showcasing the strong generalization capability of our approach. Applying our method to a more advanced detector, 1-frame FSD [6] , still results in a noticeable improvement. This enhancement is more significant compared to adding MoDAR to a detector with similar performance (i.e., 3-frame SWFormer [26] ). Furthermore, we achieve new state-of-the-art online detection results by applying our method to 7-frame FSD, attaining 83.3% AP and 75.7% APH on L1 and L2, respectively. This indicates our method's effectiveness in aggregating long-sequence information for object detection in addition to shape completion.Range Breakdown. Distant objects are more challenging to detect due to their sparsity. We further analyze the detection performance across different distance ranges. As shown in Tab. 3, our improvements over the base detector become more pronounced as the distance increases. This indicates that our method effectively addresses the sparsity issue in distant objects via shape completion.In this section, we evaluate different design choices in our method and analyze their impact on the shape completion and detection performance. All the results are based on 1-frame FSD [6] Explicit vs. Implicit. We then attempt to refine detection results using the explicit occupancy predictions. Specifically, we sample occupied voxel centers from each predicted occupancy volume and apply the RoI encoder E global to generate the final feature used for detection (more details are in the Appendix A.2). However, as demonstrated in Tab. 4, this strategy leads to a significant performance drop. Due to the non-differentiable nature of the occupancy sampling process, the detection errors cannot be back-propagated to other components when relying on explicit occupancy predictions, resulting in unstable training. In contrast, our implicit shape representation allows for joint end-to-end training of shape completion and detection, leading to better performance.Occupancy Helps Detection. Finally, we evaluate the impact of the occupancy task on detection performance. We removed the OCC head from our full model and retrained it using only the detection loss. As shown in the last row of Tab. 4, the absence of the occupancy decoder results in a noticeable decline in detection performance. This suggests that the occupancy completion task not only explicitly enriches the object shape representation but also enhances detection by contributing additional geometric information to the latent space.Training & Testing Length. Tab. 5 shows how the sequence lengths affect the performance of our method. We retrain our method using 8-frame and 16-frame tracklets, respectively. As indicated in the first 3 rows in Tab. 5, using longer sequences for training leads to better results. However, the performance improvement diminishes as the sequence length doubles.To strike a balance between performance and computational cost, we set our default training length to 32. Even trained with 32-frame tracklets, our method is flexible to handle variouslength tracklets during inference. By default, we leverage all history frames to generate predictions at each timestamp. However, we can also generate predictions using a subset of historical frames to reduce computational costs. As shown in Tab. 5, [-63,0] frames for inference achieves similar performance as using all history frames. Moreover, our method can also be extended to handle offline scenarios. When the transformer attends to all timestamps including those future ones, the performance improves further, as demonstrated in the last row of Tab. 5. In this work, we introduce a novel task, object-centric occupancy, which extends the traditional object bounding box representation to provide a more detailed description of the object shape. Compared to its scene-level counterpart, object-centric occupancy enables higher voxel resolution in large scenes by focusing on foreground objects. To facilitate object-centric occupancy learning, we build an object-centric occupancy dataset using LiDAR data and box annotations from the Waymo Open Dataset (WOD). We further propose a novel sequence-based occupancy completion network that learns from our dataset to complete object shapes from noisy object proposals. Our method achieves state-of-the-art performance on both shape completion and object detection tasks on WOD. We believe that our work will inspire future research in perception tasks in the context of autonomous driving. Our annotation pipeline is illustrated in Fig. 6 . Leveraging LiDAR scans and detection annotations from a base 3D dataset, our pipeline probes dense occupancy grids by aggregating multi-frame LiDAR point clouds and then executes occlusion reasoning to discriminate between free and unobserved voxels. Compared to ego-centric approaches [27, 31] , our methodology primarily differs by focusing on annotated objects instead of the entire scene. For each designated object, we gather points within its annotated bounding boxes over time, transform these points from sensor coordinates to the bounding box coordinates and aggregate them into a dense point cloud. Unlike scene-level occupancy, we do not transform the densified object point cloud back to each ego-vehicle coordinate for occupancy construction. Instead, we directly voxelize it under the local object coordinate system, maintaining object-centric precision. While densified object point clouds encode better shape information than a single LiDAR scan, it's important to note that unoccupied voxels (grey voxels in Fig. 6 ) does not necessarily indicate free space; they may be unobserved by the LiDAR due to occlusion. Hence, an occlusion reasoning process is required to distinguish between voxels that are truly free and those that are unobserved. Basically, an unoccupied voxel is considered free if it is traversed trough by a LiDAR ray, and unobserved otherwise. Instead of the time-consuming ray-casting operation used in [27] , we adopt a more efficient approach by leveraging range information from raw range images. Specifically, for each unoccupied voxel, we first convert its center to the range image format using sensor intrinsics and extrinsics at a specific timestamp t, yielding a 2D-pixel index (u t , v t ) and a range value r t . Next, we decide its status by comparing its range with the original range image at timestamp t:EQUATION)where R t ∈ R H×W is the raw range image captured by the LiDAR sensor at timestamp t. We do this for all timestamps to decide the final status of the voxel. And a voxel is considered unobserved only if it is not traversed by any LiDAR ray at any timestamp. Fig. 7 shows some examples of our object-centric occupancy annotations.Fig. 8 and Fig. 9 illustrate the pipelines of the single-branch model and the model using explicit occupancy for detection, respectively. The two global RoI encoders E global in Fig. 9 share the same weights. We additionally add an extra channel to each point feature to indicate whether it is from raw point clouds or from the predicted occupancy volume.Object-Centric Occupancy Free Voxels Occupied Voxels Unobserved Voxels We train our model using the Adam optimizer with an initial learning rate of 1e-4 and a batch size of 8. The model is trained for 24 epochs with the learning rate scheduled by the cosine annealing strategy. We use a transformer with 3 layers, 4 heads, and a hidden dimension of 512. The model is implemented using PyTorch and trained on 8 NVIDIA 3090 GPUs.Fig. 10 shows some examples of the occupancy prediction. Our method effectively predicts the object's shape even when it is extremely occluded. Additionally, our method effectively completes the object shape even at early timestamps, with shape completion improving as the sequence extends.we've also included several surface renderings of the predicted occupancy in Fig. 11 and Fig. 12 . These renderings were obtained by applying marching cubes to the decoded volumetric grids using a level of 0.5. The renderings demonstrate that our method can complete shapes even when the current point cloud is extremely sparse. Due to the use of 0.2m voxel size, the resolution of our predicted occupancy may not support high-quality rendering. For example, the resolution for a typical sedan (let's assume its dimensions are 4.5m* 1.8m * 1.4m) under our voxel size is 23 * 9 * 7. In contrast, common shape completion methods typically use a resolution of 128 x 128 x 128 or higher to facilitate high-quality rendering. It should be noted that for our purposes, high-quality rendering is not required. Although the selected voxel size of 0.2 meters may not provide highly detailed rendering, it is sufficient for downstream driving tasks and ensures computational affordability.Figure 11 : The renderings of predicted occupancy decoded from the shape codes for common vehicles. Top: extracted mesh from the occupancy using marching cube. Bottom: predicted occupancy and point cloud input.Figure 12 : The renderings of complex vehicles. Each row shows the rendering, the corresponding predicted occupancy and input point cloud, and another predicted occupancy with fewer input points. These results demonstrate that the predicted object occupancy can better represent complex shape structures than bounding boxes.• [NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available. • Please provide a short (1-2 sentence) justification right after your answer (even for NA).The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No] ", it is perfectly acceptable to answer "[No] " provided a proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] " or "[NA] " is not grounds for rejection. While the questions are phrased in a binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write a justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.• Delete this instruction block, but keep the section heading "NeurIPS paper checklist", • Keep the checklist subsection headings, questions/answers and guidelines below.• Do not modify the questions and only use the provided macros for your answers. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Error bars are not reported because it would be too computationally expensive.Guidelines:• The answer NA means that the paper does not include experiments. Justification: There is no potential societal impacts of the work. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper does not release data or models that have a high risk for misuse. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 1 Introduction In autonomous driving, accurate and robust 3D scene perception is crucial for safe and efficient navigation. Conventional perception systems primarily adopt 3D object bounding boxes as the perception representation [23, 6, 14, 15] . However, the limitations of 3D bounding boxes (bboxes) are becoming increasingly pronounced as the demands for perception accuracy continue to escalate. Since a 3D bbox is essentially a cuboid that encapsulates the object, it fails to capture the precise details of the object's shape, particularly for objects with irregular geometries. As shown in Fig. 1 (a), the crane is perfectly enclosed by a 3D bounding box. However, its boom, which is a long protrusion relative to the cab, results in a significant amount of unoccupied space within the 3D bounding box. Nevertheless, algorithms that employ 3D bounding boxes as a perception result inherently assume that the space within the bbox is fully occupied, thereby deeming the space enclosed by the 3D bounding box as impassable. Consequently, when addressing complex and irregularly shaped objects, bounding boxes are inadequate in providing fine-grained perceptual outcomes, which can consequently impact the precision of subsequent tasks, such as planning and control. Considering the limitation of 3D bounding boxes, occupancy representation has emerged as a promising alternative for 3D scene perception [28, 27, 31] .As shown in Fig. 1 (b), occupancy representation discretizes the 3D space into a volumetric grid, wherein each voxel is classified as occupied or free.Compared to 3D bboxes, this representation more effectively captures irregular shapes, thereby enhancing accurate planning and control. Real-time scene-level occupancy generation from sensor inputs is non-trivial, presenting challenges not only for vision-centric inputs due to the absence of depth sensing, but also for LiDAR sensors because of the sparsity of each LiDAR scan (see Fig. 2 (b) ). Thus, existing approaches [33, 34] leverage neural networks to predict occupancy in a data-driven manner. Due to computational constraints, these methods typically produce low-resolution occupancy grids for large scene perception (e.g., 200 × 200 × 16 with a voxel size of (0.4m) 3 in [27] ) or requires intensive training for implicit representation [? ? ], which remains insufficient and inefficiency for practical use.Another feasible way to build occupancy grids is directly voxelizing the LiDAR point cloud. To alleviate the sparsity problem (Fig. 2 (b)), aggregating multiple LiDAR scans is an effective way for background. However, for foreground objects, the occupancy construction becomes challenging as it requires accurate detection and tracking to compensate for their potential movements. In real-time applications, 3D detection is prone to drift, and tracking algorithms may lose or mismatch objects, resulting in inaccurate tracklets. As illustrated in Fig. 2 (d), directly aggregating point clouds from inaccurate tracklets can lead to extremely blurry shape representations. Such inaccuracies accumulate over time, progressively degrading the reliability of the shape representation.Based on these observations, we introduce object-centric occupancy as a supplement to object bounding boxes, providing a more detailed structural description for objects' intrinsic geometry. In contrast to its scene-level counterpart, object-centric occupancy exclusively focuses on foreground objects, allowing for higher voxel resolutions even in large scenes. To encourage the advancement of object-centric occupancy perception, we present a novel object-centric occupancy dataset, which is constructed from scratch using an automated pipeline. We then propose a robust sequence-based occupancy completion network. By aggregating temporal information from history observations using attention, our network effectively handles detection drifts and accurately predicts the complete objectcentric occupancy. Furthermore, our method employs an implicit shape decoder to generate dynamicsize occupancy and reduce training costs through queries on selective position. Our experiments under Waymo Open Dataset (WOD) [25] reveal that our method exhibits robust performance in completing object shapes even under noisy detection and tracking conditions. With the implicit shape descriptor, we demonstrate that performance of state-of-the-art 3D object detectors can also be improved, particularly for incomplete or distant objects.2 Related Work 
 
 Appendix:  


Decision: Post-rebuttal, all four reviewers are in favor of acceptance. The AC has examined the paper, the reviews, rebuttal, and discussion and concurs. Since the rebuttal was critical for the paper's acceptance, the AC urges the authors to carefully incorporate all information from the rebuttal into the final version of the paper.

Reviews: review: paper_summary: paper_summary: This paper addresses the limitations of 3D object bounding box representations in autonomous driving by introducing object-centric occupancy. It uses an implicit shape decoder to manage dynamic-size occupancy generation. The method demonstrates robust performance under noisy conditions, significantly enhancing detection results in the Waymo Open Dataset.
strength_weakness: strength_weakness: 1. Creating detailed occupancy for each object seems unnecessary. In most downstream tasks in autonomous driving, using bounding boxes (bboxes) is sufficient.
2. The performance improvement primarily stems from temporal feature fusion, which lacks significant technical innovation.
3. It is unclear whether the loss on occ heads in Fig. 4 enhances detection performance. The authors should compare detection performance with and without occ heads after obtaining the Shape Emb. Z to determine if occ heads contribute to learning useful features, such as yaw estimation.
questions: questions: See weaknesses.
limitations: limitations: The authors discuss the limitation concerning non-rigid objects, which is indeed a constraint. They could start by exploring whether reconstructing the noisy occupancy of non-rigid objects improves detection performance.
paper_summary: paper_summary: The manuscript introduces the idea of representing the shape of objects at higher fidelity (and independent of) the rest of the scene. This is explored in the context of autonomous vehicles research on 3d car detection and representation. The proposed model regresses a shape code and an updated 3d bounding box from a 3D bounding box tracklet (derived from any other algorithm) and the points included in it. The shape code can be queried for occupancy to produce a full shape representation during inference. The proposed approach is able to infer complete shapes from partial inputs and the updated 3D BBs improve the input 3D BBs.
strength_weakness: strength_weakness: The manuscript's related work section misses out on an existing related field of 3D CAD model retrieval (which also produces complete shapes) and shape regression from RGB (and depth data) in indoor scenes. Relevant related works include: 
    - Scan2CAD https://openaccess.thecvf.com/content_CVPR_2019/papers/Avetisyan_Scan2CAD_Learning_CAD_Model_Alignment_in_RGB-D_Scans_CVPR_2019_paper.pdf
    - SLAM++ https://www.doc.ic.ac.uk/~ajd/Publications/salas-moreno_etal_cvpr2013.pdf
    - FroDO https://openaccess.thecvf.com/content_CVPR_2020/papers/Runz_FroDO_From_Detections_to_3D_Objects_CVPR_2020_paper.pdf

I would have wanted to see a few renderings of the shape codes; This would support the claim that the model learns to complete shapes. The appendix has a few but the visualizations are hard to understand without a better renderer. Some kind of shading or edges for the 3D voxels are essential to see any kind of depth and thus shape (Fig 6 and 7). Extracting a mesh using marching cubes at the 0.5 isolevel might also work.
questions: questions: The model takes in a series of 3D BBs and outputs one updated 3D BB - at which timestamp is this 3D BB output? The latest?
limitations: limitations: The limitation of rigid objects only is addressed in the appendix. This means humans are not supported for example.
paper_summary: paper_summary: In this work, the authors propose a novel task called object-centric occupancy.
It extends the 3D detected bounding box representation to provide a more detailed description of the internal object shape.
The method provides higher voxel resolution in large scenes by focusing on foreground objects only. 
It not only achieves state-of-the-art performance on shape completion but can also help refine the object detection tasks on the Waymo Open Dataset (WOD).
strength_weakness: strength_weakness: - The experimental results are only obtained on the Waymo Open Dataset. It will be nicer to conduct the experiments on nuScenes or Argoverse 2 to validate its robustness for different datasets.
- Although the authors say it is a new task, so there are no learning baselines for shape completion, it will be interesting to compare the results with other scene occupancy methods. So that we can see the flaws of using coarse resolution quantitatively.
questions: questions: - The extrapolated results of shape completion are interesting, showing that it can achieve a performance similar to that of using GT boxes. Will it also help with 3D Object Detection results?
limitations: limitations: Yes
paper_summary: paper_summary: This paper presents a new task named object-centric occupancy completion as a fine-grained object representation to supplement the coarse-grained 3D bounding boxes. To accomplish this task, a new dataset, which annotates instance-level high-resolution occupancy, is created in an automated pipeline. This paper also introduces an implicit shape decoder to fuse multi-frame information, predict instance occupancy and refine 3D bounding boxes. Experiments on Waymo datasets above several baselines demonstrate the effectiveness of the proposed method on both occupancy prediction and 3D detection.
strength_weakness: strength_weakness: 1.	The motivation of this paper does not seem to be very reasonable. The authors claim that a. high-resolution scene-level occupancy is constrained by computational cost and foreground objects is more import, and b. 3D detection is too coarse to capture the object geometry information. So why not just predict foreground instance-level occupancy in the whole scene, instead of pursuing higher detection accuracy by using the occupancy results? 
2.	Time and memory cost bought by the proposed shape decoder are not provided. The paper is trying to make a trade-off between occupancy and detection in fine-/coarse-grained level and computational cost level. But the authors only report the occupancy and detection accuracy.
3.	Some methods, like VoxelNeXt, FSDv2, HEDNet are missing and are not compared in Table 1.
4.	Typos/mis-leading descriptions. For example, ‘Tab. 5.4’ on line 351 -> ‘Tab. 3’.
questions: questions: 1.	Why not just predict foreground instance-level occupancy in the whole scene, instead of pursuing higher detection accuracy by using the occupancy results? (the same as weakness 1)
2.	Could you provide the computational cost of your method or the proposed module?
limitations: limitations: The authors adequately addressed the limitations.

Comments: title: Post rebuttal
comment: I have read all the reviews and authors' responses. My concern has been well addressed. so I raise my rating to 5.
Thanks for the response. My concerns regarding the motivation and computation cost have been well-addressed, therefore I raise my rating to weak accept.
I have looked at the other reviews again and to me the main other problems raised by the other reviewers have also been addressed sufficiently. It is great to also see that the additional occ head helps detection accuracy (even if only slightly) and that the model can to some degree generalize over different detection front ends without retraining. These additional facts convince me that to raise my rating to weak accept.
Thank you for recognizing the strength of our method in achieving strong shape completion. We greatly appreciate your positive feedback on this aspect of our work. Given this recognition, we kindly ask if you might reconsider your score, as it would greatly support the value of our contributions.
Thanks for reminding me of those results in the appendix. Those are indeed strong completion results. I also see the data problem of obtaining complete shapes from partial observations.
Thank you for the quick response. As shown in Figure 10 of our appendix, our method demonstrates a strong ability for shape completion. Even with extremely sparse input points at an early timestamp (the first row in Figure 10), our method effectively completes the shape in a way that aligns with the sparse point observations. This level of completion cannot be achieved through simple interpolation or extrapolation. Our model’s success in this regard is due to its ability to learn the complete shape distribution from the training data.

However, as with any learning-based approach, our method's performance can be constrained by the quality of the annotations. Since our ground truth shapes are generated by aggregating points across real object sequences, the back side of an object is often 'unobserved' (see Figure 7), meaning that most of the back-side voxels are not supervised during training.

Fortunately, there is a simple yet effective way to mitigate this issue. Given the symmetry of many objects, we can fill the 'unobserved' voxels with their mirrored counterparts. Retraining our model using these mirror-aided ground truths can significantly enhance its ability to complete shapes on the back side. However, we did not employ this strategy in our main paper, as it might compromise the authenticity of the shape annotations.
I appreciate the authors responses. The attached shape renderings are from the same viewpoint as the input pointcloud as far as I can tell. This shows "weak completion" - interpolating between points and maybe some extrapolation. The real question is what the back side of those shapes look like "strong completion".

I looked at the other reviews as well and did not see anything that would change my rating so far.