Title: OctreeOcc: Efficient and Multi-Granularity Occupancy Prediction Using Octree Queries
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D Scene Understanding; Occupancy Prediction; Octree
Abstract: Occupancy prediction has increasingly garnered attention in recent years for its fine-grained understanding of 3D scenes. Traditional approaches typically rely on dense, regular grid representations, which often leads to excessive computational demands and a loss of spatial details for small objects. This paper introduces OctreeOcc, an innovative 3D occupancy prediction framework that leverages the octree representation to adaptively capture valuable information in 3D, offering variable granularity to accommodate object shapes and semantic regions of varying sizes and complexities. In particular, we incorporate image semantic information to improve the accuracy of initial octree structures and design an effective rectification mechanism to refine the octree structure iteratively. Our extensive evaluations show that OctreeOcc not only surpasses state-of-the-art methods in occupancy prediction, but also achieves a 15%-24% reduction in computational overhead compared to dense-grid-based methods.


Peer reviews:
comments: SUMMARY:
This paper proposes a new occupancy predictor that utilizes the octree structure to reduce the total number of occupancy query. The octree tree is initialized with semantic information and recursively rectified, so that it can accurately represent the structure of the driving scenes.

STRENGTHS:
1. This work gives a new way to represent occupancy instead of voxel feature or dense query, which is enlightening for later works.
2. The prediction precision of OctreeOcc exceeds FB-Occ and PanoOcc, indicating a new SOTA. The visualization results also demonstrate its ability to predict fine structures.

WEAKNESSES:
1. OctreeOcc requires extra segmentation model to provide the reference for the initialization of octree queries, which reduces the efficiency gain. More details of the segmentation model should be published.
2. The ablation study shows that the accuracy is largely improved after using the extra semantic information. A fair comparison between OctreeOcc without extra semantic information and other methods is missing.

QUESTIONS:
1. It is not clear how the high-confidence and low-confidence regions are determined in Iterative Structure Rectification. Is the confidence related to the split probability?
2. How does OctreeOcc handle the conflict between the octree queries initialized by current semantic information and the octree queries aligned from the previous frame?

LIMITATIONS:
The authors have described the limitation of this work, and there is no potential negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-03

comments: SUMMARY:
This paper introduces OctreeOcc, aiming to tackle the heavy computational demands of the dense and regular grid representations employed by the previous methods. Instead of randomly initializing the octree structure, OctreeOcc incorporates the semantic priors of images as guidance. The octree structure are further updated iteratively to correct the potential errors.

STRENGTHS:
1. Octree representation is a good solution to the heavy computation burden of the dense grid representation. OctreeOcc presents the details of the network components clearly.
2. OctreeOcc provides detailed ablation experiments.
3. OctreeOcc achieves good balance between the accuracy and the efficiency.

WEAKNESSES:
1. The image resolution and visible mask should be marked for fairly comparing with other methods. To my understanding, using the visible mask or not results in significant performance differences.
2. This paper mentioned temporal fusion in line 131. Is the temporal information obtained similar to the bevformer? Besides, how many frames are employed? (e.g., 4 frames in the PanoOcc)
3. The metrics of the Symponies in the tables should be updated. The performance differences are acceptable because Symponies uses stereo information (If OctreeOcc were to rely solely on monocular input).
4. Dose the Memory presented in the tables refer to training memory? To my understanding, the inference memory for most of the methods are highly lower than this.
5. In Table 6, query form of octree attend to the image features in a hierarchical manner, what is the number of the attention layers of the 3D voxel?

QUESTIONS:
Please see above weaknesses.

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-07

comments: SUMMARY:
This paper introduces OctreeOcc, an innovative 3D occupancy prediction framework that leverages the octree representation to adaptively capture valuable information in 3D, offering variable granularity to accommodate object shapes and semantic regions of varying sizes and complexities. The authors incorporate image semantic information to improve the accuracy of initial octree structures and design an effective rectification mechanism to refine the octree structure iteratively. Extensive evaluations show that OctreeOcc not only surpasses state-of-the-art methods in occupancy prediction, but also achieves a 15% − 24% reduction in computational overhead compared to dense-grid-based methods.

STRENGTHS:
- OctreeOcc utilises octree to represent 3D space, which is novel and no one has done it before as far as my knowledge.

- Owing to the design of octree queries, OctreeOcc improves the accuracy of occupancy prediction with the latency also decreasing.

- Extensive experiments on Occ3D and SemanticKitti demonstrate the effectiveness of OctreeOcc.

WEAKNESSES:
- The pipeline is quite complex and heavily relies on manual operations, such as the selection strategy in Iterative Structure Rectification Module. My concern is that the complicated design may reduce the generalization.

- The updating processing of octree queries is not very clear. In my understanding, if the octree mask changes, the newly generated octree queries cannot directly match the original octree queries.

- Different octree queries may have different granularity, but, as shown in Eql (3), all octree queries seem to interact with image features only referring to their centers, which ignoring the influence of granularity.

QUESTIONS:
please refer to weaknesses.

LIMITATIONS:
NaN
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors aim to tackle the problem of high memory usage in dense occupancy prediction for 3D scenes. They introduce OctreeOcc, a method that uses octree structures to make predictions more efficiently. Experimental results show that OctreeOcc reduces computational load and achieves competitive performance.\n\nSTRENGTHS:\nS1: The authors address the significant issue of high memory consumption in dense occupancy prediction, highlighting the importance of finding more efficient solutions.\n\nS2: The use of octree representation for occupancy prediction is a novel and effective approach, offering adaptive granularity for various object shapes and regions.\n\nS3: Experimental results show some reduction in memory usage and latency, demonstrating the efficiency of the proposed method.\n\nWEAKNESSES:\nW1: The proposed method seems difficult to optimize and relies on segmentation and historical information.\n\nW2: The results on the SemanticKITTI dataset do not achieve state-of-the-art performance, and the reduction in memory usage is not very significant.\n\nW3: The training process is complex and costly, requiring three days on 8 A100 GPUs, which seems less efficient compared to previous methods.\n\nQUESTIONS:\nQ1: The proposed method seems to be quite time-consuming to train. Can the authors provide a comparison of training times with other methods? This is especially relevant given the emphasis on efficiency throughout the paper.\n\nQ2: There is some confusion regarding the reported memory consumption in the experiments. For instance, the original VoxFormer [1] paper reports memory usage of less than 16GB, but the authors here report around 23GB. What accounts for this significant discrepancy? Are there differences in the settings or configurations used?\n\nQ3: The implementation details provided focus on the nuScenes dataset. Could the authors include the specifics of the implementation for the SemanticKITTI dataset as well?\n\nQ4: The paper lacks results or discussions on other commonly used datasets, such as SSCBench-KITTI360 [2]. Including these would provide a more comprehensive evaluation of the proposed method.\n\n[1] VoxFormer: Sparse Voxel Transformer for Camera-based 3D Semantic Scene Completion\n\n[2] SSCBench: Monocular 3D Semantic Scene Completion Benchmark in Street Views\n\nLIMITATIONS:\nThe authors have identified a significant limitation in their work: the dependency on the quality of the occupancy ground truth (GT). This reliance can lead to suboptimal performance if the GT data, derived from sparse LiDAR point clouds and surface reconstruction, is not accurate or complete.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}