CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds
Authors: Haiyang Wang
Lihe Ding
Shaocong Dong
Shaoshuai Shi
Aoxue Li
Jianan Li
Zhenguo Li
Liwei Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: 3D Object Detection
Point Clouds
3D Bounding Boxes
3D Deep Learning

Abstract: We present a novel two-stage fully sparse convolutional 3D object detection framework, named CAGroup3D. Our proposed method first generates some high-quality 3D proposals by leveraging the class-aware local group strategy on the object surface voxels with the same semantic predictions, which considers semantic consistency and diverse locality abandoned in previous bottom-up approaches. Then, to recover the features of missed voxels due to incorrect voxel-wise segmentation, we build a fully sparse convolutional RoI pooling module to directly aggregate fine-grained spatial information from backbone for further proposal refinement. It is memory-and-computation efficient and can better encode the geometry-specific features of each 3D proposal. Our model achieves state-of-the-art 3D detection performance with remarkable gains of +3.6% on ScanNet V2 and +2.6%  on SUN RGB-D in term of mAP@0.25. Code will be available at https://github.com/Haiyang-W/CAGroup3D.

CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds
Haiyang Wang1.6.7 Lihe Ding"; Shaocong Dong", Shaoshuai Shi
Aoxue Li Jianan LiP, Zhenguo Li", Liwei Wang
Center for Data Science, Peking University "Beijing Institute of Technology
Max Planck Institute for Informatics "Huawei Noah's Ark Lab, China
National Key Laboratory of General Artificial Intelligence, School of Intelligence Science and Technology, Peking University "Peng Cheng Laboratory "Pazhou Laboratory (Huangpu) (vanghaiyangostu, wanglw0cls} . pku edu. en, (dean. dinglihe, shaocong) @bitt edu. en sshi@upi- inf de, 1ij 1annannn@gmaaaai com (liaoxue2, Li ...ennngo (henguo)Chuaver Chuawei com
Abstract
We present novel two-stage fully sparse convolutional 3D object detection frame- work, named CAGroup3D. Our proposed method first generates some high-quality 3D proposals by leveraging the class-aware local group strategy on the object surface voxels with the same semantic predictions, which considers semantic consistency and diverse locality abandoned in previous bottom-up approaches. Then, to recover the features of missed voxels due incorrect voxel-wise seg- mentation, we build a fully sparse convolutional Rol pooling module to directly aggregate fine-grained spatial information from backbone for further proposal refinement. It is memory-and- and-computaaiionn efficient and can better encode the geonetry-speciffc features of each 3D proposal. Our model achieves state-of-the- art 3D detection performance with remarkable gains of +3.6% on ScanNet V2 and +2.6% on SUN RGB-D in term of mAP@0.25. Code will be available https:// htte:///gthubb cow Haiyang- W/CAGroup3D
1 Introduction
As crucial step towards understanding 3D visual world, 3D object detection aims to estimate the oriented 3D bounding boxes and semantic labels of objects in real 3D scenes. It has been studied for a long time in both academia and industry since it benefits various downstream applications, such as autonomous driving 2 36). robotics (55] 37 and augmented reality 33. In this paper, we focus on detecting 3D objects from unordered, sparse and irregular point clouds. Those natural characteristics make it more challenging to directly extend well-studied 2D techniques 3D detection.
Unlike 3D object detection from autonomous driving scenarios that only considers bird's eye view
(BEV) boxes 30 50 19
32 31. most of existing 3D indoor object detectors 27
38 22 5 52
typically handle this task through a bottom-up scheme, which extracts the point-wise features from input point clouds, and then groups the points into their respective instances to generate set of proposals. However, the above grouping algorithms are usually carried out in a class-agnostic manner, which abandons semantic consistency within the same group and also ignores diverse locality among different categories. For example, VoteNet (27) learns the wiint-wise center offsets and aggregates the points that vote similar semantic-iree inreeeiant local region. Though impressive, as shown in Figure Il these methods may fail in cluttered indoor scenes where various objects are close but belong to different categories. Also, the object sizes are diverse for different categories, so that a class-agnostic
Equal contribution. Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

mis-grouping of different categories seed point (table)
(b) partial over coverage
vote point (table) @ soed point (chair)
vote point (chair) @ soed point (sofa) vote point (sofa)
ball query
Class-agnostic grouping methods suffer from (a) mis- mii-giguuinn of different categories within the same local regions, (b) partial coverage of the object surfaces; outliers from the cluttered scene.
Figure 1:
local grouping may partially cover the boundary points of large objects and involve more noise outliers for small objects.
Hence, we propose CAGroup3D, a two-stage fully convolutional 3D object detection framework. Our method consists of two novel components. One is the class-aware 3D proposal generation module, which aims to generate reliable proposals by utilizing class-specific local group strategy on the object surface voxels with same semantic predictions. The other one is efficient fully sparse convolutional Rol pooling module for recovering the features of the missed surface voxels due to semantic segmentation errors, so as to improve the quality of predicted boxes.
Specifically, a backbone network with 3D sparse convolution is firstly utilized to extract descriptive voxel-wise features from raw point clouds. Based the learned features, conduct a class-aware local grouping module to cluster surface voxels into their corresponding instance centroids. Different from [27]. in order consider the semantic consistency, not only shift voxels of the same instance towards the same centroid but also predict per-voxel semantic scores. Given the contiguously distributed vote points with their semantic predictions, predicted semantic categories and vote coordinates, so as to generate class-specific 3D voxels for different categories. The voxel size of each category is adaptive its average spatial dimension. To maintain the structure of fully convolution, we apply sparse convolution as grouping operation centered on each voted voxel to aggregate adjacent voxel features in the same semantic space. Note that these grouping layers are class-dependent but share the same kernel size, thus the larger classes are preferred to be aggregated with larger local regions.
initially voxelize them according the
Secondly, given the proposal candidates, fine ameeggained specific features within 3D proposals need to be revisited from 3D backbone through certain pooling operation for the following box refinement. However, state-of-the-art pooling strategies [31] are neemoryandd-omputaan
intensive due to the hand-crafted set abstraction 25 Besides that, its max-pooling operation also harms the geometry distribution. To tackle this problem, we propose Rol-Conv pooling module, which directly adopts the well-optimized 3D sparse convolutions to aggregate voxel features from backbone. It can encode effective geometric representations with memory-efficient design for further proposal refinement.
In summary, contributions three-fold: 1) We propose novel class-aware 3D proposal generation strategy, which considers both the voxel-wise semantic consistency within the same local group and the object-level shape diversity among different categories. 2) We present Rol-Conv pooling module, an efficient fully convolutional 3D pooling operation for revisiting voxel features directly from backbone to refine 3D proposals. 3) Our approach outperforms state-of-the-art methods with remarkable gains on two challenging indoor datasets, iee,, ScanNet V2 6 and SUN RGB-D 33 demonstrating its effectiveness and generality.
2 Related Work
3D Object Detection on Point Clouds. Detecting 3D objects from point clouds challenging due to orderless, sparse and irregular characteristies. Previous approaches can be coarsely classified into two

---

lines in terms of point representations, ie.. the voxel-based methods [54 46 32 45 31 50 32 9 40
47 Voxel-based methods are mainly applied in outdoor autonomous driving scenarios where objects are distributed on the large-scale 2D ground plane. They process the sparse point clouds by efficient 3D sparse convolution, then project these 3D volumes to 2D grids for detecting bird's eye view (BEV) bboxes by 2D ConvNet. Powered by PointNet series [25] 29 point-based methods are also widely used to predict 3D bounding bboxes. Most of existing methods are in bottom-up manner, which extracts the point-wise features and groups them to obtain object features. This pipeline has been a great success for estimating 3D bboxes directly from cluttered and dense 3D scenes. However, due to the hand-crafted point sampling and computation intensive grouping scheme applied in PointNet++ [29], they are difficult to be extended to large-scale point clouds. Hence, we propose efficient fully convolutional bottom-up framework to efficiently detect 3D bboxes directly from dense 3D point clouds.
and the point-based methods [27 38 5 22
52
Feature Grouping. Feature grouping is crucial step for bottom-up 3D object detectors 27 38 22
which clusters a group of point-wise features to generate high-quality 3D bounding boxes. Among the numerous successors, voting-based framework [27 is widely used, which groups the points that vote to the same local region. Though impressive, it doesn't consider the semantic consistency, so that may fail in cluttered indoor scenes where the objects of different classes are distributed closely. Moreover, voting- based methods usually adopt a class-agoniste local region for all objects, which may incorrectly group the boundary points of large objects and involve more noise points for small objects. To address the above limitations, we present a lass-aware local grouping strategy to aggregate the points of the same category with class-specific center regions.
5 52 355,
Two-stage 3D Object Detection. Many state-of-the-an methods considered applying RCNN style detectors to the 3D scenes, which apply 3D Rol- Rol-pooling scheme or its variants (30 32
g
31
48
44
to aggregate the specific features within 3D proposals for the box refinement in a second stage. These pooling algorithms are usually equipped with set abstraction (25] to encode local spatial features,
which consists of a hand-crafted query operation (e.g., ball query 25 or vector query [ to capture
the local points and a max-pooling operation to group the assigned features. Therefore these Rol pooling modules are mostly computation expensive. Moreover, the max max-pooling operation also harms the spatial distribution information. To tackle these problems, propose Rol-Conv pooling, memory-an carry-and-comm and-computation efficient fully convolutional Rol pooling operation to aggregate the specific features for the following refinement.
3 Methodology
In this paper, we propose CAGroup3D, two-stage fully convolutional 3D object detection framework for estimating accurate 3D bounding boxes from point clouds. The overall architecture of CAGroup3D is depicted in Figure 2 Our framework consists of three major components: CNN with sparse convolution as the backbone network for point cloud feature learning ($3.1). a class-aware 3D proposal generation module for predicting high quality 3D proposals by aggregating voxel features of the same category within the class-specific local regions ($3.2) and Rol-Cony pooling module for directly extracting complete and fine-grained voxel features from the backbone to
efficient 3D voxel
revisit the miss-segmented surface voxels and refine 3D proposals, Finally, we formulate the learning objective of our framework in $3.4
3.1
3D Voxel CNN for Point Cloud Feature Learning
For generating accurate 3D proposals, we first need to learn discriminative geometric representation for describing input point clouds. Voxel CNN with 3D sparse convolution [32
45
54
13 [2| is
widely used by state-of-the-a 3D detectors thanks to its high efficiency and scalability of converting the point clouds to regular 3D volumes. In this paper, we adopt sparse convolution based backbone for feature encoding and 3D proposal generation.
3D backbone network equipped with nigh-resolution feature maps and large receptive fields is critical for accurate 3D bounding box estimation and voxel-wise semantic segmentation. The latter is closely related to the accuracy of succeeding grouping module. To maintain these two characteristics, inspired by the success of HRNett series 39
16 34 in segmentation community, we implement a 3D voxel bilateral network with dual resolution based on ResNet [5]. For brevity, we refer it as BiResNet. As shown in Figure 2 our backbone network contains two branches. One is the sparse modification of
3

---

Class-Aware 3D Proposal Generation
Input
BeReeNer
Clas Aware 3D Proposuls Generation
(1/t))
Fot (123)
VFEI
Group
Blocks
VFE.2
soer Aranck
Block2 (119)
Fest (1/2)
Voseliee
Blockl
Group.n
deranooople
Agevendation
Seat foise
weserss branch
clus-amare teaefiction
slate-oeect locel grosptng
Rol-Conv Point Cloud Feature Pooling and Box Refinement
Rol Cony Point Cloud Feature Pooling
Bax Refinement
Proparate
Sparne Alstractionn
Malipfe Sprre abamannoe Hoots
Redigrsfonce sampling
SpCrer
Figure 2: The overall architecture of CAGroup3D. (a) Generate 3D proposals by utilizing class-aware local grouping on the vote space with same semantic predictions. (b) Aggregating the specific features within the 3D proposals by the efficient Rol-Conv pooling module for the following box refinement.
ResNet18 15] where all 2D convolutions are replaced with 3D sparse convolutions. It extract multi-scale contextual information with proper downsampling modules. The other one is auxiliary branch that maintains a high-resolution feature map whose resolution is 1/2 of the input 3D voxels. Specifically, the auxiliary branch is inserted following the first stage of ResNet backbone and doesn't 't contain any downsampling operation. Similar to [39], we adopt the bridge operation between the two paths to perform the bilateral feature fusion. Finally, the fine-grained voxel-wise geometric features with rich contextual information are generated by the high-resolution branch and facilitate the following module. Experiments also demonstrate that our voxel backbone performs better than previous FPN-based ResNet 20). More architecture details are in Appendix.
3.2 Class-Aware 3D Proposal Generation
Given the voxel-wise geometric features generated by the backbone network, a bottom-up grouping algorithm is generally adopted aggregate object surface voxels into their respective ground truth instances and generate reliable 3D proposals. Voting-based grouping method (27) has shown great success for 3D object detection, which performed in class-agnostic manner. It reformulates Hough voting to learn point-wise center offsets, and then generates object candidates by clustering the points that vote similar center regions. However, this method may incorrectly group the outliers in the cluttered indoor scenarios (e.g., votes are close together but belong to different categories), which degrades the performance of 3D object detection. Moreover, due the diverse object sizes of different categories, class- agnostic local regions may mis-group the boundary points of large objects and involve more noise points for small objects.
To address this limitation, we propose the class-aware 3D proposal generation module, which first produces voxel-wise predictions (e.g., semantic maps and geometric shifts), and then clusters the object surface voxels of the same semantic predictions with class-specific local groups
Voxel-wise Semantic and Vote Prediction. After obtaining the voxel features from backbone network, two branches are constructed to output the voxel-wise semantic scores and center offset vectors. Specificallyy, the backbone network generates number of N non-empty voxels {o from backbone, where o; [ fi] with R3 and Ji E RC. A voting branch encodes the voxel feature Si to learn the spatial center offset Ar; E R3 and feature offset Aft E TRC, Based on the learned spatial and feature offset, we shift voxel o, to the center of its respective instance and generate vote

---

point p; as follow:
{p: pp. = = + Az,, fi + Afi))
(I)
The predicted offset Az, is explicitly supervised by a smooth-& loss with the ground displace- ment from the coordinate of seed voxel I; its corresponding bounding box center.
In parallel with the voting branch, we also construct a semantic branch to output semantic scores S = h for all the voxels over Nelass classes as
s; = MLP (o:) E No,i,,, for = L....,
where MLPsem(-) (-) is a one-layer multi-layer-perceptronn (MLP) network and s; indicates the semantic probability for all classes of voxel og. We adopt focal loss 21 for calculating voxel segmentation loss handle the class imbalance issue.
Notably, the vote and semantic targets of each voxel are associated with the ground-truth 3D boxes not the instance semantic masks, that it can be easily generalized to the 3D object detection datasets with bounding box annotations. To be specific, for each voxel, only the ground-trun truth bounding boxes that includes this voxel are selected. Considering the ambiguous cases that voxel is in multiple ground truth bounding boxes, only the box with the least volume is assigned to this voxel.
Class-Aware Local Grouping. This step aims produce reliable 3D proposals in a bottom-up scheme based on the above voxel-wise semantic scores { and vote predictions {pr)- To carry out grouping with semantic predictions, we first define score threshold for all categories to individually determine whether voxel belongs to a category instead of utilizing the one-hot semantic predictions. It can allow the voxel to be associated with multiple classes and thus improve the recall of semantic voxels for each category. Given the semantic scores S = {sx J= we iterate over all the classes, and slice point subset from the whole scene of each class that has the score higher than the Natass
threshold to form the class dess-deennddnt vote set {cg]/=1
{ole = {pr : r,i= I..., N)
Netass
(3)
j=1
The above semantic subset is generated a contiguous euclidean space and the vote points are distributed irregularly. To maintain the structure of pure convolutions and facilitate the succeeding class-aware convolution- based local grouping module, we individually voxelize the vote points in each semantic subset to [Vg]Naa class-specific 3D voxel size, which is proportional the class average spatial dimension. Specifically, this re-voxelization process for each class can be formulated as follows:
by employing a voxel feature encoding (VFE) layer with
(or M VFE(cy, a .dj, Aug), for j = 1,..., Nelase:
(4)
where cj is the class-dependent vote set of class index j and V, is the number of non-empty voxels after class individual re-voxelization. a dj is the class-specific voxel size, where a a predefined scale factor and d, (wz, 41.4,) is the category average spatial dimension. VFE( a dj, Aug) means that the average pooling operation adopted to voxelize vote features on j-th class subset with the voxel size dy. Importantly, the voxel size is adaptive among different categories, which is more diverse than the widely used FPN-based prediction structure.
Given the predicted vote voxels of j-th class {v.))Ki). we apply sparse convolutions with predefined kernel size ((a) on each voxel, and automatically aggregate local context inside the class to generate class-specific geometric features Ali as follow:
A =aa") aa
SparseConv (v,, (0:) (())
(5)
(1) where SparseConv 3D center." specific for different classes. A shared anchor-free head is appended to the aggregated features for predicting classification probabilities, bounding box regression parameters and confidence scores.
'uupport voselss' kemmel size) is the standard sparse 3D convolution [3] and
3.3 Rol-Cony point cloud feature pooling for 3D Proposal Refinement
Due to the semantic segmentation errors in stage-I, class-aware local grouping module will mis-group some object surface voxels. So an efficient pooling module is needed to recover the missed voxel
5

---

features and also aggregate more fine tne-grained features from backbone for proposal refinement. State- of-the-art 3D Rol pooling strategies [3 [] usually adopt set abstraction operation to encode local patterns, which is computation expensive compared to traditional convolution and hand-crafted with lots of hyper-parameter (e.g., radius, the number of neighbors). Moreover, its max-pooling operation also harms the spatial distribution information.
To tackle these limitations and hold fully convolution structure, we propose RoI-Conv pooling operation, which builds a hierarchical grouping module with well optimized 3D sparse convolutions to directly aggregate Rol-specific features from backbone for further proposal refinement. Our hierarchical structure is composed by a number of sparse abstraction block, which contains two key components: the Rol-guidence sampling for selecting a subset of input voxels within the 3D proposals, which defines the centroids of local regions, and shared sparse convolution layer for encoding local patterns into feature vectors.
Sparse Abstraction, As shown in Figure 2 the inputs to this block are number of I input voxels I (4- and .M| proposals M f fm where pm is the proposal parameters for guiding
the point sampling. The output is a number of Q pooling Rol-specific voxels Q = { )
Specifically, given the input voxels and proposals, instead of directly sampling from the whole input space, we adopt the Rol-guidence sampling to uniformly sample Gz x Gy x G2 grid points within each 3D proposal in voxel space, which are denoted as G {gg E 2²) (.C.xGxxGX... X.M G2 x GyyGG is the proposal sampling resolution, which are the parem--arameters independent on proposal sizes. Considering the overlap of different proposals,
merge the repeated grid points and generate
[ unique points set & {gx> k= where ② is the number of unique grid points. Then, with the Rol-specific points set G, we exploit sparse convolution centered on each sampled point to cover a set of neighboring input voxels (e.g., N₂ = (1h 12 (....... 1b+ for gi)) within the kernel size k(P) as:
(SparseConv (yn, Nr, k(P), if L > 0, 16,
2 = qk 4
for ls I,.... ō
(6)
if LB = 0,
where LE is the number of neighboring voxels queried by the k-th Rol-specific point with kernel size k(()), SparseConv ap center: support voxels; 'eemel size is shared sparse 3D convolution for all the proposals and only applied on the points that their neighboring voxel sets are non-empty. & means empty voxel. Then, to hold the surface geometry and reduce computation cost, we abandon the empty voxels and output the Rol-specific voxels set Q.
Rol-Conv Pooling Module. Our pooling network is equipped with two-layers sparse abstraction block and progressively abstracts the voxel features within each 3D proposal from the backbone to Rol-specific features iteratively
Q Apsrse {pm) JM m=1 7 x 7 x 7, 5),
F SparseAbs(Q' Q¹ [ x x 1, 7),
where SparseAbs((- input vosels: proposals: 'sampling resolations keenel size denotes our sparse abstraction block. Notably, to encode the voxel features of oriented proposals, we follow the transformation strategy in 322 before the last block, and normalize the input voxels belonging to each proposal to its individual canonical systems. With the Rol feature F E RC of each proposal, the refinement network predicts the size and location (i.e., bbox dimension, center and orientation) residuals relative to the proposal in Stage-I and the targets are encoded by the traditional residual-brr baseall--eeeddd-- method
31 32
3.4 Learning Objective
Our proposed approach is trained from scratch with semantic loss Cyems voting loss Loote-rog centerness loss Ceate, bounding box estimation loss Loox, classification losses Lets for Stage-I and bbox refinement loss Leeox for Stage-II, which are formulated as follows:
L Brem Cram + Lvnee + Centr
(8)
+ +hanx Lnow + Bats Let + Brebox Crebox-
Lssm a multi-class focal loss (11 used to supervise voxel-wise semantic segmentation. Lyote is a smooth-4, loss for predicting the center offset of each voxel. In term of the 3D proposal generation

---

Table 1: 3D detection results on ScanNet V2 6 and SUN RGB-D 33
The main comparison based the best results of multiple experiments, and the average value of 25 trials is given in brackets. means the multi-sensor approaches that use both point clouds and RGB images.
ScanNet V2
SUN RGB-D
Methods
Presened
mAP@0.25
mAP@0.5
mAP@0.25
mAP@0.5
F-PointNett 266
CVPR' 18
19.8
10.8
54.0
GSPN 49
CVPR'I 19
30.6
17.7
3D-SIS 77
CVPR'
40.2
22.5
Im VoVoNeeet 28
CVPR
63.4
TokenFusion
41
CVPR'22
70.8(69.8)
54.2(53.6)
64.9(64.4)
48.3(47.7)
VoteNet 27
ICCV119
58.6
33.5
57.7
3D-MPA [0
CVPR 20
64.2
49.2
HGNet 4
CVPR
61.3
34.4
61.6
MLCVNet
42
CVPR 20
64.5
41.4
59.8
GSDN 14
ECCV'20
62.8
34.8
H3DNet
52
ECCV'20
67.2
48.1
60.1
39.0
BRNet 5
CVPR'21
66.1
50.9
61.1
43.7
3DETR 24
ICCV'21
65.0
47.0
59.1
32.7
VENet 45
ICCV'21
67.7
62.5
39.2
Group-free
22
ICCV'21
69.1(68.6)
52.8(51.8)
63.0(62.6)
45.2(44.4)
RBGNet 338
CVPR'22
70.6(69.6)
55.2(54.7)
64.1(63.6)
47.2(46.3)
HyperDet3D [53
CVPR'22
70.9
57.2
63.5
47.3
FCAF3D
ECCV'22
71.5(70.7)
57.3(56.0)
64.2(63.8)
48.9(48.2)
Ours
75.1(74.5)
61.3(60.3)
66.8(66.4)
50.2(49.5)
module, we follow the same loss functions Lentr Loox and Lets defined in 7 optimize object centerness, bounding box estimation and classification respectively. For the second stage, Crebox is the residual-based smooth-{, box regression loss for 3D box proposal refinement, which contains size, box center and angle refinement loss. Besides that, we also add the same IoU loss Liou as used in stage-I, and the final box refinement loss is as follows
Lrehoxx
Σ
Lommonth e, (Ar", Ar) + Liou:
re (h....
where Ar is the predicted residual and Ar" is the corresponding ground truth. The detailed balancing factors are in Appendix.
4 Experiments
4.1 Datasets and Evaluation Metric
Our CAGroup3D evaluated on two indoor challenging 3D scene datasets, i.e., ScanNet 6 and For all datasets, we follow the standard data splits adopted in (27).
SUN RGB-D 33
ScanNet V2 contains richly-annotated 3D reconstructed indoor scenes with axis-aligned bounding box for most common 18 object categories. It contains 1201 training samples and the remaining 312
scans are left for validation. We follow 27 to sample point clouds from the reconstructed meshes.
SUN-RGB-D is a single-view indoor dataset which consists of 10,355 RGB-D images for 3D scene understanding. It contains ~5K training images annotated with the oriented 3D bounding boxes and the semantic labels for 10 categories. To feed the point data to our method, we follow [2 and convert the depth images point clouds using the provided camera parameters.
All the experiment results on both datasets are evaluated by a standard evaluation protocol 27 which uses mean average precision(mAP) with different IoU thresholds, i.e., 0.25 and 0.50,
7

---

Table 2: Effect of Semantic Prediction, Diverse Local Group, RoI-Conv and BiResNet.
Semantic Prediction
Diverse Local Group
BiResNet
RoI-Conv
mAP@0.25
mAP@0.5
68.22
53.17
69.24
54.05
72.10
57.07
73.21
57.18
74.50
60.31
Table Ablation study of the class-aware local group with different scale factor a. (k-(a)
Table 4: Ablation study the class-aware local group with different semantic threshold т.
scale factor
mAP@0.5 mAP@0.25
thres
mAP@0.5 mAP@0.25
1.00
36.38
24.66
0.20
73.51
59.50
0.60
61.49
47.41
0.10
74.28
59.97
0,20
74.21
58.77
0.08
74.38
60.10
0.15
74.50
60.31
0.06
74.51
60.27
0.10
74.01
59.03
0.04
74.38
59.98
0.05
72.98
58.10
0.02
73.53
58.85
4.2
Implementation Details
Network Architecture Details. For both datasets, we set the voxel size as 0.02m. As for the backbone, we use the same 3D voxel ResNet18 introduced in [14] as the downsample branch to extract rich contextual information and set the voxel size of high-resolution branch to 0.04m. In terms of class-aware 3D proposal generation module, we set the scale factor of re-voxelization a to 0.15. The semantic threshold r is initially set to 0.15 and then decreases by 0.02 every 10 epochs for ScanNet V2 and epochs for SUN RGB-D until it reaches the minimum value of 0.05. Moreover, the kernel size of convolution- based grouping module ((()) is We stack two sparse abstraction blocks for extracting Rol-specific representations, in where the proposal sampling resolutions are set to 7 7 X 7 and X 1 x 1, and the kernel sizes of sparse grouping (PP) are set to 5 and 7 respectively,
Training and Evaluation Scheme. Our model is trained in end-to-end manner by AdamW optimizer [23). Following the 77]. we set batch size, initial learning rate and weight decay are 16, 0.001 and 0.0001 for both datasets. Training ScanNet V2 requires 120 epochs with the learning rate decay by 10x on 80 epochs and 1 110 epochs. SUN RGB-D takes 48 epochs and learning rate decayed on 32, 44 epochs. All models are trained two NVIDIA Tesla V100 GPUs with 32 GB memory per-card. The gradnorm clip [51 18] is applied to stabilize the training dynamics. We follow the same evaluation scheme in (22), which runs training for 5 times and test each trained model for times. We report both the best and average metrics across all results.
4.3 Benchmarking Results
We report the comparison results with state-of-the-an 3D detection methods ScanNet V2 [6] and
SUN RGB-D 33 benchmark. Same as (22), we report the best and average results of 5x5 trials.
As shown in Table [ our approach leads 75.1 1 in terms of mAP@0.25 and 61.3 in terms of mAP@0.50 on SeanNet V2 (6), which is +3.6 and +4.0 better than the state-of-the-art 77. For SUN RGB-D, our approaches achieves 66.8 and 50.2, which gains +2.6 and +1.3 in terms of mAP, with 3D IoU threshold 0.25 and 0.5 respectively. Moreover, our approach outperforms the multi-sensor based methods, which is more challenging for SUN RGB-D due to its relatively poor point clouds.
4.4 Ablation Studies and Discussions
We conduct extensive ablation studys on the val sets of ScanNet to analyze individual components of our proposed method. Following [22] report the average performance of 25 trials by default.

---

Table 5: Comparison with other Rol pooling approaches.
Table 6: Ablate the depth of Rol module.
Rol Method
m.P@0.25
mAP@@..5
memory
Ge. = 1...
mAP@0.25
mAP@0.5
PointRCNN 73.65
57.83
8,054MB
{1}
72.33
58,42
Part-A² 32
74.01
58.89
6,540MB
[7,1}
74.50
60.31
Ours-SA 29
73.89
58.14
11,508MB
{7,5, 1]
73.91
60.61
Ours-SpCone
74.50
60.31
2,468MB
17,5,3,13 73.72
59.50
Table 7: Effect of different proposal sampling reso-
Table 8: Effect of using different grouping kernel sizes A(P) in the 1st sparse abstraction.
lution G2 x Gy x G2 in the 1st sparse abstraction.
G2 Gy x Gz
mAP@0.25
mAP@0.5
k(P)
mAP@0.25
mAP@0.5
3 3 x
73.56
59.61
3 X
3
73.87
59.82
x x
74.21
60.09
5
x
5
74.50
60.31
7 7
74.50
60.31
7 7
74.47
60.39
9x9x9
74.41
60.43
9 x
74.52
60.42
11x11 74.44
60.49
X
11
74.45
60.46
Effect of class-aware local grouping module. We first ablate the effects of class-aware local grouping module in Table 2 In Table 2 the base competitor (1*t row) is the fully sparse convolutional VoteNet (27 we implemented. Compared with CAGroup3D, abandons the two stage refinement, replaces the BiResNet with FPN-based variant and groups the vote voxels with the voxel size of 0.02 in a class-agnostic manner. As evidenced in the 1 and the 2nd rows, by considering the semantic consistency, our model performs better, i.e., 68.22 88.22 69 69.24, 53.17 7 54.05 on mA and mAP@0.5. Combining the semantic predictions with class spessfse local groups (3nd rows), our model achieves significant improvement, i.e., 68.22 > 72.10, 53.17 57.07. That verifies our motivation that semantic consistency within the same group and diverse locality among different categories are crucial for an effective grouping algorithm.
Our class-aware local grouping module also works well for a range of hyper-parameters, such as the scale factor a of class-specific re-voxelization and the semantic threshold т. Table 3 the performance of our model with different scale factors. We first set k((a) to and reduce the scale factor a gradually. With the decrease a, the output voxels are more fine-graid grained, which beneficial to estimate accurate bounding boxes and localize small objects. However, very small voxel size will degrade the performance. With the fixed k((*), smaller voxel sizes will lead to smaller local regions, which may mis-group the boundary object points, and thus it is hard for the network to accurately capture local object geometry. Table also ablates the effectiveness of different semantic threshold. We observe that the performance gradually improves with the decrease т. However, too small threshold can also drops the performance due to the abandonment of semantic consistency.
Effect of RoI-Conv pooling module. Table 2 demonstrates the effectiveness of two-stage refinement with Rol-Conv pooling module. By comparing the 4rd and 5th rows, find that our refinement module can really help to detect more accurate 3D bounding boxes, especially in term of mAP@0.50, i.e., 57.18 + 60.31. 3..... To further ablate the high performance of our Rol-Conv pooling module, we compare it to several pooling strategies [30, 32 widely used in 3D object detection. For a fair comparison, only switch the Rol pooling algorithm while all other settings remain unchanged, (e.g., class-aware local grouping and BiResNet). Table shows that our approach surpasses others on the performance of both detection scores and computation cost with remarkable margin. Move details about the above competitors are in Appendix. Table 6 shows the impact of stacking different number of sparse abstraction blocks with (())= except the last layer. We choose the relative shallow design of two layers as there are no obvious improvement by additional deepening. We further ablate the influence of proposal sampling resolution and sparse kernel size k(P) based on our two layers architecture. Table 7 and g show that both the larger G and k(P) can capture more fine-grained geometric details and lead to better performance. Considering the trade-off between memory usage 7 and 5. Effeec of bilateral feature learning. In Table 2 we investigate the effects of bilateral backbone (BiResNet18) by replacing it with FPN-based ResNet18 7 [4]. The 3t and 4thh rows exhibit that the
and performance improvement, our model finally sets G and k(P) of the 1et layer

---

performance drops, especially in term of mAP@0.25, from 73.21 - 72.10, when adopts FPN-based backbone. This phenomenon validates that our bilateral backbone could learn much richer contextual information while maintain the high-resolution representations.
5 Conclusion
In this paper, we propose CAGroup3D, two-stage fully convolutional 3D object detector, which generates some 3D proposals by utilizing the class-aware local grouping module the object voxels with same semantic predictions. Then, to efficiently recover the features of the missed voxels due to incorrect semantic segmentation, design fully sparse convolutional Rol pooling module, which memory-and-aa and-computation efficient and could better encode spatial information than previous max- pooling based Rol methods. Equipped with the above designs, model achieves state-of-the-art on ScanNet V2 and SUN RGB-D benchmarks with remarkable performance gains.
Limitations. CAGroup3D mainly focuses on the inter-category locality, which is class-specific and diverse among the different classes, but ignores the intra-category discriminations. Due to the incompleteness of the point cloud and the scale variance within the classes, the object spatial dimension the same class is also variational, which leads to the diverse intra-category locality. Although grouping algorithm can implicitly handle this problem in some degree by the learnable convlutional aggregation module, is still an open problem and will be studied the future.
Acknowledgement
Liwei Wang supported by National Science Foundation of China (NSFC62276005), The Major Key Project of PCL (PCL2021A12) Exploratory Research Project of Zhejiang Lab (No. 2022RC0AN02), and Project 2020BD006 supported by PKUBaidu Fund. We gratefully acknowledge the support of MindSpore
References
[1] Ronald T Azuma. A survey of augmented reality. Presence: teleoperators & virtual environ- ments, 1997.
[2] Mayank Bansal, Alex Krizhevsky, and Abhijit Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. 2018.
[3] Mark Billinghurst, Adrian Clark, and Gun Lee. A survey of augmented reality. 2015.
[4] Jintai Chen, Biwen Lei, Qingyu Song, Haochao Ying. Danny Z Chen, and Jian Wu. A hierarchical graph network for 3d object detection on point clouds. In CVPR, 2020.
[5] Bowen Cheng, Lu Sheng, Shaoshuai Shi, Ming Yang, and Dong Xu. Back-tracing representative points for voting-based 3d object detection point clouds. In CVPR, 2021.
[6] Angela Dai, Angel X. Chang, Manolis Savva, Macicj Halber, Thomas Funkhouser, and Matthias Nicßner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, 2017.
[7] Anton Konushin Danila Rukhovich, Anna Vorontsova. Feaf3d: Fully convolutional anchor-free
3d object detection. arXiv preprint arXiv:2112.00322 2021.
[8] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. AAAI, 2021.
[9] Shaocong Dong, Lihe Ding, Haiyang Wang, Tingfa Xu, Xinli Xu, Jie Wang, Ziyang Bian, Ying Wang, and Jianan Li. MsSVT: Mixed-scale sparse voxel transformer for 3d object detection on point clouds. In NeurIPS, 2022.
[10] Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias Nießner. 3d-mpa: Multi-proposal aggregation for 3d semantie instance segmentation. In CVPR, 2020.
"haps//wwwwminddspr..nll
10

---

[11] Lue Fan, Ziqi Pang. Tianyuan Zhang, Yu-X Yu-Xiong Wang, Hang Zhao, Feng Wang. Naiyan Wang. and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. In CVPR, 2022.
[12] Benjamin Graham and Laurens van der Maaten. Submanifold sparse convolutional networks. arXiv preprint arXiv: 10dv:1706.011 0000000337,00000
[13] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. 3d semantic segmentation with submanifold sparse convolutional networks. In CVPR, 2018.
[14] Jun Young Gwak, Christopher Choy, and Silvio Savarese. Generative sparse detection networks for 3d single-shot object detection. In ECCV, 2020.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[16] Yuanduo Hong, Huihui Pan, Weichao Sun, and Yisong Jia. Deep dual-resolution networks for real-time and accurate semantic segmentation of road scenes. arXiv preprint arXiv:2101.0608; 2021.
[17] Hou, Angela Dai, and Matthias Nießner. 3d-sis: 3d semantic instance segmentation of rgb-d scans. In CVPR, 2019.
[18] Jikai Jin, Bohang Zhang, Haiyang Wang, and Liwei Wang. Non-convex distributionally robust optimization: Non-asymptotic analysis. In NeurlPS, 2021.
[19] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019.
[20] Tsung-Yr Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[21] Tsung-Yr Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
[22] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In ICCV, 2021.
[23] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[24] Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. CVPR, 2021.
[25] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR, 2017.
[26] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas Guibas. Frustum pointnets for 3d object detection from rgb-d data. In CVPR, 2018
[27] Charles R Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In ICCV, 2019.
[28] Charles R Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas. Imvotenet: Boosting 3d object detection in point clouds with image votes. In CVPR, 2020.
[29] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnct++: Deep hierarchical feature learning on point sets in metric space. In NewrlPS, 2017.
[30] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In CVPR, 2019.
[31] Shaoshuai Shi, Chaoxu Guo, Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In CVPR, 2020.

---

[32] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. TPAMI, 2020.
[33] Shuran Song. Samuel Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In CVPR, 2015.
[34] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang. Deep high-resolution representation learning for human pose estimation. In CVPR, 2019.
[35] Thang Vu, Kookhoi Kim, Tung M Luu, Thanh Nguyen, and Chang D Yoo. Softgroup for 3d instance segmentation on point clouds. In CVPR, 2022.
[36] Dequan Wang, Coline Devin, Qi-Zhi Cai, Philipp Krähenbühl, and Trevor Darrell. Monocular plan view networks for autonomous driving. In IROS, 2019.
[37] Haiyang Wang, Wenguan Wang. Xizhou Zhu, Jifeng Dai, and Liwei Wang. Collaborative visual navigation. arXiv preprint arXiv:2107.011: 2021.
[38] Haiyang Wang, Shaoshuai Shi, Ze Yang, Rongyao Fang, Qi Qian, Hongsheng Li, Bernt Schiele, and Liwei Wang. Rbgnet: Ray-based grouping for 3d object detection. In CVPR, 2022.
[39] Jingdong Wang, Ke Sun, Tianheng Cheng, Borui Jiang, Chaorui Deng, Yang Zhao, Dong Liu, Yadong Mu, Mingkui Tan, Xinggang Wang. et al. Deep high-resolution representation learning for visual recognition. TPAMI, 2020.
[40] Jun Wang, Shiyi Lan, Mingfei Gao, and Larry S Davis. Infofocus: 3d object detection for autonomous driving with dynamic information modeling. In ECCV, 2020.
[41] Yikai Wang, Xinghao Chen, Lele Cao, Wenbing Huang, Fuchun Sun, and Yunhe Wang, Multi- modal token fusion for vision transformers. In CVPR, 2022.
[42] Qian Xie, Yu- Lai, Jing Wu, Zhoutao Wang, Yiming Zhang, Kai Xu, and Jun Wang. Mlevnet: Multi-level context votenet for 3d object detection. In CVPR, 2020.
[43] Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Dening Lu, Mingqiang Wei, and Jun Wang. Venet: Voting enhancement network for 3d object detection. In ICCV, 2021.
[44] Xinli Xu, Shaocong Dong, Lihe Ding, Jie Wang, Tingfa Xu, and Jianan Li. Fusionrenn: Lidar-camera fusion for two-stage 3d object detection. arXiv preprint arXiv:2209.10733 2022.
[45] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 2018.
[46] Bin Yang, Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In CVPR, 2018
[47] Hao Yang, Chen Shi, Yihong Chen, and Liwei Wang. Boosting 3d object detection via object- focused image fusion. arXiv preprint 22x.:22077.10888 2022.
[48] Zetong Yang. Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dent 3d object detector for point cloud. In ICCV, 2019.
[49] Li Yi, Wang Zhao, He Wang, Minhyuk Sung, and Leonidas Guibas. Gspn: Generative shape proposal network for 3d instance segmentation in point cloud. In CVPR, 2019.
[50] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, 2021.
[51] Bohang Zhang, Jikai Jin, Cong Fang, and Liwei Wang. Improved analysis of clipping algorithms for non-convex optimization. In NeurIPS, 2020.
[52] Zaiwei Zhang, Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. In ECCV, 2020.
12

---

[53] Yu Zheng, Yueqi Duan, Jiwen Lu, Jie Zhou, and Qi Tian. Hyperdet3d: Learning scene- conditioned 3d object detector. CVPR, 2022.
[54] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, 2018.
[55] Yuke Zhu, Roozbeh Mottaghi, Eric Kolve, Joseph J Lim, Abhinav Gupta, Li Fei-Fei, and Ali Farhadi. Target-driven visual navigation in indoor scenes using deep reinforcement learning. In ICRA, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section S
(c) Did you discuss any potential negative societal impacts your work? [No] Our work is only for academic research purpose.
(d) Have you read the ethies review guidelines and ensured that your paper conforms them? (Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Our implemen- tation details are posted on Section 4.2 and the supplemental materials. The data are public datasets and our code will be available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See Section 4.2 and Appendix.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [Yes] We report both the best and average performance of 25 trials by default.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.2
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] ScanNet V2 6 and
SUN RGB-D [33]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
We don't It include any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're re using/curating? [Yes] ScanNet V2 and SUN RGB-D 33 are two standard benchmarks.
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] ScanNet V2 6 and SUN RGB-D 33 are two standard benchmarks.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We didn't use crowdsourcing or conduct research with human subjects...
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] We didn't use crowdsourcing or conduct research with human subjects...
13

---

(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We didn't use crowdsoureing or conduct research with human subjects...
14


Reviews:
[{"review_id": "sWhvCjtssLG", "reviewer": "Reviewer_QGEb", "summary": "The authors introduced a novel two-stage 3D point cloud framework in indoor environment. In the previous bottom-up framework, the authors firstly modified the class-agnostic local grouping scheme, which may fail in clustering the object close to different categorical object. Instead of grouping class-agnostic manner as VoteNet, the proposed method grouped the voxels with the semantic consistency from the additional semantic prediction branch. Furthermore, the authors proposed an efficient ROI pooling module to compensate the miss-classified semantic prediction in the second stage. In the experimental section, the authors provided experiments on ScanNet V2 and SUN RGB-D benchmarks to demonstrate the proposed methods succeeds in grouping the objects within the same group and shows the better performance compared to existing class-agnostic methods.", "questions": "Feedbacks of weeknesses for validity of RoI pooling and opinion to reinforce other contributions.", "limitations": "I mentioned all comments including reasons and suggestions in the above sections. I recommend that the author will provide all the concerns, and improve the completeness of the paper (there is few grammatical error).", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[+] First of all, the motivation of the paper seems to be meaningful and pragmatic for indoor 3D object detection, because different types of objects may be located in a short distance in the indoor environment unlike the outdoor environment.\n\nDespite of lacks of technical novelties, the authors well modified the existing class-agnostic model to solve the desired problem. The backbone model is selected to consider the geometric characteristic of the indoor object target, and the pooling module is designed for sparse convolution to handle limitations as computational overhead and geometric information loss.\n\nAs the detection paper, the overall framework is well-described, and specifically the details for reproducing and understanding the contribution such as target assignment, handling unique set, comparisons on different kernel size etc. are well presented.\n\n\n[-] One concern is the proposed RoI pooling module. In sparse convolution, there is a hand-crafted parameter such as voxel size, and according to this factor, geometric loss occurs like existing pooling methods. \n\nAnother concern is the role of the diverse local grouping. In Table 2, this factor has the greatest impact on performance improvement compared to other factors. Among all the contributions, this factor does not look much different from operating a class-agnostic model for each class efficiently. Therefore, I recommend that the authors should provide the opinion to reinforce other contributions.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "qZ0arGpAcvO", "reviewer": "Reviewer_7kNx", "summary": "This paper presents an approach for 3D detection from voxels. The authors propose to 2 stage detection pipeline that (1) first generates high quality box proposals by considering the spatial features of the scene. Compared to prior work, this stage also uses semantic information associated with each voxel. (2) Refines the object proposals and aggregates per-object features by a ROI pooling module. While prior work uses Set Abstraction (proposed in PointNet++) or max-pooling, the authors propose a new Sparse Abstraction operator. The final model is evaluated on indoor 3D detection on the ScanNet and SUN RGB-D datasets. ", "questions": "+ How are the weights in Eqn (8) tuned? How sensitive is the model to these weights?\n+ Did you try any simpler alternatives to the re-voxelization step? For example, rather than creating separate subsets, did you try feeding in the semantic probabilities of the classes into the aggregation module (which could be class-specific)?\n+ Do you mean gIOU loss when you mention IOU loss in L237? Also, is the VoteNet baseline in Table 2 also trained with the IOU loss?\n+ Are there particular classes for which this model is better than prior work? Thin/small objects? Since the authors spend effort in accounting for each object class’s size while aggregating features, I expect this design decision may work particularly well for some classes.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n+ The authors report the results averaged over 25 trials in each table. They account for both the training time stochastic behavior as well as inference time stochastic behavior in 3D detection models. This is important for follow-up work as it (a) empirically validates the current paper’s claims; (b) helps understand the robustness of each proposed component.\n+ Strong empirical performance on indoor detection benchmarks. In particular, as Table 1 shows, the proposed model achieves a much higher mAP@0.5 than prior work.\n+ The Class-aware proposal prediction is well motivated. In particular (L143), predicting a dense semantic map before creating the proposals is a good idea since it additionally supervises the feature backbone. I also liked that the authors directly use the 3D bounding boxes for this part rather than segmentation masks (L158).\n+ The 3D backbone inspired by HRNet is a good choice for trading of resolution and compute. It also leads to good gains in performance (Table 2)\n\nWeaknesses\n+ The title suggests that the authors use point clouds directly. However, their implementation first converts the point clouds into voxels (L102). The title is misleading. Similarly, the introduction talks about challenges of using unordered point clouds which do not exist once point clouds are converted to voxels, and thus are not challenges either faced by this work or addressed by it. Please update the title/introduction.\n+ The re-voxelization step introduced in Eqn (4) seems pretty involved - (a) it requires a threshold per class which is changed with a custom schedule during training. This seems like a non-trivial hyper parameter search. (b) It additionally requires re-voxelization using using a separate parameter \\alpha. (c) Finally, it requires aggregation using a kernel k_alpha. I worry that this step will need re-tuning on different datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Q2dNQ0cdLP", "reviewer": "Reviewer_uwGH", "summary": "Existing 3D indoor object detection usually abandons semantic consistency within the same group and ignores diverse locality among different categories. To tackle this issue, the authors propose a novel class-aware proposal generation strategy that considers class-specific local groups. Besides, the authors propose an RoI pooling module to revisit the missed surface voxel features due to semantic segmentation errors. Their approach outperforms state-of-art methods on two challenging indoor datasets.", "questions": "- I am not sure about the effect of feature offsets. What is the difference between features extracted from the backbone and features added by feature offsets? Please demonstrate the design goal of feature offsets.\n- As the set of neighboring input voxels is dependent on RoI-specific points, sparse abstraction is a time-expensive process. Please compare the inference time between the proposed method and existing methods in the parts of the experiment. \n- In Table 2, it is important for a reader to see all possible combinations of important modules (e.g. class-aware grouping and RoI-Conv) and see their potential influences.  I wonder if a better backbone can alleviate the improvements of local grouping. \n- There is no qualitative analysis in the main paper. It is hard for a reader to judge what are the potential improvements. ", "limitations": "Yes.\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n+ The authors utilize a voxel feature encoding with a class-specific 3D voxel size to obtain the predicted vote voxels and generate class-specific geometric features based on them.\n+ The authors propose RoI-Conv pooling, which effectively recovers the missed surface voxel features due to the semantic segmentation errors\n+ The proposed method achieves state-of-art results with a remarkable gain on ScanNet V2 and SUN RGB-D.\n\nWeaknesses: \n- The idea of using semantic predictions and geometric shifts to get class-aware clusters is not novel, which has already been used in previous work such as SoftGroup.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OEu0QBYtZi", "reviewer": "Reviewer_nq6L", "summary": "The paper addresses the problem of 3D object detection in point cloud for indoor scenes (scannet and sun rgbd). The idea is based on voting-based methods: each point predicts its center -> clustering -> classification. The method proposes to classify points first, before center prediction and clustering. They claim contributions in the following:\n\n- Novel \"class-aware\" 3D proposal strategy as discussed above.\n\n- A refinement stage that re-voxelizes according to each point's classes. \n\n- Good results on scannet and sun rgb-d. \n\nThe results do seem strong. And both claimed novelties contributed to the improvements. I'll recommend a weak accept for now. Might change opinions based on peers' reviews. \n", "questions": "N/A", "limitations": "There is one sentence. What do the authors mean by diversity localities?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Strong results on both datasets as mentioned. Also good ablation on table 2 to demonstrate the improvements from the claimed novelties. \n\n- Good writing. \n\nWeaknesses:\n\n- Only somewhat sufficient novelty: I don't feel that reordering the steps of a voting-based method constitutes to sufficient novelty for this venue, but maybe only barely together with the 2nd-stage re-voxelization idea. A 2nd-stage detector is followed by initial points segmentation is not really new either. See [RSN: Range Sparse Net for Efficient, Accurate LiDAR 3D Object Detection]\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept