V-DETR: DETR with Vertex Relative Position Encoding for 3D Object Detection
Authors: Yichao Shen
Zigang Geng
Yuhui Yuan
Yutong Lin
Ze Liu
Chunyu Wang
Han Hu
Nanning Zheng
Baining Guo
Venue: ICLR 2024 poster (2024)
Keywords: 3D Object Detection
Detection Transformer
Relative Position Encoding
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: We introduce a highly performant 3D object detector for point clouds using the DETR framework. The prior attempts all end up with suboptimal results because they fail to learn accurate inductive biases from the limited scale of training data. In particular, the queries often attend to points that are far away from the target objects, violating the locality principle in object detection. To address the limitation, we introduce a novel 3D Vertex Relative Position Encoding (3DV-RPE) method which computes position encoding for each point based on its relative position to the 3D boxes predicted by the queries in each decoder layer, thus providing clear information to guide the model to focus on points near the objects, in accordance with the principle of locality. Furthermore, we have systematically refined our pipeline, including data normalization, to better align with the task requirements. Our approach demonstrates remarkable performance on the demanding ScanNetV2 benchmark, showcasing substantial enhancements over the prior state-of-the-art CAGroup3D. Specifically, we achieve an increase in $AP_{25}$  from $75.1\%$ to $77.8\%$ and in  ${AP}_{50}$ from $61.3\%$ to $66.0\%$.

Published as a conference paper at ICLR 2024
V-DETR: DETR WITH VERTEX RELATIVE POSITION ENCODING FOR 3D OBJECT DETECTION
Yichao Shen'r, Zigang Geng²*, Yuhui Yuan"*i, Yutong Lin', Ze Liu?, Chunyu Wang", Han Hu", Nanning Zheng"! i Baining Guo
"National Key Laboratory of Macchne Hybrid Augmented Intelligence, National Engineering Research Center for Visual Informationand Applications, and Institute Artificial Intelligence and
Roboties, Xi' an Jiaotong University
Uniivrsity of Seience and Technology of China
S MMirosoott Research Asia
ABSTRACT
We introduce a highly performant 3D object detector for point clouds using the DETR framework. The prior attempts all end up with suboptimal results because they fail to learn accurate inductive biases from the limited scale of training data. In particular, the queries often attend to points that are far away from the target objects, violating the locality principle in object detection. To address the lim- itation, we introduce a novel 3D Vertex Relative Position Encoding (3DV-RPE) method which computes position encoding for each point based on its relative position to the 3D boxes predicted by the queries in each decoder layer, thus pro- viding clear information to guide the model to focus on points near the objects, in accordance with the principle of locality. Furthermore, we have systematically refined our pipeline, including data normalization, better align with the task re- quirements. Our approach demonstrates remarkable performance on the demand- ing ScanNetV2 benchmark, showcasing substantial enhancements over the prior state-of- arr CAGroup3D. Specifically, we achieve an increase in AP25 from 75.1% 77.8% and in AP from 61.3% to 66.0%.
INTRODUCTION
3D objectdetection detection from point clouds is a challenging task that involves identifying and localizing the objects of interest present in a 3D space. This space is represented using a collec- tion of data points that have been gleaned from the surfaces of all ac- cessible objects and background in the scene. The task has significant implications for various industries, including augmented reality, gaming, robotics, and autonomous driving.
Transformers have made remarkable advancement in 2D object detec- tion, serving both powerful back- bones (Vaswani et al.; 2017 Liu
Figure The left column shows the 3D scans from the Scan- NetV2 in the reat/fronttop-down view. We display one of the ground-trui truth bounding boxes with a green 3D box. The middle col- umn shows the decoder cross-attention map hased plain DETR. Attention weights are distributed over many positions even out- side the ground-truth box. The right column shows the decoder cross- aetentattntoon map based on plain DETR + 3DV-RPE. Attention weights focus on the sparse object boundaries of the object located in the ground-truth bounding box. The color indicates the attention values: yellow for high and blue for low.
al. 2021a) and detection architec- tures (Carion et al. (2020). How-
ever, their performance in 3D detec- tion (Misra et al.) 202T) is signifi- cantly worse than the state-of-the-ar methods. Our in in-depth evaluation of 3DETR (Misra et al. 202I) revealed that the queries often attend to points that are far away from the target objects (Figure (b) shows three typical visualizations), which violates the principle of locality in object detection. The prin- ciple of locality dictates that object detection should only consider subregions of data that contain
"Core contribution.
*Corresponingg Author.

---

Published as a conference paper at ICLR 2024
the object of interest and not the entire space. Besides, the behavior is also in contrast with the success that Transformers have achieved in 2D detection, where they have been able to effectively learn the inductive biases, including locality. We attribute the discrepancy to the limited scale of training data available for 3D object detection, making it difficult for Transformers to acquire the correct inductive biases.
In this paper, we present a simple yet highly performant method for 3D object detection using the transformer architecture DETR (Carion et al., 2020). To improve locality in the cross-attention mechanism, we introduce a novel 3D Vertex Relative Position Encoding (3DV-RPE) method. It computes position encoding for each point based on its relative offsets to the vertices of the pre- dicted 3D boxes associated with the queries, providing clear positional information such as whether each point inside the boxes. This information can utilized by the model to guide cross-attention to focus on points inside the box, in accordance with the principle of locality. The prediction of these boxes is consistently refined as the decoder layers progress, resulting in increasingly accurate posi- tion encoding.
To mitigate the impact of object rotation, we propose to compute 3DV-RPE a canonical object space where all objects are consistently rotated. Particularly, for each query. we predict rotated 3D box and compute the relative offsets between the 3D points rotated in the same way, and the eight vertices of the box. This results in consistent position encoding for different instances of the same object regardless of their positions or orientations in the space, greatly facilitating the learning of the locality property in cross-attention even from limited training data. Figure | (c) visualizes the attention weights obtained by our method, We can see that the query for detecting the chair nicely attends to the points on the chair. Our experiment demonstrates that 3DV-RPE boosts the performance.
We also systematically enhance our pipeline from various aspects such as data normalization and network architectures based on our understanding of the task. For example, we propose object- based normalization, instead of the scene-based one used by the DETR series, to parameterize the 3D boxes. This is because the former is more stable for point clouds which differs from 2D detection where the sizes of the same object in images can be very different depending on the camera param- eters, impelling them to use image size to coarsely normalize the boxes. Besides, we also evaluate and adapt some of the recent advancement in 2D DETR.
We conduct thorough experiments to empirically show that our simple DETR- approach sig- nificantly outperforms the previous sfate-of-the-eee fully convolutional 3D detection methods, which helps to accelerate the convergence of the detection head architecture design for 2D and 3D detection tasks. We report the results of our approach on two challenging indoor 3D object detection bench- marks including ScanNetV2 and SUN RGB-D. Overall, compared to the DETR baseline (Misra) et al. 2021). ). our method with 3DV-RPE improves AP25/APo from 65.0%/47. .0% to 77 77.8%/66.0% respectively, and reduces the training epochs by 50%. Particularly, on Sean our approach outperforms the very recent state-of-the-ar CAGroup3D (Wang et al. 2022a) by +2.7%/44.79 measured by AP2s/AP3o so. respectively.
2 RELATED WORK
bETR- Object Detection. DETR (Carion et al. 2020) is a groundbreaking work that applies transformers Vaswani et al.) 2017) to 2D object detection, eliminating many hand-designed compo- nents such as non-maximum suppression (Neubeck & Van Gool 2006) or anchor boxes Girshick 2015 Ren et al. 2015 Lin et al. 2017 Liu et al. 2016). Many extensions of DETR have been pro- posed (Meng et al., 2021 Gao et al. 2021 1 Dai et al. al.] 202T] Wang et al. 202 Jia et al. 2022 Zhang al. 2022), such as Deformable-DEi DETR (Zhu al. (020). which uses multi scale deformable atten- tion to focus on key sampling points and improve performance on small objects. DAB-DETR (Liu al. 2022) introduces novel query formulation to enhance detection accuracy. Some recent works (Li et al., 2022; Zhang et al.] 2022, Jia et al. 2022 Chen et al. (022) achieve state-of-the- art results on object detection by using query denoising or one-to-many matching schemes, which addressed the training inefficiency of one-to-one matching. H-DETR (Jia et al.) 2022) shows that one-to-many matching can also speed up convergence on 3D object detection tasks. Following the DETR- based approach, GroupFree (Liu et al. 2021b) and 3DETR (Misra et al. 2021) built strong 3D object detection systems for indoor scenes. However, they are still inferior to other methods such as CAGroup3D (Wang et al. 2022a). In this work, we propose several critical modifications
2

---

Published as a conference paper at ICLR 2024
(a) voxelized sparse input
(b) Voting hased method
(c) Expansion based method
(d) DETR: based method
Figure 2:
(a) A simplified sparse 3D voxel space from top-down perspective. The curve shows the input
surface and the small cubes show the voxelized input. The gray five-pointed star * shows the object's center. (b) The voting scheme estimates offsets for each voxel and we color the voxels (nearer to the object's center) with yellow after voting. The dashed small cubes show the empty space after voting. (c) The generative sparse decoder (GSD) scheme enlarges the voxels around the surfaces, thus creating new voxels both inside and outside of the object (marked with yellow cubes). (d) The DETR-hased approach simply selects a small set of voxels (marked with yellow cubes) as the initial object query and iteratively predicts the boxes by refining (marked with the open yellow cireles) the object query with multiple Transformer decoder layers. We follow the DETR- based path this work.
to improve the DETR- methods and achieve new records on two indoor 3D object detection tasks.
3D Indoor Object Detection. We revisit the existing indoor 3D object detection methods that directly use raw point clouds to detect 3D boxes. We categorize them into three types based on their strategies: (i) Voting-bank -baeed methods, such VoteNet (Qi et al., 2019), MLCVNet (Xie et al. 2020) and H3DNet (Zhang et al. 2020), use a voting mechanism to shift the surface points toward the object centers and group them into object candidates. (ii) Expansion- methods, such as GSDN(Gwak et al. al. 22220), (RukhD(Rukhovich et al. 2022). and CAGroup3D( (Wang et al., (2022a), which generate virtual center features from surface features using generative sparse decoder and predict high-quality 3D region proposals. (iii) DETR-based methods, unlike these two types that require modifying the original geometry structure of the input 3D point cloud, we adopt the DETR- based approach (Liu et al. 2021b; Misra et al.) 2021) for its simplicity and generalization ability. Our experiments show that DETR has great potential for indoor 3D object detection. We show the differences between above-mentioned methods in Figure 2
3D Outdoor Object Detection. We briefly review some methods for outdoor 3D object detec- tion (ann et al. 2018 Zhou & Tuzel 2018 Lang al. 2019 Yin et al., 2021), which mostly trans- form 3D points into a bird-eye-view plane and apply 2D object detection techniques. For example, VoxeINet (Zhou & Tuzel 2018) is single-stage and end-to-end network that combines feature extraction and bounding box prediction. PointPillars (Lang et al. 2019) uses 2D convolution neu- ral network to process the flattened pillar features from Bird's 's Eye View (BEV). CenterPoint (Yin) et al. 2021) first detects centers of objects using a keypoint detector and regresses to other attributes, then refines them using additional point features on the object. However, these methods still suffer from center feature missing issues, which FSD (Fan et al. (023b) tries to address. We plan to extend our approach outdoor 3D object detection in the future, which could unify indoor and outdoor 3D detection tasks.
3
OUR APPROACH
3.1 BASELINE SETUP
Pipeline. We build our V-DETR baseline following the previous DETR-based 3D object detection methods MMisra et al. 2021 Liu et al. 2021b). The detailed steps are as follows: given 3D point cloud IxR sampled from 3D sean of an indoor scene, where the RGB values are in the first dimensions and the position XYZ values are in the last 3 dimensions. We first sample about 440K points from the original point cloud that typically has around ~20K points. Second, we use a feature encoder to process the raw sampled points and compute the point features F Third, we construct a set of 3D object queries Q send them into a plain Transformer decoder to predict set of 3D bounding boxes B We set K 1024 by default. Figure shows the overall pipeline. We present more details on the encoder architecture design, the 3D object query construction, the Hungarian matching, and loss function formulations as follows.
Encoder architecture. We choose two different kinds of encoder architecture for experiments in- cluding: (i) PointNet followed by shallow Transformer encoder adopted by Misra et al. (2021)
3

---

Published as a conference paper at ICLR 2024
or (ii) sparse 3D modification of ResNet34 followed by FPN neck adopted by Rukhovich al. 2022), , where we replace the expensive generative transposed convolution with a simple transposed
convolution within the FPN neck.
3D object query. We construct the 3D object query by combin- ing two kinds of representations as follows: first, we simply sam- ple set of K initial center po- sitions over the entire encoder output space and select their rep- resentations initialize set of 3D content query Q.. Then we use their XYZ coordinates the input point cloud space to com- pute the 3D position query O₂ with simple MLP consisting of two linear layers. We build the 3D object query by adding the 3D position query to the 3D content query.
monda supg palle
dunb snd OR
400
Traneformee Decoder
Encoder
(3DV-RPE)
Figure 3: Illustrating the overall framework of V-DETR for 3D ob- ject detection. We first encoder to extract 3D features, and then we use plain Transformer decoder estimate the 3D object queries from set initialized 3D object queries. the Transformer decoder multi-head cross-attention layer, we use a 3D vertex relative position en- coding scheme for both locality and accurate position modeling.
Hungarian matching and loss function. We choose the weighted combination of six terms in- cluding the bounding box localization regression loss, angular classification and regression loss, and semantic classification loss as the final matching cost functions and training loss functions. We illustrate the mathematical formulations as follows:
LETR = -1 GToU(b, b) >x c) + A Aa, FL,LL(((]) -s,Lhuheera,, ar) + Ao CB (C,,,,,aaa an),
where we use b, C, S. p. à (or b, c, s, I, a) to represent the predicted (or ground-truth) bounding box, box center, box size, classification score, and rotation angle respectively, e.g., represents the ground-truth semantic category of b. CE represents angle classification cross entropy loss and Lhuber represents the residual continuous angle regression loss. FL represents semantic classification focal loss.
Object-normalized box parameterization. We propose an object-normalized box reparameteri- zation scheme that differs from the original DETR (Carion et al. 2020). which normalizes the box predictions by the scene scales. We account for one key discrepancy between object size variation in 2D images and 3D point clouds, e.g., a chair's ' 2D box size may change depending on its distance to the cameras, but its 3D box size should remain consistent as the point cloud captures the real 3D) world. In the implementation, we simply reparameterize the prediction target of width and height from the original groud-truth by, and b bj. /b/, the coarsely predicted box height and width.
(-1 and b Db//b/-1 where b/, and b represent
3.2 3DV-RPE IN CANONICAL OBJECT SPACE
Position Encoding (PE) is crucial for enhancing the ability of transformers to comprehend the spatial context of the tokens. The appropriate PE strategy depends on tasks. For 3D object detection, where geometry features the primary focus, it is essential for PE to encode rich semantic positions for the points, whether they are on/off the 3D shapes of interest.
To that end, present 3DD Vertex Relative Position Encoding (3DV-RPE), novel solution specif- ically tailored for 3D object detection within the DETR framework. We modify the global plain Transformer decoder multi-head cross-attention maps follows:
Á Softmax QKT + R).
where Q and K represent the sparse query points and dense key-value points, respectively. R rep- resents the position encoding computed by 3DV-RPE that carries accurate position information.
3DV-RPE. Our key insight that, encoding a point by its relative position to the target object, which is coarsely represented by a box, is sufficient for 3D object detection. It is computed as follows:
P. MLP (FAP)))
4

---

Published as a conference paper at ICLR 2024
where AP, RKxNxx denotes the offsets between the N points and the i-th vertex of the K boxes and P. E RkxN*x represents the relative position bias term. h is the number of heads. F(-) is non-linear function. We will evaluate several alternatives for F()) in the experiments. MLP, represents an MI. based transformation that first projects the features to a higher dimension space, and then to the output features of dimension h. We obtain the final relative position bias term by adding the bias term of the eight vertices, respectively:
R Σ P..
(3)
==1
where R. encodes the relations between the 3D boxes and the points. In the subsequent section, we will introduce how compute AP, with the aid of the boxes predicted current layer.
Canonical Object Spaces. It is worth noting that the direction of the offsets are dependent on the definition of the world coordinate system and the object orientation, which complicates the learning of semantic position en- propose to trans-
VA
ArQ
y
coding. To address the limitation, form to an object coordinate system defined by the ro- tated bounding box. As illustrated in Figure 4 an offset vector the world coordinate system can be transformed the object coordinate system (rra yo) following:
Auo.
Ar
VO
Awa Aye
Teos sin A
sin 0 0 cos
Ax Ay 2
R Ap, (4)
Azg
0
O
1
where Ap is an element of AP; We use the other trans- formations in Equation 2 and Equation to get the final normalized relative position bias item that models the ro- tated 3D bounding box position information. We perform 3DV-RPE operations for different Transformer decoder layers by default. We have observed that the canonical co- ordinate transformation bears resemblance to our earlier examination of Point-RCNN (Shi et al.) 2019). In both instances, the bounding box undergoes transformation into a canonical object space. However, significant dif- ference lies in the objectives of the canonical operation. In Point-RCi RCNN, the objective is to achieve precise refine-
Figure 4: Canonical object space trans- formation for 3DV-RPE. The rectangle represents the box of the object, which de- fines an object coordinate system. The green line represents the offset from a point to the box vertex. The offset transformed to the ob- ject coordinate system is (Arg, Ayo) where the exact values can be geometrically rea- soned. Since there no rotation along the z z-axis on the current datasets, we only show the changes in the x-y plane.
ment by transforming the points contained within the 3D bounding box proposals. Conversely, our primary focus is on transforming the offset vectors to align with the object coordinate system.
Efficient implementation. A naive implementation has high GPU memory consumption due to the large number of combinations between the object queries (each object query predicts 3D bounding box) and the key-value points (output by the encoder), which makes it hard to train and deploy.
To solve this challenge, we use smaller pre-defined 3DV-RPE table of shape: T E 100100010 which represents discretized set of possible F(AP,) ) that we interpolate into. Detailed information about 3DV-RPE table settings can be found Appendix We do volumetric (5-D) sample on the transformed 3DV-RPE table follows:
P; = grid wer.ssamlleMMPP... F(AP.)).
(5)
3.3 DETR WITH 3DV-RPE
Framework. We extend the original plain Transformer decoder, which consists of a stack of decoder layers and designed for 2D object detection, to detect 3D bounding boxes from the irregular 3D points. Our approach has two steps: (i) as the first decoder layer has no access to coarse 3D bounding boxes, employ light- FFN predict the initial 3D bounding boxes and feed
5

---

Published as a conference paper at ICLR 2024
the top confident ones to the first Transformer decoder layer (e.g., { .y, w0, 1,, h))); and (ii) we update the bounding box predictions with the output of each Transformer decoder layer and use them to compute the modulation term in the multi-head cross-attention.
Figure illustrates more details of the DETR with 3DV-RPE. For instance, we employ only the 3D content query Q. as the input for the first de- coder layer and the decoder output embeddings Q from the (i - 1)-th decoder layer as the in- put for the i-th decoder layer. We also apply MLP projects to compute the absolute position encod- ings of the 3D bounding boxes by default. We set the number of decoder layers as 8 following Misra jtt al. (202T). We predict the 3D bounding box delta target based the initial prediction such as {0o, x", 39,3, 20, w 10, h1) in all the Transformer de- coder layers.
Briakie Jogl Dradlhe Dox
Layet
Add
Decode Baldddaa
Luyer 2
30 Sonniin
Tapor 1
Pattie
Visualization. Figure shows the relative position attention maps learned with the 3DV-RPE scheme. We show the attention maps for 8 vertices in the first 4 columns and the merged ones in the last column. The visualization results show that (i) our 3DV-RPE can enhance the inner 3D box regions relative to each vertex position and (ii) combining the eight rel- ative position attention maps can accurately localize the regions within the bounding box. We also show that 3DV-RPE can localize the extremity positions on the 3D object surface in the experiments.
Light PTN
Soletio
Seeaki tage bwaade
Figure 5: Illustration of the proposed V-DETR framework. We mark the modifications with yellow-colored regions and the other components, that designed following the plain DETR, with gray-colored regions.
4
EXPERIMENT
4.1 DATASETS AND METRICS
Datasets. We evaluate our approach on two chal- lenging 3D indoor object detection benchmarks including:
ScanNeiV2 (Dai et al. 2017): ScanNerV2 consists of 3D meshes recovered from RGB-D videos captured in various indoor scenes. It has about 12K training meshes and 312 validation meshes, each annotated with semantic and instance segmentation masks for around 18 classes of objects. We follow Qi et al. (2019) to extract the point clouds from the meshes.
SUN RGB-D (Song et al. 2015): SUN RGB-D is single-view RGB-D image dataset. has about 5K images for both training and validation sets. Each image is annotated with oriented 3D bounding boxes for 37 classes of objects. We follow VoteNet (Qi et al. 2019) to convert the RGB-D image to the point clouds using the camera parameters and evaluate our approach on the 10 most common classes of objects.
Metrics. We report the standard mean Average Precision (mAP) under different IoU thresholds, AP25 for 0.25 loU threshold and APso for 0.5 loU threshold.
4.2 IMPLEMENTATION DETAILS
Training, We use the AdamW optimizer (Loshchilov & Hutter 2019) with the base learning rate 7e-4, the batch size 8, and the weight decay 0. The learning rate is warmed up for epochs, then is dropped to le-6 using the cosine schedule during the entire training process. We use gradient clipping to stabilize the training. We train for 360 epochs on ScanNetV2 and 240 epochs on SUN RGB-D in all experiments except for the system-level comparisons, where train for 540 epochs on Sca We use the standard data augmentations including random cropping (at least 30K points), random sampling (100K points), random flipping (p=0.5), random rotation along the z-axis

---

Published as a conference paper at ICLR 2024
Figure 6: Visualizing the learned spatial attention maps based on 3DV-RPE. We use the small red-colored cube to represent the 3D bounding box of object, the red five-pointed star mark the eight vertices, and the entire colored cube as the input scene for simplicity. We average each P, along head dimension according Equation 2 and visualize eight vertices' learned spatial cross-attention maps (from columnfl to columnit4). We visualize the merged spatial attention maps in columni5S (from the cutaway view). The color indicates the attention values: yellow for high and blue for low. We can observe that (i) the learned spatial attention maps of each vertex enhance the regions along the internal direction starting from each vertex position, and (ii) the combined spatial attention maps can accurately enhance the internal regions inside the red-colored cubes.
(-59, 59), random translation (-0.4, 0.4), random scaling (0.6, 1.4). We also use the one-to-many matching (Jia at 2022) to speed up the convergence speed with more rich and informative positive samples.
Inference. We process the entire point clouds of each scene and generate the bounding box pro- posals. use 3D NMS suppress duplicated proposals in the one-to-many matching setting, which is not needed in the one-to-one matching setting. We also use test-time augmentation, i.e., flipping, by default unless specified otherwise.
4.3 ANALYSIS OF PERFORMANCE ENHANCEMENTS BY 3DV-RPE
Table 1: Effect of the position encoding method
Table 2: Implementation of 3DV-RPE 3DETR.
choices. Numbers in () represent results without TTA
method
#epochs
AP2s
APso
position encoding method
#epochs
APas
AP5o
3DETR
1080
65.0
47.0
None (Our Baseline)
360
68.8(67.9)
44.5(43.5)
3DETR 3DV-PRE
1080
71.2
60.8
3D box mask
360
74.0(72.9)
59. (58.3)
3DV-RPE
360
65.0(64.2) 76.7(76.2)
Table 3: Implementation of 3DV-RPE on Group- Free.
mask 3DV-RPE
360
76.0(75.3)
62.7(61.5)
None (Our Baseline)
540
71.4(70.6)
47.6 47.6(46.7)
method
#epochs
AP3s
A.
mask
540
75.1(74.2)
60.8(59.6)
Group-Free
100
69.1
52.8
3DV-RPE
540
77.8(77.4)
66.0(65.0)
3D box mask + 3DV-RPE
540
77.0(76.5)
63.5(62.4)
Group-Frre 3DV-PRE
400
72.8
62.1
Different position encoding methods. Table our 3DV-RPE ScanNetV2 with another position encoding method- - 3D box mask, which sets the relative position encoding term to -0o for positions outside the 3D bounding box and 0 otherwise. The first and fourth display the results of our baseline settings, which have been augmented with various additional enhancements introduced in the section 3.1. The results show that (i) the 3D box mask method achieves strong results on AP25, and (ii) our 3DV-RPE significantly improves over the 3D box mask method on AP method suffers from eтror accumulation from the previous decoder layers and cannot be optimized end-to-end. We also report the results of combining the 3D box mask and 3DV-RPE, which performs better than the 3D box mask scheme but worse than our 3DV-RPE. This verifies that our 3DV-RPE can learn to (i) exploit more accurate geometric structure information within the 3D bounding box and (ii) benefit from capturing useful long-range context information outside the box. Moreover, report the results with longer training epochs and observe that the gap between the 3D box mask and 3DV-RPE remains, thus further demonstrating the advantages of our approach.
essentially
50- We speculate that our 3DV-RPE performs better because the 3D box mask
7

---

Published as a conference paper at ICLR 2024
Table 4: System-level comparison with the state-of-the-art Sca and SUN RGB-D. TTA: test-time
augmentation. denotes using additional self-supervised pre-training. ** denotes using training data.
ScanNerV2
SUN RGB-D
Method
APas
APso
AP3s
AP%o
3DETR Misra al
20211
65.0
47.0
59.1
32.7
Group- Free-Freee Lis
1b
69.1
52.8
AStapehommer
Li Lt al
2023
71.1
36.6
62.2
FCAF3D Rukhovich
2022
71.5
57.3
64.2
48.9
OctFormer Wang
2023
06.2
60.6
UniDDETR Wang ct al.
2023
71.7
38.3
67.0
50.3
ConDuFormer (Duan et al.
2025
67.1
49.9
TR3D (Rakhvvich et
2023
72.9
30.3
67.1
60.4
Point GCC
iFan el.
2023a
73.1
39.6
67.7
51.0
CAGroup3D Wang al.
2022a
75.1
61.3
06.8
50.2
SWIN3D + CACrrou33
Yang
2023
76.4
63.2
V-DETR
77.4
65.0
67.5
50.4
V-DETR
77.8
66
51.1
Average Results under x riily
Group- Frce
2021h
68.6
51.8
FCAF3D Rukhovich et
2022
70.7
36.0
63.6
48.2
TR3D Rakhovich et al.
2023
72.0
57.4
66.3
49.6
CAGioop35
Wang
20228
74.5
60.3
ConDaFormer Duan et al.
2023
66.6
49.5
V-DETR
76.8
64.5
06.8
49.7
V-DETR
77.0
65.5
50.0
Implementation of 3DV-RPE on other baselines. In order to verify the universality and useful- ness of 3DV-RPE, we implement our method on other baselines, such as 3DETR (Misra al. 2021) and Group-Free (Liu et al. 2021b). The comparative analysis, as illustrated in Table 2 and Table 3 demonstrates the performance impact of integrating 3DV-RPE on the V2 benchmark under equitable conditions. It is evident from the results that the incorporation of our 3DV-RPE signifi- cantly improves performance, notably in AP in different baselines.
More analyses. We offer additional comparative analyses between our 3DV-RPE with absolute position encoding methods and other advanced attention modulation methods, to substantiate the superiority of our approach. The detailed results of these comparisons are accessible in Appendix C
4.4 COMPARISONS WITH PREVIOUS SYSTEMS
In Table 4 we compare our method with the state-of-the-an methods of most recent works at the system level, and a more detailed comparison with more previous work can be found in Appendix B These methods use different techniques, so we cannot compare them in a controlled way. According to the results, we show that our method performs the best, either measured by the highest perfor- mance or the average results under multiple trials. For example, on ScanNetV2 val set, our method achieves 2P25=77.88 8% and APso=66.0% which surpasses the latest state-of-the-aa SWIN3D + CA- Group3D (Yang et al., 2023) that reports AP25=76.4* and APso=63.2% while SWIN3D requires additional data for pertaining. Notably, Sca observe more significant gains on AP (+2.8%) that requires more accurate localization, i.e., under a higher loU threshold. We also observe consistent gains both AP25 and AP50 SUN RGB-D.
4.5 3DV-RPE ABLATION EXPERIMENTS
We conduct all the following ablation experiments on ScanNetV2 and trained with 360 epochs for efficiency, except for the ablation experiments on the coordinate system, where report the results on SUN RGB-D.
Number of vertex. Table 5 shows the effect of different numbers of vertices for computing the relative position bias term. The results show that using vertices performs the best, so we use this setting by default. We attribute their close performances to the fact that they essentially share the same minimal and maximal XYZ values when using fewer vertices such as or 4, which is caused by the zero rotation angles on ScanNetV2.
8

---

Published as a conference paper at ICLR 2024
Table Effect of the number of vertex within 3DV.
Table Effect of non non-linear transform within 3DV. RPE.
RPE.
vertex
AP25
APso
J(
AP25
APso
=(x)=1
69.6
48.2
73.4
54.8
F(z) = =(x)=2/(1+) 2/1++
76.0
62.6
76.1
63.1
F(z) tanh(a)
76.3
62.6
76.3
63.4
F(x) = +
76.6
63.0
76.7
65.0
F(x) sign(a) log Ix)
76.7
65.0
Table 7: Effect of the coordinate system on SUN RGB-D.
Table 8: Effect of the ohject-normalized hox param- eterization.
coordinate system
AP25
AP5o
object-normalize
AP26
APso
world coord.
65.8
46.9
74.9
61.1
object coord.
68.0
51.1
76.7
65.0
Non-linear transform. Table shows the effect of different non-linear transform functions. The results show that the signed log function performs the best. The signed log function magnifies small changes in smaller ranges. Therefore, we choose the signed log function by default.
Coordinate system on SUN RGB-D. We evaluate the effect of the coordinate system on calculating the relative positions in our 3DV-RPE on SUN RGB-D, which requires predicting the rotation angle along the z-axis. Table 7 shows the results. We find that transforming the relative offsets from the world coordinate system to the object coordinate system significantly improves the performance, e.g., 25 and A5 increase by +2.2% and 4.2%, respectively.
Object-normalized box parameterization. In Table 8 show the effect of using object- normalized box parameterization. We find using the object-normalized scheme significantly boosts the AP5o so from 61.1 to 65.0.
Qualitative comparisons. We show some examples of V-DETR detection results on SUN RGB-D in Figure 7 and ScanNetV2 in Figure in the Appendix B where the scenes diverse and challenging with clutter, partiality, scanning artifacts, etc. Our V-DETR performs well despite these challenges.
Encoder choice. Table compares the results of using different encoder architectures. We find that
Table 9; Effect of the encoder choice.
encoder
AP2s
A.Po
using a sparse 3D version of ResNet34 with an FPN neck achieves the best results. Therefore, we ResNet34 + FPN as our default encoder.
PointNet Tran.
73.6
60.1
ResNet34 + FPN
76.7
65.0
More ablation experiments. We provide more ablation studies the effects of using different shapes for the pre-defined 3DV-RPE table, one-to-many matching, the number of points, and other factors in the Appendix A
CONCLUSION
In this work, we have shown how make DETR-based approaches competitive for indoor 3D object detection tasks. The key contribution an effective 3D vertex relative position encoding (3DV-RPE) scheme that can model the accurate position information in the irregular sparse 3D point cloud directly. We demonstrate the advantages of our approach by achieving strong results on two challenging 3D detection benchmarks. We also plan extend our approach outdoor 3D) object detection tasks, which differ from most existing methods that rely on modern 2D DETR- based detectors by converting 3D points 2D bird-eye-view plane. We hope our approach can show the potential for unifying the object detection architecture design for indoor and outdoor 3D detection tasks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENT
We thank all the anonymous reviewers and Area Chairs for their constructive and helpful comments, which have significantly improved the quality of the paper. The work was partly supported by the National Natural Science Foundation of China (Grant No. 62088102).
REFERENCES
Amir Bar, Xin Wang, Vadim Kantorov, Colorado Reed, Roei Herzig, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Detreg: Unsupervised pretraining with region priors for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14605-14615, 2022.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, pp. 213-229, 2020.
Jintai Chen, Biwen Lei, Qingyu Song, Haochao Ying, Danny Z. Chen, and Jian Wu. A hierarchical graph network for 3d object detection point clouds. In CVPR, pp. 389-398, 2020.
Qiang Chen, Xiaokang Chen, Gang Zeng, and Jingdong Wang. Group detr: Fast training conver- gence with decoupled one-to-many label assignment. arXiv preprint arXiv: 13085, 2022.
Bowen Cheng, Lu Sheng, Shaoshuai Shi, Ming Yang, and Dong Xu. Back-tracing representative points for voting-based 3d object detection in point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In CVPR, pp. 5828- 5839, 2017.
Xiyang Dai, Yinpeng Chen, Jianwei Yang. Pengchuan Zhang, Lu Yuan, and Lei Zhang. Dynamic detr: End-to-end object detection with dynamic attention. In ICCV, pp. 2988-2997, 2021.
Lunhao Duan, Shanshan Zhao, Nan Xue, Mingming Gong, Gui-Song Xia, and Dacheng Tao. Condaformer: Disassembled transformer with local structure enhancement for 3d point cloud understanding. In Thirly-seventh Conference on Neural Information Processing Systems, 2023.
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias Nießner. 3d-mpa: Multi-proposal aggregation for 3d semantic instance segmentation. In CVPR, pp. 9028-9037, 2020.
Guofan Fan, Zekun Qi, Wenkai Shi, and Kaisheng Ma. Point-gec: Universal self-supervised 3d scene pre-training via geometry-color contrast. arXiv preprint arXiv:2305.19623, 2023a.
Lue Fan, Yuxue Yang, Feng Wang, Naiyan Wang, and Zhaoxiang Zhang. Super sparse 3d object detection. arXiv preprini arXiv:2301.022 002562,,, 2023b.
Peng Gao, Minghang Zheng, Xiaogang Wang, Jifeng Dai, and Hongsheng Li. Fast convergence of detr with spatially modulated co-attention. In ICCV, pp. 3621-3630, 2021.
Ross Girshick. Fast r-cnn. In ICCV, pp. 1440-1448, 2015.
Jun Gwak, Christopher Choy, and Silvio Savarese. Generative sparse detection networks for 3d single-shot object detection. In ECCV, pp. 297-313. Springer, 2020.
Ding Jia, Yuhui Yuan, Haodi He, Xiaopei Wu, Haojun Yu, Weihong Lin, Lei Sun, Chao Zhang, and Han Hu. Detrs with hybrid matching. arXiv preprint 2tt.::22277..88880 2022.
Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang, Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for 3d point cloud segmentation. In arXiv preprint arXiv:2203. 14508, 2022.
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In CVPR, pp. 12697-12705, 2019.
10

---

Published as a conference paper at ICLR 2024
Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr: Accelerate detr training by introducing query denoising. arXiv preprint arXiv:2203.01305. 2022.
Zechuan Li, Hongshan Yu, Zhengeng Yang, Tongjia Chen, and Naveed Akhtar. Ashapeformer:
Semanties- guided object-level active shape encoding for 3d object detection via transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1012-1021, 2023.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, pp. 2980-2988, 2017.
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes are better queries for detr. arXiv preprint arXiv:2201. 123229 2022.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng- Yang Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In ECCV, pp. 21-37. Springer, 2016.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In CVPR, pp. 10012- 10022, 2021a.
Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via trans- formers. In CVPR, pp. 1949-2958, 2021b.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Confer- ence on Learning Representations, 2019.
Depu Meng, Xiaokang Chen, Zejia Fan, Gang Zeng, Houqiang Li, Yuhui Yuan, Lei Sun, and Jing- dong Wang. Conditional detr for fast training convergence. In ICCV, 2021.
Ishan Misra, Rohit Girdhar, and Armand Joulin. An End-to-End Transformer Model for 3D Object Detection. ICCV, 2021.
Alexander Neubeck and Lue Van Gool. Efficient non-maximum suppression. In ICPR, volume 3, pp. 850-855, 2006.
Charles R Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection point clouds. ICCV, pp. 9276-9285, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Feaf3d: fully convolutional anchor-free 3d object detection. In ECCV, pp. 477-493. Springer, 2022.
Danila Rukhovich, Anna Vorontsova, and Anton Konushin. TR3D: towards real-time indoor 3d object detection. CoRR, abs/2302.02858, 2023.
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In CVPR, pp. 770-779, 2019.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgbbd scene understanding benchmark suite. In CVPR, pp. 567-576, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS, pp. 5998-6008 2017.
Haiyang Wang, Lihe Ding, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, and Liwei Wang. Cagroup3d: Class-aware grouping for 3d object detection on point clouds, arXiv preprint arX1v:2210.04264, 2022a.
11

---

Published as a conference paper at ICLR 2024
Haiyang Wang, Shaoshuai Shi, Ze Yang, Rongyao Fang, Qi Qian, Hongsheng Li, Bernt Schiele, and Liwei Wang. Rbgnet: Ray-based grouping for 3d object detection. CVPR, pp. 1100-1109, 2022b.
Peng-Shuai Wang. Octformer: Octree-based transformers for 3d point clouds.
arXiv preprint
arXiv:2305.03045, 2023.
Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: transformer-basee detector, 2021.
Query design for
Zhenyu Wang, Yali Li, Xi Chen, Hengshuang Zhao, and Shengjin Wang. Uni3detr: Unified 3d detection transformer. arXiv preprint arXiv:2310.05699, 2023.
Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Yiming Zhang, Kai Xu, and Jun Wang. Mlevnet: Multi-level context votenet for 3d object detection. In CVPR, pp. 10447-10456, 2020.
Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Dening Lu, Minggiang Wei, and Jun Wang. Venet: Voting enhancement network for 3d object detection. ICCV, pp. 3692-3701, 2021.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
Yu-Qi Yang, Yu-Xiao Guo, Jian- Yu Xiong, Yang Liu, Hao Pan, Peng-Shuai Wang, Xin Tong, and Baining Guo. Swin3d: A pretrained transformer backbone for 3d indoor scene understanding. arXiv preprint arXiv:2304.06906, 2023.
Zetong Yang, Jiang, Yanan Sun, Bernt Schiele, and Jiaya Jia. unified query-based paradigm for point cloud understanding. In CVPR, pp. 8541-8551, 2022.
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, pp. 11784-11793, 2021.
Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung- Yeung Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv 2022.
preprint arXiv:2203.0300
Zaiwei Zhang. Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. ECCV, pp. 311-32,, 2020.
Yu Zheng, Yueqi Duan, Jiwen Lu, Jie Zhou, and Qi Tian. Hyperdet3d: Learning scene-conditioned 3d object detector. In CVPR, pp- 5575-5584, 2022.
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In CVPR, pp. 4490-4499, 2018.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint 15X1v22010.04159, 2020.
12

---

Published as a conference paper at ICLR 2024
A
MORE ABLATION EXPERIMENTS AND ANALYSIS
Table 10: Effect of light-weight PFN.
Table 11: Effect of using more points.
# of points
AP25
APsa
Light weight FFN
AP2s
APso
20K
73.9
61.6
76.6
62.8
40K
75.2
62.4
76.7
100K
76.7
65.0
Table 12: Effect the one-to-many matching.
Table 13: Effect the pre-defined 3DV-RPE table shape.
points
Rquery
#repeat number
APas
AP
256
74.3
62.0
3DV-RPE table shape
AP2s
APso
40K
512
75.6
62.9
5 5
76.7
61.7
1024
62.4
10 x 10 10
76.7
65.0
100K
512
76.4
64.2
1024
76.7
65.0
50 50 30
76.7
64.3
Table 15: Inference cost comparison. We evaluate all numbers on Tesla V100 PCIe 16 GB GPU with batch size as for a fair comparison.
Table 14: Effect of voxel expansion.
method
voxel expansion
AP2s
AP
67.5
52.4
method
GPU Memory Scenew/second I Latency/scene GPU
FCAF3D
70.5
54.8
FCAF3D
7.8
76.7
65.0
CAGroup3D
2.1
480ms
1138M
Ours
Ours (light)
7.7
130ms
480M
Ours
4.2
240ms
642M
Light-weight FFN. Table 10 reports the comparison results on the effect of proposed light FFN. Ac- cording to the results, we observe that using the light-weight FFN significantly boosts the AP30 from 62.8 to 65.0, thus showing the advantages of using set of adaptive predicted initial 3D bounding boxes over a set of pre-defined 3D bounding boxes of the same size.
Number of points during training and evaluation. In Table we report the comparison results when using different number of points during training. We observe that using 100K points achieves consistently better performance, thus choose 100K points.
One-to-many matching. Table shows the comparison results when choosing different hyper- parameters for one-to-many matching scheme. For example, we find increasing the number of queries and the number ground truth repeating times even hurts the performance when training with 40K points but improves the performance when training with 100K.
Table shape. Table we show the effect of different shapes for the pre-defined 3DV-RPE table. We find that 10 10x x 10 achieves the best results. Our approach is less sensitive to the shape of the 3DV-RPE table thanks to the signed log functioon which improves the interpolation quality to some degree.
Voxel expansion. Table 14 evaluates the effect of using voxel expansion in the FPN neck when the encoder is ResNet34 FPN. We also compare our results with the recent FCAF3D method. The results show that (i) voxel expansion crucial for FCAF3D, which relies on building virtual center features; and (ii) voxel expansion degrades the performance when using DETR, which might lose the original accurate 3D surface information. Therefore, we demonstrate an important advantage of using DETR-based approaches, i.e., they do not require complicated voxel expansion operations.
Inference complexity comparison. Table 15 reports the comparison results to FCAF3D and CA- Group3D. We do not apply the test-time augmentation (TTA) ensure fair comparisons. Accord- ingly, our method achieves a better serformance-efffciieee vide light version by decreasing the number of 3D object query from 1024 to 256, which achieves
trade-off than CAGroup3D. We also pro-
13

---

Published as a conference paper at ICLR 2024
25. and AP 5 Notably, the reported latency of CAGroup3D close the numbers in their official logs but different from the numbers reported in the paper (179.3ms tested on RTX 3090 GPU). The authors of CAGroup3D have acknowledged this issue in their GitHub repository.
B MORE DETAILED SYSTEM-LEVEL COMPARISON
Table 16: Full System-level comparison with the state-of-the-art on ScanNerV2 and SUN RGB-D. TTA: test- time augmentation. denotes using additional sellf pre-training. denotes using extra training data.
ScanNerV2
SUN RGB-D
Method
APaz
AP50
APaz
AP50
VoseNet
58.6
HGNet
Chon ct al.
2020
61.3
34.4
61.6
3D-MPA
Engelmann
64.2
MLCVN
Xic
2020
64.5
41.4
59.8
GSDN
Gwak
2020
62.8
34.8
H3DNNe
Zhanp et al
2020
67.2
48.1
60.1
39.0
BRNet
al.
2021
66.1
50.9
61.1
43.7
65.0
VENes
Xie et al.
2021
67.7
62.5
30.2
Group-Free Free eL
2021b
RBGNet Wang al.
20226
70.6
65.2
61.1
17.2
HypaDDadD Zhene
2022
70.9
AStapeFonner Li al. 2023
71.1
66.6
62.2
FCAF3D Rukhovich
2022
71.5
57.3
64.2
48.9
Octoorerr Wang 2023
66.2
50.6
UnIDDETR Wang al.
2023
71.7
58.3
67.0
50.3
ConDeFoomer Duan et
67.1
49.9
TR3D Rukhovich
2023
72.9
68.3
67.1
00.4
Point-GOC* Fan al 20230
73.1
CAGroup3D Wang et al
2022a
75.1
61.3
66.8
50.2
CAGroup3D*
Yang
V-DETR
77.4
65.0
67.5
50.4
V-DETR
77.8
66.0
Average Results under x5 triels
Group-Free Liu ct a1
2021by
68.6
14.4
RBGNet
Wang al
2022b
69.9
54.7
63.6
46.3
Rukhovich
2022
TR3D
Rukhovich et al
1023
72.0
57.4
66.3
49.6
CAGroup3D
Wang
2022a
74.5
ConDaFormer
Duan et al.
2023
66.8
49.5
V-DETR
V-DETR (TTA)
77.0
65.3
67.5
50.0
Table 16 shows a more detailed comparison of our method with the state-of-the-art methods from previous works at the system level.
C MORE ANALYSIS OF 3DV-RPE ADVANTAGES
Comparison with other attention modulation methods. We summarize the comparison results with other advanced related methods including contextual relative position encoding (CRPE) (Lai et al. 2022 Yang et al. 2022). conditional cross-attention (Cond-CA) (Meng et al. 2021). dynamic anchor box cross atteetiooo (DAB-CA) (Liu et al. 2022) in Table 17 We report the comparison results under the most strong settings, i.c., 540 training epochs. Accordingly, we see that (i) both CRPE (Stratified Transformer (Lai et al. 2022 and CRPE (EQNet (Yang et al. 2022 consistently
14

---

Published as a conference paper at ICLR 2024
improve the baseline; (ii) our 3DV-RPE achieves the best performance. The reason is that the CRPE methods of center point of the 3D box, while our 3DVRRPE explicitly considers the 8x vertex points and rotated angle the 3D box. Our method encodes the box size and the six faces, thus modeling the accurate
Lai et al. 2022) and EQNet (Yang et al. 2022) only consider the
position relations between all other points and the 3D bounding box (supported by the much larger gains on 50)
Table 17: Comparison to other atlention modulation methods. We only change the decoder cross-altention scheme and keep all other settings same comparison fairness.
method
#epochs
AP25
AlPo
Baseline (w/o RPE)
5-10
71.4
47.6
Baseline + CRPE (Stratrified Transformer)
540
74.7
58.1
Baseline CRPE (FQNet)
540
73.1
54.4
Baseline + Cond-CA
540
74.7
55.8
Baseline DAB-CA
540
75.4
56.0
Baseline + 3DV-RPE
5410
77.8
66.0
Comparison with absolute position encoding methods on ScanNetV2. We summarize the com- parison results with different absolute position encoding(APE) methods, including the sin-cos ab- solute position encoding (APE w/ Sin-Cos) and the absolute position encoding using multilayer perceptron (APE w/ MLP). To minimize the gap between APE and 3DV-RPE for fair compression, we apply non-linear transformation and predefined table to APE, mirroring their usage in 3DV-RPE. depicted Table T8 our 3DV-RPE significantly outperforms all the APE variants by a large margin, especially AP50 which demands greater precision in localization.
Table 18: Comparison to absolute position encoding(APP) methods on Sca "NonLinear" refers to application of non-linear transformation, and "Predefined Table" denotes employing table and grid sampling techniques to enhance efficiency, which are both applied to 3DV-RPE.
method
#spochs
AP25
AP 50
Baseline (w/o PF)
540
71.4
47.6
Baseline + APE w/ Sin-Cos
540
71.8
47.9
Baseline APE MLP NonLinear
540
72.1
48.7
Baseline APE w/ MLP NonLinear Predefined Table
540
72.0
48.5
Baseline 3DV-RPE
540
77.8
66.0
D DETAILS ABOUT 3DV-RPE EFFICIENT IMPLEMENTATION
The transformation matrix T plays pivotal role in our proposed method, facilitating the efficient encoding of spatial relations. The matrix is initialized to encompass Nuble X Niable X Nuable grid, i.e. Nuble 10, where each grid point (i.j. k) holds diree-dimensional vector. The vector values are calculated as follows:
Tiijj, (M (Nuble 1)) M(2j - (Naable 1)) M(2k (Naabe - 1)) 11)
(6)
Nuable
Nuble -1
Nuable -
where [-M, .M] is the range of values, which avoids F(AP,) Equation 5 beyond the bounds.
15

---

Published as a conference paper at ICLR 2024
E QUALITATIVE RESULTS AND ANALYSIS
We show some qualitative examples of our V-DETR detection SUN RGB-D and ScanNetV2 in Figure ? and Figure respectively. We can observe that our method can find most of the target objects various scenes.
Figure shows the spatial cross-attention maps of our 3DV-RPE on three ScanNetV2 seenes. We that (i) our 3DV-RPE can find the 3D bounding boxes accurately and (ii) each vertex's RPE can enhance the regions inside the boxes from that vertex.
Figure 7: More qualitative results of 3D object detection on SUN RGB-D. The ground truth is shown in the first column and our method's detection results are shown in the second column.
16

---

Published as a conference paper at ICLR 2024
Figure 8: More qualitative results 3D object detection on ScanNetV2. The ground-truth shown the first column and our method's detection results are shown in the second column.
17

---

Published as a conference paper at ICLR 2024
Figure 9: Illustration of the spatial attention maps learned by our 3DV-RPE on ScanNetV2 scenes. Each scene consists of two rows. We draw green cube to mark the detected 3D bounding box and red star at its eight vertices. We average the head dimension of each P; and show the spatial cross-attention maps for eight vertices (columns 2-5). Column shows the input scene the attention values: yellow high and blue is low. We see that (i) each vertex's 's attention map highlights the regions inside the cube from that vertex, and (ii) the combined attention maps focus on the regions inside the red cubes.
the merged attention maps. The color shows
18

---

Published as a conference paper at ICLR 2024
F EFFECT OF DATA SCALE ON LEARNING THE LOCALITY INDUCTIVE BIAS FOR 2D OBJECT DETECTION
We conduct experiments the 2D detector DETR Carion al. 2020), using approximately 1% of the training data (1, 10200 images), following your suggestion. We train DETR for the same number of iterations (2, 217, 881 iterations) as the original DETR, which was trained on the full dataset for 300 epochs, while maintaining a batch size of 16. First, we observe that the mAP of the validation set drops from 44.9% to 10.4%, which closely aligns with the performance reported in Table 4 of DETReg |arr et al. (2022).
Second, in accordance with Figure of the DETR paper, we visualize the cross-attention maps for the predicted object in Figure T0 We note that these attention maps fail to focus on local object regions, especially the object extremities. Therefore, it is evident that the scale of data has substantial impact on the model's 's ability to effectively learn the locality inductive biases.
(a) full-set
(b) 1% sub-set
(c) full-set
(d) 1% sub-set
Figure 10: Illustrating the effect of training data scale on the 2D cross-are coss-attention maps with DETR. When trained using just 1% subset, it is observable that these attention maps struggle concentrate local object areas, particularly the object extremities.
19


Reviews:
[{"review_id": "vnOtCGrbh9", "reviewer": "Reviewer_YhUd", "summary": "This paper presents one improvement over DETR-based methods for 3D object detection. The idea is first to predict a coarse bounding box and then only allow attention weights to be learned within the bounding box for better-using locality as one important inductive bias for 3D object detection. Experiments are done on the ScanNetV2 and Sun RGB-D benchmarks. Results show improved performance over baselines. Extensive ablation studies are also presented.", "questions": "see weakness", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- the proposed method improves over state-of-the-art methods on ScanNetV2 and Sun RGB-D.\n- the proposed modification to the DETR-based method is valid and reasonable.\n- ablation studies are solid.", "weaknesses": "- the proposed method is more like just a small fix to DETR-based method.\n- the proposed fix is also specific to DETR-based backbones.\n- the proposed fix may be vulnerable if the first stage predicting the coarse bounding boxes fail. For example, if the bounding boxes are very off, then preventing later layers to attend to out-of-the-box regions may make it impossible to recover. \n- the paper writing can be improved. For example, the figure layouts are quite messy. The organization for Sec. 3.1 and 3.2 is a bit hard to follow. It looks like Sec. 3.1 focuses on laying out the basic pipeline of DETR and Sec. 3.2 discusses more into the contributions of the paper, but actually the content are mixed together.\n-  there are also claims that are unsupported in the paper. For example, the sentence in the introduction section \"We attribute the discrepancy to the limited scale of training data available for 3D object detection\" is not well supported. Can you use less data to train 2D detectors to show it's really the data scale issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kecfdT7r8R", "reviewer": "Reviewer_c9AZ", "summary": "The manuscript proposes a 3d object-detection-specific position encoding method that significantly improves performance for 3d DETR-like object detection.\nThe 3d position encoding encodes the relative position of key, value points to the object query points to allow the transformer to learn to attend to points inside the object bounding box more easily.\nWith the addition of the position encoding and some other tweaks, the proposed method, V-DETR, outperforms CNN-based methods. A first for transformer-based detectors in 3d.", "questions": "Page 5 has a broken figure reference.\n\nI dont understand how T in Eq(3) is initialized/set? What range do the T values take? -5 to 5 as indicated by the signed-log function?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "The proposed approach for relative position encoding is intuitive (and illustrated well in Fig 1), and experiments clearly show that it leads to a big improvement for Transformer-based methods and leads to a new state of the art wrt. to CNN-based methods as well. \n\nOverall the quality of writing and illustration is very high. The detailed pipeline visualization clearly shows the recurrent nature of the approach. The visualization of the attention for each of the corners is also very illustrating. \n\nThe manuscript pays attention to practical aspects as well: The use of a precomputed lookup table for the relative PE is a nice and practical way to safe valuable GPU memory.\n\nThe experiments are expansive and convincing. The ablations do help clarify the different choices of the hyperparameters.", "weaknesses": "The precomputed lookup table was the hardest to follow (Eq 3) since the connection to Eq 4 was not immediately obvious. One more sentence there to explicitly connect the two would be helpful. I.e. T represents a discretized set of possible \\Delta P that we interpolate into.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7gC9krwExR", "reviewer": "Reviewer_TFsP", "summary": "For the task of indoor 3D object detecion from point clouds, previous sparse detectors neglects to tackle 3D queries outside the bounding box, so the author proposed a 3D vertex positional encoding module (3DV-RPE) to guarantee the locality principle in object detection. Specifically, the proposed method encode relative position information for each query towards its assigned / predicted bounding boxes to provide clear information to guide the model to focus on points near the objects. Moreover, it utilized many widely-adopted tricks to generally improve the performance of the detector (custom backbone / loss normalization / TTA / one-to-many auxillary loss, etc). 3DV-RPE shows competitive results on ScanNetv2 and SUN-RGBD compared to previous methods.", "questions": "* How much fraction does the queries outside predicted bounding boxes account for with respect to the total number of queries? 10%? 20%? The author should provide a investigation to this problem.\n* Why the non-linear functions is designed in this way? how it is derived? is their any insights to do so?\n* Why does the PE is added in the way in Eq. 1? I think the form of matmul(Q, K) + R does not match the intended aim of the paper. Instead, I think matmul(Q+R, K) should be more proper? or add a relative PE to Q and a global PE to K?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* this paper pinpoint a interesting gap in previous sparse 3D indoor detectors that how to deal with queries outside the predicted bouding box, and try to prove that whether it is benefical to take this into account for 3D object detection.\n* this paper does a lot of work to incorporate modern network architectures (e.g., ResNet34 + FPN), training strategy improvements to make the detector performs better.", "weaknesses": "* Unfair comparision:\n    1) The author combines many technical improvements including normalizing box according to object size, one-to-many assignment as auxillary loss, a modified resnet34-fpn backbone, and even TTA. They're all irrelavant to the claimed core contribution 3DV-RPE. So to prove the proposed module is effective, the most convincing way could be adding 3DV-RPE directly onto the baseline GroupFree3D.  Considering that the 3DV-RPE  module can work in a plug-and-play manner in theory, I would expect more results based on other methods such as GroupFree3D, 3DETR or CAGroupFree3D.\n    2) in table 6, the author reports the one with 3DV-RPE + TTA (77.8 / 66.0), does all other ablation attention results are also reported with TTA?\n    3) the author reported the best results for the proposed method, how about the ablations? how many times have you run for each ablation choice in all tables? Does the fairness of the comparison is guaranteed?\n    4) Given the best set of paraters for the authors final model, change the choices in loss functions can affect the hungarian matching cost matrix, thus it's hard to say the improvement / performance drop comes from the module / improper cost weights.\n    5) Why some ablations are done with ScanNet while others use SUN-RGBD (e.g., Table 4)? Does it means the coordinates normalization works similarly on ScanNetv2?\n\n* minor contributions:\n    1) Actually I think the RPE and normalized coords are designed in similar ways: Point-RCNN has adopted to convert box to  canonical coords and do normalizations on oritentaion. Moreover, in anchor-based detectors, they already use the anchor boxes' W and H to normalize the regression targets. Here the author uses dynamic bounding boxes from predictions, which has also been explored in methods like MetaAnchor, etc.\n    2) So many un-relavent tricks to improve the detection performance. I don't like the way to do whatever it can to improve the results. Rather, the author should focus on the main contribution. After the core module being sufficiently discussed, one can further improve its results with more tricks. Here the author put all stuff together, which makes me doubt where the improvement come from.\n\n* Most of the references are before year 2023, so I think more recent works in year 2023 should be included.\n\n* I recommend against reporting results using TTA, as this leads to cutthroat competition and more potentially unfair comparisons; for example, TTA may be different in different papers, but is always written as \"TTA\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oyfZW4ziaw", "reviewer": "Reviewer_fSn3", "summary": "This paper introduces an effective enhancement to DETR-based indoor 3D object detection. The key idea is to add a relative positional embedding between the queries and points. The positional embedding, which is called 3D Vertex Relative Position Encoding (3DV-RPE), calculates the relative positional embedding under the coordinate system of each 3D bounding box generated from the query. After incorporating the positional embedding, the performance significantly increases on both ScanNetv2 and Sun-RGBD.", "questions": "1. **Baseline performance.** As mentioned in the weakness section, could you remind me where you have put the baseline performance? It is critical to recognize the improvement of 3DV-RPE. Technically, I wish to see that under the same normalization and other tricks, 3DV-RPE is indeed helpful.\n\n\n2. **Additional analysis.** With position encoding being the center of this paper, I think it necessary to conduct ablation studies on other common formats of position encoding, such as:\n*  Absolute position encoding, in both the formats you proposed like Eqn. 3 and Eqn. 4, or common sin-cos position encoding.\n* More justifications of hyper-parameters. For example, where does 10 come from in $T$'s shape? May I use another number to replace 10?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper has demonstrated the following strengths:\n\n* The approach of the paper has good support from their performance improvement on ScanNetv2 and Sun-RGBD.\n* The vertex relative position encoding (3DV-RPE) has the reasonable intuition of embedding position information for 3D detection and will likely inspire other readers.\n* The qualitative results like Figure 1 clearly illustrate the implication of the method in the paper in guiding models' attention.", "weaknesses": "* I suggest improving the order of presentation in Sec. 3.2 and Sec. 3.2. For example, I suggest moving the paragraph of \"3DV-RPE\" before talking about \"canonical object space\" and other details. When I read this part, I was quite confused by Sec. 3.2, not knowing how $R$ is generated, what is $P_i$, etc.\n\n* As position encoding is the focus of this paper, I expect the authors to analyze or conduct ablation studies on more position encoding algorithms. Details are in the \"questions\" section below. \n\n* I also haven't found the performance of the baseline without relative position encoding. In case I missed it, I suggest the authors put it into Sec. 4.3 or Sec. 4.4 for a clear ablation study.\n\nTypo on page 5, line 1: fig:rotatedRPE", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)