DeepInteraction: 3D Object Detection via Modality Interaction
Authors: Zeyu Yang
Jiaqi Chen
Zhenwei Miao
Wei Li
Xiatian Zhu
Li Zhang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Existing top-performance 3D object detectors typically rely on the multi-modal fusion strategy. This design is however fundamentally restricted due to overlooking the modality-specific useful information and finally hampering the model performance. To address this limitation, in this work we introduce a novel modality interaction strategy where individual per-modality representations are learned and maintained throughout for enabling their unique characteristics to be exploited during object detection. To realize this proposed strategy, we design a DeepInteraction architecture characterized by a multi-modal representational interaction encoder and a multi-modal predictive interaction decoder. Experiments on the large-scale nuScenes dataset show that our proposed method surpasses all prior arts often by a large margin. Crucially, our method is ranked at the first position at the highly competitive nuScenes object detection leaderboard.

Deeplnteraction: 3D Object Detection via Modality Interaction
Zeyu Yang'
Jiaqi Chen'
Zhenwei Miao?
Wei Li
Xiatian Zhu
Li Zhang'
Fudan University AAlibaba DAMO Academy
S-Lb NTU *University Surrey
Abstract
Existing cop-eerformance 3D object detectors typically rely on the multi-modal fusion strategy. This design is however fundamentally restricted due to overlooking the modality-speciff useful information and finally hampering the model perfor- mance. To address this limitation, in this work we introduce novel modality interaction strategy where individual per-modality representations are learned and maintained throughout for enabling their unique characteristics to be ex- ploited during object detection. To realize this proposed strategy, we design a DeepInteraction architecture characterized by multi-modal representational interaction encoder and multi-modal predictive interaction decoder. Experiments the large-scale nuScenes dataset show that our proposed method surpasses all prior arts often by a large margin. Crucially, our method ranked at the f first position the highly competitive nuScenes object detection leaderboard.
Introduction
3D object detection is critical for autonomous driving by localizing and recognizing decision- sensitive objects in 3D world. For reliable object detection, LiDAR and camera sensors have been simultaneously deployed to provide point clouds and RGB images for more stronger perception. The two modalities exhibit naturally strong complementary effects due to their different perceiving characteristics. Point clouds offer necessary localization and geometry information at low resolution, whilst images give rich appearance information at high resolution. Therefore, information fusion across modalities becomes particularly crucial for strong 3D object detection performance.
Existing multi-modal 3D objection detection methods typically adopt modality fusion strategy (Figure I(a)) by combining individual per-modality representations into a single hybrid representation. For example, PointPainting [38] and its variants [39, 46, 43] aggregate category scores or semantic features from the image space into the 3D point cloud space. AutoAlign [10] and VFF [23] similarly
integrate image representations into the 3D grid space. Latest alternatives [24, 30, 26] merge the image and point cloud features into a joint hird's-eye view (BEV) representation. This fusion approach is, however, structurally restricted due its intrinsic limitation of potentially dropping off large fraction of moodality-specife representational strengths due to largely imperfect information fusion into a unified representation.
To overcome the aforementioned limitations, in this work a novel modality interaction strategy (Figure 1(b) for multi-modal 3D object detection introduced. Our key idea that, instead of deriving a fused single representation, we learn and maintain two modality-specific representations throughout to enable moter-modality interaction so that both information exchange and modality- specific strengths can be achieved spontaneously. Our strategy is implemented by formulating
Li Zhang (lizhangfdel @ fudan. edu..an.dd is the corresponding author with School of Data Science, Fudan University.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Encoder
Eaceder
Fusion block
Detection head
LiDAR BEV
LIDAR DEV
fessure
Decoder
Reature
Decoder
box closs
box & class
Image feoture
Image
Scature
(a) Multi-modality fusion
(b) Multi-modality interaction
Figure I: Schematic strategy comparison. (a) Existing multi-modality fusion based 3D detection: Fusing individual per-modality representations into a single hybrid representation and from which the detection results are further decoded. (b) Our multi- modality interaction based 3D detection: Maintaining two modality-specifn representations throughout the whole pipeline with both represen- tational interaction in the encoder and predictive interaction in the decoder.
DeepInteraction architecture. It starts by mapping 3D point clouds and 2D multi-view images into LiDAR BEV feature and image perspective feature with two separate feature backbones in parallel. Subsequently, encoder interacts the two features for progressive information exchange and representation learning in a bilateral manner. To fully exploit per-modality representations, a decoder/head is further designed to conduct multi-modal predictive interaction in a cascaded manner.
Our contributions are summarized as follows: (i) We propose a novel modality interaction strategy for multi-modal 3D object detection, with the aim to resolve fundamental limitation of previous modality fusion strategy in dropping the unique perception strengths per modality. (ii) To implement our proposed strategy. we formulate a DeepInteraction architecture with a multi-moda repre- sentational interaction encoder and a multi-modal predictive interaction decoder. (iii) Extensive experiments on the nuScenes dataset show that our Deep Ineeptttattion yields new state of the art for multi-modality 3D object detection and achieves the first position at the highly competitive nuScenes leaderboard.
2 Related work
3D object detection with single modality Automated driving vehicles are generally equipped with both LiDAR and multiple surround-view cameras. But many previous methods perform 3D object detection by exploiting data captured from only a single form of sensor. For camera- based 3D object detection, since depth information is not directly accessible from RGB images, some previous works [17, 40, 37] lift 2D features into a 3D space by conducting depth estimation, followed by performing object detection in the 3D space. Another line of works [41, 28, 25, 31, 19] resort the detection Transformer [5] architecture. They leverage 3D object queries and 3D-2D correspondence to incorporate 3D computation into the detection pipelines. Despite the rapid progress of camera- based approaches, the state-of-the-art of 3D object detection is still dominated by ADDAR-based methods. Most of LiDAR-base -baeed detectors quantify point clouds into regular grid structures such as voxels [47, 44], pillars [22] or range images [2, 14, 6] before processing them. Due to the sampling characteristics of LiDAR, these grids are naturally sparse and hence fit the Transformer design. So number of approaches [32, 13] have applied the Transformer for point cloud feature extraction. Differently, several methods use the Transformer decoder or its variants as their detection head [1, 42]. 3DETR [33] adopts complete Transformer encoder architecture with less priori in design. Due to intrinsic limitations with either sensor, these methods are largely limited in performance.
Multi-modality fusion for 3D object detection Leveraging the perception data from both camera and LiDAR sensors usually leads to improved performance. This has emerged as a promising direction. Existing 3D detection methods typically perform multi-modal fusion at one of the three stages: raw input, intermediate feature, and object proposal. For example, PointPainting [38] is the pioneering input fusion method [39, 18, 46). The main idea is to decorate the 3D point clouds with the category scores or semantic features from the 2D instance segmentation network. Whilst 4D-Net [35] placed the fusion module in the point cloud feature extractor for allowing the point cloud features to dynamically attend to the image features. Im VoteNet [36] injects visual information into a set of 3D seed points abstracted from raw point clouds. The proposal based fusion methods [21, 7]

---

Coond 6..1
coand.
Meue
sottler
By
aottle
Saspling
Sumpliy
(a) Representational interaction from image LiDAR
(b) Representational interaction from LiDAR image
Figure 2: Illustration of the multi-modal representational interactions. Given two modality-speciffe representations, the image-to Srage-to-LiDee to-LiDAR feature interaction (a) spread the visual signal in the image representation to the LiDAR BEV representation, and the LiDAR-too R-to-mage feature interaction (b) takes mossl relative contexts from LiDAR representation enhance the image representations.
keep the feature extraction of two modalities independently and aggregate multi-modal features via proposals or queries at the detection head. The first two categories methods take a unilateral fusion strategy with bias 3D LiDAR modality due the superiority of point clouds distance and spatial perception. Instead, the last category fully ignores the intrinsic association between the two modalities in representation. As result, all above previous methods fail to fully exploit both modalities, in particular their strong complementary nature. Besides, couple of concurrent works have explored fusion of the two modalities in shared representation space [30, 26]. They conduct view transformation in the same way [34] as in the camera-only approach. This design however less effective exploiting the spatial cues of point clouds during view transformation, potentially compromising the quality of camera BEV representation. This gives rise to an extra need of calibrating such misalignment in network capacity.
In this work, we address the aforementioned limitations in all previous solutions with a novel multi-modal interaction strategy. The key insight behind our approach is that we maintain two modality-specifff feature representations and conduct representational and predictive interaction for maximally exploring their complementary benefits whilst preserving their respective strengths.
3
Method
We present novel modality interaction framework, dubbed DeepInteraction, for multi-modal (3D point clouds and 2D multi-camera images) 3D object detection. In contrast to all prior arts, learn two representations specific for 3D LiDAR and 2D image modalities respectively, whilst conducting multi-modal interaction through both model encoding and decoding. An overview of DeepInteraction shown in Figure 1(b). It consists of main components: An encoder with multi-modal representational interaction (Section 3.1), and a decoder with multi-modal predictive interaction (Section 3.2).
3.1 Encoder: Multi-modal representational interaction
Unlike conventional modality fusion strategy that often aggregates multi-moda inputs into hybrid feature map. individual per-modality representations are learned and maintained via multi-modal representational interaction within our encoder. Specifically, our encoder is formulated as multi- input-mulii-utppe (MIMO) structure: Taking as input two modality-speciff scene representations which are independently extracted by LiDAR and image backbones, and producing two refined representations as output. Overall, composed by stacking multiple encoder layers each with (I) multi-modal representational interaction (MMRI), (II) intra-modal representational learning (IML), and (III) representational integration.
(I) Multi-modal representational interaction (MMRI) Each encoder layer takes the representa- tions of two modalities, i.e., the image perspective representation h. and the LiDAR BEV representa- tion hns as input. Our multi-modal representational interaction aims to exchange the neighboring context in bilateral cross-modal manner, shown in Figure 2. It consists of two steps:

---

(i) Cross-modal correspondence mapping and sampling To define cross- modalityy adjacency, we first need build the pixel-to-pixel(s) correspondence between the representations h, and h To that end, we construct dense mappings between the image coordinate frame and the BEV coordinate frame p (Mp-rc and Me-op).
From image to LiDAR BEV coordinate frame Mc-pp : R² - 2 (Figure 2(a)): We first project each point (3r, 2) in 3D point cloud multi-camers images to form sparse depth map deparses followed by depth completion [20] leading to a dense depth map dienne- We further utilize ddense to back-projed each pixel in the image space into the 3D point space. This results in the corresponding
d 3D coordinate (z,, y, 2), given an image pixel (i.j) with depth dense Next, (x, y) is used to locate the corresponding BEV coordinate (ip. Jp)- We denote the above mapping T(i, )) (pp.
We obtain this correspondence via (2k + 1) x (2k + 1) sized neighbor sampling as
(ī + A Ai) Aj e +-k
From LiDAR BEV image coordinate frame Mp IR2 2 (Figure 2(b)): Given coordinate
Jp) in BEV, we first obtain the LiDAR points {(r, y, }}} within the pillar corresponding to (ip:
Then we project these 3D points into camera image coordinate frame {i. j)} according to the camera
intrinsics and extrinsics. This correspondence is obtained as: Mp-el(p:Jpp = {(i,))}.
(ii) Attention feature interaction For an image feature point query q he [ede) its cross- modality neighbors Ng. denoted as Ng- cross-attention learning:
Me spffedee) are used as the key k and value v for hip
føc-r (Pey hp) [6-11
qk Vil
softmax
v,
KUENN
where h6-d denotes indexing the element location (i,j) on the 2D representation h. This is image-to-LiDAR representational interaction.
The other way around, given a LiDAR BEV feature point as query 1 we similarly obtain its cross-modality neighbors as N₂- MoM The same process (Eq. (1)) is applied for realizing LiDAR-to-image representational interaction fop-s (hc, hp).
conduct intra-modal repre- sentational learning complementary to multi-modal interaction. The same local attention as defined in Eq. (1) is consistently applied. For either modality, we use k x k grid neighborhood the key and value. Formally, we denote fo... (he) for image representation and før-p (hp) for LiDAR representation.
(II) Intra-modal representational learning (LML) Concurrently,
(III) Representational integration
Each layer ends up by integrating the two outputs per modality:
h PF
h h))+))) hp)),
h PFN(Conca((FNNNConntth hp-+)),, h.)):
where FFN specifies feed-forward network, and Concat denotes element-wise concatenation.
3.2 Decoder: Multi-modal predictive interaction
Beyond considering the multi-modal interaction at the representation-levdl we further introduce decoder with multi-modal predictive interaction (MMPI) to maximize the complementary effects in prediction. As depicted in Figure 3(a), our core idea is enhance the 3D object detection of one modality conditioned on the other modality. In particular, the decoder is built by stacking multiple multi-moda predictive interaction layers, within which predictive interactions are formulated in an alternative and progressive manner. Similar to the decoder of DETR [5]. we cast the 3D object detection set prediction problem. Here, we define set of N object queries { and the resulting N object predictions {(b,, ( where bn and c, denote the predicted bounding box and category for the n-th prediction.

---

Mul- Ceal-Nootsse ReLU
K
Rol Fests.
RolAlign
+
MMPI- 177--mmnge
MIMP--LiDA
Muli-heal self-untenttie
Adé & Nomm
boyer
Addd Nori
PFN
PFN
Add
EEN
6
(a) Predictive interaction decoder
(b) Multi-modal predictive interaction layer (MMPI)
Figure 3: Illustration of our generates predictions via (b) progressively interacting with two modality sppccifc representations.
predictive interaction. Our predictive interaction decoder (a)
Multi-modal predictive interaction layer (MMPI) For the 1-th decoding layer, the set prediction is N computed by taking the object queries { {Qh-1)) Jn=z and the bounding box predictions { (
==1 from previous layer as inputs and enabling interaction with the intensified image h', or LiDAR h representations (h', if is odd, h2 if I is even). We formulate the nulti-modal predictive interaction
layer (Figure 3(b) for specific modality as follows:
(I) Multi-modal predictive interaction on image representation (MMPI-image) Taking as input {
3D object proposals { (b--1) and object queries
generated by the previous layer, n=1
n=1
this layer leverages the image representation h' for further prediction refinement. To integrate the previous predictions {o! (b!--1)
we first extract N Region of Interest (Rol) [15] features {Rn, x==1 x=1 from the image representation h'.. where R2 E SXSXC is the extracted Rol feature for the n-th query, (S S) is Rol size, and C' is the number of channels of Rol features. Specifically, for each 3D bounding box, we project onto image representation h', to get the 2D convex polygon and take the minimum axis-aligned cireumscribed rectangle. We then design multi-modal predictive
interaction operator that maps {Q% [1) into the parameters of a series of X convolutions
N
n=1 {R,,
and then applies them consecutively
N n=1 The resulted interactive representation is further J/=1' n=1'
used to obtain the updated object query (Q)
(II) Multi-modal predictive interaction on LiDAR representation (MMPI-LiDAR) This layer shares the same design the above except that takes input LiDAR representation instead. With regards to the Rol for LiDAR representation, we project the 3D bounding boxes from previous layer to the LiDAR BEV representation h', and take the minimum exis-aligned rectangle. It is worth mentioning that due to the scale of objects autonomous driving scenarios is usually tiny in the BEV coordinate frame, enlarge the scale of the 3D bounding box by times. The shape of Rol features cropped from the LiDAR BEV representation h is also set to be 5 x S x C. Here is the the number of channels of Rol features as well as the height of BEV representation. The multi-modal predictive interaction layer on LiDAR representation is stacked on the above image counterpart.
For object detection, feed- forward network is appended on the {Q%) N for each multi-modal predictive interaction layer to infer the locations, dimensions, orientations and velocities. During training. the matching cost and loss function as [1] are applied.
4 Experiments
4.1 Experimental setup
Dataset We evaluate our approach the nuScenes dataset [3]. It provides point clouds from 32-beam LiDAR and images with a resolution of 1600 x 900 from surrounding cameras. The total of 1000 scenes, where each sequence is roughly 20 seconds long and annotated every 0.5 second, is
5

---

Table I: Comparison with state-of-the-art methods the nuScenes test Metrics: mAP(%) NDS(%)) (6). 'L' and 'C" represent LiDAR and camera, respectively. f denotes test-time augmentation is
used. $ denotes that test-time augmentation and model ensemble both are applied for testing.
Backbones
validation
Test
Method
Modality
Image
LiDAR
mAPt
NDS1
mAPt
NDS1
BEVDet4D [17]
C
Swin-Base
42.1
54.5
45.1
56.9
BEVFormer [25]
C
V99
48.1
56,9
Ego3RT [31]
C
V99
47.8
53.4
42.5
47.9
PolarFormer [19]
C
50.0
56.2
49.3
57.2
CenterPoint [45]
L
VoxelNet
59.6
60.3
67.3
Focals Conv
L
VoxeINct-FocalsCom
61.2
68.1
63.8
70.0
Transfusion-L_ [1]
L
VoxelNet
65.1
70.1
65.5
70.2
LargeKemel3D [9]
L
LoxclNet-LaggeKemmel3
63.3
69.1
65.3
70.5
FUTR3D [7]
L+C
RI01
VoxelNet
64.5
68.3
PointAugmenting [39]t
L4C
DLA34
VoxelNet
66.8
71.0
MVP [46]
L+C
DLA34
VoxelNet
67.1
70.8
66.4
70.5
AutoAlignV2 [10]
L+C
CSPNet
VoxelNet
67.1
71.2
68.4
72.4
TransFusion [1]
L+C
R50
VoxeINet
67.5
71.3
68.9
71.6
BEVFusion [26]
L+C
Swin-Tiny
VoxelNet
67.9
71.0
69.2
BEVFusion [30]
L+C
Swin-Tiny
VoxeINet
68.5
71.4
70.2
72.9
DeepInteractton-baas
L+C
R50
VoxelNet
69.9
72.6
70.8
73.4
Focals Conv-F [8]
L+C
R50
VoxeINet-
67.1
71.5
70.1
73.6
LargeKernel3--F [9]
L+C
R50
VoxeINet-Large
71.1
74.2
DeepInteraction-large
L+C
Swin-Tiny
VoxelNet
72.6
74.4
74.1
75.5
BEVFusion-e [30]$
L+C
Swin-Tiny
VoxeINet
73.7
74.9
75.0
76.1
DeepInteraction-eS
L+C
Swin-Tiny
VoxelNet
73.9
75.0
75.6
76.3
officially split into train/val/test set with 700/150/150 scenes. For the 3D object detection task, 1.4M objects in scenes annotated with 3D bounding boxes and classified into 10 categories: car, truck, bus, trailer, construction vehicle, pedestrian, motorcycle, bicycle, barrier, and traffic cone.
Metric Mean average precision (mAP) [12] and nuScenes detection score (NDS) [3] are used as the evaluation metric of 3D detection performance. The final mAP is computed by averaging over the distance thresholds of 0.5m, Im, 2m, 4m across 10 classes. NDS is a weighted average of mAP and other attribute metrics, including translation, scale, orientation, velocity, and other box attributes.
4.2 Implementation details
Our implementation is based on the public code base mindetection3d [11]. For the image branch backbone, we use simple ResNet-50 [16] and initialize from the instance segmentation model Cascade Mask R-CNN [4] pretrained on COCO [27] and then nulmage [3], which is same as Transfusion []]. To save the computation cost, we rescale the input image to 1/2 of its original size before feeding into the network, and freeze the weights of image branch during training. The voxel size is set to (0.075m, 0.075m., 0.2m)) and the detection range is set to [-54m, 54m] for X and axis and [-5m, 3m] for axis. The representational interaction encoder is composed by stacking two representational interaction layers. For the multi-modal predictive interaction decoder, we use cascaded decoder layers. We set the query number to 200 for training and testing and use the same query initialization method as Transfusion [1]. The above configuration is termed DeepInteraction-baal
We also adopt another two widely used settings for online submission, i.e., test-time augmentation (TTA) and model ensemble. In the following, we refer to the two set- tings as DeepInteraction-larg DeepInteraction-L
In particular, uses Swin-Tiny [29] as image backbone, and doubles the number of chan- nel for each convolution block in LiDAR backbone. The voxel size of DeepInteraction-lagge
and DeepInteraction-e respectively.

---

RIOH
RORT
BACK-IGH
LIDAR BEV
Construction
Motorcyole
Figure 4: Qualitative results on nuScenes val set. In LiDAR BEV (right), green boxes are the ground-truth and blue boxes are the predictions. Best viewed when zooming in.
set to [0.5m, 0.5m, 0.2m]. Following the common practice, we use double flipping and rotation with yaw angles [0 25. =1255% ) for test-time augmentation. DeepInteraction ensembles multiple Deeplnteraaction-lagge models with input LiDAR BEV grid size between [0.5m, 0.5mm and [1.5m, 1.5m].
For data augmentation, following TransFusion [1] we adopt random rotation with a range of r E [-m/4, T/ /4], random scaling with factor of r [0.9, 1.1], random translation with standard deviation 0.5 in three axis, and random horizontal flipping. We also use the class-balanced re- sampling in CBGS [48] to balance the class distribution for nuScenes. Following [1], we adopt two stage training recipe. We take TransFusion-L [1] as our LiDAR-onle only baseline. We use Adam optimizer with oye-eycle learning rate policy, with max learning rate 1 x 10 weight decay 0.01 and momentum 0.85 to 0.95, following CBGS [48]. Our LiDAR-only baseline is trained for 20 epochs and LiDAR-image fusion for epochs with batch size of 16 using 8 NVIDIA V100 GPUs.
4.3 Comparison to the state of the art
Performance We compare with vfateoffthe-aae alternatives on the nuScenes test set. As shown in Table 1, DeepInteraction achieves new state-of-the-at performance under all settings. The base variant without TTA and model ensemble, DespInteraction. base, with simple ResNet-50 image backbone, surpasses all the prior arts as well as the concurrent work BEVFusion [30] even with Swin-Tiny image backbone. DeepInteraction-lar F [9] with the same TTA and test time augmentation (single model) by considerable margin. Our ensemble version DeepInteraction
beats the closest rival LargeKerneI3D.
achieves first rank among all the solutions on the nuScenes leaderboard. These results verify the performance advantages of our multi-modal interaction approach. Per-category results are shown Appendix A. Qualitative results are provided in Figure and Appendix A.4.
Run time We compare inference speed tested NVIDIA V100, A6000 GPUs and A100 separately. As shown in Table 2, our method achieves the best performance while running faster than alternative maitting-bssed [39] and query-based [7] fusion approaches. This validates superior trade-off between detection performance and inference speed achieved by our method. As found in [39], feature extraction for multi-view high resolution camera images contributes the most of overall latency in multi-modal 3D detector. Indeed, from Table 3(c) we observe that increasing the number of decoder layers only brings negligible extra latency, which concurs with the same conclusion.
4.4 Ablations on the decoder
Multi-modal predictive interaction layer vs. DETR [5] decoder layer In Table 3(a) we evaluate the design of decoder layer by comparing our multi-modas predictive interaction (MMPI) with DETR [5] decoder layer. Note the DETR decoder layer means the conventional Transformer deocder layer is used to aggregate multi-modal information same as in Transfusion [1]. We further test mixing design: using vanilla DETR decoder layer for aggregating features in LiDAR representation and our MMPI for aggregating features in image representation (second row). It is evident that MMPI
7

---

Table 2: Run time comparison. If not specified, the performance and efficiency evaluated on the nuScenes val set.
Method
mAP(%))
NDS(
FPS(A100)T
FPS(A6000)
FPS(V100)
PointAugmenting [39]
66.8 (Test)
71.0 (Test)
1.4
2.8
2.3
FUTR3D [7]
64.2
68.0
4.5
2.3
1.8
Transfusion [1]
67.5
71.3
6.2
5,5
3.8
DeepInteraction
69.9
72.6
4.9
3.1
2.6
Table 3: Ablation studies on the decoder. The mAP and NDS are evaluated on the nuScenes val set.
LiDAR Image
mAP
NDS
Modality
mAP
NDS
DETR [5] DETR [5]
68.6
71.6
Fully LiDAR
69.2
72.2
DETR [5]
MMPI
69.3
72.1
LiDAR-image alternating
69.9
72.6
MMPI
MMPI
69.9
72.6
(a) The type of decoder layer.
(b) Single vs. multiple representations.
of decoder layers
mAP
NDS
FPST
Train
Inference
mAP
NDS
(LiDAR-only)
65.1
70.1
8.7
200
69.9
72.6
69.5
72.3
2.8
200
300
70.1
72.7
69.7
72.5
2.7
400
70.0
72.6
69.8
72.5
2.7
200
69.7
72.5
69.9
72.6
2.6
300
300
69.9
72.6
69.7
72.1
2.5
400
70.0
72.6
(c) The number of decoder layers.
(d) The number of queries.
significantly superior over DETR by improving 1.3% mAP and 1.0% NDS, with combinational flexibility in design.
How many representatoon/modaalitss?? In Table 3(b), evaluate the effect of using different numbers of pespresetaaioon/mmdaaiiiiii in decoding. We compare our MMPI using both representations in an alternating manner with variant using LiDAR representation in all decoder layers. It is observed that using both representations is beneficial, verifying our design consideration.
Number of decoder layers As shown in Table 3(c), increasing the number of decoder layers up to 5 layers can consistently improve the performance whilst introducing negligible latency. LiDAR-only denotes the Transfusion-L [1] baseline.
Number of queries Since our query embeddings are initialized in a non con-parametric and input- dependent manner as in [1], the number of queries are adjustable during inference. In Table 3(d), we evaluate different combinations stable over different choices with 200/300 for training/test the best setting.
query numbers for training and test. Overall, the performance
4.5 Ablations on the encoder
Multi-modal representational interaction vs. fusion To precisely demonstrate the superiority of multi-modal representational interaction, compare naive version of our DeepInteraction with conventional representational fusion strategy as presented in Transfusion [1]. We limit our DeepInterace using the same number of encoder and decoder layers [1] for a fair comparison. Table 4(c) shows that our representational interaction is clearly more effective.
Encoder design We ablate the design of our encoder with focus on multi-modal representational in- teraction (MMRI) and intra-modal representational learning (IML). We have couple observations from Table 4(a): (1) Our MMRI can significantly improve the performance over IML; (2) MMRI and IMI can work well together for further performance gain, As seen from Table 4(b), stacking our encoder layers for iterative MMRI is beneficial.

---

Objects in the image
Before interaction
After interaction
(b)
(c)
Figure 5: Illustrations of the heatmaps predicted from BEV representations before and after repre- sentational interactions. All samples from the nuScenes val split. (a) A case of the occluded tiny object. (b) A case of small object at long distance. (c) An example of two adjacent barriers which connected together in LiDAR point clouds and thus it is difficult to have their instance-level understanding without the help of visual clues.
Qualitative results of representational interaction To gain more insight about the effect of multi-modal representational interaction (MMRI), we visualize the heatmaps of challenging cases. We observe from Figure 5 that without the assistance of MMRI, some objects cannot be detected when using LiDAR only (the middle column). The locations of these objects are highlighted by red circles in the heatmap and white bounding boxes in the RGB image below.
Table 4: Ablation studies on the representational interaction encoder. The mAP and NDS are evaluated on the nuScenes val set.
IML
MMRI
mAP
NDS
# of encoder layers
mAP
NDS
68.1
71.9
w/o
66.4
70.7
69.5
72.5
67.7
71.2
69.9
72.6
69.9
72.6
(a) Encoder design. IML: Intra-modal learning; MMRI: Multi-modal representational interaction.
(b) The number of encoder layers.
Method
mAP
NDS
Representational fusion
67.5
71.3
Representational interaction (Ours)
68.7
71.9
(c) Our representational interaction vs. conventional representational fusion (e.g., Transfusion [1]..

---

Table 5: Evaluation different LiDAR backbones, The mAP and NDS evaluated the nuScenes
val set.
Methods
Modility
mAP
NDS
Methods
Modility
MAP
NDS
PointPillars [22]
L
46.2
59.1
VoxelNet [47]
L
52.6
63.0
+Transfusion-L [1]
L
54.5
62.7
+Transfusion-L. [I]
L
65.1
70.1
+Transfusion [1]
L+C
58.3
64.5
+Transfusion [1]
L+C
67.5
71.3
+Deeplnteraction
L+C
60.0
65.6
+DeepInteraction
L+C
69.9
72.6
(a) Comparison between pillar-based methods.
(b) Comparison between voxel-based methods.
Table 6: Comparison with the LiDAR-only baseline Transfusion-L. [1] on nuScenes val split.
Method
mAP
NDS
Car
Truck
C.V.
Bus
T.L.
B.R.
M.T.
Bike
Ped.
T.C.
Transfusion-L. [1]
65.1
70.1
86.5
59.6
25.4
74.4
42.2
74.1
72.1
56.0
86.6
74.1
DeepInteraction
69.9
72.6
88.5
64.4
30.1
79.2
44.6
76.4
79.0
67.8
88.9
80.0
Concretely, the sample (a) suggests that camera information is helpful to recover partially obscured tiny objects with sparse observation in the point cloud. The sample (b) shows a representative case where some distant objects be recognized successfully due to the help of visual information. From the sample (c), we observe that the centers of some barriers yield a more distinct activation in the heatmap after representational interaction. This is probably due that it is too difficult to locate the boundaries several consecutive barriers from LiDAR point clouds only.
4.6 Ablation on LiDAR backbones
We examine the generality of our framework with two different LiDAR backbones: PointPillars [22] and VoxelNet [47]. For PointPillars, we set the voxel size to (0.2m, 0.2m) while keeping the For fair comparison, we use the same number of queries as TransFusion [1]. As shown Table 5, due to the proposed multi-modal interaction strategy, DeepInteraction exhibits consistent improvements over LiDAR-only baseline using either backbone (by 5.5% mAP for voxel-based backbone, and 4.4% mAP for pillar-based backbone). This manifests the generality of our DeepInteraction across varying point cloud encoder.
remaining settings same as DeepInteraction-bast
4.7 Performance breakdown
To demonstrate more fine-grained performance analysis, compare our DeepInteraction and our LiDAR-only baseline Transfusion [1] at the category level in terms of mAP on nuScenes val set. We can see from Table that our fusion approach achieves remarkable improvements on all the categories, especially on tiny or rare object categories 1.8% mAP for bicycle, +6.9% mAP for motorcycle, and +5.9% mAP for traffic cone).
Conclusion
In this work, we have presented novel 3D object detection method DeepInteraction for exploring the intrinsic multi-moda complementary nature. This key idea is to maintain two codality-speccife representations and establish interactions between them for both representation learning and pre- dictive decoding. This strategy designed particularly to resolve the fundamental limitation of existing unilateral fusion approaches that image representation are insufficiently exploited due to their auxiliary-source role treatment. Extensive experiments demonstrate our proposed DeepInteracti yields new state of the art the nuScenes benchmark dataset and achieves the first position at the highly competitive nuScenes 3D object detection leaderboard.
Acknowledgement This work was supported in part by National Natural Science Foundation of China (Grant No. 6210020439), Lingang Laboratory (Grant No. LG-QS-202202-007 Natural Science Foundation of Shanghai (Grant No. 227R 1407500), Science and Technology Innovation 2030 Brain Science and Arain-Inspired Intelligence Project (Grant No. 2021ZD02200204) and Shanghai Municipal Science and Technology Major Project (Grant No. 2018SHZDZX01),
10

---

References
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In CVPR, 2022. 2, 2,5,0 6, 7, 8, 10
[2] Alex Bewley, Pei Sun, Thomas Mensink, Dragomir Anguelov, and Cristian Sminchisescu. Range conditioned dilated convolutions for scale invariant 3d object detection. arXiv preprint. 2020. 2
[3] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishman, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020. 6,
[4] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: high quality object detection and instance segmentation. IEEE TPAMI, 2019. 6
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020. 2, 4, 8
[6] Yuning Chai, Pei Sun, Jiquan Ngiam, Weiyue Wang, Benjamin Caine, Vijay Vasudevan, Xiao Zhang, and Drago Anguelov. To the point: Efficient 3d object detection in the range image with graph convolution kernels. In CVPR, 2021. 2
[7] Xuanyao Chen, Tianyuan Zhang, Yue Wang, Yilun Wang, and Hang Zhao. Futr3d: A unified sensor fusion framework for 3d detection. arXiv preprint, 2022. 2, 6, 7, 8
[8] Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection. arXiv preprint, 2022. 6
[9] Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Scaling up kernels in 3d cnns. arXiv preprint, 2022. 6, 7
[10] Zchui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinghong Jiang, Feng Zhao, Bolei Zhou, and Hang Zhao. Autoalign: Pixel-instance feature aggregation for multi-modal 3d object detection. arXiv preprint, 2022. 1,
[11]
MMDetection3D Contributors. MMDetection33:: Open next-generation platform for general 3D object detection. https://gi com/ open mm www.aa/mmmmeeeteeeeeeeee 2020.
[12] Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2010. 6
[13] Lue Fan, Ziqi Pang. Tianyuan Zhang, Yu-Xiong Wang, Hang Zhao, Feng Wang. Naiyan Wang. and Zhaoxiang Zhang. Embracing single stride 3d object detector with sparse transformer. arXiv preprint, 2021.
[14] Lue Fan, Xuan Xiong, Feng Wang, Naiyan Wang, and ZhaoXiang Zhang. Rangedet: In defense of range view for lidar-based 3d object detection. In ICCV, 2021.
[15] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. ICCV, 2017.
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[17] Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du. Bevdet: High- performance multi- camera 3d object detection in bird-eye-views arXiv preprint, 2021. 2, 6
[18] Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3d object detection. In ECCV, 2020. 2
[19] Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers, arXiy preprint, 2022. 6
11

---

[20] Jason Ku, Ali Harakeh, and Steven L Waslander. In defense of classical image processing: Fast depth completion on the cpu. In CRV, 2018. 4
[21] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Waslander. Joint 3d proposal generation and object detection from view aggregation. In IROS, 2018. 2
[22] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In CVPR, 2019. 2, 10
[23] Yanwei Li, Xiaojuan Qi. Yukang Chen, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. Voxel field fusion for 3d object detection. In CVPR, 2022.
[24] Yingwei Li. Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Bo Wu, Yifeng Lu, Denny Zhou, al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. arXiv preprint, 2022.
[25] Zhiqi Li. Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: Learning bird's-cye-viers representation from multi-camera images via spatiotemporal transformers. arXiv preprint, 2022. 6
[26] Tingting Liang, Hongwei Xie, Kaicheng Yu. Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. BEVFusion: A Simple and Robust LiDAR- Cameaa Fusion Framework. arXiv preprint, 2022. 1,
[27] T. Y. Lin, M. Maire, S. Belongie, J. Hays, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 6
[28] Yingfei Liu, Tiancai Wang. Xiangyu Zhang, and Jian Sun. Petr: Position embedding transfor- mation for multi-view 3d object detection. arXiv preprint, 2022. 2
[29] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
6
[30] Zhijian Liu, Haotian Tang, Alexander Amini, Xingyu Yang. Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's s-yeeeey view representation. arXiv preprint, 2022. 1, 6,7
[31] Jiachen Lu, Zheyuan Zhou, Xiatian Zhu, Hang Xu, and Li Zhang. Learning ego 3d representation ray tracing. In ECCV, 2022. 2,
[32] Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In CVPR, 2021. 2
[33] Ishan Misra, Rohit Girdhar, and Armand Joulin. An End-to-End Transformer Model for 3D Object Detection. In ICCV, 2021. 2
[34] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. ECCV, 2020. 3
[35] AJ Piergiovanni, Vincent Casser, Michael S Ryoo, and Anelia Angelova. 4d-net for learned multi-modal alignment. In CVPR, 2021. 2
[36] Charles R Qi, Xinlei Chen, Or Litany, and Leonidas Guibas. Imvotenet: Boosting 3d object detection in point clouds with image votes. In CVPR, 2020. 2
[37] Cody Reading, Ali Harakeh, Julia Chae, and Steven L. Waslander. Categorical depth distribu- tionnetwork for monocular 3d object detection. In CVPR, 2021.
[38] Sourabh Vora, Alex Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In CVPR, 2020. 1.2
[39] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmentation for 3d object detection. In CVPR, 2021. 1, 2, 7,8
12

---

[40] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, 2019. 2
[41] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In CoRL, 2022.
[42] Yue Wang and Justin M. Solomon. Object dgenn: 3d object detection using dynamic graphs. In NeurlPS, 2021. 2
[43] Shaoqing Xu, Dingfu Zhou, Jin Fang. Junbo Yin, Bin Zhou, and Liangjun Zhang. Fusion- Painting: Multimodal fusion with adaptive attention for 3d object detection. ITSC, 2021.
[44] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 2018.
[45] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In CVPR, 2021. 6
[46] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Multimodal virtual point 3d detection. NeurlPS, 2021. 1, 2,
[47] Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. CVPR, 2018. 2,
[48] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint, 2019. 7
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 4.
(b) Did you describe the limitations of your work? [Yes] See supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "nttt4noTrg", "reviewer": "Reviewer_2uMx", "summary": "The paper introduces DeepInteraction, a new approach to fuse rgb and depth input to do 3D object detection. Prior works only add rgb features to the Lidar point cloud, while DeepInteraction treat them equally and fuse them in the transformer layers. Experiments are conducted on nuScenes benchmark and DeepInteraction achieves state-of-the-art performance.\n", "questions": "See weaknesses.", "limitations": "Limitations are discussed in the paper.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- DeepInteraction achieves state-of-art performance on nuScenes benchmark. Under the same evaluation setting (without ensemble), it generally improves mAP by 2 points whether test-time augmentation is used or not.\n- The experiments on nuScenes are extensive. It's interesting to see DeepInteraction has reasonable performance whether the 3D backbone is PointPillars or VoxelNet, and VoxelNet works better.\n\nWeaknesses:\n\n- How does the approach work on other automonous driving datasets? I'm convinced DeepInteraction helps recover tiny objects which are just several pixels in the Lidar points. But are such cases really general in other benchmarks? Or, are there any reasons it cannot be run on other benchmarks?\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "I3ixA8LORn0", "reviewer": "Reviewer_83YX", "summary": "(1) The paper reveals and gives an analysis of the unilateral limitation of feature aggregation on different modalities for existing methods.\n(2) The paper proposes a new framework for bilateral feature interaction and association on different modalities.\n(3) The paper achieves state-of-the-art results on nuScenes benchmark.", "questions": "Authors are encouraged to provide additional experiments for the two concerns I list in the weakness part:\n\n(1) The comparison of speed and memory.\n(2) Some visual results(e.g., heatmap of two sources) to convince the effect of the proposed bilateral framework.", "limitations": "The authors have presented the potential societal impacts and limitations in their supplementary material. Overall, they look good to me.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(1) The paper is well-written and easy to follow, especially for demonstrating its motivation in the introduction session.\n(2) The experiments are sufficient and convincing for both comparing with previous methods and ablation studies.\n\nWeaknesses:\n(1) The paper lacks the comparison of speed and memory between different methods. For the autonomous driving scenarios whether the model can run in real-time is also important. I'm curious about whether the DETR-based framework can run as fast as, at least not slower too much compared with previous SOTAs.\n(2) Although the proposed bilateral system is reasonable, it would be great to show some visualizations of the heat map of the features from the two modalities to help readers get a deeper understanding on how the visual information from the two sources is well selected and utilized. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "7CQ3GATLyZn", "reviewer": "Reviewer_F1dJ", "summary": "This work tackles the task of 3D object detection in autonomous driving scenarios. The claimed key contribution is the proper combination of two input modalities: RGB camera images and point clouds from LiDAR scanners (unlike prior methods which treat one of the modalities as auxiliary). The proposed method outperforms prior methods on the nuScenes benchmark. The method is analyzed on the same dataset.", "questions": "(1) Which experiment in the ablation study shows that the bilateral feature fusion is indeed the key component necessary for the improved scores?\n\n(2) How does the model perform on other datasets e.g. KITTI 3D object detection?\n\n**Minor Details**\n\n- Fig.2: inconsistent notation \\phi <-> \\varphi in (a) and (b).\n- l.59 ‘preform’ → ‘perform’\n- reference [5] and [6] are the same\n", "limitations": "Potential negative societal impact is not discussed (there is a section in the supplementary titled as such but discusses only research impact). Limitations are only discussed on an abstract meta level. Overall, the two sections (E, F) in the supplementary are not adding any real value to the paper and seem to exist only to tick the required boxes.", "rating": 1, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The paper makes a strong point that the bilateral fusion between camera and lidar modalities is the key part that contributes to the strong performance of the proposed model. From the ablation study alone it is not directly clear if the improvements come from the change from unilateral to bilateral modality information flow or from stronger backbones which seem to have the biggest impact according to Table (e).\n\n- The method is evaluated only on a single dataset. There are multiple other popular 3D object detection benchmarks, e.g., [KITTI](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d).\nEvaluation on more datasets would show the generalization capability of the model.\n\n- Important related work on image depth fusion and transformer-based 3D object detection appears to be missing, e.g., [ImVoteNet](https://arxiv.org/pdf/2001.10692.pdf), [3DETR](https://arxiv.org/abs/2109.08141) [Misra et al. ICCV 2021].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["Research Integrity Issues (e.g., plagiarism)", "Responsible Research Practice (e.g., IRB, documentation, research ethics)"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept