Title: DH-Fusion: Depth-Aware Hybrid Feature Fusion for Multimodal 3D Object Detection
Venue: NeurIPS 2024
Accepted: no
Keywords: Depth Aware

Multi Modality

3D Object Detection
Abstract: State-of-the-art LiDAR-camera 3D object detectors usually focus on feature fusion. However, they neglect the factor of depth while designing the fusion strategy. In this work, we for the first time point out that different modalities play different roles as depth varies via statistical analysis and visualization. Based on this finding, we propose a Depth-Aware Hybrid Feature Fusion (DH-Fusion) strategy that guides the weights of point cloud and RGB image modalities by introducing depth encoding at both global and local levels. Specifically, the Depth-Aware Global Feature Fusion (DGF) module adaptively adjusts the weights of image Bird's-Eye-View (BEV) features in multi-modal global features via depth encoding. Furthermore, to compensate for the information lost when transferring raw features to the BEV space, we propose a Depth-Aware Local Feature Fusion (DLF) module, which adaptively adjusts the weights of original voxel features and multi-view image features in multi-modal local features via depth encoding. Extensive experiments on the nuScenes dataset demonstrate that our DH-Fusion method surpasses previous state-of-the-art methods w.r.t. NDS. Moreover, our DH-Fusion is more robust to various kinds of corruptions, outperforming previous methods on nuScenes-C w.r.t. both NDS and mAP.

Abstract

State-of-the-art LiDAR-camera 3D object detectors usually focus on feature fusion. However, they neglect the factor of depth while designing the fusion strategy. In this work, we for the first time point out that different modalities play different roles as depth varies via statistical analysis and visualization. Based on this finding, we propose a Depth-Aware Hybrid Feature Fusion (DH-Fusion) strategy that guides the weights of point cloud and RGB image modalities by introducing depth encoding at both global and local levels. Specifically, the Depth-Aware Global Feature Fusion (DGF) module adaptively adjusts the weights of image Bird's-Eye-View (BEV) features in multi-modal global features via depth encoding. Furthermore, to compensate for the information lost when transferring raw features to the BEV space, we propose a Depth-Aware Local Feature Fusion (DLF) module, which adaptively adjusts the weights of original voxel features and multi-view image features in multi-modal local features via depth encoding. Extensive experiments on the nuScenes dataset demonstrate that our DH-Fusion method surpasses previous state-of-the-art methods w.r.t. NDS. Moreover, our DH-Fusion is more robust to various kinds of corruptions, outperforming previous methods on nuScenes-C w.r.t. both NDS and mAP.

1 Introduction

3D object detection has a wide range of applications in the fields of autonomous driving and robotics. A large number of previous works have successfully focused on using a single modality, such as point cloud or images, to design efficient 3D object detectors. However, the performance of these detectors reaches a bottleneck due to the limitations of modality characteristics. For instance, the point cloud modality can only provide rich geometric information while lacks detailed semantic information; the image modality can only provide rich texture information while lacks three-dimensional spatial information. To address the aforementioned issues, we are highly motivated to obtain comprehensive information that represents objects by designing a LiDAR-camera 3D object detector.

In recent years, LiDAR-camera 3D object detection develops rapidly. Some works [1, 4, 28, 33, 67] 27 propose effective methods to integrate information from two modalities at the feature level. However, 28 they all overlook an important factor of depth in their fusion strategies. To understand how point 29 cloud and image information vary with depth, we first conduct statistical and visualization analysis 30 on the nuScenes-mini dataset [3], and find that: (1) The number of points representing objects at 31 near range is relatively large, which allows us to accurately determine the object's location, size, and 32 category, even without the aid of images. As shown in Fig. 1a, there is an average of 163.7 points per 33 object within 0-10 meters, which is a substantial number. We also visualize a car at 6.8 meters in 34 Fig. 1b ① and find it encompasses a considerable number of points, well representing the shape. In 35 contrast, some background noise in the image may interfere with detection (Fig. 1b ② ). (2) As the 36

5.9e4

10.4

1

Number of Points Per Object

0-10      10-20      20-30     30-40     40-50

Depth (m)

20

10

30

40

100

200

1

4.0e3

8.0e3

1.2e4

1.6e4

1.0e6

2.3e7

1.0e5

1.6e4

7.8e3

4.1e3

163.9

2.9

0.4

0.3

Point

1.0e7

Number of Pixels Per Object

150

(a) Statistical chart

Image

6.8m

④

(2)

③

①

②

42.1m

35.3m

6.8m

42.1m

35.3m

(b) Visualization

depth increases, the number of points representing objects decreases rapidly. As shown in Fig. 1a, 37

the number of points within 30-50 meters falls below one per object, meaning that many objects are 38

even not represented by any points, such as the object at 42.1 meters in Fig. 1b ③ . In contrast, the 39

complete objects may still be observed on the image, as in Fig. 1b ④ , where the image information 40

becomes more important. To address the above problems, we propose a feature fusion strategy that 41

adaptively adjusts the importance of the two modalities based on depth. 42

Specifically, we propose a novel method for multi-modal 3D object detection, namely Depth-Aware 43

Hybrid Feature Fusion (DH-Fusion). The innovation lies in adaptively adjusting the weights of 44

features by introducing depth encoding to hybrid feature fusion at both global and local levels. The 45

fusion strategy consists of two crucial components: Depth-Aware Global Feature Fusion (DGF) 46

module and Depth-Aware Local Feature Fusion (DLF) module. In DGF, we take point cloud Bird's47

Eye-View (BEV) features and image BEV features as inputs, and dynamically adjust the weights of 48

image BEV features based on depth during fusion by utilizing a global-fusion transformer encoder 49

with a depth encoder. To compensate for the information lost when transforming raw features to 50

BEV space, we enhance the fused BEV features at a lower cost by utilizing the original instance 51

features. In DLF, we obtain 3D boxes by utilizing a Region Proposal Network (RPN). Then, the 52

3D boxes are projected into both LiDAR voxel features and multi-view image features to crop out 53

corresponding local instance features with more detailed information. Afterward, we take these as 54

inputs and dynamically adjust the weights of local multi-view image features and local LiDAR voxel 55

features based on depth through the use of a local-fusion transformer encoder with the depth encoder. 56

57

58

In the end, we update local features for each object on the global feature map to enhance the detailed instance information of multi-modal global features for detection.

Our contributions are summarized as follows. 59

We for the first time point out that depth is an important factor to consider while fusing LiDAR 60 point cloud features and RGB image features for 3D object detection. From our statistical and 61 visualization analysis, we can see that image features play different roles as depth varies. 62

We propose a depth-aware hybrid feature fusion strategy that dynamically adjusts the weights of 63 features during feature fusion by introducing depth encoding at both global and local levels. The 64 above strategy can obtain high-quality features for detection, fully leveraging the advantages of 65 different modalities at various depths. 66

Our method is evaluated on the nuScenes [3] dataset and a more challenging nuScenes-C [13] 67

68

dataset, outperforming previous multi-modal methods and being robust to various kinds of data

69

corruptions.

2 Related Work 70

Since our method is based on conducting 3D object detection using data from multiple modalities, 71

including point cloud and images, we briefly review recent works in the following fields: LiDAR72

based 3D object detection, camera-based 3D object detection, and LiDAR-camera 3D object detection. 73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

2.1 LiDAR-based 3D Object Detection

LiDAR-based 3D object detectors only take the point cloud as input. Based on their different data representations, they can be divided into point-based [44-46, 64, 65], voxel-based [12, 22, 61, 68, 71], and point-voxel-based [17, 42, 43] methods. The feature extraction networks of point-based methods typically extract features directly from the point cloud through a point-based backbone [40], such as PointRCNN [44]. The voxel-based methods first convert the point cloud into voxels and then extract voxel features through a 3D sparse convolution network [14], such as VoxelNet [71]. Point-voxelbased methods like PV-RCNN [42] combine the above two methods to extract and fuse point and voxel features. The purpose of these approaches is to capture the geometric spatial information of the point cloud. However, point cloud is sparse and incomplete, lacking detailed texture information, which greatly limits the detection performance.

2.2 Camera-based 3D Object Detection

Camera-based 3D object detectors only take images as inputs. Depending on the form of inputs, they can be divided into monocular [2, 24, 32, 41, 47, 55], stereo [6, 25, 30, 48, 70], and multi-view [19, 27, 56, 62] 3D object detectors. Early works like FCOS3D [55] input a monocular image and utilize 2D object detectors to directly predict 3D bounding boxes, but these approaches have limited capability in capturing spatial information. Subsequently, stereo and multi-view 3D object detectors are proposed to obtain more precise depth information by constructing spatial relationships among multiple images, such as Stereo RCNN [25] and BEVDet [19]. These methods successfully achieve purely visual 3D object detection, but they do not perform as well as LiDAR-based methods, because the spatial depth information provided by images is not as direct and precise as that provided by point cloud.

2.3 LiDAR-Camera 3D Object Detection

LiDAR-camera 3D object detectors take point cloud and images as inputs, and can be classified into early-fusion-based [50, 52, 57, 59, 69], intermediate-fusion-based [1, 4, 28, 33, 67], and latefusion-based [37, 38] 3D object detectors based on the location of multi-modal information fusion [36].

Early-fusion-based methods perform at the point level, where the typical approach involves enhancing the raw point cloud with semantic information extracted from images. PointPainting [50] and FusionPainting [59] decorate the raw point cloud with semantic scores from 2D semantic segmentation. Similarly, PointAugmenting [52] enhances the raw point cloud using features extracted from a 2D semantic segmentation network. However, early-fusion-based methods are sensitive to alignment errors between the two modalities.

Intermediate-fusion-based methods perform at the feature level. Transfusion [1] first proposes to utilize the transformer for fine-grained fusion from LiDAR BEV features and multi-view image features. FUTR3D [5] encode each modality using deformable attention [73] in its own coordinate and concatenate them for fusion. BEVFusion [28, 33] projects both point cloud and images to BEV space for BEV feature fusion. SparseFusion [58] extracts instance-level features from both two modalities separately, and fuse them to perform detection. Similarly, ObjectFusion [4] utilizes 3D proposals from LiDAR modality to extract instance-level features for fusion. CMT [60] proposes the simultaneous interaction between the object queries and multi-modal features in the transformer encoder and decoder. IS-Fusion [67] proposes feature fusion at both the instance level and scene level. The intermediate-fusion-based methods gradually become a mainstream approach due to the diversity of fusion strategies.

Late-fusion-based methods perform at the bounding box level. Typically, CLOCs [37] obtains 2D and 3D bounding boxes by separately using 2D and 3D object detectors, and then combine them to achieve more accurate 3D bounding boxes. However, the interaction between modalities in late-fusion-based methods is very limited, which constrains model performance.

These multi-modal methods successfully outperform single-modal methods. However, their feature fusion methods do not take depth into account. In contrast, our approach introduces depth information to guide the hybrid feature fusion, boosting the performance of the detector.

Multi-view Images

3D Encoder

2D Encoder

…

…

…

Decoder

Detection

Head

Global Feature Fusion

Local Feature Fusion

Depth Encoder

Input Encoding

Depth-Aware Global Feature Fusion

Depth-Aware Local Feature Fusion

Decoding

G

O

Select

Select

Project

Project

View

Transfer

Compress

G

O

G

B

G

B

L

O

G

B

L

B

L

O

G

B

ˆ

G

B

M

Select

Depth Encoder

Depth Encoder

M

Sine and

Cosine

De

000

H

I1

3 Methodology 125

In this section, we first give an overview of our proposed multi-modal 3D object detector, and then 126 provide a detailed introduction to our proposed feature fusion method. 127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

3.1 Overview

We propose a multi-modal 3D object detection method via Depth-Aware Hybrid Feature Fusion (DH-Fusion). As illustrated in Fig. 2, our approach consists of two important feature fusion modules: Depth-Aware Global Feature Fusion (DGF) and Depth-Aware Local Feature Fusion (DLF). In the following, we briefly describe the detection pipeline.

Inputs. First, we take the point cloud P and multi-view images I as inputs, where point cloud consists of a set of points: P = { P 1 , P 2 , · · · , P N l } , and each point has four dimensions: X-axis, Y-axis, Z-axis, and intensity; the multi-view images comprise N c images: I = { I 1 , I 2 , · · · , I N c } , each image captured by its corresponding camera.

Input Encoding. For the point cloud P , we use a 3D encoder to extract raw global voxel features V G O ; for the multi-view images I , we use a 2D encoder to extract image features of all views I G O .

Hybrid Feature Fusion. Then, for voxel features V G O , we compress the height dimension to obtain point cloud BEV features V G B ; for image features I G O , we transform their perspective view to bird's eye view to obtain image BEV features I G B . To fully leverage the features from two modalities, we design a DGF module that aims to dynamically adjust the weights of image BEV features based on depth values during feature fusion. Please refer to Sec. 3.2 for more details. To compensate for the information lost when transforming raw features to BEV space, we propose a DLF module that, based on depth, utilizes the raw features to enhance the detailed information of each object instance in global multi-modal features. It consists of three processes: local feature selection, local feature fusion, and merging local features into global features. First, we obtain the local multi-modal BEV features F L B , local voxel features V L O , and local multi-view image features I L O , by cropping the corresponding global features based on the 3D boxes obtained from an RPN; then, it dynamically and individually adjusts the weights of each local feature of V L O and I L O based on depth values during feature fusion; finally, we update local features for each object on the global feature map. Please refer to Sec. 3.3 for more details. In this way, we obtain enhanced multi-modal global features for detection.

Decoding. Based on the enhanced multi-modal global features ˆ F G B that contain rich semantic and 154 spatial information, we utilize a transformer decoder and a detection head to predict the object 155 categories and 3D bounding boxes. 156

157

158

159

160

161

162

163

167

168

169

170

171

172

173

174

175

176

177

Cross Attention

Add & Norm

FFN

Add & Norm

Multiply & Norm

Global-Fusion Transformer

Q

K

V

G

B

G

B

Depth Encoder

G

B

一

Cross Attention

Add & Norm

Depth Encoder

Multiply & Norm

…

Projection

Cross Attention

Add & Norm

Depth Encoder

Multiply & Norm

…

…

Projection

Cat & Conv

Merge

Local Feature Selection

Local-Fusion Transformer

Q

K

V

Q

K

V

G

O

G

O

L

O

G

B

L

B

L

O

G

B

ˆ

11

11

11

11

11

11

11

个

个

3.2 Depth-Aware Global Feature Fusion

As shown in Fig. 3, the DGF module consists of a global-fusion transformer with a depth encoder. In the following, we provide a detailed explanation of each component.

3.2.1 Depth Encoder

We introduce depth encoding (DE) in feature fusion to dynamically adjust the weights of image BEV features during fusion. First, we build a depth matrix M to store the depth value of each position element p k represented as:

where ( x k , y k ) are the positional coordinates, d k is the depth value, and n is the number of elements. 164 Then, we use Euclidean distance to calculate the distance between every element's spatial location 165 ( x k , y k ) and the ego coordinate element's location ( x n 2 , y n 2 ) : 166

where we denote E ( · ) as the Euclidean distance calculation. The depth matrix M serves as a lookup table to avoid redundant computation of depth values. Since the size of the BEV features is large and the depth distribution is simple, to avoid introducing additional parameters, the depth encoding De is obtained by applying sine and cosine functions [49] to the depth matrix.

3.2.2 Global-Fusion Transformer

In the global-fusion transformer, we take the point cloud BEV features V G B ∈ R W × H × C and image BEV features I G B ∈ R W × H × C as inputs, and integrate the depth encoding obtained above by multiplying it with the point cloud BEV features, forming the query Q G V = N ( V G B × Conv ( De )) , where Conv ( · ) is a convolution operation to align with the channels of V G B , and N ( · ) is a normalization layer. The image BEV features are queried as the corresponding key K G I and value V G I . We utilize the multi-head cross attention to achieve the interacted feature ˆ V G B based on depth:

where CA ( · ) indicates the multi-head cross attention. Afterward, we aggregate the information from 178 both modalities to obtain the fused features F G B : 179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

where N ( · ) is a normalization layer; FFN ( · ) specifies a feed-forward network containing two convolution operations. In this way, we obtain fused features in which the image features play different roles as the depth varies.

3.3 Depth-Aware Local Feature Fusion

As shown in Fig. 4, the DLF module consists of a local feature selection and a local-fusion transformer with the depth encoder. In the following, we provide a detailed explanation of each component.

3.3.1 Local Feature Selection

To compensate for the information lost when transforming point cloud features and image features to BEV space, we enhance the instance details of fused BEV features F G B using instance features from raw voxel features V G O and multi-view image features I G O . Specifically, we utilize an RPN to regress t 3D boxes based on the BEV features F G B . We directly crop the global fused BEV features F G B based on the regressed 3D boxes to obtain the local fused BEV features F L B ∈ R c × t . On the other hand, we project the 3D boxes onto the raw voxel features and multi-view image features to obtain their corresponding local features before global fusion, preserving richer information for each object instance. Specifically, we utilize the voxel pooling operation [12], followed by a 3D convolution operation and a linear layer, to extract local voxel features V L O ∈ R c × t ; we transform the 3D boxes from bird's eye view to perspective view, and utilize the RoI Align operation [15], followed by a linear layer, to extract instance image features I L O ∈ R c × t . By doing this, we obtain the hybrid (before & after global fusion) local features, which will be sent to the subsequent fusion module.

3.3.2 Local-Fusion Transformer

In the local-fusion transformer, the weights of each local raw feature are dynamically adjusted based on depth values during feature fusion, and we update local features for each object on the global feature map. Specifically, we take the local multi-modal BEV features F L B , local voxel features V L O , and local multi-view image features I L O as inputs, and integrate the depth encoding by multiplying it with the local multi-modal BEV features, forming the query Q L F . The local multi-view image features and local voxel features are respectively queried as the corresponding key K L I , K L V and value V L I , V L V . The two multi-head cross-attention modules are utilized to achieve the interacted features ˆ Q L F , ˆ Q L F ′ . Note that the computation process of multi-head cross attention is similar to that described in Sec. 3.2.2 and is omitted here. Afterward, we aggregate the above features:

where Cat ( · ) is the concatenation operation; Conv ( · ) is used to align with the feature channels of global fused BEV features F G B . As a result, we obtain enhanced local features by dynamically calling back rich information in raw modalities at various depths. Afterward, we update the global features F G B by inserting the enhanced local features at corresponding locations.

4 Experiments

In this section, we will first introduce the dataset and evaluation metrics, followed by the implementation details. Then, we will compare our method with the state-of-the-art methods on nuScenes and also present results on a more challenging dataset of nuScenes-C with data corruptions. Finally, we will show the ablation studies and qualitative results. More experiments are provided in Appendix A.2.

4.1 Experimental Setup

Datasets and evaluation metrics. We evaluate our proposed DH-Fusion on the nuScenes benchmark [3] and a more challenging dataset of nuScenes-C [13] with data corruptions. nuScenes dataset provides 700 scene sequences for training, 150 scene sequences for validation, and 150 scene sequences for testing. Each sequence contains 40 frames of 32-beam LiDAR data, and each frame

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

has six corresponding images covering a 360-degree field of view. It offers calibration matrices that facilitate accurate projection of 3D points onto 2D pixels, and contains 10 object categories that are commonly encountered within autonomous driving. nuScenes-C dataset provides 27 corruptions with 5 severities on the nuScenes validation set, including corruptions at the weather, sensor, motion, object, and alignment level. We use the nuScenes detection scores (NDS) and mean Average Precision (mAP) to evaluate our detection results, where NDS is a comprehensive metric in nuScenes that combines object translation, scale, orientation, velocity, and attribute errors.

Implementation details. We implement the proposed DH-Fusion with PyTorch [39] under the open-source framework MMDetection3D [10]. Specifically, for the LiDAR branch, we use VoxelNet [71] with FPN [61] as the 3D encoder. The voxel size is set to [0.075m, 0.075m, 0.1m], and the range of point cloud is [-54m, 54m] along the X-axis, [-54m, 54m] along the Y-axis, and [-3m, 5m] along the Z-axis. For the image branch, we use the ResNet18 [16], ResNet50 [16], and SwinTiny [34] with FPN [29] as the 2D image encoder of DH-Fusion-light, -base, -large, respectively. Correspondingly, the resolution of input images is resized to 256 × 704, 320 × 800, and 384 × 1056. Additionally, we utilize BEVPoolV2 [18] to obtain image BEV features. Following [33], the feature size W × H is set to 180 × 180, the channel C is set to 128, and the channel c is also set to 128. The multi-head cross attention is implemented with 8 heads, and the FFN contains 2 MLP layers with a hidden dimension of 128. Following [58], the number of regressed 3D boxes t is set to 200. More implementation details are provided in Appendix A.1.

4.2 Comparison to the State of the Art

Aiming for a fair comparison, we categorize previous methods based on the types of 2D backbones into ResNet50-based, SwinTiny-based, and others, and provide three versions of our proposed method, named DH-Fusion-light, DH-Fusion-base, and DH-Fusion-large. The results are shown in Tab. 1. (1) Compared with the ResNet50-based methods, our DH-Fusion-base outperforms the top method FocalFormer3D [7] by up to 1 pp w.r.t. NDS under the same configuration. Specifically, we reach 74.0% w.r.t. NDS and 71.2% w.r.t. mAP on the validation set, and 74.7% w.r.t. NDS and 71.7% w.r.t. mAP on the test set, while maintaining comparable inference speed of 8.7 FPS on a 3090 GPU. (2) Compared with the SwinTiny-based methods and others, our DH-Fusion-large outperforms the top method IS-Fusion [67] under the same configuration, and runs 2x faster than it. Specifically, we reach 74.4% w.r.t. NDS on the validation set, and 75.4% w.r.t. NDS on the test set, while achieving a faster inference speed of 5.7 FPS on a 3090 GPU, indicating that our proposed method is both more effective and efficient. (3) Furthermore, our DH-Fusion-light surpasses the typical BEVFusion [33] by up to 1 pp w.r.t. all metrics using a lighter 2D backbone, and achieves a real-time inference speed of 13.8 FPS. Overall, our method achieves higher detection accuracy and faster inference speed.

4.3 Robustness to Corruptions

We further implement some experiments on the nuScenes-C [13] dataset to evaluate the model's robustness under various corruptions, including changes in weather, data loss or temporal-spatial misalignment in multi-modal inputs, etc. The results for different kinds of corruptions are shown in Tab. 2, and more detailed results for each fine-grained corruption are shown in Appendix A.2.3. We find that our DH-Fusion-light still achieves an average performance of 68.67 % w.r.t. NDS and 63.07 % w.r.t. mAP under various corruptions, which only decreases by 4.63 pp w.r.t. NDS and 6.68 pp w.r.t. mAP, compared to its performance without corruptions. Performance drop is smaller than that observed with previous methods including BEVFusion [28] across all kinds of corruptions, indicating that our DH-Fusion-light possesses superior robustness. Furthermore, we observe that our DH-Fusion-light is particularly robust against weather and object corruptions, where the performance drop is less than 3pp. The more stable performance indicates that our method is more friendly to practical applications, where data corruption may occur.

4.4 Ablation Studies

We conduct ablation studies to first demonstrate the effect of each component of DH-Fusion, then to demonstrate the effect of depth encoding in DGF and DLF, and finally to assess the impact of multiplying depth encoding. All method variants are implemented on the nuScenes validation dataset.

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

Effect of DGF and DLF. To demonstrate the effect of DGF and DLF, we conduct experiments by integrating the components one by one into the baseline, BEVFusion [33]. The results are shown in Tab. 3. We find that our DGF improves the baseline performance by 1.0 pp w.r.t. NDS and 0.9 pp w.r.t. mAP. This demonstrates that dynamically adjusting the weights of the image BEV features during fusion is effective for 3D object detection. Additionally, our DLF improves the baseline performance by 1.3 pp w.r.t. NDS and 0.8 pp w.r.t. mAP, which indicates that dynamically adjusting the weights of the local raw instance features based on depth during fusion effectively compensates for the information loss caused by the transformation of global features into the BEV feature space. The results of integrating both components show an improvement of 1.9 pp w.r.t. NDS and 1.3 pp w.r.t. mAP, well verifying the benefits of dynamically fusing global and local hybrid features based on depth.

Effect of depth encoding in DGF and DLF. To evaluate the effectiveness of our depth encoding, we conduct experiments where the depth encoding is removed from the DGF and DLF modules, respectively. The results are shown in Tab. 4. When removing the depth encoding from Baseline+DGF, the performance drops by 0.6 pp w.r.t. NDS and 0.4 pp w.r.t. mAP. Similarly, when removing the depth encoding from Baseline+DLF, the performance also decreases by 1.1 pp w.r.t. NDS and 0.9 pp w.r.t. mAP. These results indicate that our depth encoding is effective. Furthermore, we observe that removing the depth encoding from the DLF module results in a larger performance drop, suggesting that depth encoding plays a more crucial role in local feature fusion.

Impact of different operations for depth encoding. We conduct experiments with different operations of depth encoding, including concatenation, summation, and multiplication. The results in Tab. 5, show that the multiplication operation consistently outperforms the summation and concatenation operations w.r.t. both metrics. The superior performance of multiplication can be attributed to its ability to more effectively modulate the feature maps based on depth information. Unlike summation, which simply shifts the feature values, or concatenation, which increases the dimensionality without direct interaction, multiplication allows for more interaction between the

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

(a) Attention weights

(b) Average map

0.5

0.4

Feature Values

0.3

0.2

0.1

0.0

10

20

Depth

40

50

BEV Feature

BEVFusion

DH-Fusion

（

Ours

）

Left Front Image

depth encoding and features, leading to better feature representation and ultimately improving the detection performance.

4.5 Qualitative Results

To better understand how depth encoding affects the feature fusion, in Fig. 5, we plot a curve to observe how the attention weights applied on the image BEV features in our DGF module vary with depth, and visualize the average attention map. It is evident that the weights of the image BEV features stay low in near range, but go up significantly as depth increases when the depth is larger than 40 meters. This trend supports our hypothesis that the image modality would become more important as depth increases. In this way, our depth encoding allows the model to dynamically adjust the weights of image BEV features based on depth.

We also compare the detection results of our DH-Fusion method with the baseline BEVFusion [33] in Fig. 6, where we clearly find that our method better localizes those distant objects compared to BEVFusion. These results demonstrate that our proposed multi-modal fusion strategy based on depth is more effective for detection. Besides, we exhibit the corresponding BEV feature maps, where our method shows a stronger feature response for the foreground objects, especially for distant ones. That is why our feature fusion strategy can provide higher-quality detection results. More qualitative results can be found in Appendix A.3.

5 Conclusion

In this paper, we for the first time point out that different modalities play different roles as depth varies 319 via statistical analysis and visualization. Based on this finding, we propose a feature fusion strategy 320 for multi-modal 3D object detection, namely Depth-Aware Hybrid Feature Fusion (DH-Fusion), that 321 dynamically adjusts the weights of features during feature fusion by introducing depth encoding at 322 both global and local levels. Extensive experiments on the nuScenes dataset demonstrate that our 323 DH-Fusion method surpasses previous state-of-the-art methods w.r.t. NDS. Moreover, our DH-Fusion 324 is more robust to various kinds of corruptions, outperforming previous methods on the nuScenes-C 325 dataset w.r.t. both NDS and mAP. Our method uses an attention-based approach to interact with 326 the two modalities, making the detection results sensitive to modality loss. We plan to further 327 explore feature fusion methods that are robust to modality loss. Although our method improves 328 detection performance, emergency plans still need to be implemented in practical applications to 329 ensure personnel safety. 330

References 331

Bai, X., Hu, Z., Zhu, X., Huang, Q., Chen, Y., Fu, H., Tai, C.L.: Transfusion: Robust lidar332 camera fusion for 3d object detection with transformers. In: CVPR (2022) 333

Brazil, G., Liu, X.: M3d-rpn: Monocular 3d region proposal network for object detection. In: 334 ICCV (2019) 335

Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V .E., Xu, Q., Krishnan, A., Pan, Y ., Baldan, 336 G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. In: CVPR (2020) 337

Cai, Q., Pan, Y., Yao, T., Ngo, C.W., Mei, T.: Objectfusion: Multi-modal 3d object detection 338 with object-centric fusion. In: ICCV (2023) 339

Chen, X., Zhang, T., Wang, Y., Wang, Y., Zhao, H.: Futr3d: A unified sensor fusion framework 340 for 3d detection. In: CVPR (2023) 341

Chen, Y., Liu, S., Shen, X., Jia, J.: Dsgn: Deep stereo geometry network for 3d object detection. 342 In: CVPR (2020) 343

Chen, Y., Yu, Z., Chen, Y., Lan, S., Anandkumar, A., Jia, J., Alvarez, J.M.: Focalformer3d: 344 focusing on hard instance for 3d object detection. In: ICCV (2023) 345

Chen, Z., Li, Z., Zhang, S., Fang, L., Jiang, Q., Zhao, F.: Deformable feature aggregation for 346 dynamic multi-modal 3d object detection. In: ECCV (2022) 347

Chiu, H.k., Prioletti, A., Li, J., Bohg, J.: Probabilistic 3d multi-object tracking for autonomous 348 driving. arxiv 2020. arXiv preprint arXiv:2001.05673 (2020) 349

Contributors, M.: MMDetection3D: OpenMMLab next-generation platform for general 3D 350 object detection. https://github.com/open-mmlab/mmdetection3d (2020) 351

Deng, J., Dong, W., Socher, R., Li, L.J., Li, K., Fei-Fei, L.: Imagenet: A large-scale hierarchical 352 image database. In: CVPR (2009) 353

Deng, J., Shi, S., Li, P., Zhou, W., Zhang, Y., Li, H.: V oxel r-cnn: Towards high performance 354 voxel-based 3d object detection. In: AAAI (2021) 355

Dong, Y., Kang, C., Zhang, J., Zhu, Z., Wang, Y., Yang, X., Su, H., Wei, X., Zhu, J.: Bench356 marking robustness of 3d object detection to common corruptions. In: CVPR (2023) 357

Graham, B., Engelcke, M., Van Der Maaten, L.: 3d semantic segmentation with submanifold 358 sparse convolutional networks. In: CVPR (2018) 359

He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: CVPR (2017) 360

He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR 361 (2016) 362

Hu, J.S., Kuai, T., Waslander, S.L.: Point density-aware voxels for lidar 3d object detection. In: 363 CVPR (2022) 364

Huang, J., Huang, G.: Bevpoolv2: A cutting-edge implementation of bevdet toward deployment. 365 arXiv:2211.17111 (2022) 366

Huang, J., Huang, G., Zhu, Z., Ye, Y., Du, D.: Bevdet: High-performance multi-camera 3d 367 object detection in bird-eye-view. arXiv:2112.11790 (2021) 368

Huang, J., Ye, Y., Liang, Z., Shan, Y., Du, D.: Detecting as labeling: Rethinking lidar-camera 369 fusion in 3d object detection. arXiv arXiv:2311.07152 (2023) 370

Jiao, Y., Jie, Z., Chen, S., Chen, J., Ma, L., Jiang, Y .G.: Msmdfusion: Fusing lidar and camera 371 at multiple scales with multi-depth seeds for 3d object detection. In: CVPR (2023) 372

373

[22]

Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: Fast encoders

for object detection from point clouds. In: CVPR (2019) 374

Lee, Y., Hwang, J.w., Lee, S., Bae, Y., Park, J.: An energy and gpu-computation efficient 375 backbone network for real-time object detection. In: CVPR workshops (2019) 376

Li, B., Ouyang, W., Sheng, L., Zeng, X., Wang, X.: Gs3d: An efficient 3d object detection 377 framework for autonomous driving. In: CVPR (2019) 378

Li, P., Chen, X., Shen, S.: Stereo r-cnn based 3d object detection for autonomous driving. In: 379 CVPR (2019) 380

Li, Y., Chen, Y., Qi, X., Li, Z., Sun, J., Jia, J.: Unifying voxel-based representation with 381 transformer for 3d object detection. In: NeurIPS (2022) 382

Li, Z., Wang, W., Li, H., Xie, E., Sima, C., Lu, T., Qiao, Y., Dai, J.: Bevformer: Learning 383 bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In: 384 ECCV (2022) 385

Liang, T., Xie, H., Yu, K., Xia, Z., Lin, Z., Wang, Y ., Tang, T., Wang, B., Tang, Z.: Bevfusion: 386 A simple and robust lidar-camera fusion framework. In: NeurIPS (2022) 387

Lin, T.Y., Dollár, P., Girshick, R., He, K., Hariharan, B., Belongie, S.: Feature pyramid networks 388 for object detection. In: CVPR (2017) 389

Liu, Y., Wang, L., Liu, M.: Yolostereo3d: A step back to 2d for efficient stereo 3d detection. In: 390 ICRA. IEEE (2021) 391

Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: 392 Hierarchical vision transformer using shifted windows. In: ICCV (2021) 393

Liu, Z., Wu, Z., Tóth, R.: Smoke: Single-stage monocular 3d object detection via keypoint 394 estimation. In: CVPR (2020) 395

Liu, Z., Tang, H., Amini, A., Yang, X., Mao, H., Rus, D.L., Han, S.: Bevfusion: Multi-task 396 multi-sensor fusion with unified bird's-eye view representation. In: ICRA (2023) 397

Liu, Z., Mao, H., Wu, C.Y., Feichtenhofer, C., Darrell, T., Xie, S.: A convnet for the 2020s. In: 398 CVPR (2022) 399

Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint 400 arXiv:1711.05101 (2017) 401

Mao, J., Shi, S., Wang, X., Li, H.: 3d object detection for autonomous driving: A comprehensive 402 survey. IJCV (2023) 403

Pang, S., Morris, D., Radha, H.: Clocs: Camera-lidar object candidates fusion for 3d object 404 detection. In: IROS (2020) 405

Pang, S., Morris, D., Radha, H.: Fast-clocs: Fast camera-lidar object candidates fusion for 3d 406 object detection. In: WACV (2022) 407

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., 408 Gimelshein, N., Antiga, L., et al.: Pytorch: An imperative style, high-performance deep learning 409

410

library. In: NeurIPS (2019)

Qi, C.R., Yi, L., Su, H., Guibas, L.J.: Pointnet++: Deep hierarchical feature learning on point 411 sets in a metric space. In: NeurIPS (2017) 412

Qin, Z., Wang, J., Lu, Y.: Monogrnet: A geometric reasoning network for monocular 3d object 413 localization. In: AAAI (2019) 414

Shi, S., Guo, C., Jiang, L., Wang, Z., Shi, J., Wang, X., Li, H.: Pv-rcnn: Point-voxel feature set 415 abstraction for 3d object detection. In: CVPR (2020) 416

Shi, S., Jiang, L., Deng, J., Wang, Z., Guo, C., Shi, J., Wang, X., Li, H.: Pv-rcnn++: Point-voxel 417 feature set abstraction with local vector representation for 3d object detection. IJCV (2022) 418

Shi, S., Wang, X., Li, H.: Pointrcnn: 3d object proposal generation and detection from point 419 cloud. In: CVPR (2019) 420

Shi, S., Wang, Z., Shi, J., Wang, X., Li, H.: From points to parts: 3d object detection from point 421 cloud with part-aware and part-aggregation network. IEEE TPAMI (2020) 422

Shi, W., Rajkumar, R.: Point-gnn: Graph neural network for 3d object detection in a point cloud. 423 In: CVPR (2020) 424

Shi, X., Ye, Q., Chen, X., Chen, C., Chen, Z., Kim, T.K.: Geometry-based distance decomposi425 tion for monocular 3d object detection. In: ICCV (2021) 426

Sun, J., Chen, L., Xie, Y ., Zhang, S., Jiang, Q., Zhou, X., Bao, H.: Disp r-cnn: Stereo 3d object 427 detection via shape prior guided instance disparity estimation. In: CVPR (2020) 428

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., 429 Polosukhin, I.: Attention is all you need. In: NeurIPS (2017) 430

Vora, S., Lang, A.H., Helou, B., Beijbom, O.: Pointpainting: Sequential fusion for 3d object 431 detection. In: CVPR (2020) 432

Wang, C.Y., Liao, H.Y.M., Wu, Y.H., Chen, P.Y., Hsieh, J.W., Yeh, I.H.: Cspnet: A new 433 backbone that can enhance learning capability of cnn. In: CVPR workshops (2020) 434

Wang, C., Ma, C., Zhu, M., Yang, X.: Pointaugmenting: Cross-modal augmentation for 3d 435 object detection. In: CVPR (2021) 436

Wang, H., Shi, C., Shi, S., Lei, M., Wang, S., He, D., Schiele, B., Wang, L.: Dsvt: Dynamic 437 sparse voxel transformer with rotated sets. In: CVPR (2023) 438

Wang, H., Tang, H., Shi, S., Li, A., Li, Z., Schiele, B., Wang, L.: Unitr: A unified and efficient 439 multi-modal transformer for bird's-eye-view representation. In: ICCV (2023) 440

Wang, T., Zhu, X., Pang, J., Lin, D.: Fcos3d: Fully convolutional one-stage monocular 3d 441 object detection. In: ICCV (2021) 442

Wang, Y., Guizilini, V.C., Zhang, T., Wang, Y., Zhao, H., Solomon, J.: Detr3d: 3d object 443 detection from multi-view images via 3d-to-2d queries. In: Robot Learning (2022) 444

Wu, H., Wen, C., Shi, S., Li, X., Wang, C.: Virtual sparse convolution for multimodal 3d object 445 detection. In: CVPR (2023) 446

Xie, Y., Xu, C., Rakotosaona, M.J., Rim, P., Tombari, F., Keutzer, K., Tomizuka, M., Zhan, W.: 447 Sparsefusion: Fusing multi-modal sparse representations for multi-sensor 3d object detection. 448 In: ICCV (2023) 449

Xu, S., Zhou, D., Fang, J., Yin, J., Bin, Z., Zhang, L.: Fusionpainting: Multimodal fusion with 450 adaptive attention for 3d object detection. In: ITSC (2021) 451

Yan, J., Liu, Y., Sun, J., Jia, F., Li, S., Wang, T., Zhang, X.: Cross modal transformer via 452 coordinates encoding for 3d object dectection. In: ICCV (2023) 453

Yan, Y., Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors (2018) 454

Yang, C., Chen, Y., Tian, H., Tao, C., Zhu, X., Zhang, Z., Huang, G., Li, H., Qiao, Y., Lu, L., 455 et al.: Bevformer v2: Adapting modern image backbones to bird's-eye-view recognition via 456 perspective supervision. In: CVPR (2023) 457

Yang, H., Zhang, S., Huang, D., Wu, X., Zhu, H., He, T., Tang, S., Zhao, H., Qiu, Q., Lin, B., 458 He, X., Ouyang, W.: Unipad: A universal pre-training paradigm for autonomous driving. In: 459 CVPR (2024) 460

Yang, Z., Sun, Y., Liu, S., Shen, X., Jia, J.: Ipod: Intensive point-based object detector for point 461 cloud. arXiv:1812.05276 (2018) 462

Yang, Z., Sun, Y., Liu, S., Shen, X., Jia, J.: Std: Sparse-to-dense 3d object detector for point 463 cloud. In: ICCV (2019) 464

Yang, Z., Chen, J., Miao, Z., Li, W., Zhu, X., Zhang, L.: Deepinteraction: 3d object detection 465 via modality interaction. In: NeurIPS (2022) 466

Yin, J., Shen, J., Chen, R., Li, W., Yang, R., Frossard, P., Wang, W.: Is-fusion: Instance-scene 467 collaborative fusion for multimodal 3d object detection. In: CVPR (2024) 468

Yin, T., Zhou, X., Krahenbuhl, P.: Center-based 3d object detection and tracking. In: CVPR 469 (2021) 470

Yin, T., Zhou, X., Krähenbühl, P.: Multimodal virtual point 3d detection. In: NeurIPS (2021) 471

You, Y., Wang, Y., Chao, W.L., Garg, D., Pleiss, G., Hariharan, B., Campbell, M., Wein472

berger, K.Q.: Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. 473 arXiv:1906.06310 (2019) 474

Zhou, Y., Tuzel, O.: Voxelnet: End-to-end learning for point cloud based 3d object detection. 475 In: CVPR (2018) 476

Zhu, B., Jiang, Z., Zhou, X., Li, Z., Yu, G.: Class-balanced grouping and sampling for point 477 cloud 3d object detection. arXiv preprint arXiv:1908.09492 (2019) 478

Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: Deformable transformers 479 for end-to-end object detection. arXiv preprint arXiv:2010.04159 (2020) 480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope. The claims are clearly stated and are consistent with the theoretical and experimental results presented in the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations of our method, specifically that using an attentionbased approach to interact with the two modalities makes the detection results sensitive to modality loss.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provide detailed theoretical statements and formulas along with their descriptions in the paper.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide a detailed experimental setup in the paper, and the training and testing details are provided in the supplementary material to ensure the reproducibility of our results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in

some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: We release the experimental details in the paper, and the code will be released after the paper is accepted.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide a detailed experimental setup in the paper, and the training and testing details are provided in the supplementary material.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We provide data explanations and statistical methods for obtaining statistical results in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide hardware computer resources for training and testing.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in our paper complies with NeurIPS ethical standards in all aspects.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss that although our method has good performance, practical applications need to ensure personnel safety.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The model of the paper dos not address the issues mentioned in the guidelines. Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We have annotated the cited papers and datasets in our paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

741 742 743 744 745 746 747 748 749 750 751 752 753 754 755 756 757 758 759 760 761 762 763 764 765 766 767 768 769 770 771 772 773 774 775 776 777 778 779 780 781 782 783 784 785 786 787 788 789 790

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

799

800

801

802

803

804

805

806

807

808

809

810

811

812

813

814

815

816

817

818

819

820

821

822

823

824

825

826

827

828

829

830

831

832

833

A Appendix

A.1 Additional Implementation Details

During training, we adopt a one-stage strategy like DAL [20]. The whole pipeline is trained for a total of 20 epochs with the AdamW optimizer [35] loading from the pre-trained weights from the ImageNet [11] classification task only. Meanwhile, we use CBGS [72] to resample the training data, and the one-cycle learning policy with a maximum learning rate of 2 . 0 × 10 -4 . The batch size is set to 8 on 4 3090 RTX GPUs. We adopt random flipping along both X and Y-axis, the random scaling in [0.95, 1.05], and random rotation in [π /8, π /8] to augment the LiDAR data, and the random rotation in [-5.4 ◦ , 5.4 ◦ ] and random resizing in [-0.06, 0.44] to augment the images. During evaluation, we test a single model without any data augmentation on a single 3090 RTX GPU.

A.2 Additional Experiments

A.2.1 3D Multi-Object Tracking Experiments

We evaluate our DH-Fusion on the nuScenes tracking benchmark for 3D multi-object tracking (MOT) task. Following ObjectFusion [4], we adopt the same tracking-by-detection algorithm that uses velocity-based closest point distance matching, which is more effective than 3D Kalman filter [9]. For fair comparisons, we report the results of our DH-Fusion-light capable of real-time detection on the nuScenes validation set, as shown in Tab. 6. We find that our DH-Fusion-light outperforms BEVFusion [33] and ObjectFusion [4] by 2.0 pp and 0.6 pp w.r.t. AMOTA. These results demonstrate that our DH-Fusion provides 3D detection boxes of higher quality, benefiting the downstream task of 3D MOT.

A.2.2 Evaluation at Different Depths

Since our fusion strategy is depth-aware, it is necessary to validate our method at different depths. Following [4], we categorize annotation and prediction ego distances into three groups: Near (020m), Middle (20-30m), and Far (>30m). As shown in Tab. 7, compared to ObjectFusion [4], our DH-Fusion-light consistently improves performance across all depth ranges. Specifically, our method achieves a 47.1 mAP in the long range (>30m), surpassing ObjectFusion by 5.5 pp w.r.t. mAP. These results indicate that our method is more effective across different depths, especially in detecting distant objects.

A.2.3 Detailed Results on the nuScenes-C

We further provide the detailed results of each fine-grained corruption on nuScenes-C in Tab. 8. The results are highly consistent with the average values of each kind of data corruption.

A.3 More Visualization

As an extension of Fig. 6 in the manuscript, we provide additional examples of 3D object detection results and BEV features from our baseline, BEVFusion [33], and our DH-Fusion. In various samples, our method consistently achieves higher accuracy and recall in 3D detection results, with

stronger feature responses for distant objects compared to BEVFusion. These results demonstrate the 834 effectiveness of the proposed method in dynamically adjusting the weights of features based on depth 835 during fusion at both global and local levels. 836

BEVFusion

Front Left

Front

Front Right

BEVFeature

Back Left

Back

Back Right

DH-Fusion (Ours)

Front Left

Front

Front Right

BEV Feature

BackLeft

Back

Back Right

BEVFusion

Front Left

Front

Eront

Righ

Point

BEV

Cloud

Feature

Back

Left

Back

Back Right

DH-Fusion (Ours)

FrontLeft

Front

Point

BEV (

Cloud

Feature

Back

Left

Back

Back Right

BEVFusion

Fronit

Front

FrontRight

BEVFeature

Back Left

Back

DH-Fusion (Ours)

Frontl

Front

Front Right

BEVFeature

Back Left

Back

BEVFusion

FrontLeft

Front Right

Point

BEV

Cloud

Feature

Back

eft

DH-Fusion (Ours)

FrontLeft

Front

Front Right

Point

BEV

Cloud

Feature:

Back

eft

Back Ri

ht

Peer reviews:
comments: SUMMARY:
This study reveals that modalities have varying impacts depending on depth, leading to the proposal of DH-Fusion. This method
dynamically adjusts feature weights using depth encoding, improving multi-modal 3D object detection. Results on nuScenes show DHFusion outperforms prior methods.

STRENGTHS:
1. This paper is well-presented. The structure is clear and easy to follow.
2. Comprehensive experiments on the nuScenes dataset are conducted to validate the effectiveness of the proposed DH-Fusion.

WEAKNESSES:
1. Lake of Novelty: The Depth Encoder in DH-Fusion is similar to the 3D Position Encoders in PETR (PETR: Position embedding transformation for multi-view 3d object detection). The Depth-Aware Global Feature Fusion (DGF) module and Depth-Aware Local Feature Fusion (DLF) module in DH-Fusion are analog to the Hierarchical Scene Fusion (HSF) module and Instance-Guided Fusion (IGF) module in IS-Fusion (IS-Fusion: Instance-scene collaborative fusion for multimodal 3d object detection). In conclusion, the contribution of this work seems like "A+B," which is limited.
2. For the nuScenes test leaderboard, DH-Fusion achieved a Top 10 ranking only with 384x1056 image size and SwinTiny backbone.
Please provide results when using larger 900x1600 image size and ConvNeXtS backbone.

QUESTIONS:
1.Please analyze the theoretical reasons for DH-Fusion's robustness advantage against various corruptions, as in nuScenes-C.
2. In Table 1, for experiments on the nuScenes dataset, It's necessary to include metrics like mATE, mASE, mAOE, mAVE, and mAAE as
done in other papers
3.Do the authors plan to release the code and provide pre-trained, especially nuscenes test leaderboard models for further research?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a LiDAR-camera modality feature fusion method based on depth encoding for robust 3D object detection. Based on the observation that the LiDAR and camera modality information should have dynamic relative importance depending on the distance of object to be detected, the paper proposed a Depth-Aware Hybrid Feature Fusion (DH-Fusion) strategy which consists of a Depth-Aware Global Feature Fusion (DGF) module and a Depth-Aware Local Feature Fusion (DLF) module. Experiment on the public nuScenes and nuScenes-C dataset demonstrates that the proposed method is robust to various kinds of corruptions and achieves SOTA performance on 3D object detction.\n\nSTRENGTHS:\n1. The idea of depth-aware multimodality feature fusion for 3D object detection is reasonable, especially for the detection of distant objects.\n2. The ablation study clearly demonstrates the effectiveness of the proposed DGF&DLF module when using BEVFusion as baseline\n3. The presentation is clear and the ability of the proposed method on the detection of distant object in Figure 6 is impressive\n\nWEAKNESSES:\n1. How about the algorithm's performance on small object detection? small object could be normal-sized object at far distance or small-sized object in near distance, is it possible that the proposed depth-aware module hurts the detection performance of small-sized object in near distance? since according to Figure 5, LiDAR modality will have relatively larger weights at near distance, but it is in low resolution, so not good for small object detection.\n2. Compare with SOTA, the achieved performance improvement is not that significant. as shown in table 1, the performance gap between the proposed method and IS-Fusion is small and IS-Fusion even achieves slightly better mAP, it is not clear whether the proposed method can achieve similar performance improvement as indicated in ablation study when using IS-Fusion as baseline.\n3. In Figure 5(b), it would be good to add a color bar to indicate the magnitude corresponding to each color\n\nQUESTIONS:\nThe main concern is on the experiment verification of the proposed method, as listed in the weakness part. I may adjust my rating if such concerns are well addressed\n\nLIMITATIONS:\nThere is no paragraph explaining the weakness of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel strategy for LiDAR-camera 3D object detection that emphasizes the importance of depth information in feature fusion processes. The authors argue that different modalities, such as LiDAR point clouds and RGB images, contribute variably at different depths, and this variation has been overlooked in previous works. The key contribution is the Depth-Aware Hybrid Feature Fusion (DH-Fusion) strategy that dynamically adjusts the weights of point cloud and image features based on depth encoding at both global and local levels. The DH-Fusion method surpasses previous state-of-the-art methods in terms of NDS on the nuScenes dataset and demonstrates robustness to various data corruptions. In general, the design is reasonable and performance is impressive.\n\nSTRENGTHS:\n1.\tThe paper is well-structured, with a clear abstract, introduction, methodology, experiments, and conclusion sections that logically flow from one to the next.\n2.\tThe authors effectively communicate complex ideas through clear language and comprehensive illustrations, aiding the reader's understanding of the proposed method.\n3.\tThe motivation of design is clear and experiments are extensive.\n4.\tThe idea of depth encoding for dynamical fusion is interesting and reasonable.\n5.\tThe performance is very impressive and the robustness makes the method more applicable to challenging scene.\n\nWEAKNESSES:\nThe paper has no obvious weakness except they didn't do experiments on other datasets.\nBut I think the nuScenes is already large enough to demonstrate the general effectiveness.\n\nQUESTIONS:\nI noticed this paper encodes depth information using cosine functions, but I haven't seen experiments validating the impact of cosine functions. Would there be a significant performance drop if distances were used directly instead of cosine functions?\n\nLIMITATIONS:\nThere is no discussion of limitation in main text, but a justification is given in Checklist: using an attention-based approach to interact with the two modalities makes the detection results sensitive to modality loss.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces DH-Fusion, a novel Depth-Aware Hybrid Feature Fusion strategy for multimodal 3D object detection that leverages LiDAR and camera data. The key innovation lies in dynamically adjusting the weights of point cloud and RGB image features based on depth encoding at both global and local levels. The authors propose two modules: Depth-Aware Global Feature Fusion (DGF) and Depth-Aware Local Feature Fusion (DLF), which enhance feature integration and compensate for information loss during the transformation to Bird's-Eye-View (BEV) space. Experiments on the nuScenes dataset demonstrate that DH-Fusion surpasses state-of-the-art methods in terms of Novelty Detection Score (NDS) and is more robust to data corruptions, as evidenced by superior performance on the nuScenes-C dataset.\n\nSTRENGTHS:\n1. The paper proposes a novel feature fusion strategy that adaptively adjusts the weights of LiDAR point cloud and RGB image features based on depth\n2. The introduction of depth encoding at both global and local levels allows for more nuanced and context-aware feature integration, enhancing the detector's ability to understand the scene's depth structure.\n\nWEAKNESSES:\n1. The authors only present results on nuScenes dataset. The alogrithms should be also evaluated on other prevailing public dataset like KITTI.\n2. The depth-aware fusion might be tailored to the specific characteristics of the training dataset, potentially leading to overfitting and reduced performance on diverse or unseen data.\n3. While the paper includes ablation studies, a more extensive set of experiments that isolate the impact of different components of the system could provide deeper insights.\n\nQUESTIONS:\nWhat is the computational complexity of the DH-Fusion model, and how does it compare with other state-of-the-art methods in terms of runtime and resource usage?\n\nLIMITATIONS:\n1.  While the method shows strong performance on the nuScenes dataset, its generalizability to other datasets or varied real-world conditions might require further investigation.\n2. The paper does not provide a detailed discussion on the computational efficiency, which is crucial for practical applications, especially in terms of processing time and resource usage.\n3 .The method assumes high-quality, synchronized data from LiDAR and camera sensors, which might not always be guaranteed in real-world scenarios.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}