PathFusion: Path-consistent Lidar-Camera Deep Feature Fusion
Authors: Lemeng Wu
Dilin Wang
Meng Li
Yunyang Xiong
Raghuraman Krishnamoorthi
qiang liu
Vikas Chandra
Venue: Submitted to ICLR 2023 (2023)

Abstract: Fusing camera with LiDAR is a promising technique to improve the accuracy of 3D detection due to its complementary physical properties.
While most existing methods focus on fusing camera features directly with raw LiDAR point clouds or shallow 3D features, it is observed that direct deep 3D feature fusion achieves inferior accuracy due to feature mis-alignment. The mis-alignment that originates from the feature aggregation across large receptive fields becomes increasingly severe for deep network stages. In this paper, we propose PathFusion to enable path-consistent LiDAR-camera deep feature fusion. PathFusion introduces a path consistency loss between shallow and deep features, which encourages the 2D backbone and its fusion path to transform 2D features in a way that is semantically aligned with the transform of the 3D backbone. We apply PathFusion to the prior-art fusion baseline, Focals Conv, and observe more than 1.2% mAP improvements on the nuScenes test split consistently with and without testing-time augmentations. Moreover, PathFusion also improves KITTI AP 3D (R11) by more than 0.6% on moderate level.

Under review as a conference paper at ICLR 2023
PATHFUSION: PATH-CONSISTENT LIDAR-CAMERA DEEP FEATURE FUSION
Anonymous authors Paper under double-blind review
ABSTRACT
Fusing camera with LiDAR is promising technique improve the accuracy of 3D detection due to the complementary physical properties. While most existing methods focus on fusing camera features directly with raw LiDAR point clouds or shallow 3D features, it is observed that direct deep 3D feature fusion achieves inferior accuracy due feature mis- alignment. The mis- aligmmetn that originates from the feature aggregation across large receptive fields becomes increasingly severe for deep network stages. In this paper, we propose PathFusion to enable path-consistent LiDAR-camest deep feature fusion. PathFusion introduces path consistency loss between shallow and deep features, which encourages the 2D backbone and its fusion path to transform 2D features in way that is semantically aligned with the transform of the 3D backbone. We apply PathFusion to the prior- art fusion baseline, Focals Conv, and observe more than 1.2% mAP improvements on the nuScenes test split consistently with and without resting-time augmentations. Moreover, PathFusion also improves KITTI AP3D (R11) by more than 0.6% on moderate level.
INTRODUCTION
LiDARs and cameras are widely used in autonomous driving to provide complementary information for 3D detection (e.g., Caesar et al. 2020, Sun et al.] 20200] Geiger et al. 2013). LiDAR point clouds capture better geometry information but suffer from low resolution due the power and hardware limitations. By contrast, cameras capture dense and colored images with more semantic information but usually lack the shape and depth information for geometry reasoning. As a result, recent methods <i al. 2022c Vora et al. 2020 Liang et al., 2022 Li ct al. 2022b) propose to fuse 2D and 3D features from LiDAR point clouds and camera images to enable accurate and robust 3D detection.
LiDAR and camera fusion requires to project the features into the same space. Previous works have proposed to either lift 2D camera features into 3D space (e.g.. Vora et al. 2020 Wang et al. 2021a
Li all 2022b) Chen et al. 2022c a or to align both 2D and 3D features in a common representation
space like bird-eye-view (BEV) (e.g., et al. 2022c Liang al. 2022
One important question for feature fusion is to select the correct fusion stage and we illustrate popular choices in Figure M Shallow fusion in Figure ! (a) (Chen al.) 2022a Vora al. 2020 Chen et
2022d) fuses camera features with raw LiDAR points directly or shallow LiDAR features. Although shallow fusion benefits from the LiDAR and camera calibration for better feature alignments, the camera features are forced to go through multiple modules which are specialized for 3D feature extractions instead 2D camera processing (Li al. 2022b). e.g., voxelization. By contrast, deep fusion (Xu et al. 2021b Li et al. 2022b in Figure I (b) enables more dedicated LiDAR and camera feature processing. However, since multiple LiDAR points are usually voxelized together and further aggregated with neighboring voxels before fusion,
voxel may correspond to number of camera features, making the feature alignment very ambiguous. The feature mis-alignmmnt significantly degradates the overall network accuracy and forces the majority of existing works to choose shallow fusion (Chen et al. 2022a).
In this paper, we propose PathFusion to augment existing shallow fusion methods and enable fusing deep LiDAR and camera features. PathFusion introduces a novel path consistency loss to regularize the 2D feature extraction and projection and explicitly encourages the alignments between LiDAR

---

Under review as a conference paper at ICLR 2023
Shallow (2D) Shallow (3D) Fusion
Deep (2D) Deep (3D) Fusion
(c) Shalllow
Deep Fusion with regulartzation (ours)
Shallow
Shallow
Detector
Shallow
Deep
Layers
Layers
Layers
Layers
Layers
Layers
LIDAR Points (3D)
LIDAR Poinss (3D)
|LIDAR Foints (3D)
Fuxion
Fusinn
Faxion
Shallow
Shallow
Deep
Shallow
Deep
Layers
Layers
Layers
regularization
Emages
Images
Emagges
(2D)
(2D)
Figure I: Overview of three different strategies to fuse the camera and LiDAR features: (a) shallow fusion accurately fuses the 2D feature to the shallow 3D feature: (b) deep fusion projects the 2D features to the 3D feature space; (c) our method proposes path consistency loss to mitigate the feature mis-alignment problem and enables augmentation of shallow fusion methods with deep fusion for better accuracy.
and camera deep features. As shown in Figure (c), we construct different paths between 2D and 3D features and minimize the semantic differences of features from different paths enforce feature alignments. PathFusion can be easily applied to prior-art fusion methods, c.g., Focals Conv (Chen al., 2022a), and demonstrates consistent accuracy improvements. Compared to existing piror-of- the-art shallow fusion method, FocalConv-F, PathFusion achieves 0.62% AP3D (R11) improvement on KITTI and 1.2% mAP and 0.7% NDS improvement on nuScenes test set.
The rest of the paper is organized as follows. Section 2 introduces related work and section 3 describes the background of 2D and 3D feature fusion. Section 4 provides motivation example on the challenge of deep feature fusion and we present our method in section 5, Section 6 summarizes our results.
RELATED WORK
3D Object Detection with LiDAR or Camera 3D object detection targets at predicting 3D bounding box and can be conducted based on 3D point clouds from LiDARs 2D images from
cameras. LiDAR- methods mainly encode the raw LiDAR point clouds (Qi et al. 2017a b). or the voxelized into sparse voxel (Zhou & Tuzel 2018 process the input source multi-resolution features. Based on those 3D features, various single-stage, two-stage 3D detection heads (Shi et al.
2020 2019 Deng al., 2021 [Bhattccharyaa & Czarnecki 2020 Yan al. 2018 Yang et al. 2020
He et al. 2020 heng et al. 2021 Lang et al. 2019; Yin et al. 202 la Yan el al. 2018 Shi et al.
2021
Wang et 2020) are proposed predict the 3D bounding boxes target objects.
Another series of works focus on camera-based method which encodes single-view or multi-view
images with 2D backbones such as ResNet (He al.) 2016). Due lack of depth information.
2D>tt-3D detection heads are devised to enhance 21) features with implicitly or explicitly predicted
depth to generate the 3D bounding box (Liu al. 2022a b Wang al. 2021e
Huang al. 202T
Reading et al. 2021 Xie al. 2022
Huang & Huang 2022 Li et al. 2022c Wang et al. 2021b)
LiDAR-camera Fusion
Because of the complementary properties of LiDARs and cameras, recent methods propose jointly optimizing both modalities and achieving superior accuracy compared to
LiDAR- camera-only methods. As in Figure (a) and (b), these methods can be largely divided
into two categories depending on the fusion stages: (a) shallow fusion decorates the point clouds or
shallow LiDAR features with image features to enrich the LiDAR inputs with the image semantic
prior (Vora et al. 2020 Yin et al., 20216 Xu et al. 2021a Chen et al. 2022c a. Wu et al. 2022 Li
al. 2022a Chen et al.] 2022a Liang et al. 8a); (b) deep fusion lifts image features into
space and combines them in the middle deep stages of the backbone (Liang et al. 2018b Huang
ct al. 2020).
Shallow fusion methods, e.g., Focals Conv (Chen et al. 2022al
LargeKerneI3D (Chen et al. 2022b
have achieved state-of oftte-of-hh-arr accuracy while deep fusion models suffer from the increasingly severe mis- mis-alignment between camera and LiDAR features. Recently, Transfusion (Bai et al. 2022 and DeepFusion (Li et al. 2022b) propose to align the LiDAR and camera features with transformer and leverages cross-attention to dynamically capture the correlations between image and LiDAR features
2

---

Under review as a conference paper at ICLR 2023
3D Branch
Stage
Stage
Stage 3
Stage
Detector
LiDAR Points
Pi
P2
P3
P4 Lifting operator
2D Branch
Stage
Stage
Stage
Stage 4
Camera Images
Figure 2: A generic 3D detection network with 2D feature fusion at different stages.
during fusion. However, the computation cost of attention scales linearly with the product of the number of voxels and the number of pixels, which significantly increases the model inference latency. For example, the attention module in Deepfusion (Li et al. 2022b) contributes to more than 30% latency increase.
Path and Cycle consistency To encourage the feature alignments in our work, we introduce a regularization term during training based on path consistency. Path and cycle consistency are common ways validate whether samples in graph
match after passing paths cycles. In deep
learning community, they are used in image matching (Zhou el al., 2015a 2016 2015b Dwibedi al. 2019), domain transfer and adaptation (Zhu al. 2017 Hoffman al. 2018 Yi al. 2017) and regularizing the better feature quality in various of general tasks, including segmentation and detection (Zhagg et al. 2019 Wang et al. 2013 2014). For the first time, our work introduces path consistency into LiDAR and camera feature extraction and fusion to enable accurate 3D object detection.
BACKGROUND: 2D AND 3D FEATURE FUSION
The goal of Lidar-Camen feature fusion to improve 3D detection tasks with additional dense 2D semantic information captured by cameras. A generic 3D detection network with multi-stage 2D feature fusion shown Figure 2 The network mainly consists three components: 3D feature extraction branch, 2D feature extraction branch, and 2D-to-3D feature lifting.
3D Feature Extraction Given a point cloud with a collection of points, i.e., {p:) (p: E RR3), detect objects of interest from the point cloud, typical pre-processing step is first apply voxelization on the points and transform the point cloud representation to a voxel volume representation. Denote 3/u E Hxxx the transformed voxel volume, with H, W, D representing the height, width, and depth in the 3D space, respectively. The 3D detection network is often built with stack of 3D feature backbone (e.g. Qi et al. 2017b Zhou & Tuzel 2018 Mao et al. 2021b) and a detector head (e.g. |ang et al.) 2019; Shi et al. 2020 Yin et al. 2021a Deng et al. 2021
with yo as the input. The 3D backbone often multiple stages with down-sampling between each stage. Denote E C x xD, the feature maps produced by stage where C₁ represents the
channel size and H4, W,, D, denote the corresponding height, width and depth, respectively, after down-sampling.
2D Feature Extraction Assume an RGB image, To 3x xW" is captured along with the point cloud, with H' x W' as the image resolution. Similarly, with modern feature backbone (c.g., ResNet50), one can extract 2D features for To at different stages. Denote 2; E R extracted 2D feature stage i, where C/ is the channel size and H W the feature map size. The line of 2D and 3D feature fusion work aims at lifting 2D features (æ;) to the 3D feature space and fusing the lifted features with 3D features {y:} to improve 3D detection tasks.
11x xW; the
2D-to-3D Feature Lifting One immediate challenge is that 2D and 3D feature maps have different spatial resolutions and relate to different locations in the real-world 3D space. Without loss of generality, consider fusing 2D feature map v, from the stage in the 2D branch with 3D feature
3

---

Under review as a conference paper at ICLR 2023
CxH'xW'
Cx 1X
H2 x W/
Fusion
Upsampling
Upsampled 2D feature at pixel-level resolution
2D feature from stage
Fused 3D feature
Figure 3: An illustration of lifting features from 2D to 3D at a deep stage. The upsampling typically implemented with feature pyramid network (Lin et al. 2017). Image is from (Geiger al.
2013
map y/ from the stage in the 3D branch. Existing works (e.g., Chen et al. 2022a: Li et al. 2022b
Chen et al. 2022b) often first upsample x; to the input pixel-level resolution !f x using a feature pyramid network (Lin et al. 2017). Then for each voxel in the 3D volume 1/s,
use the camera
parameters and query the corresponding 2D pixel-level feature for each voxel (e.g., Vora et al. 2020
al 20226 Bai al. 2022 Chen et al. 2022a) See Figure f demonstration. Note that each voxel might correspond to a region of a different size. In this work, we follow (Chen et al. (022a) and take the feature vector the center as the representation of the patch for simplicity. Finally, the fused 3D features could be a summation or concatenation between the lifted 2D features and original 3D features from the 3D backbone.
MOTIVATING EXAMPLE: CHALLENGES OF DEEP FUSION
Feature fusion at the input level is intuitive and straightforward, each point from LiDAR can accurately locate its corresponding 2D pixels using the camera parameters. However, the accurate mapping between 2D and 3D is lost due feature aggregation in both 2D and 3D backbone. Especially for deep layers, each voxel relates to larger respective field, which has pixels that are far away from each other the 3D space (e.g., car and background trees Figure 3 Ideally, one would like each voxel and its corresponding 2D feature patch to encode the same part of the physical world. However, there is no guarantee of such correlation in the network since 2D and 3D features are both involved with non-linear transformations and down-sampling. Naively assigning each voxel with 2D features that are semantically unrelated might cause confusion of the 3D backbone and consequently degrade the 3D detection performance.
We verify our hypothesis in Figure (a). We closely follow the setting in prior-art Focals Conv (Chen al. 2022a), and alternate feature fusion from stage to stage 4 as shown in Figure 2 In this case, we use VoxelNet (Zhou & Tuzel 2018) with Focals Conv as the 3D backbone and ResNer-50 (He al. 2016 as the 21) backbone. Both the 2D and 3D backbones have four stages. We report AP (R11) score on KITTI (Geiger et al. 2013). As shown in Figure (a), the model trained with shallow feature fusion at stage outperforms the baseline with fusion, demonstrating the benefits of camera features for 3D detection. However, the performance drops consistently when fusing features deeper layers due to potential semantic mis-alignment between 2D and 3D features. We propose novel training regularization to improve deep feature fusion in the sequel.
5 METHOD: DEEP FUSION WITH PATH CONSISTENCY REGULARIZATION
We denote V/ as an ideal semantic alignment between 2D features and 3D features yı, in way that there exists a lifting operator P., such that the lifted 2D features ÿ = P; o 2; and y, are semantically aligned. We denote T Rs ys and y semantically aligned, in sense that and y: has a high similarity in the feature space.
For feature fusion between camera and LiDAR, 2g Yo holds at the input layer where the mapping from 2D to 3D is exact, and P. can be constructed accurately with camera parameters. Assume It Ye hold for features at stage namely, there exists P, such that i Pt 2, ss yr. It remains to show 2++1 ~ Bt+1 so that we can conclude features at all stages are semantically aligned from a mathematical induction point of view.
4

---

Under review as a conference paper at ICLR 2023
85.5
85.22
Y/ y,
Fi+1
y4+1
w/o fusion
APso R11 score
85.0
84.85
Stage 1+1
Loss
84.5
Branch
Pe
P
Lifting operator
84.0
f+1
83.49
83.5
83.11
X
Stage
X,+1
83.0
Stage Stage Stage Stage 4
2D Brunch
(a)
(b)
Figure 4: (a) Illustration of performance degradation with naive deep feature fusion. Results are the KITTI val split. The baseline setup without feature fusion achieves a 84.93% of AP3D(R11) (b) Illustration of out path-consistent loss.
In the following, we first show how Ia and Ye is transformed in the 2D branch and 3D branch, respectively. Then we present a path-consistent loss that encourages the network to learn semantically aligned 2D and 3D features, that is 2441 V1+1 (or Pi+1 2(+1 (t+1).
2D path On the 2D branch, 2, is first evolved with stack of 2D convolutions layers (denoted as f(+1) and then lifted by Pe+1. More precisely, the resulting 3D features can be written as following.
yon = Pr+1 o 3t+11 = Pr+1 o fe+1 o o
where + represents the output from ft+1. Here we denote A o as applying operator A on the output from B. See the red path in Figure 4 (b) for illustration.
3D path The 3D counterpart ye simply transformed by number of 3D convolution layers (we denoted as F(t1).
y3D Fi+1 y Fi+1 or Fi+1 P. o 24. Here we replace yt ÿt ot by construction. See the blue path in Figure (b) for an example.
(2)
Path-consistent regularization Our path-consistent regularization is simply defined minimizing the distance between the lifted features YED and the 3D features yaD such that 2141 ~ M4+1 as follows,
L consistency loss(y22. VaD) loss loss(P(41 fi+1 oz,
path
path
where loss(-) denotes the loss function.
the consistency regularization to minimize loss(Pt+1 o fi+1 It; Fi+1 yt). However, in practice, the 3D voxel feature y: is often much sparser compared to the 2D feature 2,, This loss is only defined where y/ has values. Meanwhile, Fi-1 o yt represents geometry based features and P.-1 Fr+ represents color- features, it is non-trivial to design an appropriate loss function to measure the distance before these two. While our consistency loss only depends on the same input zz. the information from both paths is dense. Meanwhile, simple loss function like negative cosine similarity or & distance is sufficient since features are originated from the same color feature space.
An alternative implementation
Path consistency for multi-stage networks In case of mutli-stage feature fusion wherein features are fused at both shallow and layers. Assume there are in total n stages, we apply the path consistency defined in Eqn. for all stages, that is,
Coonsistency
Coonsistency
Σ
loss loss(Prtz f4t o Bi,
F Pora)
2D path
3D path
We refer Eqn 4
path consistency loss.
5

---

Under review as a conference paper at ICLR 2023
Algorithm We augment the standard 3D detection loss with our path consistency loss. Overall, training objective can be formulated as follows,
L3D + conniistency
(5)
Here LaD is the main loss for the 3D task that affects all the parameters, Leonsistency path- consistent regularization, and a is hyper-parameter that controls the magnitude of the regularization. As our goal is to regularize the 2D) branch to improve the 3D detection task, only back-propagate our path consistency loss through the lifting operators and the 2D backbone while stopping the corresponding gradients on the 3D backbone.
EXPERIMENTS
We evaluate method, PathFusion, both KITTI (eegger et al, 2013 and nuScenes (Caesar et al. 2020). We choose Focals Conv (Chen et al.] 2022a) as our base model. We add our path path-consistent loss top of Focals Conv training to facilitate feature fusion between the 2D and 3D branches. In alignment with Focals Conv, we found that naively fusing LiDAR and camera features at deeper layers leads significant performance degradation (see e.g.. Figure 4 (a)). However, our path- consistent regularization learns semantically aligned transforms from 2D to 3D, yielding consistent improvements more deep features are fused. Specifically, comparing to Focals Conv with the same detector VoxeI-RCNN, we improved the AP (R11) from 85.22% to 85.84% on KITTI and improved mAP from 70.1% 71.3% on nuScenes testing split, respectively.
PathFusion settings Focals Conv fuses stage features from the 2D branch. We denote this implementation as Focals Conv-F. On top of the Focals Conv-F setting, we add deep-level feature do not apply the path
fusion with our path-consistent regularization. As defined in Eqn. consistency loss on stage 1, as the misalignment issue is only significant at deeper layers.
We choose cosine as our consistency loss and specified. Additionally, our consistency regularization only affects parameters stage-wise. For example, our consistency loss computed at the end of stage i only back-propagates to parameters in stage and the corresponding lifting operator P..
0.01 in Eqn. 5 default unless otherwise
6.1 RESULTS ON KITTI
Dataset We follow the setting KITTI (Geiger al. 2013) by splitting the dataset into 3,717 for training and 3,769 for validation. Each KITTI example contains one LiDAR input together with two camera inputs. We only use the LiDAR input and RGB from the left camera for fair comparison with Focals Conv (Chen et al. 2022a).
Settings Following the setup in Focals Conv (Chen et al., 2022a), we use Voxel RCNN Deng et al. 2021) as the overall detection framework, in which
VoxelNet (hou & Tuzel 2018) with
Focals Conv as the 3D backbone and ResNet-50 (He et al. 2016) as the 2D backbone. Following
Focals Cony, the ResNet-50 is pretrained on COCO (Lin al. 2014) using Deeplabv3 (Chen et al.
2017). To enable deep-level fusion, we introduce feature fusion in stage and keep the original shallow-level fusion in stage 1.
We strictly follow the training setting used in Focals Conv-F and train the model for 80 epochs with batch size of 16, and Precision (mAP) metric. The mAP calculates the 3D bounding box recall score on 11 positions. The metric referred as (R!1).
Adam optimizer with learning rate 0.01. We report the mean Average
Results We summarize results in Table I We report the AP R (R11) score evaluated in easy, moderate and hard levels following the previous works (Chen et al. 2022a Deng et al. 2021 Shi from Table I our method outperforms Focals Conv- F by 0.41 %, 0.62%, 0.12% h in three levels and achieves 90. 23, 85.84% and 85.31% on (R1 1). These results suggest that our proposed Path Fusion learns better feature fusion and thus provides consistent improvement.
et al. 2020). As

---

Under review as a conference paper at ICLR 2023
Method
Modality
Easy
Moderate
Hard
PointPillars Lanng et al.
2019)
86.62
76.06
68.91
SECOND Yan et al. 20
88.61
78.62
77.22
Point R-CNN (Shi et al.
88.88
78.63
77.38
Part-A² Shi al.; 202
89.47
79.47
78.54
3DSSD Yang et al. 2020
89.71
79.45
78.67
STD Yang et al.] 2019)
89.70
79.80
79.30
SA-SSD He et al., 2020)
L
90.15
79.91
78.78
PV-RCNN (Shi et al. (2200)
89.35
83.69
78.70
VoTr-TSD (Mao et al. [202 bb
L
89.04
84.04
78.68
Pyramid-PV (Ma et al. (02
89.37
84.38
78.84
Voxel R-CNN (Degg et al..
2021
I.
89.41
84.54
78.93
Focals Conv Chen et al. 2022a)
89.52
84.93
79.18
LargeKemel3D Chen et al. 2022b
L
89.52
85.07
79.32
F-PonttNet Qi al. 2018)
83.76
70.92
63.65
Neit (Zhao et al.
2019
I.C
85.62
72.05
64.19
3D-CVF (Yoo et al.) 2020))
LC
89.67
79.88
78.47
Focals Conv-F (Chen et al. 2022a)
LC
89.82
85.22
85.19
PathFusion (ours)
LC
90.23
85.84
85.31
Table 1:
Comparison on KITTI Results are APıd (R11) for Car on the val split. L represents LiDAR input and LC represents both LiDAR and camera input. Focals Conv-F denotes the implementation with shallow fusion on stage in Figure
6.2 RESULTS NUSCENES
Dataset nuScenes Caesar et al. 2020 is large- 3D object detection dataset for self-driving. consists of 700 scenes for training 150 scenes for validation and 150 scenes for testing. Each scene has sequence of frames and each frame contains data from LiDAR and 6 cameras. In total, nuScenes contains 1.4M camera images and 390K LiDAR sweeps.
Settings Following Focals Conv-F (Chen et al. 2022a). and use ResNet-50) pretrained using Deeplab (Chen et al. 2017) on COCO (Lin et al., 2014) as the 2D backbone. We train the model for 20 epochs with batch size 32 and Adam Optimizer with 0.001 learning rate. We apply cosine scheduler to decrease the learning rate to le le-4, For the data augmentation, same as Focals Conv-F, we apply rotation, flip, translate, rescale and ground truth sampling augmentation during training. We extend the Focals Conv-F to fuse 2D and 3D features at both stage and stage (see Figure
VoxeINet as 3D backbone
Results We evaluate on both the nuScenes validation and test set, and report our results in Table and Table respectively. On the validation split, we achieve 1.5% improvement mAP and 0.7% improvement on NDS (NuScenes Detection Score), respectively, compared to Focals Conv-F.
We further evaluate our model the nuScenes test server. Following Liu al. (022c): Liang et al. (202)); [Cen et al.] (022a),, we report results with and without testing time data augmentation. For the testing-time augmentations, we follow Chen al. (022a) and use double flip and rotation with yaw angles in |-6. 6.. 0%, 6.25%]. We observe consistent improvements on Focals Conv across all the settings by regularizing the training with our path-consistent loss. Specifically, our best result outperforms Focals Conv-F by 1.2%, 1.3%, 1.2% mAP and 0.8%, 0.8%, 0.7% NDS on the test set for the settings without and with testing-time data augmentation.
6.3 ABLATION STUDIES
Choices of fusion configurations To study the importance of feature fusion deep layers, alternate the feature fusion at different stages.
https//wwww.msseeess.rrg

---

Under review as a conference paper at ICLR 2023
Method
mAP
NDS
Car
Truck
Bus
Trailer
Ped
Mot
Byc
T.C.
Focals Conv-F
63.8
69.4
86.5
58.5
72.4
41.2
23.9
86.0
69.0
55.2
76.8
69.1
PathFusion (ours)
63.3
70.1
86.8
61.4
72.1
42.3
26.6
87.0
75.2
61.0
77.5
66.2
Table Comparison with other methods on nuScenes val split without testing-tinne time augmentations.
Method
Modality
mAP
NDS
Car
Truck
Bus
Truiler
C.V.
Ped
Mot
Byc
T.C.
Bar
PointPillars (Lang 2019
L
30,5
45.3
68.4
23.0
28.2
23.4
4.1
59.7
27.4
1.1
30.8
38,9
3DSSD Yang 2020
42.6
56.4
81.2
47.2
61.4
30.5
12.6
70.2
36.0
8.6
31.1
47,9
CBGS Zhu 2019
52.8
63.3
81.1
48.5
54.9
42.9
10.5
80.1
51.5
22.3
70,9
65.7
HotSpol
( Chen 20206
1.
59.3
66.0
83.1
50.9
56.4
53.3
23.0
81.3
63.5
36.6
73.0
71.6
CVCNET Chen
2020a
L
58.2
66.6
82.6
49.5
59.4
51.1
16.2
83.0
61.8
38.8
69.7
69.7
CenterPoint Yim et
58.0
65.5
84.6
51.0
60.2
53.2
17.5
83,4
53.7
28.7
76.7
70.9
CenterPoint
I. 60.3
67.3
85.2
53.5
63.6
56.0
20.0
84.6
59.5
30.7
78.4
71.1
Focals Conv Chen et al.
2022a
I. 63.8
70.0
86.7
56.3
67.7
59.5
23.8
87.5
64.5
36.3
81.4
74.1
SargeKerel331 Chen
2022h
70.5 65.3
85.9
55.3
66.2
60.2
26.8
85.6
72.5
46.6
80.0
74.3
PointPainting Vora et al.
2020
L.C
46.4
58.1
77.9
35.8
36.2
37.3
15..8
73.3
41.5
24.1
62.4
60.2
3DCVF Yoo al. 2020
LC
52.7
62.3
83.0
45.0
48.8
49.6
15.9
74.2
51.2
30.4
62.9
65.9
FusionPantiing Xu ct al
2021b
LC
66.3
70.4
86.3
58.5
66.8
59.4
27.7
87.5
71.2
51.7
84.2
70.2
MVF Yin et al.
2021e
LC
66.4
70.5
86.8
58.5
67.4
57.3
26.1
89.1
70.0
49.3
85.0
74.8
PointAuugeettng (Wang et
2021a
LC
66.8
71.0
87.5
57.3
65.2
60.7
28.0
87.9
74.3
50.9
83.6
72.6
Focals Conv-F Chen et al.
2022a
I.C
67.8
71.8
86.5
57.5
68.7
60.6
31.2
87.3
76.4
52.5
84.6
72.3
PathFusion (ours)
I.C
69.0
72.6
87.5
59.8
69.3
62.0
34.7
87.6
77.3
53.7
85.3
72.9
Foculs Conv-Fi
LC
68.9
72.8
86.9
59.3
68.7
62.5
32.8
87.8
78.5
53.9
85.5
72.8
PothFusion (ours)
LC
70.2
73.6
88.0
61.4
69.4
64.1
34.9
88.5
80,2
54.8
85.9
73.2
Focals Con-FF
LC
70.1
73.6
87.5
60.0
69.9
64.0
32.6
89.0
81.1
59.2
85.5
71.8
PuthFusion (ours)
LC
71.3
74.3
88.7
62.6
70.1
64.9
35.9
89.8
82.5
59.6
86.0
72.4
Table 3: Comparison on the nuScenes test set. L represents LiDAR input and LC represents both LiDAR and camera input. denotes double flip and denotes double flip and rotation testing-time augmentation, respectively.
We first evaluate on the KITTI dataset. As can see from Table without path-consistent regularization, the best detection result is achieved when only the feature from stage is fused. In contrast, method leads to consistent improvements. And our best results are achieved by fusing all the stages. Meanwhile, fusing both shallow features from stage and deeper features from stage leads to similar performance compared to the result from fusing all the stages.
We further evaluate on nuScenes. Here, to reduce the experiment turnaround time, we follow Focals Conv (Chen al. 2022ah and only train from Table 5 without our path-consistent regularization, both the mAP and NDS scores decrease significantly when deep features are fused. This trend is in consistent with what observed on KITTI. However, our method learns better feature fusion and achieves 0.7% improvements with path-consistent regularization.
1 split of the full training set. As we
Method
w/o Fusion
Stage
Stage
Stage 3
Stage 4
Stage 1&2
Stage 1&3
Stage 1&4
All Stages
w/o path consistency
84.93
85.22
84.85 55
83,49
83.11
85.01
84.85
84.02
83.97
w/ path consistency
85.69
85.76
85.78
85.79
85.82
85.84
85.88
Relative improvements
0.84
2.27
2.67
0.78
0.97
1.82
1.91
Table 4: Ablation Study on different fusion configurations. Results are AP3n(R1)) on the KITTI validation set. All Stages represent fusing Stage 1-4.
Robustness of our method We found our method leads to significantly improved training stability, Specifically, we use the official Focals Conv-F repo and repeat the experiments 10 times with different random seeds and report the AP (R11 1) at the moderate level. We report our findings in Table 6 Compared with Focals Conv-F, the results of our method has small standard deviation
8

---

Under review as a conference paper at ICLR 2023
MAP
NDS
Shallow-level fusion
61.7
67.2
Deep-level fusion
60.7
65.6
Shallow & Deep level fusion
61.5
67.0
+ path consistency
62.4
67.8
Table 5: Improvement over multi-modal baseline trained on nuScenes and evaluate on val set.
(i.e., 0.13), our worst run also outperfroms the best run of Focal Convs-F. We report the average performance of our method in Table
Method
Mean
Min
Max
Focals Conv-F
85.06 0.34
84.59
85.43
PathFusion (ours)
85.84 + 0.13
85.70
86.07
Table 6: Comparison KITTI with 10 random trials. Results are AP3D (R11) at the moderate level. Min and Max denotes the minimum and maximum 3D (R11) achieved out of the runs.
Impact of gradient stopping on 3D branch Gradient stopping is an important trick to regularize the corresponding parameter part of path consistency loss properly. The reason is two-fold: I) without gradient stopping, the path-consistent alone has trivial solutions by producing zero activations for both the 2D and 3D branches. 2) the task interest is 3D detection; hence, is intuitive only to regularize the 2D branch facilitate the 3D task without interfering with the 3D branch. To verify, we remove the gradient stopping on the 3D backbone and evaluate on the nuScenes 1 4 split. We found the mAP and NDS drop significantly from 62.4%, 67.8% 56.7% and 63.0%, respectively, validating the effectiveness gradient stopping on the 3D branch.
Consistency loss design We experiment with the cosine and loss as our consistency loss function. And in the meantime, we study the impact of our consistency coefficient a and test on the 1/4 nuScenes split Table 7 As we can see in Table 7 our method outperforms the baseline (mAP 61.7% and NDS 67. )7.2%) in all settings. And the choice of cosine loss with coefficient of 0.01 provides the best performance.
Loss Type
Loss weight (a)
mAp
NDS
0.1
62.1
67.5
Cosine
0.01
62.4
67.8
0.001
62.3
67.7
0.1
61.8
67.3
0.01
62.0
67.5
0.001
62.0
67.4
Table Ablation study on the impact of different path-consistent loss design and impact of the loss coefficient a. The result trained nuScenes / and evaluate val set.
7 CONCLUSION
In this work, we proposed a path consistency loss improve deep fusion between LiDAR features and camera features. Our method works by encouraging the 2D branch to follow the transformations learned the 3D branch and hence, producing complementary information that is semantically aligned to the 3D features. We applied method to improve prior-art Focals Conv, and our method leads to significant improvements on both KITTI and nuScenes datasets. Specifically, our PathFusion achieves 71.3% mAP the nuScenes test set, 1.2% better than the result from Focals Conv-F.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1090-1099, 2022.
Prarthana Bhattacharyya and Krzysztof Czarnecki. Deformable PV-RCNN: improving 3d object detection with learned deformations. In ECCV Workshop, 2020.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR. pp. 11618-11628, 2020.
Liang-Chiel Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:/706.05558 2017.
Qi Chen, Lin Sun, Ernest Cheung, and Alan L. Yuille. Every view counts: Cross-view consistency in voxelization. In NeurlPS, 2020a.
3d object detection with hybrrdd-ylindriial-spheeeee
Qi Chen, Lin Sun, Zhixin Wang, Kui Jia, and Alan Yuille. Object as hotspots: An anchor- free 3d object detection approach via firing of hotspots. In ECCV, volume 12366, pp. 68-84, 2020b.
Yukang Chen, Yanwei Li, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Focal sparse convolutional networks for 3d object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022a.
Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Scaling up kernels in 3d cnns. arXiv preprint arXiv:2206. 10555, 2022b.
Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinghong Jiang, Feng Zhao Bolei Zhou, and Hang Zhao. Autoalign: Pixel-instance feature aggregation for multi-modal 3d object detection. arXiv preprint arXiv:2201.0666 .06493, 2022c.
Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Autoalignv2: Deformable feature aggregation for dynamic multi-modal 3d object detection. arXiv preprint arXiv:2207.103166 2022d.
Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel R-CNN: towards high performance voxel-based 3d object detection. In AAAI, pp. 1201-1209, 2021.
Debidatta Dwibedi, Yusuf Aytar, Jonathan Tompson, Pierre Sermanet, and Andrew Zisserman. Temporal cycle-consistency learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1801-1810, 2019.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The KITTI dataset. Int. J. Robotics Res., 32(1 (::11)1:1311111: 2013.
Chenhang He, Hui Zeng, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang. Structure aware single-stage 3d object detection from point cloud. In CVPR, pp. 11870-11877 2020.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016.
Judy Hoffman, Eric Tzeng. Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros. and Trevor Darrell. Cycada: Cycle-conee cede--consistent adversarial domain adaptation. In International conference on machine learning. pp. 1989-1998. Pmlr. 2018.
Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues in multi-camera 3d object detection. arXiv preprint arXiv:2203.1705: 2022.
Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du. Bevdet: perfommcr multi-camera 3d object detection in vid-eyy-view. arXiv preprint arXiv:2 2021.
10

---

Under review as a conference paper at ICLR 2023
Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3d object detection. In ECCV, volume 12360, pp. 35 35-52, 2020.
Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom, Pointpil- lars: Fast encoders for object detection from point clouds. CVPR, pp. 12697-12705, 2019.
Yanwei Li, Xiaojuan Qi, Yukang Chen, Liwei Wang, Zeming Li, Jian Sun, and Jiaya Jia. Voxel field fusion for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), Pp. 1120-1122 129, June 2022a.
Yingwei Li, Adams Wei Yu, Tianjian Meng, Ben Caine, Jiquan Ngiam, Daiyi Peng, Junyang Shen, Yifeng Lu, Denny Zhou, Quoe V Le, et al. Deepfusion: Lidar-camera deep fusion for multi-modal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17182-17191, 2022b.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Beyformer: Learning bird's-eye-ve v-ew representation from multi-camera images via spatiotemporal transformers. arXiv preprint arXiv:2203. 2022c.
Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi- 3d object detection. In ECCV, volume 1 11220, pp. 663-678, 201 8a.
Ming Liang. Bin Yang. Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In Proceedings of the European conference on computer vision (ECCV), pp. 641-656, 2018b.
Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: simple and robust lidar-camera fusion framework. arXiv preprint arXiv:2205.1311 13790, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2117-2125, 2017.
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. arXiv preprint 2Xiv:2203.05625, 2022a.
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petrv2: A unified framework for 3d perception from multi-camera images. arXiv preprint arXiv:2206.01256, 2022b.
Zhijian Liu, Haotian Tang, Alexander Amini, Xingyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified sird's-eye view representation. arXiv, 2022c.
Jiageng Mao, Minzhe Niu, Haoyue Bai, Xiaodan Liang, Hang Xu, and Chunjing Xu. Pyramid R-CNN: towards better performance and adaptability for 3d object detection. In ICCV, 2021a.
Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng. Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In ICCV, 2021b.
Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J. Guibas. Frustum pointnets for 3d object detection from RGB-D data. CVPR, pp. 918-927, 2018
Charles Ruizhongtai Qi. Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In CVPR. pp. 77-85, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning point sets in metric space. In NeurlPS, pp. 5099-5108, 2017b.
11

---

Under review as a conference paper at ICLR 2023
Cody Reading, Ali Harakeh, Julia Chae, and Steven L Waslander. Categorical depth distribution network for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8555- 2021.
Shaoshuai Shi, Xiaogang Wang. and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In CVPR, pp. 770-779, 2019.
Shaoshuai Shi, Chaoxu Guo, Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN: poxnt-voxel feature set abstraction for 3d object detection. In CVPR, pp. 10526-10535, 2020.
Shaoshuai Shi. Zhe Wang, Jianping Shi, Xiaogang Wang. and Hongsheng Li. From points parts: 3d object detection from point cloud with part-aware and part-aggregation network. T-PAMI, 43 (8) (8):2647-266, 2021.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, pp. 2443-2451, 2020.
Sourabh Vora, Alex H. Lang, Bassam Helou, and Oscar Beijhom. Pointpainting: Sequential fusion for 3d object detection. In CVPR, pp. 4603-461 2020.
Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmen- tation for 3d object detection. In CVPR, pp. 1799--1803, 2021a.
Fan Wang, Qixing Huang, and Leonidas J Guibas. Image co- co-segmentation via consistent functional maps. In Proceedings of the IEEE international conference on computer vision, pp. 849-856, 2013.
Fan Wang. Qixing Huang, Maks Ovsjanikov, and Leonidas J Guibas. Unsupervised multi-class joint image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1442-119,, 2014.
Tai Wang, Xinge Zhu, Jiangmiao Pang. and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 913-922, 2021b.
Yue Wang, Alireza Fathi, Abhijit Kundu, David A. Ross, Caroline Pantofaru, Thomas A. Funkhouser, and Justin Solomon. Fillar-based object detection for autonomous driving. In ECCV, volume 12367, pp. 18-34, 2020.
Yue Wang, Vitor Guizilini, Tianyuan Zhang, Yilun Wang. Hang Zhao, and Justin M. Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In The Conference on Robot Learning (CoRL), 2021c
Xiaopei Wu, Liang Peng, Honghui Yang, Liang Xie, Chenxi Huang, Chengqi Deng, Haifeng Liu, and Deng Cai. Sparse fuse dense: Towards high quality 3d detection with depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 5418 1118-5427,00 2022.
Enze Xie, Zhiding Yu, Daquan Zhou, Jonah Philion, Anima Anandkumar, Sanja Fidler, Ping Luo, and Jose M Alvarez. M 2bev: Multi-camera joint 3d detection and segmentation with unified birds-eye view representation. arXiv preprint arXiv:2204. 0.00888888 2022.
Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Zhou Bin, and Liangjun Zhang. Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3047-3054. IEEE, 2021a.
Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Bin Zhou, and Liangjun Zhang. Fusionpainting: Multimodal fusion with adaptive attention for 3d object detection. In /TSC, pp. 3047-3054, 2021b.
12

---

Under review as a conference paper at ICLR 2023
Yan Yan, Yuxing Mao, and Bo Li. SECOND: sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018.
Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. STD: sparse-to-dense 3d object detector for point cloud. In ICCV, pp. 1951-1960, 2019.
Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR, pp. 11037-11045, 2020.
Zili Yi, Hao Zhang, Ping Tan, and Minglun Gong. Dualgan: Unsupervised dual learning for image- to-image translation. Proceedings of the IEEE international conference on computer vision, pp. 2849-2857, 2017.
Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühh. Center-based 3d object detection and tracking. In CVPR, pp. 11784-11793, 2021a.
Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Multimodal virtual point 3d detection. Advances in Neural Information Processing Systems, 34:16494-16507, 2021b.
Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Multimodal virtual point 3d detection. In NeurlPS, 2021c.
Jin Hyeok Yoo, Yecheol Kim, Ji Song Kim, and Jun Won Choi. 3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. In ECCV, volume 12372, pp. 720-736, 2020.
Zaiwei Zhang, Zhenxiao Liang. Lemeng Wu, Xiaowei Zhou, and Qixing Huang. Path-invariant map networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11084-11094, 2019.
Xin Zhao, Zhe Liu, Ruolan Hu, and Kaiqi Huang. 3d object detection using scale invariant and feature reweighting networks. In AAAI, pp. 9267-9274, 2019.
Wu Zheng, Weiliang Tang, Sijin Chen, Li Jiang, and Chi-Wing Fu. CIA-SSD: confident iou-aware single-stage object detector from point cloud. In AAAI, pp. 3555-3562, 2021.
Tinghui Zhou, Yong Jae Lee, Stella X Yu, and Alyosha A Efros. Flowweb: Joint image set alignment by weaving consistent, pixel-wise correspondences. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 1191-1200, 2015a.
Tinghui Zhou, Philipp Krahenbuhl, Mathieu Aubry, Qixing Huang. and Alexei A Efros. Learning dense correspondence via 3d-guided cycle consistency. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 117-126, 2016.
Xiaowei Zhou, Menglong Zhu, and Kostas Daniilidis. Multi-image matching via fast alternating minimization. In Proceedings of the IEEE international conference computer vision, pp. 4032 4040, 2015b.
Yin Zhou and Oncel Tuzel. VoxeInet: End-to-end learning for point cloud based 3d object detection. In CVPR, pp. 4490-4499, 2018.
Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. CoRR, htss/190..09497 2019.
Jun Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-imann to-image translation using cycle-consistent adversarial networks. Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
13


Reviews:
[{"review_id": "e08EEqeVMDb", "reviewer": "Reviewer_W3vh", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Empirical results show deep 3d fusion of lidar and camera has inferior performance compared to early stage fusion. A possible reason is the misalignment in deep fusion when deep features have large receptive fields and include overlapping regions of large areas. This paper aims to enable path-consistency between early features and deep features to enforce alignment in deep features. Path consistency is negative cosine similarity or l1 distance between features of lifted 2D features and 3D features whose input is lifted 2D features from previous stage . The experiments show 1.2%+ mAP on nuscenes test set and 0.6%+ on KITTI AP3d moderate compared to its baseline Focals Conv.", "summary_of_the_review": "The biggest concern is the results only show marginal gain compared to baseline. Without latency comparison to other fusion methods and other sota lidar-only methods, the proposed method has not justify the potential of wide application. ", "strength_and_weaknesses": "Strength:\nThe approach is reasonable. Paper is generally well-written. Ablation study is extensive.\nWeaknesses:\n- The biggest concern is the results only show marginal gain compared to baseline. Especially on KITTI the results are known to be volatile.\n- This paper claims other fusion methods are not optimized in model complexity but no latency comparison is provided.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OoKecSjATsl", "reviewer": "Reviewer_3wCP", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new technique called PathFusion that aims to fuse the LiDAR and camera features, especially the deep layer features. In order to fuse the features at multiple stages, the method design a path-consistency loss to lift and align the 2D features in the camera branch to the 3D features in the LiDAR branch. With the path-consistent regularization, the PathFusion further improves the detection accuracy on both KITTI and NuScenes datasets compared to the baseline Focals Conv method. ", "summary_of_the_review": "This paper present PathFusion to fuse the multi-stage features between LiDAR and camera branches to improve 3D detection. A path consistency regularization is designed to align the 2D image and 3D lidar features. Experiments validate the performance of the proposed method. However, the contribution is limited, and the practice of forced alignment is doubtful. Considering the above factors, I vote for the weak reject in this round of review.", "strength_and_weaknesses": "Strength: \n+ The motivation is good. Compared to the previous methods that only fuse the features in shallow layers, this method considers the fusion of deep-layer features which is valuable to multi-modal detection.\n+ The method is easy to follow and reproducable. The definition of path consistency loss is simple. \n+ Experiments on KITTI and NuScenes datasets demonstrate the robustness of the proposed method.\n\nWeaknesses:\n- The contribution is slightly incremental. The main part of this work is the design of path consistency regularization.\n- The forced alignment of the camera and LiDAR features is doubtful. As mentioned in the introduction, the LiDAR branch mainly extracts the geometry features, while the image branch produces the semantic features. Though the 2D semantic features are transformed into the 3D space, the distribution of lifted features remains intrinsic differences. Instead of aligning the features, learning an accurate transformation is more critical. \n- The description of the lift operator is unclear. Are the 2D features in different stages upsampled to the origin resolution before the lift transformation? Is that possible to adjust calibration parameters to directly lift 2D features to 3D features?\n- In practice, the point cloud and image cannot be aligned perfectly due to the synchronization error and motion distortion. For these cases, the shallow fusion is not reliable. Is deep fusion helpful in such situations?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "F9rR1sMOcI", "reviewer": "Reviewer_gP9m", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The submitted manuscript considers the task of fusing camera and lidar data to produce 3D object detections. The paper focuses on a specific type of fusion mechanism where 2D camera features are mapped to 3D for inference and discusses the pros and cons of late vs. early fusion. The main contribution of the paper is a loss function that penalizes inconsistencies across different paths through the network. The experiments demonstrate that the proposed loss function can lead to improved performance on both the KTTI and NuScenes benchmarks. ", "summary_of_the_review": "To conclude, \n\n*  (pros) the proposed loss function is intuitively appealing,\n*  (pros) the results demonstrate that the loss function improves performance,\n*  (con) the presented solution is not state-of-the-art,\n*  (con) some aspects are not sufficiently well explored,\n* (con) the paper could be more well written.\n\nMy current recommendation is marginally below acceptance, but I would be willing to change that if the paper is adjusted. ", "strength_and_weaknesses": "The proposed idea (the loss function) is interesting, and I believe it has some merit. The fact that the loss function improves performance on both KITTI and nuScenes data is, of course, a significant strength. At the same time, it is also a simple idea, and I don’t think the authors have done enough to motivate that it is important. \n\nA few aspects that would be good to discuss further: \n\n* How general is this idea? The authors only demonstrate it on a very specific architecture called FocalsConv, and I personally doubt that FocalsConv will be commonly used in the future. Can the authors mention other networks that the loss function could be combined with to motivate that the generality of the idea goes beyond FocalsConv. Ideally, the authors should include such examples in the evaluation, but I am not that is necessary. \n\n* What is the relation between the proposed loss and self-supervised learning? My understanding is that the loss function can be used in a self-supervised manner. In the current version, this has not been discussed or explored in a satisfying manner. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject