Unleash the Potential of Image Branch for Cross-modal 3D Object Detection
Authors: Yifan Zhang
Qijian Zhang
Junhui Hou
Yixuan Yuan
Guoliang Xing
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D object detection
3D point cloud
TL;DR: To fully exploit the potential of the image branch in cross-modal 3D object detection.

Abstract: To achieve reliable and precise scene understanding, autonomous vehicles typically incorporate multiple sensing modalities to capitalize on their complementary attributes. However, existing cross-modal 3D detectors do not fully utilize the image domain information to address the bottleneck issues of the LiDAR-based detectors. This paper presents a new cross-modal 3D object detector, namely UPIDet, which aims to unleash the potential of the image branch from two aspects. First, UPIDet introduces a new 2D auxiliary task called normalized local coordinate map estimation. This approach enables the learning of local spatial-aware features from the image modality to supplement sparse point clouds. Second, we discover that the representational capability of the point cloud backbone can be enhanced through the gradients backpropagated from the training objectives of the image branch, utilizing a succinct and effective point-to-pixel module. Extensive experiments and ablation studies validate the effectiveness of our method. Notably, we achieved the top rank in the highly competitive cyclist class of the KITTI benchmark at the time of submission. The source code is available at https://github.com/Eaphan/UPIDet.

Unleash the Potential of Image Branch for Cross-modal 3D Object Detection
Yifan Zhang', Qijian Zhang', Junhui Hou'; Yixuan Yuan²*, and Guoliang Xing" !City University of Hong Kong. "The Chinese University of Hong Kong (yzhang3362-co (,tp:33662..jjjiii. qijjihangg---- eeyymmittttuuu......m......mmm.e edu. hk; jh. hou@city.. edu edu. hk; yxyuan@ee. cuhk, edu. hk; guu.hhhgggiiiig glxinggie cuhk. edu. .h
Abstract
To achieve reliable and precise scene understanding, autonomous vehicles typically incorporate multiple sensing modalities to capitalize on their complementary tributes. However, existing tross-modal 3D detectors do not fully utilize the image domain information address the bottleneck issues of the LiDAR-baser detectors. This paper presents new cross- moosl 3D object detector, namely UPIDet, which aims to unleash the potential of the image branch from two aspects. First, UPIDet introduces a new 2D auxiliary task called normalized local coordinate map estima- tion. This approach enables the learning of local spatial-awan features from the image modality to supplement sparse point clouds. Second, we discover that the representational capability of the point cloud backhone can be enhanced through the gradients backpropagated from the training objectives of the image branch, utilizing a succinct and effective point-to-pixel module. Extensive experiments and ablation studies validate the effectiveness of our method. Notably, achieved the top rank the highly competitive cyclist class of the KITTI benchmark the time of submission. The source code is available at http::/giiuub.coo//aapppp UPDDe...
Introduction
In recent years, there has been increasing attention from both academia and industry towards 3D object detection, particularly in the context of autonomous driving scenarios [11, 57, 56, 58]. Two dominant data modalities 3D point clouds and 2D RGB images demonstrate complementary properties, with point clouds encoding accurate structure and depth cues but suffering from sparsity, incompleteness, and non- con-uniformity. On the other hand, RGB images convey rich semantic features and has already developed powerful learning architectures [10], but pose challenges in reliably modeling spatial structures. Despite the efforts made by previous studies [43, 40] devoted to cross-modal learning, how to reasonably mitigate the substantial gap between two modalities and utilize the information from the image domain to complement single-modal detectors remains an open and significant issue.
Currently, the performance of single-modal detectors is limited by the sparsity of observed point clouds. As shown in Figure (b), the performance of detectors on objects with fewer points (LEVEL_2) 2) significantly lower than that on objects with denser points (LEVEL_ 1). And Fig. (a) shows the distribution of point clouds in single object for the Waymo dataset. It can be observed that a substantial proportion of objects have less than 50 point clouds, making challenging to deduce the 3D bounding boxes with seven degrees of freedom from very sparse point clouds, as discussed in detail in Sec.3. While the observed point cloud can sparse due to heavy occlusion or low reflectivity material, their contour and appearance can still be clear in RGB images (as shown in the example Fig.1 (c)). Despite previous work exploring the use of semantic image features to enhance features extracted from LiDAR [40, 12), they have not fundamentally improved the spatial
*Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

in
25%
Vehicle
Pedestrian
Cydlit
20%
15%
10%
596
0%
6-20
23.50
51-100
101-180
181-250
251-350
351-500
S01-10P0
>1000
Vehicle Pedestrian
Cyclir
(a)
(D)
(c)
Figure 1: (a) Distribution of number of points in single object in Waymo Dataset.(b) Performance of the Second on Waymo [38]. (c) Point cloud and image of a black car and part of their correspondence.
representation limited by the sparsity of the point cloud. There is a class of work that attempts to generate pseudo LiDAR points as a supplement to sparse point clouds but these models commonly rely on depth estimators trained on additional data [43].
On the other hand, the performance of LiDAR-baser detection frameworks is limited by the inherent challenges of processing irregular and unordered point clouds, which makes it difficult for neural network backbones to learn effectively from 3D LiDAR data. Consequently, the representation capability of these backbone networks is still relatively insufficient, despite the performance boost achieved by previous tross-modes methods[12 [12, 40] that incorporate discriminative 2D semantic information into the 3D detection pipeline. However, there is no evidence that these methods actually enhance the representation capability of the 3D LiDAR backbone network, which is also one of the most critical influencing factors.
This paper addresses the first issue mentioned above by leveraging local-spatial aware information from RGB images. As depicted in Fig. 1 (c), predicting the 3D bounding box of a reflective black car from the sparse observed point cloud is challenging. By incorporating image information, we can not only distinguish the foreground points but also determine the relative position of the point cloud within the object, given the known 3D-to-2D correspondence. To learn local spatial-aware feature representations, we introduce new 2D auxiliary task called normalized local coordinate (NLC) map estimation, which provides the relative position of each pixel inside the object. Ultimately, we expect to improve the performance of the cross- cross-modal detector by utilizing both the known global coordinates points and their estimated relative position within an object.
On the other hand, the depth of point cloud serves as complementary source to RGB signals, providing 3D geometric information that is resilient to lighting changes and aids in distinguishing between various objects. In light of this, we present succinct point-to-pixel feature propagation module that allows 3D geometric features extracted from LiDAR point clouds to flow into the 2D image learning branch. Unlike the feature-level enhancements brought by existing pixel-to-point propagation, this method enhances the representation ability of 3D LiDAR backbone networks. Remarkably, this is an interesting observation because the back- proo-prop gradients from the 2D image branch built upon established 2D convolutional neural networks (CNNs) with impressive learning ability can effectively boost the 3D LiDAR branch.
We conduct extensive experiments on the prevailing KITTI [8] benchmark dataset. Compared with of-te-of- single-modal and cross-modal 3D object detectors, our framework achieves remarkable performance improvement. Notably, our method ranks 1st on the cyclist class of KITTI 3D detection benchmark Conclusively, our main contributions are three-fold:
We demonstrate that the representational capability of the point cloud backbone can be en- hanced through the gradients backpropagated from the training objectives of the image branch, utilizing a succinet and effective point-to-pixel module.
We reveal the importance of specific 2D auxiliary tasks used for training the 2D image learning branch which has not received enough attention in previous works, and introduce 2D NLC

---

map estimation to facilitate learning spatial detection performance.
spatial-aware aware features and implicitly boost the overall 3D
For the first time, we provide a rigorous analysis of the performance bottlenecks of single-modal detectors from the perspective of degrees of freedom (DOF) of the 3D bounding box.
2 Related Work
LiDAR-basee 3D Object Detection could be roughly divided into two categories. 1) Voxel-based
3D detectors typically voxelize the point clouds into grid-structure forms of a fixed size [61]. [47] introduced a more efficient sparse convolution to accelerate training and inference. 2) Point-based 3D detectors consume the raw 3D point clouds directly and generate predictions based on (downsam- pled) points. [32] applied point-based feature extractor and generated high-quality proposals on foreground points. 3DSSD [49] adopted a new sampling strategy named F-FPS as supplement to D-FPS to preserve enough interior points of foreground instances. It also built one-stage anchor-fie free 3D object detector based on feasible representative points. [55] and [2] introduced semantic aware down-sampling strategies preserve foreground points much as possible. [36] proposed encode the point cloud by fixed-radius near-neighbor graph.
Camera-based 3D Object Detection. Farly works [25, 24] designed monocular 3D) detectors by referring to 2D detectors [31, 39] and utilizing 2D-3D geometric constraints. Another way is to convert images to Isdud representations via monocular depth estimation and then resort to LiDAR-based methods [43]. [3] built dense 2D-3D correspondence by predicting normalized object coordinate map [42] and adopted uncertainty-driett
PnP to estimate object location. Our proposed NLC map estimation differs in that it aims to extract local spatial features from image features and does not rely labeled dense depth maps and estimated Rol boxes. Recently, multi-view detectors like BEVStereo [17] have also achieved promising performance, getting closer to LiDAR- based methods.
Cross-Modal 3D Object Detection can be roughly divided into four categories: proposal-level result- level, point-level, and pscudo-lidar-bssed approaches. Proposal-level fusion methods [5, 14, 19] adopted a two-stage framework to fuse image features and point cloud features corresponding to the same anchor or proposal. For result-level fusion, [26] exploited the geometric and semantic consistencies of 2D detection results and 3D detection results to improve maglle-modal detectors. Obviously, both proposal-Ievel and decision-level fusion strategies coarse and do not make full use of correspondence between LiDAR points and images. Pseudo-LiDAR based methods [52, 45, 19] employed depth estimatioo/complltion and converted the estimated depth map to pseudo-LiDAR points to complement raw sparse point clouds. However, such methods require extra depth map annotations that are high-cost. Regarding point-level fusion methods, [18] proposed the continuous fusion layer to retrieve corresponding image features of nearest 3D points for each grid in BEV feature map. Especially, [40] and [12] retrieved the semantic scores features by projecting points to image plane. However, neither semantic segmentation nor 2D detection tasks can enforce the network to learn 3D-spatial-awars image features. By contrast, we introduce NLC Map estimation as an auxiliary task to promote the image branch to learn local spatial information to supplement the sparse point cloud. Besides, [1] proposed TransFusion, a transformer- based detector that performs fine-grained fusion with attention mechanisms on the BEV level. [27] and [20] estimated the depth of aulti-view images and transformed the camera feature maps into the BEV space before fusing them with LiDAR BEV features.
3 Preliminary
This section introduces the definition of the NLC map and analyzes the performance bottlenecks of single-modal detectors in terms of degrees of freedom (DOF). The goal of the cross-modal 3D detector is to estimate the bounding box parameters, including the object dimensions (w. I, h), center location (Fes Yes 2.), and orientation angle 0. The input consists of an RGB image E R where H and W denote the height and width of image, respectively. Additionally, the input includes a 3D LiDAR point cloud pali = 1..... N, where N is the number of points, and each point P. E R is defined by its 3D location (Zp, Up, 2p) in the LiDAR coordinate system and the reflectance value p.
3

---

Normalized Local Coordinate (NLC) System. We define the NLC system as follows: we take the center of an object as the origin, align the r-axis towards the head direction of its ground-truth (GT) bounding box, and then normalize the local coordi- nates with respect to the size of the GT bounding box [7, 34]. Figure 2 (a) shows an example of the NLC system for a typical car. With the geome- try relationship between the imput RGB image and point cloud, we can project the NLCs to the 2D
0.1.1
@ 1.
01.0.1
1.0
0.0.%
(b) Figure 2: Illustration of the (a) NLC system and
(a)
(b) 2D NLC map.
image plane and construct 2D NLC map with three channels corresponding to the three spatial dimensions, as illustrated in F(b.
Specifically, to build the correspondence between the 2D and 3D modalities, we project the observed points from the 3D coordinate system to the 2D coordinate system on the image plane:
de [u 11 = K [ Ti -Tp Up 2p 1]²
(1)
where u, v, de denote the corresponding coordinates and depth on the image plane, R E 3x33 and T E 33x1 denote the rotation matrix and translation matrix of the LiDAR relative to the camera, and K 12 3x3 the camera intrinsic matrix.
Given the point cloud and the bounding box of an object, LiDAR coordinates of foreground points can be transformed into the NLC system by proper translation, rotation, and scaling, i.e.,
ALC H
11/1
0
1/w
0
Scs
T0.5
0.5
1/1
0
sin0
ssinf coso
in
2p de
9/
00.5
0.5
(2)
0
0
1/K
0.5
0
0
1/b
1
0.5
where (xp) (x -pp 22p
LNC. (,NCC) and (1p-s ypos
LCS)) denote the coordinates of points in NLC
"p
system and local coordinate system, respectively.
Discussion. Suppose that the bounding box is unknown, but the global coordinates and NLCs of points are known. We can build set of equations to solve for the parameters of the box, namely Fe: yes Ze: W. 1, h.0. B. To solve for all seven parameters, we need at least seven points with known global coordinates and NLCs construct the equations. However, as shown in Fig. 1, the number of points for some objects is too small to estimate the 3D box with 7 degrees of freedom. Moreover, is challenging to infer NLCss of points with only LiDAR data for objects far away with low reflectivity, as the point clouds are sparse. However, RGB images can still provide visible contours and appearances for these cases. Therefore, we propose estimate the NLCs of observed points based on enhanced image features. We can then retrieve the estimated NLCs based on the 2D-3D correspondence. Finally, we expect the proposed cross-modal detector achieve higher detection accuracy with estimated NLCs and known global coordinates.
4 Proposed Method
Overview. Architecturally, the processing pipeline for cross-modal 3D object detection includes an image branch and a point cloud branch, which learn feature representations from 2D RGB images and 3D LiDAR point clouds, respectively. Most existing methods only incorporate pixel-wise semantic cues from the 2D image branch into the 3D point cloud branch for feature fusion. However, we observe that point-based networks typically show insufficient learning ability due to the essential difficulties in processing irregular and unordered point cloud data modality [53]. Therefore, we propose to boost the expressive power of the point cloud backbone network with the assistance of the 2D image branch. Our ultimate goal is not to design more powerful point cloud backbone network structure, but to make full use of the available resources in this cross- moosl application scenario.
As shown in Fig.3, in addition to pixel-to-poin feature propagation explored by mainstream point- level fusion methods[18, 12], we employ point-to-pixx propagation allowing features to flow inversely from the point cloud branch to the image branch. We achieve bidirectional feature propagation in a multi-stage fashion. In this way, not only can image features propagated via the pixel-to-point module provide additional information, but gradients backpropagated from the training objectives of the image branch can boost the representation ability of the point cloud backbone. We also employ auxiliary tasks to train the pipeline, aiming enforce the network to Iearn rich semantic and spatial

---

Image Backbone
RGB Image
Scmanti Scome wation
NLC Map Esimation
Samantic Segmentation
Center Estimaiin
Mll. D
Point Cloud
Cloud Backbone
Fused Poirt Fash.ree
Boxes Prodiction
Auniliar Tasks
Sidrrcttionl Propagation
RPN
Rol-grid Pooling
Detect Head
Figure 3: Flowchart of the proposed UPIDet for cross-modal 3D object detection. It contains two branches separately extracting features from input 2D RGB images and 3D point clouds. UPIDet is mainly featured with (1) the point-to-pixe module that enables gradient backpropagation from 2D branch to 3D branch and (2) the supervision task of NLC map estimation for promoting the image branch to exploit local spatial features.
representations. In particular, propose NLC map estimation promote the image branch to learn spatial-aware features, providing a necessary complement to the sparse spatial representation extracted from point clouds, especially for distant highly occluded cases. Appendix A.1 provides detailed network structures the image and point cloud backbones.
4.1 Bidirectional Feature Propagation
The proposed bidirectional feature propagation consists of pixel-to-point module and point-to- pixel module, which bridges the two learning branches that operate on RGB images and LiDAR point clouds. Functionally, the point-to-pias to-pixel module applies grid-level interpolation point-wise features to produce 2D feature map, while the pixel-to-point module retrieves the corresponding 2D image features by projecting 3D points to the 2D image plane.
In general, we partition the overall 2D and 3D branches into the same number of stages. This allows us to perform bidirectional feature propagation at each stage between the corresponding layers of the image and point cloud learning networks. Without loss of generality, we focus on a certain stage where we can acquire 2D feature map F CC x W with C2D channels and dimensions H' x W from the image branch and a CaD dimeenniiii embedding vectors G {g: the corresponding N' 3D points.
N' C } J== of
Point-to-Pixel Module. As illustrated Fig. 4, we design the point-to-pixel module to propagate geometric information from the 3D point cloud branch into the 2D image branch. Formally,
Czo
H' x W x Cgo
H' x W x Cap
H' x W' (Cop + Cap) H' x W
Conv
Interpolation
Point Features
H' W!
Fused Image Features
Ayg Pooling
Output Feature To Targe: Pixel
Projection
Retrieve Paint Fealures
Image Features
Figure 4: Illustration of the proposed point-to to-x-el module.
5

---

aim to construct 2D feature map F E RRCDD by querying from the acquired point-wise embeddings G. More specifically, for each pixel position (r, over the targeted 2D grid of F with dimensions H' x W" we collect the corresponding geometric features of points that are projected inside the range of this pixel and then perform feature aggregation by avg-pooling AVG(
P21 AvG li n.),
(3)
where n counts the number of points that can be projected inside the range of the pixel located at the r row and the td column. For empty pixel positions where no points are projected inside, we define its feature value as O.
After that, instead of directly feeding the "interpolated" 2D feature map FP into the subsequent 2D convolutional stage, we introduce few additional refinement procedures for feature smoothness as described below:
Ffeee = CONV(CAT|(COV((F22) F)),
where CAT(:,-] ]] and CONV(-) stand for feature channel concatenation and the convolution operation, respectively. The resulting Ffuse is further fed into the subsequent layers the 2D image branch.
Pixel-to-Point Module. This module propagates semantic information from the 2D image branch into the 3D point cloud branch. We project the corresponding N' 3D points onto the 2D image plane, obtaining their planar coordinates denoted as X = {æ; E J Considering that these 2D coordinates are continuously and irregularly distributed over the projection plane, we apply bilinear interpolation to compute exact image features at projected positions, deducing a set of point- wise embeddings F = { (x;; E Ci = 1, .... N". Similar our practice in the point-to-pp toxpeel module, the interpolated 3D point-wise features F 12P need to be refined by the following procedures:
Gfuse = MLP (CAT MLP 12P 01), G] :0))
(5)
where MLP(:; 0) denotes shared multi-layer perceptrons parameterized by learnable weights O, and the resulting Gfuse further passes through the subsequent layers of the 3D point cloud branch.
Remark. Unlike some prior work that introduces complicated bidirectional fusion mechanisms [16, 21, 23] to seek more interaction between the two modalities and finally obtain more comprehensive features, we explore concise point-to-pixel propagation strategy from the perspective of directly improving the representation capability of the 3D LiDAR backbone network. We believe that concise but effective point-to-pixel propagation strategy will better demonstrate that with feasible 21) auxiliary tasks, the gradients backpropagated from the training objectives of the image branch can boost the representation ability of the point cloud backbone.
4.2 Auxiliary Tasks for Training
The semantic and spatial-aware information is essential for determining the category and boundary of objects. To promote the network to uncover such information from input data, we leverage auxiliary tasks when training the whole pipeline. For the point cloud branch, following SA-SSD [9], we introduce 3D semantic segmentation and center estimation to learn structure-aware features. For the image branch, propose NLC map estimation in addition to 2D semantic segmentation.
NLC Map Estimation. Vision-based semantic information is useful for networks to distinguish foreground from background. However, existing detectors face a performance bottleneck due to limited localization accuracy in distant or highly occluded cases where the spatial structure incomplete due to the sparsity of point clouds. To address this, we use NLC maps as supervision to learn the relative position each pixel inside the object from the image. This auxiliary task can drive the image branch to learn local spatial-awan features that complement the sparse spatial representation extracted from point clouds. Besides, it can augment the representation ability of the point cloud branch.
Remark. The proposed NLC map estimation shares the same objective with pseudo-LiDAR-basee methods, which aim to enhance the spatial representation limited by the sparsity of point clouds. However, compared to learning a pseudo-LiDAR representation via depth completion, our NLC map estimation has several advantages: 1) the local NLCC is easier to learn than the global depth owing to its scale-invariant property at different distances; 2) our NLC map estimation can be naturally incorporated and trained with the detection pipeline end-to-end, while it is non-trivial to

---

optimize the depth estimator and LiDAR based detector simultaneously in pseudo-LiDAR based methods although can be implemented somewhat complex way [30]; and 3) pseudo-LiDAR representations require ground-truth dense depth images as supervision, which may not be available in reality, whereas our method does not require such information.
Semantic Segmentation. Leveraging semantics of 3D point clouds has demonstrated effectiveness in point-based detectors, owing the explicit preservation of foreground points at downsampling [55, 2]. Therefore, we introduce the auxiliary semantic segmentation tasks not only in the point cloud branch but also in the image branch to exploit richer semantic information. Besides, additional semantic features extracted from images can facilitate the network to distinguish true positive candidate boxes from false positives.
4.3
Loss Function
The NLC map estimation task is optimized with the loss function defined
N Npos Σ
LNLC =
p. NLC
DNLC In 1
(6)
where p; the 7th LiDAR point, Npos is the number of foreground LiDAR points, || is the Huber-loss, 1p indicates the loss is calculated only with foreground points, and p MLC and p D.IC
are the NLCs of ground-truth and prediction at corresponding pixel for foreground points.
We the standard cross-entropy loss to optimize both 2D and 3D semantic segmentation, denoted as 2 and L respectively. And the loss of center estimation is computed as
Letr
1
Σ
lpp Apill 1p.
Npos
where Ap; the target offsets from points to the corresponding object center, and Ap, denotes the output of center estimation head.
Besides the auxiliary tasks, we define the loss the RPN stage Lypn following [2]. In addition, we also adopt commonly used proposal refinement loss Leeon defined in [33]. In all, the loss for optimizing the overall pipeline in an end-to-end manner is written as
Leotal
Lrgn + Lrean A LN + 22 La + Xa
(8)
where Å1. dg. λ3 and d are
that are empirically set to 1.
Experiments
5.1 Experiment Settings
Datasets and Metrics. We conducted experiments on the prevailing KITTI benchmark dataset, which contains two modalities of 3D point clouds and 2D RGB images. Following previous works [33]. we divided all training data into two subsets, i.e., 3712 samples for training and the rest 3769 for validation. Performance evaluated by the Average Precision (AP) metric under IoU thresholds of 0.7, 0.5, and 0.5 for car, pedestrian, and cyclist categories, respectively. We computed APs with 40 sampling recall positions by default, instead 11. For the 2D auxiliary task semantic segmentation, used the instance segmentation annotations as provided in [29]. Besides, also conducted experiments on the Waymo Open Dataset (WOD) [38], which can be found Appendix A.3.
Implementation Details. For the image branch, we used ResNet18 [10] as the backbone encoder, followed by a decoder composed of pyramid pooling module [59] and several upsampling layers to give the outputs of semantic segmentation and NLC map estimation. For the point cloud branch, we deployed a point-based network like [49] with extra FP layers and further applied semantic- guided farthest point sampling (S-FPS) [2] in SA layers. Thus, we implemented bidirectional feature propagation between the top of each SA or layer and their corresponding locations at the image branch. Please refer to Appendix A.I for more details.
7

---

Table 1: Performance comparisons on the KITTI test set, where the best and the second best results are highlighted in bold and underlined, respectively.
3D Car (177)
Ped. (loU=0.5)
3D Cyc. (lo3=0.5)
Method
Modality
mAP
Easy
Mod.
Hard
Basy
Mod.
Hard
Fasy
Mod.
Hard
PoinRRCNN [32]
LiDAR
86.96
75.64
70.70
47.98
39.37
36.01
74.96
58.82
52.53
60.33
PointPillars [15]
LIDAR
82.58
74.31
68.99
51.45
41.92
38. 89
77.10
58. 65
51.92
60.65
TANet [22]
LIDAR
84.39
75.94
68.82
53.72
44,34
40.49
75.70
59.44
52.53
61.71
IA-SSD [55]
LIDAR
88.34
80.13
75.04
46,51
39.03
35.60
78.35
61.94
55.70
62.29
STD [48]
LÍDAR
87.95
79.71
75.09
53.29
42.47
38.35
78.69
61.59
55.30
63.60
Point-GNi Soit-GNN [35]
LIDAR
88.33
79.47
72.29
51.92
43.77
40.14
78.60
63.48
57.08
63.90
Part-A2 [34]
LIDAR
87.81
78.49
73.51
53.10
43.35
40.06
79.17
63.52
56.93
63.99
PV-RCNN [33]
LÍDAR
90.25
81.43
76.82
52.17
43.29
40.29
78.60
63.71
57.65
64.91
3DSSD [49]
LÍDAR
88.36
79.57
74.55
54.64
44.27
40.23
82.48
64.10
56.90
65.01
HotSpoiNet
LIDAR
87.60
78.31
73.34
53.10
45.37
41.47
82.59
6.595
59.00
65.
PDV [11]
LIDAR
90.43
81.86
77.36
47.80
40.56
38.46
83.04
67.81
60.46
65.31
[5]
LIDAR+RBB
74.97
63.63
54,00
MMF [19]
LIDAR+RGIB
77.43
70.22
AVOD-FPN [14]
LiDAR+RGi
83.07
71.76
65.73
50.46
42.27
39.04
63.76
50.55
44.93
56.84
F-PointNet [28]
LiDAR+RGB
82.19
69.79
60.59
50.53
42.15
38.08
72.27
56.12
49.01
57.86
PointPainting [40]
LIDAR+RGB
82.11
71.70
67.08
50.32
40.97
37.84
77.63
63.78
55.89
60.81
F-ConvNet [44]
LiDAR+RGB
87.36
76.39
66.69
52.16
43.38
38,80
81.98
65.07
56.54
63.15
CAT-Det [54]
LiDAR+RGB
89.87
81.32
76.68
54.26
45.44
41.94
83.68
68. 68.81
61.45
67.05
EQ-PVRCNN [50]
LiDAR+RGB
90.13
82.01
27.53
55.84
47.02
42.94
85.41
69.10
62.30
68.03
UPIDet (Ours)
LiDAR+RGB
89.13
82,97
80.05
55.59
48.77
46.12
86.74
74.32
67.45
70.13
Table 2: Ablative experiments on different feature exploitation and propagation schemes.
3D Car (IoU=0.7)
3D Ped.
(IoU=0.5)
3D Cyc. (IoU=0.5)
Exp.
Method
mAP
Fasy
Mod.
Hard
Easy
Mod.
Hard
Fasy
Mod.
Hard
Single-Modal
91.92
85.22
82.98
68.82
61.47
56.39
91.93
74.56
69.58
75.88
(b)
Point-to-pixel
91.83
85.11
82.91
70.66
63.57
58.19
92.78
76.31
71.77
77.01
(c) Pixel-to-point
92.07
85.79
82.95
69.46
65.53
59.56
94.39
75.24
72.23
77.47
(d)
Bidirectional
92.63
85.77
83.13
72.68
67.64
62.25
94.39
77.77
71.47
78.64
5.2
Comparison with State-of-the-Art Methods
We submitted our results to the official KITTI website, and the results compared in Table 1. Our UPIDet outperforms existing state-of-the-art methods on the KITTI test set by a remarkable margin, with absolute increase of 2.1 % mAP over the second best method EQ-PVRCNN, Notably, at the time of submission, we achieved the top rank on the KITTI 3D detection benchmark for the cyclist class. We observe that UPIDet exhibits consistent and more significant improvements on the "moderate" and "hard" levels, where objects are distant or highly occluded with sparse points. We attribute these performance gains to our bidirectional feature propagation mechanism, which more adequately exploits complementary information between multi-modalities, and the effectiveness of our proposed 2D auxiliary tasks (as verified in Sec.5.3). Furthermore, actual visual results (presented in Appendix A.6) demonstrate that UPIDet produces higher-quality 3D bounding boxes in various scenes.
5.3 Ablation Study
We conducted comprehensive ablation studies validate the effectiveness and explore the impacts of key modules involved in the overall learning framework.
Effectiveness of Feature Propagation Strategies. We performed detailed ablation studies on specific multi-modal feature exploitation and interaction strategies. We started by presenting a singlc-modal baseline (Table 2(a)) that only preserves the point cloud branch of our UPIDet for both training and inference. Based on our complete bidirectional propagation pipeline (Table2(d)), we explored another two variants as shown (Table2(b)) and (Table2(c)), solely preserving the point-to-pixel and pixel-to -oee-tto-point feature propagation in our 3D detectors, respectively. Note that in Table2(b), the point- to-pixel feature flow was only enabled during training, and we detached the point-to-pixel module as well as the whole image branch during inference. Empirically, we can draw several conclusions that strongly support our preceding claims. First, the performance of Table 2(a) is the worst among all variants, which reveals the superiority of cross-moda learning. Second, combining (able2(a)
8

---

Table
Ablative experiments on key modules of our UPIDet. LiDAR: the point cloud branch trained
without auxiliary tasks; Img: the 2D RGB image backbone; 3D Seg: 3D semantic segmentation; 3D
Ctr: 3D center estimation; 2D Seg: 2D semantic segmentation; NLC:: 2D NL.C map estimation.
Exp.
LiDAR 3D Seg
3D Ctr
Img
2D Seg
NLC
Car
Ped.
Cyc.
mAP
(a)
87.40
59.02
79.11
75.18
(b)
86.75
59.90
78.57
75.07
86.56
61.06
78.58
75.40
86.71
62.23
78.68
75.88
(c)
86.91
63.23
79.00
76.38
(1)
87.10
64.52
79.94
77.18
(g)
87.18
67.52
81.21
78.64
and Tbble2(b), the mAP is largely boosted from 75.88% to 77.01 %. Considering that during the inference stage, these two variants have identical forms of input and network structure, the resulting improvement strongly indicates that the representation ability of the 3D LiDAR branch is indeed mature CNN architecture with 2D auxiliary
strengthened. In other words, the joint optimization task supervision can guide the point cloud branch to learn more discriminative point features. Third, comparing Table2(b) and Table2(c) with Table2(d), we can verify the superiority bidirectional propagation (78 64%) over the unidirectional schemes (77.01% % and 77.47%). If we particularly pay attention Table2(c) and Table2(d), we can conclude that our newly proposed point-to to to-pixel feature propagation direction further brings a 1.17% mAP increase based on the previously explored pixel-to-point paradigm.
Effectiveness of Image Branch, Comparing Table 3 (d) with Tables 3 (g) the performance is stably boosted from 75.88% to 78.64%. the gradual addition of the 2D image branch and two auxiliary tasks including 2D NLC map estimation and semantic segmentation. These results indicate that the additional semantic and geometrie features learned from the image modality are indeed effective supplements to the point cloud representation, leading to significantly improved 3D object detection performances.
Effectiveness of Auxiliary Tasks. Comparing Table 3 (a) with Tables (b)-(d), the mAP boosted by 0.70% when incorporating 3D semantic segmentation and 3D center estimation, and can be further improved by 2.76% after introducing 2D semantic segmentation and NLC map estimation (Tables (g)- However, only integrating image features without 2D auxiliary tasks (comparing results of Tables 3 (d) and (e)) brings limited improvement of 0.5%. This observation shows that the 2D) auxiliary tasks, especially the proposed 2D NLC map estimation, do enforce the image branch to learn complementary information for the detector.
15.35L
13 51
6.2
632
332
6:04
1:54
6:4
6.9
104
6.2
Ewe
8.9
Enbt
Her
but
Moceen
Her
Cyclet
Figure Quantitative results of NLC map estimation on the KITTI val set.
Performance of NLC Map Estimation. The NLC map estimation task is introduced guide the image branch to learn local apatial-aware features. We evaluated the predicted NLC of pixels containing at least one projected point with mean absolute error (MAE) for each object:
1
MAE
Σ
NLC
N.CC 4p.
1p.: 9 (ar, (ar 2).
Nobj
where Notj is the number of LiDAR points inside the ground-truth box of the object, 1p: indicates
that the evaluation is only calculated with foreground points inside the box, dp.
NC and 4p.
NLC are the

---

normalized local coordinates of the ground truth and the prediction at the corresponding pixel for the foreground point. Finally, we obtained the mean value MAE, for all instances, namely m We report the metric over different difficulty levels for three categories on the KITTI validation set in Figure We can observe that, for the car class, the mMAF error is only 0.0619, i.e., +-6.19 19 em error meter. For the challenging pedestrian and cyclist categories, the error becomes larger due to the smaller size and non-rigid shape.
6 Conclusion
We have introduced a novel cross-modal 3D detector named UPIDet that leverages complementary information from the image domain in two ways. First, demonstrated that the representational power of the point cloud backbone can be enhanced through the gradients backpropagated from the training loss of the image branch, utilizing succinct and effective point-to-pixel module. Second, we proposed NLC map estimation as an auxiliary task to facilitate the learning of local spatial-aware representation in addition to semantic features by the image branch. Our approach achieves state- of-the-art results on the KITTI detection benchmark. Furthermore, extensive ablative experiments demonstrate the robustness of UPIDet against sensor noises and its ability to generalize LiDAR signals with fewer beams. The promising results also indicate the potential of joint training between 3D object detection and other 2D scene understanding tasks. We believe that our novel perspective will inspire further investigations into multi-tat taak multi-mode learning for scene understanding in autonomous driving.
Acknowledgement
This work was supported in part by the Hong Kong Research Grants Council under Grants CityU 11219422, CityU 11202320, and CityU 11218121 12111,1, in part by the Hong Kong Innovation and Tech- nology Fund under Grant HHP//17/21, and part by the CityU Strategic Research Grant CityU 11210523. This work used the computational facilities provided by the Computing Services Centre the City University of Hong Kong. Besides, we thank the anonymous reviewers for their invaluable feedback that improved our manuscript.
References
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongho Fu, and Chiew-Lan Tai. Trans- fusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 1090-1099, 2022.
[2] Chen Chen, Zhe Chen, Jing Zhang, and Dacheng Tao. Sasa: Semantics-aummentee set abstraction for point-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 1, pp. 221-229, 2022.
[3] Hansheng Chen, Yuyao Huang, Wei Tian, Zhong Gao, Lu Xiong. Monorun: Monocular 3d object detection by reconstruction and uncertainty propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10379-10388, 2021.
Qi Chen, Lin Sun, Zhixin Wang, Kui Jia, and Alan Yuille. Object hotspots: An anchor-free 3d object detection approach via firing of hotspots. In European Conference on Computer Vision, pp. 84. Springer, 2020.
[5] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 1907-1915, 2017.
[6] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance boxel-based 3d object detection. In Proceedings the the AAAI Conference
Artificial Intelligence, pp. 1201-1209, 2021.
[7] Gil Elbaz, Tamar Avraham, and Anath Fischer. 3d point cloud registration for localization using deep neural network auto-encode In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 4631 13440 2017.
10

---

[8] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3354-3361, 2012.
[9] Chenhang He. Hui Zeng. Jianqiang Huang. Xian-Sheng Hua, and Lei Zhang. Structure aware single-stage 3d object detection from point cloud. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 11873-11882, 2020.
[10] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-778, 2016.
[11] Jordan SK Hu. Tianshu Kuai, and Steven L Waslander. Point density- aware voxels for lidar 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Patiern Recognition, pp. 8469 8478, 2022.
[12] Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3d object detection. In European Conference on Computer Vision, pp. 35 52. Springer, 2020.
[13] D.P. Kingma and J.L., Ba. Adam: A method for stochastic optimization. In International Conference on Learning Reprexentations, pp. 1-15, 2015.
[14] Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Waslander. Joint 3d proposal generation and object detection from view aggregation. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1-8. IEEE, 2018.
[15] A.H. Lang. S. Vora, H. Caesar, L. Zhou, Yang, and O. Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 12689-122222 12697, 2019.
[16] Haoran Li, Yaran Chen, Qichao Zhang. and Dongbin Zhao. Bifnet: Bidirectional fusion network for road segmentation. IEEE transactions on cybernetics, 52(9))8617 8628, 2021.
[17] Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang. Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth es- timation in multi-view object detection with dynamic temporal stereo. arXiv preprint arXiv:2209. 10248, 2022.
[18] Ming Liang. Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In European Conference on Computer Vision, pp. 641-656, 2018
[19] Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-sensor fusion for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 7345-7353, 2019.
[20] Tingting Liang. Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin. Yongtao Wang, Tao Tang, Bing Wang. and Zhi Tang. Bevfusion: A simple and robust lidar-cament fusion framework. arXiv preprint arXiv:2205.131 13790. 2022.
[21] Haisong Liu, Tao Lu, Yihui Xu, Jia Liu, Wenjie Li, and Lijun Chen. Camliflow: bidirectional camera-lida fusion for joint optical flow and scene flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5791 58011, 2022.
[22] Zhe Liu, Xin Zhao, Tengteng Huang, Ruolan Hu, Yu Zhou, and Xiang Bai. Tanet: Robust 3d object detection from point clouds with triple attention. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 11677- -1684, 2020.
[23] Zhe Liu, Bingling Li, Xiwu Chen, Xi Wang, Xiang Bai, et al. Epnet++: Cascade bi-directional fusion for multi-modal 3d object detection. arXiv preprint 22. 2021.
[24] Fabian Manhardt, Wadim Kehl, and Adrien Gaidon. Roi-10d: Monocular lifting of 2d detection to 6d pose and metric shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2069-2078. 2019.
[25] Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7074-7082, 2017.
[26] Su Pang. Daniel Morris, and Hayder Radha. Clocs: Camera-lidar object candidates fusion for 3d object detection. In 2020 JEEE/RS/ International Conference on Intelligent Robots and Systems, pp. 10386-1039) IEEE, 2020.
11

---

[27] Jonah Philion and Sanja Fidler. Lift. splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting 3d. In European Conference Computer Vision, pp. 194-210. Springer, 2020.
[28] Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 918-927,20 2018.
[29] Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Amodal instance segmentation with kins dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3014-3023, 2019.
[30] Rui Qian, Divyansh Garg. Yan Wang, Yurong You, Serge Belongie, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. End-to-end pseudo-lida for image- based 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1881-550 18900 2020.
[31] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28:91-99, 2015.
[32] S. Shi, X. Wang, and H. Li. Pointrenn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 770-779, 2019.
[33] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-remn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
[34] Shaoshuai Shi, Zhe Wang. Jianping Shi, Xiaogang Wang. and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEFE Transactions on Pattern Analysis and Machine Intelligence, 4348):647722644 2020.
[35] W. Shi and R. Rajkumar. Point-gnn: Graph neural network for 3d object detection a point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1708-1716, 2020.
[36] Weijing Shi and Raj Rajkumar. Point-gnm: In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1711-1719, 2020.
Graph neural network for 3d object detection in point cloud.
[37] Leslic N Smith. Cyclical learning rates for training neural networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 464-472. IEEE, 2017.
[38] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Eitinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, Shlens, Z. Chen, and D. Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2443-2451, 2020.
[39] Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9627-9636, 2019.
[40] Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4604-4612 2020.
[41] Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmentation for 3d object detection. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11794-1 11803, 2021.
[42] He Wang. Srinath Sridhar, Jingwei Huang. Julien Valentin, Shuran Song, and Leonidas J Guibas. Normal- ized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 2642-2651, 2019.
[43] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8445-8488008 8453, 2019.
12

---

[44] Zhixin Wang and Kui Jia. Frustum convnet: Sliding frustums to aggregate local point-wise wise features for amodal 3d object detection. In 2019 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1742-1749. IEEE, 2019.
[45] Xiaopei Wu, Liang Peng. Honghui Yang. Liang Xie, Chenxi Huang, Chengqi Deng. Haifeng Liu, and Deng Cai. Sparse fuse dense: Towards high quality detection with depth completion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3418-5427, June 2022.
[46] Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensot fusion for 3d bounding box estimation, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 244 244,253,222 2018.
[47] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10): 3337, 2018.
[48] Zetong Yang. Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dent 3d object detector for point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
1951 1951-1960, 2019.
[49] Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11040-1 11048, 2020.
[50] Zetong Yang, Li Jiang, Yanan Sun, Bernt Schicle, and Jiaya Jia, A unified query- query-based paradigm for point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 8541-8551, 2022.
[51] Jin Hyeok Yoo. Yecheol Kim, Jisong Kim, and Jun Won Choi. 3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusson for 3d object detection. European Conference on Computer Vision pp- 720-736. Springer, 2020.
[52] Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. In International Conference Learning Representations, pp. 1-13, 2020.
[53] Qijian Zhang, Junhui Hou, and Yue Qian. Multi-view vision-to-geonn to-toooertyyy knowledge transfer for 3d point cloud shape analysis. arXiv preprint or arXhv:2007.03/28 2022.
[54] Yanan Zhang, Jiaxin Chen, and Di Huang. Cat-det: Contrastively augmented transformer for multi- modal 3d object detection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 908-917, 2022.
[55] Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points equal: Learning highly efficient point-based detectors for 3d lidar point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18953-18962, 2022.
[56] Yifan Zhang, Junhui Hou, and Yixuan Yuan. comprehensive study of the robustness for lidar-based 3d object detectors against adversarial attacks. International Journal of Computer Vision, 2023.
[57] Yifan Zhang, Qijian Zhang, Zhiyu Zhu, Junhui Hou, and Yixuan Yuan. Glenet: Boosting 3d object detectors with generative label uncertainty estimation. International Journal of Computer Vision, pp. 1-21, 2023.
[58] Yifan Zhang, Zhiyu Zhu, and Junhui Hou. Spatial-temporal enhanced transformer towards multi-frame 3d object detection. arXiv preprint arXiv:2307.0034
2023.
[59] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2881 2881-2890, 2017.
[60] Wu Zheng, Weiliang Tang, Li Jiang, and Chi-Wing Fu. Se-ssd: Self-ensembling single-stage object detector from point cloud. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 14494- 14503, 2021.
[61] Y. Zhou and O. Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4490-4499, 2018
13

---

A Appendix
In this appendix, provide the details omitted from the manuscript due to space limitation. We organize the appendix as follows.
Section A. Implementation details.
Section A.2: More quantitative results.
Section A.3: Experimental results on Waymo dataset.
Section A.4: More ablation studies.
Section A.5: Efficiency analysis.
Section A.6: Visual results of 3D object detection.
Section A.7: Visual results of 2D semantic segmentation.
Section A.8: Details on the official KITTI test leaderboard.
A.1 Implementation Details
Network Architecture. Fig. Al illustrates the architectures of the point cloud and image backbone networks. For the encoder of the point cloud branch, we further show the details of multi-scale grouping (MSG) network in Table A1. Following 3DSSD [49], we take key points sampled by the 3rd SA layer generate vote points and estimate the 3D box. Then we feed these 3D boxes and features output by the last FP layer to the refinement stage. Besides, we adopt density-awar Rol grid pooling [11] encode point density as an additional feature. Note that 3D center estimation [9] aims to learn the relative position of each foreground point to the object center, while the 3D box is estimated based on sub-sampled points. Thus, the auxiliary task of 3D center estimation differs from 3D box estimation and can facilitate learning structure-awee aware features of objects.
Table Al: Details of set abstraction layers in the point cloud branch. We report the sampling strategy used in the sampling operation, ball radius of group operation, "nquery" that denotes the number of group points, and dimensions of the unit PointNet layer for multi-scale grouping. The features at different scales concatenated and dimensionally reduced to the specific output channels.
Layer
Sampling Stratogy
Radius
nquery
Feature Dimension
Output Channels
1d SA
D.FPS
[0.2, 0.4, 0.8]
[32, 32, 64]
[[16, 16, 32), [16, 16, 32), [32, 32, 64]] [[64, 64, 128), 64, 128), [64, 96, 128]]
2dd SA
D-FPS S-FPS
10.4,0.8 0.8. 1.6]
[32, 64]
128
3rd SA
D.PS & FPS
[1.6, 3.2, 4.8)
[64, 64, 128)
[[128, 128, 256], [128, 196, 256), [128, 256, 256]]
256
Through the experiments on KITTI dataset, we adopted Adam [13] (3/=0.9, 32=0.99) to optimize our UPIDet. We initialized the learning rate as 0.003 and updated it with the one-cycle policy [37]. And trained the model for a total of 80 epochs in end-to-end manner. In our experiments, the batch size was set to 8, equally distributed on 4 NVIDIA 3090 GPUs. We kept the input image with the original resolution and padded it to the size of 1248 x 376, and down-sampled the input point cloud to 16384 points during training and inference. Following the common practice, we set the detection range of the x, y. and z axis [0m, 70.4m), [-40m, 40m] and [-3m, 1m]. respectively.
Training Details.
We applied common data augmentation strategies global and object levels. The global-level augmentation includes random global flipping, global scaling with random scaling factor between 0.95 and 1.05, and global rotation around the z-axis with random angle in the range of [- - 4]. Each of the three augmentations was performed with a 50% probability for each sample. The object-level augmentation refers to copying objects from other scenes and pasting them to current scene [47]. In order to perform sampling synchronously on point clouds and images, we utilized the instance masks provided in [29]. Specifically, we pasted both the point clouds and pixels of sampled objects to the point cloud and images of new scenes, respectively.
Data Augmentation.
A.2 More quantitative Results
Performance on KITTI Val Set. We also reported the performance of our UPIDet on all three classes of the KITTI validation set in Table A2, where can be seen that our UPIDet also achieves the highest mAP of 77.73%, which is obviously higher than the second best method CAT-Det.
14

---

Input RGB Image (H, 3)
Input Point Cloud
(N. 4)
C1
(D FPS)
Level
(i1/2, W22644)
Bi-Propag
(N/4, 64)
MaxPool &
(D-FTS&S-FFFS)
Level 2
KH/4, W /4.129)
Bi Propag
[N/16, 128)
C3
(D-F78& S-FPS)
(8/8, W /8,256)
Level
Bi-Propag
IN/32, 256)
C48C5&PPM
UpSempling x2
FP
Level
/1/ W /4.2266
Bi-rrppgg
IN/16, 128)
UpSampling x2
FP
Level
(#/2, W2,64)
Bi-Propag
(N/4, 64)
UpSampling x2
FP
Level
(H,W.69
Bi-Propag
64)
Conv
Conv
MLP
MLP
(H1, W.3)
(H,W, C)
(N.C)
(N, 3)
Estimation
Sementic Segmentation
Semantic Segmentation
Estimation
Figure A1: The detailed architecture of 2D and 3D backbones. We adopt ResNet18 as the encoder of the image branch, followed by a decoder with Pyramid Pooling Module (PPM) and several up- sampling blocks. C1, C2, C3, C4, and C5 denote convolutional layers of different stages in ResNet. Extra convolutional layers are deployed after each up-sampling layer. For the point cloud branch, adopt the PointNet++ structure. SA: set abstraction layer, D-FPS: 3D Euclidean farthest point sampling, S-FPS: semantic-guinn guidde farthest point sampling, FP: feature propagation layer, MLP: shared multi-layer perceptron. Besides, "Bi-Propag" denotes the proposed bidirectional feature propagation between the 2D and 3D backbones.
ristance-based
Table A2: Quantitative comparisons on the KITTI validation set under the evaluation metric of 3D Average Precision (AP) calculated with 11 sampling recall positions. We highlight the best and the second best results in bold and underlined, respectively.
3D Car (IoU=0.7)
3D Ped. (loU=0.5)
3D Cyc. (loU=0.5)
Method
Modality
mAP
Lasy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
PoinPillars [15]
LIDAR
86.46
77.28
74.65
37.75
52.29
47.90
80.05
62.68
59.70
66.53
SECOND [47]
LiDAR
88.61
78.62
77.22
56.55
$2.98
47.73
80.58
67.15
63.10
68.06
3DSSD [49]
LiDAR
88.55
78.45
77.30
58.18
54.31
49.56
86.25
70.48
65.32
69.82
PointRCNN [32]
LiDAR
88.72
78.61
77.82
62.72
53.85
50.24
86.84
71.62
65.59
70.67
PV-RCNN [33]
LIDAR
89.03
83.24
78.59
63.71
$7.37
52.84
86.06
69.48
64.50
71.65
TANet [22]
LiDAR
88.21
77.85
75.62
70.80
63.45
58.22
85.98
64.95
60.40
71.72
Part-42 [34]
LiDAR
89.55
79.40
78.84
60.05
.4
85.50
69.90
65.48
72.20
AVOD-FPN
LIDAR+RGB
84.41
74.44
68.65
$8.80
49.70
PointFusion [46]
LiDAR+RGB
77.92
63.00
53.27
33.36
28.04
23.38
49.34
29.42
26.98
42.75
F-PointNet [28)
LiDAR+RGB
83.76
70.92
63.65
70.00
61.32
53.59
77.15
56.49
$3.37
65.58
CLOCs [26]
LiDAR+RGB
89.49
79.31
77.36
62.88
56.20
50.10
87.57
67.92
63.67
70.50
EPNct [12]
LiDAR+RGB
88.76
78.65
78.32
66,74 74
$9.29
54.82
83.88
65.50
62.70
70,96
CAT-Det [54]
LiDAR+RGB
90.12
81.46
79.15
74.08
66.35
58.92
87.66 64
72,82
68.20
75.42
UPIDet (Ours)
LIDR++GGB
89.73
86.40
79.31
71.77
68.49
62.52
89.24
76.91
75.18
77.73
Performance of Single-Class Detector. Quite few methods [6, 60] train models only for detection. Empirically, the single-class detector performs better in the car class compared with multi-class detectors. Therefore, also provided performance of UPIDet trained only for the class, and compared it with several state-of-tth-arr methods in Table A3.
Generalization to Asymmetric Backbones. As shown in Fig. A1, originally adopted an encoder-decoder network in the LiDAR branch that architecturally similar the image backbone. Nevertheless, it is worth clarifying that our approach is not limited to symmetrical structures and
15

---

Table A3: Comparison with state-of-the- the-tht--rtt art methods on the KITTI val set for car 3D detection.
All results are reported by the average precision with 0.7 IoU threshold. R11 and R40 denotes AP calculated with and 40 recall sampling recall points, respectively.
(%)
RD (%)
Method
Modal
Fasy
Mod.
Hard
Fasy
Mod.
Hard
Voxel R-CNN [6]
LIDAR
89.41
84.52
78.93
92.38
85.29
82.86
PV-RCNN [33]
LiDAR
89.35
83.69
78.7
92.57
84.83
82.69
SA-SSD [9]
LiDAR
90.15
79.91
78.78
93.14
84.65
81.86
SE-SSD [60]
LIDAR
90.21
85.71
79.22
93.19
86.12
83,31
GLENet-VR [57]
LiDAR
89.93
86.46
79.19
93.51
86.10
83.60
MV3D [S]
LIDAR+RGB
71.29
62.68
56.56
3D-CVF [51]
LiDAR+RGB
89.67
79.88
78.47
UPIDet (Ours)
LiDAR+RGB
89.72
86.52
79.34
93.03
86.46
83.73
can be generalized to different point-based backbones. Here, we replaced the 3D branch of the original framework with an efficient single-stage Sircctor-SAAA. [2], using a backbone only with the encoder in the LiDAR branch, which is asymmetric with the encoder-decoder structure of the image backbone. Accordingly, the proposed bidirectional propagation only performed between the 3D backbone and the encoder of the 2D image backbone. The experimental results are shown in Table A4. We can observe that the proposed method works well even when the two backbones are asymmetric, which demonstrates the satisfactory generalization ability of our method for different LiDAR backbones.
Table A4: Bidirectional propagation is also effective when the 2D and 3D backbones are asymmetric. Here we adopt a single-stage detector [2] whose backbone includes only an encoder, which is asymmetric with the encoder-decoder network in the image branch.
3D Car (lnl:=0.7)
3D Ped. (loU=0.5)
3D Cyc. (loU=0.5)
Method
mAP
Easy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
SASA
92.17
84,90
82.57
66.75
61.40
56.00
89.91
74.05
69.41
75.24
Ours (SASA)
92.11
85.67
82.99
70.52
63.38
58.16
91.58
74.81
70.22
76.61
Table A5: 3D detection results on the Waymo Open Dataset validation set. "_" denotes that the results not reported in their papers.
Vehicle LI
Vehicle L2
Pedestrian LI Pedestrian L2
Cyclist LI
Cyclist L2 mAP mAPH 74.41
Method
mAP mAPH
mAP MAPH
mAP mAPH
mAP mAPH
mAP mAPH
PointAugmenting
67.41
62.7
75.42
70.55
76.29
TransFusion
65.14
64.00
67.40
Ours
78.36
77.91
69.45
69.04
76.32
71.67
65.93
61.81
79.64
78.55
76.36
75.26
A.3 Results on Waymo Open Dataset
The Waymo Open Dataset [38] is large-scale dataset for 3D object detection. It contains 798 sequences (15836 frames) for training, and 202 sequences (40077 frames) for validation. According to the number of points inside the object and the difficulty of annotation, the objects are further divided into two difficulty levels: LEVEL_ and LEVEL_2. Following common practice, we adopted the metrics of mean Average Precision (mAP) and mean Average Precision weighted by heading accuracy (mAPH), and reported the performance on both LEVEL I and LEVEL 2. We set the detection range to [-75.2m. 75.2m] for and y axis, and [-2m, 4m] for axis. Following [41] and [1]. the training on Waymo dataset consists of two stages to allow flexible augmentations. First, we only trained the LiDAR branch without image inputs and bidirectional propagation for six epochs. We enabled the copy-and-past augmentation in this stage. Then, we trained the whole pipeline for another 30 epochs, during which the copy-and-pant segmentation head is disabled, since ground- -tuth segmentation maps are not provided [38].
disabled. Note that the image semantic
As shown in Table A5, our method achieves substantial improvement compared with previous of the-arts. Particularly, unlike existing approaches including PoimtAugmenting [41] ] and TransFusion [1] where the camera backbone is pre-trained on other datasets and then frozen, trained the entire
16

---

pipeline in an end-to-end manner. It can be seen that even without the 2D segmentation auxiliary task, our method still achieves higher accuracy under all scenarios except "Ped L2", demonstrating its advantage.
A.4 More Ablation Studies
Table A6: Effect of the semantic-guided SA layer. Compared with the single-modal baseline, UPIDet can better exploit image semantics and preserve more foreground points during downsampling.
Method
Single-Modal
UPIDet (Ours)
Improvement
SA Layer
FG Rate
Instance Recall
FG Rate
Instance Recall
FG Rate
Instance Recall
Level-2
15.87
97.92
20.70
98.23
+4.83
+0.37
Level-3
29.73
97.35
38.03
97.82
+8.29
+0.47
Table A7: Comparison between our multi-task training methods and the single-modal modal 2D semantic segmentation baseline (PSPNet). The results show that point features effectively improve the segmentation performance on pedestrian and cyclist classes.
Method
Car
Pes.
Cyc.
mloU
PSPNet [59]
77.49
30.45
23.83
43.92
UPIDet (Ours)
78.45
36.15
30.42
48.34
Effect of Semantic-guided Point Sampling, When performing downsampling in the SA layers of the point cloud branch, adopted S-FPS [2] to explicitly preserve as many foreground points as possible. We report the percentage of sampled foreground points and instance recall (i.e., the ratio of instances that have at least one point) in Table A6, where it can be seen that exploiting supplementary semantic features from images leads to substantial improvement of the ratio of sampled foreground points and better instance recall during S-FPS.
Influence on 2D Semantic Segmentation. We also aimed to demonstrate that the 2D-3D joint learning paradigm benefits not only the 3D object detection task but also the 2D semantic segmentation task, As shown in Table A7, the deep interaction between different modalities yields an improvement of 4.42% mloU. The point features
naturally complement RGB image features by providing 3D geometry and semantics, which are robust to illumination changes and help distinguish different classes of objects, for 2D visual information. The results suggest the potential of joint training between 3D object detection and more 2D scene understanding tasks in autonomous driving.
Conditional Analysis. To better figure out where the improvement comes from when using additional image features, we compared UPIDet with the single-modal detector on different occlusion levels and distant ranges. The results shown in Table A8 and Table A9 include separate APs for objects belonging to different occlusion levels and APs for moderate class in different distance ranges. For car detection, our UPIDet achieves more accuracy gains for long-distance and highly occluded objects, which suffer from the sparsity of observed LiDAR points. The cyclist and pedestrian are much more difficult categories on account of small sizes, non-rigid structures, and fewer training samples. For these two categories, UPIDet still brings consistent and significant improvements on different levels in extremely difficult cases.
Table A8: Performance breakdown over different occlusion levels. As defined by the official website of KITTI, occlusion levels 0, 1, and 2 correspond to fully-visible samples, paclly-oe samples, and samples that are difficult to see, respectively.
Class
Car
Pedestrian
Cyclist
Occlusion
Level-0
Level-1
Level-2
Level-0
Level-1
Level-2
Level-0
Levei-1
Level-2
Single-Modal
91.98
77.18
55.41
67.44
26.76
6.10
91.23
24.66
1.74
UPIDet (Ours)
92.26
77.44
58.39
74.00
35.13
7.99
92.89
30.02
2.53
Improvement
+0.28
+0.26
+2.97
+6.56
+8.38
+1.89
+1.66
+5.36
+0.79
Generalization to Sparse LiDAR Signals. We also compared our UPIDet with the single-modal baseline on LiDAR point clouds with various sparsity. practice, following Lieudo-LiDAR++ [52], simulated the 32- 32-beam, 16-beam, and 8-beam LiDAR signals by selecting LiDAR points whose
17

---

Table A9: Performance breakdown over different distances.
Class
Car
Podestrian
Cyclist
Distance
0-20m
20-40m
40m-Inf
0-20m
20-40m
40m-Inf
0-20m
20-40m
40m-Inf
Single-Modal
96.28
85,21
43.91
71.28
38.42
1.63
93.61
61.56
34.48
UPIDet (Ours)
96,36
86.48
49.88
76.56
45.72
2.46
94.12
67.27
39.10
Improvement
+0.07
+1.27
+5.97
+5.28
+7.30
+0.83
+0.51
+5.71
+4.62
elevation angles fall within specific intervals. As shown in Table A10, the proposed UPIDet outper- forms the single-modal bascline under all settings. The consistent improvements suggest our method can generalize to sparser signals. Besides, the proposed UPIDet significantly performs better than the baseline in the setting of LiDAR signals with fewer beams, demonstrating the effectiveness of our method in exploiting the supplementary information in the image domain.
Table A10: Comparison with single-modal baselines under LiDAR signals with different beams,
where we report AP3D R40 on the KITTI validation set.
LiDAR Beams
Modal
Car
Ped.
Cyc.
mAP
LIDAR
86.71
62.23
78.68
75,87
64
LiDAR RGB
87.18
67.52
81.21
78.64
Improvemnnt
+0.47
+5.29
+2.53
+2.76
LiDAR
83.49
57.83
70.82
70.71
LiDAR RGB
84.47
62.56
73.93
73.65
Improvement
+0.98
+4.73
+3.11
+2.94
LDDAR
79.80
53.84
60.31
64.71
16
LIDAR RGB
80.78
59.44
67.35
69.19
Improvemnnt
+0.99
+5.60
+6.84
+4,48
LIDAR
64.42
22.57
43.19 13.19
43.39
8
LiDAR RGB
67.09
31.03
47.97
48.70
Improvement
+2.66
+8.46
+4.78
+5.30
Robustness against Input Corruption. We also conducted extensive experiments verify the robustness of our UPIDet to sensor perturbation. Specifically, we added Gaussian noises to the reflectance value of points or RGB images. Fig. A2 shows that the mAP value of our cross-modal UPIDet is consistently higher than that of the single-moda modal baseline and decreases slower with the LiDAR noise level increasing. Particularly, as listed in Table A11, when the variance of the LiDAR noise is set to 0.15, the perturbation affects our cross-modal UPIDet much less than the single-modal detector. Besides, even applying the corruption to both LiDAR input and RGB images, the mAP value of our UPIDet only drops by 2.49%.
Table All: Performances (mAPs) of the single-modal baseline and our UPIDet on the KITTI val under input corruptions of simulated LiDAR and image noise sampled from the Gaussian distribution. Note that the image noise is only applicable to
77
detectors.
Cormuptions Type
Modal
Ped.
Cyc.
No Comuptiin
LIDAR
86,71
62.23
78.68
75.87
0.10
020
LIDAR Noise
LDDAR
84.37
49.19
77.27
70.28
Wetance Cont
Corruption
+
87,18
67.52
Figure A2: Comparisons of noise robust- ness between the single-modal baseline
LADAR Noise
LiDAR + RGB
86.82
65.61
77.63
76.69
Image Noise
LIDAR + RGB
87.14
66.26
77.97
77.12
and our UPIDet.
LIDAR Tmage Noise
LiDAR RGB
86.60
65.42
76.44
76.15
Effectiveness of Multi-stage Interaction. As mentioned before, both 2D and 3D backbones adopt an encoder-decoder structure, and we perform bidirectional feature propagation at both downsampling and upsampling stages. Here, we conducted experiments to verify the superiority of the multi-stage interaction over single-stage interaction. As shown in Table A12, only performing the bidirectional feature propagation in the encoder (i.e., Table A (b)) or the decoder (i.e., Table A A12 (c)) leads to worse performance than that of performing the module in both stages (i.c., Table A12 (d)).
18

---

Table A12: Ablative experiments on the multi-stage manner of bidirectional propagation, where SA and FP denote applying bidirectional propagation at downsampling (in the encoder) and upsampling (in the decoder) stages the point cloud branch, respectively.
Stage
3D Car (loU=0.7)
3D Ped. (105=0.5)
3D Cyc. (15U=0.5)
mAP
SA
FP
Easy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
91.92
85.22
82.98
68.82
61.47
56.39
91.93
74.56
69.58
75.88
V
92.67
85.86
83.40
70.94
65.29
60.35
93.60
75.60
71.04
77.64
92.19
85.44
83.27
69.02
63.41
58.47
92.85
76.39
71.91
76.99
V
92.63
85.77
83.13
72.68
67.64
62.25
94.39
77.77
71.47
78.64
A.5
Efficiency Analysis
We also compared the inference speed and number of parameters of the proposed UPIDet with state-of-the-art cross-modal approaches Table A13. Our UPIDet has about the same number of parameters CAT-Det [54], but much higher inference speed at 9.52 frames per second single GeForce RTX 2080 Ti GPU. In general, our UPIDet is inevitably slower than some single- modal detectors, but it achieves good trade-off between speed and accuracy among cross-modal approaches.
Table A13: Comparison of the number of network parameters, inference speed, and detection accuracy of different multi-modal methods on the KITTI test set.
Method
Params (M)
Frames per second
mAP (%)
AVOD-FPN [14]
38.07
10.00
56.84
F-PointNet [28]
12.45
6.25
57.86
EPNet [12]
16.23
5.88
CAT-Det [54]
23.21
3.33
67.05
UPIDet (Ours)
24.98
9.52
70.13
A.6 Visual Results of 3D Object Detection
In Figure A3, we present the qualitative comparison of detection results between the single-modal baseline and UPIDet. We can observe that the proposed UPIDet shows better localization capability than the single-modal baseline in challenging cases. Besides, we also show qualitative results of UPIDet on the KITTI test split in Figure A4. We can clearly observe that our UPIDet performs well in challenging cases, such as pedestrians and cyclists (with small sizes) and highly- occluded cars.
A.7
Visual Results of 2D Semantic Segmentation
Several examples are shown in Figure A5. For distant cars in the first and the second row as well the pedestrian in the sixth row, the size of objects small and PSPNet tends to treat them as background, while our UPIDet is able to correct such errors. In the third row, our UPIDet finds the dim cyclist missed by PSPNet. Our UPIDet also performs better for the highly occluded objects as shown in the fourth and the fifth lines. This observation shows the 3D feature representations extracted from point clouds can boost 2D semantic segmentation, since the image- based method is sensitive to illumination and can hardly handle corner cases with only single-mode inputs.
A.8 Details on Official KITTI Test Leaderboard
We submitted the results of UPIDet the official KITTI website, and ranks 1 the 3D object detection benchmark for the cyclist class. Figure A6 shows the screenshot of the leaderboard. Figure A7 illustrates the precision-recd -eccal curves along with AP scores on different categories of the KITTI test set. The samples of the KITTI test set are quite different from those of bhining/validation set in terms of scenes and camera parameters, so the impressive performance of our UPIDet on the test set demonstrates it also achieves good generalization.
19

---

Figure A3: Qualitative comparison between singlemoddl baseline and our multi modal UPIDet. For each comparison, from top to bottom, we have the image, detection results of single-modal baseline, and detection results of UPIDet. We use red, green, and yellow to denote the ground-truth, true positive and false positive bounding boxes, respectively. We highlight some objects images with red circles, which are detected by UPIDet but missed by the single-mon modal method.
20

---

Figure A4: Extra qualitative results of UPIDet on the KITTI test set. The predicted bounding boxes of car, pedestrian, and cyclist are visualized in green, cyan, and yellow, respectively. We also show the corresponding projection of boxes on images. Best viewed color and zoom for more details.
Image
(b) Ground Truth
PSPNet
(d) Ours
Figure A5: Visual results of 2D semantic segmentation the KITTI val set. The prediction boxes shown in green for car, cyan for pedestrian, and yellow for cyclist. Best viewed in color. Compared with PSPNet, cross-modal UPIDet produces accurate and detailed results.
21

---

Method
Setting
Code
Moderate
Hard
Runtime
Environmnt
BiFroDet
74.32%
86.74%
67.45
0.1.5
(Python CIC-
TED
88.82%
66.84%
(C/C+.)
(C/Co+)
87:91
66.17
95.75
62.73
(Python)
HUE)
34.00
62.57
34.68%
62.21
GN
SAREE
34.80
62.26
0.03s
EO.P.RON
62.30
VoCo
59.00
(Python C/C++)
Figure A6: Screenshot of the KITTI 3D object detection benchmark for cyclist class on August 15th, 2022. Note: When we submit the results, the temporary name is BiProDet.
Au2 1o3 21 21)
Reaal
Newlll
(3) Easy category
Moderate car category
(c) Hard category
144
Parh-
201
02
Rocal
Recal
Ncalll
(d) Easy pedestrian category
(e) Moderaae pedestrian category
(1) Hard pedestrian category
(e-t m1)
Fi (1)
Rocall
Easy cyclist category
(h) Moderate cyclist category
(0 Hard syclist category
Figure A7: Precision-recall curves of different methods on the KITTI 3D object detection test set on Aug. 1hth 2022. We also report APs in different categories for each method.
22


Reviews:
[{"review_id": "Aqiz49KSPB", "reviewer": "Reviewer_bgiL", "summary": "This paper proposes a method for multimodal (image and LiDAR) 3D object detection. The main purpose of the proposed method is to enhance the image branch. The authors design the task of NLC Map estimation, which is to predict the normalized local coordinates of points within a ground-truth box. The prediction happens on the image plane and the points are projected (carrying their NLC labels). In addition, image semantic segmentation is also adopted as an auxiliary task for the image branch. The image and LiDAR branch fuse their features during the forward of their backbones. The LiDAR branch predicts 3D instances based on the RPN proposals and RoI-pooled fused features.", "questions": "Would it be more straightforward to use local coordinate prediction in the LiDAR branch like Part-A^2? NLC estimation can also be performed for the points if they are not projected to the image plane?", "limitations": "The authors did not address the limitations.\n", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Although the idea of predicting local coordinates within a bounding box is similar to Part-A^2, it is interesting to see that it can effectively enhance the image branch. I am also wondering if additionally introducing local coordinate prediction for the LiDAR branch can further boost performance.\n2. The proposed method achieves SOTA results on the KITTI dataset, and the effectiveness of the auxiliary tasks are proved by ablation study.\n", "weaknesses": "1. The experiments are only conducted on the KITTI dataset, in fact, there are a bunch of recent multimodal 3D detection methods that provide results on the NuScenes dataset, so it would be more convincing with comparisons on NuScenes.\n2. The bidirectional feature propagation module is simply gathering features from another modality by coordinate projection and feature transformation, which is not significantly novel.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer bgiL\n\nWe sincerely appreciate the reviewer's time and effort in reviewing our paper. Thanks for your valuable comments and recognition of our work. In the following, we will comprehensively address your concerns.\n\n### **Comment 1:** *The experiments are only conducted on the KITTI dataset. The results would be more convincing with comparisons on NuScenes.*\n\n**Response:** We appreciate your suggestion for conducting experiments on the NuScenes dataset. However, due to **limited** computational resources, achieving competitive performance on NuScenes in such a short time is very difficult. The baseline point-based detector (the LiDAR branch) we used performs **poorly** on NuScenes or Waymo, further increasing the experiment workload. Frankly speaking, when preparing this paper, we have made great efforts to improve the performance of point-based detectors on the Waymo dataset and achieve **comparable** accuracy to the BEV-based detectors. Despite the missing experiments on NuScenes, we posit our method **sheds light on** point-based methods by offering new insights and an effective framework. Also, we believe the strong results on KITTI and Waymo sufficiently demonstrate the effectiveness of our proposed method and validate our motivation.\n\n### **Comment 2:** *The bidirectional feature propagation module is simply gathering features from another modality by coordinate projection and feature transformation, which is not significantly novel.*\n\n**Response:** The key motivation of our point-to-pixel module is **not** to propose structurally novel fusion mechanisms. Instead, we aim to **directly boost the representation capability of the point cloud backbone** network by back-propagating **gradients** from the training objectives of the image branch, due to the relatively weak representation capability of current point-based backbones for processing irregular 3D point clouds (see Lines 39-46 of the manuscript). We experimentally and exactly demonstrated our claim in Table 2 of our manuscript, where the mAP improves from 75.88% (Table2(a)) to 77.01% (Table2(b)) by adding the point-to-pixel module, indicating strengthened representation ability of the point cloud backbone. To our knowledge, exploiting **2D** auxiliary tasks to improve the representation capability of **3D** backbones has not been identified before. The concise point-to-pixel design best demonstrates this capability. Besides, **Reviewer gAXu** also acknowledged such a manner is non-trivial and interesting.\n\n### **Comment 3:** *Would it be more straightforward to use local coordinate prediction in the LiDAR branch like Part-A^2?*\n\n**Response:** Thank you for the valuable suggestion. We conducted experiments by predicting NLCs directly in the LiDAR branch, which improves mAP from 77.18% to 77.68% (see table below). By contrast, supervising 2D NLC map estimation further boosts mAP to 78.64%, showing superiority in exploiting local spatial-aware features from the image branch to complement sparse point cloud representations. The results validate the advantage of our design.\n\n\n| Setting | Car    | Ped.   | Cyc.   | mAP    |\n| :------ | :----- | :----- | :----- | :----- |\n| w/o NLC | 87\\.10 | 64\\.52 | 79\\.94 | 77\\.18 |\n| 3D NLC  | 87\\.23 | 65\\.19 | 80\\.62 | 77\\.68 |\n| 2D NLC  | 87\\.18 | 67\\.52 | 81\\.21 | 78\\.64 |"}, {"review_id": "iiCW96JT00", "reviewer": "Reviewer_gAXu", "summary": "This work addresses the task of 3D object detection from LiDAR and cameras. Their main contribution is developing a joint 2D and 3D stream architecture, with simple bidirectional feature flow in the backbones. To improve this, they propose to predict NLC maps in the image stream. The proposed components demonstrate performance improvement on the KITTI dataset.", "questions": "- Please reference the weaknesses section for questions.\n- I also want to ask if any augmentations are done in 2D. For instance, LR flipping may introduce ambiguity in predicting left vs right side of an object.\n", "limitations": "Authors do not appear to have included a limitations section. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The paper is clear and easy to read, with diagrams to explain crucial parts.\n- The NLC prediction significantly improves performance, supporting the core hypothesis in the paper.\n- As opposed to some existing works with joint 2D-3D backbones, the proposed bidirectional feature flow is simple without bells and whistles and complicated components.\n- It is a non-trivial and interesting result that point-to-pixel flow during training only can still improve 3D detection performance, showing that 2D gradients can help 3D model learning.\n", "weaknesses": "- The statement “However, there is no evidence that these methods actually enhance the representation capability of the 3D LiDAR backbone network, which is also one of the most critical influencing factors.” in L41 - L46 (as well as its encompassing paragraph) is unclear to me and seems unsubstantiated. Many existing works (PointPainting, AutoAlign, etc) have enhanced the point cloud/3D voxels with 2D semantics before/while using the 3D backbone network and have demonstrated significant improvements in detection metrics by training end-to-end. \n- I am slightly concerned that all of the ablative results shown are on the KITTI dataset, which is a small-scale dataset with low diversity. I would appreciate some ablations on at least the key components (NLC prediction, point-to-pixel flow) on the Waymo dataset. Nonetheless, it is impressive that a point-based method can achieve such strong performance on Waymo.\n- Point-to-pixel flow during training only does improve 3D detection performance, which seems to demonstrate that training a 2D CNN can help the 3D network learn better features. However, this is slightly confounded with the possibility that *NLC* supervision is what is improving 3D detection when point-to-pixel flow is added, not the 2D CNN. It would strengthen the paper if point-to-pixel flow improves performance even when 3D NLC supervision (similar to that in PartA2) is done.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer gAXu\nWe sincerely appreciate the time and effort in evaluating our manuscript. Your meticulous review and thoughtful critiques truly reflect your deep domain expertise and diligence as a reviewer. In what follows, we will address your remaining concerns comprehensively and clearly.\n\n### **Comment 1:** *The statement \"there is no evidence that these methods actually enhance the representation capability of the 3D LiDAR backbone network\" is not clear and seems unsubstantiated. Many existing works (PointPainting, AutoAlign, etc.) have enhanced the point cloud/3D voxels with 2D semantics before/while using the 3D backbone network and have demonstrated significant improvements.*\n\n**Response:** Sorry for the confusion caused. Prior works have indeed shown performance gains from **feature-level** fusion. By contrast, our key contribution is demonstrating that with feasible 2D auxiliary tasks, the **gradients** back-propagated from the training objectives of the image branch can boost the **representation ability** of the point cloud backbone. As analyzed in Table 2 of the manuscript, the mAP improves from 75.88% (Table2(a)) to 77.01% (Table2(b)) by adding the point-to-pixel module, precisely indicating the strengthened representation ability of the point cloud backbone because **only** the point cloud branch is used during inference. To the best of our knowledge, it has never been identified previously that **2D** auxiliary supervision tasks can be used to improve **3D** backbone.\n\n### **Comment 2:** *I would appreciate some ablations on at least the key components (NLC prediction, point-to-pixel flow) on the Waymo dataset. Nonetheless, it is impressive that a point-based method can achieve such strong performance on Waymo.*\n\n**Response:** Thank you for your valuable comments. We agree with the reviewer that the ablation study will be more convincing on the Waymo dataset. As suggested, we conducted experiments on 20% Waymo training data, verifying consistent improvements from point-to-pixel propagation and 2D NLC supervision (see table below). We also appreciate you noting the strong Waymo results of our **point-based** approach. In fact, we made great efforts to improve the performance of point-based detectors on Waymo. And we found that it is **crucial** to develop a separate head for each category like CenterPoint, aiming to learn the biases of different categories and solve the category imbalance problem.\n\n\nTable T0: Effect of the key components in BiProDet on WOD val set. We report the results of APH on LEVEL 2.\n\n| Exp. | Point2Pixel | 2D NLC | Veh.   | Ped.   | Cyc.   | Mean      |\n| :--: | :-: | :----: | :----: | :----: | :----: | :-------: |\n| (a)  | &#10006;   | &#10006;      | 65\\.67 | 57\\.34 | 71\\.64 | 64\\.88 |\n| (b)  | &#10004;   | &#10006;      | 66\\.67 | 58\\.36 | 72\\.39 | 65\\.80 |\n| (c)  | &#10004;   | &#10004;       | 67\\.75 | 59\\.25 | 73\\.9  | 66\\.96 |\n\n\n### **Comment 3:** *It is slightly confounded with the possibility that NLC supervision is what is improving 3D detection when point-to-pixel flow is added, not the 2D CNN. It would strengthen the paper if point-to-pixel flow improves performance even when 3D NLC supervision (similar to that in Part-A^2) is done.*\n\n**Response:** Thanks for your valuable suggestion. We have experimentally demonstrated the effectiveness of point-to-pixel propagation in Table 2 of our manuscript. Note the mAP is largely boosted from 75.88% (Table2(a)) to 77.01% (Table2(b)) with only a **single variable**, i.e., the point-to-pixel module is added. And following your suggestions, we further conducted the ablative experiment of point-to-pixel propagation when 2D NLC supervision is **replaced** with 3D NLC supervision. As shown in the table below, the point-to-pixel module improves the mAP consistently by 0.7%, validating its efficacy independent of NLCs supervision type.\n \n| Exp. | 3D NLC | P2I | Veh.   | Ped.   | Cyc.   | Mean   |\n| :--: | :----: | :-: | :----: | :----: | :----: | :----: |\n| (a)  | &#10004; | &#10006;   | 87\\.09 | 63\\.94 | 79\\.91 | 76\\.98 |\n| (b)  | &#10004;  | &#10004;   | 87\\.23 | 65\\.19 | 80\\.62 | 77\\.68 |\n\n### **Comment 4:** *I also want to ask if any augmentations are done in 2D. For instance, LR flipping may introduce ambiguity in predicting left vs right side of an object.*\n\n**Response:** We appreciate you raising this important point. We do not perform any data augmentation on the image input. The LR flipping does introduce ambiguity in predicting the relative location of pixels inside an object. "}, {"review_id": "SJR3U3KFe8", "reviewer": "Reviewer_5jwD", "summary": "This paper propose a multi-modal usion-based 3d object detector named BiProDet. BiProDet adopts a bidirectional feature propagation mechanism, i.e.,  point-to-pixel module and pixel-to-point module. Besides, BiProDet propose a new auxiliary task called Normalized Local Coordinate (NLC) map.", "questions": "1. The proposed method is closely related to EPNet++. It would be better to provide a more detailed comparison between these two approaches.\n2. BEV representation is receiving increasing attention recently and achieves sota results on many benchmarks. It would be better to compare the proposed method with BEV-base fusion methods (e.g. BEVFusion).\n3. It would be more convincing to evaluate the proposed method on more datasets besides KITTI,  e.g., Nuscenes.\n3. Why NLC is more beneficial for small objects such as ped. and cyc.?", "limitations": "The proposed fusion method is not as flexible as the BEV representation. And the comparison with BEV-based methods is missing.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written, and the ablation study demonstrate the effectiveness of the proposed method.", "weaknesses": "1. Comparison with some closely related work is missing. For example, the proposed method is similar to EPNet++ ( [21] in the submitted manuscript) in several aspects. Both of them design bidirectional feature fusion and adopt semantic segmentation as an auxiliary task.\n2. BEV representation is receiving increasing attention recently. However,  a comparison with these methods is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer 5jwD\n\nWe sincerely appreciate the time and effort you have dedicated to providing such insightful and comprehensive feedback on our work. The comments you have raised help us identify areas for improvement in our work. The review process has been informative in refining our paper.\n\n### **Comment 1:** *Comparison with some closely related work is missing. It would be better to provide a more detailed comparison between the proposed method and EPNet++.*\n\n**Response:** Thanks for pointing out this related paper. It is also worth mentioning that the authors were **indeed** unaware of the preprint version of EPNet++ in Arxiv at the early stage of this project. Here we summarize the differences between our work and EPNet++:\n-  EPNet++ proposes an LI-Fusion layer to enable more interaction between the two modalities and finally obtains more comprehensive features. By contrast, we explore point-to-pixel propagation from the perspective of **directly improving the representation capability of the 3D LiDAR backbone** network. Specifically, by designing feasible 2D auxiliary tasks, the **gradients** back-propagated from the training objectives of the image branch can boost the representation ability of the point cloud backbone. We experimentally and exactly demonstrated our claim in Table 2 of our manuscript. To the best of our knowledge, it has never been identified previously that **2D** auxiliary tasks can be used to improve the **3D** backbone.\n-  We also verify that the bidirectional propagation benefits **not only** the 3D object detection task **but also** the 2D semantic segmentation task (see Table 7 of the supplementary material). The decent results show the potential of 2D-3D joint learning between 3D object detection and more 2D scene understanding tasks.\n-  We design a **concise yet effective** bidirectional propagation strategy without bells and whistles, which achieves significantly better performance than EPNet++ [4], according to its reported results on the KITTI test set. And we make the source code publicly available before EPNet++.\n\n### **Comment 2:** *It would be better to compare the proposed method with BEV-base fusion methods. It would be more convincing to evaluate the proposed method on more datasets besides KITTI, e.g., Nuscenes.*\n\n**Response**: We appreciate the suggestion for including experiments on the NuScenes dataset. To be honest, it is very **difficult** to achieve competitive performance on the large-scale Nuscenes dataset in such a short time with limited GPU resources. Especially, the baseline **point-based** detector we used performs **poorly** on NuScenes or Waymo, which increases the experiment workload. Actually, when preparing this work, we have made great efforts to improve the performance of point-based detectors on the Waymo dataset to be **comparable** to the BEV-based detectors. Despite the missing experiments on NuScenes, we posit our method sheds light on point-based methods by offering new insights and an effective framework. Also, we believe the strong results on KITTI and Waymo sufficiently demonstrate the effectiveness of our proposed method and validate our motivation.\n\n\n### **Comment 3:** *Why NLC is more beneficial for small objects such as ped. and cyc.?*\n\n**Response:** Insightful observation! Small objects like pedestrians tend to have **sparser** point clouds and be more sensitive to occlusion. While their LiDAR representations may be incomplete, the contour and appearance cues could still be clear in RGB images. By learning local spatial-aware features from images supervised by 2D NLC maps, we can better complement the sparse LiDAR observations for small objects. Thus, the local spatial information from images provides greater benefits for small objects with sparser point clouds.\n\n### **Comment 4:** *The proposed fusion method is not as flexible as the BEV representation.*\n\n**Response:** While BEV methods show flexibility on large datasets, point-based approaches also have advantages, like **preserving** fine details without voxelization. In this work, we made great efforts to improve the performance of point-based detectors on the Waymo dataset to be comparable to the BEV-based detectors. For example, we found that it is crucial to develop a separate head for each category like CenterPoint, aiming to learn the biases of different categories and solve the category imbalance problem. We believe our method **sheds light on** point-based methods by offering new insights and an effective framework."}, {"review_id": "klnlHBDslb", "reviewer": "Reviewer_s1rE", "summary": "This paper studies 3D object detection with multi-modal inputs (image and point cloud). This study uses a two-stage 3D object detection pipeline and proposes two approaches for further performance improvements. Firstly, the authors propose a bidirectional feature module to fuse point cloud and image features. Secondly, the authors propose several 2D and 3D auxiliary tasks to improve representation learning. The experiments validated the proposed approaches on KITTI and Waymo Open Dataset.", "questions": "1. Why the performance on the EASY category of KITTI is inferior compared with [45]? Can the authors further improve it?\n2. Can the authors provide quantitative comparisons with [1] on the Nuscenes dataset?\n3. have the authors attempted other fusion approaches instead of the proposed one? What will be the performance difference?", "limitations": " The authors have not addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. the ablation study is well-designed and extensive to validate the effectiveness of the proposed NLC map representation and fusion network.\n2. the idea of using NLC map is well-motivated.\n3. the performance on KITTI and Waymo Open Dataset is competitive in several metrics. ", "weaknesses": "1. the technical novelty is somewhat limited. Bidirectional feature fusion is not new, for example, see \"FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation\". The proposed fusion approach is similar to this paper, which is not cited. In addition, using auxiliary loss to improve representation learning is also not very novel.\n2. Auxiliary tasks require extra labels. Do other approaches in Table 1 use semantic masks as supervision?\n3. Compared with [45], the performance in the EASY category is inferior.\n4. Previous fusion-base approaches such as [1] provided evaluations on the Nuscenes dataset yet this paper does not.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer s1rE\n\nWe sincerely appreciate you raising insightful points that helped improve our work. The comments have helped us better articulate the key contributions and value of our work.\n\n### **Comment 1:** *The technical novelty is somewhat limited. Bidirectional feature fusion is not new.*\n\n**Response:** We agree that bidirectional fusion techniques have been explored in prior works. However, the main contribution is **not** proposing a structurally new fusion module, e.g., improving the bidirectional fusion module with an attention mechanism. Instead, we aim to **directly boost the representation capability of the point cloud backbone** network by back-propagating **gradients** from the training objectives of image branch through the proposed point-to-pixel module, due to the relatively weak representation capability of current point-based backbones for processing irregular 3D point clouds (see Lines 39-46 of the manuscript). We experimentally demonstrated our claim in Table 2 of our manuscript, where the mAP largely boosted from 75.88% (Table2(a)) to 77.01% (Table2(b)), **precisely** indicating that the point-to-pixel propagation indeed strengthens the representation ability of the 3D LiDAR branch. To the best of our knowledge, it has never been identified previously that 2D auxiliary tasks can improve the representation ability of 3D backbone. Besides, **Reviewer gAXu** also acknowledged such a manner is non-trivial and interesting.\n\n### **Comment 2:** *Using auxiliary loss to improve representation learning is not very novel.*\n\n**Response:** Previous works have used 3D and 2D auxiliary losses to improve the representation ability of 3D and 2D backbones respectively. And our key finding is that the gradients back-propagated from the training objectives of the image branch can boost the representation ability of the point cloud backbone (see Table 2 of our manuscript). To the best of our knowledge, it has **never** been identified previously that **2D** auxiliary tasks could so effectively improve the representation power of **3D** backbone.\n\n### **Comment 3:** *Auxiliary tasks require extra labels. Do other approaches in Table 1 use semantic masks as supervision?*\n\n**Response:** Thanks for raising a fair point. Among the several methods listed in Table 1, MMF used extra depth completion supervision and PointPainting used extra semantic segmentation labels. Besides, it is **common** to use extra image segmentation labels in cross-modal 3D detectors, such as EPNet and MVX-Net.\n\n### **Comment 4:** *Compared with [45], the performance in the EASY category is inferior. Can the authors further improve it?*\n\n**Response:** Thank you for the detailed review. Our method particularly benefits challenging cases with **sparser** point clouds, which are more likely to occur for distant or occluded objects in the moderate and hard categories. Specifically, in this work, we learn local spatial-aware features in the image branch under the supervision of 2D NLC map supervision, which serves as a complement to the sparse point clouds. While [45] presents a novel embedding-querying paradigm and achieves higher APs in the easy category, our overall performance **significantly** surpasses it. Following your suggestion, we will explore ways to improve the AP for easy cases by incorporating stronger image backbones and leveraging the recent advances in the field of 3D object detection.\n\n### **Comment 5:** *Previous fusion-base approaches such as [1] provided evaluations on the Nuscenes dataset yet this paper does not.*\n\n**Response:** We appreciate the suggestion for including experiments on the NuScenes dataset. To be honest, it is very difficult to achieve competitive performance on the large-scale Nuscenes dataset in such a short time with limited GPU resources. Especially, the baseline **point-based** detector we used performs **poorly** on NuScenes or Waymo, which increases the experiment workload. Actually, when preparing this work, we have made great efforts to improve the performance of point-based detectors on the Waymo dataset to be **comparable** to the BEV-based detectors. Despite the missing experiments on NuScenes, we posit our method sheds light on point-based methods by offering new insights and an effective framework. Also, we believe the strong results on KITTI and Waymo sufficiently demonstrate the effectiveness of our proposed method and validate our motivation.\n\n### **Comment 6:** *Have the authors attempted other fusion approaches instead of the proposed one? What will be the performance difference?*\n\n**Response:** We have not tried to compare the performance with other fusion approaches, as our main contribution is **not** proposing a structurally new fusion method, e.g., improving the bidirectional fusion module with an attention mechanism.  Instead, we explore point-to-pixel propagation with the goal of **directly enhancing the representation capability of the 3D LiDAR backbone** network by back-propagating gradients from the training objectives of the image branch. The minimalist point-to-pixel design best reveals this effect, which has never been identified previously to the best of our knowledge. The core ideas could be integrated with other fusion approaches in the future. But we believe our work provides valuable insights despite not comparing the performance with previous fusion methods."}, {"review_id": "QBL7uX49OX", "reviewer": "Reviewer_DmuL", "summary": "This paper proposes a novel cross-model 3D detector BiProDet, which leverages the information from image domain in two ways. First, it proposes point-to-pixel bidirectional propagation strategy to boost the representation ability of the point cloud backbone. Second, it introduces NLC map estimation as an auxiliary task for training to learn local spatial-aware features of image branch to supply sparse point clouds.  The BiProDet exhibits consistent and more significant improvements on the \"moderate\" and \"hard\" levels, where objects are distant or highly occluded with sparse points and ranked 1st on the KITTI 3D detection benchmark for the cyclist class.\nIn my opinion, this paper discusses of the reinforcement mechanism of image branch for 3D detector deeply, and a bidirectional propagation is designed to enhance the performance of the backbone, which is very innovative. And a wealth of ablation experiments has been done to prove the effectiveness of the algorithm proposed in this paper.", "questions": "None", "limitations": "None", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper has several strengths in terms of originality, quality, clarity, and significance. \n\n1. It introduces a 2D auxiliary task called normalized local coordinate (NLC) map estimation to improve the performance of the cross-modal detector by providing the relative position of each pixel inside the object.\n\n2. The paper proposes a novel point-to-pixel feature propagation mechanism that allows 3D geometric features from LiDAR point clouds to enhance the representation ability of 2D image learning.\n\n3. It provides a analysis of the performance bottlenecks of single-modal\n   detectors.", "weaknesses": "1. The motivation is weak. As the proposed method point-to-pixel is just a fusion strategy in cross-modal modeling, which also been explored in many works such as BEVFusion[1]. So it is relatively weak to serve as the motivation of the paper.\n\n2. The proposed specific 2D auxiliary tasks have been explored in TiG-BEV [2]. Althogh the format of the 2D tasks is not the same, the core of the methods is similar.\n\n[1] https://github.com/mit-han-lab/bevfusion\n\n[2] TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Response to Reviewer DmuL\n\nWe sincerely appreciate the reviewer for your time and effort in reviewing our paper. In the following, we will comprehensively address your concerns.\n\n### **Comment 1:** *The motivation is weak. As the proposed method point-to-pixel is just a fusion strategy in cross-modal modeling, which also been explored in many works such as BEVFusion[1]. So it is relatively weak to serve as the motivation of the paper.*\n\n**Response:** We agree that the bidirectional fusion mechanisms have been explored before. However, the motivation of our work is **not** to obtain stronger fused feature representation with trivial structural innovation of the point-to-pixel module like previous works. Instead, recent research [R1] shows that the representation capability of point-based backbone networks for 3D point clouds is still relatively **insufficient** due to the irregular characteristic, which limits the performance of 3D detectors. Therefore, we aim to directly **boost the representation capability of the point cloud backbone** network by back-propagating **gradients** from the training objectives of the image branch through the proposed point-to-pixel module. We experimentally demonstrated our claim in Table 2 of our manuscript, where the mAP largely boosted from 75.88% (Table2(a)) to 77.01% (Table2(b)), **precisely** indicating that the point-to-pixel propagation indeed strengthens the representation ability of the 3D LiDAR branch. To the best of our knowledge, it has never been identified previously that **2D** auxiliary tasks can be used to improve the representation ability of the **3D** backbone. Besides, **Reviewer gAXu** also acknowledged such a manner is non-trivial and interesting. \n\n[R1] Zhang et al., PointMCD: Boosting Deep Point Cloud Encoders via Multi-view Cross-modal Distillation for 3D Shape Recognition, TMM'23.\n\n\n### **Comment 2:** *The proposed specific 2D auxiliary tasks have been explored in TiG-BEV [2]. Although the format of the 2D tasks is not the same, the core of the methods is similar.*\n\n**Response:** We agree with the reviewer that the adopted 2D supervision tasks are similar. The camera-based TiG-BEV utilizes inner-depth supervision to enhance the understanding of object-wise spatial structures. By contrast, our cross-modal method incorporates the image branch to learn local spatial-aware features, complementing the information the sparse point clouds provide. It is worth noting that we have made our article and source code publicly available **prior** to the release of TiG-BEV.\n\nIt would be better to **holistically** understand the two key components, i.e., the NLC map estimation and point-to-pixel module,  rather than in isolation. On one hand, we can learn extra information from the image branch with auxiliary tasks. On the other hand, we demonstrate that the gradients back-propagated from the image branch through the point-to-pixel module can boost the representation ability of the point cloud backbone. The suitable 2D tasks and the point-to-pixel module are **intricately linked**, and it is only by combining them that the greatest performance improvement can be achieved.\n"}]

Author rebuttal:
### General Response
We thank all reviewers for your time and constructive comments. Here we want to summarize a few key clarifications concerning the contributions of our work again:

**(1) The novelty of our work.** The key motivation of our point-to-pixel module is not to propose structurally novel fusion mechanisms. Instead, we aim to **directly boost the representation capability of the point cloud backbone** network by back-propagating **gradients** from the training objectives of the image branch, due to the relatively weak representation capability of current point-based backbones for processing irregular 3D point clouds (see Lines 39-46 of the manuscript). We experimentally demonstrated our claim in Table 2 of our manuscript, where the mAP improves from 75.88% (Table2(a)) to 77.01% (Table2(b)) by adding the point-to-pixel module, **precisely** indicating the strengthened representation ability of the point cloud backbone because **only** the point cloud branch is used during inference. To our knowledge, exploiting **2D** auxiliary tasks to improve **3D** backbones has not been identified before. The concise point-to-pixel design best demonstrates this capability.

It would be better to **holistically** understand the two key components, i.e., the NLC map estimation and point-to-pixel module, rather than in isolation. On one hand, we can learn local spatial-aware information from the image branch with NLC map supervision. On the other hand, we demonstrate that the gradients back-propagated from the image branch through the point-to-pixel module can boost the representation ability of the point cloud backbone. The suitable 2D tasks and the point-to-pixel module are **intricately** linked, and it is only by combining them that the highest detection accuracy can be achieved.

Besides, we demonstrate that the 2D-3D joint learning paradigm benefits **not only** the 3D object detection task **but also** the 2D semantic segmentation task (see Table 7 of the supplementary material). The point features can naturally complement RGB image features by providing 3D geometry and semantics, which are robust to illumination changes and help distinguish different classes of objects for 2D visual information. The results suggest the potential of joint training between 3D object detection and more 2D scene understanding tasks in autonomous driving.

**(2) Experiments on the NuScenes dataset.** Due to limited computational resources, achieving competitive performance on NuScenes in such a short time is very difficult. The employed baseline **point-based** detector (the LiDAR branch) performs **poorly** on NuScenes or Waymo, increasing the experiment workload. Frankly speaking, when preparing this paper, we have made great efforts to improve the performance of point-based detectors on the Waymo dataset to **comparable** accuracy to the BEV-based detectors. Despite the missing experiments on NuScenes, we posit our method **sheds light on** point-based methods by offering new insights and an effective framework. In conclusion, we believe the strong results on KITTI and Waymo sufficiently demonstrate the effectiveness of our proposed method and validate our motivation.

 We will make the reviews and author discussion public. Besides, we will include the newly added experiments and analysis in the final manuscript/supplementary material.

Meta-review: All but one reviewer vote for acceptance. Reviewers praise the simplicity and effectiveness of the proposed approach. This AC votes for acceptance, but urges the authors to add comparisons on additional datasets as requested by multiple reviewers.
Decision: Accept (poster)