Title: Unleash the Potential of Image Branch for Cross-modal 3D Object Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D object detection

3D point cloud
Abstract: To achieve reliable and precise scene understanding, autonomous vehicles typically incorporate multiple sensing modalities to capitalize on their complementary attributes. However, existing cross-modal 3D detectors do not fully utilize the image domain information to address the bottleneck issues of the LiDAR-based detectors. This paper presents a new cross-modal 3D object detector, namely UPIDet, which aims to unleash the potential of the image branch from two aspects. First, UPIDet introduces a new 2D auxiliary task called normalized local coordinate map estimation. This approach enables the learning of local spatial-aware features from the image modality to supplement sparse point clouds. Second, we discover that the representational capability of the point cloud backbone can be enhanced through the gradients backpropagated from the training objectives of the image branch, utilizing a succinct and effective point-to-pixel module. Extensive experiments and ablation studies validate the effectiveness of our method. Notably, we achieved the top rank in the highly competitive cyclist class of the KITTI benchmark at the time of submission. The source code is available at https://github.com/Eaphan/UPIDet.

Unleash the Potential of Image Branch for Cross-modal 3D Object Detection

Abstract

To achieve reliable and precise scene understanding, autonomous vehicles typically incorporate multiple sensing modalities to capitalize on their complementary attributes. However, existing cross-modal 3D detectors do not fully utilize the image domain information to address the bottleneck issues of the LiDAR-based detectors. This paper presents a new cross-modal 3D object detector, namely UPIDet, which aims to u nleash the p otential of the i mage branch from two aspects. First, UPIDet introduces a new 2D auxiliary task called normalized local coordinate map estimation. This approach enables the learning of local spatial-aware features from the image modality to supplement sparse point clouds. Second, we discover that the representational capability of the point cloud backbone can be enhanced through the gradients backpropagated from the training objectives of the image branch, utilizing a succinct and effective point-to-pixel module. Extensive experiments and ablation studies validate the effectiveness of our method. Notably, we achieved the top rank in the highly competitive cyclist class of the KITTI benchmark at the time of submission. The source code is available at https://github.com/Eaphan/UPIDet.

1 Introduction

In recent years, there has been increasing attention from both academia and industry towards 3D object detection, particularly in the context of autonomous driving scenarios [11, 57, 56, 58]. Two dominant data modalities - 3D point clouds and 2D RGB images - demonstrate complementary properties, with point clouds encoding accurate structure and depth cues but suffering from sparsity, incompleteness, and non-uniformity. On the other hand, RGB images convey rich semantic features and has already developed powerful learning architectures [10], but pose challenges in reliably modeling spatial structures. Despite the efforts made by previous studies [43, 40] devoted to cross-modal learning, how to reasonably mitigate the substantial gap between two modalities and utilize the information from the image domain to complement single-modal detectors remains an open and significant issue.

Currently, the performance of single-modal detectors is limited by the sparsity of observed point clouds. As shown in Figure 1 (b), the performance of detectors on objects with fewer points (LEVEL_2) is significantly lower than that on objects with denser points (LEVEL_1). And Fig.1 (a) shows the distribution of point clouds in a single object for the Waymo dataset. It can be observed that a substantial proportion of objects have less than 50 point clouds, making it challenging to deduce the 3D bounding boxes with seven degrees of freedom from very sparse point clouds, as discussed in detail in Sec.3. While the observed point cloud can be sparse due to heavy occlusion or low reflectivity material, their contour and appearance can still be clear in RGB images (as shown in the example in Fig.1 (c)). Despite previous work exploring the use of semantic image features to enhance features extracted from LiDAR [40, 12], they have not fundamentally improved the spatial

Number of Points in GT

25%

Vehicle

Pedestrian

Cyclist

76

L1L2

20%

72

15%

Ratio

68

10%

64

5%

60

0%

56

0-5

6-20

21-50

51-100

101-180

181-250

251-350

351-500

501-1000

>1000

Vehicle

Pedestrian

Cyclist

(a)

(b)

(c)

representation limited by the sparsity of the point cloud. There is a class of work that attempts to generate pseudo LiDAR points as a supplement to sparse point clouds but these models commonly rely on depth estimators trained on additional data [43].

On the other hand, the performance of LiDAR-based detection frameworks is limited by the inherent challenges of processing irregular and unordered point clouds, which makes it difficult for neural network backbones to learn effectively from 3D LiDAR data. Consequently, the representation capability of these backbone networks is still relatively insufficient, despite the performance boost achieved by previous cross-modal methods[12, 40] that incorporate discriminative 2D semantic information into the 3D detection pipeline. However, there is no evidence that these methods actually enhance the representation capability of the 3D LiDAR backbone network, which is also one of the most critical influencing factors.

This paper addresses the first issue mentioned above by leveraging local-spatial aware information from RGB images. As depicted in Fig. 1 (c), predicting the 3D bounding box of a reflective black car from the sparse observed point cloud is challenging. By incorporating image information, we can not only distinguish the foreground points but also determine the relative position of the point cloud within the object, given the known 3D-to-2D correspondence. To learn local spatial-aware feature representations, we introduce a new 2D auxiliary task called normalized local coordinate (NLC) map estimation, which provides the relative position of each pixel inside the object. Ultimately, we expect to improve the performance of the cross-modal detector by utilizing both the known global coordinates of points and their estimated relative position within an object.

On the other hand, the depth of point cloud serves as a complementary source to RGB signals, providing 3D geometric information that is resilient to lighting changes and aids in distinguishing between various objects. In light of this, we present a succinct point-to-pixel feature propagation module that allows 3D geometric features extracted from LiDAR point clouds to flow into the 2D image learning branch. Unlike the feature-level enhancements brought by existing pixel-to-point propagation, this method enhances the representation ability of 3D LiDAR backbone networks. Remarkably, this is an interesting observation because the back-propagated gradients from the 2D image branch built upon established 2D convolutional neural networks (CNNs) with impressive learning ability can effectively boost the 3D LiDAR branch.

We conduct extensive experiments on the prevailing KITTI [8] benchmark dataset. Compared with state-of-the-art single-modal and cross-modal 3D object detectors, our framework achieves remarkable performance improvement. Notably, our method ranks 1 st on the cyclist class of KITTI 3D detection benchmark 2 . Conclusively, our main contributions are three-fold:

We demonstrate that the representational capability of the point cloud backbone can be enhanced through the gradients backpropagated from the training objectives of the image branch, utilizing a succinct and effective point-to-pixel module.

We reveal the importance of specific 2D auxiliary tasks used for training the 2D image learning branch which has not received enough attention in previous works, and introduce 2D NLC

map estimation to facilitate learning spatial-aware features and implicitly boost the overall 3D detection performance.

For the first time, we provide a rigorous analysis of the performance bottlenecks of single-modal detectors from the perspective of degrees of freedom (DOF) of the 3D bounding box.

2 Related Work

LiDAR-based 3D Object Detection could be roughly divided into two categories. 1) Voxel-based 3D detectors typically voxelize the point clouds into grid-structure forms of a fixed size [61]. [47] introduced a more efficient sparse convolution to accelerate training and inference. 2) Point-based 3D detectors consume the raw 3D point clouds directly and generate predictions based on (downsampled) points. [32] applied a point-based feature extractor and generated high-quality proposals on foreground points. 3DSSD [49] adopted a new sampling strategy named F-FPS as a supplement to D-FPS to preserve enough interior points of foreground instances. It also built a one-stage anchor-free 3D object detector based on feasible representative points. [55] and [2] introduced semantic-aware down-sampling strategies to preserve foreground points as much as possible. [36] proposed to encode the point cloud by a fixed-radius near-neighbor graph.

Camera-based 3D Object Detection. Early works [25, 24] designed monocular 3D detectors by referring to 2D detectors [31, 39] and utilizing 2D-3D geometric constraints. Another way is to convert images to pseudo-lidar representations via monocular depth estimation and then resort to LiDAR-based methods [43]. [3] built dense 2D-3D correspondence by predicting normalized object coordinate map [42] and adopted uncertainty-driven PnP to estimate object location. Our proposed NLC map estimation differs in that it aims to extract local spatial features from image features and does not rely on labeled dense depth maps and estimated RoI boxes. Recently, multi-view detectors like BEVStereo [17] have also achieved promising performance, getting closer to LiDAR-based methods.

Cross-Modal 3D Object Detection can be roughly divided into four categories: proposal-level, resultlevel, point-level, and pseudo-lidar-based approaches. Proposal-level fusion methods [5, 14, 19] adopted a two-stage framework to fuse image features and point cloud features corresponding to the same anchor or proposal. For result-level fusion, [26] exploited the geometric and semantic consistencies of 2D detection results and 3D detection results to improve single-modal detectors. Obviously, both proposal-level and decision-level fusion strategies are coarse and do not make full use of correspondence between LiDAR points and images. Pseudo-LiDAR based methods [52, 45, 19] employed depth estimation/completion and converted the estimated depth map to pseudo-LiDAR points to complement raw sparse point clouds. However, such methods require extra depth map annotations that are high-cost. Regarding point-level fusion methods, [18] proposed the continuous fusion layer to retrieve corresponding image features of nearest 3D points for each grid in BEV feature map. Especially, [40] and [12] retrieved the semantic scores or features by projecting points to image plane. However, neither semantic segmentation nor 2D detection tasks can enforce the network to learn 3D-spatial-aware image features. By contrast, we introduce NLC Map estimation as an auxiliary task to promote the image branch to learn local spatial information to supplement the sparse point cloud. Besides, [1] proposed TransFusion, a transformer-based detector that performs fine-grained fusion with attention mechanisms on the BEV level. [27] and [20] estimated the depth of multi-view images and transformed the camera feature maps into the BEV space before fusing them with LiDAR BEV features.

3 Preliminary

This section introduces the definition of the NLC map and analyzes the performance bottlenecks of single-modal detectors in terms of degrees of freedom (DOF). The goal of the cross-modal 3D detector is to estimate the bounding box parameters, including the object dimensions ( w , l , h ), center location ( x c , y c , z c ) , and orientation angle θ . The input consists of an RGB image X ∈ R 3 × H × W , where H and W denote the height and width of image, respectively. Additionally, the input includes a 3D LiDAR point cloud p i | i = 1 , ..., N , where N is the number of points, and each point p i ∈ R 4 is defined by its 3D location ( x p , y p , z p ) in the LiDAR coordinate system and the reflectance value ρ .

Normalized Local Coordinate (NLC) System.

We define the NLC system as follows: we take the center of an object as the origin, align the x -axis towards the head direction of its ground-truth (GT) bounding box, and then normalize the local coordinates with respect to the size of the GT bounding box [7, 34]. Figure 2 (a) shows an example of the NLC system for a typical car. With the geometry relationship between the input RGB image and point cloud, we can project the NLCs to the 2D

image plane and construct a 2D NLC map with three channels corresponding to the three spatial dimensions, as illustrated in Fig.2(b).

Specifically, to build the correspondence between the 2D and 3D modalities, we project the observed points from the 3D coordinate system to the 2D coordinate system on the image plane:

where u , v , d c denote the corresponding coordinates and depth on the image plane, R ∈ R 3 × 3 and T ∈ R 3 × 1 denote the rotation matrix and translation matrix of the LiDAR relative to the camera, and K ∈ R 3 × 3 is the camera intrinsic matrix.

Given the point cloud and the bounding box of an object, LiDAR coordinates of foreground points can be transformed into the NLC system by proper translation, rotation, and scaling, i.e.,

where ( x NLC p , y NLC p , z NLC p ) and ( x LCS p , y LCS p , z LCS p ) denote the coordinates of points in NLC system and local coordinate system, respectively.

Discussion. Suppose that the bounding box is unknown, but the global coordinates and NLCs of points are known. We can build a set of equations to solve for the parameters of the box, namely x c , y c , z c , w, l, h, θ . To solve for all seven parameters, we need at least seven points with known global coordinates and NLCs to construct the equations. However, as shown in Fig. 1, the number of points for some objects is too small to estimate the 3D box with 7 degrees of freedom. Moreover, it is challenging to infer NLCs of points with only LiDAR data for objects far away or with low reflectivity, as the point clouds are sparse. However, RGB images can still provide visible contours and appearances for these cases. Therefore, we propose to estimate the NLCs of observed points based on enhanced image features. We can then retrieve the estimated NLCs based on the 2D-3D correspondence. Finally, we expect the proposed cross-modal detector to achieve higher detection accuracy with estimated NLCs and known global coordinates.

4 Proposed Method

Overview . Architecturally, the processing pipeline for cross-modal 3D object detection includes an image branch and a point cloud branch, which learn feature representations from 2D RGB images and 3D LiDAR point clouds, respectively. Most existing methods only incorporate pixel-wise semantic cues from the 2D image branch into the 3D point cloud branch for feature fusion. However, we observe that point-based networks typically show insufficient learning ability due to the essential difficulties in processing irregular and unordered point cloud data modality [53]. Therefore, we propose to boost the expressive power of the point cloud backbone network with the assistance of the 2D image branch. Our ultimate goal is not to design a more powerful point cloud backbone network structure, but to make full use of the available resources in this cross-modal application scenario.

As shown in Fig.3, in addition to pixel-to-point feature propagation explored by mainstream pointlevel fusion methods[18, 12], we employ point-to-pixel propagation allowing features to flow inversely from the point cloud branch to the image branch. We achieve bidirectional feature propagation in a multi-stage fashion. In this way, not only can image features propagated via the pixel-to-point module provide additional information, but gradients backpropagated from the training objectives of the image branch can boost the representation ability of the point cloud backbone. We also employ auxiliary tasks to train the pipeline, aiming to enforce the network to learn rich semantic and spatial

ImageBackbone

RGBImage

SemanticSegmentation

NLCMapEstimation

3DSemanticSegmentation

CenterEstimation

Point

000...00

CloudBackbone

FusedPointFeatures

3DBoxesPrediction

Auxiliary Tasks

Bidirectional Feature Propagation

RPN

Rol-grid Pooling

DetectHead

representations. In particular, we propose NLC map estimation to promote the image branch to learn spatial-aware features, providing a necessary complement to the sparse spatial representation extracted from point clouds, especially for distant or highly occluded cases. Appendix A.1 provides detailed network structures of the image and point cloud backbones.

4.1 Bidirectional Feature Propagation

The proposed bidirectional feature propagation consists of a pixel-to-point module and a point-topixel module, which bridges the two learning branches that operate on RGB images and LiDAR point clouds. Functionally, the point-to-pixel module applies grid-level interpolation on point-wise features to produce a 2D feature map, while the pixel-to-point module retrieves the corresponding 2D image features by projecting 3D points to the 2D image plane.

In general, we partition the overall 2D and 3D branches into the same number of stages. This allows us to perform bidirectional feature propagation at each stage between the corresponding layers of the image and point cloud learning networks. Without loss of generality, we focus on a certain stage where we can acquire a 2D feature map F ∈ R C 2 D × H ′ × W ′ with C 2 D channels and dimensions H ′ × W ′ from the image branch and a set of C 3 D -dimensional embedding vectors G = { g i ∈ R C 3 D } N ′ i =1 of the corresponding N ′ 3D points.

Point-to-Pixel Module. As illustrated in Fig. 4, we design the point-to-pixel module to propagate geometric information from the 3D point cloud branch into the 2D image branch. Formally, we

N'×C3D

H' × W'× C3D

H'× W'×(C3D + C2D)

Grid

Conv

Conv

Interpolation

PointFeatures

H' × W' × C2D

FusedImage

Features

Avg

Pooling

OutputFeature

Projection

Retrieve

TofargetPixel

ImageFeatures

PointFeatures

aim to construct a 2D feature map F P 2 I ∈ R C 3 D × H ′ × W ′ by querying from the acquired point-wise embeddings G . More specifically, for each pixel position ( r, c ) over the targeted 2D grid of F P 2 I with dimensions H ′ × W ′ , we collect the corresponding geometric features of points that are projected inside the range of this pixel and then perform feature aggregation by avg-pooling AVG ( · ) :

where n counts the number of points that can be projected inside the range of the pixel located at the r th row and the c th column. For empty pixel positions where no points are projected inside, we define its feature value as 0 .

After that, instead of directly feeding the "interpolated" 2D feature map F P 2 I into the subsequent 2D convolutional stage, we introduce a few additional refinement procedures for feature smoothness as described below:

where CAT [ · , · ] and CONV ( · ) stand for feature channel concatenation and the convolution operation, respectively. The resulting F fuse is further fed into the subsequent layers of the 2D image branch.

Pixel-to-Point Module. This module propagates semantic information from the 2D image branch into the 3D point cloud branch. We project the corresponding N ′ 3D points onto the 2D image plane, obtaining their planar coordinates denoted as X = { x i ∈ R 2 } N ′ i =1 . Considering that these 2D coordinates are continuously and irregularly distributed over the projection plane, we apply bilinear interpolation to compute exact image features at projected positions, deducing a set of point-wise embeddings F I 2 P = { F ( x i ) ∈ R C 2 D | i = 1 , ..., N ′ } . Similar to our practice in the point-to-pixel module, the interpolated 3D point-wise features F I 2 P need to be refined by the following procedures:

where MLP ( · ; θ ) denotes shared multi-layer perceptrons parameterized by learnable weights θ , and the resulting G fuse further passes through the subsequent layers of the 3D point cloud branch.

Remark . Unlike some prior work that introduces complicated bidirectional fusion mechanisms [16, 21, 23] to seek more interaction between the two modalities and finally obtain more comprehensive features, we explore a concise point-to-pixel propagation strategy from the perspective of directly improving the representation capability of the 3D LiDAR backbone network. We believe that a concise but effective point-to-pixel propagation strategy will better demonstrate that with feasible 2D auxiliary tasks, the gradients backpropagated from the training objectives of the image branch can boost the representation ability of the point cloud backbone.

4.2 Auxiliary Tasks for Training

The semantic and spatial-aware information is essential for determining the category and boundary of objects. To promote the network to uncover such information from input data, we leverage auxiliary tasks when training the whole pipeline. For the point cloud branch, following SA-SSD [9], we introduce 3D semantic segmentation and center estimation to learn structure-aware features. For the image branch, we propose NLC map estimation in addition to 2D semantic segmentation.

NLC Map Estimation. Vision-based semantic information is useful for networks to distinguish foreground from background. However, existing detectors face a performance bottleneck due to limited localization accuracy in distant or highly occluded cases where the spatial structure is incomplete due to the sparsity of point clouds. To address this, we use NLC maps as supervision to learn the relative position of each pixel inside the object from the image. This auxiliary task can drive the image branch to learn local spatial-aware features that complement the sparse spatial representation extracted from point clouds. Besides, it can augment the representation ability of the point cloud branch.

Remark . The proposed NLC map estimation shares the same objective with pseudo-LiDAR-based methods, which aim to enhance the spatial representation limited by the sparsity of point clouds. However, compared to learning a pseudo-LiDAR representation via depth completion, our NLC map estimation has several advantages: 1) the local NLC is easier to learn than the global depth owing to its scale-invariant property at different distances; 2) our NLC map estimation can be naturally incorporated and trained with the detection pipeline end-to-end, while it is non-trivial to

optimize the depth estimator and LiDAR-based detector simultaneously in pseudo-LiDAR based methods although it can be implemented in a somewhat complex way [30]; and 3) pseudo-LiDAR representations require ground-truth dense depth images as supervision, which may not be available in reality, whereas our method does not require such information.

Semantic Segmentation. Leveraging semantics of 3D point clouds has demonstrated effectiveness in point-based detectors, owing to the explicit preservation of foreground points at downsampling [55, 2]. Therefore, we introduce the auxiliary semantic segmentation tasks not only in the point cloud branch but also in the image branch to exploit richer semantic information. Besides, additional semantic features extracted from images can facilitate the network to distinguish true positive candidate boxes from false positives.

4.3 Loss Function

The NLC map estimation task is optimized with the loss function defined as

where p i is the i th LiDAR point, N pos is the number of foreground LiDAR points, ∥·∥ H is the Huber-loss, 1 p i indicates the loss is calculated only with foreground points, and p NLC i and ˆ p NLC i are the NLCs of ground-truth and prediction at corresponding pixel for foreground points.

We use the standard cross-entropy loss to optimize both 2D and 3D semantic segmentation, denoted as L 2 D sem and L 3 D sem respectively. And the loss of center estimation is computed as

where ∆ p i is the target offsets from points to the corresponding object center, and ∆ˆ p i denotes the output of center estimation head.

Besides the auxiliary tasks, we define the loss of the RPN stage L rpn following [2]. In addition, we also adopt commonly used proposal refinement loss L rcnn as defined in [33]. In all, the loss for optimizing the overall pipeline in an end-to-end manner is written as

where λ 1 , λ 2 , λ 3 , and λ 4 are hyper-parameters that are empirically set to 1.

5 Experiments

5.1 Experiment Settings

Datasets and Metrics. We conducted experiments on the prevailing KITTI benchmark dataset, which contains two modalities of 3D point clouds and 2D RGB images. Following previous works [33], we divided all training data into two subsets, i.e., 3712 samples for training and the rest 3769 for validation. Performance is evaluated by the Average Precision (AP) metric under IoU thresholds of 0.7, 0.5, and 0.5 for car, pedestrian, and cyclist categories, respectively. We computed APs with 40 sampling recall positions by default, instead of 11. For the 2D auxiliary task of semantic segmentation, we used the instance segmentation annotations as provided in [29]. Besides, we also conducted experiments on the Waymo Open Dataset (WOD) [38], which can be found in Appendix A.3.

Implementation Details. For the image branch, we used ResNet18 [10] as the backbone encoder, followed by a decoder composed of pyramid pooling module [59] and several upsampling layers to give the outputs of semantic segmentation and NLC map estimation. For the point cloud branch, we deployed a point-based network like [49] with extra FP layers and further applied semanticguided farthest point sampling (S-FPS) [2] in SA layers. Thus, we implemented bidirectional feature propagation between the top of each SA or FP layer and their corresponding locations at the image branch. Please refer to Appendix A.1 for more details.

5.2 Comparison with State-of-the-Art Methods

We submitted our results to the official KITTI website, and the results are compared in Table 1. Our UPIDet outperforms existing state-of-the-art methods on the KITTI test set by a remarkable margin, with an absolute increase of 2.1% mAP over the second best method EQ-PVRCNN. Notably, at the time of submission, we achieved the top rank on the KITTI 3D detection benchmark for the cyclist class. We observe that UPIDet exhibits consistent and more significant improvements on the "moderate" and "hard" levels, where objects are distant or highly occluded with sparse points. We attribute these performance gains to our bidirectional feature propagation mechanism, which more adequately exploits complementary information between multi-modalities, and the effectiveness of our proposed 2D auxiliary tasks (as verified in Sec.5.3). Furthermore, actual visual results (presented in Appendix A.6) demonstrate that UPIDet produces higher-quality 3D bounding boxes in various scenes.

5.3 Ablation Study

We conducted comprehensive ablation studies to validate the effectiveness and explore the impacts of key modules involved in the overall learning framework.

Effectiveness of Feature Propagation Strategies. We performed detailed ablation studies on specific multi-modal feature exploitation and interaction strategies. We started by presenting a single-modal baseline (Table 2(a)) that only preserves the point cloud branch of our UPIDet for both training and inference. Based on our complete bidirectional propagation pipeline (Table2(d)), we explored another two variants as shown in (Table2(b)) and (Table2(c)), solely preserving the point-to-pixel and pixel-to-point feature propagation in our 3D detectors, respectively. Note that in Table2(b), the pointto-pixel feature flow was only enabled during training, and we detached the point-to-pixel module as well as the whole image branch during inference. Empirically, we can draw several conclusions that strongly support our preceding claims. First , the performance of Table 2(a) is the worst among all variants, which reveals the superiority of cross-modal learning. Second , combining Table2(a)

and Table2(b), the mAP is largely boosted from 75.88% to 77.01%. Considering that during the inference stage, these two variants have identical forms of input and network structure, the resulting improvement strongly indicates that the representation ability of the 3D LiDAR branch is indeed strengthened. In other words, the joint optimization of a mature CNN architecture with 2D auxiliary task supervision can guide the point cloud branch to learn more discriminative point features. Third , comparing Table2(b) and Table2(c) with Table2(d), we can verify the superiority of bidirectional propagation (78.64%) over the unidirectional schemes (77.01% and 77.47%). If we particularly pay attention to Table2(c) and Table2(d), we can conclude that our newly proposed point-to-pixel feature propagation direction further brings a 1.17% mAP increase based on the previously explored pixel-to-point paradigm.

Effectiveness of Image Branch. Comparing Table 3 (d) with Tables 3 (e)-(g), the performance is stably boosted from 75.88% to 78.64%, as the gradual addition of the 2D image branch and two auxiliary tasks including 2D NLC map estimation and semantic segmentation. These results indicate that the additional semantic and geometric features learned from the image modality are indeed effective supplements to the point cloud representation, leading to significantly improved 3D object detection performances.

Effectiveness of Auxiliary Tasks. Comparing Table 3 (a) with Tables 3 (b)-(d), the mAP is boosted by 0.70% when incorporating 3D semantic segmentation and 3D center estimation, and can be further improved by 2.76% after introducing 2D semantic segmentation and NLC map estimation (Tables 3 (d)-(g)). However, only integrating image features without 2D auxiliary tasks (comparing results of Tables 3 (d) and (e)) brings limited improvement of 0.5%. This observation shows that the 2D auxiliary tasks, especially the proposed 2D NLC map estimation, do enforce the image branch to learn complementary information for the detector.

0.20

mMAEx

mMAEy

mMAE>

0.20

mMAEx

mMAEy

mMAE

0.20

mMAEx

mMAEy

mMAE

16.5%

16.8%

16.9%

18.0%

17.7%

17.9%

0.16

0.16

0.16

12.5%

12.7%

12.9%

0.12

0.12

0.12

11.1%

11.2%

9.3%

9.0%

0.04

0.08

5.0%

7.0%

4.3%

5.7%

8.0%

4.8%

6.7%

5.0%

0.04

0.08

4.5%

5.0%

5.5%

0.08

0.04

4.4%

6.3%

6.5%

0.00

Easy

Moderate

Hard

0.00

Easy

Moderate

Hard

0.00

Easy

Moderate

Hard

(a) Car

(b) Pedestrian

(c) Cyclist

Performance of NLC Map Estimation. The NLC map estimation task is introduced to guide the image branch to learn local spatial-aware features. We evaluated the predicted NLC of pixels containing at least one projected point with mean absolute error (MAE) for each object:

where N obj is the number of LiDAR points inside the ground-truth box of the object, 1 p i indicates that the evaluation is only calculated with foreground points inside the box, q NLC p i and ˆ q NLC p i are the

normalized local coordinates of the ground-truth and the prediction at the corresponding pixel for the foreground point. Finally, we obtained the mean value of MAE q for all instances, namely mMAE q . We report the metric over different difficulty levels for three categories on the KITTI validation set in Figure 5. We can observe that, for the car class, the mMAE error is only 0.0619, i.e., ± 6 . 19 cm error per meter. For the challenging pedestrian and cyclist categories, the error becomes larger due to the smaller size and non-rigid shape.

6 Conclusion

We have introduced a novel cross-modal 3D detector named UPIDet that leverages complementary information from the image domain in two ways. First, we demonstrated that the representational power of the point cloud backbone can be enhanced through the gradients backpropagated from the training loss of the image branch, utilizing a succinct and effective point-to-pixel module. Second, we proposed NLC map estimation as an auxiliary task to facilitate the learning of local spatial-aware representation in addition to semantic features by the image branch. Our approach achieves stateof-the-art results on the KITTI detection benchmark. Furthermore, extensive ablative experiments demonstrate the robustness of UPIDet against sensor noises and its ability to generalize to LiDAR signals with fewer beams. The promising results also indicate the potential of joint training between 3D object detection and other 2D scene understanding tasks. We believe that our novel perspective will inspire further investigations into multi-task multi-modal learning for scene understanding in autonomous driving.

References

Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1090-1099, 2022.

Chen Chen, Zhe Chen, Jing Zhang, and Dacheng Tao. Sasa: Semantics-augmented set abstraction for point-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 1, pp. 221-229, 2022.

Hansheng Chen, Yuyao Huang, Wei Tian, Zhong Gao, and Lu Xiong. Monorun: Monocular 3d object detection by reconstruction and uncertainty propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10379-10388, 2021.

Qi Chen, Lin Sun, Zhixin Wang, Kui Jia, and Alan Yuille. Object as hotspots: An anchor-free 3d object detection approach via firing of hotspots. In European Conference on Computer Vision , pp. 68-84. Springer, 2020.

Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1907-1915, 2017.

Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence , pp. 1201-1209, 2021.

Gil Elbaz, Tamar Avraham, and Anath Fischer. 3d point cloud registration for localization using a deep neural network auto-encoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 4631-4640, 2017.

Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 3354-3361, 2012.

Chenhang He, Hui Zeng, Jianqiang Huang, Xian-Sheng Hua, and Lei Zhang. Structure aware single-stage 3d object detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 11873-11882, 2020.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 770-778, 2016.

Jordan SK Hu, Tianshu Kuai, and Steven L Waslander. Point density-aware voxels for lidar 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8469-8478, 2022.

Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3d object detection. In European Conference on Computer Vision , pp. 35-52. Springer, 2020.

D.P. Kingma and J.L. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations , pp. 1-15, 2015.

Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander. Joint 3d proposal generation and object detection from view aggregation. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 1-8. IEEE, 2018.

A.H. Lang, S. Vora, H. Caesar, L. Zhou, J. Yang, and O. Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 12689-12697, 2019.

Haoran Li, Yaran Chen, Qichao Zhang, and Dongbin Zhao. Bifnet: Bidirectional fusion network for road segmentation. IEEE transactions on cybernetics , 52(9):8617-8628, 2021.

Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo. arXiv preprint arXiv:2209.10248 , 2022.

Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi-sensor 3d object detection. In European Conference on Computer Vision , pp. 641-656, 2018.

Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-sensor fusion for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7345-7353, 2019.

Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. arXiv preprint arXiv:2205.13790 , 2022.

Haisong Liu, Tao Lu, Yihui Xu, Jia Liu, Wenjie Li, and Lijun Chen. Camliflow: bidirectional camera-lidar fusion for joint optical flow and scene flow estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 5791-5801, 2022.

Zhe Liu, Xin Zhao, Tengteng Huang, Ruolan Hu, Yu Zhou, and Xiang Bai. Tanet: Robust 3d object detection from point clouds with triple attention. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pp. 11677-11684, 2020.

Zhe Liu, Bingling Li, Xiwu Chen, Xi Wang, Xiang Bai, et al. Epnet++: Cascade bi-directional fusion for multi-modal 3d object detection. arXiv preprint arXiv:2112.11088 , 2021.

Fabian Manhardt, Wadim Kehl, and Adrien Gaidon. Roi-10d: Monocular lifting of 2d detection to 6d pose and metric shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2069-2078, 2019.

Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estimation using deep learning and geometry. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7074-7082, 2017.

Su Pang, Daniel Morris, and Hayder Radha. Clocs: Camera-lidar object candidates fusion for 3d object detection. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 10386-10393. IEEE, 2020.

Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European Conference on Computer Vision , pp. 194-210. Springer, 2020.

Charles R Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 918-927, 2018.

Lu Qi, Li Jiang, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Amodal instance segmentation with kins dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 3014-3023, 2019.

Rui Qian, Divyansh Garg, Yan Wang, Yurong You, Serge Belongie, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. End-to-end pseudo-lidar for image-based 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 5881-5890, 2020.

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems , 28:91-99, 2015.

S. Shi, X. Wang, and H. Li. Pointrcnn: 3d object proposal generation and detection from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 770-779, 2019.

Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 10529-10538, 2020.

Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(8):2647-2664, 2020.

W. Shi and R. Rajkumar. Point-gnn: Graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1708-1716, 2020.

Weijing Shi and Raj Rajkumar. Point-gnn: Graph neural network for 3d object detection in a point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1711-1719, 2020.

Leslie N Smith. Cyclical learning rates for training neural networks. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision , pp. 464-472. IEEE, 2017.

P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, V. Vasudevan, W. Han, J. Ngiam, H. Zhao, A. Timofeev, S. Ettinger, M. Krivokon, A. Gao, A. Joshi, Y. Zhang, J. Shlens, Z. Chen, and D. Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2443-2451, 2020.

Zhi Tian, Chunhua Shen, Hao Chen, and Tong He. Fcos: Fully convolutional one-stage object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 9627-9636, 2019.

Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 4604-4612, 2020.

Chunwei Wang, Chao Ma, Ming Zhu, and Xiaokang Yang. Pointaugmenting: Cross-modal augmentation for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 11794-11803, 2021.

He Wang, Srinath Sridhar, Jingwei Huang, Julien Valentin, Shuran Song, and Leonidas J Guibas. Normalized object coordinate space for category-level 6d object pose and size estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2642-2651, 2019.

Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8445-8453, 2019.

Zhixin Wang and Kui Jia. Frustum convnet: Sliding frustums to aggregate local point-wise features for amodal 3d object detection. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems , pp. 1742-1749. IEEE, 2019.

Xiaopei Wu, Liang Peng, Honghui Yang, Liang Xie, Chenxi Huang, Chengqi Deng, Haifeng Liu, and Deng Cai. Sparse fuse dense: Towards high quality 3d detection with depth completion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 5418-5427, June 2022.

Danfei Xu, Dragomir Anguelov, and Ashesh Jain. Pointfusion: Deep sensor fusion for 3d bounding box estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 244-253, 2018.

Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors , 18(10): 3337, 2018.

Zetong Yang, Yanan Sun, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Std: Sparse-to-dense 3d object detector for point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 1951-1960, 2019.

Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 11040-11048, 2020.

Zetong Yang, Li Jiang, Yanan Sun, Bernt Schiele, and Jiaya Jia. A unified query-based paradigm for point cloud understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 8541-8551, 2022.

Jin Hyeok Yoo, Yecheol Kim, Jisong Kim, and Jun Won Choi. 3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. In European Conference on Computer Vision , pp. 720-736. Springer, 2020.

Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q. Weinberger. Pseudo-lidar++: Accurate depth for 3d object detection in autonomous driving. In International Conference on Learning Representations , pp. 1-13, 2020.

Qijian Zhang, Junhui Hou, and Yue Qian. Multi-view vision-to-geometry knowledge transfer for 3d point cloud shape analysis. arXiv preprint arXiv:2207.03128 , 2022.

Yanan Zhang, Jiaxin Chen, and Di Huang. Cat-det: Contrastively augmented transformer for multimodal 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 908-917, 2022.

Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 18953-18962, 2022.

Yifan Zhang, Junhui Hou, and Yixuan Yuan. A comprehensive study of the robustness for lidar-based 3d object detectors against adversarial attacks. International Journal of Computer Vision , 2023.

Yifan Zhang, Qijian Zhang, Zhiyu Zhu, Junhui Hou, and Yixuan Yuan. Glenet: Boosting 3d object detectors with generative label uncertainty estimation. International Journal of Computer Vision , pp. 1-21, 2023.

Yifan Zhang, Zhiyu Zhu, and Junhui Hou. Spatial-temporal enhanced transformer towards multi-frame 3d object detection. arXiv preprint arXiv:2307.00347 , 2023.

Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2881-2890, 2017.

Wu Zheng, Weiliang Tang, Li Jiang, and Chi-Wing Fu. Se-ssd: Self-ensembling single-stage object detector from point cloud. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 14494-14503, 2021.

Y. Zhou and O. Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 4490-4499, 2018.

A Appendix

In this appendix, we provide the details omitted from the manuscript due to space limitation. We organize the appendix as follows.

Section A.1: Implementation details.

Section A.2: More quantitative results.

Section A.3: Experimental results on Waymo dataset.

Section A.4: More ablation studies.

Section A.5: Efficiency analysis.

Section A.6: Visual results of 3D object detection.

Section A.7: Visual results of 2D semantic segmentation.

Section A.8: Details on the official KITTI test leaderboard.

A.1 Implementation Details

Network Architecture. Fig. A1 illustrates the architectures of the point cloud and image backbone networks. For the encoder of the point cloud branch, we further show the details of multi-scale grouping (MSG) network in Table A1. Following 3DSSD [49], we take key points sampled by the 3 rd SA layer to generate vote points and estimate the 3D box. Then we feed these 3D boxes and features output by the last FP layer to the refinement stage. Besides, we adopt density-aware RoI grid pooling [11] to encode point density as an additional feature. Note that 3D center estimation [9] aims to learn the relative position of each foreground point to the object center, while the 3D box is estimated based on sub-sampled points. Thus, the auxiliary task of 3D center estimation differs from 3D box estimation and can facilitate learning structure-aware features of objects.

Training Details. Through the experiments on KITTI dataset, we adopted Adam [13] ( β 1 =0.9, β 2 =0.99) to optimize our UPIDet. We initialized the learning rate as 0.003 and updated it with the one-cycle policy [37]. And we trained the model for a total of 80 epochs in an end-to-end manner. In our experiments, the batch size was set to 8, equally distributed on 4 NVIDIA 3090 GPUs. We kept the input image with the original resolution and padded it to the size of 1248 × 376 , and down-sampled the input point cloud to 16384 points during training and inference. Following the common practice, we set the detection range of the x , y , and z axis to [0m, 70.4m], [-40m, 40m] and [-3m, 1m], respectively.

Data Augmentation. We applied common data augmentation strategies at global and object levels. The global-level augmentation includes random global flipping, global scaling with a random scaling factor between 0.95 and 1.05, and global rotation around the z -axis with a random angle in the range of [ -π/ 4 , π/ 4 ]. Each of the three augmentations was performed with a 50% probability for each sample. The object-level augmentation refers to copying objects from other scenes and pasting them to current scene [47]. In order to perform sampling synchronously on point clouds and images, we utilized the instance masks provided in [29]. Specifically, we pasted both the point clouds and pixels of sampled objects to the point cloud and images of new scenes, respectively.

A.2 More quantitative Results

Performance on KITTI Val Set. We also reported the performance of our UPIDet on all three classes of the KITTI validation set in Table A2, where it can be seen that our UPIDet also achieves the highest mAP of 77.73%, which is obviously higher than the second best method CAT-Det.

Input RGBImage

(H, W,3)

(N,4)

C1

SA (D-FPS)

Level 1

(H /2,W /2,64)

Bi-Propag

(N/4, 64)

MaxPool&C2

SA(D-FPS&S-FPS)

Level 2

(H/4,W/4,128)

Bi-Propag

(N/16, 128)

SA(D-FPS&

S-FPS)

(H /8,W

Level 3

Bi-Propag

(N/32, 256)

/8,256)

C4&C5&PPM

FP

UpSampling x2

Level 4

(H/4,W/4,256)

Bi-Propag

(N/16, 128)

UpSampling x2

FP

Level 5

(H /2,W /2,64)

Bi-Propag

(N/4, 64)

UpSampling x2

FP

Level 6

(H,W,64)

Bi-Propag

(N,64)

Conv

↓Conv

MLP

MLP

(H,W,3)

(H,W,C)

(N,C)

(N,3)

NLC Map Estimation

2D Semantic Segmentation

3D Semantic Segmentation

3D Center Estimation

Performance of Single-Class Detector. Quite a few methods [6, 60] train models only for car detection. Empirically, the single-class detector performs better in the car class compared with multi-class detectors. Therefore, we also provided performance of UPIDet trained only for the car class, and compared it with several state-of-the-art methods in Table A3.

Generalization to Asymmetric Backbones. As shown in Fig. A1, we originally adopted an encoder-decoder network in the LiDAR branch that is architecturally similar to the image backbone. Nevertheless, it is worth clarifying that our approach is not limited to symmetrical structures and

can be generalized to different point-based backbones. Here, we replaced the 3D branch of the original framework with an efficient single-stage detector-SASA [2], using a backbone only with the encoder in the LiDAR branch, which is asymmetric with the encoder-decoder structure of the image backbone. Accordingly, the proposed bidirectional propagation is only performed between the 3D backbone and the encoder of the 2D image backbone. The experimental results are shown in Table A4. We can observe that the proposed method works well even when the two backbones are asymmetric, which demonstrates the satisfactory generalization ability of our method for different LiDAR backbones.

A.3 Results on Waymo Open Dataset

The Waymo Open Dataset [38] is a large-scale dataset for 3D object detection. It contains 798 sequences (15836 frames) for training, and 202 sequences (40077 frames) for validation. According to the number of points inside the object and the difficulty of annotation, the objects are further divided into two difficulty levels: LEVEL_1 and LEVEL_2. Following common practice, we adopted the metrics of mean Average Precision (mAP) and mean Average Precision weighted by heading accuracy (mAPH), and reported the performance on both LEVEL_1 and LEVEL_2. We set the detection range to [-75.2m, 75.2m] for x and y axis, and [-2m, 4m] for z axis. Following [41] and [1], the training on Waymo dataset consists of two stages to allow flexible augmentations. First, we only trained the LiDAR branch without image inputs and bidirectional propagation for six epochs. We enabled the copy-and-paste augmentation in this stage. Then, we trained the whole pipeline for another 30 epochs, during which the copy-and-paste is disabled. Note that the image semantic segmentation head is disabled, since ground-truth segmentation maps are not provided [38].

As shown in Table A5, our method achieves substantial improvement compared with previous state-ofthe-arts. Particularly, unlike existing approaches including PointAugmenting [41] and TransFusion [1] where the camera backbone is pre-trained on other datasets and then frozen, we trained the entire

pipeline in an end-to-end manner. It can be seen that even without the 2D segmentation auxiliary task, our method still achieves higher accuracy under all scenarios except 'Ped L2', demonstrating its advantage.

A.4 More Ablation Studies

Effect of Semantic-guided Point Sampling. When performing downsampling in the SA layers of the point cloud branch, we adopted S-FPS [2] to explicitly preserve as many foreground points as possible. We report the percentage of sampled foreground points and instance recall (i.e., the ratio of instances that have at least one point) in Table A6, where it can be seen that exploiting supplementary semantic features from images leads to substantial improvement of the ratio of sampled foreground points and better instance recall during S-FPS.

Influence on 2D Semantic Segmentation. We also aimed to demonstrate that the 2D-3D joint learning paradigm benefits not only the 3D object detection task but also the 2D semantic segmentation task. As shown in Table A7, the deep interaction between different modalities yields an improvement of 4.42% mIoU. The point features can naturally complement RGB image features by providing 3D geometry and semantics, which are robust to illumination changes and help distinguish different classes of objects, for 2D visual information. The results suggest the potential of joint training between 3D object detection and more 2D scene understanding tasks in autonomous driving.

Conditional Analysis. To better figure out where the improvement comes from when using additional image features, we compared UPIDet with the single-modal detector on different occlusion levels and distant ranges. The results shown in Table A8 and Table A9 include separate APs for objects belonging to different occlusion levels and APs for moderate class in different distance ranges. For car detection, our UPIDet achieves more accuracy gains for long-distance and highly occluded objects, which suffer from the sparsity of observed LiDAR points. The cyclist and pedestrian are much more difficult categories on account of small sizes, non-rigid structures, and fewer training samples. For these two categories, UPIDet still brings consistent and significant improvements on different levels even in extremely difficult cases.

Generalization to Sparse LiDAR Signals. We also compared our UPIDet with the single-modal baseline on LiDAR point clouds with various sparsity. In practice, following Pseudo-LiDAR++ [52], we simulated the 32-beam, 16-beam, and 8-beam LiDAR signals by selecting LiDAR points whose

elevation angles fall within specific intervals. As shown in Table A10, the proposed UPIDet outperforms the single-modal baseline under all settings. The consistent improvements suggest our method can generalize to sparser signals. Besides, the proposed UPIDet significantly performs better than the baseline in the setting of LiDAR signals with fewer beams, demonstrating the effectiveness of our method in exploiting the supplementary information in the image domain.

Robustness against Input Corruption. We also conducted extensive experiments to verify the robustness of our UPIDet to sensor perturbation. Specifically, we added Gaussian noises to the reflectance value of points or RGB images. Fig. A2 shows that the mAP value of our cross-modal UPIDet is consistently higher than that of the single-modal baseline and decreases slower with the LiDAR noise level increasing. Particularly, as listed in Table A11, when the variance of the LiDAR noise is set to 0.15, the perturbation affects our cross-modal UPIDet much less than the single-modal detector. Besides, even applying the corruption to both LiDAR input and RGB images, the mAP value of our UPIDet only drops by 2.49%.

Effectiveness of Multi-stage Interaction. As mentioned before, both 2D and 3D backbones adopt an encoder-decoder structure, and we perform bidirectional feature propagation at both downsampling and upsampling stages. Here, we conducted experiments to verify the superiority of the multi-stage interaction over single-stage interaction. As shown in Table A12, only performing the bidirectional feature propagation in the encoder (i.e., Table A12 (b)) or the decoder (i.e., Table A12 (c)) leads to worse performance than that of performing the module in both stages (i.e., Table A12 (d)).

A.5 Efficiency Analysis

We also compared the inference speed and number of parameters of the proposed UPIDet with state-of-the-art cross-modal approaches in Table A13. Our UPIDet has about the same number of parameters as CAT-Det [54], but a much higher inference speed at 9.52 frames per second on a single GeForce RTX 2080 Ti GPU. In general, our UPIDet is inevitably slower than some singlemodal detectors, but it achieves a good trade-off between speed and accuracy among cross-modal approaches.

A.6 Visual Results of 3D Object Detection

In Figure A3, we present the qualitative comparison of detection results between the single-modal baseline and our UPIDet. We can observe that the proposed UPIDet shows better localization capability than the single-modal baseline in challenging cases. Besides, we also show qualitative results of UPIDet on the KITTI test split in Figure A4. We can clearly observe that our UPIDet performs well in challenging cases, such as pedestrians and cyclists (with small sizes) and highlyoccluded cars.

A.7 Visual Results of 2D Semantic Segmentation

Several examples are shown in Figure A5. For distant cars in the first and the second row as well as the pedestrian in the sixth row, the size of objects is small and PSPNet tends to treat them as background, while our UPIDet is able to correct such errors. In the third row, our UPIDet finds the dim cyclist missed by PSPNet. Our UPIDet also performs better for the highly occluded objects as shown in the fourth and the fifth lines. This observation shows the 3D feature representations extracted from point clouds can boost 2D semantic segmentation, since the image-based method is sensitive to illumination and can hardly handle corner cases with only single-modal inputs.

A.8 Details on Official KITTI Test Leaderboard

We submitted the results of our UPIDet to the official KITTI website, and it ranks 1 st on the 3D object detection benchmark for the cyclist class. Figure A6 shows the screenshot of the leaderboard. Figure A7 illustrates the precision-recall curves along with AP scores on different categories of the KITTI test set. The samples of the KITTI test set are quite different from those of training/validation set in terms of scenes and camera parameters, so the impressive performance of our UPIDet on the test set demonstrates it also achieves good generalization.

80000000

0000 0000 0

(a) Image

(c)PSPNet

(d) Ours

Method

Setting

Code

Moderate

Easy

Hard

Runtime

Environment

BiProDet

74.32 %

86.74 %

67.45 %

0.1 s

GPU @2.5 Ghz(Python +C/C++)

2

74.12 %

88.82%

66.84 %

0.1 s

1core@2.5Ghz(C/C++)

3

CasA++

code

73.79 %

87.76 %

66.84 %

0.1 s

......

1 core @2.5 Ghz(C/C++)

4

CasA

code

73.47 %

87.91 %

66.17 %

0.1 s

1 core @ 2.5 Ghz (C/C++)

5

SGNet

70.40 %

86.75 %

62.73 %

0.09 s

GPU @ 2.5 Ghz (Python)

6

HMFI

code

70.37 %

84.02 %

62.57 %

0.1 s

1core @2.5Ghz(C/C++)

CAD

69.94 %

......

84.68 %

62.21 %

0.1 s

GPU @2.5Ghz(Python +C/C++)

8

SARFE

......

69.67 %

84.88%

62.26 %

0.03 s

1 core @2.5 Ghz(C/C++)

9

EQ-PVRCNN

code

69.10 %

85.41 %

62.30 %

0.2 s

GPU @2.5Ghz(Python +C/C++)

10

VoCo

69.00 %

82.74 %

62.46 %

0.1 s

1core @2.5Ghz(Python +C/C++)

1.0

1.0

80

0.8

0.8

Ours[82.97]

Precision

0.6

PDV[90.43]

Precision

0.6

PDV[81.86]

PDV[77.36]

3DSSD[88.36]

3DSSD[79.57]

3DSSD[74.55]

PV-RCNN[76.82]

0.4

Part-A²[87.81]

0.4

Part-A²[78.49]

0.4

Part-A2[73.51]

CAT-Det[89.87]

CAT-Det[81.32]

CAT-Det[76.68]

0.2

F-Convnet[82.11]

PointPainting[82.11]

0.2 -

PointPainting[71.70]

F-Convnet[71.70]

0.2

PointPainting[67.08]

F-Convnet[67.08]

F-PointNet[82.19]

F-PointNet[69.79]

F-PointNet[60.59]

AVOD-FPN[83.07]

AVOD-FPN[71.76]

AVOD-FPN[65.73]

0.0+

0.0

0.2

0.4

Recall

0.6

0.8

1.0

0.0+

0.0

0.2

0.4

Recall

0.6

0.8

1.0

0.0

0.0

0.2

0.4

Recall

0.6

8'0

(a) Easy car category

(b)Moderate car category

(c) Hard car category

1.0

1.0

Ours[48.77]

1.0

Ours[46.12]

3DSSD[44.27]

PDV[40.56]

3DSSD[40.23]

PDV[38.46]

80

0.8

PV-RCNN[43.29]

0.8

PV-RCNN[40.29]

STD[42.47]

Part-A2[43.35]

Part-A²[40.06]

PDV[47.80]

Precision

0.6

CAT-Det[45.44]

CAT-Det[41.94]

0.4

PV-RCNN[52.17]

Part-A²[53.10]

STD[53.29]

3DSSD[54.64]

0.4

AVOD-FPN[42.27]

F-Convnet[40.97]

PointPainting[40.97]

F-PointNet[42.15]

0.4

F-Convnet[37.87]

PointPainting[37.87]

F-PointNet[38.08]

AVOD-FPN[39.04]

CAT-Det[54.26]

0.2

F-Convnet[50.32]

PointPainting[50.32]

0.2

0.2

F-PointNet[50.53]

AVOD-FPN[50.46]

0.0

0.0

0.2

0.4

Recall

0.6

0.8

1.0

0.0

0.2

0.4

Recall

0.6

0.8

1.0

0.0

0.0

0.2

0.4

Recall

0.6

80

1.0

(d) Easy pedestrian category

(e) Moderate pedestrian category

(f) Hard pedestrian category

0.8

0.8

0.8

Ours[86.74]

Ours[74.32]

Ours[67.45]

Precision

0.6

PDV[83.04]

Precision

0.6

PDV[67.81]

PDV[60.46]

3DSSD[82.48]

[0695]95500

PV-RCNN[78.60]

PV-RCNN[57.65]

0.4

[698]5

Part-A²[79.17]

0.4

STD[61.59]

Part-A²[63.52]

0.4

STD[55.30]

Part-A²[56.93]

0.2

CAT-Det[83.68]

F-Convnet[77.63]

PointPainting[77.63]

0.2

F-Convnet[63.78]

CAT-Det[68.81]

0.2

CAT-Det[61.45]

F-Convnet[55.89]

PointPainting[55.89]

F-PointNet[72.27]

F-PointNet[56.12]

F-PointNet[49.01]

AVOD-FPN[63.76]

AVOD-FPN[50.55]

AVOD-FPN[44.93]

0.0+

0.0

02

0.4

0.6

0.8

1.0

0.0+

0.0

0.2

0.4

0.6

0.8

1.0

0.0+

0.0

0.2

0.4

0.6

0.8

1.0

Recall

Recall

Recall

(h) Moderate cyclist category

(i) Hard cyclist category

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel cross-model 3D detector BiProDet, which leverages the information from image domain in two ways. First, it proposes point-to-pixel bidirectional propagation strategy to boost the representation ability of the point cloud backbone. Second, it introduces NLC map estimation as an auxiliary task for training to learn local spatial-aware features of image branch to supply sparse point clouds.  The BiProDet exhibits consistent and more significant improvements on the \"moderate\" and \"hard\" levels, where objects are distant or highly occluded with sparse points and ranked 1st on the KITTI 3D detection benchmark for the cyclist class.\nIn my opinion, this paper discusses of the reinforcement mechanism of image branch for 3D detector deeply, and a bidirectional propagation is designed to enhance the performance of the backbone, which is very innovative. And a wealth of ablation experiments has been done to prove the effectiveness of the algorithm proposed in this paper.\n\nSTRENGTHS:\nThe paper has several strengths in terms of originality, quality, clarity, and significance. \n\n1. It introduces a 2D auxiliary task called normalized local coordinate (NLC) map estimation to improve the performance of the cross-modal detector by providing the relative position of each pixel inside the object.\n\n2. The paper proposes a novel point-to-pixel feature propagation mechanism that allows 3D geometric features from LiDAR point clouds to enhance the representation ability of 2D image learning.\n\n3. It provides a analysis of the performance bottlenecks of single-modal\n   detectors.\n\nWEAKNESSES:\n1. The motivation is weak. As the proposed method point-to-pixel is just a fusion strategy in cross-modal modeling, which also been explored in many works such as BEVFusion[1]. So it is relatively weak to serve as the motivation of the paper.\n\n2. The proposed specific 2D auxiliary tasks have been explored in TiG-BEV [2]. Althogh the format of the 2D tasks is not the same, the core of the methods is similar.\n\n[1] https://github.com/mit-han-lab/bevfusion\n\n[2] TiG-BEV: Multi-view BEV 3D Object Detection via Target Inner-Geometry Learning\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies 3D object detection with multi-modal inputs (image and point cloud). This study uses a two-stage 3D object detection pipeline and proposes two approaches for further performance improvements. Firstly, the authors propose a bidirectional feature module to fuse point cloud and image features. Secondly, the authors propose several 2D and 3D auxiliary tasks to improve representation learning. The experiments validated the proposed approaches on KITTI and Waymo Open Dataset.\n\nSTRENGTHS:\n1. the ablation study is well-designed and extensive to validate the effectiveness of the proposed NLC map representation and fusion network.\n2. the idea of using NLC map is well-motivated.\n3. the performance on KITTI and Waymo Open Dataset is competitive in several metrics.\n\nWEAKNESSES:\n1. the technical novelty is somewhat limited. Bidirectional feature fusion is not new, for example, see \"FFB6D: A Full Flow Bidirectional Fusion Network for 6D Pose Estimation\". The proposed fusion approach is similar to this paper, which is not cited. In addition, using auxiliary loss to improve representation learning is also not very novel.\n2. Auxiliary tasks require extra labels. Do other approaches in Table 1 use semantic masks as supervision?\n3. Compared with [45], the performance in the EASY category is inferior.\n4. Previous fusion-base approaches such as [1] provided evaluations on the Nuscenes dataset yet this paper does not.\n\nQUESTIONS:\n1. Why the performance on the EASY category of KITTI is inferior compared with [45]? Can the authors further improve it?\n2. Can the authors provide quantitative comparisons with [1] on the Nuscenes dataset?\n3. have the authors attempted other fusion approaches instead of the proposed one? What will be the performance difference?\n\nLIMITATIONS:\nThe authors have not addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper propose a multi-modal usion-based 3d object detector named BiProDet. BiProDet adopts a bidirectional feature propagation mechanism, i.e.,  point-to-pixel module and pixel-to-point module. Besides, BiProDet propose a new auxiliary task called Normalized Local Coordinate (NLC) map.

STRENGTHS:
The paper is well written, and the ablation study demonstrate the effectiveness of the proposed method.

WEAKNESSES:
1. Comparison with some closely related work is missing. For example, the proposed method is similar to EPNet++ ( [21] in the submitted manuscript) in several aspects. Both of them design bidirectional feature fusion and adopt semantic segmentation as an auxiliary task.
2. BEV representation is receiving increasing attention recently. However,  a comparison with these methods is missing.

QUESTIONS:
1. The proposed method is closely related to EPNet++. It would be better to provide a more detailed comparison between these two approaches.
2. BEV representation is receiving increasing attention recently and achieves sota results on many benchmarks. It would be better to compare the proposed method with BEV-base fusion methods (e.g. BEVFusion).
3. It would be more convincing to evaluate the proposed method on more datasets besides KITTI,  e.g., Nuscenes.
3. Why NLC is more beneficial for small objects such as ped. and cyc.?

LIMITATIONS:
The proposed fusion method is not as flexible as the BEV representation. And the comparison with BEV-based methods is missing.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work addresses the task of 3D object detection from LiDAR and cameras. Their main contribution is developing a joint 2D and 3D stream architecture, with simple bidirectional feature flow in the backbones. To improve this, they propose to predict NLC maps in the image stream. The proposed components demonstrate performance improvement on the KITTI dataset.\n\nSTRENGTHS:\n- The paper is clear and easy to read, with diagrams to explain crucial parts.\n- The NLC prediction significantly improves performance, supporting the core hypothesis in the paper.\n- As opposed to some existing works with joint 2D-3D backbones, the proposed bidirectional feature flow is simple without bells and whistles and complicated components.\n- It is a non-trivial and interesting result that point-to-pixel flow during training only can still improve 3D detection performance, showing that 2D gradients can help 3D model learning.\n\nWEAKNESSES:\n- The statement “However, there is no evidence that these methods actually enhance the representation capability of the 3D LiDAR backbone network, which is also one of the most critical influencing factors.” in L41 - L46 (as well as its encompassing paragraph) is unclear to me and seems unsubstantiated. Many existing works (PointPainting, AutoAlign, etc) have enhanced the point cloud/3D voxels with 2D semantics before/while using the 3D backbone network and have demonstrated significant improvements in detection metrics by training end-to-end. \n- I am slightly concerned that all of the ablative results shown are on the KITTI dataset, which is a small-scale dataset with low diversity. I would appreciate some ablations on at least the key components (NLC prediction, point-to-pixel flow) on the Waymo dataset. Nonetheless, it is impressive that a point-based method can achieve such strong performance on Waymo.\n- Point-to-pixel flow during training only does improve 3D detection performance, which seems to demonstrate that training a 2D CNN can help the 3D network learn better features. However, this is slightly confounded with the possibility that *NLC* supervision is what is improving 3D detection when point-to-pixel flow is added, not the 2D CNN. It would strengthen the paper if point-to-pixel flow improves performance even when 3D NLC supervision (similar to that in PartA2) is done.\n\nQUESTIONS:\n- Please reference the weaknesses section for questions.\n- I also want to ask if any augmentations are done in 2D. For instance, LR flipping may introduce ambiguity in predicting left vs right side of an object.\n\nLIMITATIONS:\nAuthors do not appear to have included a limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method for multimodal (image and LiDAR) 3D object detection. The main purpose of the proposed method is to enhance the image branch. The authors design the task of NLC Map estimation, which is to predict the normalized local coordinates of points within a ground-truth box. The prediction happens on the image plane and the points are projected (carrying their NLC labels). In addition, image semantic segmentation is also adopted as an auxiliary task for the image branch. The image and LiDAR branch fuse their features during the forward of their backbones. The LiDAR branch predicts 3D instances based on the RPN proposals and RoI-pooled fused features.\n\nSTRENGTHS:\n1. Although the idea of predicting local coordinates within a bounding box is similar to Part-A^2, it is interesting to see that it can effectively enhance the image branch. I am also wondering if additionally introducing local coordinate prediction for the LiDAR branch can further boost performance.\n2. The proposed method achieves SOTA results on the KITTI dataset, and the effectiveness of the auxiliary tasks are proved by ablation study.\n\nWEAKNESSES:\n1. The experiments are only conducted on the KITTI dataset, in fact, there are a bunch of recent multimodal 3D detection methods that provide results on the NuScenes dataset, so it would be more convincing with comparisons on NuScenes.\n2. The bidirectional feature propagation module is simply gathering features from another modality by coordinate projection and feature transformation, which is not significantly novel.\n\nQUESTIONS:\nWould it be more straightforward to use local coordinate prediction in the LiDAR branch like Part-A^2? NLC estimation can also be performed for the points if they are not projected to the image plane?\n\nLIMITATIONS:\nThe authors did not address the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}