SlowGuess/NeurIPS_2023_OCR huggingface dataset:
optimize the depth estimator and LiDAR-based detector simultaneously in pseudo-LiDAR based methods although it can be implemented in a somewhat complex way [30]; and 3) pseudo-LiDAR representations require ground-truth dense depth images as supervision, which may not be available in reality, whereas our method does not require such information.
Semantic Segmentation. Leveraging semantics of 3D point clouds has demonstrated effectiveness in point-based detectors, owing to the explicit preservation of foreground points at downsampling [55, 2]. Therefore, we introduce the auxiliary semantic segmentation tasks not only in the point cloud branch but also in the image branch to exploit richer semantic information. Besides, additional semantic features extracted from images can facilitate the network to distinguish true positive candidate boxes from false positives.
4.3 Loss Function
The NLC map estimation task is optimized with the loss function defined as
$$
L _ {N L C} = \frac {1}{N _ {\text {p o s}}} \sum_ {i} ^ {N} \left\| \boldsymbol {p} _ {i} ^ {N L C} - \hat {\boldsymbol {p}} _ {i} ^ {N L C} \right\| _ {H} \cdot \mathbf {1} _ {\boldsymbol {p} _ {i}}, \tag {6}
$$
where  $\pmb{p}_i$  is the  $i^{th}$  LiDAR point,  $N_{pos}$  is the number of foreground LiDAR points,  $\| \cdot \|_H$  is the Huber-loss,  $\mathbf{1}_{\pmb{p}_i}$  indicates the loss is calculated only with foreground points, and  $\pmb{p}_i^{NLC}$  and  $\hat{\pmb{p}}_i^{NLC}$  are the NLCs of ground-truth and prediction at corresponding pixel for foreground points.
We use the standard cross-entropy loss to optimize both 2D and 3D semantic segmentation, denoted as  $L_{sem}^{2D}$  and  $L_{sem}^{3D}$  respectively. And the loss of center estimation is computed as
$$
L _ {c t r} = \frac {1}{N _ {p o s}} \sum_ {i} ^ {N} \| \Delta \boldsymbol {p} _ {i} - \Delta \hat {\boldsymbol {p}} _ {i} \| _ {H} \cdot \mathbf {1} _ {\boldsymbol {p} _ {i}}, \tag {7}
$$
where  $\Delta p_{i}$  is the target offsets from points to the corresponding object center, and  $\Delta \hat{p}_i$  denotes the output of center estimation head.
Besides the auxiliary tasks, we define the loss of the RPN stage  $L_{rpn}$  following [2]. In addition, we also adopt commonly used proposal refinement loss  $L_{rcnn}$  as defined in [33]. In all, the loss for optimizing the overall pipeline in an end-to-end manner is written as
$$
L _ {t o t a l} = L _ {r p n} + L _ {r c n n} + \lambda_ {1} L _ {N L C} + \lambda_ {2} L _ {s e m} ^ {2 D} + \lambda_ {3} L _ {s e m} ^ {3 D} + \lambda_ {4} L _ {c t r}, \tag {8}
$$
where  $\lambda_1, \lambda_2, \lambda_3$ , and  $\lambda_4$  are hyper-parameters that are empirically set to 1.
Datasets and Metrics. We conducted experiments on the prevailing KITTI benchmark dataset, which contains two modalities of 3D point clouds and 2D RGB images. Following previous works [33], we divided all training data into two subsets, i.e., 3712 samples for training and the rest 3769 for validation. Performance is evaluated by the Average Precision (AP) metric under IoU thresholds of 0.7, 0.5, and 0.5 for car, pedestrian, and cyclist categories, respectively. We computed APs with 40 sampling recall positions by default, instead of 11. For the 2D auxiliary task of semantic segmentation, we used the instance segmentation annotations as provided in [29]. Besides, we also conducted experiments on the Waymo Open Dataset (WOD) [38], which can be found in Appendix A.3.
Implementation Details. For the image branch, we used ResNet18 [10] as the backbone encoder, followed by a decoder composed of pyramid pooling module [59] and several upsampling layers to give the outputs of semantic segmentation and NLC map estimation. For the point cloud branch, we deployed a point-based network like [49] with extra FP layers and further applied semantic-guided farthest point sampling (S-FPS) [2] in SA layers. Thus, we implemented bidirectional feature propagation between the top of each SA or FP layer and their corresponding locations at the image branch. Please refer to Appendix A.1 for more details.
Table 1: Performance comparisons on the KITTI test set, where the best and the second best results are highlighted in bold and underlined, respectively.
Method	Modality	3D Car (IoU=0.7)	3D Ped. (IoU=0.5)	3D Cyc. (IoU=0.5)	mAP
Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
PointRCNN [32]	LiDAR	86.96	75.64	70.70	47.98	39.37	36.01	74.96	58.82	52.53	60.33
PointPillars [15]	LiDAR	82.58	74.31	68.99	51.45	41.92	38.89	77.10	58.65	51.92	60.65
TANet [22]	LiDAR	84.39	75.94	68.82	53.72	44.34	40.49	75.70	59.44	52.53	61.71
IA-SSD [55]	LiDAR	88.34	80.13	75.04	46.51	39.03	35.60	78.35	61.94	55.70	62.29
STD [48]	LiDAR	87.95	79.71	75.09	53.29	42.47	38.35	78.69	61.59	55.30	63.60
Point-GNN [35]	LiDAR	88.33	79.47	72.29	51.92	43.77	40.14	78.60	63.48	57.08	63.90
Part-A2 [34]	LiDAR	87.81	78.49	73.51	53.10	43.35	40.06	79.17	63.52	56.93	63.99
PV-RCNN [33]	LiDAR	90.25	81.43	76.82	52.17	43.29	40.29	78.60	63.71	57.65	64.91
3DSSD [49]	LiDAR	88.36	79.57	74.55	54.64	44.27	40.23	82.48	64.10	56.90	65.01
HotSpotNet [4]	LiDAR	87.60	78.31	73.34	53.10	45.37	41.47	82.59	65.95	59.00	65.19
PDV [11]	LiDAR	90.43	81.86	77.36	47.80	40.56	38.46	83.04	67.81	60.46	65.31
MV3D [5]	LiDAR+RGB	74.97	63.63	54.00	-	-	-	-	-	-	-
MMF [19]	LiDAR+RGB	88.40	77.43	70.22	-	-	-	-	-	-	-
AVOD-FPN [14]	LiDAR+RGB	83.07	71.76	65.73	50.46	42.27	39.04	63.76	50.55	44.93	56.84
F-PointNet [28]	LiDAR+RGB	82.19	69.79	60.59	50.53	42.15	38.08	72.27	56.12	49.01	57.86
PointPainting [40]	LiDAR+RGB	82.11	71.70	67.08	50.32	40.97	37.84	77.63	63.78	55.89	60.81
F-ConvNet [44]	LiDAR+RGB	87.36	76.39	66.69	52.16	43.38	38.80	81.98	65.07	56.54	63.15
CAT-Det [54]	LiDAR+RGB	89.87	81.32	76.68	54.26	45.44	41.94	83.68	68.81	61.45	67.05
EQ-PVRCNN [50]	LiDAR+RGB	90.13	82.01	77.53	55.84	47.02	42.94	85.41	69.10	62.30	68.03
UPIDet (Ours)	LiDAR+RGB	89.13	82.97	80.05	55.59	48.77	46.12	86.74	74.32	67.45	70.13
Table 2: Ablative experiments on different feature exploitation and propagation schemes.
Exp.	Method	3D Car (IoU=0.7)	3D Ped. (IoU=0.5)	3D Cyc. (IoU=0.5)	mAP
Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
(a)	Single-Modal	91.92	85.22	82.98	68.82	61.47	56.39	91.93	74.56	69.58	75.88
(b)	Point-to-pixel	91.83	85.11	82.91	70.66	63.57	58.19	92.78	76.31	71.77	77.01
(c)	Pixel-to-point	92.07	85.79	82.95	69.46	65.53	59.56	94.39	75.24	72.23	77.47
(d)	Bidirectional	92.63	85.77	83.13	72.68	67.64	62.25	94.39	77.77	71.47	78.64
We submitted our results to the official KITTI website, and the results are compared in Table 1. Our UPIDet outperforms existing state-of-the-art methods on the KITTI test set by a remarkable margin, with an absolute increase of  $2.1\%$  mAP over the second best method EQ-PVRCNN. Notably, at the time of submission, we achieved the top rank on the KITTI 3D detection benchmark for the cyclist class. We observe that UPIDet exhibits consistent and more significant improvements on the "moderate" and "hard" levels, where objects are distant or highly occluded with sparse points. We attribute these performance gains to our bidirectional feature propagation mechanism, which more adequately exploits complementary information between multi-modalities, and the effectiveness of our proposed 2D auxiliary tasks (as verified in Sec.5.3). Furthermore, actual visual results (presented in Appendix A.6) demonstrate that UPIDet produces higher-quality 3D bounding boxes in various scenes.
We conducted comprehensive ablation studies to validate the effectiveness and explore the impacts of key modules involved in the overall learning framework.
Effectiveness of Feature Propagation Strategies. We performed detailed ablation studies on specific multi-modal feature exploitation and interaction strategies. We started by presenting a single-modal baseline (Table 2(a)) that only preserves the point cloud branch of our UPIDet for both training and inference. Based on our complete bidirectional propagation pipeline (Table2(d)), we explored another two variants as shown in (Table2(b)) and (Table2(c)), solely preserving the point-to-pixel and pixel-to-point feature propagation in our 3D detectors, respectively. Note that in Table2(b), the point-to-pixel feature flow was only enabled during training, and we detached the point-to-pixel module as well as the whole image branch during inference. Empirically, we can draw several conclusions that strongly support our preceding claims. First, the performance of Table 2(a) is the worst among all variants, which reveals the superiority of cross-modal learning. Second, combining Table2(a)
Table 3: Ablative experiments on key modules of our UPIDet. LiDAR: the point cloud branch trained without auxiliary tasks; Img: the 2D RGB image backbone; 3D Seg: 3D semantic segmentation; 3D Ctr: 3D center estimation; 2D Seg: 2D semantic segmentation; NLC: 2D NLC map estimation.
Exp.	LiDAR	3D Seg	3D Ctr	Img	2D Seg	NLC	Car	Ped.	Cyc.	mAP
(a)	✓	87.40	59.02	79.11	75.18
(b)	✓	✓	86.75	59.90	78.57	75.07
(c)	✓	✓	86.56	61.06	78.58	75.40
(d)	✓	✓	✓	86.71	62.23	78.68	75.88
(e)	✓	✓	✓	✓	86.91	63.23	79.00	76.38
(f)	✓	✓	✓	✓	✓	87.10	64.52	79.94	77.18
(g)	✓	✓	✓	✓	✓	✓	87.18	67.52	81.21	78.64
and Table2(b), the mAP is largely boosted from  $75.88\%$  to  $77.01\%$ . Considering that during the inference stage, these two variants have identical forms of input and network structure, the resulting improvement strongly indicates that the representation ability of the 3D LiDAR branch is indeed strengthened. In other words, the joint optimization of a mature CNN architecture with 2D auxiliary task supervision can guide the point cloud branch to learn more discriminative point features. Third, comparing Table2(b) and Table2(c) with Table2(d), we can verify the superiority of bidirectional propagation  $(78.64\%)$  over the unidirectional schemes  $(77.01\%$  and  $77.47\%)$ . If we particularly pay attention to Table2(c) and Table2(d), we can conclude that our newly proposed point-to-pixel feature propagation direction further brings a  $1.17\%$  mAP increase based on the previously explored pixel-to-point paradigm.
Effectiveness of Image Branch. Comparing Table 3 (d) with Tables 3 (e)-(g), the performance is stably boosted from  $75.88\%$  to  $78.64\%$ , as the gradual addition of the 2D image branch and two auxiliary tasks including 2D NLC map estimation and semantic segmentation. These results indicate that the additional semantic and geometric features learned from the image modality are indeed effective supplements to the point cloud representation, leading to significantly improved 3D object detection performances.
Effectiveness of Auxiliary Tasks. Comparing Table 3 (a) with Tables 3 (b)-(d), the mAP is boosted by  $0.70\%$  when incorporating 3D semantic segmentation and 3D center estimation, and can be further improved by  $2.76\%$  after introducing 2D semantic segmentation and NLC map estimation (Tables 3 (d)-(g)). However, only integrating image features without 2D auxiliary tasks (comparing results of Tables 3 (d) and (e)) brings limited improvement of  $0.5\%$ . This observation shows that the 2D auxiliary tasks, especially the proposed 2D NLC map estimation, do enforce the image branch to learn complementary information for the detector.
Figure 5: Quantitative results of NLC map estimation on the KITTI val set.
Performance of NLC Map Estimation. The NLC map estimation task is introduced to guide the image branch to learn local spatial-aware features. We evaluated the predicted NLC of pixels containing at least one projected point with mean absolute error (MAE) for each object:
$$
\mathrm {M A E} _ {q} = \frac {1}{N _ {o b j}} \sum_ {i} ^ {N} \left| q _ {\boldsymbol {p} _ {i}} ^ {N L C} - \hat {q} _ {\boldsymbol {p} _ {i}} ^ {N L C} \right| \cdot \mathbf {1} _ {\boldsymbol {p} _ {i}}, q \in \{x, y, z \}, \tag {9}
$$
where  $N_{obj}$  is the number of LiDAR points inside the ground-truth box of the object,  $\mathbf{1}_{\pmb{p}_i}$  indicates that the evaluation is only calculated with foreground points inside the box,  $q_{\pmb{p}_i}^{NLC}$  and  $\hat{q}_{\pmb{p}_i}^{NLC}$  are the
normalized local coordinates of the ground-truth and the prediction at the corresponding pixel for the foreground point. Finally, we obtained the mean value of  $\mathrm{MAE}_q$  for all instances, namely  $\mathrm{mMAE}_q$ . We report the metric over different difficulty levels for three categories on the KITTI validation set in Figure 5. We can observe that, for the car class, the mMAE error is only 0.0619, i.e.,  $\pm 6.19$  cm error per meter. For the challenging pedestrian and cyclist categories, the error becomes larger due to the smaller size and non-rigid shape.
We have introduced a novel cross-modal 3D detector named UPIDet that leverages complementary information from the image domain in two ways. First, we demonstrated that the representational power of the point cloud backbone can be enhanced through the gradients backpropagated from the training loss of the image branch, utilizing a succinct and effective point-to-pixel module. Second, we proposed NLC map estimation as an auxiliary task to facilitate the learning of local spatial-aware representation in addition to semantic features by the image branch. Our approach achieves state-of-the-art results on the KITTI detection benchmark. Furthermore, extensive ablative experiments demonstrate the robustness of UPIDet against sensor noises and its ability to generalize to LiDAR signals with fewer beams. The promising results also indicate the potential of joint training between 3D object detection and other 2D scene understanding tasks. We believe that our novel perspective will inspire further investigations into multi-task multi-modal learning for scene understanding in autonomous driving.
This work was supported in part by the Hong Kong Research Grants Council under Grants CityU 11219422, CityU 11202320, and CityU 11218121, in part by the Hong Kong Innovation and Technology Fund under Grant MHP/117/21, and in part by the CityU Strategic Research Grant CityU 11210523. This work used the computational facilities provided by the Computing Services Centre at the City University of Hong Kong. Besides, we thank the anonymous reviewers for their invaluable feedback that improved our manuscript.
[1] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1090-1099, 2022.
[2] Chen Chen, Zhe Chen, Jing Zhang, and Dacheng Tao. Sasa: Semantics-augmented set abstraction for point-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 1, pp. 221-229, 2022.
[3] Hansheng Chen, Yuyao Huang, Wei Tian, Zhong Gao, and Lu Xiong. Monorun: Monocular 3d object detection by reconstruction and uncertainty propagation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10379-10388, 2021.
[4] Qi Chen, Lin Sun, Zhixin Wang, Kui Jia, and Alan Yuille. Object as hotspots: An anchor-free 3d object detection approach via firing of hotspots. In European Conference on Computer Vision, pp. 68-84. Springer, 2020.
[5] Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1907-1915, 2017.
[6] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 1201-1209, 2021.
[7] Gil Elbaz, Tamar Avraham, and Anath Fischer. 3d point cloud registration for localization using a deep neural network auto-encoder. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4631–4640, 2017.
In this appendix, we provide the details omitted from the manuscript due to space limitation. We organize the appendix as follows.
Section A.1: Implementation details.
Section A.2: More quantitative results.
- Section A.3: Experimental results on Waymo dataset.
Section A.4: More ablation studies.
Section A.5: Efficiency analysis.
Section A.6: Visual results of 3D object detection.
- Section A.7: Visual results of 2D semantic segmentation.
- Section A.8: Details on the official KITTI test leaderboard.
Network Architecture. Fig. A1 illustrates the architectures of the point cloud and image backbone networks. For the encoder of the point cloud branch, we further show the details of multi-scale grouping (MSG) network in Table A1. Following 3DSSD [49], we take key points sampled by the  $3^{rd}$  SA layer to generate vote points and estimate the 3D box. Then we feed these 3D boxes and features output by the last FP layer to the refinement stage. Besides, we adopt density-aware RoI grid pooling [11] to encode point density as an additional feature. Note that 3D center estimation [9] aims to learn the relative position of each foreground point to the object center, while the 3D box is estimated based on sub-sampled points. Thus, the auxiliary task of 3D center estimation differs from 3D box estimation and can facilitate learning structure-aware features of objects.
Table A1: Details of set abstraction layers in the point cloud branch. We report the sampling strategy used in the sampling operation, ball radius of group operation, "nquery" that denotes the number of group points, and dimensions of the unit PointNet layer for multi-scale grouping. The features at different scales are concatenated and dimensionally reduced to the specific output channels.
Layer	Sampling Strategy	Radius	nquery	Feature Dimension	Output Channels
1stSA	D-FPS	[0.2, 0.4, 0.8]	[32, 32, 64]	[[16, 16, 32], [16, 16, 32], [32, 32, 64]]	64
2ndSA	D-FPS & S-FPS	[0.4, 0.8, 1.6]	[32, 32, 64]	[[64, 64, 128], [64, 64, 128], [64, 96, 128]]	128
3rdSA	D-FPS & S-FPS	[1.6, 3.2, 4.8]	[64, 64, 128]	[[128, 128, 256], [128, 196, 256], [128, 256, 256]]	256
Training Details. Through the experiments on KITTI dataset, we adopted Adam [13] ( $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.99$ ) to optimize our UPIDet. We initialized the learning rate as 0.003 and updated it with the one-cycle policy [37]. And we trained the model for a total of 80 epochs in an end-to-end manner. In our experiments, the batch size was set to 8, equally distributed on 4 NVIDIA 3090 GPUs. We kept the input image with the original resolution and padded it to the size of  $1248 \times 376$ , and down-sampled the input point cloud to 16384 points during training and inference. Following the common practice, we set the detection range of the  $x$ ,  $y$ , and  $z$  axis to  $[0m, 70.4m]$ ,  $[-40m, 40m]$  and  $[-3m, 1m]$ , respectively.
Data Augmentation. We applied common data augmentation strategies at global and object levels. The global-level augmentation includes random global flipping, global scaling with a random scaling factor between 0.95 and 1.05, and global rotation around the  $z$ -axis with a random angle in the range of  $[- \pi /4, \pi /4]$ . Each of the three augmentations was performed with a  $50\%$  probability for each sample. The object-level augmentation refers to copying objects from other scenes and pasting them to current scene [47]. In order to perform sampling synchronously on point clouds and images, we utilized the instance masks provided in [29]. Specifically, we pasted both the point clouds and pixels of sampled objects to the point cloud and images of new scenes, respectively.
A.2 More quantitative Results
Performance on KITTI Val Set. We also reported the performance of our UPIDet on all three classes of the KITTI validation set in Table A2, where it can be seen that our UPIDet also achieves the highest mAP of  $77.73\%$ , which is obviously higher than the second best method CAT-Det.
Figure A1: The detailed architecture of 2D and 3D backbones. We adopt ResNet18 as the encoder of the image branch, followed by a decoder with Pyramid Pooling Module (PPM) and several upsampling blocks. C1, C2, C3, C4, and C5 denote convolutional layers of different stages in ResNet. Extra convolutional layers are deployed after each up-sampling layer. For the point cloud branch, we adopt the PointNet++ structure. SA: set abstraction layer, D-FPS: 3D Euclidean distance-based farthest point sampling, S-FPS: semantic-guided farthest point sampling, FP: feature propagation layer, MLP: shared multi-layer perceptron. Besides, "Bi-Propag" denotes the proposed bidirectional feature propagation between the 2D and 3D backbones.
Table A2: Quantitative comparisons on the KITTI validation set under the evaluation metric of 3D Average Precision (AP) calculated with 11 sampling recall positions. We highlight the best and the second best results in bold and underlined, respectively.
Method	Modality	3D Car (IoU=0.7)	3D Ped. (IoU=0.5)	3D Cyc. (IoU=0.5)	mAP
Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
PointPillars [15]	LiDAR	86.46	77.28	74.65	57.75	52.29	47.90	80.05	62.68	59.70	66.53
SECOND [47]	LiDAR	88.61	78.62	77.22	56.55	52.98	47.73	80.58	67.15	63.10	68.06
3DSSD [49]	LiDAR	88.55	78.45	77.30	58.18	54.31	49.56	86.25	70.48	65.32	69.82
PointRCNN [32]	LiDAR	88.72	78.61	77.82	62.72	53.85	50.24	86.84	71.62	65.59	70.67
PV-RCNN [33]	LiDAR	89.03	83.24	78.59	63.71	57.37	52.84	86.06	69.48	64.50	71.65
TANet [22]	LiDAR	88.21	77.85	75.62	70.80	63.45	58.22	85.98	64.95	60.40	71.72
Part-A2 [34]	LiDAR	89.55	79.40	78.84	65.68	60.05	55.44	85.50	69.90	65.48	72.20
AVOD-FPN [14]	LiDAR+RGB	84.41	74.44	68.65	-	58.80	-	-	49.70	-	-
PointFusion [46]	LiDAR+RGB	77.92	63.00	53.27	33.36	28.04	23.38	49.34	29.42	26.98	42.75
F-PointNet [28]	LiDAR+RGB	83.76	70.92	63.65	70.00	61.32	53.59	77.15	56.49	53.37	65.58
CLOCs [26]	LiDAR+RGB	89.49	79.31	77.36	62.88	56.20	50.10	87.57	67.92	63.67	70.50
EPNet [12]	LiDAR+RGB	88.76	78.65	78.32	66.74	59.29	54.82	83.88	65.50	62.70	70.96
CAT-Det [54]	LiDAR+RGB	90.12	81.46	79.15	74.08	66.35	58.92	87.64	72.82	68.20	75.42
UPIDet (Ours)	LiDAR+RGB	89.73	86.40	79.31	71.77	68.49	62.52	89.24	76.91	75.18	77.73
Performance of Single-Class Detector. Quite a few methods [6, 60] train models only for car detection. Empirically, the single-class detector performs better in the car class compared with multi-class detectors. Therefore, we also provided performance of UPIDet trained only for the car class, and compared it with several state-of-the-art methods in Table A3.
Generalization to Asymmetric Backbones. As shown in Fig. A1, we originally adopted an encoder-decoder network in the LiDAR branch that is architecturally similar to the image backbone. Nevertheless, it is worth clarifying that our approach is not limited to symmetrical structures and
Table A3: Comparison with state-of-the-art methods on the KITTI val set for car 3D detection. All results are reported by the average precision with 0.7 IoU threshold. R11 and R40 denotes AP calculated with 11 and 40 recall sampling recall points, respectively.
Method	Modal	AP3D|R11 (%)	AP3D|R40 (%)
Easy	Mod.	Hard	Easy	Mod.	Hard
Voxel R-CNN [6]	LiDAR	89.41	84.52	78.93	92.38	85.29	82.86
PV-RCNN [33]	LiDAR	89.35	83.69	78.7	92.57	84.83	82.69
SA-SSD [9]	LiDAR	90.15	79.91	78.78	93.14	84.65	81.86
SE-SSD [60]	LiDAR	90.21	85.71	79.22	93.19	86.12	83.31
GLENet-VR [57]	LiDAR	89.93	86.46	79.19	93.51	86.10	83.60
MV3D [5]	LiDAR+RGB	-	-	-	71.29	62.68	56.56
3D-CVF [51]	LiDAR+RGB	-	-	-	89.67	79.88	78.47
UPIDet (Ours)	LiDAR+RGB	89.72	86.52	79.34	93.03	86.46	83.73
can be generalized to different point-based backbones. Here, we replaced the 3D branch of the original framework with an efficient single-stage detector—SASA [2], using a backbone only with the encoder in the LiDAR branch, which is asymmetric with the encoder-decoder structure of the image backbone. Accordingly, the proposed bidirectional propagation is only performed between the 3D backbone and the encoder of the 2D image backbone. The experimental results are shown in Table A4. We can observe that the proposed method works well even when the two backbones are asymmetric, which demonstrates the satisfactory generalization ability of our method for different LiDAR backbones.