SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table A4: Bidirectional propagation is also effective when the 2D and 3D backbones are asymmetric. Here we adopt a single-stage detector [2] whose backbone includes only an encoder, which is asymmetric with the encoder-decoder network in the image branch.
Method	3D Car (IoU=0.7)	3D Ped. (IoU=0.5)	3D Cyc. (IoU=0.5)	mAP
Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
SASA	92.17	84.90	82.57	66.75	61.40	56.00	89.91	74.05	69.41	75.24
Ours (SASA)	92.11	85.67	82.99	70.52	63.38	58.16	91.58	74.81	70.22	76.61
Table A5: 3D detection results on the Waymo Open Dataset validation set. “-” denotes that the results are not reported in their papers.
Method	Vehicle L1
mAP mAPH	Vehicle L2
mAP mAPH	Pedestrian L1
mAP mAPH	Pedestrian L2
mAP mAPH	Cyclist L1
mAP mAPH	Cyclist L2
mAP mAPH
PointAugmenting	67.41	-	62.7	-	75.42	-	70.55	-	76.29	-	74.41	-
TransFusion	-	-	-	65.14	-	-	-	64.00	-	-	-	67.40
Ours	78.36	77.91	69.45	69.04	76.32	71.67	65.93	61.81	79.64	78.55	76.36	75.26
A.3 Results on Waymo Open Dataset
The Waymo Open Dataset [38] is a large-scale dataset for 3D object detection. It contains 798 sequences (15836 frames) for training, and 202 sequences (40077 frames) for validation. According to the number of points inside the object and the difficulty of annotation, the objects are further divided into two difficulty levels: LEVEL_1 and LEVEL_2. Following common practice, we adopted the metrics of mean Average Precision (mAP) and mean Average Precision weighted by heading accuracy (mAPH), and reported the performance on both LEVEL_1 and LEVEL_2. We set the detection range to  $[-75.2\mathrm{m}, 75.2\mathrm{m}]$  for  $x$  and  $y$  axis, and  $[-2\mathrm{m}, 4\mathrm{m}]$  for  $z$  axis. Following [41] and [1], the training on Waymo dataset consists of two stages to allow flexible augmentations. First, we only trained the LiDAR branch without image inputs and bidirectional propagation for six epochs. We enabled the copy-and-paste augmentation in this stage. Then, we trained the whole pipeline for another 30 epochs, during which the copy-and-paste is disabled. Note that the image semantic segmentation head is disabled, since ground-truth segmentation maps are not provided [38].
As shown in Table A5, our method achieves substantial improvement compared with previous state-of-the-arts. Particularly, unlike existing approaches including PointAugmenting [41] and TransFusion [1] where the camera backbone is pre-trained on other datasets and then frozen, we trained the entire
pipeline in an end-to-end manner. It can be seen that even without the 2D segmentation auxiliary task, our method still achieves higher accuracy under all scenarios except "Ped L2", demonstrating its advantage.
A.4 More Ablation Studies
Table A6: Effect of the semantic-guided SA layer. Compared with the single-modal baseline, UPIDet can better exploit image semantics and preserve more foreground points during downsampling.
Method	Single-Modal	UPIDet (Ours)	Improvement
SA Layer	FG Rate	Instance Recall	FG Rate	Instance Recall	FG Rate	Instance Recall
Level-2	15.87	97.92	20.70	98.23	+4.83	+0.31
Level-3	29.73	97.35	38.03	97.82	+8.29	+0.47
Table A7: Comparison between our multi-task training methods and the single-modal 2D semantic segmentation baseline (PSPNet). The results show that point features effectively improve the segmentation performance on pedestrian and cyclist classes.
Method	Car	Pes.	Cyc.	mIoU
PSPNet [59]	77.49	30.45	23.83	43.92
UPIDet (Ours)	78.45	36.15	30.42	48.34
Effect of Semantic-guided Point Sampling. When performing downsampling in the SA layers of the point cloud branch, we adopted S-FPS [2] to explicitly preserve as many foreground points as possible. We report the percentage of sampled foreground points and instance recall (i.e., the ratio of instances that have at least one point) in Table A6, where it can be seen that exploiting supplementary semantic features from images leads to substantial improvement of the ratio of sampled foreground points and better instance recall during S-FPS.
Influence on 2D Semantic Segmentation. We also aimed to demonstrate that the 2D-3D joint learning paradigm benefits not only the 3D object detection task but also the 2D semantic segmentation task. As shown in Table A7, the deep interaction between different modalities yields an improvement of  $4.42\%$  mIoU. The point features can naturally complement RGB image features by providing 3D geometry and semantics, which are robust to illumination changes and help distinguish different classes of objects, for 2D visual information. The results suggest the potential of joint training between 3D object detection and more 2D scene understanding tasks in autonomous driving.
Conditional Analysis. To better figure out where the improvement comes from when using additional image features, we compared UPIDet with the single-modal detector on different occlusion levels and distant ranges. The results shown in Table A8 and Table A9 include separate APs for objects belonging to different occlusion levels and APs for moderate class in different distance ranges. For car detection, our UPIDet achieves more accuracy gains for long-distance and highly occluded objects, which suffer from the sparsity of observed LiDAR points. The cyclist and pedestrian are much more difficult categories on account of small sizes, non-rigid structures, and fewer training samples. For these two categories, UPIDet still brings consistent and significant improvements on different levels even in extremely difficult cases.
Table A8: Performance breakdown over different occlusion levels. As defined by the official website of KITTI, occlusion levels 0, 1, and 2 correspond to fully-visible samples, partly-occluded samples, and samples that are difficult to see, respectively.
Class	Car	Pedestrian	Cyclist
Occlusion	Level-0	Level-1	Level-2	Level-0	Level-1	Level-2	Level-0	Level-1	Level-2
Single-Modal	91.98	77.18	55.41	67.44	26.76	6.10	91.23	24.66	1.74
UPIDet (Ours)	92.26	77.44	58.39	74.00	35.13	7.99	92.89	30.02	2.53
Improvement	+0.28	+0.26	+2.97	+6.56	+8.38	+1.89	+1.66	+5.36	+0.79
Generalization to Sparse LiDAR Signals. We also compared our UPIDet with the single-modal baseline on LiDAR point clouds with various sparsity. In practice, following Pseudo-LiDAR++ [52], we simulated the 32-beam, 16-beam, and 8-beam LiDAR signals by selecting LiDAR points whose
Table A9: Performance breakdown over different distances.
Class	Car	Pedestrian	Cyclist
Distance	0-20m	20-40m	40m-Inf	0-20m	20-40m	40m-Inf	0-20m	20-40m	40m-Inf
Single-Modal	96.28	85.21	43.91	71.28	38.42	1.63	93.61	61.56	34.48
UPIDet (Ours)	96.36	86.48	49.88	76.56	45.72	2.46	94.12	67.27	39.10
Improvement	+0.07	+1.27	+5.97	+5.28	+7.30	+0.83	+0.51	+5.71	+4.62
elevation angles fall within specific intervals. As shown in Table A10, the proposed UPIDet outperforms the single-modal baseline under all settings. The consistent improvements suggest our method can generalize to sparser signals. Besides, the proposed UPIDet significantly performs better than the baseline in the setting of LiDAR signals with fewer beams, demonstrating the effectiveness of our method in exploiting the supplementary information in the image domain.
Table A10: Comparison with single-modal baselines under LiDAR signals with different beams, where we report  $\mathrm{AP}_{3\mathrm{D}|\mathrm{R40}}$  on the KITTI validation set.
LiDAR Beams	Modal	Car	Ped.	Cyc.	mAP
64	LiDAR	86.71	62.23	78.68	75.87
LiDAR + RGB	87.18	67.52	81.21	78.64
Improvement	+0.47	+5.29	+2.53	+2.76
32	LiDAR	83.49	57.83	70.82	70.71
LiDAR + RGB	84.47	62.56	73.93	73.65
Improvement	+0.98	+4.73	+3.11	+2.94
16	LiDAR	79.80	53.84	60.51	64.71
LiDAR + RGB	80.78	59.44	67.35	69.19
Improvement	+0.99	+5.60	+6.84	+4.48
8	LiDAR	64.42	22.57	43.19	43.39
LiDAR + RGB	67.09	31.03	47.97	48.70
Improvement	+2.66	+8.46	+4.78	+5.30
Robustness against Input Corruption. We also conducted extensive experiments to verify the robustness of our UPIDet to sensor perturbation. Specifically, we added Gaussian noises to the reflectance value of points or RGB images. Fig. A2 shows that the mAP value of our cross-modal UPIDet is consistently higher than that of the single-modal baseline and decreases slower with the LiDAR noise level increasing. Particularly, as listed in Table A11, when the variance of the LiDAR noise is set to 0.15, the perturbation affects our cross-modal UPIDet much less than the single-modal detector. Besides, even applying the corruption to both LiDAR input and RGB images, the mAP value of our UPIDet only drops by  $2.49\%$ .
Table A11: Performances (mAPs) of the single-modal baseline and our UPIDet on the KITTI val set under input corruptions of simulated LiDAR and image noise sampled from the Gaussian distribution. Note that the image noise is only applicable to multi-modal detectors.
Corruptions Type	Modal	Car	Ped.	Cyc.	mAP
No Corruption	LiDAR	86.71	62.23	78.68	75.87
LiDAR Noise	LiDAR	84.37	49.19	77.27	70.28
No Corruption	LiDAR + RGB	87.18	67.52	81.21	78.64
LiDAR Noise	LiDAR + RGB	86.82	65.61	77.63	76.69
Image Noise	LiDAR + RGB	87.14	66.26	77.97	77.12
LiDAR + Image Noise	LiDAR + RGB	86.60	65.42	76.44	76.15
Figure A2: Comparisons of noise robustness between the single-modal baseline and our UPIDet.
Effectiveness of Multi-stage Interaction. As mentioned before, both 2D and 3D backbones adopt an encoder-decoder structure, and we perform bidirectional feature propagation at both downsampling and upsampling stages. Here, we conducted experiments to verify the superiority of the multi-stage interaction over single-stage interaction. As shown in Table A12, only performing the bidirectional feature propagation in the encoder (i.e., Table A12 (b)) or the decoder (i.e., Table A12 (c)) leads to worse performance than that of performing the module in both stages (i.e., Table A12 (d)).
Table A12: Ablative experiments on the multi-stage manner of bidirectional propagation, where SA and FP denote applying bidirectional propagation at downsampling (in the encoder) and upsampling (in the decoder) stages of the point cloud branch, respectively.
Stage	3D Car (IoU=0.7)	3D Ped. (IoU=0.5)	3D Cyc. (IoU=0.5)	mAP
SA	FP	Easy	Mod.	Hard	Easy	Mod.	Hard	Easy	Mod.	Hard
(a)	-	-	91.92	85.22	82.98	68.82	61.47	56.39	91.93	74.56	69.58	75.88
(b)	✓	-	92.67	85.86	83.40	70.94	65.29	60.35	93.60	75.60	71.04	77.64
(c)	-	✓	92.19	85.44	83.27	69.02	63.41	58.47	92.85	76.39	71.91	76.99
(d)	✓	✓	92.63	85.77	83.13	72.68	67.64	62.25	94.39	77.77	71.47	78.64
A.5 Efficiency Analysis
We also compared the inference speed and number of parameters of the proposed UPIDet with state-of-the-art cross-modal approaches in Table A13. Our UPIDet has about the same number of parameters as CAT-Det [54], but a much higher inference speed at 9.52 frames per second on a single GeForce RTX 2080 Ti GPU. In general, our UPIDet is inevitably slower than some single-modal detectors, but it achieves a good trade-off between speed and accuracy among cross-modal approaches.
Table A13: Comparison of the number of network parameters, inference speed, and detection accuracy of different multi-modal methods on the KITTI test set.
Method	Params (M)	Frames per second	mAP (%)
AVOD-FPN [14]	38.07	10.00	56.84
F-PointNet [28]	12.45	6.25	57.86
EPNet [12]	16.23	5.88	-
CAT-Det [54]	23.21	3.33	67.05
UPIDet (Ours)	24.98	9.52	70.13
A.6 Visual Results of 3D Object Detection
In Figure A3, we present the qualitative comparison of detection results between the single-modal baseline and our UPIDet. We can observe that the proposed UPIDet shows better localization capability than the single-modal baseline in challenging cases. Besides, we also show qualitative results of UPIDet on the KITTI test split in Figure A4. We can clearly observe that our UPIDet performs well in challenging cases, such as pedestrians and cyclists (with small sizes) and highly-occluded cars.
A.7 Visual Results of 2D Semantic Segmentation
Several examples are shown in Figure A5. For distant cars in the first and the second row as well as the pedestrian in the sixth row, the size of objects is small and PSPNet tends to treat them as background, while our UPIDet is able to correct such errors. In the third row, our UPIDet finds the dim cyclist missed by PSPNet. Our UPIDet also performs better for the highly occluded objects as shown in the fourth and the fifth lines. This observation shows the 3D feature representations extracted from point clouds can boost 2D semantic segmentation, since the image-based method is sensitive to illumination and can hardly handle corner cases with only single-modal inputs.
A.8 Details on Official KITTI Test Leaderboard
We submitted the results of our UPIDet to the official KITTI website, and it ranks  $\mathbf{1}^{st}$  on the 3D object detection benchmark for the cyclist class. Figure A6 shows the screenshot of the leaderboard. Figure A7 illustrates the precision-recall curves along with AP scores on different categories of the KITTI test set. The samples of the KITTI test set are quite different from those of training/validation set in terms of scenes and camera parameters, so the impressive performance of our UPIDet on the test set demonstrates it also achieves good generalization.
Figure A3: Qualitative comparison between single-modal baseline and our multi-modal UPIDet. For each comparison, from top to bottom, we have the image, detection results of single-modal baseline, and detection results of UPIDet. We use red, green, and yellow to denote the ground-truth, true positive and false positive bounding boxes, respectively. We highlight some objects in images with red circles, which are detected by UPIDet but missed by the single-modal method.
Figure A4: Extra qualitative results of UPIDet on the KITTI test set. The predicted bounding boxes of car, pedestrian, and cyclist are visualized in green, cyan, and yellow, respectively. We also show the corresponding projection of boxes on images. Best viewed in color and zoom in for more details.
(c) PSPNet
Figure A5: Visual results of 2D semantic segmentation on the KITTI val set. The prediction boxes are shown in green for car, cyan for pedestrian, and yellow for cyclist. Best viewed in color. Compared with PSPNet, our cross-modal UPIDet produces more accurate and detailed results.
Method	Setting	Code	Moderate	Easy	Hard	Runtime	Environment
1	BiProDet	74.32 %	86.74 %	67.45 %	0.1 s	GPU @ 2.5 Ghz (Python + C/C++)
2	TED	74.12 %	88.82 %	66.84 %	0.1 s	1 core @ 2.5 Ghz (C/C++)
3	CasA++	code	73.79 %	87.76 %	66.84 %	0.1 s	1 core @ 2.5 Ghz (C/C++)
4	CasA	code	73.47 %	87.91 %	66.17 %	0.1 s	1 core @ 2.5 Ghz (C/C++)
5	SGNet	70.40 %	86.75 %	62.73 %	0.09 s	GPU @ 2.5 Ghz (Python)
6	HMFI	code	70.37 %	84.02 %	62.57 %	0.1 s	1 core @ 2.5 Ghz (C/C++)
7	CAD	69.94 %	84.68 %	62.21 %	0.1 s	GPU @ 2.5 Ghz (Python + C/C++)
8	SARFE	69.67 %	84.88 %	62.26 %	0.03 s	1 core @ 2.5 Ghz (C/C++)
9	EQ-PYRCNN	code	69.10 %	85.41 %	62.30 %	0.2 s	GPU @ 2.5 Ghz (Python + C/C++)
10	VoCo	69.00 %	82.74 %	62.46 %	0.1 s	1 core @ 2.5 Ghz (Python + C/C++)
Figure A6: Screenshot of the KITTI 3D object detection benchmark for cyclist class on August 15th, 2022. Note: When we submit the results, the temporary name is BiProDet.
(a) Easy car category
(b) Moderate car category
(c) Hard car category
(d) Easy pedestrian category
(e) Moderate pedestrian category
(f) Hard pedestrian category
(g) Easy cyclist category
Figure A7: Precision-recall curves of different methods on the KITTI 3D object detection test set on Aug.  $15^{th}$ , 2022. We also report APs in different categories for each method.
(h) Moderate cyclist category
(i) Hard cyclist category