SlowGuess/NeurIPS_2023_OCR huggingface dataset:
As illustrated in the lower part of Figure 2, we perform the diffusion inference step in the teacher model to denoise noisy bounding box candidates for unlabeled data. To be specific, we randomly generate the initial noisy data  $\mathbf{x}_t = \{\tilde{\mathbf{b}}_s,\tilde{\mathbf{l}}\}_{t}$  from a Gaussian distribution and extract  $N_{b}$  candidate features  $\mathbf{f}^{obj} = [f^{obj};\tilde{\mathbf{l}} ]$  as those in the diffusion training step. Then, the decoder would output model predictions based on  $\mathbf{f}^{obj}$ , followed by the DDIM sampling [17] to iteratively denoise  $\mathbf{x}_t$  and achieve better states, e.g., closer to  $\mathbf{x}_0$ . Finally, we utilize the denoised model predictions and generate pseudo-labels for unlabeled data. Algorithm 1 shows the overall inference procedure.
Bounding box renewal. During the iterative DDIM sampling step, having a filtering scheme can early reject some low-quality candidates that are unlikely to be the final pseudo-labels. Therefore, we apply a bounding box renewal mechanism at each sampling step similar to [10] and replace them by sampling new box sizes with class label distributions from a Gaussian distribution. Specifically, we adopt the filtering scheme originally used in SSL for 3D object detection [53], where the confidence score of each candidate is estimated based on the three metrics: objectness, classification confidence, and intersection-over-union (IoU) scores.
3.5 Overall Procedure in SSL
Training objectives for SSL. Our training procedure consists of two phases: pre-training and SSL training. We begin by training the proposed Diffusion-SS3D through supervised learning using labeled data  $\{\mathbf{p}_i^l,\mathbf{y}_i^l\}_{i = 1}^{N_l}$  to pre-train our detector. Next, we initialize the teacher an student models with the same detector weights, and then jointly train the detector using both labeled data  $\{\mathbf{p}_i^l,\mathbf{y}_i^l\}_{i = 1}^{N_l}$  and unlabeled data  $\{\mathbf{p}_i^u\}_{i = 1}^{N_u}$ , where each training batch includes a mixture of labeled and unlabeled scenes. The student model receives both labeled and unlabeled data after strong augmentation, while the unlabeled data with weak augmentation are fed into the teacher model to produce pseudo-labels  $\tilde{\mathbf{y}}^u$  through iterative diffusion sampling. The overall loss function  $\mathcal{L}$  for the student model is defined as  $\mathcal{L} = \mathcal{L}_l(\mathbf{p}^l,\mathbf{y}^l) + \lambda \mathcal{L}_u(\mathbf{p}^u,\tilde{\mathbf{y}}^u)$ , where the  $\mathcal{L}_l$  and  $\mathcal{L}_u$  are the detection loss functions used in [53] for bounding box regression and classification.  $\lambda$ , set as 2 following [53], is the loss weight to balance the importance of labeled and unlabeled data.
Model evaluation. We utilize the student model to perform the testing stage. Similar to the inference step in Section 3.4, randomly generated bounding box candidates are denoised through the DDIM sampling and bounding box renewal steps to produce final 3D object detection results.
Datasets. We evaluate our method on two benchmarks, including the ScanNet [12] and SUN RGB-D [46] datasets, with the evaluation settings adopted in the prior semi-supervised 3D object detection works [16, 53, 63]. ScanNet [12], a widely used 3D indoor scene benchmark dataset, consists of 1,201 training and 312 validation indoor scenes reconstructed from 2.5 million high-quality RGB-D images. Following the previous work [16, 53, 63], we primarily focus on the 18 semantic classes. SUN RGB-D [46], another 3D benchmark dataset, comprises 5,285 training and 5,050 validation scenes, where we evaluate our model on the 10 object classes.
Evaluation metrics. We split both benchmarks into labeled and unlabeled data for SSL, using labeled data ratios of  $5\%$ ,  $10\%$ , and  $20\%$  for ScanNet and  $1\%$ ,  $5\%$ , and  $10\%$  for SUN RGB-D. We evaluate our results using mean average precision (mAP) and report mAP@0.25 (mAP with the 3D IoU threshold of 0.25) and mAP@0.5 scores. We run the experiments under three random data splits and report our averaged performance and the standard deviation.
Implementation details. In this work, we employ PointNet++ [31] as the encoder and IoU-aware VoteNet [53] as the decoder. In the pre-training phase, we only use labeled data with a batch size of 4 to train the diffusion model. The model is trained for 900 epochs with an initial learning rate of 0.005. Like [16, 53], the learning rate then decays at the 400th, 600th, and 800th epochs with a factor of 0.1. In the phase of semi-supervised learning, a batch is composed of 4 labeled and 8 unlabeled data. The pre-trained model is used for initializing both the teacher and student models. The student model is trained for 1,000 epochs using the AdamW optimizer, with an initial learning rate of 0.005. Like [16, 53], the learning rate decays at the 400th, 600th, 800th, and 900th epochs with factors of 0.3, 0.3, 0.1, and 0.1, respectively.
For the diffusion process, we set the maximum timesteps to 1000 and the number of proposal boxes  $N_{b}$  to 128. We note that during random sampling from the Gaussian distribution, objects are typically present in a small space of a 3D scene. Therefore, different from DiffusionDet [10], which sets the mean of the sampling sizes to 0.5, we use a smaller value of 0.25. For noisy labels, the mean of random sampling is adjusted to the reciprocal of the class number. Unless otherwise specified, we perform two DDIM sampling [45] steps in the teacher model to generate pseudo-labels, as well as during evaluation to produce final results. For fair comparisons, we follow 3DIOUMatch [53] to employ a pseudo-label filtering scheme as a post-processing step on top of our generated pseudo-labels via the diffusion process.
Table 1: Results on the ScanNet val set with  $5\%$ ,  $10\%$ ,  $20\%$ , and  $100\%$  labeled data.
Model	5%	10%	20%	100%
mAP @ 0.25	mAP @ 0.5	mAP @ 0.25	mAP @ 0.5	mAP @ 0.25	mAP @ 0.5	mAP @ 0.25	mAP @ 0.5
VoteNet [30]	27.9 ± 0.5	10.8 ± 0.6	36.9 ± 1.6	18.2 ± 1.0	46.9 ± 1.9	27.5 ± 1.2	57.8	36.0
SESS [63]	32.0 ± 0.7	14.4 ± 0.7	39.5 ± 1.8	19.8 ± 1.3	49.6 ± 1.1	29.0 ± 1.0	61.3	38.8
3DloUMatch [53]	40.0 ± 0.9	22.5 ± 0.5	47.2 ± 0.4	28.3 ± 1.5	52.8 ± 1.2	35.2 ± 1.1	62.9	42.1
Diffusion-SS3D	43.5 ± 0.2	27.9 ± 0.3	50.3 ± 1.4	33.1 ± 1.5	55.6 ± 1.7	36.9 ± 1.4	64.1	43.2
Gain (mAP)	3.5↑	5.4↑	3.1↑	4.8↑	2.8↑	1.7↑	1.2↑	1.1↑
Table 2: Results on the SUN RGB-D val set with  $1\%$ ,  $5\%$ ,  $10\%$ , and  $20\%$  labeled data.
Model	1%	5%	10%	20%
mAP @ 0.25	mAP @ 0.5	mAP @ 0.25	mAP @ 0.5	mAP @ 0.25	mAP @ 0.5	mAP @ 0.25	mAP @ 0.5
VoteNet [30]	18.3 ± 1.2	4.4 ± 0.4	29.9 ± 1.5	10.5 ± 0.5	38.9 ± 0.8	17.2 ± 1.3	45.7 ± 0.6	22.5 ± 0.8
SESS [63]	20.1 ± 0.2	5.8 ± 0.3	34.2 ± 2.0	13.1 ± 1.0	42.1 ± 1.1	20.9 ± 0.3	47.1 ± 0.7	24.5 ± 1.2
3DIOUMatch [53]	21.9 ± 1.4	8.0 ± 1.5	39.0 ± 1.9	21.1 ± 1.7	45.5 ± 1.5	28.8 ± 0.7	49.7 ± 0.4	30.9 ± 0.2
Diffusion-SS3D	30.9 ± 1.0	14.7 ± 1.2	43.9 ± 0.6	24.9 ± 0.3	49.1 ± 0.5	30.4 ± 0.7	51.4 ± 0.8	32.4 ± 0.6
Gain (mAP)	9.0↑	6.7↑	4.9↑	3.8↑	3.6↑	1.6↑	1.7↑	1.5↑
Tables 1 and 2 show the results of Diffusion-SS3D on the ScanNet and SUN RGB-D datasets with different amounts of labeled data. Overall, our method performs favorably against state-of-the-art approaches, including VoteNet [30], SESS [63], and 3D IoUMatch [53]. In particular, comparing to the main baseline, 3D IoUMatch, Diffusion-SS3D achieves consistent performance improvement across all settings, especially when fewer labeled data are available. For example, for challenging scenarios like  $5\%$  ScanNet and  $1\%$  SUN RGB-D settings, our method improves the mAP@0.5 metric by more than  $5\%$  in both settings. For higher labeled data ratios or in the fully-supervised setting (i.e.,  $100\%$ ), Diffusion-SS3D demonstrates a consistent  $1\%$  improvement in both mAP metrics. The results demonstrate the effectiveness of using the diffusion model to produce more reliable pseudo-labels in SSL and in the fully-supervised scenario (even though it is not our main focus).
In addition, we apply our diffusion model to a data augmentation method, OPA [16], which enhances 3D object detection in SSL via learning a point augmentor in the teacher-student framework. Since OPA works on data augmentation on the data input side, we integrate it into our Diffusion-SS3D framework and conduct experiments to show the complementary benefit. Specifically, when using  $5\%$  labeled data in ScanNet, our method achieves a mAP@0.25 of  $44.6\%$  and a mAP@0.5 of  $28.1\%$ , which are  $+2.7\%$  and  $+3.1\%$  better than the OPA approach, respectively. This demonstrates the usefulness of our approach combined with other data augmentation techniques. More details and results are provided in the supplementary materials.
4.2 Ablation Study and Analysis
Diffusion on object sizes and labels. In Table 3, we delve into the impact of our proposed diffusion model by focusing on two primary aspects: diffusion on object sizes and class label distributions. With the denoising process in object sizes, i.e., ID(2), the performance is improved significantly compared to the baseline without the diffusion model in ID(1). Moreover, adding label denoising further enhances the quality of pseudo-labels and thus achieves the best performance in ID(4). Interestingly, we find that the effectiveness of size denoising is slightly better than label denoising, i.e., comparing ID(2) and (3). Here, we consider this a task-specific behavior, where the parameters to be denoised may have different influences on various perception tasks.
DDIM and box renewal. As described in Section 3.4, DDIM sampling [45] and a box renewal mechanism are used to infer pseudo-labels in the teacher model. In Table 4, we validate the importance of these two components. First, comparing ID(1) and (2), DDIM sampling contributes via denoising object sizes and class label distributions to obtain better qualities of bounding box candidates for pseudo-label generation. Then, comparing ID(1) and (3), it shows that box renewal also has the denoising capability. We note that the effectiveness of box renewal is also enhanced through the diffusion training process, where it utilizes the trained diffusion decoder to integrate updated bounding box features and label distributions. Finally, the best performance is achieved in ID(4) with both DDIM and box renewal, validating our diffusion model's capability in producing reliable pseudo-labels.
Table 3: Ablation study on the effect of box size and class label distribution in the diffusion process.
ID	Diffusion on Sizes	Diffusion on Labels	ScanNet 5%	SUNRGBD 1%
mAP @ 0.25	mAP @ 0.5	mAP @ 0.25	mAP @ 0.5
(1)	40.5 ± 1.2	22.8 ± 0.8	21.9 ± 1.4	8.0 ± 1.5
(2)	✓	43.1 ± 0.3	26.3 ± 0.2	30.5 ± 0.3	13.8 ± 0.7
(3)	✓	42.6 ± 0.6	26.0 ± 0.5	30.3 ± 0.4	13.3 ± 0.2
(4)	✓	✓	43.5 ± 0.2	27.9 ± 0.3	30.9 ± 1.0	14.7 ± 1.2
Table 4: Ablation study on the effect of DDIM and box renewal.
ID	DDIM	Box Renewal	ScanNet 5%	SUNRGBD 1%
mAP @ 0.25	mAP @ 0.5	mAP @ 0.25	mAP @ 0.5
(1)	40.5 ± 1.2	22.8 ± 0.8	21.9 ± 1.4	8.0 ± 1.5
(2)	✓	42.8 ± 0.5	26.6 ± 0.9	30.2 ± 0.7	13.9 ± 1.3
(3)	✓	42.3 ± 0.4	26.8 ± 0.3	29.5 ± 0.9	12.8 ± 0.6
(4)	✓	✓	43.5 ± 0.2	27.9 ± 0.3	30.9 ± 1.0	14.7 ± 1.2
Table 5: Ablation study on various diffusion sampling steps.
ID	Num. DDIM	ScanNet 5%	SUNRGBD 1%
mAP @ 0.25	mAP @ 0.5	mAP @ 0.25	mAP @ 0.5
(1)	1	42.6 ± 0.5	26.2 ± 0.7	30.3 ± 0.7	12.7 ± 0.2
(2)	2	43.5 ± 0.2	27.9 ± 0.3	30.9 ± 1.0	14.7 ± 1.2
(3)	4	44.0 ± 1.1	27.5 ± 1.0	28.0 ± 1.1	12.9 ± 0.3
Table 6: Ablation study on various scaling factors for SNR.
ID	Scaling Factor	ScanNet 5%	SUNRGBD 1%
mAP @ 0.25	mAP @ 0.5	mAP @ 0.25	mAP @ 0.5
(1)	1.0	43.2 ± 0.3	26.6 ± 0.9	30.5 ± 0.8	13.2 ± 0.5
(2)	2.0	43.5 ± 0.2	26.7 ± 0.2	31.3 ± 1.1	14.2 ± 1.4
(3)	4.0	43.5 ± 0.2	27.9 ± 0.3	30.9 ± 1.0	14.7 ± 1.2
Number of DDIM steps. We study how the diffusion sampling steps, i.e., DDIM [45], affect the performance. In general, having more sampling steps may better remove noises and obtain outputs closer to ground truths. However, since we address SSL, the denoising steps may have a different influence on unlabeled data because we do not have its ground truth for training the diffusion model (e.g., a distribution shift from labeled data). In Table 5, we find that having two DDIM steps improves the denoising process, but having four steps may slightly degrade the performance, which is an interestingly finding for SSL. Hence we adopt two sampling steps in all the experiments. Note that, the observed sensitivity of DDIM steps to different datasets is partially attributed to limited labeled data, which may make diffusion sampling steps more sensitive when inferring unlabeled data distribution.
Signal-to-noise ratio. The signal scaling factor is a critical parameter that can affect the signal-to-noise ratio (SNR) during the diffusion process. Table 6 explores the impact of various scaling factors. Overall, our method performs stably with different values, and we choose 4.0 as our final setting in all the experiments. It is also worth noting that, in our context like class label distributions, e.g., the 18 classes in ScanNet, there are much fewer parameters compared to more complicated diffusion tasks, e.g., image generation. Therefore, we find that having a higher SNR can be slightly beneficial and make training more stable, as it reduces the difficulty of the denoising process.
Pseudo-label quality. To assess the enhanced quality of pseudo-labels generated by Diffusion-SS3D, we evaluate metrics on unlabeled training data during model training, via the teacher model that generates pseudo-labels. In Table 7, we demonstrate that overall the pseudo-label quality of Diffusion-SS3D outperforms 3D IoUMatch [53] by more than  $8\%$  improvement in mAP and recall rate. Notably, our diffusion model achieves better quality in earlier epochs and maintains stability throughout the entire training process. It is important to highlight that the semi-supervised performance continues to improve as the model learns from both labeled and unlabeled data over extended training periods. These results show the ability of the diffusion model to consistently generate high-quality pseudolabels. For visual comparisons of the generated pseudo-labels, please refer to the supplementary materials.
Table 7: Results regarding the quality of pseudo-labels generated for unlabeled training data by the teacher model during the training process.
Model	Metric	ScanNet 5%
Epoch 100	Epoch 200	Epoch 400	Epoch 800	Epoch 1000
3D IoUMatch	mAP @ 0.5	14.08	17.85	21.73	22.17	22.42
Recall @ 0.5	27.86	31.49	35.24	36.61	35.25
Diffusion-SS3D	mAP @ 0.5	29.98	30.09	30.86	31.01	30.93
Recall @ 0.5	43.73	44.14	45.06	44.72	44.17
Table 8: Ablation study on the effect of the diffusion component and various sampling strategies.
ID	Diffusion	Sampling Stragety	ScanNet 5%
mAP @ 0.25	mAP @ 0.5
(1)	Random	40.2 ± 1.5	22.3 ± 1.1
(2)	Farthest Point	40.5 ± 1.2	22.8 ± 0.8
(3)	✓	Random	43.1 ± 0.6	27.4 ± 0.6
(4)	✓	Farthest Point	43.5 ± 0.2	27.9 ± 0.3
Farthest point sampling. In our approach, point sub-sampling is employed to narrow the search space and limit freedom in generating noisy boxes, facilitating the diffusion learning process. We explore different sampling strategies, including random point sub-sampling and farthest point sampling (FPS), to select representative centers, as shown in Table 8. Comparing ID(1) and ID(2) reveals similar performance when applying random sampling and FPS to the 3D IoUMatch [53] model without the diffusion component. Similarly, ID(3) and ID(4) show comparable results for random sampling and FPS with our approach. Notably, our diffusion model, utilizing both FPS and random sampling, outperforms the 3D IoUMatch [53] baseline. In practice, we apply FPS following the methods of VoteNet [30] and 3D IoUMatch [53].
4.3 More Discussions
Object orientation. In this paper, we focus on denoising object size and class labels, while considering orientations can be another potential direction for our diffusion model. However, incorporating orientation data is challenging due to the discrepancy in orientation quality between ground truths and predictions, as noted in [61]. For example, ScanNet [12] does not provide orientations, i.e., all objects are assigned with orientations as 0. Similarly, in SUN RGB-D [46], orientation data varies inconsistently across scenes, complicating the denoising process. Exploring diffusion models capable of handling diverse data types, including noisy ground truths and orientation information, remains an intriguing topic for future research.
Outdoor datasets. Diffusion-SS3D excels on indoor benchmark datasets like ScanNet [12] and SUN RGB-D [46] against existing state-of-the-art methods [16, 53]. Although our approach is not restricted to specific datasets (i.e., integrating diffusion model within a general teacher-student framework), exploring outdoor benchmark datasets like KITTI [14] and Waymo Open Dataset [49] may require hyperparameter changes since some object properties (e.g., location, density) in outdoor would be different from the indoor scenario.
Societal impact. Since the diffusion models require more computational powers for training and inference (runtime in frame-per-second is presented in the supplementary materials), optimizing efficiency is crucial to reduce the environmental footprint for real-time applications and large-scale deployments.
In this paper, we address the challenge of generating high-quality pseudo-labels for 3D object detection in semi-supervised learning. We propose Diffusion-SS3D, a new perspective of employing the diffusion model to produce reliable pseudo-labels via adding random noise to 3D object sizes and class label distributions, and subsequently reversing the diffusion process. We then incorporate the diffusion model into a teacher-student framework, which facilitates the learning process with denoised bounding box candidates towards better pseudo-labels. We evaluate the effectiveness of Diffusion-SS3D on the ScanNet and SUN RGB-D benchmark datasets and demonstrate the state-of-the-art performance, with extensive analysis of how the diffusion model affects SSL performance. Our method explores a broader range of reasonable pseudo-label candidates for SSL in 3D object detection, showing the potential of the diffusion model in this domain.
autonomous driving: Waymo open dataset. In CVPR, 2020. 10
[50] Peng Tang, Chetan Ramaiah, Yan Wang, Ran Xu, and Caiming Xiong. Proposal Learning for Semi-Supervised Object Detection. In WACV, 2021. 3
[51] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning results. NIPS, 2017. 2, 3
[52] Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional Score-based Diffusion Models for Probabilistic Time Series Imputation. NIPS, 2021. 3
[53] He Wang, Yezhen Cong, Or Litany, Yue Gao, and Leonidas J Guibas. 3DIoUMatch: Leveraging IoU Prediction for Semi-Supervised 3D Object Detection. In CVPR, 2021. 1, 2, 3, 4, 7, 8, 9, 10
[54] Haiyang Wang, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, Liwei Wang, et al. CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds. NIPS, 2022. 1
[55] Yan Wang, Junbo Yin, Wei Li, Pascal Frossard, Ruigang Yang, and Jianbing Shen. SSDA3D: Semi-supervised Domain Adaptation for 3D Object Detection from Point Cloud. arXiv preprint arXiv:2212.02845, 2022. 1
[56] Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised Data Augmentation for Consistency Training. NIPS, 2020. 3
[57] Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-End Semi-Supervised Object Detection with Soft Teacher. In ICCV, 2021. 1, 3
[58] Tjin Yan, Hongwei Zhang, Tong Zhou, Yufeng Zhan, and Yuanqing Xia. ScoreGrad: Multivariate Probabilistic Time Series Forecasting with Continuous Energy-based Generative Models. arXiv preprint arXiv:2106.10121, 2021. 3
[59] Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3DSSD: Point-based 3D Single Stage Object Detector. In CVPR, 2020. 1
[60] Junbo Yin, Jin Fang, Dingfu Zhou, Liangjun Zhang, Cheng-Zhong Xu, Jianbing Shen, and Wenguan Wang. Semi-supervised 3D Object Detection with Proficient Teachers. In ECCV, 2022. 1
[61] Hong-Xing Yu, Jiajun Wu, and Li Yi. Rotationally Equivariant 3D Object Detection. In CVPR, 2022. 10
[62] Peiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jia, Bo Pang, Ruigi Gao, Yixin Zhu, Song-Chun Zhu, and Ying Nian Wu. Latent Diffusion Energy-Based Model for Interpretable Text Modeling. arXiv preprint arXiv:2206.05895, 2022. 3
[63] Na Zhao, Tat-Seng Chua, and Gim Hee Lee. SESS: Self-Ensembling Semi-Supervised 3D Object Detection. In CVPR, 2020. 1, 2, 3, 4, 7, 8
[64] Hongyu Zhou, Zheng Ge, Songtao Liu, Weixin Mao, Zeming Li, Haiyan Yu, and Jian Sun. Dense Teacher: Dense Pseudo-Labels for Semi-supervised Object Detection. In ECCV, 2022. 1
[65] Yin Zhou and Oncel Tuzel. VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection. In CVPR, 2018. 1
[66] Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete Contrastive Diffusion for Cross-Modal Music and Image Generation. arXiv preprint arXiv:2206.07771, 2022.3