SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cheng-Ju Ho1 Chen-Hsuan Tai1 Yen-Yu Lin1 Ming-Hsuan Yang2,3 Yi-Hsuan Tsai3
$^{1}$ National Yang Ming Chiao Tung University  $^{2}$ University of California at Merced  $^{3}$ Google
Semi-supervised object detection is crucial for 3D scene understanding, efficiently addressing the limitation of acquiring large-scale 3D bounding box annotations. Existing methods typically employ a teacher-student framework with pseudolabeling to leverage unlabeled point clouds. However, producing reliable pseudolabels in a diverse 3D space still remains challenging. In this work, we propose Diffusion-SS3D, a new perspective of enhancing the quality of pseudo-labels via the diffusion model for semi-supervised 3D object detection. Specifically, we include noises to produce corrupted 3D object size and class label distributions, and then utilize the diffusion model as a denoising process to obtain bounding box outputs. Moreover, we integrate the diffusion model into the teacher-student framework, so that the denoised bounding boxes can be used to improve pseudolabel generation, as well as the entire semi-supervised learning process. We conduct experiments on the ScanNet and SUN RGB-D benchmark datasets to demonstrate that our approach achieves state-of-the-art performance against existing methods. We also present extensive analysis to understand how our diffusion model design affects performance in semi-supervised learning. The source code will be available at https://github.com/luluho1208/Diffusion-SS3D.
3D object detection is a crucial computer vision task that predicts oriented bounding boxes and semantic classes of 3D objects for geometrical scene understanding, and thus it is essential to 3D applications such as autonomous driving, AR/VR, and robot navigation. Recent works [26, 30, 35, 39, 40, 41, 54, 59, 65] have made significant progress in the field of 3D object detection. However, most existing approaches heavily rely on labeled 3D point clouds, which requires substantial manual annotations. To overcome this limitation, semi-supervised learning (SSL) methods for 3D object detection have emerged, utilizing unlabeled 3D point clouds to supplement scarce labeled data and further improve detector performance.
Existing semi-supervised learning methods for 3D object detection [16, 53, 55, 60, 63] focus on generating pseudo-labels for 3D bounding boxes as supervisory signals on unlabeled data. However, different from the 2D cases where pseudo-labels are limited to the 2D image coordinates [7, 22, 25, 43, 44, 57, 64], the potential object locations in the 3D geometric space can be more diverse, causing the difficulty in generating high-quality pseudo-labels (i.e., pseudo bounding boxes) for 3D object detection. Consequently, an ensuing question arises: Is there an effective approach for generating pseudo-labels that account for the vast 3D space in the realm of semi-supervised 3D object detection? As shown in Figure 1(a), existing methods [16, 53, 63] usually adopt a teacher-student framework to provide pseudo-labels. In this case, bounding box candidates are solely based on model predictions, making it challenging to adjust pseudo bounding boxes, e.g., their sizes. In addition, pseudo-label quality may not be satisfactory when the model fails to output sufficient predictions (i.e., lower recall rate).
To mitigate this problem, we consider other sources to produce more reliable pseudo-labels for unlabeled data, and hence propose to denoise bounding boxes corrupted by random noise via the
Figure 1: Pseudo-label Generation. (a) Prior works [16, 53, 63] apply a teacher-student framework where the point features serve as the only source for pseudo-label generation from the decoder output. (b) In our method, Diffusion-SS3D also adopts the teacher-student framework (the student model is omitted from the figure to save the space) and further integrates noisy object sizes and noisy label distributions in the denoising process (i.e., diffusion sampling). This enables more reliable pseudo-labels through the iterative refinement of denoising via the diffusion model.
diffusion model [13, 17, 45, 48]. There have been prior works using the diffusion model for 2D object detection [10] with purely random bounding boxes. However, applying the same process to 3D scenes is not trivial due to the vast 3D space, e.g., leading to a low recall rate of the ground-truth objects covered by purely random 3D object bounding boxes. To tackle this issue and generate plausible bounding box candidates, we include random noise in the 3D object size for representative points from the 3D model, in which each point indicates one potential object location. Therefore, our diffusion model starts from more accurate bounding box candidates with various object sizes, which may increase the chance (i.e., higher recall rate) of discovering better pseudo-labels for unlabeled data. Then, through the denoising process in our diffusion model, pseudo-labels are refined to be more reliable. Another challenge lies in less accurate class predictions for pseudo bounding boxes. To address it, we introduce a label denoising process, in which we add random noise to the class label distribution for each bounding box candidate, and then our diffusion model would denoise the class label to improve pseudo class label prediction.
We refer to our diffusion method for semi-supervised 3D object detection as Diffusion-SS3D, where we include random noise to 3D object size and class label distributions in the diffusion model to generate reliable pseudo bounding boxes (see Figure 1(b)). For SSL, we utilize the teacher-student framework [16, 53, 63] to incorporate labeled and unlabeled data into the training process, where the student model is trained based on (pseudo) ground truths and then update the teacher model with the exponential moving average (EMA) scheme [51] to produce pseudo-labels. To integrate the diffusion model into this framework, while training the student model, we regulate Gaussian noises to generate noisy object sizes and class labels as corrupted (pseudo) ground truths, followed by a decoder for predicting bounding box outputs. For the teacher model, we randomly generate object sizes and class label distributions, and apply diffusion sampling for denoising and producing pseudo-labels on unlabeled data. Then, during testing, we can utilize the diffusion sampler that denoises random object sizes and class labels to obtain bounding box predictions from the student model.
The proposed method, Diffusion-SS3D, explores a wider range of reasonable pseudo-label candidates for SSL in 3D object detection. We conduct experiments on the ScanNet [12] and SUN RGB-D [46] benchmark datasets to demonstrate that our approach achieves state-of-the-art performance. For example, our method improves the prior work by more than  $6\%$  in mAP@0.5 when using only the  $1\%$  labeled data on SUN RGB-D, and more than  $5\%$  when using  $5\%$  labeled data on ScanNet. This shows the benefit of adopting the proposed diffusion model to improve the pseudo-label quality. Moreover, we present extensive analysis to understand how our diffusion model design affects performance in SSL, e.g., ablation study of our diffusion model components, sampler steps, and signal-to-noise ratio (SNR). The main contributions of this work can be summarized as follows:
- We present Diffusion-SS3D, the first method to utilize the diffusion model for semi-supervised 3D object detection, treating the task as a denoising process for improving the quality of pseudo-labels.
- We propose to introduce the random noise to 3D object size and class label distributions for producing more plausible pseudo bounding boxes, by a means to integrate the diffusion model into the teacher-student framework for SSL.
- We present extensive experimental results to analyze the main factors in the diffusion model design that may affect 3D detector performance in SSL, as well as demonstrating state-of-the-art performance against existing methods.
Semi-supervised Learning. Semi-supervised learning (SSL) enhances models by incorporating unlabeled data during training. Typically SSL methods can be categorized into two groups: consistency regularization [5, 6, 21, 37, 56] and pseudo-labeling [22, 43, 51]. Among these approaches, the teacher-student framework [43, 51] is an effective self-training method, using two identical networks where the teacher model is guided by the student model via the exponential moving average mechanism. It also leverages asymmetric augmentations, using weak augmentation for the teacher model to ensure high-quality pseudo-label supervision and strong augmentation for the student model to enrich data variance for training.
Semi-supervised Learning for Object Detection. For 2D object detection in SSL, similarly, existing methods adopt consistency regularization [19, 50] or teacher-student learning framework [7, 25, 43, 44, 57]. For 3D object detection, the primary paradigm in SSL [16, 29, 53, 63] employs the teacher-student framework with pseudo-labeling. For example, SESS [63] enforces consistency across different augmentations as regularization, while 3D IoUMatch [53] enhances the pseudo-label quality through confidence-based filtering with an IoU estimation module. Recently, OPA [16] introduces a trainable object-level augmentor to enrich the variance of foreground objects and improve data augmentation. DetMatch [29] and MVC-MonoDet [24] tackle the 3D SSL task via the multi-modal and multi-view settings respectively. However, most of these methods rely on the teacher model for pseudo-label generation, limiting the ability to identify bounding boxes that are not covered by model predictions. In this work, we propose to leverage the diffusion model to denoise from noisy object sizes and class labels as another source to provide high-quality pseudo-labels for bounding boxes.
Diffusion Models for Visual Recognition. Diffusion models [17, 42, 47, 48] have attracted significant attention owing to their powerful denoising ability in generative models. Numerous fields have leveraged diffusion models, such as natural language processing [3, 18, 23, 38, 62], multimodal data generation [4, 32, 34, 36, 66], and temporal data modeling [1, 9, 20, 33, 52, 58]. Despite great success in synthesis tasks, diffusion models are less explored for visual recognition [2, 8, 10, 15, 27]. LRA [8] is introduced to mitigate the influence of noisy labels by utilizing neighbor consistency to create pseudo-clean labels for diffusion training. [27] utilizes diffusion models originally designed for image generation as a unified representation learner and demonstrates their effectiveness in extracting discriminative features for classification tasks. Moreover, DiffusionInst [15] formulates the instance segmentation task as a noise-to-filter problem, and utilizes the denoising capabilities of a diffusion model to generate better instance masks. DiffusionDet [10] casts object detection as a noise-to-box task, where high-quality object bounding boxes are produced by gradually denoising randomly generated proposals. In this work, we leverage the diffusion model in a different problem setting for semi-supervised learning, aiming to provide a new perspective of producing more reliable pseudo-labels for 3D object detection.
3.1 Problem Definition and Preliminaries
Semi-supervised 3D object detection. 3D object detection aims to recognize and locate objects of interest in a 3D point cloud scene consisting of  $N_{p}$  points  $\mathbf{p} \in \mathbb{R}^{N_p \times 3}$ , represented by their bounding boxes and semantic classes. Semi-supervised learning involves  $N_{l}$  labeled scenes  $\{\mathbf{p}_i^l, \mathbf{y}_i^l\}_{i=1}^{N_l}$  and  $N_{u}$  unlabeled scenes  $\{\mathbf{p}_i^u\}_{i=1}^{N_u}$ , where typically  $N_{l} < N_{u}$ . Ground-truth annotations  $\mathbf{y}_i^l$  contain  $K$  objects of interest  $\{\mathbf{b}_i^k, \mathbf{l}_i^k\}_{k=1}^{K}$  in  $\mathbf{p}_i^l$ , where  $\mathbf{b}$  and  $\mathbf{l}$  represent a set of bounding box parameters and semantic class labels respectively, with a total number of  $N_{cls}$  classes. Specifically, we formulate the bounding box  $\mathbf{b} = \{\mathbf{b}_c, \mathbf{b}_s, \mathbf{b}_o\}$ , where  $\mathbf{b}_c = \{c_x, c_y, c_z\}$  denotes the center coordinates,  $\mathbf{b}_s = \{s_l, s_w, s_h\}$  represents the object size, and  $\mathbf{b}_o$  corresponds to the bounding box orientation.
Figure 2: Overview of Diffusion-SS3D. Diffusion-SS3D employs the teacher-student learning framework combined with an asymmetric augmentation mechanism to incorporate both labeled and unlabeled data during training. In the student model, we establish a diffusion training process by introducing corrupted (pseudo) ground-truths by having the noisy object size  $\tilde{\mathbf{b}}_s$  and noisy class label distributions  $\tilde{\mathrm{I}}$ . Then, based on the features  $f^{obj}$  within boxes, the student decoder is trained to predict the bounding box output supervised by (pseudo) ground truths (see Figure 3 for details). For the teacher model updated by the student model via exponential moving average (EMA), we establish the diffusion inference process that denoises the randomly generated box sizes  $\tilde{\mathbf{b}}_s$  and class label distributions  $\tilde{\mathrm{I}}$  via iteratively applying diffusion sampling to output refined bounding box candidates. Following a filtering mechanism, the teacher decoder produces final pseudo-labels for unlabeled data.
Diffusion model. Diffusion models [13, 17, 45, 48] are generative models that simulate a diffusion process to represent data distributions. In the forward process, noise is progressively introduced to an initial data point, producing intermediate samples that converge to the target data distribution. The inference process estimates the posterior distribution of the initial data point from observed sample sequences by reversing the diffusion process. In [45], the diffusion forward process transforms an initial data sample  $\mathbf{x}_0$  into a noisy sample  $\mathbf{x}_t$  at time  $t$  by adding noise governed by the noise variance schedule  $\beta_t$  [17], with  $\alpha_t \coloneqq 1 - \beta_t$  and  $\bar{\alpha}_t \coloneqq \prod_{s=1}^t \alpha_s$ . The forward process to generate a noisy sample  $\mathbf{x}_t$  from  $\mathbf{x}_0$  can be defined by
$$
q \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {0}\right) = \mathcal {N} \left(\mathbf {x} _ {t}; \sqrt {\bar {\alpha} _ {t}} \mathbf {x} _ {0}, (1 - \bar {\alpha} _ {t}) \mathbf {I}\right), \tag {1}
$$
where  $\mathcal{N}$  denotes a Gaussian distribution and  $\mathbf{I}$  is an identity matrix. The model is trained to predict  $\mathbf{x}_0$  from  $\mathbf{x}_t$  by minimizing the  $l_2$  loss between  $\mathbf{x}_0$  and model outputs. During inference,  $\mathbf{x}_0$  is obtained by gradually denoising from  $\mathbf{x}_t$ , i.e.,  $\mathbf{x}_t \to \mathbf{x}_{t - \Delta} \to \ldots \to \mathbf{x}_0$ .
In this paper, we propose Diffusion-SS3D, a diffusion model for semi-supervised 3D object detection. We incorporate random noise for 3D object size and class label distributions in the diffusion model, and then reverse the diffusion process to generate reliable pseudo bounding boxes. Specifically, we define data samples as a set of bounding box parameters  $\mathbf{x}_0 = \{\mathbf{b}_s,1\}$ , representing the 3D object size and one-hot semantic label respectively. Then our detector is trained to predict  $\mathbf{x}_0$  from noisy bounding box parameters  $\mathbf{x}_t$ , conditioned on the features of the given point cloud  $\mathbf{p}$ . In SSL, we use this process to denoise bounding box candidates and generate final pseudo-labels for unlabeled data.
3.2 Algorithm Overview
Existing SSL methods for 3D object detection [16, 53, 63] utilize the teacher-student framework with asymmetric data augmentation to integrate labeled and unlabeled data in training. One critical factor is to generate pseudo-labels for unlabeled data, in which these approaches would solely rely on model outputs, which may restrict the models from generating better pseudo-labels (i.e., lower recall rate). Motivated by the challenge of creating high-quality pseudo-labels, we propose Diffusion-SS3D, a new perspective of producing reliable pseudo-labels via the diffusion model in the teacher-student framework. Figure 2 presents an overview of our method.
Figure 3: We illustrate the process of feeding RoI features with noisy object size and label distributions to the decoder before the diffusion sampling step. First, the input point cloud  $\mathbf{p}$  is processed through the encoder, and the Farthest Point Sampling (FPS) is performed to obtain  $N_{b}$  representative points  $\{m_i\}_{i = 1}^{N_b}$  (represented in red points). Concurrently, the noisy object size  $\tilde{\mathbf{b}}_s$  and the noisy label distribution  $\tilde{\mathbf{I}}$  are sampled from a Gaussian distribution. With representative points serving as centers of bounding boxes, RoI features  $f^{obj}$  within each bounding box are extracted and then concatenated with noisy label distributions, together as the input to the decoder to predict bounding box output.
In the student model, the goal is to supervise model predictions, i.e.,  $\hat{\mathbf{y}}^l$  and  $\hat{\mathbf{y}}^u$ , for labeled data with ground truths  $\mathbf{y}^l$  and unlabeled data with pseudo-labels  $\tilde{\mathbf{y}}^u$ , respectively. More importantly, we establish the training step for the diffusion process by generating the noisy data  $\mathbf{x}_t = \{\tilde{\mathbf{b}}_s,\tilde{\mathbf{l}}\}_{t}$  from ground truths of the labeled data, forming noisy bounding box candidates, where  $\tilde{\mathbf{b}}_s$  is the noisy object size and  $\tilde{\mathbf{l}}$  is the noisy class label distribution. Then, we use the noisy bounding boxes, along with the corresponding features from the encoder, as the input to the decoder for making the predictions. Similarly, for the unlabeled data, the noisy bounding boxes can be generated from pseudo-labels. In each training iteration, the student model also updates the teacher model via an exponential moving average (EMA) mechanism for both the encoder and decoder.
In the teacher model, we aim to generate more reliable pseudo-labels for unlabeled data via the diffusion model. We start with randomly generated object sizes and class label distributions to form noisy bounding boxes, and then adopt the decoder to output the predictions similar to the process in the student model. Subsequently, we employ the diffusion sampling method [45] to iteratively denoise and produce more accurate object sizes and class labels for pseudo-label generation.
3.3 Diffusion Model Training
In this section, we introduce the components and procedure used to train the proposed DiffusionSS3D framework, including the details of how we process point cloud features with generated noisy bounding boxes and class label distributions to train the diffusion model.
Detector encoder. The encoder takes a 3D point cloud scene  $\mathbf{p} \in \mathbb{R}^{N_p \times 3}$  as input, executing only once during the diffusion training process to extract the feature representations  $f \in \mathbb{R}^C$  for each point  $m \in \mathbb{R}^3$  of  $\mathbf{p}$  as a  $C$ -dimensional feature vector.
Detector decoder. Given a set of  $N_{b}$  noisy object sizes  $\tilde{\mathbf{b}}_s \in \mathbb{R}^{N_b \times 3}$  and noisy class labels  $\tilde{\mathbf{l}} \in \mathbb{R}^{N_b \times N_{cls}}$ , associated with their extracted RoI-features  $f^{obj}$  within the object from the encoder, the decoder predicts bounding box regression values and classification results. Details of how the features are extracted are described in the following.
RoI-feature extraction with noisy bounding boxes. We aim to denoise from noisy bounding boxes to train the diffusion model. However, completely sampling random bounding boxes as the noisy data [10] may be too challenging for our case due to the vast 3D space. To increase the chance of producing plausible bounding box candidates with noise, we consider using representative point locations  $\{m_i\}$  obtained from the 3D model, in which each point indicates a potential object location. Then, based on these points, we generate noisy bounding boxes by adding random noises to the 3D object size and class label distributions. The overall procedure is illustrated in Figure 3.
Algorithm 1: Teacher Model	Algorithm 2: Student Model
def pseudo_label(pc, steps, T):
# Extract points and features
pts, feats = Teacher encoder(pc)	def object_prediction(pc, gts, pls):
# Extract points and features
pts, feats = Student encoder(pc)
# Noisy sizes and class labels
sizes_t = normal(mean=0, std=1) # [Nb, 3]
labels_t = normal(mean=0, std=1) # [Nb, Ncls]	# Pad to Nb bounding boxes
sizes = prepare_size(gts, pls) # [Nb, 3]
labels = prepare_label(gts, pls) # [Nb, Ncls]
# Uniform sample step size
times = reversedlinespace(-1, T, steps))
time_pairs = list(zip(times[-1], times[1]))	# Signal scaling
sizes = (sizes * 2 - 1) * size_scale
labels = (labels * 2 - 1) * label_scale
for tCUR, t_next in zip(time_pairs):
# Generate noisy boxes
boxes_t = concat(FPS(plt), sizes_t)	# Corrupt ground truths
t = randint(0, T) # time_step
sizes_crt = corrupt(sizes, t)
labels_crt = corrupt(labels, t)
# Predict pseudo labels
pls = TeacherDecoder(feats, boxes_t,
lables_t, tCUR)	# Generate noisy boxes
boxes_crt = concat(FPS(plt), sizes_crt)
# Estimate sizes_t and labels_t at t_next
sizes_t, labels_t = ddim(sizes_t, labels_t,
plt, t CUR, t_next)	# Predict object candidates
preds = Student.decoder(feats, boxes_crt,
labels_crt, t)
# Box renewal
sizes_t, labels_t = box_renewal(sizes_t,
lables_t)	# Supervise by (pseudo) ground truths
loss = detector_loss(preds, gts, pls)
Student = update(Student, loss)
# Filtering
pls = filter(pls)	# Update teacher model
Teacher = EMA(Teacher, Student)
return pls	return preds
linespace:generate evenly spaced values	corrupt(x, t): sqrt(alpha_cumprod(t)) * x +
sqrt(1 - alpha_cumprod(t)) * noise
alpha_cumprod(t): Πi=1^n αi
Among all  $M$  representative points generated by the encoder, we perform the Farthest Point Sampling (FPS) algorithm [31] to obtain a small set of  $N_{b}$  sampled points  $\{m_i\}_{i = 1}^{N_b}$  that maximally cover this 3D scene. We then generate  $N_{b}$  noisy bounding boxes by treating the sampled points  $\{m_i\}_{i = 1}^{N_b}$  as object centers, where each box has a noisy object size  $\tilde{\mathbf{b}}_s$  and noisy class label distributions  $\tilde{\mathbf{I}}$ . From the encoded features  $\{f_i\}$ , we process features within each bounding box to form the RoI-features  $\{f_i^{obj}\}_{i = 1}^{N_b} \in \mathbb{R}^{N_b \times C}$ . By concatenating the RoI-features and noisy class label distributions  $\mathbf{f}_i^{obj} = [f_i^{obj}; \tilde{\mathbf{I}}] \in \mathbb{R}^{N_b \times (C + N_{cls})}$ , the decoder takes  $\mathbf{f}_i^{obj}$  as a noisy condition to predict the regression values for bounding boxes and classification results.
Diffusion training procedure. As illustrated in the upper part of Figure 2, the diffusion model is trained in the student model, where it involves adding Gaussian noise to (pseudo) ground truths and training the student decoder to reverse this process. In addition to corrupting (pseudo) ground truths, we generate random bounding boxes and class labels so that total  $N_{b}$  noisy candidates are produced. Similar to [28], the magnitude of the noise is controlled by a cosine schedule that progressively lessens  $\alpha_{t}$  in (1) along the iterative steps. Note that the coordinates of the ground truth box also need to be adjusted through signal scaling. This is critical as the signal-to-noise ratio (SNR) plays an important role in the performance of the diffusion model [11]. Then, the student model is trained by minimizing the loss between decoder's predictions and (pseudo) ground truths of labeled/unlabeled data. We present the training procedure in Algorithm 2.
3.4 Diffusion Inference for Pseudo-Labels