Diffusion-SS3D: Diffusion Model for Semi-supervised 3D Object Detection
Authors: Cheng-Ju Ho
Chen-Hsuan Tai
Yen-Yu Lin
Ming-Hsuan Yang
Yi-Hsuan Tsai
Venue: NeurIPS 2023 poster (2023)
Keywords: Semi-supervised learning
3D object detection
diffusion model
TL;DR: Diffusion-SS3D enhances the quality of pseudo-labels via the diffusion model for semi-supervised 3D object detection.

Abstract: Semi-supervised object detection is crucial for 3D scene understanding, efficiently addressing the limitation of acquiring large-scale 3D bounding box annotations. Existing methods typically employ a teacher-student framework with pseudo-labeling to leverage unlabeled point clouds. However, producing reliable pseudo-labels in a diverse 3D space still remains challenging. In this work, we propose Diffusion-SS3D, a new perspective of enhancing the quality of pseudo-labels via the diffusion model for semi-supervised 3D object detection. Specifically, we include noises to produce corrupted 3D object size and class label distributions, and then utilize the diffusion model as a denoising process to obtain bounding box outputs. Moreover, we integrate the diffusion model into the teacher-student framework, so that the denoised bounding boxes can be used to improve pseudo-label generation, as well as the entire semi-supervised learning process. We conduct experiments on the ScanNet and SUN RGB-D benchmark datasets to demonstrate that our approach achieves state-of-the-art performance against existing methods. We also present extensive analysis to understand how our diffusion model design affects performance in semi-supervised learning. The source code will be available at https://github.com/luluho1208/Diffusion-SS3D.

Diffusion-SS3DD: Diffusion Model for Semi-supervised 3D Object Detection
Cheng-Ju Ho!
Chen-Hsuan Tai
Yen-Yu Lin
Ming-Hsuan Yang
Yi-Hsuan Tsai
National Yang Ming Chiao Tung University
University of California at Merced
GGogle
Abstract
Semi-supervised object detection is crucial for 3D scene understanding, efficiently addressing the limitation of acquiring large-scale 3D bounding box annotations. Existing methods typically employ a teacher-student framework with pseudo- labeling leverage unlabeled point clouds. However, producing reliable pseudo- labels in a diverse 3D space still remains challenging. In this work, we propose Diffusion-SS3I
perspective of enhancing the quality of pseudo-labels via the diffusion model for semi supervised 3D object detection. Specifically, we include noises to produce corrupted 3D object size and class label distributions, and then utilize the diffusion model as a denoising process to obtain bounding box outputs. Moreover, framework, so that the denoised bounding boxes can be used to improve pscudo- label generation, as well the entire semi-supervised learning process. We conduct experiments on the ScanNet and SUN RGB-D benchmark datasets to demonstrate that our approach achieves state-of-the- performance against existing methods. We also present extensive analysis to understand how our diffusion model design affects performance in sumi-supervised learning. The source code will be available at https: / / g thub. wwwwcommmmlaaaeeee lul 208 Diffusion-SSSS
integrate the diffusion model into the teacher-student
Introduction
3D object detection is a crucial computer vision task that predicts oriented bounding boxes and semantic classes of 3D objects for geometrical scene understanding, and thus it is essential to 3D applications such as autonomous driving, AR/VR, and robot navigation. Recent works [26, 30, 35, 39, 40, 41, 54, 59, 65] have made significant progress in the field of 3D object detection. However, most existing approaches heavily rely on labeled 3D point clouds, which requires substantial manual annotations. To overeome this limitation, semi-supervised learning (SSL) methods for 3D object detection have emerged, utilizing unlabeled 3D point clouds to supplement scarce labeled data and further improve detector performance.
Existing semi cemi-supervised learning methods for object detection [16, 53, 55, 60, 63] focus on generating useudo-labels for 3D bounding boxes as supervisory signals on unlabeled data. However, different from the 2D cases where pseudo-labels are limited to the 2D image coordinates [7, 22, 25, 43, 44, 57, 64], the potential object locations in the 3D geometric space can be more diverse, causing the difficulty in generating high-quality pseudo-label (i.e., pseudo bounding boxes) for 3D object detection. Consequently,
ensuing question arises: Is there effective approach for generating pseudo-labels that account for the vast 3D space in the realm of remi-supervised 3D object detection? As shown in Figure I(a), existing methods [16, 53, 63] usually adopt teacher-student framework to provide pseudo-labels. In this case, bounding box candidates are solely based on model predictions, making it challenging to adjust pseudo bounding boxes, e.g., their sizes. In addition, pseudo-label quality may not be satisfactory when the model fails to output sufficient predictions (i.e., lower recall rate).
To mitigate this problem, we consider other sources to produce more reliable pseudo-labels for unlabeled data, and hence propose to denoise bounding boxes corrupted by random noise via the
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Unlabeled Data
Pseudo-Labels
Uniabeled Data
Pseudo-Labes
Point
Point
Teacher Encoder
Point Featuree
Teacher Decocirr
Teacher Encoder
Point Features
Teacher Decader
EMA
EMA
Diftusion
Sampling
Encoder
Decoder
Notey Object Sizes
Gauesian Nolse
Noisy Claas Labels
(a) Prior work
DiffusionSSS0 (ours)
Figure 1: Pseudo-label Generation. (a) Prior works [16, 53, 63] apply teacher-student framework where the point features serve as the only source for pseudo-label generation from the decoder output. (b) In our method, Diffusion-SS3D also adopts the teacher-student framework (the student model omitted from the figure to save the space) and further integrates noisy object sizes and noisy label distributions in the denoising process (i.e., diffusion sampling). This enables more reliable pseudo-labels through the iterative refinement of denoising via the diffusion model.
diffusion model [13, 17, 45, 48]. There have been prior works using the diffusson model for 2D object detection [10] with purely random bounding boxes. However, applying the same process to 3D scenes not trivial due to the vast 3D space, e-g., leading to low recall rate of the ground-truth objects covered by purely random 3D object bounding boxes. To tackle this issue and generate plausible bounding box candidates, we include random noise in the 3D object size for representative points from the 3D model, in which each point indicates one potential object location. Therefore, our diffusion model starts from more accurate bounding box candidates with various object sizes, which may increase the chance (i.e., higher recall rate) of discovering better leeudo-labels for unlabeled data. Then, through the denoising process in our diffusion model, pscudo-labels are refined to be more reliable. Another challenge lies in less accurate class predictions for pseudo bounding boxes. To address it, we introduce a label denoising process, in which we add random noise to the class label distribution for each bounding box candidate, and then our diffusion model would denoise the class label to improve pseudo class label prediction.
We refer to our diffusion method for semi- supirsuper 3D object detection as Diffusion-SS3L where we include random noise to 3D object size and class label distributions in the diffusion model to generate reliable pseudo bounding boxes (see Figure 1(b)). For SSL, we utilize the teacher-student framework [16, 53, 63] to incorporate labeled and unlabeled data into the training process, where the student model is trained based on (pseudo) ground truths and then update the teacher model with the exponential moving average (EMA) scheme [51] to produce pscudo-labels. To integrate the diffusion model into this framework, while training the student model, we regulate Gaussian noises to generate noisy object sizes and class labels as corrupted (pseudo) ground truths, followed by a decoder for predicting bounding box outputs. For the teacher model, we randomly generate object sizes and class label distributions, and apply diffusion sampling for denoising and producing pseudo-labelt on unlabeled data. Then, during testing, we can utilize the diffusion sampler that denoises random object sizes and class labels to obtain bounding box predictions from the student model.
The proposed method, Diffusion-SS3D, explores wider range of reasonable pseudo-label candidates for SSL in 3D object detection. We conduct experiments on the ScanNet [12] and SUN RGB-D [46] benchmark datasets to demonstrate that our approach achieves of performance. For example, our method improves the prior work by more than 6% in mAP@0.5 when using only the 1% labeled data on SUN RGB-D, and more than 5% when using 5% labeled data ScanNet. This shows the benefit of adopting the proposed diffusion model to improve the pseudo-labe quality. Moreover, we present extensive analysis understand how our diffusion model design affects performance in SSL, e.g., ablation study of our diffusion model components, sampler steps, and signal-to-nn -o-isee ratio (SNR). The main contributions of this work can be summarized follows:
We present Diffusion-SS332 the first method to utilize the diffusion model for semi-supervised 3D object detection, treating the task as a denoising process for improving the quality of pscudo-labels.

---

We propose to introduce the random noise to 3D object size and class label distributions for producing more plausible pseudo bounding boxes, by means to integrate the diffusion model into the teacher-student framework for SSL..
We present extensive experimental results to analyze the main factors in the diffusion model design that may affect 3D detector performance in SSL, as well as demonstrating state-of-the-art performance against existing methods.
2 Related Work
Semi-supervised Learning. Semi supervisee learning (SSL) enhances models by incorporating unlabeled data during training. Typically SSL methods can be categorized into two groups: consis- tency regularization [5, 6, 21, 37, 56] and pseudo-labeling [22, 43, 51]. Among these approaches, the teacher-student framework [43, 51] is an effective self-training method, using two identical networks where the teacher model is guided by the student model via the exponential moving average mechanism. It also leverages asymmetric augmentations, using weak augmentation for the teacher model to ensure high-quality pseudo-label supervision and strong augmentation for the student model to enrich data variance for training.
Semi-supervised Learning for Object Detection. For 2D object detection in SSL, similarly, existing methods adopt consistency regularization [19, 50] or teacher-student learning framework [7, 25, 43, 44, 57J. For 3D object detection, the primary paradigm in SSL [16, 29, 53, 63] employs the teacher-student framework with pseudo-labeling. For example. SESS [63] enforces consistency across different augmentations as regularization, while 3DtoUMatch [53] enhances the pseudo-label quality through confidence-based filtering with an IoU estimation module. Recently, OPA [16] introduces a trainable object-level augmentor to enrich the variance of foreground objects and improve data augmentation. DetMatch [29] and MVC-MonoDet [24] tackle the 3D SSL task via the multi-modal and multi-view settings respectively. However, most of these methods rely on the teacher model for pseudo-label generation, limiting the ability identify bounding boxes that not covered by model predictions. In this work, we propose to leverage the diffusion model to denoise from noisy object sizes and class labels as another source to provide high-quality pseudo-labels for bounding boxes.
Diffusion Models for Visual Recognition. Diffusion models [17, 42, 47, 48] have attracted significant attention owing to their powerful demoising ability in generative models. Numerous fields have leveraged diffusion models, such as natural language processing [3, 18, 23, 38, 62], multimodal data generation [4, 32, 34, 36, 66], and temporal data modeling [1. 9. 20, 33, 52, 58]. Despite great success in synthesis tasks, diffusion models are less explored for visual recognition [2. 8, 10. 15, 27].
LRA [8] is introduced to mitigate the influence of noisy labels by utilizing neighbor consistency to create pseudo-clean labels for diffusion training. [27] utilizes diffusion models originally designed for image generation as a unified representation learner and demonstrates their effectiveness in extracting discriminative features for classification tasks. Moreover, DiffusionInst [15] formulates the instance segmentation task as noise-to- filter problem, and utilizes the denoising capabilities of a diffusion model to generate better instance masks. DiffusionDet [10] casts object detection as a noise-to-box task, where high-quality object bounding boxes are produced by gradually denoising randomly generated proposals. In this work, we leverage the diffusion model a different problem setting for semi-supervised learning, aiming provide new perspective of producing more reliable pseudo-labels for 3D object detection.
3 Methodology
3.1 Problem Definition and Preliminaries
Semi-supervised 3D object detection. 3D object detection aims to recognize and locate objects of interest in 3D point cloud scene consisting of N, points p E , x3 represented by their bounding boxes and semantic classes. Semi-supervised learning involves Ni labeled scenes {p v and N₂, unlabeled scenes {p." J.m, where typically N; << Nu. Ground-truth annotations y! contain K k=1 in p/. where and represent a set of bounding box parameters and semantic class labels respectively, with total number of Nets classes. Specifically, we formulate the bounding box
objects of interest (by 1 K
(be, b,, bo) where b. = CCa Cap 02) denotes the center coordinates, b, = (st, sw, } represents the object size, and b, corresponds to the bounding box orientation.
3

---

Student Model
L
Ground
Ground Truths
Noay Sizee
b
Novey Labels
yf
Truths
Strong Augmentation
Student Encoder
J
Student
y
Lu
y
Psaudo- Labels
Decoder
p p
Input
EMA
EMA
fobj
Teacher Encoder
Teacher Decoder
Weak Augmentation
Pseudo-Label Generation
Naryy Siee b
Nokey Labels
I
Gouosian Nose
Diffusion Sampling
Teacher Model
Overview of Diffusion-SS3D. Diffusion-SS3D employs the teacher-student learning framework combined with asymmetric augmentation mechanism incorporate both labeled and unlabeled data during training. In the student model, we establish a diffusion training process by introducing corrupted (pseudo) ground-truths by having the noisy object size b, and noisy class label distributions I. Then, based on the features frok within boxes, the student decoder is trained to predict the bounding box output supervised by (pseudo) ground truths (see Figure for details). For the teacher model updated by the student model via exponential moving average (EMA), we establish the diffusion inference process that denoises the randomly generated box sizes b, and class label distributions via iteratively applying diffusion sampling to output refined bounding box candidates. Following a filtering mechanism, the teacher decoder produces final pscudo-labels for unlabeled data.
Figure 2:
Diffusion model. Diffusion models [13, 17, 45, 48] are generative models that simulate diffusion process to represent data distributions. In the forward process, noise is progressively introduced to an initial data point, producing intermediate samples that converge the target data distribution. The inference process estimates the posterior distribution of the initial data point from observed sample sequences by reversing the diffusion process. In [45], the diffusion forward process transforms initial data sample
Xo into a noisy sample X, at time by adding noise governed by the noise variance schedule B, [17]. with Or == 1 Br and ae == I Ot.. The forward process to generate a noisy sample x, from X can be defined by
(x, xx) = N (xx: Vazxa, II)
(1)
where N denotes Gaussian distribution and I is an identity matrix. The model is trained to predict Xo from x, by minimizing the l2 loss between Xg and model outputs. During inference, Xo is obtained by gradually denoising from Xx, i.e., + X
Xo-
In this paper, we propose Diffusion-SS3D. diffusion model for semi-supervised 3D object detection. We incorporate random noise for 3D object size and class label distributions in the diffusion model, and then reverse the diffusion process to generate reliable pscudo bounding boxes. Specifically, we define data samples as set of bounding box parameters X = {b,, Il. representing the 3D object size and one-ho hot semantic label respectively. Then our detector is trained to predict Xg from noisy bounding box parameters Xt, conditioned on the features of the given point cloud In SSL, we use this process denoise bounding box candidates and generate final pseudo-labels labels for unlabeled data.
3.2 Algorithm Overview
Existing SSL methods for 3D object detection [16, 53, 63] utilize the reacher-student framework with asymmetric data augmentation integrate labeled and unlabeled data training. One critical factor is to generate pseudo-labele for unlabeled data, in which these approaches would solely rely on model outputs, which may restrict models from generating better pseudo-labels (i.e., lower recall rate). Motivated by the challenge of creating high-quality pseudo-labels, we propose Diffusion-SS3D. new perspective of producing reliable useudo-labels via the diffusion model the teacher-student framework. Figure 2 presents overview of our method.

---

pols I]
Encoder
Decader
Point Cloud p
Representative
Rol Features fobb
Model Predictions
Points and Features
Nolsy Object Size
Gaussian
Distribution
Noisy Distribution
i hit
We illustrate the process of feeding Rol features with noisy object size and label distribu- tions to the decoder before the diffusion sampling step. First, the input point cloud is processed through the encoder, and the Farthest Point Sampling (FPS) is performed obtain N₂ representative points {m, N,, (represented in red points). Concurrently, the noisy object size b, and the noisy label distribution are sampled from a Gaussian distribution. With representative points serving as centers of bounding boxes, Rol features fals within each bounding box are extracted and then concatenated with noisy label distributions, together as the input to the decoder to predict bounding box output.
Figure 3:
In the student model, the goal is supervise model predictions, i.e., y' and ý", for labeled data with ground truths y' and unlabeled data with pseudo-labels y", respectively. More importantly, we establish the training step for the diffusion process by generating the noisy data x, = (b., Ī)r from ground truths of the labeled data, forming noisy bounding box candidates, where b, is the noisy object size and I is the noisy class label distribution. Then, we use the noisy bounding boxes, along with the corresponding features from the encoder, as the input to the decoder for making the predictions. Similarly, for the unlabeled data, the noisy bounding boxes can be generated from pscudo-labels. In each training iteration, the student model also updates the teacher model via an exponential moving average (EMA) mechanism for both the encoder and decoder.
In the teacher model, we aim to generate more reliable pseudo-labels for unlabeled data via the diffusion model. We start with randomly generated object sizes and class label distributions to form noisy bounding boxes, and then adopt the decoder output the predictions similar the process in the student model. Subsequently, we employ the diffusion sampling method [45] to iteratively denoise and produce more accurate object sizes and class labels for pseudo-label generation.
3.3 Diffusion Model Training
In this section, we introduce the components and procedure used train the proposed Diffusion- SS3D framework, including the details of how we process point cloud features with generated noisy bounding boxes and class label distributions train the diffusion model.
Detector encoder. The encoder takes a 3D point cloud scene e IR N333 as input, executing only once during the diffusion training process to extract the feature representations f E RC for each point m E R of p as a C-dimensional feature vector.
Detector decoder. Given a set of N₂ noisy object sizes b, R.Nxx3 and noisy class labels ie N associated with their extracted Rol-features fobj within the object from the encoder, the decoder predicts bounding box regression values and classification results. Details of how the features are extracted are described in the following.
RoI-feature extraction with noisy bounding boxes. We aim to denoise from noisy bounding boxes to train the diffusion model. However, completely sampling random bounding boxes as the noisy data [10] may be too challenging for our case due to the vast 3D space. To increase the chance of producing plausible bounding box candidates with noise, we consider using representative point locations }m;} obtained from the 3D model, in which each point indicates a potential object location. Then, based on these points, we generate noisy bounding boxes by adding random noises to the 3D object size and class label distributions. The overall procedure is illustrated in Figure 3.
5

---

Algorithm 1: Teacher Model
Algorithm 2: Student Model
def paeudo, _geudo_label lpe, stepe, T): Extract points and features ptc, feats Teacher. oncoder
def prject.prediction(pp. gie, ple):
Extract pointe festures pts, feats -
Noisy sizes and class labels sizes_t = normal (mean-0, std-1) [N2. 3] labela_t
a Pad No bounding boxes
sizes = prepara_aiza(gts, pls) # [N6, 3]
(mean=0, std=1) [N,, Na..
labela prepare_labal(tta, pla) [Ns, Nail
Uniform sample step tines reversed T, stepa))
Signal ecaling
sizes (cizes 2 1) size_scale
tine_paira [sat(eip(simea[[-1] timea[t:]))
label: (labels 2 1) label_scale
for t_aur, t_next in zip(tina.paira):
# Corrupt ground trutha randint T) tine_atep sizee_crpt - corrupt corrupt (sizes, t) labele_crpt corrupt
Generate noiay boxes
bozee_t - concat (FPa(p)),,,,,,,,, sizes_t)
# Predict paeudo labels
pla Teacher decordrddcddrrreeo (feats, boxea_t, lables_t, t_cur)
# Generate noiay boxea baxes_crpt concat (FPS(tt)) sizes_crpt)
Estinate sizes_t labele_t t.next sizes_t, labels_t ddin(sizes_tt labels_t, pls, t_cur, t_next)
Predict object candidates
preds Seudent.decoder(ff (feate, boxes_crpt, labels_crpt,
Box renewal
# Supervise by (pseudo) ground truths lasa detactor_losa (preaa gta, pla) Student update 10s2)
eizec_t, labels_t bos_renewalliizee. labela_t)
Filtoring pla filter(pla)
# Update teacher nodel Teacher ENA (Teacher, Student)
return ple
return preda
Sinespace:generas evenly spaced values
(xr t)aaqt( alpha_ cupps_cumprod(t)) x sqrt alpha__cuprodde moise
(t)
Among all M representative points generated by the encoder, we perform the Farthest Point Sampling (FPS) algorithm [31] to obtain a small set of N,, sampled points {m: that maximally cover this 3D scene. We then generate N₂, noisy bounding boxes by treating the sampled points (m;) = as object centers, where each box has a noisy object size b, and noisy class label distributions I. From the encoded features {Li], we process features within each bounding box to form the Rol- features (obj) No
1==1 E NNxC By concatenating the Rol-features and noisy class label distributions = [ I E (C the decoder takes pe as noisy condition to predict the regression values for bounding boxes and classification results.
Diffusion training procedure. As illustrated in the upper part of Figure 2, the diffusion model is trained the student model, where involves adding Gaussian noise to (pseudo) ground truths and training the student decoder to reverse this process. In addition to corrupting (pseudo) ground truths, we generate random bounding boxes and class labels so that total N₂ noisy candidates are produced. Similar to [28], the magnitude of the noise is controlled by cosine schedule that progressively lessens ar (1) along the iterative steps. Note that the coordinates of the ground truth box also need to be adjusted through signal scaling. This is critical as the signal-to-noise ratio (SNR) plays an important role in the performance of the diffusion model [11]. Then, the student model is trained by minimizing the loss between decoder's predictions and (pseudo) ground truths of labeled/unlabeled data. We present the training procedure in Algorithm
3.4 Diffusion Inference for Pseudo- Labels
As illustrated in the lower part of Figure 2, we perform the diffusion inference step in the teacher model to denoise noisy bounding box candidates for unlabeled data. To be specific, we randomly generate the initial noisy data x, = (b., 1}, from a Gaussian distribution and extract N₂, candidate features fobj [fobi i as those in the diffusion training step. Then, the decoder would output model predictions based on robj, followed by the DDIM sampling [17] to iteratively denoise x, and achieve better states, e.g., closer to Xg. Finally, we utilize the denoised model predictions and generate pseudo-labels for unlabeled data. Algorithm I shows the overall inference procedure.

---

Bounding box renewal. During the iterative DDIM sampling step, having filtering scheme can early reject some low-quality candidates that are unlikely to be the final pseudo-labels. Therefore, we apply a bounding box renewal mechanism at each sampling step similar to [10] and replace them by sampling new box sizes with class label distributions from a Gaussian distribution. Specifically, we adopt the filtering scheme originally used in SSL for 3D object detection [53], where the confidence score of each candidate is estimated based on the three metrics: objectness, classification confidence, and intersection--ver--iiio (loU) scores.
3.5 Overall Procedure in SSL
Training objectives for SSL. Our training procedure consists of two phases: pre-training and SSL training. We begin by training the proposed Diffusion-SS3D through supervised learning using labeled data (p). y. to pre-train our detector. Next, we initialize the teacher an student models with the same detector weights, and then jointly train the detector using both labeled data {p'. y:) and unlabeled data {p ) where each training batch includes mixture of labeled and unlabeled scenes. The student model receives both labeled and unlabeled data after strong augmentation, while the unlabeled data with weak augmentation are fed into the teacher model produce pseudo-labels y" through iterative diffusion sampling. The overall loss function L for the student model is defined as L = Li y²) + (p", 5²"), where the Ci and Lu are the detection loss functions used in [53] for bounding box regression and classification. X, set as 2 following [53], is the loss weight to balance the importance of labeled and unlabeled data.
Model evaluation. We utilize the student model to perform the testing stage. Similar to the inference step in Section 3.4, randomly generated bounding box candidates are denoised through the DDIM sampling and bounding box renewal steps to produce final 3D object detection results.
4 Experiments
Datasets. We evaluate our method on two benchmarks, including the ScanNet [12] and SUN RGB- D [46] datasets, with the evaluation settings adopted in the prior semi-supervised 3D object detection works [16, 53, 63]. ScanNet [12], widely used 3D indoor scene benchmark dataset, consists of 1, 1,201 training and 312 validation indoor scenes reconstructed from 2.5 million high-quality RGB-D images. Following the previous work [16, 53, 63], primarily focus on the 18 semantic classes. SUN RGB-D [46], another 3D benchmark dataset, comprises 5,285 training and 5,050 validation scenes, where we evaluate our model on the 10 object classes.
Evaluation metrics. We split both benchmarks into labeled and unlabeled data for SSL., using labeled data ratios of 5%, 10%, and 20% for ScanNet and 1 1%, 5%, and 10% for SUN RGB-D. We evaluate our results using mean average precision (mAP) and report mAP@0.25 (mAP with the 3D IoU threshold of 0.25) and mAP@0.5 scores. We run the experiments under three random data splits and report our averaged performance and the standard deviation.
Implementation details. In this work, we employ PointNet++ [31] as the encoder and IoU-aware VoteNet [53] as the decoder. In the pre-training phase, we only use labeled data with a batch size of 4 to train the diffusion model. The model is trained for 900 epochs with an initial learning rate of 0.005. Like [16, 53], the learning rate then decays at the 400th, 600th, and 800th epochs with factor 0. In the phase of supervisd learning, a batch composed of 4 labeled and 8 unlabeled data. The pre-trained model is used for initializing both the teacher and student models. The student model is trained for 1,000 epochs using the Adam optimizer, with an initial learning rate of 0.005. Like [16, 53], the learning rate decays at the 400th, 600th, 800th, and 900th epochs with factors of 0.3, 0.3, 0.1, and 0.1, respectively.
For the diffusion process, we set the maximum timesteps to 1000 and the number of proposal boxes No 128. We note that during random sampling from the Gaussian distribution, objects are typically present in a small space of 3D scene. Therefore, different from DiffusionDet [10], which sets the mean of the sampling sizes to 0.5, we use a smaller value of 0.25. For noisy labels, the mean of random sampling is adjusted to the reciprocal of the class number. Unless otherwise specified, we perform two DDIM sampling [45] steps in the teacher model to generate pseudo-labels, as well as during evaluation to produce final results. For fair comparisons, we follow 3DloU [53] to employ pseudo-label filtering scheme as a post-processing step on top of our generated pseudo-labels via the diffusion process.
7

---

Table 1: Results on the ScanNet val set with 5%, 10%, 20%, and 100% labeled data.
Model
mAP @ 0.25
mAP 6 0.5
mAP 0.25
10%
MAP @ 0.5
mAP @ 0.25
20%
mAP @ 0.5
mAP @ 0.25
100%
mAP 0.5
[30]
0..9.05
27.1
57.8
SESS [63]
3..0
1444
39.5 1.8
=
49.6
29.0
61.3
38.8
IMithlMath [53]
400+099
0.5
0.
2..3=
52.8 1.2
35.2 1.1
62.9
42.1
Diffusion
43.
Guin (mAP)
3.5
541
3.11
4.87
2.87
1.71
1:21
1.11
Table 2: Results on the SUN RGB-D val set with 1%, 5% 10%, and 20% labeled data.
20%
Model
mAP 6 0:25
MAP 6.0.5
MAP 8 0.25
MAP
MAP & 0:25
mAP 6:
mAP 8 0.25
MAP @ 0.5
WoteNer [30]
SESS [63]
2..
5.3
34.2 2.0
13.1
42.1 1.1
20.
1..
24.5 1
3DloUMatch [53]
21.9 144
8.0
1.
2.7
4 1.5
18.
49.7
30.9
Diffusion-SS3D
14.7+1.
51,4+08
0.64+0.
Gain (mnAP)
9.01
6.7
4.9
381
3.6
16t
1.71
1.51
4.1 Main Results
Tables and 2 show the results of Diffusion-SS3D on the ScanNet and SUN RGB-D datasets with different amounts of labeled data. Overall, our method performs favorably against state-of-the-ar approaches, including VoteNet [30], SESS [63], and 3DIoUMatch [53]. In particular, comparing the main baseline, 3DtoUMatch, Diffusion-SS3D achieves consistent performance improvement across all settings, especially when fewer labeled data are available. For example, for challenging scenarios like 5% ScanNet and I 1% SUN RGB-D settings, our method improves the mAP@0.5 metric by more than 5% in both settings. For higher labeled data ratios or in the fully-supervised setting (i.e., 100%), Diffusion-SS33 demonstrates a consistent 1% improvement in both mAP metrics. The results demonstrate the effectiveness of using the diffusion model to produce more reliable pseudo-labels in SSI. and the fully-supervised scenario (even though it not our main focus).
In addition, we apply our diffusion model to a data augmentation method, OPA [16], which enhances 3D object detection in SSL via learning a point augmentor in the teacher-student framework. Since OPA works on data augmentation on the data input side, we integrate it into our Diffusion-SS3D framework and conduct experiments show the complementary benefit. Specifically, when using 5% labeled data in ScanNet, our method achieves mAP@0.25 of 44.6% and mAP@0.5 of 28. 1%, which are +2.7% and +3.1% better than the OPA approach, respectively. This demonstrates the usefulness of our approach combined with other data augmentation techniques. More details and results are provided in the supplementary materials.
4.2 Ablation Study and Analysis
Diffusion object sizes and labels. In Table 3, we delve into the impact our proposed diffusion model by focusing on two primary aspects: diffusion on object sizes and class label distributions. With the denoising process in object sizes, i.e., ID(2), the performance is improved significantly compared to the baseline without the diffusion model in ID(1). Moreover, adding label denoising further enhances the quality of pseudo-labed and thus achieves the best performance in ID(4). Interestingly, we find that the effectiveness of size denoising is slightly better than label denoising, i.e., comparing ID(2) and (3). Here, we consider this a task-specific behavior, where the parameters to denoised may have different influences on various perception tasks.
DDIM and box renewal. As described in Section 3.4, DDIM sampling [45] and box renewal mechanism are used to infer pseudo-labels in the teacher model. In Table 4, we validate the importance of these two components. First, comparing ID(1) and (2). DDIM sampling contributes via denoising object sizes and class label distributions to obtain better qualities of bounding box candidates for labudo-label generation. Then, comparing ID(1) and (3), it shows that box renewal also has the denoising capability. We note that the effectiveness of box renewal is also enhanced through the diffusion training process, where it utilizes the trained diffusion decoder to integrate updated bounding box features and label distributions. Finally, the best performance is achieved in ID(4) with both DDIM and box renewal, validating our diffusion model's capability in producing reliable pseudo-labels.

---

Table 3: Ablation study on the effect of box size and class label distribution in the diffusion process.
Diftusion Sizes
Diffusion on Labels
ScanNet 5%
SUNRGBD 1%
mAP 6 0.25
mAP 6 0.5
mAP @ 0.25
mAP @ 0.5
(1)
40.5
2288
2199
8.0
(2)
43.1
26.3
30.5
13.8
(3)
42.6 0.6
26.0 0.5
132
(4)
43.5 0.2
27.9 + 0.3
30.9 1.0
14.7 1.2
Table Ablation study on the effect of DDIM and box renewal.
ScanNet 5%
SUNRGBD 1%
ID
DDIM
Box Renewal
mAP @ 0.25
mAP 0.5
mAP 0.25
mAP @ 0.5
(1)
40.5 1.2
12.80...
1..9
8.0 1.5
2
42.8 + 0.5
2.96
0.72
13.9 1.3
42.3 = 0.4
26.8
2.95
12.8 0.6
(4)
43.5 0.2
27.9 0.3
30.9 + 1.0
14.7 1.2
Table 5: Ablation study on various diffusion sampling steps.
ScanNet 5%
SUNRGBD 1%
Num. DDIM
mAP @ 0.25
mAP @ 0.5
mAP @ 0.25
mAP @ 0.5
1
42.6 43.5 0.2
26.2
30.3 0.7
0.27
2
2
27.9 + 0.3
30,9 1.0
14.7 + 1.2
4
44.0 1.1
27.5 # 1.0
28.0 1.1
12.9 0.3
Table 6: Ablation study various scaling factors for SNR.
ScanNet 5%
SUNRGBD 1%
ID
Scaling Factor
mAP 0.25
mAP @ 0.5
mAP @ 0,25
mAP @ 0.5
1.0
43.2 0.3
26.6 + 0.9
30.5 = 0.8
13.2 0.5
(2)
2.0
43.5 + 0.2
26.7 + 0.2
31.3 + 1.1
14.2 # 1.4
4.0
43.5 0.2
27.9 + 0.3
30.9 = 1.0
14.7 1.2
Number of DDIM steps. We study how the diffusion sampling steps, i... DDIM [45], affect the performance. In general, having more sampling steps may better remove noises and obtain outputs closer to ground truths. However, since we address SSL, the denoising steps may have different influence on unlabeled data because we do not have its ground truth for training the diffusion model (e.g., distribution shift from labeled data). In Table 5, we find that having two DDIM steps improves the denoising process, but having four steps may slightly degrade the performance, which is an interestingly finding for SSL. Hence we adopt two sampling steps in all the experiments. Note that, the observed sensitivity of DDIM steps to different datasets is partially attributed to limited labeled data, which may make diffusion sampling steps more sensitive when inferring unlabeled data distribution.
Signal-to-noise ratio. The signal scaling factor is a critical parameter that can affect the signal-to- noise ratio (SNR) during the diffusion process. Table explores the impact of various scaling factors. Overall, our method performs stably with different values, and we choose 4.0 as our final setting in all the experiments. It also worth noting that, in our context like class label distributions, e.g., the 18 classes in ScanNet, there are much fewer parameters compared to more complicated diffusion tasks, e.g., image generation. Therefore, we find that having a higher SNR can be slightly beneficial and make training more stable, as it reduces the difficulty of the denoising process.
Pseudo-label quality. To assess the enhanced quality of pseudo-labels generated by Diffusion-SS3D we evaluate metrics on unlabeled training data during model training, via the teacher model that generates pseudo-labels. In Table 7, we demonstrate that overall the pseudo-label quality of Diffusion- SS3D outperforms 3DtcUMatch [53] by more than 8% improvement in mAP and recall rate. Notably, our diffusion model achieves better quality in earlier epochs and maintains stability throughout the entire training process. It is important to highlight that the semi- cumi-suerreieed performance continues to improve as the model learns from both labeled and unlabeled data over extended training periods. These results show the ability of the diffusion model to consistently generate high-quality pseudo- labels. For visual comparisons of the generated pseudo-labels, please refer to the supplementary materials.

---

Table 7: Results regarding the quality of pseudo-labels generated for unlabeled training data by the teacher model during the training process.
ScanNet 5%
Model
Metric
Lpcch 100
Lpoch 200
Lpoch 400
Epuch 800
Lpoch 1000
MAP @ 0.5
14.08
17.85
21.73
22.17
22.42
3DloUMathh
Recall @ 0.5
27.86
31.49
35.24
36.61
35.25
mAP @
29.98
30,09
30.86
31.01
30.93
Diffusion-SS3D
Recall 0.5
43.73
44.14
45.06
44.72
44.17
Table 8: Ablation study on the effect of the diffusion component and various sampling strategies.
ScanNet 5%
ID
Diffusion
Sampling Stragety
mAP @ 0.25
mAP @ 0.5
Random
40.2 4 1.5
22.3 1.1
(2)
Farthest Point
40.5 # 1.2
22.8
(3)
Random
43.1
27.4 0.6
(4)
Farthest Point
43.5
27.9 + 0.3
Farthest point sampling. In our approach, point sub-sampling is employed to narrow the search space and limit freedom generating noisy boxes, facilitating the diffusion learning process. We explore different sampling strategies, including random point sub-sampling and farthest point sampling (FPS), to select representative centers, as shown in Table 8. Comparing ID(1) and ID(2) reveals similar performance when applying random sampling and FPS to the 3DtcUMatch [53] model without the diffusion component. Similarly, ID(3) and ID(4) show comparable results for random sampling and FPS with our approach. Notably, our diffusion model, utilizing both FPS and random sampling, outperforms the 3DtcUMatch [53] baseline. In practice, we apply FPS following the methods of VoteNet [30] and 3DIoUMatch [53].
4.3 More Discussions
Object orientation. In this paper, we focus on denoising object size and class labels, while consider- ing orientations can be another potential direction for diffusion model. However, incorporating orientation data is challenging due to the discrepancy in orientation quality between ground truths and predictions. as noted in [61]. For example, ScanNet [12] does not provide orientations, i.e., all objects are assigned with orientations as O. Similarly, in SUN RGB-D [46], orientation data varies inconsistently across scenes, complicating the denoising process. Exploring diffusion models capable of handling diverse data types, including noisy ground truths and orientation information, remains an intriguing topic for future research.
Outdoor datasets. Diffusion-SS3D excels on indoor benchmark datasets like ScanNet [12] and SUN RGB-D [46] against existing state-of-the-art methods [16, 53]. Although our approach is not restricted to specific datasets (i.e., integrating diffusion model within general teacher-student framework), exploring outdoor benchmark datasets like KITTI [14] and Waymo Open Dataset [49] may require hyperparameter changes since some object properties (c.g., location, density) in outdoor would be different from the indoor scenario.
Societal impact. Since the diffusion models require more computational powers for training and inference (runtime in frame-per pane-oee- is presented in the supplementary materials), optimizing efficiency is crucial to reduce the environmental footprint for real-time applications and large-scale deployments.
Conclusions
In this paper, we address the challenge of generating high-quality pseudo-labels for 3D object detection in semi-supervised learning. We propose Diffusion-SS3D, a new perspective of employing the diffusion model to produce reliable pseudo-labels. via adding random noise 3D object sizes and class label distributions, and subsequently reversing the diffusion process We then incorporate the diffusion model into a teacher-student framework, which facilitates the learning process with denoised bounding box candidates towards better pseudo-labels. We evaluate the effectiveness of Diffusion-SSS SS3D on the ScanNet and SUN RGB-D benchmark datasets and demonstrate the state-of- the-art performance, with extensive analysis of how the diffusion model affects SSL performance. Our method explores a broader range of reasonable pseudo-labe candidates for SSL 3D object detection, showing the potential of the diffusion model in this domain.
10

---

References
[1] Juan Miguel Lopez Alcaraz and Nils Strodthoff. Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models. arXiv preprint arXiv:2208. 09399, 2022. 3 [2] Tomer Amit, Eliya Nachmani, Tal Shaharbany, and Lior Wolf. SegDiff: Image Segmentation
with Diffusion Probabilistic Models. arXiv preprint wrXiv:2112.00333 2021. 3
[3] Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg.
Structured Denoising Diffusion Models in Discrete State-Spaces. NIPS, 2021. 3
[4] Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended Diffusion for Text-driven Editing of Natural Images. In CVPR, 2022. 3
[5] David Berthelot, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Kihyuk Sohn, Han Zhang, and Colin Raffel. ReMixxaach: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoring. arXiv preprint arXiv:1911.09: 1.09785, 2019. 3
[6] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raffel. MixMatch: A Holistic Approach to Semi-Supervised Learning. NIPS, 2019. 3 [7] Binbin Chen, Weijie Chen, Shicai Yang. Yunyi Xuan, Jie Song, Di Xie, Shiliang Pu, Mingli Song, and Yueting Zhuang. Label Matching Semi-Supervised Object Detection. In CVPR, 2022. 1,
[8] Jian Chen, Ruiyi Zhang, Tong Yu, Rohan Sharma, Zhiqiang Xu, Tong Sun, and Changyou Chen. sabell-etrieval-auggeee arXiv:2305. 19518, 2023.
diffusion models for learning from noisy labels. arXiv preprint
[9] Nanxin Chen, Yu Zhang, Heiga Zen, Ron J Weiss, Mohammad Norouzi, and William Chan. WaveGrrd: Estimating Gradients for Waveform Generation. arXiv preprint arXiv:2009.007 2020. 3
[10] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. DiffusionDet: Diffusion Model for Object Detection. arXiv preprint arXiv:22 1.00788, 2022, 3,5,7
[11] Ting Chen, Lala L.,, Saurabh Saxena, Geoffrey Hinton, and David Fleet. A Generalist Framework for Panoptic Segmentation of Images and Videos. arXiv preprint 10X.0:22
2022. 6
[12] Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes. In CVPR, 2017. 2,
7. 10
[13] Prafulla Dhariwal and Alexander Nichol. Diffusion Models Beat GANs on Image Synthesis.
NIPS, 2021. 2, 4
[14] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. IJRR, 2013. 10
[15] Zhangxuan Gu, Haoxing Chen, Zhuoer Xu, Jun Lan, Changhua Meng, and Weiqiang Wang. DiffusionInst: Diffusion Model for Instance Segmentation. arXiv preprint arXiv:2212.0217
2022. 3
[16] Cheng-Ju Ho, Chen- Tai, Yi-Hsuan Tsai, Yen- Yu Lin, and Ming- Hingg-ssann Yang. Learning Object-level Point Augmentor for Semi-supervised 3D Object Detection. BMVC, 2022. 1, 2, 3,
4. 7, 8, 10
[17] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising Diffusion Probabilistic Models. NIPS,
2020. 2. 4, 6
[18] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax Flows and Multinomial Diffusion: Learning Categorical Distributions. NIPS, 2021. 3
[19] Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak. Consistency- based Semi-supervised
Learning for Object detection. NIPS, 2019. 3
[20] Zhifeng Kong. Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Din: Versatile
Diffusion Model for Audio Synthesis. arXiv preprint arXiv:2009.0976 2020. 3
[21] Samuli Laine and Timo Aila. Temporal Ensembling for Semi-SSperviser Learning. arXiv
preprint 1rXiv:1610.02244
2016. 3
[22] Dong-Hyun Lee et al. Pseudo-Label : The Simple and Efficient Semi-Supervised Learning
Method for Deep Neural Networks. In ICML, 2013. 1.3
[23] Xiang Li, John Thickstun, Ishaan Gulrajani, Percy Liang, and Tatsunori Hashimoto.
Diffusion-LM Improves Controllable Text Generation. NIPS, 2022. 3
[24] Qing Lian, Yanbo Xu, Weilong Yao, Yingcong Chen, and Tong Zhang. Semi-supervised
Monocular 3D Object Detection by Multi-view Consistency. In ECCV, 2022. 3
[25] Yen-Cheng Liu, Chih-Yan Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen
Wu, 7solt Kira, and Peter Vajda. Unbiased Teacher for Superviidd Object Detection.
arXiv preprint arXiv: 09400 2021. 1,3

---

[26] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-Free 3D Object Detection via Transformers. In ICCV, 2021. 1
[27] Soumik Mukhopadhyay, Matthew Gwilliam, Vatsal Agarwal, Namitha Padmanabhan, Archana Swaminathan, Srinidhi Hegde, Tianyi Zhou, and Abhinav Shrivastava. Diffusion models beat gans on image classification. arXiv preprint (xXxv23307087000 2023. 3
[28]
Alexander Quinn Nichol and Prafulla Dhariwal. Improved Denoising Diffusion Probabilistic
Models. In ICML, 2021. 6
[29]
Jinhyung D. Park, Chenfeng Xu, Yiyang Zhou, Masayoshi Tomizuka, and Wei Zhan. DetMatch: Two Teachers are Better Than One for Joint 2D and 3D Semi- comi-Supervised Object Detection. In ECCV, 2022. 3
[30]
Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas. Deep Hough Voting for 3D
Object Detection in Point Clouds. In ICCV, 2019. 1, 8, 10
[31] Charles Ruizhongtai Qi. Li Yi. Hao Su, and Leonidas Guibas. PointNet++: Deep Hicrarchical
Feature Learning on Point Sets in a Metrie Space. NIPS, 2017. 6.
[32] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hicrarchical Text-Conditional Image Generation with CLIP Latents. arXiv preprint arXiv:2204.06125 2022.
3
[33] Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, and Roland Vollgraf. Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows. arXiv
preprint 20xiv:2002.06100 103, 2020. 3
[34]
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-Resolution Image Synthesis with Latent Diffusion Models. In CVPR, 2022. 3
[35] Danila Rukhovich, Anna Vorontsova, and Anton Konushin. FCAF3D: Fully Convolutional Anchor-Free 3D Object Detection. In ECCV, 2022.
[36] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic
Text-to-lmage Diffusion Models with Deep Language Understanding. NIPS, 2022. 3
[37] Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learning. NIPS, 2016. 3
[38] Nikolay Savinov, Junyoung Chung, Mikolaj Binkowski, Erich Elsen, and Aaron van den Oord. Step-unrolled Denoising Autoencoders for Text Generation. arXiv preprini arXiv:21
2021. 3
[39] Shaoshuai Shi, Li Jiang, Jiajun Deng, Zhe Wang, Chaoxu Guo, Jianping Shi, Xiaogang Wang, and Hongsheng Li. PV-RCNN++: Point-Voxel Feature Set Abstraction With Local Vector
Representation for 3D Object Detection. IJC,, 2023. 1
[40] Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. PointRCN: 3D Object Proposal Genera-
tion and Detection from Point Cloud. In CVPR, 2019. 1
[41] Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From Points to Parts: 3D Object Detection from Point Cloud with Part-aware and Part-aggregation Network. TPAMI, pages 2647-2664, 2020.
[42] lascha Sohi-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep Unsuper- vised Learning using Nonequilibrium Thermodynamics. In /CMI., 2015. 3
[43] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. FixMatch: Simplifying Semi- Supervised Learning with Consistency and Confidence. NIPS, 2020. 1,3
[44] Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Yu Lee, and Tomas Pfis- ter. A Simple Semi- -eeiiSSpeee Learning Framework for Object Detection. arXiv preprint arXiv:2005.0473
2020. 1,
[45] Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising Diffusion Implicit Models. arXiv
preprint arXiv:2010. /0220000.0 2020. 2, 4, 5, 7,
[46] Shuran Song, Samuel P Lichtenberg. and Jianxiong Xiao. SUN RGB-D: A RGB-D Scene
Understanding Benchmark Suite. In CVPR, 2015. 2, 7, 10
[47] Yang Song and Stefano Ermon. Generative Modeling by Estimating Gradients of the Data Distribution. NIPS, 2019. 3
[48] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-Bard Based Generative Modeling through Stochastic Differential Equations. arXiv preprint 2rXiv:2011.13456 2020. 2, 3.
[49] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai. Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for
12

---

autonomous driving: Waymo open dataset. In CVPR, 2020. 10
[50] Peng Tang, Chetan Ramaiah, Yan Wang. Ran Xu, and Caiming Xiong. Proposal Learning for
Semi -upervvsed Object Detection. In WACV, 2021. 3
[51] Antti Tarvainen and Harri Valpola. Mean teachers are better role models: Weight-average
consistency targets improve semi-supprvised deep learning results. NIPS, 2017. 2, 3
[52] Yusuke Tashrro, Jiaming Song, Yang Song, and Stefano Ermon. CSDI: Conditional Score-based
Diffusion Models for Probabilistic Time Series Imputation. NIPS, 2021. 3
[53] He Wang, Yezhen Cong. Or Litany, Yue Gao, and Leonidas J Guibas. 3DioUMatch: Leveraging IoU Prediction for Seni--Spervieed 3D Object Detection. In CVPR, 2021. 1, 2. 3, 4, 7. 8, 9, 10 [54] Haiyang Wang, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, Liwei Wang. et al. CAGroup3D: Class-Aware Grouping for 3D Object Detection on Point Clouds. NIPS,
2022.
[55] Yan Wang, Junbo Yin, Wei Li, Pascal Frossard, Ruigang Yang, and Jianbing Shen. SSDA3D: Semi-supervised Domain Adaptation for 3D Object Detection from Point Cloud. arXiv preprint
arXiv.221.002845, 2022. 1
[56] Qizhe Xie, Zihang Dai, Eduard Hovy. Thang Luong, and Quoe L.e. Unsupervised Data Augmentation for Consistency Training. NIPS, 2020. 3
[57] Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng Liu. End-to-End Semi-Supervised Object Detection with Soft Teacher. In ICCV,
2021. 3.3
[58] Tijin Yan, Hongwei Zhang, Tong Zhou, Yufeng Zhan, and Yuanqing Xia. ScoreGrad: Multivari- ate Probabilistic Time Series Forecasting with Continuous Energy-based Generative Models. arXiv preprint arXiv: 106. 10121, 2021. 3
[59] Zetong Yang. Yanan Sun, Shu Liu, and Jiaya Jia. 3DSSD: Point-based 3D Single Stage Object
Detector. In CVPR. 2020. 1
[60] Junbo Yin, Jin Fang. Dingfu Zhou, Liangjun Zhang. Bheng-Zhong Xu, Jianbing Shen, and Wenguan Wang. Semi-supervised 3D Object Detection with Proficient Teachers. In ECCV,
2022.
[61]
Hong-Xing Yu, Jiajun Wu, and Li Yi. Rotationally Equivariant 3D Object Detection. In CVPR,
2022. 10
[62]
Peiyu Yu, Sirui Xie, Xiaojian Ma, Baoxiong Jía, Bo Pang, Ruigi Gao, Yixin Zhu, Song-Chun Zhu, and Ying Nian Wu. Latent Diffusion Energy-Based Based Model for Interpretable Text Modeling,
arXiv preprinl urXiv:2206.000 05895, 2022. 3
[63] Na Zhao, Tat-Seng Chua, and Gim Hee Lee. SESS: Self-Ennembling Sunii-upervieed 3D Object Detection. In CVPR, 2020. 1, 2, 7,8
[64]
Hongyu Zhou, Zheng Ge. Songtao Liu, Weixin Mao, Zeming Li, Haiyan Yu, and Jian Sun. Dense Teacher: Dense Pseudo-Labels for Semi-supervised Object Detection. In ECCV, 2022. I
[65] Yin Zhou and Oncel Tuzel. VoxelNet: End-to-End Learning for Point Cloud Based 3D Object
Detection. In CVPR, 2018.
[66] Ye Zhu, Yu Wu, Kyle Olszewski, Jian Ren, Sergey Tulyakov, and Yan Yan. Discrete Contrastive Diffusion for Mross-Modal Music and Image Generation. arXiv preprint 0777712206.07771,
2022. 3
13


Reviews:
[{"review_id": "GS2zv4cwxh", "reviewer": "Reviewer_7AAY", "summary": "The paper proposes a novel algorithm utilizing the diffusion model in 3D object detection for generating pseudo-labels for semi-supervised learning. Technically, it adopts the previous method of teacher-student architecture, but extends it by introdusing diffusion to generate pseudo-labels and denoising technique for class labels and object size distribution. The model incorporates both labeled and unlabeled data through asymetric augmentation mechanism.", "questions": "Please discuss the role of the teacher network.", "limitations": "I also encourage authors to add societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The novel algorithm creates high-quality pseudo-labels for unlabled data and higly increases the performance of the current techniques in 3D object detection. \n- The qualitative and quantitative results of the proposed approach demonstrate good improvements over existing methods.  This indicates the effectiveness and superiority of the proposed approach in achieving better performance and accuracy for 3D object detection.\n- Also, the source code is available.\n", "weaknesses": "- There needs to be ablative study for farthest point sampling. I am curious this provides the confidential results. I encourage authors to visualize its results.\n\n- It would be helpful if there's additional ablative study for the baseline that does not use any teacher network but train the student network using the proposed method. I am curious why teacher model is necessary.\n\n- Less qualitative results. I encourage authors to add more qualitative results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback and we address each question below.\n\n**Q1: Farthest point sampling.**\n\nWe first note that we follow the common implementation in VoteNet and 3DIoUMatch to apply farthest point sampling (FPS) in our framework. To study the impact of FPS, we further conduct an experiment to use the random point sub-sampling method. Results shown in the table below indicate that both FPS and random sampling provide competitive performance improvement compared to the 3DIoUMatch baseline. We will add this result to the final version.\n\n| ID | Diffusion | Sampling | ScanNet 5% mAP@0.25 | ScanNet 5% mAP@0.5 |\n|:---:|:---:|:---:|:---:|:---:|\n| (1) |  | Random | 40.2 ± 1.5 | 22.1 ± 1.1 |\n| (2) |  | FPS | 40.0 ± 0.9 | 22.5 ± 0.5 |\n| (3) | ✓ | Random | 43.1 ± 0.6 | 27.4 ± 0.6 |\n| (4) | ✓ | FPS | 43.5 ± 0.2 | 27.9 ± 0.3 |\n\nTo visualize the point sampling results via FPS, Figure 3 of the main paper illustrates one example. The red points in the top of figure are sampled points, in which we consider them as potential object centers to further produce random object bounding boxes in the diffusion process. We will clarify this in the final version.\n\n**Q2: Why teacher model is necessary.**\n\nFor semi-supervised setting, the teacher-student framework plays a key role to handle unlabeled data, where the teacher model is utilized to generate pseudo-labels from unlabeled data, serving as a supervisory signal for the learning process in the student model. In our approach, we utilize the common teacher-student framework and integrate the diffusion process, so that the diffusion sampling occuring in the teacher model generates pseudo-labels. While it may be possible to only use a single student model and generate pseudo-labels, it would not be an effective way in semi-supervised learning (e.g., the VoteNet baseline) due to the absence of a more stable teacher model (i.e., updated via an exponential moving average scheme from the student model) for pseudo-label generation. Note that, all the main methods in the paper (e.g., SESS, 3DIoUMatch) use the same teacher-student framework for fair comparisons.\n\nTo realize the suggested experiment without the teacher model, we conduct an experiment of using 100% labeled data via the proposed diffusion model. In this way, there is no need to generate pseudo-labels from unlabeled data, so that the teacher model is not critical anymore. Specifically, we train our diffusion model with labeled data using the same way as the student model does (top of Figure 2 in the main paper). During inference, random data distribution is then generated (see Figure 3 of the main paper) and denoised via DDIM sampling to produce final predictions. We show the results in the table below on ScanNet, where our method performs better than the baseline without diffusion by more than 1%. Although the fully-supervised setting is not our main focus in this paper, this demonstrates the potential of introducing the diffusion process in more settings.\n\n| Model | 100% mAP@0.25 | 100% mAP@0.5 |\n|:---:|:---:|:---:|\n| VoteNet | 57.8 | 36.0 |\n| SESS | 61.3 | 38.8 |\n| 3DIoUMatch | 62.9 | 42.1 |\n| Diffusion-SS3D | **64.1** | **43.2** |\n| Gain (mAP) | +1.2 | +1.1 |\n\n**Q3: Qualitative results.**\n\nIn our supplementary materials, Figure 1 and 2 demonstrate the effectiveness of our diffusion process, showcasing the progressive improvement from random noisy boxes to final predictions. In addition, we have included more visual comparisons of generated pseudo-labels in the rebuttal pdf file. We will show more qualitative results in the final version as suggested.\n\n**Q4: Societal impact.**\n\nSince the diffusion model requires more computational powers for training and inference (runtime in frame-per-second is presented in Table 4 of the supplementary material), the increased energy consumption may cause an environmental impact. Therefore, it is worth exploring more eco-friendly computing strategies to reduce the environmental footprint. We will add more discussions to the final version."}, {"review_id": "phxVo0FghB", "reviewer": "Reviewer_TXhm", "summary": "This paper proposes a semi-supervied 3D object detection framework, named Diffusion-SS3D. Diffusion-SS3D introduces diffusion process to improve the quality of pseudo-labels. The authors perform experiments on ScanNet and SUN RGB-D benchmark to verify the effectiveness. ", "questions": "Please see the weaknesses.", "limitations": "The method proposed by the author has achieved good results in indoor datasets, e.g., ScanNet and SUN RGB-D. What about the performance on outdoor datesets? And what about the performance of the proposed method in multi-camera 3D detection methods?", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ The motivation of this paper makes sense.\n+ The experimental results are promising.\n+ The combination of semi-supervised 3D object detection and diffusion is interesting and novel. ", "weaknesses": "- The statements in the method part are too long. The writing should be organized again. Try to use more detailed figure to introduce your method.\n- How to evaluate the quality of pseudo-labels? It’s better to provide statistical data or visualization to support your claims.\n- In Tables 5 and 6, it seems that the diffusion steps and scaling factors for SNR are sensitive to different datasets. What is the reason for the above? Please provide some insights or analysis on this phenomenon (maybe the amount of labeled data is limited?). ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback and we address each question below.\n\n**Q1: Pseudo-label quality.**\n\nTo validate whether the quality of pseudo-labels is improved, we evaluate the metrics on unlabeled training data during model training, via the teacher model that generates pseudo-labels. In the table below, we show that overall the pseudo-label quality of our Diffusion-SS3D is better than 3DIoUMatch by more than 8% improvement in mAP and recall rate. In addition, we observe that our diffusion model is able to achieve a better quality in earlier epochs, and then become stable during the entire training process. Note that, we still observe the improvement of semi-supervised performance during training, since the model needs to be trained longer and learn from both labeled and unlabeled data. These results support our claim that the diffusion model can generate high-quality pseudo-labels and we will add them to the final version. In addition, we have included more visual comparisons of generated pseudo-labels in the rebuttal pdf file.\n\n| ScanNet 5% | Metric | Epoch 100 | Epoch 200 | Epoch 400 | Epoch 600 | Epoch 800 | Epoch 1000 |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 3DIoUMatch | mAP@0.5 | 14.08 | 17.85 | 21.73 | 21.95 | 22.17 | 22.42 |\n|  | Recall@0.5 | 27.86 | 31.49 | 35.24 | 36.22 | 36.61 | 35.25 |\n| Diffusion-SS3D | mAP@0.5 | 29.98 | 30.09 | 30.86 | 30.55 | 31.01 | 30.93 |\n|  | Recall@0.5 | 43.73 | 44.14 | 45.06 | 44.75 | 44.72 | 44.17 |\n\n**Q2: Sensitivity in Tables 5 and 6.**\n\nAs discussed in Ln 291-297 of the main paper, one reason for the sensitivity of diffusion sampling steps (Table 5 of the main paper) to different datasets is indeed the limited labeled data as the reviewer suggested. Since the diffusion training process is mainly guided by the limited labeled data, it makes diffusion sampling steps more sensitive when inferring on unlabeled data distribution. Similarly for Table 6 and Ln 298-304 of the main paper, since the scaling factor determines the difficulty of noisy data (i.e., how difficult to denoise), having limited labeled data would also make the training process more challenging. We will clarify this in the final version.\n\nNevertheless, in both Table 5 and 6 of the main paper, no matter whether there are slight deviations across different settings, we can conclude that using DDIM step equal to 2 and the scaling factor equal to 4.0 should be a general rule-of-thumb that achieves better results compared to the baseline without diffusion. \n\n**Q3: Outdoor dataset.**\n\nDue to limited time to experiment with a new training framework (i.e., the outdoor dataset needs a different baseline than VoteNet we use in the paper), we will include the results in the final version. Note that, by design of the proposed diffusion model in a general teacher-student framework, our method should not be limited to certain datasets. However, we do recognize that some changes may be required as some object properties (e.g., location, density) in outdoor would be different from the indoor scenario.\n\nFor the multi-camera setting, it is an interesting direction as it may provide more information, e.g., pseudo-label consistency from multi-views in the diffusion training process, which can be considered as a future work as it is beyond the scope of this work. We will include these discussions in the final version.\n\n**Q4: Writings.**\n\nThank you for the feedback. We will improve the writings and figures in the final version."}, {"review_id": "xbowJdIPUl", "reviewer": "Reviewer_PTFH", "summary": "In this paper, the author argues that previous 3D semi-supervised detection methods relied solely on teacher models, which cannot generate sufficiently reliable pseudo-labels. Therefore, the author proposes a method called Diffusion-SS3D. This method enables diffusion learning to remove noise from corrupted 3D object size and class label distributions, thereby optimizing the pseudo-labels generated by the teacher model and obtaining more reliable pseudo-labels. The proposed method achieves state-of-the-art performance on the ScanNet and SUN RGB-D datasets.\n", "questions": "1.I am curious to know how much difference there would be in the performance if we trained diffusion and generated pseudo-labels using a sampled object size and class distribution based on the prior knowledge obtained from the dataset labels.\n\n2.Since diffusion is just one module within the entire teacher-student framework, the metrics on the dataset may not directly indicate the reliability of diffusion as a source of pseudo-labels. Could the authors design an experiment that demonstrates the improved quality of pseudo-labels obtained through diffusion denoising?\n", "limitations": "1.The authors did not provide a detailed discussion on the limitations of the method. They only briefly mentioned in the final part of the experiment section that diffusion does not perform well in denoising orientations. In this case, could further exploration be conducted to investigate the conditions under which the proposed method performs well in terms of ground truth and predictions?\n\n2.From my perspective, the diffusion model has certain computational costs, and it may be worth discussing the practicality of the method \nfrom this angle.\n", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Compared to previous works such as 3DIoUMatch, the approach of utilizing diffusion for denoising instead of relying solely on threshold filtering to generate better pseudo-labels is intriguing.\n", "weaknesses": "1.In my opinion, the paper suggests that diffusion can generate more reliable pseudo-labels, but it does not clearly explain why the pseudo-labels generated by diffusion are more reliable than those generated by previous works.\n\n2.In Table 4, the performance of using only Box Renewal is not provided. I believe that both DDIM and Box Renewal have certain denoising capabilities. If the paper considers DDIM to be effective, it should provide the results of using only Box Renewal to further demonstrate this viewpoint.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback and we address each question below.\n\n**Q1: Why are the pseudo-labels generated by diffusion are more reliable?**\n\nIn Figure 1 of the main paper, we illustrate the fundamental difference between the conventional framework and our diffusion model in pseudo-label generation, which results in different pseudo-label qualities. First, prior works like 3DIoUMatch are designed to refine pseudo-labels purely based on model outputs, in which the objects may not be discovered if the model cannot output sufficient predictions (i.e., lower recall rate).\n\nOn the other hand, our diffusion process starts with random box sizes and label distributions, which do not depend on model predictions, so there is a higher chance (i.e., higher recall rate) we can find more objects (please refer to the table provided in Q4). Then, through the denoising process in our diffusion model, pseudo-labels are refined to be more reliable. We will add this discussion to the paper.\n\n\n**Q2: Performance of box renewal.**\n\nWe experiment with only using box renewal in our diffusion framework, as shown in the table below. We observe that both DDIM and box renewal have the denoising capability. Note that the effectiveness of box renewal also benefits from the diffusion training process, in which box renewal requires the trained diffusion decoder (as shown in Algorithm 1 of the main paper), so that the decoder can take the updated bounding box features and label distributions from box renewal. Therefore, we would emphasize that the entire diffusion training framework makes major improvements rather than a single DDIM or box renewal component, i.e., box renewal may not be solely a method without the diffusion training step. We will add this result with more discussions to the final version.\n\n| ID | DDIM | Box Renewal | ScanNet 5% mAP@0.25 | ScanNet 5% mAP@0.5 |\n|:---:|:---:|:---:|:---:|:---:|\n| (1) |  |  | 40.5 ± 1.2 | 22.8 ± 0.8 |\n| (2) | ✓ |  | 42.8 ± 0.5 | 26.6 ± 0.9 |\n| (3) |  | ✓ | 42.3 ± 0.4 | 26.8 ± 0.3 |\n| (4) | ✓ | ✓ | **43.5** ± 0.2 | **27.9** ± 0.3 |\n\n**Q3: Prior knowledge in diffusion training.**\n\nThanks for the suggestion. Using prior knowledge in the diffusion process is interesting, but this may require more studies. For instance, we may consider using category-specific random sampling based on the possible object size as the prior. However, this may be non-trivial as we do not assume to know which category to perform random sampling during inference. In practice, we experiment and find that randomly sampling the object size within 1/4 of the entire scene is more effective than using a larger object size. This supports that having some prior knowledge should be still a helpful cue and we will consider this as a future work.\n\n**Q4: Pseudo-label quality.**\n\nTo validate whether the quality of pseudo-labels is improved, we evaluate the metrics on unlabeled training data during model training, via the teacher model that generates pseudo-labels. In the table below, we show that overall the pseudo-label quality of our Diffusion-SS3D is better than 3DIoUMatch by more than 8% improvement in mAP and recall rate. In addition, we observe that our diffusion model is able to achieve a better quality in earlier epochs, and then become stable during the entire training process. Note that, we still observe the improvement of semi-supervised performance during training, since the model needs to be trained longer and learn from both labeled and unlabeled data. These results support our claim that the diffusion model can generate high-quality pseudo-labels and we will add the results to the final version. In addition, we have included more visual comparisons of generated pseudo-labels in the rebuttal pdf file.\n\n| ScanNet 5% | Metric | Epoch 100 | Epoch 200 | Epoch 400 | Epoch 600 | Epoch 800 | Epoch 1000 |\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 3DIoUMatch | mAP@0.5 | 14.08 | 17.85 | 21.73 | 21.95 | 22.17 | 22.42 |\n|  | Recall@0.5 | 27.86 | 31.49 | 35.24 | 36.22 | 36.61 | 35.25 |\n| Diffusion-SS3D | mAP@0.5 | 29.98 | 30.09 | 30.86 | 30.55 | 31.01 | 30.93 |\n|  | Recall@0.5 | 43.73 | 44.14 | 45.06 | 44.75 | 44.72 | 44.17 |\n\n**Q5: Limitations.**\n\nIn diffusion model training, it relies on ground-truth information to perform the denoising process. For the datasets we experiment on, we find that the orientation information may not be available or inaccurate. For example, ScanNet does not provide orientations, i.e., all objects being asigned with orientations as 0. For SUN RGB-D, orientations are provided inconsistently across different scenes, which causes the training difficulty in denoising orientations (Ln 308-310). We will consider this as a future work to explore noisy data for diffusion models.\n\nFor other limitations, since the diffusion model requires more computational powers for training and inference (runtime in frame-per-second is presented in Table 4 of the supplementary material), optimizing efficiency is crucial for real-time applications and large-scale deployments. In the meantime, the increased energy consumption may cause an environmental impact, so it is worth exploring more eco-friendly computing strategies to reduce the environmental footprint. We will include more discussions in the final version.\n\n**Q6: Computational costs.**\n\nOne of the limitations of the diffusion model is its computational costs as described above (highlighted in Table 4 of the supplementary material). For example, compared to our baseline method without diffusion, i.e., 3DIoUMatch, our runtime speed is decreased by 28.8% (from 65.54 FPS to 46.64 FPS) while the performance in mAP@0.5 relatively improves by 61.3% (from 8% to 12.9%) on SUN RGB-D (Ln 61-65 of the supplementary material). We will add more discussions regarding this limitation, e.g., adjusting the diffusion sampling steps to achieve a trade-off between accuracy and efficiency as shown in Table 4 of the supplementary material."}, {"review_id": "gN5rwD0yZL", "reviewer": "Reviewer_powF", "summary": "This paper introduces a novel approach to enhance the accuracy of pseudo-labels and inference results in semi-supervised 3D object detection through the utilization of diffusion models. The authors propose the integration of diffusion models from two perspectives: 3D object sizes and class label distributions. The application of diffusion models to 3D object sizes aims to improve the recall rate of ground-truth objects, while the denoising process of class label distributions addresses the issue of low accuracy in category predictions for pseudo-labels. The paper includes comprehensive ablation studies and detailed analysis to validate the effectiveness of the proposed approach.", "questions": "Is there any existing research that demonstrates the effectiveness of denoising classification logits through the diffusion process on label distributions in other classification-based tasks, such as image classification? It would be beneficial to discuss in the related work section if there are any existing studies.", "limitations": "The authors did not discuss the limitations or potential negative social impact of the proposed method.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The authors present a comprehensive and detailed explanation of the algorithm, providing readers with a clear understanding of the proposed method and its underlying principles.\n2. The use of diffusion models to improve the quality of pseudo labels in semi-supervised 3D object detection is novel. \n", "weaknesses": "1. The authors should include a comparison on the 20% labeled data, as it is a commonly used setting in the literature, such as in SESS and 3DIoUMatch. Additionally, it would be beneficial to include comparisons at larger labeled ratios, such as 50% and 100%, to provide a more comprehensive analysis of the proposed method's performance.\n\n2. Although the use of representative points in the proposed diffusion model for 3D object sizes is mentioned as a distinction from the 2D object detection approach [9], it is a straightforward idea as point sub-sampling is commonly employed in 3D object detection to reduce the search space. Moreover, it would be valuable to investigate the effects of using different sub-sampling techniques, such as random sub-sampling, and compare them with FPS sampling.\n\n3. The diffusion model for refining bounding boxes appears similar to the IoU Optimization technique proposed in 3DIoUMatch, which aims to optimize the center and size of bounding boxes. It would be informative to clarify whether the IoU Optimization technique has been applied to the pseudo labels when comparing them with the results from 3DIoUMatch.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback and we address each question below.\n\n**Q1: Comparisons on the 20% labeled data or larger labeled ratios.**\n\nFor ScanNet, the results of using 20% labeled data are reported in Table 1 of the main paper. For SUN RGB-D, we show in the table below that our Diffusion-SS3D performs better than 3DIoUMatch by more than 1.5% in both mAP@0.25 and mAP@0.5. Due to limited time and computational resources, we present the new results of using one random data split and will include full results using more random splits in the final version.\n\n| SUN RGB-D | 20% mAP@0.25 | 20% mAP@0.5 |\n|:---:|:---:|:---:|\n| VoteNet | 45.7 ± 0.6 | 22.5 ± 0.8 |\n| SESS | 47.1 ± 0.7 | 24.5 ± 1.2 |\n| 3DIoUMatch | 49.7 ± 0.4 | 30.9 ± 0.2 |\n| Diffusion-SS3D | **51.3** | **32.7** |\n| Gain (mAP) | +1.6 | +1.8 |\n\nFor higher ratios of labeled data or the fully-supervised setting (i.e., 100%), we did run the 100% setting on ScanNet and the results are shown in the table below. We find that our Diffusion-SS3D has more than 1% improvement in both mAP metrics. In addition, we also include the results of 20% labeled data as reference (also shown in Table 1 of the main paper). Note that, although the fully-supervised setting is not our main focus in this paper, these results demonstrate the potential of introducing the diffusion process in more settings.\n\n| ScanNet | 20% mAP@0.25 | 20% mAP@0.5 | 100% mAP@0.25 | 100% mAP@0.5 |\n|:---:|:---:|:---:|:---:|:---:|\n| VoteNet | 46.9 ± 1.9 | 27.5 ± 1.2 | 57.8 | 36.0 |\n| SESS | 49.6 ± 1.1 | 29.0 ± 1.0 | 61.3 | 38.8 |\n| 3DIoUMatch | 52.8 ± 1.2 | 35.2 ± 1.1 | 62.9 | 42.1 |\n| Diffusion-SS3D | **55.6** ± 1.7 | **36.9** ± 1.4 | **64.1** | **43.2** |\n| Gain (mAP) | +2.8 | +1.7 | +1.2 | +1.1 |\n\n**Q2: Effects of using different sub-sampling techniques.**\n\nAlthough we are aware of point sub-sampling as a common strategy in 3D object detection, we find that leveraging this scheme in our diffusion model can reduce the search space and decrease the degree of freedom in generated noisy boxes, facilitating the diffusion learning process, in which the 2D case [9] does not have the similar issue.\n\nWe show more results in the table below regarding the point sampling strategies. With the suggested random sub-sampling, we find that results are similar to the ones using FPS (i.e., within 0.5% differences in mAP), for both models with and without our diffusion component. In practice, we follow the implementation of the VoteNet and 3DIoUMatch methods that use FPS. We will add these results in the final version.\n\n| ID | Diffusion | Sampling | ScanNet 5% mAP@0.25 | ScanNet 5% mAP@0.5 |\n|:---:|:---:|:---:|:---:|:---:|\n| (1) |  | Random | 40.2 ± 1.5 | 22.1 ± 1.1 |\n| (2) |  | FPS | 40.0 ± 0.9 | 22.5 ± 0.5 |\n| (3) | ✓ | Random | 43.1 ± 0.6 | 27.4 ± 0.6 |\n| (4) | ✓ | FPS | 43.5 ± 0.2 | 27.9 ± 0.3 |\n\n**Q3: IoU optimization technique from 3DIoUMatch.**\n\nWe first note that the 3D IoU prediction in 3DIoUMatch is used as a filtering scheme based on model outputs to find high-quality pseudo-labels. In contrast, our diffusion model learns to denoise from random box sizes and labels to form pseudo-labels. Therefore, applying 3DIoUMatch's filtering scheme complements our method as a post-processing step. In practice, since we consider 3DIoUMatch as our baseline, we do use the same filtering step in our framework after the diffusion steps. We will clarify it in the final version.\n\n**Q4: Existing works using diffusion process on label distributions.**\n\nThanks for the suggestion. Since using diffusion models for recognition tasks are relatively new, we have tried our best to include the work we are aware of in the Related Work section of the main paper. We do find two concurrent works [A, B] (both published on arXiv after the NeurIPS submission deadline) that involve the diffusion process for image classification. [A] uses the embedding of the image generation model for learning a classifier, while [B] tackles the learning process of noisy labeled data via the diffusion model. We will include and discuss both works in the final version.\n\n[A] Mukhopadhyay et al., Diffusion Models Beat GANs on Image Classification, arXiv:2307.08702, July 2023.\n\n[B] Chen et al., Label-Retrieval-Augmented Diffusion Models for Learning from Noisy Labels, arXiv:2305.19518, May 2023.\n\n**Q5: Limitations and social impact.**\n\nSince the diffusion model requires more computational powers for training and inference (runtime in frame-per-second is presented in Table 4 of the supplementary material), optimizing efficiency is crucial for real-time applications and large-scale deployments. In the meantime, the increased energy consumption may cause an environmental impact, so it is worth exploring more eco-friendly computing strategies to reduce the environmental footprint. We will include more discussions in the final version."}, {"review_id": "g2AcXyFKIn", "reviewer": "Reviewer_5wDL", "summary": "The paper addresses the problem of semi-supervised 3D object detection (3DOD) where the aim is to train a 3DOD model using only few labelled and a lot of unlabelled data. On top of previous approaches, the authors propose to use the diffusion mechanism to enhance the pseudo labels generated by a teacher model for the unlabeled scenes. The main idea is to denoise noisy initial bounding boxes and class label distributions via the diffusion mechanism and thereby obtain better pseudo labels, which are used to train a student model. Additionally, the authors use the diffusion-based denoising to enhance the initial output of the student model during inference. Experiments on ScanNet and SUN RGB-D show that the approach outperforms previous approaches.", "questions": "1. The authors have evaluated their method on two indoor datasets. However, many recent works, e.g., [a], [b], mainly evaluate on outdoor datasets for 3DOD. Have the authors tried, how their method performs on such datasets? Maybe it makes sense to mention as a limitation that the method’s efficacy has not yet been established for such outdoor datasets.\n2. The authors claim that the pseudo labels are improved by the diffusion mechanism. Is there an evaluation that supports this claim, i.e., an evaluation of the pseudo labels of the author’s method vs. a baseline without diffusion?\n3. As far as I understood, the authors make use of the diffusion mechanism in two ways: First, the pseudo labels are improved and second as a refinement technique of 3DOD prediction during inference. I was wondering, if the second aspect is limited to semi-supervised 3DOD? Could it also be used to enhance fully supervised 3DOD?\n\n[a] Park et al., “DetMatch: Two Teachers are Better than One for Joint 2D and 3D Semi-Supervised Object Detection,” ECCV 2022.\n\n[b] Lian et al., “Semi-supervised Monocular 3D Object Detection by Multi-view Consistency,” ECCV 2022.", "limitations": "Limitations of the method have been discussed to some degree. I think it would be interesting to additionally state if the method has limitations w.r.t. outdoor datasets and/or in terms of added complexity during training or inference.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Abstract and introduction provide a clear motivation for the considered problem as well as for the proposed solution. The contributions over previous works are clearly stated and discussed.\n2. The method description and the mathematical definitions are clear and easy to follow. Figures and the algorithm are nicely prepared and facilitate the method description’s clarity.\n3. The proposed method clearly outperforms the considered baselines. \n4. Various ablation studies show the effect of various method components and parameters.\n5. Nice qualitative examples in the supplementary. It would be nice to refer to them in the main paper.", "weaknesses": "1. For a large part of the paper it remained a bit unclear to me, if the author’s method is applied during training or during inference. In the abstract and introduction, it first sounds like the method is applied only during training, but later on it becomes clear that the diffusion mechanism is also applied during inference to refine the initial predictions. I think this should be clearly stated in abstract and introduction.\n2. In the SOTA comparison it remains unclear to me, which part of the improvement is due to the enhanced pseudo labels during training and which part is due to the refinement of predictions during inference. It would help, if the baseline without the author’s method could be added to the Table. Additionally, it would help, if a variant of the method using only training-side improvement but not the prediction enhancement during inference would be added. Thereby, it becomes clear, how much improvement can be achieved without additional inference complexity and how much improvement can be achieved due to enhancement of the predictions at the cost of additional inference complexity.\n\nAlthough, there are a few weaknesses in this paper, I think this work could present a valuable contribution to NeurIPS, if the mentioned issues can be addressed in the rebuttal.\n\nMinor comments and suggestions:\n\n3. In Figure 1, it would help the clarity if the abbreviations such as EMA were not abbreviated.\n4. Section 3.1: the symbol l is used two times with different meaning: First, it is an index for labelled (line 116) and then as an index for length (line 121).\n5. The second paragraph in Section 4.1 discusses results which I could not find in any of the tables. It would help to add the comparison to, e.g., Tables 1 and 2.\n6. The baselines in the ablations in Tables 3 and 4 are exactly equal to the results of 3DIoUMatch. Did the authors resimulate those results? I think it would make sense to resimulate them for the ablation studies to avoid impact of differing hardware or hyperparameters on the ablation results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive feedback and we address each question below.\n\n**Q1: Diffusion applied during training or inference.**\n\nOur diffusion mechanism is applied during both the training and inference stages. However, like most diffusion models, our Diffusion-SS3D is applied to denoise randomly generated inputs during inference. To be specific, our diffusion model is first trained under the SSL setting. Then, during inference, the model would denoise from randomly generated object sizes and label distributions via DDIM sampling to generate final predictions.\n\nAlthough the diffusion model is designed to perform denoising during inference, it is still possible to directly use our learned decoder and output model predictions without the denoising step. We show results in the table below on ScanNet, where the DDIM step equal to 0 indicates no denoising step during inference. Our model still performs competitively due to our diffusion model training stage. The performance is improved further using more DDIM steps. We will clarify this in the final version and add more results.\n\nDDIM steps in inference|5% mAP@0.25|5% mAP@0.5|\n|:---:|:---:|:---:|\n|0|42.8 ± 0.5|26.8 ± 0.6|\n|1|43.1 ± 0.4|27.0 ± 0.7|\n|2|43.5 ± 0.2|27.9 ± 0.3|\n\n**Q2: Improvement due to the enhanced pseudo labels during training or inference, and its complexity.**\n\nAs indicated in Q1 above, we report results of having the diffusion process in both training and inference or merely training on ScanNet, as well as the baseline without our method (i.e., 3DIoUMatch). Overall, with diffusion model training but without the denoising step during inference, the results are improved significantly from the 3DIoUMatch baseline while having the same runtime speed. With the denoising process during inference, the results are improved further.\n\n|Diffusion training|DDIM in inference|5% mAP@0.25|5% mAP@0.5|\n|:---:|:---:|:---:|:---:|\n|||40.5 ± 1.2|22.8 ± 0.8|\n|✓||42.8 ± 0.5|26.8 ± 0.6|\n|✓|✓|43.5 ± 0.2|27.9 ± 0.3|\n\nRegarding the inference complexity, we show results in Table 4 of the supplementary material. For example, compared to 3DIoUMatch, our runtime speed can be decreased by 28.8% (from 65.54 FPS to 46.64 FPS) while the performance in mAP@0.5 relatively improves by 61.3% (from 8% to 12.9%) on SUN RGB-D (Ln 61-65 of the supplementary material).\n\n**Q3: Minor comments.**\n\nWe will revise the paper as suggested, including EMA in Figure 1 and notations in Section 3.1.\n\nFor the second paragraph of Section 4.1, we discuss the usage of applying existing augmentation methods. Due to limited space, we only present this discussion in the text and include Table 3 in the supplementary material. We will improve the presentation.\n\nFor baselines in Tables 3 and 4, we re-run 3DIoUMatch with comparable implementations and hardware on ScanNet. We obtain very similar results to the original ones shown in the table below.\n\n3DIoUMatch|5% mAP@0.25|5% mAP@0.5|\n|:---:|:---:|:---:|\nOriginal |40.0 ± 0.9|22.5 ± 0.5|\nRe-simulated |40.5 ± 1.2|22.8 ± 0.8|\n\n**Q4: Outdoor dataset.**\n\nDue to limited time to experiment with a new training framework (i.e., the outdoor dataset needs a different baseline than VoteNet used in the paper), we will include the results in the final version. Note that, by design of the proposed diffusion model in a general teacher-student framework, our method should not be limited to certain datasets. However, we do recognize that some changes may be needed as some object properties (e.g., location, density) in outdoor would be different from the indoor scenario. We will add this discussion to the final paper.\n\nIn addition, thanks for providing the references [a, b], and we will discuss them in the final paper. Note that, [a] operates a multi-view setting using stereo or video data, while [b] considers multi-modal data with both 2D and 3D labeled data, which are different from the setting of this work.\n\n**Q5: Pseudo-label quality.**\n\nTo validate whether the quality of pseudo-labels is improved, we evaluate the metrics on unlabeled training data during model training, via the teacher model that generates pseudo-labels. In the table below, overall the pseudo-label quality of our Diffusion-SS3D is better than 3DIoUMatch by more than 8% improvement in mAP and recall rate. In addition, we observe that our diffusion model achieves a better quality in earlier epochs, and then become stable during the entire training process. Note that, we still observe the improvement of SSL performance during training, since the model needs to be trained longer and learn from both labeled and unlabeled data. We will add them to the final version.\n\n| ScanNet 5% |Metric|Epoch 100|Epoch 200|Epoch 400|Epoch 800| Epoch 1000|\n|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| 3DIoUMatch | mAP@0.5 |14.08|17.85|21.73|22.17|22.42|\n|  | Recall@0.5 |27.86|31.49|35.24|36.61|35.25|\n| Diffusion-SS3D | mAP@0.5 | 29.98 | 30.09|30.86| 31.01 |30.93 |\n|  | Recall@0.5 | 43.73 | 44.14 |45.06|44.72|44.17|\n\n**Q6: Fully-supervised setting.**\n\nOur method is indeed not limited to only the SSL setting, and we can still train the model in fully-supervised setting without using our components relevant to unlabeled data. Specifically, we train the diffusion model with 100% labeled data using the same way as the student model does (top of Figure 2 in the main paper). During inference, random data distribution is then generated (Figure 3 of the main paper) and denoised via DDIM sampling to produce final predictions.\n\nWe show the result in the table below on ScanNet, where our method performs better than the baseline without diffusion by more than 1%. Although the fully-supervised setting is not our main focus, this demonstrates the potential of introducing the diffusion process in more settings.\n\n|Model|100% mAP@0.25|100% mAP@0.5|\n|:---:|:---:|:---:|\n|SESS|61.3|38.8|\n|3DIoUMatch|62.9|42.1|\n|Diffusion-SS3D|64.1|43.2|\n|Gain (mAP)|+1.2|+1.1|\n\n**Q7: Limitations.**\n\nWe will add more discussions as in Q2 and Q4."}]

Author rebuttal:
Thanks for your constructive feedback.

In addition to addressing individual questions in each rebuttal, we include more example results of generated pseudo-labels in the pdf file, in comparisons with the 3DIoUMatch baseline that does not use the diffusion model like our method.

If there are any further inquiries, please notify us by the end of Author-Reviewer Discussion Stage (Aug 16th).

Meta-review: All five reviewers recommend acceptance. The authors did a good job with their rebuttal, the reviewers concur that most concerns were addressed. For the benefit of the readers, please include the important points from the rebuttal in the final paper.
Decision: Accept (poster)