SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In our implementation, we follow DDIM [32] setting  $\sigma_t = 0$ . The resulting model becomes an implicit probabilistic model [67], where the generation process becomes deterministic given  $\mathbf{y}_T$ .
B t-SNE visualization of the DDIM generation process
Figure B.1: The t-SNE visualization of the CLIP feature space during the reverse generation process of a conditional diffusion model using an 7-step DDIM on the CIFAR-10 dataset. The process begins at time  $t = 7$ , with the sampling of the latent representation of the label from the latent distribution  $\mathcal{N}(f_q(\mathbf{x}), \mathbf{I})$ . Through a series of multi-step reverse operations, the latent distribution is transformed into the conditional distribution of labels. The data points are color-coded according to the entry with the highest value in intermediate/final label vectors, and the ground truth class labels are represented by distinct markers.
C Experimental setup and details
C.1 Real-world dataset details
WebVision comprising 2.4 million images that were crawled using Google and Flickr search engines, with the ILSVRC12 taxonomy. Following prior studies, we trained our model on the initial 50 classes from the Google image subset of Webvision and tested it on the validation sets of both Webvision and ILSVRC12.
Food-101N consists of 310k food images collected from the internet with the Food-101 [68] taxonomy, and has an estimated label noise level of  $20\%$ , making it an ideal dataset to evaluate the robustness of our method under real-world noisy labels. We assessed the classification accuracy on the curated label set of Food-101, which contains around 25k images.
Clothing1M contains 1 million images of clothes obtained from shopping websites. Based on the keywords in the surrounding text, the images are automatically classified into 14 classes with  $\sim 40\%$  estimated noise level. The dataset includes a clean training set, validation set, and test set with manually refined labels, consisting of approximately  $47.6\mathrm{k}$ ,  $14.3\mathrm{k}$ , and  $10\mathrm{k}$  pictures, respectively. We discarded the clean training set and only used the noisy label data for training.
To present the hyperparameter settings of our neural network, we first give a description of our neural network design. As shown in Figure C.1, the network consists of a frozen  $f_{p}$  encoder, a ResNet encoder, and a series of feed forward layers. Features encoded by the two encoders are combined with time embedding via hadamard product and passed through a series of feed-forward networks, batch normalization, and softmax activation to predict the noise term  $\epsilon_{\theta}$ .
In our experiments, we use ResNet34 for CIFAR10 and CIFAR100, and use ResNet50 for real-world datasets as the trainable encoder (blue ResNet block in Figure C.1). The dimensions of all feed-forward layers are set to 512 for CIFAR datasets and 1024 for real-world datasets, respectively. We train LRA-diffusion models for 200
Figure C.1: The network architecture for conditional diffusion models. The input to the network consists of four elements:  $\mathbf{y}_t$ ,  $f_p(\mathbf{x})$ ,  $\mathbf{x}$ , and the time embedding for  $t$ , represented by pink blocks. The blue blocks in the figure represent the trainable network components.
epochs with Adam optimizer. The batch size is 256. We used a learning rate schedule that included a warmup phase followed by a half-cycle cosine decay. The initial learning rate is set to 0.001. Following [15], we applied data augmentation in the training, including resizing, random horizontal flip, and random cropping. To retrieve the nearest neighbors, we set  $k = 10$  based on our tests using a range of  $k$  values from 1 to 100 on the validation sets. The KNN accuracy remained relatively stable for  $k$  between 10 and 50, and then starts to decline due to reduced label consistency among neighbors. Based on these results, we infer that our LRA diffusion model is less sensitive to variations in  $k$  within this range. All experiments are conducted using four NVIDIA Titan V GPUs.
D Additional ablation study
D.1 Classifier feature conditioning for accuracy enhancement
To demonstrate how our method can enhance a trained classifier's performance by using its features as conditional information, we conduct an ablation study examining the impact of conditional diffusion and KNN on the trained classifier. Specifically, we train classifiers, denoted as  $\eta (\mathbf{x})$ , using the standard method at various noise levels. We then remove the classification head and utilize the remaining model  $f_{\eta}$  as the  $f_{p}$  encoders in our conditional diffusion models.
The experimental results shown in Figure D.1 indicate that these techniques can improve test accuracy. We observe that when the noise level is below  $55\%$ , the conditional diffusion model (green) achieves a  $\sim 1\%$  improvement over the standard method. Moreover, when the LRA method is applied concurrently (purple), test accuracy can be further enhanced. This improvement occurs because learning from neighbors' labels reduces the noise level during training, as evidenced by the comparison between KNN results (blue) and clean label percentage (gray).
However, when the noise level exceeds  $55\%$ , the use of diffusion and LRA-diffusion methods does not seem advantageous. This limitation arises because the distribution of labels in the neighborhood becomes too corrupted for KNN to effectively improve the proportion of clean labels during training, as illustrated by the intersection of the blue and gray curves in the figure. We argue this does not diminish the practical value of our method because a dataset with more than  $50\%$  label noise is not meaningful in practice.
D.2 Effects of pseudo-label construction strategies
We also conduct comparative experiments using another method that utilizes neighbor labels: replacing the one-hot label vector with the mean vector of the neighbor's labels as the prediction target, which we call Mean-diffusion. We found that it can achieve higher accuracy when the noise level is higher than  $55\%$ . This may be due to the increase in the diversity of neighbor labels. The sampling-based LRA-diffusion will need
Figure D.1: Test accuracy of seven methods on the CIFAR-10 dataset with different levels of PMD label noise. Other than the already introduced method names, here Diffusion is a conditional diffusion model using the feature  $f_{\eta}$ . The clean label percentage is represented by the gray line.
to learn a more complex multi-modal distribution, but Mean-diffusion only needs to learn a point estimate. However, when the noise level is lower than  $55\%$ , we found that LRA-diffusion is slightly more accurate than Mean-diffusion. A possible explanation is that the distribution of  $\mathbf{y}_0$  in LRA-diffusion contains only n one-hot labels. In contrast,  $\mathbf{y}_0$  in Mean-diffusion is more diverse ( $n^k / k!$  possible mean vectors for  $n$  classes and  $k$  neighbors). In conclusion, LRA-diffusion has higher performance with less noisy labels. On the other hand, Mean-diffusion has faster and more stable convergence and is more robust for high noise level. However, they tend to perform similarly when the noise level is too high or too low since neighbors' labels will become the same or too corrupted.
D.3 Robustness of SimCLR feature conditioning
Finally, we use the SimCLR model as the encoder  $f_{p}$  in our conditional diffusion model (listed as SimCLR LRA-diffusion in Figure D.1), to showcase the effectiveness of our proposed LRA-diffusion method in utilizing prior knowledge from pre-trained image representations to enhance the test accuracy and robustness. The experimental results (red) show that its test accuracy significantly surpasses other settings until the noise level reaches  $65\%$ . Beyond this point, the labels in the neighborhood become too corrupted to provide additional supervision information.
D.4 Effects of CLIP feature conditioning strategies
Table D.1: Classification accuracy (\%) of linear probing, KNN, and diffusion model using pre-trained CLIP feature on real-world label noise datasets.
Method	Webvision	ILSVRC2012	Food-101N	Clothing1M
KNN	81.88%	82.12%	91.73%	65.70%
linear prob + Mean label	68.56%	68.76%	90.68%	65.75%
linear prob + sample label	54.84%	56.80%	89.56%	55.52%
diffusion (Mean label)	83.96%	82.24%	93.13%	71.79%
diffusion (sample label)	84.16%	82.56%	93.42%	71.65%
We carried out an ablation study employing various strategies to integrate the CLIP model for classification on real-world datasets. The results demonstrate that the superior performance of our diffusion-based method does not simply reflect the strength of the CLIP features. We show results of linear probing and KNN using CLIP features in Table D.1 to further demonstrate the effectiveness of our algorithm design. The results suggested that
utilizing linear probing with LRA and mean labels degrades the pre-trained feature space, leading to diminished performance in comparison to the unsupervised KNN approach. On the other hand, the diffusion process can effectively incorporate the CLIP feature space to achieve higher performance.