SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Implementation details. Consistency models are trained by consistency distillation in CIFAR-10/CIFAR-100 datasets and consistency training in the SVHN/ColoredMNIST datasets. We set the hyperparameters consistent with the author's choice [21] on CIFAR-10. We implement the code on the PyTorch [47] framework. All results are averaged across 3 random seeds. We refer to the Appendix for further details.
2We use the author's code in https://github.com/ahsanMah/msma.
Table 3: One-class classification results (AUROC) of Projection Regret (PR) against various baselines in the CIFAR-10 one-class classification task. Bold denotes the best method.
Method	Category	Plane	Car	Bird	Cat	Deer	Dog	Frog	Horse	Ship	Truck	Mean
MemAE [43]	Autoencoder	-	-	-	-	-	-	-	-	-	-	0.609
AnoGAN [44]	GAN	0.671	0.547	0.529	0.545	0.651	0.603	0.585	0.625	0.758	0.665	0.618
OCGAN [45]	GAN	0.757	0.531	0.640	0.620	0.723	0.620	0.723	0.575	0.820	0.554	0.657
DROCC [46]	Adv. Generation	0.817	0.767	0.667	0.671	0.736	0.744	0.744	0.714	0.800	0.762	0.742
P-KDGAN [49]	GAN	0.825	0.744	0.703	0.605	0.765	0.652	0.797	0.723	0.827	0.735	0.738
LMD [18]	Diffusion Model	0.770	0.801	0.719	0.506	0.790	0.625	0.793	0.766	0.740	0.760	0.727
Rot+Trans [50]	Self-supervised	0.760	0.961	0.867	0.772	0.911	0.885	0.884	0.961	0.943	0.918	0.886
Projection Regret (ours)	Diffusion Model	0.864	0.946	0.789	0.697	0.840	0.825	0.854	0.889	0.929	0.919	0.855
PR + (Rot+Trans) (ours)	Hybrid	0.829	0.971	0.889	0.803	0.925	0.905	0.911	0.963	0.954	0.933	0.908
Figure 4: Hyperparameter analysis of Projection Regret on CIFAR-10 vs CIFAR-100 task. (left): Ablation on hyperparameter  $\alpha$  and  $\beta$ . The proposed time ensemble performs better than the best hyperparameter. (middle): Ablation on hyperparameter  $n_{\alpha}$ . (right): Ablation on hyperparameter  $n_{\beta}$ .
Table 1 shows the OOD-detection performance against diffusion-model-based methods. Projection Regret consistently outperforms MSMA and LMD on 8 out of 9 benchmarks. While Projection Regret mainly focuses on OODs with similar backgrounds, Projection Regret also outperforms baseline methods on the task where they show reasonable performance (e.g., CIFAR-10 vs SVHN).
In contrast, MSMA struggles on various benchmarks where the ID and the OOD share similar backgrounds (e.g., CIFAR-10 vs CIFAR-100). While LMD performs better in such benchmarks, LMD shows underwhelming performance in SVHN vs (CIFAR-10 or CIFAR-100) and even underperforms over input likelihood [6]. To analyze this phenomenon, we perform a visualization of the SVHN samples that show the highest abnormality score and their inpainting reconstructions in the Appendix. The masking strategy of LMD sometimes discards semantics and thereby reconstructs a different image. In contrast, Projection Regret shows near-perfect OOD detection performance in these tasks.
We further compare Projection Regret against OOD detection methods based on alternative generative models (e.g., EBM [24], GLOW [42]) in Table 2. We also denote the category of each baseline method. Among the various generative-model-based OOD detection methods, Projection Regret shows the best performance on average. Significantly, while CLEL [24] requires an extra self-supervised model for joint modeling, Projection Regret outperforms CLEL in 3 out of 4 tasks.
We finally compare Projection Regret against various one-class classification methods in Table 3. Projection Regret achieves the best performance and outperforms other reconstruction-based methods consistently. Furthermore, when combined with the contrastive learning method [50] by simply multiplying the anomaly score, our approach consistently improves the performance of the contrastive training-based learning method. Specifically, Projection Regret outperforms the contrastive training-based method in airplane class where the data contains images captured in varying degrees. As such, contrastive training-based methods underperform since they use rotation-based augmentation to generate pseudo-outlier data.
Hyperparameter analysis. We analyze the effect of hyperparameters for Projection Regret under the CIFAR-10 vs CIFAR-100 task. First, we conduct experiments with varying the timestep index
Table 4: Ablation Study on each component of Projection Regret. In-distribution dataset is the CIFAR-10 dataset. we report AUROC. Gains are computed against LMD [18].
Method	SVHN	CIFAR100	LSUN	ImageNet
LMD [18]	0.979	0.620	0.734	0.686
Projection (i.e., \( S_{\Pi}^{LPIPS} \))	0.966 (-1.3%)	0.667 (+4.7%)	0.762 (+2.8%)	0.718 (+3.2%)
Projection Regret (\( \alpha = 9, \beta = 8 \))	0.986 (+0.7%)	0.760 (+14.0%)	0.828 (+9.4%)	0.795 (+10.9%)
Projection Regret (Ensemble)	0.993 (+1.4%)	0.775 (+15.5%)	0.837 (+10.3%)	0.814 (+12.8%)
Table 5: AUROC of Projection Regret given different distance metric choices. Bold and underline denotes the best and second best methods.
Method	Distance d	SVHN	CIFAR100	LSUN	ImageNet
LMD [18]	0.979	0.620	0.734	0.686
Projection Regret	SSIM [27]	0.328	0.629	0.650	0.620
LPIPS [20]	0.993	0.775	0.837	0.814
UNet (proposed)	0.917	0.734	0.865	0.815
hyperparameters  $\alpha$  and  $\beta$ . As mentioned in Section 3.2, the ensemble performs better (i.e., 0.775 in Table 1) than the best hyperparameter selected (i.e., 0.760 when  $\alpha = 9$ ,  $\beta = 8$ ).
Furthermore, we perform ablation on the ensemble size hyperparameter  $n_{\alpha}$  and  $n_{\beta}$ . While selecting a larger ensemble size improves the performance, we show reasonable performance can be achieved in smaller  $n_{\alpha}$  or  $n_{\beta}$  (e.g.,  $n_{\alpha} = 40$ ,  $n_{\beta} = 5$  or  $n_{\alpha} = 20$ ,  $n_{\beta} = 10$ ) without much sacrifice on the performance.
Ablation on the components of Projection Regret. We further ablate the contribution of each component in Projection Regret under the CIFAR-10 dataset. First, we set the best-performing timestep index hyperparameter  $(\alpha = 9, \beta = 8)$  under the CIFAR-10 vs CIFAR-100 detection task. Furthermore, we also compare the projection baseline method, i.e.,  $S_{\Pi}^{\mathrm{LPIPS}}(\cdot, t_{\beta}; \pmb{\theta})$  where  $\beta = 8$ .
The performance of each method is shown in Table 4. We also denote the absolute improvements over LMD [18]. Our proposed baseline method  $S_{\Pi}^{\mathrm{LPIPS}}$  already improves over LMD in 3 out of 4 tasks, especially in the CIFAR-10 OOD dataset with similar background statistics. Furthermore, we observe consistent gains on both Projection Regret with and without ensemble over baselines. The largest gain is observed in Projection Regret over the projection baseline  $S_{\Pi}^{\mathrm{LPIPS}}$  (e.g.,  $9.3\%$  in the CIFAR-10 vs CIFAR-100 detection task).
Diffusion-model-based distance metric. While we mainly use LPIPS [20] as the distance metric  $d$ , we here explore alternative choices on the distance metric.
We first explore the metric that can be extracted directly from the pre-trained diffusion model. To this end, we consider intermediate feature maps of the U-Net [51] decoder of the consistency model  $f_{\theta}$  since the feature maps may contain semantic information to reconstruct the original image [52]. Formally, let  $F^{(\ell)}(\mathbf{x},t,\mathbf{z})\in \mathbb{R}^{C_\ell\times H_\ell\times W_\ell}$  be the  $\ell$ -th decoder feature map of  $f_{\theta}(\mathbf{x} + t\mathbf{z},t)$ . Motivated by semantic segmentation literature [53, 54] that utilize the cosine distance between two feature maps, we define our U-Net-based distance metric  $d_{\mathrm{UNet}}(\cdot ,\cdot)$  as follows:
$$
d _ {\mathrm {U N e t}} \left(\mathbf {x}, \mathbf {x} ^ {\prime}\right) = \mathbb {E} _ {\mathbf {z} \sim \mathcal {N} \left(\mathbf {0}, \mathbf {I}\right)} \left[ \sum_ {\ell = 1} ^ {L} \frac {1}{H _ {\ell} W _ {\ell}} \sum_ {i j} d _ {\cos} \left(F _ {:, i, j} ^ {(\ell)} \left(\mathbf {x}, t _ {\gamma}, \mathbf {z}\right), F _ {:, i, j} ^ {(\ell)} \left(\mathbf {x} ^ {\prime}, t _ {\gamma}, \mathbf {z}\right)\right) \right], \tag {5}
$$
where  $d_{\cos}(\mathbf{u},\mathbf{v}) = \left\| \mathbf{u} / \| \mathbf{u}\| _2 - \mathbf{v} / \| \mathbf{v}\| _2\right\| _2^2$  is the cosine distance and  $t_\gamma$  is a timestep hyperparameter. We set  $t_\gamma$  small enough to reconstruct the original image, e.g.,  $t_\gamma = 0.06$  (i.e.,  $\gamma = 3$ ), for evaluation.
We report the result in Table 5. We also test the negative SSIM index [27] as the baseline distance metric. While LPIPS performs the best, our proposed distance  $d_{\mathrm{UNet}}$  shows competitive performance against LPIPS and outperforms LMD and MSMA in CIFAR-10 vs (CIFAR-100, LSUN or ImageNet) tasks. On the other hand, SSIM harshly underperforms LPIPS and  $d_{\mathrm{UNet}}$ .
Using the diffusion model for full-step projection. While the consistency model is used for full-step projection throughout our experiments, a diffusion model can also generate full-step projection  $\Pi_{\theta}^{t}(\mathbf{x})$
Table 6: Performance (AUROC) of Projection Regret computed in the EDM [29] with Heun's 2nd order sampler. Performance of Projection Regret under consistency model [21] under the same hyperparameter is included for comparison.
Model	SVHN	CIFAR100	LSUN	ImageNet
nα=10, nβ=5
EDM [29]	0.986	0.763	0.825	0.799
CM [21]	0.990	0.761	0.816	0.797
nα=40, nβ=10
CM [21]	0.993	0.775	0.837	0.814
Table 7: OOD detection results (TNR at  $95\%$  TPR, AUROC) under more severe background bias. In-distribution dataset is the colored MNIST dataset in Ming et al. [55] with  $r = 0.45$ . The result in Ming et al. [55] is included for reference. Higher is better in all metrics.
Method	Spurious OOD dataset	LSUN	iSUN	Textures
Ming et al. [55]	0.696/0.867	0.901/0.959	0.933/0.982	0.937/0.985
Projection Regret	0.958/0.990	1.000/1.000	1.000/1.000	1.000/1.000
with better generation quality through multi-step sampling. For verification, we experiment with the EDM [29] trained in the CIFAR-10 dataset and perform full-step projection via the 2nd-order Heun's sampler. Since sampling by diffusion model increases the computation cost by  $2\beta$  times compared to the consistency model, we instead experiment on the smaller number of ensemble size  $n_{\alpha} = 10$ ,  $n_{\beta} = 5$  and report the result of Projection Regret with a consistency model computed in the same hyperparameter setup as a baseline.
We report the result in Table 6. We denote the result of Projection Regret utilizing the full-step projection based on the diffusion model and the consistency model as EDM and CM, respectively. Despite its excessive computation cost, Projection Regret on EDM shows negligible performance gains compared to that on CM.
Projection Regret helps robustness against spurious correlations. While we mainly discuss the problem of generative model overfitting into background information in the unsupervised domain, such a phenomenon is also observed in supervised learning where the classifier overfits the background or texture information [56]. Motivated by the results, we further evaluate our results against the spurious OODs where the OOD data share a similar background to the ID data but with different semantics. Specifically, we experiment on the coloredMNIST [40] dataset following the protocol of [55]. We report the result in Table 7. Note that spurious OOD data have the same background statistics and different semantic (digit) information. Projection Regret achieves near-perfect OOD detection performance. It is worth noting that Projection Regret does not use any additional information while the baseline proposed by [55] uses label information.
Fast sampling of diffusion models. While we use full-step projection of the image for Projection Regret, this induces excessive sampling complexity in standard diffusion models [26, 57, 28]. Hence, we discuss works that efficiently reduce the sampling complexity of the diffusion models. PNDM [58] and GENIE [59] utilize multi-step solvers to reduce the approximation error when sampling with low NFEs. Salimans and Ho [60] iteratively distill the 2-step DDIM [61] sampler of a teacher network into a student network.
Out-of-distribution detection on generative models. Generative models have been widely applied to out-of-distribution detection since they implicitly model the in-distribution data distribution. A straightforward application is directly using the likelihood of the explicit likelihood model (e.g., VAE [41]) as a normality score function [6]. However, Nalisnick et al. [33] discover that such models may assign a higher likelihood to OOD data. A plethora of research has been devoted to explaining the aforementioned phenomenon [62, 7, 63, 64]. Serrà et al. [7] hypothesize that deep generative models assign higher likelihood on the simple out-of-distribution dataset. Kirichenko et al. [63] propose that flow-based models' [42] likelihood is dominated by local pixel correlations.
Finally, diffusion models have been applied for OOD detection due to their superior generation capability. Mahmood et al. [17] estimate the  $\ell_2$  distance between the input image and its single-step projection as the abnormality score function. Liu et al. [18] applies checkerboard masking to the input image and performs a diffusion-model-based inpainting task. Then, reconstruction loss between the inpainted image and the original image is set as the abnormality score function.
Reconstruction-based out-of-distribution detection methods. Instead of directly applying pretrained generative models for novelty detection, several methods[43, 46, 45] train a reconstruction model for out-of-distribution. MemAE [43] utilizes a memory structure that stores normal representation. DROCC [46] interprets one-class classification as binary classification and generates boundary data via adversarial generation. Similarly, OCGAN [45] trains an auxiliary classifier that controls the generative latent space of the GAN model. While these methods are mainly proposed for one-class classification, we are able to outperform them with significant gains.
Debiasing classifiers against spurious correlations. Our objective to capture semantic information for novelty detection aligns with lines of research that aim to debias classifiers against spurious correlations [40, 56, 65, 66, 67]. Geirhos et al. [56] first observe that CNN-trained classifiers show vulnerability under the shift of non-semantic regions. Izmailov et al. [66] tackles the issue by group robustness methods. Motivated by research on interpretability, Wu et al. [67] apply Grad-CAM [68] to discover semantic features. Ming et al. [55] extend such a line of research to out-of-distribution detection by testing OOD detection algorithms on spurious anomalies.
In this paper, we propose Projection Regret, a novel and effective out-of-distribution (OOD) detection method based on the diffusion-based projection that transforms any input image into an in-distribution image with similar background statistics. We also utilize recursive projections to reduce background bias for improving OOD detection. Extensive experiments demonstrate the efficacy of Projection Regret on various benchmarks against the state-of-the-art generative-model-based methods. We hope that our work will guide new interesting research directions in developing diffusion-based methods for not only OOD detection but also other downstream tasks such as image-to-image translation [69].
**Limitation.** Throughout our experiments, we mainly use the consistency models to compute the full-step projection via one network evaluation. This efficiency is a double-edged sword, e.g., consistency models often generate low-quality images compared to diffusion models. This might degrade the detection accuracy of our Projection Regret, especially when the training distribution  $p_{\mathrm{data}}$  is difficult to learn, e.g., high-resolution images. However, as we observed in Table 6, we strongly believe that our approach can provide a strong signal to detect out-of-distribution samples even if the consistency model is imperfect. In addition, we also think there is room for improving the detection performance by utilizing internal behaviors of the model like our distance metric  $d_{\mathrm{UNet}}$ . Furthermore, since diffusion models use computation-heavy U-Net architecture, our computational speed is much slower than other reconstruction-based methods.
Negative social impact. While near-perfect novelty detection algorithms are crucial for real-life applications, they can be misused against societal agreement. For example, an unauthorized data scraping model may utilize the novelty detection algorithm to effectively collect the widespread data. Furthermore, the novelty detection method can be misused to keep surveillance on minority groups.
This work is fully supported by LG AI Research.
[59] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. In NeurIPS, 2022.
[60] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
[61] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[62] Sungik Choi and Sae-Young Chung. Novelty detection via blurring. In ICLR, 2020.
[63] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail to detect out-of-distribution data. In NeurIPS, 2020.
[64] Lily H. Zhang, Mark Goldstein, and Rajesh Ranganath. Understanding failures in out-of-distribution detection with deep generative models. In ICML, 2021.
[65] Junhyun Nam, Hyuntak Cha, Sungsoo Ahn, Jaeho Lee, and Jinwoo Shin. Learning from failure: training debiased classifier from biased classifier. In NeurIPS, 2020.
[66] Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson. On feature learning in the presence of spurious correlations. In NeurIPS, 2022.
[67] Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: concept-aware mitigation of spurious correlations. In ICML, 2023.
[68] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
[69] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. In ICLR, 2023.
[70] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. In NeurIPS, 2017.
A Full algorithm
We refer to Algorithm 1 for the efficient computation of our method.
Algorithm 1 Projection Regret with Consistency Models (PyTorch-like Pseudo-code)
f(x, t): consistency model
# t_alpha, t_beta: the projection timesteps
# n_alpha, n_beta: the projection ensemble sizes
# repeat(A, n): repeat tensor A n times along axis=0
# d(a, b): distance function, e.g., LPIPS
def Projection(x, t):
    return f(x + t * randn_like(x), t)
def ProjectionRegret(x): # x: [1, 3, H, W]
    xProj = Projection(repeat(x, n_alpha * n_beta), t_beta)
    y = Projection(repeat(x, n_alpha), t_alpha)
    yProj = Projection(repeat(y, n_beta), t_beta)
    dx = d(repeat(x, n_alpha * n_beta), x_proj).mean()
    dy = d(repeat(y, n_beta), y_proj).mean()
    return dx - dy
We train the baseline EDM [29] model for 390k iterations for all datasets. Then, we perform consistency distillation [21] for the CIFAR-10 and CIFAR-100 [30] datasets and consistency training for the SVHN [31] dataset. The resulting FID score [70] of the CIFAR-10/CIFAR-100 dataset is 5.52/7.0 and 8.0 for the SVHN dataset (the model trained by consistency training usually shows higher FID than the one trained by consistency distillation). While training the base EDM diffusion model and the consistency model, we use 8 A100 GPUs. During the out-of-distribution detection, we use 1 A100 GPU. For the ensemble size hyperparameter  $n_{\alpha}$  and  $n_{\beta}$ , we set  $n_{\alpha} = 40$ ,  $n_{\beta} = 10$  for CIFAR-10/SVHN experiments and  $n_{\alpha} = 100$ ,  $n_{\beta} = 5$  for CIFAR-100 experiment. As mentioned in the main paper, we set our base hyperparameter that shows the best performance against the rotated in-distribution dataset and set the ensemble configurations around it. We use 8 hyperparameter configurations for the CIFAR-10 and CIFAR-100 datasets and 4 hyperparameter configurations for the SVHN dataset. To be specific, chosen configurations are  $\mathcal{C} = \{(7,6), (7,7), (8,7), (8,8), (9,8), (9,9), (10,9), (10,10)\}$  for CIFAR-10/CIFAR-100 dataset and  $\mathcal{C} = \{(10,10), (11,10), (11,11), (12,11)\}$  for the SVHN dataset.
C Visualization of LMD in SVHN dataset
We further reconstruct the output of the LMD [18] when SVHN is the in-distribution dataset. Following the practice of Liu et al. [18], we set the mask as the alternating checkboard mask throughout the experiments in Section 4.2. We visualize a grid of 9 samples and their reconstructions where LMD shows the highest abnormality score in Figure 5. We can see LMD's inconsistent reconstruction in such data, thereby failing to reconstruct consistent images. Furthermore, as mentioned in Section 4.2, sometimes the checkboard strategy discards the in-distribution data's semantics and thereby fails to reconstruct the original image.
(a) Original images
(b) Masked images
(c) Reconstructed images
Figure 5: Visualization of LMD [18] in the SVHN [31] dataset. (a) Original SVHN images where LMD outputs the highest abnormality score. (b) The masked images following the practice of Liu et al. [18]. (c) Their reconstruction images through LMD.