Projection Regret: Reducing Background Bias for Novelty Detection via Diffusion Models
Authors: Sungik Choi
Hankook Lee
Honglak Lee
Moontae Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: Novelty detection
out-of-distribution detection
consistency models
diffusion models
score-based generative models
TL;DR: We propose an efficient out-of-distribution detection model based on pre-trained diffusion model and the LPIPS metric

Abstract: Novelty detection is a fundamental task of machine learning which aims to detect abnormal (*i.e.* out-of-distribution (OOD)) samples. Since diffusion models have recently emerged as the de facto standard generative framework with surprising generation results, novelty detection via diffusion models has also gained much attention. Recent methods have mainly utilized the reconstruction property of in-distribution samples. However, they often suffer from detecting OOD samples that share similar background information to the in-distribution data. Based on our observation that diffusion models can *project* any sample to an in-distribution sample with similar background information, we propose *Projection Regret (PR)*, an efficient novelty detection method that mitigates the bias of non-semantic information. To be specific, PR computes the perceptual distance between the test image and its diffusion-based projection to detect abnormality. Since the perceptual distance often fails to capture semantic changes when the background information is dominant, we cancel out the background bias by comparing it against recursive projections. Extensive experiments demonstrate that PR outperforms the prior art of generative-model-based novelty detection methods by a significant margin.

Projection Regret: Reducing Background Bias for Novelty Detection via Diffusion Models
Sungik Choi Hankook Lee'
Honglak Leel
Moontae Lee
LL Al Research
University of Illinois Chicago
(sungik. choi, hankook. lee, honglak. lee,, moontae 01gesearees.
Abstract
Novelty detection is fundamental task of machine learning which aims to detect abnormal (i.e. out-of-distribution (OOD)) samples. Since diffusion models have recently emerged as the de facto standard generative framework with surprising generation results, novelty detection via diffusion models has also gained much attention. Recent methods have mainly utilized the reconstruction property of in-distribution samples. However, they often suffer from detecting OOD samples that share similar background information to the in-distribution data. Based on our observation that diffusion models can project any sample to an in-distribution sample with similar background information, we propose Projection Regret (PR), efficient novelty detection method that mitigates the bias of semanica information. To be specific, PR computes the perceptual distance between the test image and its diffusion- based projection to detect abnormality. Since the perceptual distance often fails to capture semantic changes when the background information is dominant, cancel out the background bias comparing against recursive projections. Extensive experiments demonstrate that PR outperforms the prior art of generraive-moddl-bassee novelty detection methods by significant margin.
Introduction
Novelty detection [1], also known out-of-distribution (OOD) detection, is fundamental machine learning problem, which aims to detect abnormal samples drawn from far from the training distribution (i.e., in-distribution). This plays vital role in many deep learning applications because the behavior of deep neural networks on OOD samples is often unpredictable and can lead to erroneous decisions [2]. Hence, the detection ability is crucial for ensuring reliability and safety in practical applications, including medical diagnosis [3], autonomous driving [4), and forecasting [5].
The principled way identify whether test sample is drawn from the training distribution Paata (x) or not is to utilize an explicit or implicit generative model. For example, one may utilize the likelihood function directly [6] or its variants 17, 8] as an OOD detector. Another direction is to utilize the generation ability, for example, reconstruction loss [9] or gradient representations of the reconstruction loss [10]. It is worth noting that the research direction of utilizing generative models for diitt--iisttiuttio detection has become increasingly important in recent years as generative models have been successful across various domains, e.g., vision [11, 1, 12] and language [13], but also they have raised various social issues including deepfake [14] and hallucination [15].
Among various generative frameworks, diffusion models have recently emerged as the most popular framework due to their strong generation performance, e.g. high-resolution images [16], and its wide applicability. e.g., text-to-image synthesis [11]. Hence, they have also gained much attention as attractive tool for novelty detection. For example, Mahmood et al. [17] and Liu et al. [18] utilize the reconstruction error the OOD detection metric using diffusion models with Gaussian noises [17] or checkerboard masking [18]. Somewhat unexpectedly, despite the high-quality generation results,
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Canehaann Melel
In-diserbution (Church)
+A01o, (2.0
0.381
0.278
0.387
x 116 (x)
11. (x')
Spa 0.381
0.387 = -0.006 Indistribution
Sm 0.278 0.244 0.034
Out-oof-isttibuution (Brigge
Out-of-distibution
(a) Projection ITg
(b) Projection Regret Sax
Figure 1: Conceptual illustration of framework. (a) Given image Xp. the projection of the image, IT, (xp). is defined by diffusing Xy until an intermediate timestep and then reversing the process using the consistency model fe. The projection 1I, can map an out-of-distribution image to an in-distribution one with similar backgrounds when is neither too early nor too late. (b) Our proposed method, Projection Regret, computes the abnormality score Spp. by capturing the semantic difference between a test image and its projection. We further utilize the two-step projection to cancel out the background bias, especially for the case when background information is dominant.
their detection performance still lags behind other generative models, e.g., energy-based models [19]. We find that these undesired results are related to the background bias problem: the reconstruction error is often biased by non-semantic (i.e., background) information. Namely, the prior methods often struggle to detect against CIFAR-10 10 ones).
images with similar backgrounds (e.g., CIFAR-100 images
Contribution. To reduce the background bias, we first systemically examine the behavior of diffusion models. We observe that the models can change semantic information to in-distribution without a major change to background statistics by starting from an intermediate point of the diffusion process and reversing the process, which referred to as projection in this paper (see Figure la). Based on this observation, suggest using the perceptual distance [20] between test image and its projection to detect whether the test image is -ul-of-distribution..
Although our projection slightly changes the background statistics, the perceptual distance sometimes fails to capture semantic differences when the background information dominant over the semantics. To ve this issue, we propose Projection Regret, which further reduces the background bias by canceling out the background information using recursive projections (see Figure lb). We also utilize an ensemble of multiple projections for improving detection performance. This can be calculated efficiently via the consistency model [21], which supports the reverse diffusion process via only one network evaluation.
Through extensive experiments, we demonstrate the effectiveness of Projection Regret (PR) under various out-of oot-of-dd of-distribution benchmarks. More specifically, PR outperforms the prior art of diffusion- based OOD detection methods with large margin, e.g., 0.620 - 0.775 detection accuracy (AUROC) against a recent method, LMD [18]. under the CIFAR- 10 CIFAR- 100 detection task (see Table 1). Furthermore, PR shows superiority over other types of generative models (see Table 2), including EBMs [22, 23, 24] and VAFS [25). We also propose perceptual distance metric using underlying features (i.e., feature maps in the U-Net architecture [26)) of the diffusion model. We demonstrate that Projection Regret with this proposed metric shows comparable results to that of LPIPS [20] and outperforms other distance metrics, e.g., SSIM [27] (see Table 5).
To sum up, our contributions are summarized as follows:
We find that the perceptual distance between the original image and its projection is competitive abnormality score function for OOD detection (Section 3.1).
We also propose Projection Regret (PR) that reduces the bias of non-semantic information the score (Section 3.2) and show its efficacy across challenging OOD detection tasks (Section 4.2).
We also propose an alternative perceptual distance metric that is computed by the decoder features of the pre-trained diffusion model (Section 4.3).

---

2 Preliminaries
2.1 Problem setup
Given a training distribution Pdata (x) on the data space X, the goal of out-of-distribution (OOD) detection is to design abnormality score function S(x) E R which identifies whether is drawn from the training distribution (i.e., S(x) & T) or not (i.e, S(x) > T) where T is a hyperparameter. For notation simplicity, "A B" denotes an OOD detection task where A and B are in-distribution (ID) and out-of di-tiiibibtttoonn (OOD) datasets, respectively. For evaluation, we use the standard metric, area under the receiver operating characteristic curve (AUROC).
In this paper, we consider a practical scenario in a generative model po(x) pre-trained on Past (x) is only available for OOD detection. Namely, we do not consider (a) training detector with extra information (e.g., label information and outlier examples) nor (b) accessing internal training data at the test time. As many generative models have been successfully developed across various domains and datasets in recent years, our setup is also widely applicable to real-world applications as well.
2.2 Diffusion and consistency models
Diffusion models (DMs). A diffusion model is defined over a diffusion process starting from the data distribution X₂ Paum (x) with stochastic differential equation (SDE) and its reverse-time process can be formulated as probability flow ordinary differential equation (ODE) [28, 29]:
Forward SDE:
dx; pxx,, t)dt + (t))d,,,
Reverse ODE:
dx; [u(x,, t) (t) log log.:.xx) dt,
where f E [0, T]. T > 0 is a fixed constant, (4(:;)) and (-) are the drift and diffusion coefficients, respectively, and w, is the standard Wiener process. The variaine-exploding (VE) diffusion model
learns a time-dependent denoising function Dx(,,, t) by minimizing the following denoising score matching objective [29]:
Lex := (er-
PX + 022,1) x12]
(1)
where d, depends on (t). Following the practice of Karras al. [29]. set o, t and discretize the timestep into = to < t1 < < ty = T for generation based on the ODE solver.
Consistency models (CMs). A few months ago, Song et al. [21] proposed consistency models, new type of generative model, built on top of the consistency property of the probability flow (PF) ODE trajectories: points (x,, E) on the same PF ODE trajectory map to the same initial point x,, ie., the model aims to learn a consistency function fe (xx t) = X,, This function fe can be trained by trajectory-wise distillation. The trajectory can be defined by the ODE solver of a diffusion model (i.e., consistency distillation) or a line (i.e., consistency training). For example, the training objective for the latter case can be formulated as follows:
Lox == EE- 1o [d(fo (x 0z, (7,z,44). t4), fo(x or., z, ((+1))]
(2)
where is the distance metric, e.g., &g or LPIPS [20]. and 0 is the target parameter obtained by exponential moving average (EMA) of the online parameter 0. The main advantage of CMs over DMs is that CMs support one-shot generation. In this paper, mainly CMs due their efficiency, but our method is working with both diffusion and consistency models (see Table 6).
3 Method
In this section, we introduce Projection Regret (PR), novel and effective OOD detection framework based on diffusion model. Specifically, we utilize diffusion- based projection that maps an OOD image to an in-distribution one with similar background information (Section 3.1) and design an abnormality score function Spg that reduces background bias using recursive projections (Section 3.2).
3.1 Projection
Although previous OOD detection methods based on generative models have been successfully applied to several benchmarks [8, 17], e.g., CIFAR-10 SVHN they still struggle to detect out-of- distribution samples of similar backgrounds to the in-distribution data (e.g., CIFAR-10 vs CIFAR-100). We here examine this challenging scenario with diffusion models.
3

---

CIFAR100
(Out-o--iitriuttion)
SVHN (Out-d-distriutioon
8
CIFAR10 (In-distribution
Original images
Full-step Projection [x)
Single-step Projection T (x)
Figure 2: Projection of the OOD and the in-distribution image. We show full-step projection and
single-step projection results for in-distribution (CIFAR-IO) and OOD (CIFAR-100, SVHN) images.
CIFAR-10 CIPAR-100
CIFAR-10 SVHN
S-PUS
0.65
SCPIPS
0.004
0.00
0.8
55
0.20
0.4
0.9
0.0
0.0
10
Figure 3: AUROC of the OOD detection task under different design choices and time-steps. (Left): CIFAR- 10 vs CIFAR-100 OOD detection. (Right): CIFAR-10 vs SVHN OOD detection. Our proposed abnormality score (red) is the best choice.
Given discrete timesteps to < t1 < ... <t = T, we first suggest two projection schemes, The (x, z) and H1's (x. z), by reversing the diffusion process until t = f,-1 and f c, respectively, starting from the noisy input x,, := f,z. Note that T can be implemented by only the diffusion- based denoising function Do while Hī can be done by multi-step denoising with Dg or single-step generation with the consistency model fo as described in Section 2.2. We refer wo and IT as single-step and full-step projections, respectively, and the latter illustrated in Figure la. We simply use T (x) and 11 (x) with N' I) unless otherwise stated.
We here systemically analyze the projection schemes TTJ and II's with the denoising function D9 and the consistency function fo. respectively, trained on CIFAR-10 [30] for various timesteps t E (c, T]. Note that both projections require only network evaluation. Figure 2 shows the projection results of SIFAR-10 100 [30] and SVHN [31] OOD images where f; x 1.09 (i.e. i = 7, N = 17, and = 80). One semantic changes in the images while their background statistics have not changed much. Furthermore, the shift from the in-distribution CIFAR-10 dataset is less drastic than the OOD dataset. We also observe that the projections (i) do not change any information when 4; is small, but (ii) generate a completely new image when & is too large, as shown in Figure la. Moreover, we observe that the single-step projection T outputs more blurry image than the full-step projection II';
Based on these observations, we propose projection based OOD detection scores by measuring the reconstruction error between input and its projection with distance metric d(-,)) +):
(x 6; 0) = Ez-NNO. [d (x, (x, z))]
(x, L; 0) = EZ-N((.)) (d (x, (x, z))] (3)
For the distance d, in this paper, we use the common metrics in computer vision: the l, distance and the perceptual distance, I.PIS [20]. One may expect that the latter is fit for our setup since focuses on semantic changes more than background changes. Other choices are also studied in Section 4.3.

---

We report the AUROC on the CIFAR-IO vs (CIFAR-ION 100 or SVHN) tasks in Figure 3. As expected. the perceptual distance better than the &g distance when & is not too small. Our proposed metric, Sh PIPS shows the best performance in the intermediate time step and this even exceeds the performance of crffusionn-mdel-baaae Furthermore, the performance of S degrades in larger time step due to the blurry output of the diffusion model Da. On the other hand, Sh² and S consistently struggle to detect the CIFAR-100 dataset and even output smaller abnormality scores on the SVHN dataset.
OOD detection methods [17, 18] in the CIFAR-I0 vs CIFAR- 100 task.
3.2 Projection Regret
While the perceptual distance performs better than the E2 distance, it may fail to capture the semantic change when background information dominant. For example, we refer to Figure 1b for the failure case of P While the projection of the OOD (i.e., bridge) data shows in-distribution (i.e., church) semantics, the area of the changed region is relatively small. Hence, Sh StPPPS outputs a lower detection score for the OOD data than the in-distribution data.
To reduce such a background bias, we propose Projection Regret, a novel and effective detection method using recursive projections inspired by the observations in Section 3.1: (i) projected image x' =TT'a (x) can be considered as an in in-distribution image with similar background to the original image x, and (ii) its recursive projection II' (x') may not change semantic information much because of (i). Therefore, the change in background statistics from x to IT' (x) would be similar to that from x' to 11'2 (x'), in other words, d(x, 11 (x)) z semantic difference + d(x', 11's (x))..
Based on this insight, we design our Projection Regret score, Spa. as follows:
Spa(x, a, 3:0) 0) = S [x, tg: 0) (z [$ (11' 2x, tg: 0)]
where a and 3 are timestep hyperparameters. We mainly use LPIPS [20] for the distance metric d. It is worth noting that Projection Regret is a plug-and-play module: any pre-trained diffusion model and any distance metric d can be used without extra training and information. The overall framework is illustrated in Figure 1 and this can be efficiently implemented by the consistency model fo and batch-wise computation described in Algorithm 1.
Ensemble. It is always challenging to select the best hyperparameters for the OOD detection task since we do not know the OOD information in advance. Furthermore, they may vary across different OOD datasets. To tackle this issue, we also suggest using an ensemble approach with multiple timestep pairs. Specifically, given a candidate set of pairs C {(o,, {)) we use the sum of all scores, ie., Spa(x; 0) = = La,,s)ec Spa(x, a, 33 0). in our main experiments (Section 4.2) with a roughly selected C. We discuss the effect of each component in Section 4.3.
4 Experiment
In this section, we evaluate the out-of-distribution (OOD) detection performance of our Projection Regret. We first describe our experimental setups, including benchmarks, baselines, and implementa- tion details of Projection Regret (Section 4.1.. We then present our main results on the benchmarks (Section 4.2) and finally show ablation experiments of our design choices (Section 4.3).
4.1 Experimental setups
We follow the standard OOD detection setup [33, 34, 35]. We provide further details in the Appendix.
Datasets. We examine Projection Regret under extensive in-distribution (ID) out-of-distribure (OOD) evaluation tasks. We use CIFAR-10/100 [30] and SVHN [31] for ID datasets, and CIFAR- 10/100, SVHN, LSUN [36]. ImageNet [37], Textures [38], and Interpolated 10 [19] for OOD datasets. Note that CIFAR- 10, 100, LSUN, and ImageNet have similar background information [39], so it is challenging to differentiate one from the other (e.g., CIFAR- 10 vs ImageNet).
Following Tack et al. [39], we remove the artificial noise present in the resized LSUN and ImageNet
'We first search the best hyperparameter that detects rotated in in-distribution data OODs [32]. Then, we set C. around the found hyperparameter.
5

---

Table Out-of-distribution detection performance (AUROC) under various in-distribution vs out-of- distribution tasks. Bold and underline denotes the best and second best methods.
C10
C100
SVHN
Method
SVHN CIFARI00
LSUN
ImageNet
SVHN
C10
LSUN
C1000
Diffission Models (26, 28]
Inpur Likelihood [6]
0,180
0.520
0.199
0.495
0.974
0.970
Input Complexity [7]
0.870
0.568
0.792
0.468
0.973
0.976
Likelibood Regret [8]
0,904
0.546
0.896
0.484
0.805
0.821
MSMA [17]
0.992
0.579
0.587
0.716
0.974
0,426
0.400
0.976
0,979
LMD [18]
0.992
0.607
0.985
0.568
0.914
0.876
Consistency Models [21]
MSMA [17]
0.707
0.570
0.605
0.578
0.643
0.506
0.559
0.985
0.981
LMD [18]
0,979
0.620
0.734
0.686
0.968
0.573
0.678
0.792
Projection Regret (ours)
0.775
0.837
0.814
0.945
0.577
0.682
0.993
Table 2:
detection performance (AUROC) of Projection Regret against various generative models. Bold and underline denotes the best and second best methods.
Method
Category
SVHN
CIFAR100
CF10 Interp.
Textures
Avg Rank.
PixcICNN+ [48]
Autoregressive
0.32
0.63
0.71
0.33
4.25
GLOW [42]
Flow-based
0.24
0.55
0,51
0.27
7.25
NVAE [25]
VAE
0.56
0.64
IGEBM [19]
EBM
0.63
0.50
0.70
0.48
5.25
VAEBM [23]
VAE EBM
0.83
0.62
0.70
4.33
Improved CD [22]
EBM
0.91
0.83
0.65
0.88
3.25
CLEL [24]
EBM
0.98
0.71
0.72
0.94
Projection Regret (ours)
Diffusion model
0.99
0.78
0.80
0.91
1.5
datasets. We also remove overlapping categories from the OOD datasets for each task. We also apply Projection Regret in a one-class classification benchmark on the CIFAR-10 dataset. In this benchmark, construct 10 subtasks where each single class in the corresponding subtask are chosen as in-distribution data and rest of the classes are selected as outlier distribution data. Finally, we also test Projection Regret in ColoredMNIST [40] benchmark where classifiers biased towards spurious background information. All images are 32x32.
Baselines. We compare Projection Regret against various snnerative-model-basseee OOD detection methods. For the diffusion diffusion-model- model- basclines, we compare Projection Regret against MSMA [17] and LMD [18]. We refer Section 5 for further explanation. We implement MSMA? in the original NCSN and implement LMD in the consistency model. Furthermore, we experiment with the ablation of MSMA that compares & distance between the input and its full-step projection instead of the single-step projection. Since likelihood evaluation available in Score-SDE [28], we also compare Projection Regret with input likelihood [6]. input complexity [7], and likelihood-regret [8] from the results of [18]. Finally, we compare Projection Regret with OOD detection methods that are based on alternative generative models, including VAE [41], GLOW [42], and energy-basee models (EBMs) [19, 23, 22, 24]. Finally, for one-class classification, various reconstruction and generative model-based methods are included for reference (e.g. [43, 44, 45, 46)).
Implementation details. Consistency models are trained by consistency distillation in CIFAR- 10/CIFAR- 100 datasets and consistency training in the SVHN/ColoredMNNIT datasets. We set the hyperparameters consistent with the author's choice [21] on CIFAR-10. We implement the code on the PyTorch [47] framework. All results are averaged across random seeds. We refer to the Appendix for further details.
We use the author's code in https: com/ awt.//ah/aaaaaa......

---

Table 3: One-class classification results (AUROC) of Projection Regret (PR) against various
baselines in the CIFAR-10 one-class classification task. Bold denotes the best method.
Method
Calegory
Plane
Car
Bird
Cat
Deer
Dog
Frog
Horse
Ship
Truck
Mean
McmAB [43]
Autsencoder
AnoGAN [44]
GAN
0.671
0.547
0.529
0.545
0.651
0.603
0.585
0.625
0.758
0.665
0.618
OCGAN [45]
GAN
0.757
0.531
0.640
0.620
0.723
0.620
0.723
0.575
0.820
0.554
0.657
DROCC [46]
Adv. Generation
0.817
0.767
0.667
0.671
0.736
0.744
0.744
0.714
0.800
0.762
0.742
P-KDGAN1499
GAN
0.825
0.744
0.703
0.605
0.765
0.652
0.797
0.723
0.827
0.735
0.738
LMD [18]
Diffusion Model
0.770
0.801
0.719
0.506
0.790
0.625
0.793
0,766
0.740
0.760
0.727
Rot+ Truns [50]
Self-sueerviee
0.760
0.961
0.867
0.772
0.911
0.885
0.884
0.961
0.943
0.918
0.886
Projection Regret (ours)
Diffusion Model
0.864
0.946
0.789
0.697
0.840
0.825
0.854
0.889
0.929
0.919
0.855
PR (Roce Trans) (ours)
Ilybrid
828
0.971
0.889
0.803
0.925
0.905
0.911
0.963
0.954
0.933
0,908
CIFAR-10 GFAR-100
40.
10).
CIAAR-19
0.699
0:897
0.57
0856
0.045
0.76
0g- 10
0, 4.
0.780
3.575
5.721
0,797
0.686
0.872
0.700
NAROO
AUROC
5.707
0.75
0718
0.798
0.76
0.54
6.729
0.7.41
5.74
0.722
0.68
0.500
Figure 4: Hyperparameter analysis of Projection Regret CIFAR-10 CIFAR-100 task. (left): Ablation on hyperparameter and 3. The proposed time ensemble performs better than the best hyperparameter. (middle): Ablation hyperparameter na. (right): Ablation on hyperparameter ng.
4.2
Main results
Table 1 shows the OOD-detection performance against Regret consistently outperforms MSMA and LMD on 8 out of benchmarks. While Projection Regret mainly focuses OODs with similar backgrounds, Projection Regret also outperforms baseline methods on the task where they show reasonable performance (e.g., CIFAR-10 vs SVHN).
methods. Projection
In contrast, MSMA struggles various benchmarks where the ID and the OOD share similar backgrounds (e.g., CIFAR-10 vy CIFAR-100), While LMD performs better in such benchmarks, LMD shows underwhelming performance in SVHN (CIFAR-10 or CIFAR-100) and even underperforms over input likelihood [6]. To analyze this phenomenon, we perform visualization of the SVHN samples that show the highest abnormality score and their inpainting reconstructions in the Appendix. The masking strategy of LMD sometimes discards semantics and thereby reconstructs a different image. In contrast, Projection Regret shows near-perfect OOD detection performance in these tasks.
We further compare Projection Regret against OOD detection methods based on alternative generative models (e.g., EBM [24], GLOW [42]) in Table 2. We also denote the category of each baseline method. wwwettiiie-ooe----eeeee OOD detection methods, Projection Regret shows the
Among the various best performance on average. Significantly, while CLEE [24] requires an extra self-supervised model for joint modeling, Projection Regret outperforms CLEL out of 4 tasks.
We finally compare Projection Regret against various one-class classification methods in Table 3. Projection Regret achieves the best performance and outperforms other reconstruction-bant methods consistently. Furthermore, when combined with the contrastive learning method [50] by simply multiplying the anomaly score, our approach consistently improves the performance of the contrastive training- based learning method. Specifically, Projection Regret outperforms the contrastive training- based method in airplane class where the data contains images captured in varying degrees. As such, contrastive training-bast based methods underperform since they use Station-basedd augmentation to generate pseudo-outlir data.
4.3 Analysis
Hyperparameter analysis. We analyze the effect of hyperparameters for Projection Regret under the CIFAR-10 CIFAR-100 100 task. First, we conduct experiments with varying the timestep index
7

---

Table Ablation Study on each component of Projection Regret. In In-distribution dataset the
CIFAR-10 10 dataset. we report AUROC. Gains are computed against LMD [18].
Method
SVHN
CIFARIO0
LSUN
ImageNet
LMD [18]
0.979
0.620
0.734
0.686
Projection (i... P
0,966 (-1.3%)
0.667 (+4.7%)
0.762 (+2.8%)
0.718 (+3.2%)
Projection Regret (a 9, 8 8)
0,986 (+0.7%)
0.760 (+14.0%)
0.828 (+9,4%)
0.795 (+10.9%)
Projection Regret (Ensemble)
0.993 (+1.4%)
0.775 (+15.5%)
0.837 (+10.3%)
0.814 (+12.8%)
Table 5: AUROC of Projection Regret given different distance metric choices. Bold and underline denotes the best and second best methods.
Method
Distance d
SVHN
CIFARI00
LSUN
ImageNet
LMD [18]
0,979
0.620
0.734
0,686
SSIM [27]
0.328
0.629
0.650
0.620
Projection Regret
LPIPS [20]
0.993
0.775
0.837
0,814
UNet (proposed)
0.917
0.734
0,865
0,815
hyperparameters a and B. As mentioned in Section 3.2, the ensemble performs better (i.e., 0.775 in 9, 8).
Table 1) than the best hyperparameter selected (i.e., 0.760 when
Furthermore, we perform ablation on the ensemble size hyperparameter nh and ng. While selecting a larger ensemble size improves the performance, we show reasonable performance can be achieved in smaller na or n& (e.g.. na = 40, ng 5 or aa 20, ng 10) without much sacrifice on the performance.
Ablation the components of Projection Regret. We further ablate the contribution of each component in Projection Regret under the CIFAR-10 dataset. First, we set the best- rest-performing timestep index hyperparameter (a B 8) under the CIFAR-10 CIFAR-100 detection task. Furthermore, we also compare the projection baseline method, i.e., SLPIPS (-, tg; 0) where B = 8.
The performance of each method is shown in Table 4. We also denote the absolute improvements over LMD [18]. Our proposed baseline method SLPPP already improves over LMD in 3 out of tasks, especially in the CIFAR- 10 OOD dataset with similar background statistics. Furthermore, observe consistent gains on both Projection Regret with and without ensemble over baselines. The largest gain is observed in Projection Regret over the projection baseline St (e.g., 9.3% in the CIFAR-10 ys CIFAR-100 detection task).
Diffusion-model-basee distance metric. While we mainly use LPIPS [20] as the distance metric d, we here explore alternative choices on the distance metric.
We first explore the metric that can be extracted directly from the pre-trained diffusion model. To this end, we consider intermediate feature maps of the U-Net [51] decoder of the consistency model fe since the feature maps may contain semantic information to reconstruct the original image [52]. Formally, let (() f, z) E CR x Hex Wt be the f-th decoder feature map of fo + tz, t). Motivated by semantic segmentation literature [53, 54] that utilize the cosine distance between two feature maps, we define our U-Net-based distance metric dunet (s,
+) as follows:
dynerr (x,x) (O.T)(
Σ 1=1
Edcos deos ())
F.2
(5)
H&We
|lu/lu/l/ l 1|2 is the cosine distance and t., is a timestep hyperparameter.
where deos (u, v) We set f. small enough reconstruct the original image, e.g., t, 0.06 (i.e., 7 3), for evaluation.
We report the result in Table 5. metric. While LPIPS performs the best, our proposed distance dumee shows competitive performance against LPIPS and outperforms LMD and MSMA in CIFAR-10 ys (CIFAR-100, LSUN or ImageNet) tasks. On the other hand, SSIM harshly underperforms LPIPS and dusat-
We also test the negative SSIM index [27] as the baseline distance
Using the diffusion model for full-step projection. While the consistency model is used for full-step projection throughout our experiments, a diffusion model can also generate full-step projection 11', (x)
8

---

Table 6: Performance (AUROC) of Projection Regret computed in the EDM [29] with Heun's 's 2nd order sampler. Performance of Projection Regret under consistency model [21] under the same hyperparameter included for comparison.
Model
SVHN CIFARI00 LSUN
ImageNet
aa 10, n2
HDM [29]
0.986
0.763 0.761
0.825
0.799
CM [21]
0.990
0.816
0.797
40, n2
10
CM [21]
0.993
0.775
0.837
0.814
Table 7: OOD detection results (TNR at 95% TPR, AUROC) under more severe background bias. In-distribution dataset is the colored MNIST dataset in Ming al. [55] with r 0.45. The result in Ming et al. [55] is included for reference. Higher is better in all metrics.
Method
Spurious OOD dataset
LSUN
iSUN
Textures
Ming et al. [55] Projection Regret
0.69//0.867 0. 0.958/0.990
0.933/0.988 982 0,901/0.959
0.937/0,985
1.000/1.00 1000
1.000/1 .000 1.000/1.000
with better generation quality through multi-step sampling. For verification, we experiment with the EDM [29] trained in the CIFAR-10 dataset and perform full-step projection via the 2nd-order Heun's sampler. Since sampling by diffusion model increases the computation cost by 2,3 times compared to the consistency model, we instead experiment on the smaller number of ensemble size na 10, n, 5 and report the result of Projection Regret with consistency model computed in the same hyperparameter setup as baseline.
We report the result in Table 6. We denote the result of Projection Regret utilizing the full-step projection based on the diffusion model and the consistency model as EDM and CM, respectively. Despite its excessive computation cost, Projection Regret on EDM shows negligible performance gains compared to that on CM.
Projection Regret helps robustness against spurious correlations. While we mainly discuss the problem generative model overfitting into background information in the unsupervised domain, such phenomenon is also observed in supervised learning where the classifier overfits the background or texture information [56]. Motivated by the results, we further evaluate our results against the spurious OODs where the OOD data share similar background to the ID) data but with different semantics. Specifically, we experiment on the coloredMNIST [40] dataset following the protocol of [55]. We report the result in Table 7. Note that spurious OOD data have the same background statistics and different semantic (digit) information. Projection Regret achieves near-perfect OOD detection performance. It is worth noting that Projection Regret does not any additional information while the baseline proposed by [55] uses label information.
5 Related Works
Fast sampling of diffusion models. While we use full-step projection of the image for Projection Regret, this induces excessive sampling complexity in standard diffusion models [26, 57, 28]. Hence, we discuss works that efficiently reduce the sampling complexity of the diffusion models. PNDM [58] and GENIE [59] utilize multi-step solvers to reduce the approximation error when sampling with low NFEs. Salimans and Ho [60] iteratively distill the 2-step DDIM [61] sampler of a teacher network into student network.
didor--distribution detection on generative models. Generative models have been widely applied to cuttof-distributiee detection since they implicitly model the in-distribution data distribution. A straightforward application directly using the likelihood of the explicit likelihood model (e.g.. VAE [41]) as a normality score function [6]. However, Nalisnick et al. [33] discover that such models may assign a higher likelihood to OOD data. plethora of research has been devoted to explaining the aforementioned phenomenon [62, 7, 63, 64). Serrà et al. [7] hypothesize that deep generative models assign higher likelihood on the simple out-of-distribution dataset. Kirichenko et al. [63] propose that flow-based models' [42] likelihood is dominated by local pixel correlations.

---

Finally, diffusion models have been applied for OOD detection due to their superior generation capability. Mahmood et al. [17] estimate the &g distance between the input image and its single-step projection as the abnormality score function. Liu et al. [18] applies checkerboard masking to the input image and performs a diffusion--oodll-asee inpainting task. Then, reconstruction loss between the inpainted image and the original image is set as the abnormality score function.
Reconstruction-bast basee out-o--distribution detection methods. Instead of directly applying pre- trained generative models for novelty detection, several methods[43, 46, 45] train a reconstruction model for distr---ditiittiionn. MemAE [43] utilizes a memory structure that stores normal representa- tion. DROCC [46] interprets one-class classification as binary classification and generates boundary data via adversarial generation. Similarly, OCGAN [45] trains an auxiliary classifier that controls the generative latent space of the GAN model. While these methods are mainly proposed for one-class classification, we are able to outperform them with significant gains.
Debiasing classifiers against spurious correlations. Our objective to capture semantic information for novelty detection aligns with lines of research that aim to debias classifiers against spurious correlations [40, 56, 65, 66, 67]. Geirhos et al. [56] first observe that CNN-trained classifiers show vulnerability under the shift of non-semantic regions. Izmailov et al. [66] tackles the issue by group robustness methods. Motivated by research on interpretability, Wu et al. [67] apply Grad-CAM [68] discover semantic features. Ming al. [55] extend such line of research to out-of-distribution detection by testing OOD detection algorithms on spurious anomalies.
6 Conclusion
In this paper, we propose Projection Regret, a novel and effective -uito--distribution (OOD) detection method based on the diffusion-based projection that transforms any input image into an in-distribution image with similar background statistics. We also utilize recursive projections to reduce background bias for improving OOD detection. Extensive experiments demonstrate the efficacy of Projection Regret on various benchmarks against the state-of-the-ar that our work will guide new interesting research directions in developing diffusion-basee methods for not only OOD detection but also other downstream tasks such as image-to-image translation [69].
methods. We hope
Limitation. Throughout our experiments, we mainly use the consistency models to compute the full- step projection via one network evaluation. This efficiency a double-edged sword, e.g., consistency models often generate low-quality images compared to diffusion models. This might degrade the detection accuracy of our Projection Regret, especially when the training distribution Pdaa difficult to learn, e.g., high-resolution images. However, as observed in Table 6. strongly believe that our approach can provide a strong signal to detect -odtt-ddistribttionn samples even if the consistency model is imperfect. In addition, we also think there is room for improving the detection performance by utilizing internal behaviors of the model like our distance metric duvet- Furthermore, since diffusion models computation-heat U-Net architecture, computational speed is much slower than other reconstruction based methods.
Negative social impact. While near-perfect novelty detection algorithms are crucial for real-life applications, they can be misused against societal agreement. For example, an unauthorized data scraping model may utilize the novelty detection algorithm to effectively collect the widespread data. Furthermore, the novelty detection method can be misused to keep surveillance on minority groups
Acknowledgments and Disclosure of Funding
This work is fully supported by LG Al Research.
10

---

References
[1] Victoria Hodge and Jim Austin. survey of outlier detection methodologies. Artificial intelligence review, 22:85-126, 2004.
[2] lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adver- sarial examples. ICLR, 2015.
[3] Jasper Linmans, Jeroen van der Laak, and Geert Litjens. Efficient out-of-distribution detection in digital pathology using multi-head convolutional neural networks. In MIDL, pages 465-478, 2020.
[4] Julia Nitsch, Masha Itkina, Ransalu Senanayake, Juan Nieto, Max Schmidt, Roland Siegwart, Mykel J Kochenderfer, and Cesar Cadena. Out-of-distribution detection for automotive percep- tion. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pages 2938 2943. IEEE, 2021.
[5] Ramneet Kaur, Kaustubh Sridhar, Sangdon Park, Susmit Jha, Anirban Roy, Oleg Sokolsky, and Insup Lee. Codit: Conformal out-o--distribution detection in time-series data. arXiv preprint 12X1:2207..17669 2022.
[6] C. M. Bishop. Novelty detection and neural network validation. In ICANN. 1994.
[7] Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F. Núñez, and Jordi Luque. Input complexity and out-of-distribution detection with likelihood-based generative models. In ICLR, 2020.
[8] Zhisheng Xiao, Qing Yan, and Yali Amit. Likelihood regret: An -idi-ii-iittitttionnn detection score for variational autoencoder. In NeurIPS, 2020.
[9] Houssam Zenati, Manon Romain, Chuan Sheng Foo, Bruno Lecouat, and Vijay Ramaseshan Chandrasekhar. Adversarially learned anomaly detection. In ICDM, 2018.
[10] Gukyeong Kwon Mohit Prabhushankar, Dogancan Temel, and Ghassan AlRegib. Backpropa- gated gradient representations for anomaly detection. In ECCV, 2020.
[11] Robin Rombach, Andreas Blattman, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, 2022.
[12] Wenhu Chen, Hexiang Hu, Chitwan Saharia, and William W. Cohen. Re-imagen.retrieval augmented text-to-image generator. In ICLR, 2023.
[13] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. arXiv preprint arXiv:2005. 14165, 2020.
[14] Apruva Gandhi and Shomik Jain. Adversarial perturbations fool deepfake detectors. In IJCNN, 2020.
[15] Shiyue Zhang and Mohit Bansal. Addressing semantic drift in question generation for semi- supervised question answering. In EMNLP, 2019.
[16] Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J. Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. arXiv preprint arXiv:2104.07633 2021.
[17] Ahsan Mahmood, Junier Oliva, and Martin Andreas Styner. Multiscale score matching for out-of-distribution detection. In ICLR, 2021.
[18] Zhenzhen Liu, Jin Peng Zhou, Yufan Wang, and Kilian Q. Weinberger. Unsupervised out-of- distribution detection with diffusion inpainting. arXiv preprint 103266,,00. 2023,
11

---

[19] Yilun Du and Igor Mordatch. Implicit generation and generalization in meergy-bssed models. In NeurIPS, 2019.
[20] Richard Zhang, Philip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreason- able effectiveness of deep features as perceptual metric. In CVPR, 2018.
[21] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. In ICML, 2023.
[22] Yilun Du, Shuang Li, Joshua Tenenbaum, and Igor Mordatch. Improved contrastive divergence training of energy-based models. In ICMI.,,222
[23] Zhisheng Xiao, Karsten Kreis, Jan Kautz, and Arash Vahdat. Vaebm: A symbiosis between variational autoencoders and energy-based models. In ICLR, 2021.
[24] Hankook Lee, Jongheon Jeong, Sejun Park, and Jinwoo Shin. Guiding energy- -asedd models via contrastive latent variables. In ICLR, 2023.
[25] Arash Vahdat and Jan Kautz. Nvac: A deep hicrarchical variational autoencoder. In NeurIPS, 2020.
[26] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
[27] Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. Image quality assessment: From error visibility to structural similarity. IEEE Transactions Image Processing, 2004.
[28] Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021.
[29] Tero Karras, Miia Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion based generative models. In NeurlPS, 2022.
[30] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
[31] Yuval Netzer, Tao Wang. Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop Deep Learning and Unsupervised Feature Learning, 2011.
[32] Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. In ICLR, 2018
[33] Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? In ICLR, 2019.
[34] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of diitt---iittiittinn image detection in neural networks. In ICLR, 2018
[35] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting outtoo--distribution samples and adversarial attacks. In NeurIPS, 2018.
[36] Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 15X.v31506.03335, 2015.
[37] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
[38] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
[39] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. In NeurIPS, 2020.
12

---

[40] Martin Arjovsky, Léon Bottou, Ishann Gulrajani, and David Lopez-Paz. Invariant risk mini- mization, arXiv preprint arXiv:190..02893, 2019.
[41] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In NeurIPS, 2014.
[42]
Durk P. Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In NeurIPS, 2018.
[43] Dong Gong, Lingqiao Liu, Vuong Le. Budhaditya Saha, Moussa Reda Mansour, Svetha Venkatesh, and Anton van den Hengel. Memorizing normality to detect anomaly: memory- augmented deep autoencoder for unsupervised anomaly detection. In ICCV, 2019.
[44] Thomas Schlegl, Philipp Seeböck, Sebastian M. Waldstein, Ursula Schmidt -rffurth and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In IPMI, 2017.
[45] Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In CVPR, 2019.
[46]
Sachin Goyal, Aditi Raghyunathan, Moksh Jain, Harsha Simhadri, and Prateek Jain. Droce: Deep robust one-class classification. In ICML, 2020.
[47] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang. Zachary De Vito,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In NeurlPS, 2019.
[48] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P. Kingma. Pixelcnn++: pixelenn implementation with discretized logistic mixture likelihood and other modifications. In ICLR, 2017.
[49] Zhiwei Zhang, Shifeng Chen, and Lei Sun. P-kdgan: Progressive knowledge distillation with gans for one-class novelty detection. In IJCAI, 2020.
[50] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. In NeurlPS, 2019.
[51] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[52] Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label-efficient semantic segmentation with diffusion models. In ICLR, 2022.
[53] Gen Li, Varun Jampani, Laura Sevilla-Lara, Deqing Sun, Jonghyun Kim, and Joongkyu Kim. Adaptive prototype learning and allocation for few-shot segmentation. In CVPR, 2021.
[54] Bo Pang, Yizhuo Li, Yifan Zhang, Gao Peng, Jiajun Tang, Kaiwen Zha, Jiefeng Li, and Cewu Lu. Unsupervised representation for semantic segmentation by implicit cycle-attention contrastive learning. In AAAI, 2022.
[55] Yifei Ming, Hang Yin, and Yixuan Li. On the impact of spurious correlation for out-of- distribution detection. In AAAI, 2022.
[56] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. ICLR, 2019.
[57] Jonathan Ho, Ajay Jain, and Picter Abbeel. Denoising diffusion probabilistic models. In NeurlPS, 2020.
[58] Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In ICLR, 2022.
13

---

[59] Tim Dockhorn, Arash Vahdat, and Karsten Kreis. Genie: Higher-order denoising diffusion solvers. In NeurlPS, 2022.
[60] Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In ICLR, 2022.
[61] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021.
[62] Sungik Choi and Sae- Young Chung. Novelty detection via blurring. ICIR, 2020.
[63] Polina Kirichenko, Pavel Izmailov, and Andrew Gordon Wilson. Why normalizing flows fail to detect
data. In NeurIPS, 2020.
[64] Lily H. Zhang, Mark Goldstein, and Rajesh Ranganath. Understanding failures out-of- distribution detection with deep generative models. In ICML, 2021.
[65] Junhyun Nam. Hyuntak Cha, Sungsoo Ahn, Jacho Lee, and Jinwoo Shin. Learning from failure: training debiased classifier from biased classifier. In NeurlPS, 2020.
[66] Pavel Izmailov, Polina Kirichenko, Nate Gruver, and Andrew Gordon Wilson. On feature learning in the presence of spurious correlations. In NeurlPS, 2022.
[67] Shirley Wu, Mert Yuksekgonul, Linjun Zhang, and James Zou. Discover and cure: concept- aware mitigation of spurious correlations. In ICML, 2023.
[68] Ramprasaath R. Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based localization. In ICCV, 2017.
[69] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to-image translation. ICLR, 2023.
[70] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In NeurIPS, 2017.
14

---

Supplementary Material
Projection Regret: Reducing Background Bias for Novelty Detection via Diffusion Models
A Full algorithm
We refer Algorithm for the efficient computation of method.
Algorithm 1 Projection Regret with Consistency Models (PyTorch-like Pseudo-code)
# fxx, t): consistency model
# t_alpha, t_beta: the projection timesteps
# n_alpha, n_beta: the projection ensemble sizes
# (A, n): repeat tensor A n times along axis=0
# d(a, b): distance function, e.g., LPIPS
Projection t): return f(x
(andd_11ie(x))
def ProjectionRegrettss (x):
[1, H, W]
x_proj = Projection (repeat (x, n_alpha * n_beta), t_beta)
= projection(repeatt (x, n_alpha), t_alpha) n_beta), t_beta)
y-proj = (rreecioorrrppattteeeeee
dx = d (xepeat (x, n_alpha n_beta), x_proj).mean() mean()
dy (repeat (y, n_beta), y_proj).mean mean() return dx dy
B Implementation details
We train the baseline EDM [29] model for 390k iterations for all datasets. Then, we perform consis- tency distillation [21] for the CIFAR-I0 10 and CIFAR 100 [30] datasets and consistency training for the SVHN [31] dataset. The resulting FID score [70] of the CIFAR 10--0///IFARR 100 dataset is 5.52/7.0 and 8.0 for the SVHN dataset (the model trained by consistency training usually shows higher FID than the trained by consistency distillation). While training the base EDM diffusion model and the con- sistency model, we use A100 GPUs. During the out-of-distribution detection, we use A100 GPU. For the ensemble size hyperparameter n, and Rgg, set n, 40, n, 10 for experiments and n4, = 100, n for CIFAR- 100 experiment. As mentioned in the main paper, set base hyperparameter that shows the best performance against the rotated in-distribution dataset and set the ensemble configurations around it. We use 8 hyperparameter configurations for the CIFAR-10 and CIFAR-100 datasets and 4 hyperparameter configurations for the SVHN dataset. To be specific. chosen configurations are C = {7, 6). (7, 7), (8, 7), (8, 8), (9, 8), (9, 9), (10, 9). (10, 10)} for CIFAR-IO/CIFAR 100 dataset and {110, 10), (11, 10), (11, 11), (12, 11)} for the SVHN dataset.
C Visualization of LMD in SVHN dataset
We further reconstruct the output of the LMD [18] when SVHN is the in-distribution dataset. Follow- ing the practice of Liu et al. [18], we set the mask as the alternating checkboard mask throughout the experiments in Section 4.2, We visualize a grid of 9 samples and their reconstructions where LMD shows the highest abnormality such data, thereby failing to reconstruct consistent images. Furthermore, as mentioned in Section 4.2, sometimes the checkboard strategy discards the in-distribution data's 's semantics and thereby fails to reconstruct the original image.
in Figure 5. We can see LMD's inconsistent reconstruction in
15

---

56
9
26
38
6
80
10
10
(a) Original images
(b) Masked images
(c) Reconstruced images
Figure 5: Visualization of LMD [18] in the SVHN [31] dataset. (a) Original SVHN images where LMD outputs the highest abnormality score. (b) The masked images following the practice of Liu al. [18]. (c) Their reconstruction images through LMD.
16


Reviews:
[{"review_id": "yLUaKR7PUg", "reviewer": "Reviewer_mygo", "summary": "This paper proposes Projection Regret (PR) to mitigate the bias of background information for novelty detection. As an effective perceptual distance, it is able to detect abnormality by reducing the effect of dominant background using recursive projections. Experimental results show the effectiveness of the proposed novelty detection framework. ", "questions": "Refer to the Weaknesses.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Good motivation to explain the background bias problem.\n2. well written and easy to follow this paper.\n3. The effectiveness of Projection Regret. \n", "weaknesses": "My main concerns lie in the following two aspects: (1) About the detection running time. Diffusion models are used in the proposed method, which leads to the increase of inference time when detecting the abnormality of test images. (2) It lacks experimental comparison on other practical industry product detection tasks, such as MVTecAD and BTAD benchmark, which seem to be more challenging to evaluate the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer mygo,\n\nWe sincerely thank you for your helpful feedback and insightful comments. In what follows, we address your concerns one by one.\n___\n**[Q1]** About the detection running time. Diffusion models are used in the proposed method, which leads to an increase of inference time when detecting the abnormality of test images.\\\n**[A1]** During inference, our method (Projection Regret, PR) is very efficient because PR only requires three projections and each projection can be computed by one forward pass using a consistency model [1] that enables one-shot generation, as described in Algorithm 1. The second-best baseline, LMD [2], on the other hand, requires multiple (sequential) sampling steps for inpainting. As a result, PR is 1.8x faster than LMD.\n\n---\n**[Q2]** It lacks experimental comparison on other practical industry product detection tasks, such as MVTecAD and BTAD benchmark, which seem to be more challenging to evaluate the effectiveness of the proposed method. \\\n**[A2]** We first note that industrial anomaly detection (IAD) like MvTecAD and BTAD benchmarks are not within our scope since they differ from our target task: unsupervised novelty detection (UND), also known as out-of-distribution (OOD) detection. To be specific, UND aims to detect OOD (i.e., novel or unseen) semantic categories/shapes, while the IAD task aims to detect a localized defect in a pixel-wise manner. Therefore, their sample distributions are also significantly different. For example, OOD samples of the IAD task only differ in the small localized region against the ID samples, but OOD samples of the UND task may differ in shapes, background, and so on. Due to the aforementioned reasons, the UND and IAD tasks have been investigated in different research directions, e.g., formulating why the novelty detection model overfits to non-semantic information in UND [3] and formulating abnormality score in local pixels in IAD [4,5].  We think that IAD is an interesting extension of our work and leave the extension for future work. \n___\n**References** \\\n[1] Consistency models, ICML 2023 \\\n[2] Unsupervised out-of-distribution detection with diffusion inpainting, ICML 2023 \\\n[3] Novelty detection via blurring, ICLR 2020 \\\n[4] Towards total recall in industrial anomaly detection, CVPR 2022 \\\n[5] Pushing the limits of few-shot anomaly detection in industry vision: Graphcore, ICLR 2023\n"}, {"review_id": "xstrl58SRM", "reviewer": "Reviewer_ZGie", "summary": "This paper presents a novel approach to detecting and handling novelty in data using a generative diffusion model, with a specific focus on addressing biased backgrounds. The proposed method aims to transform noisy samples into perfect ones by leveraging the capabilities of the diffusion model. The central idea is that by reversing the effects of noise on an inlier sample (x+noise), the resulting output will closely resemble the original sample (X). Conversely, if X were to be treated as an out-of-distribution (OOD) sample, the diffusion model trained on similar OOD backgrounds could project the samples onto a background resembling the inlier data.\n\nUnlike traditional methods, such as auto-encoders that rely on reconstruction error, this approach emphasizes the significance of high and meaningful reconstruction errors for anomaly detection. Since the primary focus is on the foreground, which constitutes a small portion of the images, the reconstruction error becomes a more reliable indicator of anomalies.\n\nTo validate the effectiveness of the proposed method, the authors conducted evaluations on various datasets, including CIFAR-10, CIFAR-100, SVHN, and ImageNet. Comparative analyses were performed against existing generative models commonly employed for novelty detection.", "questions": "The idea and problem mentioned in this paper are interesting. However, the authors need to consider the following points: (1) They should review documents in the field of fairness for anomaly detection and those that attempt to mitigate bias. (2) There are benchmarks for evaluating methods biased against the background, such as Hard ImageNet. While these datasets may not be explicitly designed for out-of-distribution (OOD) detection, evaluating the proposed method on such data would be beneficial. Although CIFAR-10 and CIFAR-100 not have meaningful background concepts, additional evaluations on datasets with more pronounced background biases would provide valuable insights. (3) The comparison with existing methods needs improvement.\"", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "\nRecently, there has been growing concern that existing methods for image classification, which are generally not specific to one-class classification or novelty detection tasks, face significant challenges. Additionally, studies in the field have highlighted the problem of bias in the background for novelty detection. In response, this paper addresses this ongoing challenge by proposing a solution.\n\nThe paper leverages the reverse step of the diffusion model to restore noisy outliers, with the goal of aligning the background of these recovered samples more closely with the inlier samples. This approach is intriguing and tackles a specific aspect of the problem.\n\nThe results demonstrate the effectiveness of the proposed idea.", "weaknesses": "One weakness of this paper is that the main idea of using projection and reconstruction error for novelty detection is not novel. There have been previous papers, such as \"Adversarially One-Class Classifier for Novelty Detection\" (CVPR 2018), that have presented similar ideas but with different approaches. Thus, the high-level novelty of the proposed method is questionable. However, the method is novel. \n \nFurthermore, the paper claims the method is less biased towards the background. However, the datasets used for evaluation, such as CIFAR and SVHN, are not known for having extreme biases toward the background. To address the bias problem effectively, evaluating the method on datasets that exhibit strong biases in the background would be more appropriate, such as \"Hard ImageNet: Segmentations for Objects with Strong Spurious Cues.\" Other papers specifically tackle the issue of spurious correlations and out-of-distribution detection, such as \"On The Impact Of Spurious Correlation For Out-of-distribution Detection\" by Yifei Ming (2021). It would be valuable to compare the results of the proposed method with those papers that specifically address the bias problem rather than just comparing against other generative models.\n\nMoreover, the proposed method may fail in cases where the OOD samples have backgrounds that are very similar to the inliers but differ in the main concept or foreground. In such scenarios, the background would be ideally recovered using the proposed method, leading to false positives where OOD samples are detected as inliers.\n\n\n\nAt the moment there are a lot of methods that try to handle the same problem to the reconstruction error can be a reliable score for novelty detection for example, see the Mem-autoencoder paper for novelty or anomaly detection. \n\n\nAddressing these weaknesses and providing a more thorough comparison with relevant papers and datasets would strengthen the overall contribution of the proposed method.\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ZGie,\n\nWe sincerely thank you for your helpful feedback and insightful comments. In what follows, we address your concerns one by one.\n___\n**[Q1]** One weakness of this paper is that the main idea of using projection and reconstruction error for novelty detection is not novel. There have been previous papers, such as \"Adversarially One-Class Classifier for Novelty Detection\" [1], that have presented similar ideas but with different approaches. Thus, the high-level novelty of the proposed method is questionable. \\\n**[A1]** We ​​politely disagree with the reviewer’s opinion: our main idea is to use the **“recursive projection”** and the **“regret term”**, and it is clearly different from the previous works, e.g., [1]. As we discussed in Section 3.2, a simple reconstruction approach like [1] may fail to capture semantic changes when background information is dominant. Hence, our idea plays a crucial role in novelty detection as we empirically verified in our ablation experiment (Table 3). We strongly believe that our work is a non-trivial extension of the previous reconstruction-based approaches, i.e., our idea is novel and effective compared to them. We respectfully ask you to reconsider your position on our high-level novelty.\n___\n**[Q2]** Furthermore, the paper claims the method is less biased towards the background. However, the datasets used for evaluation, such as CIFAR and SVHN, are not known for having extreme biases toward the background. To address the bias problem effectively, evaluating the method on datasets that exhibit strong biases in the background would be more appropriate, such as \"Hard ImageNet: Segmentations for Objects with Strong Spurious Cues.\" ...\\\n**[A2]** Thank you for the valuable suggestion on experiment setups. As you suggested, we validate the effectiveness of our method on a spurious out-of-distribution (OOD) benchmark, ColorMNIST [2], presented by Ming et al. [3] where digits are given as semantic information. In this benchmark, different digits with the same background color are given as spurious OOD samples. Furthermore, in-distribution digits and background information are highly correlated (r=0.45) in the training setup. As we reported in Table 6 (see Global Response PDF), our method significantly outperforms [3] even without using label information. This is because our method relies on generative modeling which aims to learn every pixel information rather than collapsing into a shortcut solution. It is also worth noting that the methods [2-3] that address the bias problem in classifiers cannot be directly applied to unsupervised OOD detection since the above methods require label information.\n\nWe believe this discussion further strengthens our contribution, especially for various practical and challenging scenarios. We will incorporate this experimental result and discussion into the final manuscript.\n___\n**[Q3]** Moreover, the proposed method may fail in cases where the OOD samples have backgrounds that are very similar to the inliers but differ in the main concept or foreground. In such scenarios, the background would be ideally recovered using the proposed method, leading to false positives where OOD samples are detected as inliers.\\\n**[A3]**  We clarify that such a scenario is not our failure case because our detection score is based on the distance between a test image and its projection. If an out-of-distribution (OOD) sample has background statistics that are very similar to the inliers but differ in the main concept, then our projection of the OOD sample becomes similar to an inlier one, which leads to a large distance between the OOD sample and its projection due to their semantic distance. In contrast, an in-distribution sample has a small distance because its projection has similar semantic information.\n\nThis claim is also supported by our experimental results. First, although the ColorMNIST dataset [2] presented in [A2] includes such OOD samples (i.e., different digits but the same background colors), our method achieves near-perfect OOD detection performance as reported in Table 6 (see Global Response PDF). In addition, we think that CIFAR-10 vs CIFAR-100 is another representative task of discriminating OOD samples with similar backgrounds. In this task, we outperform the previous best diffusion-model-based method, LMD [4], by a large margin. These experimental results verify that our method can detect OOD samples that have similar backgrounds to the inliers.\n___\n**[Q4]** Comparison against Mem-autoencoder paper.\\\n**[A4]** We appreciate you introducing the related work, Mem-autoencoder (MemAE) [5]. For the comparison against [5], we experiment with Projection Regret and other competitive baselines in the one-class classification task on CIFAR-10 and report their results in Table 7 (see Global Response PDF). Our method greatly outperforms the reconstruction-based baselines, including MemAE, across all classes. For example, Projection Regret outperforms MemAE by 25\\%.\n___\n**[Q5]** Providing a more thorough comparison with relevant papers and datasets would strengthen the overall contribution of the proposed method.\\\n**[A5]** We appreciate the reviewer for the introduction of related fields and methods. We will include the related research on spurious correlations [2] and reconstruction-error-based novelty detection methods [1,5] further in the final manuscript.\n___\n**References**\\\n[1] Adversarially learned one-class classifier for novelty detection, CVPR 2018\\\n[2] Invariant Risk Minimization, Arxiv 2019\\\n[3] On the impact of spurious correlation for out-of-distribution detection, AAAI 2022\\\n[4] Unsupervised out-of-distribution detection with diffusion inpainting, ICML 2023\\\n[5] Memorizing normality to detect anomaly: memory-augmented deep autoencoder for unsupervised anomaly detection, ICCV 2019\n\n"}, {"review_id": "B8tFr6IvgJ", "reviewer": "Reviewer_9KBe", "summary": "This paper discusses a method in machine learning known as Novelty Detection, which is used to identify abnormal or out-of-distribution (OOD) samples. The authors suggest that diffusion models, a popular generative framework due to their strong generation performance, has recently become an attractive tool for novelty detection.\n\nHowever, the authors note a problem: while these models excel at generating high-quality results, they often struggle to detect OOD images with similar backgrounds. This issue, referred to as 'background bias', can lead to inaccurate novelty detection.\n\nTo address this issue, the paper proposes 'Projection Regret' (PR), a method which reduces the impact of background bias and improves the accuracy of novelty detection. PR calculates the perceptual distance between a test image and its 'projection' (an in-distribution sample with similar background information created by the diffusion model). To further mitigate the effect of dominant background information, PR uses recursive projections to cancel out the background bias.\n\nThe paper also introduces an ensemble of multiple projections for improved detection performance, calculated efficiently via a consistency model. A new perceptual distance metric using underlying features of the diffusion model is proposed and compared to other metrics, showing promising results.\n\nIn conclusion, the paper's main contributions are:\n1. Identifying the issue of background bias in novelty detection via diffusion models.\n2. Proposing a solution, Projection Regret (PR), which mitigates this bias and enhances OOD detection.\n3. Introducing an alternative perceptual distance metric computed from decoder features of the pre-trained diffusion model.\n\nExtensive experiments demonstrate that PR significantly outperforms previous diffusion-based novelty detection methods, showing potential for future applications.", "questions": "**Questions:**\n\n1. Could you provide more details on the computational requirements of PR? For instance, what is the time complexity and how does it scale with the size of the dataset?\n\n2. Would PR be equally effective in real-world applications where data might be more complex or noisy compared to standard datasets like CIFAR-10 or CIFAR-100?\n\n3. Could you delve deeper into the issue of background bias? Understanding its origins and why it affects diffusion models specifically could strengthen the paper.\n\n4. Have you investigated the robustness of PR against various types of noise and distortions? If not, do you anticipate that the method would be robust against such challenges?\n\n**Suggestions:**\n\n1. Consider incorporating additional evaluation metrics such as precision-recall curves or F1 scores for a more comprehensive evaluation of PR's performance.\n\n2. It would be beneficial to demonstrate how PR performs with real-world datasets, such as medical imaging data or self-driving car sensory data. This would help readers understand its applicability and effectiveness in practical scenarios.\n\n3. Detailed information about computational efficiency, including time complexity and scalability aspects, would be useful for potential users assessing suitability for their specific use-cases.\n\n4. Conducting a robustness analysis against various types of noise and distortions would further validate the efficacy of PR under different challenging conditions.\n\n", "limitations": "The authors have addressed some limitations of their work, particularly the issue of background bias in diffusion models and how their proposed method, Projection Regret (PR), mitigates this. However, a few additional limitations could be further discussed:\n\n1. **Computational Resources:** While the authors mention that PR can be calculated efficiently, the paper does not provide specific details on computational requirements or scalability. As advanced machine learning models often require significant computational resources, an evaluation of this aspect would be helpful for potential users.\n\n2. **Robustness:** The paper does not explicitly discuss the robustness of PR to noise or distortions in the data. Given that real-world data can often be noisy or imperfect, discussing the limitations of PR in handling such situations would be beneficial.\n\nRegarding the broader societal impacts, the paper does not directly address this point. While novelty detection in machine learning can have multiple positive societal impacts, like enhancing medical diagnosis or improving safety in autonomous driving, it might also have potential negative impacts:\n\n1. **Privacy concerns:** Improving the ability of machines to identify novel information could potentially lead to privacy concerns, as more accurate models might be misused for surveillance or unauthorized data collection.\n\n2. **System Misuse:** In high-stakes applications, an over-reliance on automated novelty detection systems that might still make mistakes could lead to serious consequences.\n\nThe authors could improve their paper by addressing these points, possibly in a new section dedicated to limitations and broader societal impacts. This would help readers appreciate the full context of the research, including its potential downsides.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "**Originality:** The paper presents a novel method called Projection Regret (PR) to address the problem of background bias in novelty detection using diffusion models. This is a creative combination of existing ideas, especially the use of perceptual distance and recursive projections to cancel out dominant background information. The proposed alternative perceptual distance metric using decoder features of the pre-trained diffusion model also adds to the originality.\n\n**Quality:** The research appears to be of high quality. The authors have carried out extensive experiments to demonstrate the effectiveness of their proposed method. They have also compared PR with other existing methods for novelty detection, showing that it outperforms them by a significant margin.\n\n**Clarity:** The paper is well written and structured. The authors clearly outline the problem of background bias in novelty detection, explain the concept behind PR, and provide detailed explanations of their experimental setup and results. They also do a good job of explaining complex concepts in an understandable way.\n\n**Significance:** The significance of this work lies in its potential to improve the accuracy of novelty detection in machine learning, which has broad applications in many fields, including medical diagnosis, autonomous driving, and forecasting. By addressing the issue of background bias, the proposed PR method could enhance the reliability and safety of deep learning applications. Additionally, the proposed perceptual distance metric could serve as a useful tool for researchers working on similar problems in the future.", "weaknesses": "While the paper presents a significant contribution to the field, there are a few areas where it could be improved:\n\n**1. Evaluation metrics:** While the paper does an excellent job of comparing with existing methods and demonstrating the superiority of PR, it might benefit from more diverse evaluation metrics. At present, the paper primarily focuses on detection accuracy (AUROC). Incorporating additional measures such as precision-recall curves or F1 scores could provide a more comprehensive evaluation.\n\n**2. Real-world applications:** The paper demonstrates PR's effectiveness using standard datasets (CIFAR-10 vs CIFAR-100 etc.), but it would be valuable to see how PR performs in real-world scenarios. It is crucial to know how the method copes with the complexity and variability found in actual application scenarios including medical imaging or self-driving car data.\n\n**3. Deeper exploration of background bias:** Although the authors have proposed an innovative solution to tackle the issue of background bias, they could delve deeper into this problem. Understanding the nature of this bias, its origins, and why it is particularly problematic for diffusion models can further enhance the paper's impact.\n\n**4. Computational efficiency:** While the authors mention that PR can be calculated efficiently via a consistency model, specific data about computational requirements, time complexity, and scalability is missing. Providing these details can help readers assess whether PR is suitable for their specific use-cases, especially those that require real-time processing or deal with large-scale datasets.\n\n**5. Robustness analysis:** The paper could benefit from a detailed robustness analysis of PR against various types of noise and distortions. This would allow potential users to understand the limits of the approach and potential pitfalls in practical applications. \n\nBy addressing these points, the paper could strengthen its contributions and appeal to a broader audience within the machine learning community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear reviewer 9KBe,\n\nWe sincerely thank you for your helpful feedback and insightful comments. In what follows, we address your concerns one by one.\n___\n**[Q1]** Could you provide more details on the computational requirements of PR? For instance, what is the time complexity, and how does it scale with the size of the dataset?\\\n**[A1]** As shown in Algorithm 1, our algorithm (Projection Regret or PR) performs 3 forward computations of the consistency model. Therefore, the time complexity is linear to the size of the dataset. In a real-time setting, our algorithm takes 0.395s/sample in the CIFAR-10 dataset and the consistency model version of LMD [1] takes 0.705s on the same dataset.\n___\n **[Q2]** Would PR be equally effective in real-world applications where data might be more complex or noisy compared to standard datasets like CIFAR-10 or CIFAR-100?\\\n**[A2]** We scale Projection Regret into a large-scale LSUN domain where the dataset is more complex. To be specific, we set the bedroom class as the in-distribution (ID) dataset and set bridge/church/classroom classes as the out-of-distribution (OOD) dataset. We summarize the OOD detection result in Table 8 (see Global Response PDF). Our method outperforms the second-best method, LMD [1], with a significant gap. This result shows the potential of PR in real-world applications.\n___\n**[Q3]** Could you delve deeper into the issue of background bias? Understanding its origins and why it affects diffusion models specifically could strengthen the paper.\\\n**[A3]**  The background bias issue originates from the fact that we do not have a perfect distance metric that only captures the semantic difference between two images. While the LPIPS distance metric [2] mainly used in our work captures such a semantic change better than $\\ell_{2}$ distance, it often captures other information (e.g., background), especially when the main object is relatively small (see L49-L50). Furthermore, when we apply the metric on various datasets that significantly differ from ImageNet, e.g., SVHN or LSUN, it is not likely that the LPIPS distance metric fully captures the semantic change between the input data and its projection. In such scenarios, the distance metric could be biased by the background information as shown in Figure 1(b). Hence, we think the background bias issue is problematic due to an imperfect distance metric. \n\nTo reduce the background bias, we propose Projection Regret using the unique characteristic of diffusion models: diffusion-based projection can map any OOD sample to an ID one with a similar background. Therefore, we rather regard diffusion models (or consistency models) as the remedy to compensate for the imperfect distance metric. Furthermore, it is evident that OOD detection methods based on alternative generative models, e.g. GANs or GLOW, are also vulnerable to OOD with a similar background (see CIFAR-10 vs CIFAR-100 OOD detection results in Table 2). Hence, we think background bias is not limited to the diffusion model but a common problem of unsupervised OOD detection methods.\n___\n**[Q4]** Have you investigated the robustness of PR against various types of noise and distortions? If not, do you anticipate that the method would be robust against such challenges?\\\n**[A4]** We appreciate the reviewer for the motivating suggestion. We further test PR’s robustness under the corruption of the in-distribution data. To be specific, we set **corrupted** CIFAR-10 data [3] as pseudo-in-distribution data and perform an OOD detection task against **uncorrupted** SVHN/CIFAR-100/LSUN/ImageNet as OOD datasets. For the corruption strategy, we use (shot noise, defocus blur, fog, and elastic transform) for the representative strategy of (noise, blur, weather, and digital) in varying corruption levels of (1,2,3). We provide the visualization of the corrupted data in Figure 5 (see Global Response PDF). \n\nWe present the result in Table 9 (see Global Response PDF). As corruption intensity increases, the OOD detection performance decreases. While strong corruptions (e.g. shot noise, fog) deteriorate OOD detection performance significantly, our method is relatively robust to weak corruption (e.g. defocus blur). Hence, our method would be at least robust to imperceptible corruption.\n___\n**[Q5]** Consider incorporating additional evaluation metrics such as precision-recall curves or F1 scores for a more comprehensive evaluation of PR's performance.\\\n**[A5]**  Widely used popular metrics on OOD detection (TNR at 95\\% TPR, detection accuracy, AUPR_IN, and AUPR_OUT) are included in the CIFAR-10 vs (SVHN, CIFAR-100, LSUN, ImageNet) OOD detection experiments in Table 10 (see Global Response PDF). The result is consistent and we will add the full results in the final manuscript.\n___\n**[Q6]** Improving the ability of machines to identify novel information could potentially lead to privacy concerns, as more accurate models might be misused for surveillance or unauthorized data collection. In high-stakes applications, an over-reliance on automated novelty detection systems that might still make mistakes could lead to serious consequences.\\\n**[A6]** We appreciate the reviewer for elaborating on the limitations of our method applied in real-life scenarios. We will incorporate this discussion into the limitation section in the final draft.\n___\n**References**\\\n[1] Unsupervised out-of-distribution detection with diffusion inpainting, ICML 2023\\\n[2] The unreasonable effectiveness of deep features as a perceptual metric, CVPR 2018\\\n[3] Benchmarking neural network robustness to common corruptions and perturbations, ICLR 2019\n"}, {"review_id": "gTh1mGghBu", "reviewer": "Reviewer_SZQH", "summary": "Recent methods have mainly utilized the reconstruction property of in-distribution samples to detect OOD by diffusion models. However, they often suffer from detecting OOD samples that share similar background information to the in-distribution data. Based on the observation that diffusion models can project any sample to an in-distribution sample with similar background information, the paper proposes Projection Regret (PR), an efficient novelty detection method that mitigates the bias of non-semantic information. To be specific, PR computes the perceptual distance between the test image and its recursive diffusion-based projection to detect the abnormality.", "questions": "see above", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is written well and is easy to understand.\n2. The studied problem is very important.\n3. The results seem to outperform state-of-the-art.", "weaknesses": "1. From the introduction, the projection cannot change the background information too much, which seems to be harmful on near-OOD detection, such as Cifar10 vs Cifar100. Since c100 and c10 has a similar background, then wouldn't this projection create a barrier further for solving the OOD detection problem?\n2. The design and hyperparameter search (including the ensemble) seems to be very important on the results. How do the authors get the best configuration on a new ID/OOD pair? Is there any sensitivity analysis on more pairs? i.e., doe the current Figure 3 generalize to most ID/OOD pairs?\n3. The computational budget is a little bit concerning in the algorithm. Is there any concrete comparison or explanation?\n4. More results on large-scale benchmarks (i.e., ImageNet) and large (diffusion) models seem to be more meaningful for practical concerns and interpretability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer SZQH,\n\nWe sincerely thank you for your helpful feedback and insightful comments. In what follows, we address your concerns one by one.\n___\n**[Q1]** From the introduction, the projection cannot change the background information too much, which seems to be harmful on near-OOD detection, such as Cifar10 vs Cifar100. Since c100 and c10 has a similar background, then wouldn't this projection create a barrier further to solving the OOD detection problem?\\\n**[A1]** We clarify that our projection of an out-of-distribution (OOD) sample changes its semantics from OOD to in-distribution (see Figure 1(a)). Since our detection score is based on the semantic distance metric between a test image and its projection, our algorithm will output a high abnormality score on the OOD dataset even if the background information is similar. \n\nThis claim is evidenced by our empirical results: our algorithm significantly outperforms baselines on CIFAR-10 vs CIFAR-100 (see Table 1). To further address your concern, we additionally experiment with an additional OOD benchmark, the ColorMNIST dataset [1,2] where spurious OOD samples have **the same background statistics** (i.e., same background colors) but different semantics (i.e., different digits) compared to the in-distribution data. In this dataset, our algorithm achieves near-perfect OOD detection performance as shown in Table 6 (see Global Response PDF). These experimental results verify that our method can detect OOD samples that have similar backgrounds to the in-distribution samples.\n___\n**[Q2]** The design and hyperparameter search (including the ensemble) seems to be very important in the results. How do the authors get the best configuration on a new ID/OOD pair? Is there any sensitivity analysis on more pairs? i.e., doe the current Figure 3 generalize to most ID/OOD pairs?\\\n**[A2]** We choose the hyperparameters for each in-distribution (ID) data without any access to out-of-distribution (OOD) samples as described in footnote 1 on page 5. To be specific, we consider rotated in-distribution samples as synthetic OOD ones and choose the best hyperparameters in this synthetic OOD detection task. We find that the hyperparameters selected from the synthetic task are well generalized across other OOD datasets, as we verified in our experiments. Therefore, for a new ID dataset, one can find the best configuration itself, and the configuration can be used across various OOD pairs.\n\nFor your information, we further provide the sensitivity analysis of Projection Regret on more ID/OOD pairs: CIFAR-10 vs (SVHN, LSUN, ImageNet) datasets in Figure 6 (see Global Response PDF). Somewhat interestingly, the hyperparameter chosen by our synthetic OOD detection task ($\\alpha$=9, $\\beta$=8) also achieves the best performance on LSUN and ImageNet datasets or a small gap (0.005 AUROC) against the best performance on the SVHN dataset. We further improve this by ensembling across multiple configurations and the ensemble outperforms the best-searched hyperparameter (see Table 3).\n\nFinally, as the reviewer asked about the generalization property of Figure 3 to other ID/OOD pairs, we also test our projection with varying timesteps. As shown in Figure 6 (see Global Response PDF), the trend in the LSUN and ImageNet OOD datasets shows a similar trend to CIFAR-100 and SVHN shown in Figure 3. Therefore, we think the hyperparameters are not sensitive to ID/OOD pairs, which is a merit of our framework.\n___\n**[Q3]** The computational budget is a little bit concerning in the algorithm. Is there any concrete comparison or explanation?\\\n**[A3]** Our algorithm is efficient because it only requires three projections and each projection can be computed by one forward pass using a consistency model [3] that enables one-shot generation, as described in Algorithm 1. As a result, even with our ensemble technique described in L162-L167, our algorithm is 1.8x faster than the second-best baseline, LMD [4] (0.395s vs 0.705s per sample, respectively).\n___\n**[Q4]** More results on large-scale benchmarks (i.e., ImageNet) and large (diffusion) models seem to be more meaningful for practical concerns and interpretability.\\\n**[A4]** To show our algorithm’s scalability in large-scale datasets with high-resolution images, we construct an OOD detection task on the LSUN domain. Specifically, we set the bedroom class of LSUN as an in-distribution dataset and apply the pre-trained consistency model to detect bridge/church/classroom OOD datasets. We also experiment with LMD as a baseline. As reported in Table 8 (see Global Response PDF), our method significantly outperforms LMD by a large margin across all large-scale LSUN OOD detection tasks. We also observe that our method achieves better AUROC in relatively far OOD datasets (bridge, church) against near OOD datasets (classroom). Hence, this additional experimental result shows the scalability and the potential of practical applications.\n___\n\n**References** \\\n[1] Invariant risk minimization, arXiv 2019\\\n[2] On the impact of spurious correlation for out-of-distribution detection, AAAI 2022\\\n[3] Consistency Models, ICML 2023\\\n[4] Unsupervised out-of-distribution detection with diffusion inpainting, ICML 2023\n\n"}]

Author rebuttal:
Dear reviewers and ACs,

We sincerely appreciate your valuable time and effort spent reviewing our manuscript.

As reviewers highlighted, our work aims at an important problem **(Reviewer ZGie, 9KBe, SZQH)** with an interesting/novel method **(Reviewer ZGie,9KBe)**, strong empirical results **(Reviewer mygo, ZGie, 9KBe, SZQH)**, and a well-written/easy-to-follow writeup **(Reviewer mygo, 9KBe, SZQH)**.

We appreciate your constructive comments on our manuscript. We have carefully addressed the comments with the following additional discussions and experiments: 
- **[Reviewer ZGie, SZQH]** Performance of Projection Regret on spurious OOD datasets with the same background information (Table 6). 
- **[Reviewer ZGie]** Performance of Projection Regret in CIFAR-10 one-class classification benchmark (Table 7). 
- **[Reviewer 9KBe, SZQH]** Performance of Projection Regret applied to larger diffusion models with larger dataset size (Table 8). 
- **[Reviewer 9KBe]** Robustness analysis on Projection Regret against common corruptions (Table 9, Figure 5). 
- **[Reviewer 9KBe]** Performance of Projection Regret measured in alternative metrics (Table 10). 
- **[Reviewer SZQH]** Sensitivity analysis of Projection Regret and the motivation experiment extended to multiple OOD datasets (Figure 6). 

We hope our response sincerely addresses all the reviewers’ concerns.

Thank you very much.

Best regards, \
Authors.


Meta-review: The proposed work on novelty detection by reducing background bias has received no strong support for acceptance, but at the same time none of the reviewers is against acceptance. AC has read the paper as well and believes the authors have adequately addressed the reviewer concerns on background bias, concerning computational efficiency of the diffusion-based approach, and a lacking experimental justification on real-world datasets. While the work may leave something to be desired, for example a more thorough qualitative analysis of background bias, e.g. using the datasets suggested by Reviewer mygo, the current paper is of sufficient interest. Discussion with the SAC further strengthened AC in recommending acceptance, given that all promised discussions and experiments will be added to the camera-ready version.
Decision: Accept (poster)