Out-of-distribution Detection with Diffusion-based Neighborhood
Authors: Luping Liu
Yi Ren
Xize Cheng
Zhou Zhao
Venue: Submitted to ICLR 2023 (2023)
Keywords: OOD detection
diffusion model

Abstract: Out-of-distribution (OOD) detection is an important task to ensure the reliability and safety of deep learning and the discriminator models outperform others for now. However, the feature extraction of such models must compress the data and lose certain information, leaving room for bad cases and malicious attacks. However, despite effectively fitting the data distribution and producing high-quality samples, generative models lack suitable indicator scores to match with discriminator models in the OOD detection tasks. In this paper, we find that these two kinds of models can be combined to solve each other's problems. We introduce diffusion models (DMs), a kind of powerful generative model, into OOD detection and find that the denoising process of DMs also functions as a novel form of asymmetric interpolation. This property establishes a diffusion-based neighborhood for each input data. Then, we perform discriminator-based OOD detection based on the diffusion-based neighborhood instead of isolated data. In this combination, the discriminator models provide detection metrics for generation models and the diffusion-based neighborhood reduces the information loss of feature extraction. According to our experiments on CIFAR10 and CIFAR100, our new methods successfully outperform state-of-the-art methods. Our implementation is put in the supplementary materials.

Under review as a conference paper at ICLR 2023
OUT-OF-DISTRIBUTION DETECTION WITH DIFFUSION- BASED NEIGHBORHOOD
Anonymous authors Paper under double-blind review
ABSTRACT
Dut-of-distribution (OOD) detection is an important task to ensure the reliability and safety of deep learning and the discriminator models outperform others for now. However, the feature extraction of the discriminator models must compress the data and lose certain information, leaving room for bad cases and malicious attacks. In this paper, provide new assumption that the discriminator models are more sensitive to some subareas of the input space and such unfair treatment creates bad cases and overconfidence areas. Under this assumption, new detection methods and indicator scores. For detection methods, we introduce diffusion models (DMs) into OOD detection. We find that the denoising process (DDP) of DMs also functions as a novel form of asymmetric interpolation, which is suitable to enhance the input and reduce the overconfidence areas. For indicator scores, we find that the features of the discriminator models of OOD inputs occur sharp changes under DDP and use the norm of this dynamic change as our indicator scores. Therefore, we develop a new framework to combine the discriminator and generation models to do OOD detection under our new assumption. The discriminator models provide proper detection spaces and the generation models reduce the overconfidence problem. According to our experiments on CIFARIO and CIFAR 100, our new methods successfully outperform state-of-the-ard methods. Our implementation is put in the supplementary materials.
design
INTRODUCTION
-ittof-distribution (OOD) detection is an important task for deep models that helps the models determine their capability boundary and keep them from being fooled by OOD data. It has strong connection with many real-world machine-learning applications, such as cybersecurity (Xin et al. 2018), medical diagnosis (Latif et al. 2018 Guo et al. 2020) and autopilot (Geiger et al. 2012)) The existing methods for OOD detection can be generally categorized into discriminator based and generation based methods. The discriminator-bet based methods (Wang al. (2022) use the logit the feature space to do that. The generation- based methods (An & Cho 2015 Nalisnick et al. 2019) use the reconstruction difference in data space or density estimation in latent space to do that.
The diseriminator-based methods can extract useful features and make the detection faster and better most cases. However, such extraction and compression lose some information and leave room for bad cases and malicious attacks (Goodfellow et al. 2014 Amodei et al. 2016). The seneration-based methods can capture the whole data distribution but lack effective indicator scores to compete with the SOTA discriminator-basee. methods, partly because of the curse of dimensionality. Previous works mostly concentrate on solving these challenges using only one kind of model. For discriminator based methods, |Wang et al. (2022) combine the information from both features and logits. [Sehwag et al. (2020) use self-supervised learning to improve feature extraction. For generation-bant Nalisnick et al. (2019) use the typicality set design better indicator scores. Jiang et al. (2022 use statistical methods in the latent space, such as the Kolmoggrov-Smmioo test.
methods,
In addition to overcoming the problems of each kind of model by itself, we find that generative and discriminative models can be combined and solve each other's problems. We provide new assumption that the discriminator models are more sensitive to some subareas of the input space to explain the existence of bad cases and overconfidence areas. To solve this problem, we get inspiration from water quality detection in the real world. They use some fixed detectors and make the water
explain our moti- vation

---

Under review as a conference paper at ICLR 2023
flow by stirring. Then these detectors monitor large area of water. For OOD detection, the discriminator models also concentrate on some fixed subareas of the whole input space. Therefore, also want to "stir" the input to improve detection accuracy and reduce overconfidence areas. We find that generation models are good choice to be the "stirring" operators. Under such operators, the results of InD data remain normal at all times and that of OOD data expose anomalies.
To design suitable generation strategies that can enhance discrim- inator models, we introduce diffuusin models (DMs), which play an important role in generation models, into OOD detection. DMs have created many state-of- seaee-of-hh--art art generation results, including (Vahdat et al.] 2021;00 Ho et al.) (022).. We dive into the structure of DMs and find that the diffusion denoising process (DDP) of
Ingut
Diffusion model
Nossv magges
DMs can be an ideal choice for the "stirring" operator we men- tioned above. Because can adjust any level of feature space and provides tools to keep the adjustment under control by using the denoising and interpolation properties. Such an operator "stirs" the input and needs be resampled several times to make the result convergent, which builds a neighborhood of input data,
Difi Difiinechorn
called the diffusion-based neighborhood (DiftNB). According our above analysis, the feature of OOD data explores anomalies, which means that the feature can change sharply. We can detect such anomalies by simple Euclidean distance between several different features of DiANB. Our pipeline is in Figure !
Input
neighbors
Feature space
L2 loss
InD or 00D
We choose ten representative methods to compare with our meth- ods on two representative datasets: CIFARIO and CIFAR100. According to our experiments, our new methods outperform existing models and methods in most cases. Our work has the following contributions:
Figure 1: The pipeline of our de- tection method.
We provide a new assumption that the discriminator models are more sensitive to some subareas for OOD detection. We analyze why it causes the overconfidence problem and how to solve it.
We find that the diffusion denoising process of invertible diffusion models a novel kind of asymmetric interpolation, which can keep the InD data relatively unchanged and provide tools to control the direction of the denoising process.
We develop a framework to combine the discriminator and generation models, which uses a ResNet extract features and the diffusion denoising process of a diffusion model to reduce overconfidence areas. Our methods get competitive OOD detection results with SOTA methods.
2
BACKGROUND
In this section, we first introduce existing methods for OOD detection. Then, we show the develop- ment of diffusion models related paper. Because of the limited space in the main paper, more related works about diffusion models can be found in Appendix A.I
2.1 OUT-OF-DISTRIBUTION DETECTION
OOD detection is an important task that can help neural networks to determine their capability boundary. More specifically, let X = {zi..... In
be group of images from the in-distribution (InD) p. We want build detector that f( =n) 1, Vi, p(x,) ) 2 a and f 2n,) = 0, p(x:)) & Here, controls the decision boundary. When we get another group of data Y Y= [m1 }, we decide whether this group from InD p or an unknown distribution q based on the results of f. If n 1, this is pointwise OOD detection, and if > 2, this is group OOD detection. In general, the existing OOD detection methods can be categorized into discriminator-bue based and generation- methods.
Discriminator based methods design indicator scores based on the output of discriminator models. Some methods can be used without modifying the model. ODIN (Liang al. 2018) uses temperature scaling and the softmax results to detect OOD samples. ViM (Wang et al., 2022) combines the information of features and logits. KNN (Sun et al. 2022) includes the kth nearest neighbor of the
2

---

Under review as a conference paper at ICLR 2023
input in feature space into the detection process. Some methods try to improve the detection ability in the training process. G-ODIN (Hsu et al., 2020) designs a new loss function. ConfGAN (Sricharan & Srivastava 2018) generates OOD data using GANs to help the discriminator models to determine the boundary. PixMix Hendrrycks et al. 2022) uses data augmentation to improve the detection results.
SSI> (Sehwag al. 2020) uses self-supervised learning improve feature extraction.
Generation-based methods use the reconstruction difference in the input space and the density estimation in the latent space to do OOD detection. [An Cho (2015) use the reconstruction ability of VAES. Some methods assume that the generation models can reconstruct the in-distribution data better. Some methods use the Distribution transformation capability of generation models and transfer the input distribution into simple Gaussian distribution. The likelihood of the input becomes direct choice, but Nalisnick et al. (2018) finds that OOD data can also locate the high-likelihood area. Nalisnick et al. (209)) find the Inl data is concentrated in the typical set instead of the high likelihood area and design new methods using the typical set. [Serrà et al. (2019) find that we can use input complexity to correct the bias of likelihood. In addition to the likelihood, many existing statistical methods can detect whether a distribution obeys standard Gaussian distribution, ZZhagg et al. (2020) uses KL-divergence to detect OOD data. Jiang et al. (2022) nonparametric statistics method called the Kolmogorov-Sminn -iiinov test.
2.2 DIFFUSION MODEL
Classical diffusion model DMs build a transformation from Gaussian distribution image distri- bution through multistep denoising process. Given a data distribution (xu)), the diffusion process satisfies Markov process as following Ho et (2020):
T ((F1T/20)) = II N NNNV(11 B,xr-1,
( = N(Va, (1 ā.).).
Here, T 1000, which is the max iteration step. 3r E (0, 1), which controls the speed of adding noise. Additionally, a, 1 3r, a₂ I[ 0% Bi 1a The objective function is defined by:
T- + yarl1-ā. and Br
1-4u-1
T-a,
87
-le colvarto + v1 ūge, !)[??
4 = Ezg.e
ac(1
a.)
Here, is an estimate of the noise e. After we get well-trained eg, according to Song et al. (202 la). the denoising process Denoising Diffusion Probabilistic Models (DDPMs) and Denoising Diftusion Implicit Models (DDIMs) satisfies:
Ft
VI ()e
V
+ v ū:-8 of t) deee-
Vāa
Here, 5 is the iteration step size. If o; equals one, Equation (3) represents the denoising process of DDPMs; if o, equals zero, this equation represents the denoising process of DDIMs. In Appendix we will further describe how to make the iteration of the diffusion model fast and invertible.
A.3
Classifier-free guidance Ho Salimans (021) show simple and effective way generate
conditional samples called classifier-free guidance. It adds a condition embedding e into eg in the training process and changes the final estimation of noise as:
Eo (Zt, c) (1 w)e t) (x)(
(4)
Here, w is the guidance weight, which controls the balance between realness and diversity.
3 DIFFUSION-BASER NEIGHBORHOOD FOR OOD DETECTION
In this section, we first theoretically define our task, which to design proper additional operators for fixed detector. We assume that the detector only concentrates some subarea of the input
3

---

Under review as a conference paper at ICLR 2023
space. Then we use toy example to show how to design proper additional operators and find that reconstruction operators are potential candidates for general cases. After that, we use invertible diffusion models and the diffusion denoising process (DDP) to develop new asymmetry interpolation to satisfy the requirements of the additional operators. Then we use a new toy example to show what will happen under DDP for more complex situations and use the norm of the dynamic change under DDP as our detection score. Both DDP and our new score can be applied to any black box detector f and we introduce classifier-free guidance for the multi-class condition. In the end, we provide our general methods and visualize their actual effect when the detector is pre-trained ResNet18.
3.1 THEORETICAL SETTING
We first transfer the OOD detection task and the idea behind water quality detection mentioned in Section D into a theoretical problem. Following the definitions in Section 2. the detector is function f that can be an analytic function or a neural network. The detection problem is that given a distribution p
certain space U, let {a E Upp(x) > of be W and want to find detector that satisfies f- (1) = W exactly. The idea behind water quality detection to use fixed detector and make water flow by stirring. The fixed detector means that the detector may treat bad cases as normal and (1) is strictly bigger than W. The stirring operator means that we can add additional operators to help the detection. Therefore, we can define a group of operators {gr satisfying if W and Oiet, (f g.) (1) C f The first condition means that all fo It can identify the InD data correctly and the second condition means that the Over, (f o g.) (1) becomes smaller by adding more 9;- Now we introduce the theoretical version of our problem:
gz) = f()
Given a fixed subarea W C C. a single value function f satisfying W fI) how can we design operators {9: (ii), to minimize Diet, (f 9.) (1)?
For our task. f is a neural network and the difference between (1) and W is the overconfidence area. Here, we assume all training data belong to one class. To design proper {9i be,, we need to locate what causes such an overconfidence problem. We find that discriminator models are similar to the fixed water detector and only concentrate on some subareas of the whole input space. Here, we provide the main assumption in our paper:
For discriminator model, different channels concentrate on different subareas of the input space. Some subareas are more important than others.
Here, we emphasize that even in the class-related area, some subareas are more important than others. To show why this assumption is reasonable, We dive into the structure of discriminator model and find that the linear operators, including the convolutional layers and the following pooling layers, play an important role, According to the singular value decomposition, every linear operator LA : R" - R", < n, can be transferred to restriction operator LA under proper change of basis and S is a m-ddimensional subspace. The input data is bounded in our task, so the subspace becomes a bounded area in the input space. Different linear kernels in layer concentrate on different S; and the total sensitive area of this layer is Usetg S₁. Some subareas can be included by more S., which are more important than the remaining area.
3.2 SINGLE-LLAER MODEL.
The whole input space may be covered by the support area of all kernels S; and some areas are covered more times than others. For simplification, we can transfer the problem to a restriction operator. The key idea is to set a threshold N and mask the area that is covered by less than N support areas S. Therefore, the data pass through layer of neural network as it had first passed through a masking and some information is then ignored.
make the assump- tion clear
For the convenience of theoretical analysis, we choose a simple neural network as f. which only contains one convolutional block and full connection layer. The support area of each convolutional kernel in the input space is S, and the total support area is Users S.. Here, we simplify it to a simple restriction operator, namely, each S; is a square and all 5; are disjoint with each other. Our input space is the image space and our training set contains only one element zero (the empty image).
Ty. Ts is the index set of {ø=} and (St) respectively. They can he finite or infinite.
A restriction operator is that Vf. fls(x) f(x if x E S, else fls(x) 0. S is called the support area.
4

---

Under review as a conference paper at ICLR 2023
Input
Output
Input
Output
Not zero
Not zero
Move
Zero
Mix
Not
Zero
Recon
Not zero
Figure 2: The left side shows the detection process of toy example and bad under this setting. The right side provides three different operators to correct the result of this bad case.
Our InD is just uniform distribution in spherical neighborhood of zero and target is to detect whether a new input is InD or OOD, namely, detect whether it falls in the neighborhood of zero. We show this toy example the left side of Figure and the third row shows bad under this setting, which looks like zero after the mask operator. On the right side of Figure 2 we use three kinds of additional operators moving, mixing and reconstruction solve this problem.
The first two can perfectly solve the toy example in Figure and can be extended to any restriction operator support different S. We put the proof in Appendix A.5 The key idea of these solutions is that they move or mix information from one place to another. Then the detector f can get the whole information from small support area S. Such ideas can generalized to more complex cases and the third reconstruction operator in Figure Figurepplays central role here. We find that the reconstruction strategy successfully contains the moving operator (the face of the cat moves the left and down), the mix operator (the boundary of each small box becomes unclear) and the semantic level moving operator (the color of the cat is lighter). We can move or mix the information at both the semantic and pixel level instead of only the pixel level now. Therefore, provides a candidate {9t with potential advantages for real detectors f. such as ResNet.
3.3 DIFFUSION DENOISING PROCESS
Such reconstruction strategy can be finished with any kind of generation model. However, this strategy also has its weakness. Figure we can detect the OOD examples more precisely using a reconstruction operator. However, we also need to satisfy the condition W f namely, to keep the pure white picture from being dirty. This challenge tells that we need to add more control to the reconstruction strategy. In the following, to solve this challenge. We dive into the structure of diffusion models and show that DDP a kind of interpolation under the invertible condition. Therefore, DDP combines the benefit of denoising and interpolation and provides many powerful tools to control the reconstruction process. Due to space limitations, put the analysis of enhancing invertibility in Appendix A.3 and directly analyze interesting applications of DDP under the invertible condition.
introduce the diffusion-denoissng process (DDP)
Interpolation Many previous pa- pers use diffusion models to inter- polate inputs, but our method asym1 different from the existing sym- asym2 metric one using spherical linear
0.1
0.2
0.3
0.5
0.6
0.7
0.9
interpolation (Shoemake) 1985). sym
We combine the invertible diffu-
sion models and the diffusion de- Figure 3: Different interpolation results using two fixed images. noising process to get new asymmetric interpolation. Let us assume that To is an image and is Gaussian noise, which is the reverse of an image æ1, and use æp and to get Zt. The diffusion- denoising process shown in Algorithm] and we define DDP as (xx,, t, 0). When equals zeros, do not add any noise to the images, remove the total image 2n and only leave the noise e, we will do a full denoising process to this noise. Because this denoising process is invertible, can get the original image T1. Therefore, the outputs
get the original image Zp. and when t equals T.
5

---

Under review as a conference paper at ICLR 2023
of DDP gradually change from Ty to ø1- Figure shows the three different interpolation results. The first row uses the cat as 2o and the second row uses the car as 2o- We can find that 2o can be better preserved with respect to the symmetric interpolation in the third row.
DDP is a kind of reconstruction strategy, in the meanwhile, it is also a denoising operator and an interpolation operator. That DDP is denoising operator means that it keep the InD data relatively unchanged and pull the OOD data to the high-density area of InD, which is a distribution- sensitive property. That DDP an interpolation operator means that even if we cannot reconstruct the input perfectly, we can at least control the direction of change. Both these properties give us more possibilities to control DDP and solve the above weakness. Finally, we get the following property:
The invertible diffusion-denoising process is an asymmetric distributionn-sessiine interpolation.
3.4 MULTI-LAYER MODEL
In the above section, we show how to design (gi and our choice invertible DDP. Here, we show the effect of DDP in more complex example to design a proper detection score. We consider the last convolutional layer of multi-layer model. In this setting, we face a major change. The imput of this layer has many channels. To be more theoretical, the input feature is in mmxxx the input feature space is mmxi and the input feature is combination of e celements IR m Each of them is the output of a previous convolutional kernel. Therefore, single input becomes points in the feature space instead of a single point.
make the assump- tion clear
For the convenience of theoretical analysis, we abstract the whole input space into 2-dimensional space and the ideal fea- ture space belonging to the input data is the light blue square. The point in each image represents e = 6 features of a single input and the corresponding arrow shows the movement of these features under DDP. We use the dark blue area to show the features used by the convolutional kernels of the last con- volutional layer. For better visualization, the output of each convolutional kernel is a single value, the number of points N that fall in its sensitive area. Again, we assume that all the sen- sitive areas form a mask. This example is just a simplification of the process to compute confidence.
Feature (low)
Confidence (high)
For a normal InD input (the first row), its features are uniformly distributed in the feature area, which ensures that N maintains a dynamic balance under the perturbation of DDP in the feature space. For normal OOD input (the second row), DDP pulls them to the high-density area and N increases at the same time. For more challenging OOD data (the third row), is not InD and its features are relatively sparse in the ideal feature set. However, all these features happen to fall in the dark blue area at the same time by coincidence or man mane, which causes the over-confidence problem. Such imbalance breaks the dynamic balance between the dark blue area and the remaining and causes rapid decline of N under the perturbation of DDP. Therefore, all two kinds of OOD input can be successfully detected by the change of confidence. These analyses can be extended to any middle layer by replacing the single value output with vector output.
Figure 4: The change in different levels of feature space under the dis- turbance of DDP.
3.5 DIFFUSION-BASE NEIGHBORHOOD
The first toy example tells us that we can use DDP to enhance the detection process and the second toy example tells us what will happen in feature spaces under DDP. Detecting the dynamic change of confidence still requires the output is single-value. We can generalize this idea to detect the change in any feature space of the model by using the norm of the change the score. After we remove the limitations of the form of the output, both DDP and the dynamic change score can be applied to any ResNet.
solve the gap
black box detector f. such
The final step is to take multiple categories into account and avoid category migration for InD input. Here, we need the help of the interpolation property and there exist two choices. First, search

---

Under review as a conference paper at ICLR 2023
Algorithm 1 Diffusion-denooisin process
Algorithm 3 Conditional neighborhood
Input: Images Zo, generative interval [0, 7]. gen-
Input: Images timestep t
erative gap for = T. T....
y = FC(ResNet(z))
do
2: (N
2:
#i = var (Tt vI (xc
3: Pnoise = v I + vI dee
3:
c= c= co(2s.))
4: Wneighhor (Innise y/ 1,
4: It-5 vo + vI ac-56
5: Return Tineighhor
5: end for
6:
Return 200
Algorithm OOD detection
Algorithm 2 Unconditional neighborhood
Input: Images æ; diffusion-based neighborhood
Fneighhor fea = ResNet(<)
Input: Images I, timestep
1: Iknn KNN(2, ftraining data))
2: Icancighbor ResNet((Fnnighboi
2: (( 0,
3: score = Σ fea - feassighbor
3: Tnoise vaxz + VI Cref
4: if score >> 5 then
4: Tncighhor = ④ (Zacise t, 0)
5:
Return OOD
5: Return Zhoiihhorr
end if
1.0
0.6
InD
0.0
-05
0.0
0OD
-02
-044
-2
-0.2
0.0
0.2
0.4
0. 0.
(b)
(c)
Figure 5: The influence on the different levels under DDP, including the image space, the first and the fourth feature spaces of the ResNet18. 88 In Fig (a), the first column is the original input and the remaining is the DifINB them. In Fig (b, c), the blue and green arrow is the feature change of InD data and OOD data respectively under DDP after dimensionality reduction using PCA. Fig (b) and (c) show the results the first and the fourth feature spaces of a ResNet18 respectively.
the kth nearest neighbor of the input in the input space and generate the corresponding noises of them. We interpolate these noises with the original input using DDP. Another more interesting choice is that we can train a conditional diffusion model, and fix the class condition to the class of inpu Then all noise are corresponding to the images in the same class. We can interpolate the input with any noise, instead of searching for it first. What's more, we can choose several noises for each input and all the results of DDP become a neighborhood of the input. We call this neighborhood the diffusion- based neighborhood (DifINB). Then we use discriminator model to the dynamic change in the feature spaces and determine the OOD samples based on that. We put our algorithm in Algorithm 4
In Figure we show the DifNB of different inputs using DDP and the influence of DDP on the different feature spaces of a pre-trained ResNet18. The DiffNB has more obvious semantic change for OOD input. For example, rickshaw becomes horse in Fig[5 On the other hand, the semantic information DiftNB of the InD data is relatively unchanged. Correspondingly, We can find that change in low-level features (the first feature space, in Fig 5b is similar but the change in high-level semantic features (the fourth feature space. in Fig 5c becomes small when the input is InD and relatively big when the input is OOD.
4 EXPERIMENT
In this section, first show the detailed setting of our experiments. Then we offer OOD detection results, including our method and existing representative methods. After that, we provide ablation study results to show the contributions of each item and hyperparameter in new scores.
When the label is unavailable, we the discriminator model generate pscudo label.

---

Under review as a conference paper at ICLR 2023
Table The AUROC results of different methods. We train the models on the training data for 160k epochs and test the results on the test data. We use the conditional version method and the guidance weight is 2. We set the disturbance degree = 300, the repeat size and use the logic space as our detection space. The higher results are better and the bold results are the best in each case.
InD
cifar 10
cifar100 svhn
OOD
cifar100
tin
svhn
texture
place
cifar10
tin
texture
place
avg
ODIN
77.76
79.65
73.41
80.76
82.61
78.1
81.33
70.97
79.31
79.76
78.37
EBO
86.19
88.61
88.42
86.88
89.62
79.07
82.46
77.81
77.84
80.16
83.71
ReAct
86.37
88.91
89.52
88.19
90.1
73.48
79.63
84.45
83.58
76.94
84.12
MLS
86.14
88.53
88.47
86.89
89.5
79.18
82.59
77.68
77.94
80.29
83.72
VIM
87.19
88.86
97.28
96.03
90.03
71.54
78.34
81.15
87.41
75.77
85,36
KNN
89.62
91.48
95.07
92.84
91.86
76.48
83.33
82.09
83.69
79.03
86.55
G-ODIN
88.75
90.7
98.05
95.45
91.86
72.79
81.38
89.85
89.41
77.44
87.57
CSI
87.36
89.64
94.52
89.82
88.44
69.43
72.83
77.14
59.38
69.1
79.77
CutMix
85.72
87.99
90.14
86.51
90.28
78.6
82,43
84.05
77.26
78.53
84.15
PixMix
90.62
92.6
97.33
95.8
92.23
75.77
81.86
93.79
84.36
78.88
88.32
Diff (ours)
90.53
92.85
95.09
93.66
92.65
76.43
84.23
84,96
80.64
78.7
86.97
4.1
SETUP
We evaluate our methods on the most recent OOD detection benchmarks, OpenOOD benchmarks YYang et al., 2022p. We images from six different datasets, which are filtered ensure that the in-distribution and the OOD do not have overlapping data. We use the CIFAR10 (Krizhvssyy et al.) 2009) and CIFAR 100 InD samples. For the CIFAR 10 dataset, we use CIFARIOU Tinylmagenet Krizhevsky et al. 2017). SVHN (Netzer et al. 2011. Texture and Places365 (Zhou et al. 2017) as OOD data. For the CIFARION 100 dataset, the OOD datasets are the same, except for swapping CIFAR 100 for CIFARIO as the OOD dataset. For fair comparison, we first train discriminator and generation models using the training set. we evaluate the results by calculating the area under the receiver operating characteristic curve (AUROC) dataset and the test set of others, to avoid the influence of model overfitting. All images from different datasets are resized into 32 x 32. The discriminator models pre-trained R from OpenOOD. The diffusion model used in this paper is just the classical model from DDPMs. We use pre-trained unconditional models and train the conditional version by ourselves.
(2006) between the test set of the Inl
4.2 OUT-OF-DISTRIBUTION DETECTION
We put the results in Table D We choose ten representative baselines. The first seven methods do not adjust the discriminator model similar to our method. ODIN (Liang et al.) 2018) uses temperature scaling and gradient- based input perturbation. EBO (Liu et al. 2020)
energy- based function. ReAct (Sun et al. 2021) uses rectified activation. MLS (Hendrycks et al. 2019) uses maximum logit scores. VIM (Wang et al. 2022) combines the information of feature space and logic space. KNN (Sun et al. (022) uses the nearest neighbor in the feature space. All these methods are post-hoc methods and we outperform them in all cases of CIFARIO and two cases of CIFARIOK We also compare our methods with four methods with additional training on the discriminator model. G-ODIN (Hsu al. 2020) decomposes the posterior model the probability of InD. CSI (Tack et al., 2020) explores the effectiveness of contractive learning objectives. CutMix (Yun al.) 2019) and PixMix (Hendrycks et al. 2022) are two new kinds of data augmentation to improve the capability of models. Our method outperforms them in three cases and gets competitive results in the others.
140
120
Our method performs worse than the SOTA methods when the test dataset is SVHN. A performance bottleneck is that, in addition to density esti- mation, DDP also has a lazy strategy in the denoising process. tends to keep the smooth area unchanged. In Figure we show this phenomenon using a simple case. We resize an InD image to r x r and then resize it back, which pulls the InD data away. However, the reconstruction error
36
Figure 6: The bad denois- ing cases for DDP.
8

---

Under review as a conference paper at ICLR 2023
42.0
425
950
g220
4.0
41.5
125
715
924
750
10
0.3
200
43. 668,
53c
50tc
340-150
60
300
350
620
540
553
(a)
(b)
(c)
(d)
Figure 8: The AUROC results under different guidance weights, training steps, disturbance degrees and resampling sizes.
decrease instead of increase under this operator. This phenomenon also occurs when the input is the relatively simple SVHN dataset.
4.3 ABLATION STUDY
Here, we show the influence of each item and hyperparameter on our scores. And all the setting is the same as the main experiments in Table except for the ablation object.
Detection space In Figure 7 we compare the results when we use
90-
different detection spaces, including the input image space, the different level feature spaces, and the logit space. We get the best results when we use the high-level feature or the logit. Here, the logit has 10 dimensions and is much smaller than the high-level feature space (512 dimensions). This shows that DDP can reduce information loss successfully. The low-level features and the image space get relatively bad results, the main reason is the information still redundant at these levels.
Iaae Faat
1ns3 logt
Figure 7: The results using different detection spaces.
Condition In Figure 8a we compare our the unconditional and conditional methods. The main problem is that CIFARIO0 has much more classes, which makes separating the feature space become more difficult and unconditional diffusion models cannot keep the interpolation in a single class. We also test different class weights w, we find that higher class weight can get relatively better results. This shows that realness is more important than diversity in the OOD detection task. In addition, we also find that the conditional version improves the detection results on CIFAR more obviously, which means the conditional control is important especially when the number of classes is big.
Training In Figure 8b] we find that although the training process of diffusion models is relatively computation-cost to achieve the best FID results, the OOD detection does not need the models to be 100 percent well-trained (200k steps). After 40k step training, we can get relatively good results and the improvement of FID does not help the OOD detection after that.
Timestep In Figure 8c we determine how to choose the best in DDP. We find that the best choice t 300 and this is consistent with our examples in Figure 4 When < 300, the difference caused by DDP is still not obvious enough. When 2 300, the information start to lose because the noise item accounts for larger and larger proportion, which limits the OOD detection results.
Resampling In Figure 8d we determine the influence of the repeated sampling size. According to analysis, the consistent detection results maintained by dynamic balance, therefore, need to resample several times to remove the random error in DDP. We find that more is better and 4 times resampling is good enough.
DISCUSSION
In this paper, we start with an assumption to explain the overconfidence problem. Then we combine discriminator and generation models to solve it. Under the setting of the first toy example, such strategy can perfectly solve the OOD detection problem. Although we cannot say the general cases can also be perfectly solved, competitive results on CIFAR10 and CIFARI00 using the combination a ResNet and a diffusion model. Our approach has good interpretability and a solid theoretical background. We believe that this strategy opens a new door to developing more powerful OOD detection methods and has the potential to be applied to OOD generalization and other related tasks.
show how to use this idea in the abstract feature space and get

---

Under review as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
Our implementation is put in the supplementary materials. We will publicize our method on GitHub once paper accepted.
REFERENCES
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. Concrete problems in ai safety. arXiv preprint arXiv: 16066665655 2016.
Jinwon An and Sungzoon Cho. Variational autoencoder based anomaly detection using reconstruction probability. Special Lecture on IE, 2(1):1-18, 2015.
Jacob Austin, Daniel D Johnson, Jonathan Ho, Daniel Tarlow, and Rianne van den Berg. Structured denoising diffusson models in discrete statesppcces Advances in Neural Information Processing 2021.
Systems,
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: analytic estimate of the optimal reverse variance in diffusion probabilistic models. In International Conference on Learning Representations, 2022.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34, 2021.
Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don't 't know by virtual outlier synthesis. arXiv preprint arXiv:2202.01199 2022.
Emilien Dupont, Arnaud Doucet, and Yee Whye Teh. Augmented neural odes. Advances in Neural Information Processing Systems, 32, 2019.
Tom Fawcett. An introduction to roc analysis. Pallern recognition letters, 2788):861-874, 2006.
ZongYuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Gamavi. Generative openmax for multi- class open classification. arXiv preprint arXiv: arXiv:1707.07418. 2017.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference computer vision and pattern recognition, pp. 3354-3361. IEEE, 2012.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Peng Guo, Zhiyun Xue, 7ac Mtema, Karen Yeates, Ophira Ginsburg, Maria Demarco, Rodney Long, Mark Schiffman, and Sameer Antani. Ensemble deep learning for cervix image selection toward improving reliability in automated cervical precancer screening. Diagnostics, 10(7):451, 2020.
Dan Hendrycks, Steven Basart, Mantas Mazeika, Mohammadreza Mostajabi, Jacob Steinhardt, and Dawn Song. Scaling out-of-distribution detection for real-world settings. arXiv preprint arXiv:1911.11132. 2019.
Dan Hendrycks, Andy Zou, Mantas Mazcika, Leonard Tang, Bo Li, Dawn Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16783-16792. 2022.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurlPS 202 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 6840-6851, 2020.
10

---

Under review as a conference paper at ICLR 2023
Jonathan Ho, Tim Salimans, Alexey A Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. In ICLR Workshop on Deep Generative Models for Highly Structured Data, 2022.
Yen-Chang Hsu, Yilin Shen, Hongxia Jin, and Zsolt Kira. Generalized odin: Detecting out-of- distribution image without learning from out-ofddistribution data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10951-10960, 2020.
Dihong Jiang, Sun Sun, and Yaoliang Yu. Revisiting flow generative models for out-of-distribution detection. In International Conference on Learning Representations, 2022.
Zhifeng Kong. Wei Ping. Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations. 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009,
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Communications of the ACM, 60(6):84 90, 2017.
Max WY Lam, Jun Wang, Dan Su, and Dong Yu. Bddm: Bilateral denoising diffusion models for fast and high-quality speech synthesis. In International Conference Learning Representations, 2021.
Siddique Latif, Muhammad Usman, Rajib Rana, and Junaid Qadir. Phonoeardiographic sensing using deep learning for abnormal heartbeat detection. IEEE Sensors Journal, 18(22):9393-9400) 2018.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibratee classifiers for detecting oft-of-distrribttoo samples. arXiv preprint arXiv: 2017.
Shiyu Liang, Yixuan Li, and R Srikant. Enhancing the reliability out-of-distribution image detection in neural networks. In International Conference Learning Representations, 2018.
Luping Liu, Yi Ren, Zhijie Lin, and Zhou Zhao. Pseudo numerical methods for diffusion models manifolds. In International Conference on Learning Representations, 2022.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of -fitt-iidittiittiinn detection. Advances in Neural Information Processing Systems, 33:21464-2144 2020.
Petr Marek, Vishal Ishwar Naik, Vincent Auvray, and Anuj Goyal. Oodgan: Generative adversarial network for out-of-domain data generation. arXiv preprint arXiv:2 (ttiv:2104.02440 2021.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't know? arXiv preprint 11Xi::810000936. 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee whye Teh, and Balaji Lakshminarayanan. Detecting out-of- distribution inputs to deep generative models using typicality. arXiv preprint arXiv:1906.02900 2019.
Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In Proceedings of the European Conference on Computer Vision (ECCV). pp. 613-628, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International Conference Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204. 06/25, 2022.
11

---

Under review as a conference paper at ICLR 2023
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. In International Conference on Learning Representations. 2022.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supervised outlier detection. In International Conference on Learning Representations, 2020.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque Input complexity and out-of-disrribttoon detection with likelihood-based generative models. In International Conference on Learning Representations, 2019.
Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphies and interactive techniques, pp. 245-254. 1985.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 202
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021b.
Kumar Sricharan and Ashok Srivastava. Building robust classifiers through generation of confident out of distribution examples. arXiv preprint 1112v:1812.002239 2018.
Yiyou Sun, Chuan Guo, and Yixuan Li. React: Out-of-distribution detection with rectified activations. Advances in Neural Information Processing Systems, 34:144-157 2021.
Yiyou Sun, Yifei Ming, Xiaojin Zhu, and Yixuan Li. Out-of-distribution detection with deep nearest neighbors. arXiv preprint arXiv:2204.06500 2022.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin, Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 33:11839-119 11852, 2020.
Arash Vahdat, Karsten Kreis, and Jan Kautz. Score-based generative modeling in latent space. Advances in Neural Information Processing Systems, 34, 2021.
Haoqi Wang, Zhizhong Li, Litong Feng, and Wayne Zhang. Vim: Out-of-distribution with virtual- logit matching. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4921-4930, 2022.
Yang Xin, Lingshuang Kong, Zhi Liu, Yuling Chen, Yanmiao Li, Hongliang Zhu, Mingcheng Gao, Haixia Hou, and Chunhua Wang. Machine learning and deep learning methods for cybersecurity. leee access, 6:35365-35381 2018
Jingkang Yang, Pengyun Wang, Dejian Zou, Zitang Zhou, Kunyuan Ding, Wenxuan Peng, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, al. Openood: Benchmarking generalized out-of- distribution detection. arXiv preprint, 2022.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pp.
2019.
Yufeng Zhang, Wanwei Liu, Zhenbang Chen, Ji Wang, Zhiming Liu, Kenli Li, and Hongmei Wei. Out-of-distribution detection with distance guarantee in deep generative models. arXiv preprint arXiv:2002.03338,, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence,
2017.
12

---

Under review as a conference paper at ICLR 2023
A
APPENDIX
A.1
RELATED WORK
Diffusion Model Denoising Diffusion Probabilistic Models (DDPMs) Ho et al. (020) successfully generate high-quality images and make DMs become popular. For now, DMs can not only generate unconditional high-quality images but also are applied to many different fields. For the conditional generation task, DMs can do interpolation. manipulation, image-editing, style transformation and
text-conditional generation Ramesh et al. (2021) 2022). For different data types, DMs can do text Austin et al. (2021), audio Kong et al. (2020): Lam et al. (2021) and video generation.
The main challenge for DMs is that they require hundreds to thousands of iterations to produce high- quality results, which limits the application of DMs. After DDPMs, many works try to make DMs faster and better. Some of them focus the denoising equations of DMs. Nichol & Dhariwal 20211 design a better time schedule for the denoising process. Liu et al. (2022) provide new numerical methods for the denoising process. [Bao et al. (2022) find analytic results for the variance of the denoising process. Some of them try to design new training strategies and new models. [Salimans & Ho (2022) use distillation to accelerate DMs. Dhariwal & Nichol (202I) change the model structure from Unet to GAN to make each iteration step more powerful. W What's more, [Song et al. (2021b) first shows that DMs can be rewritten as two neural differential equations Chen et al. (2018): Dupont et al.) (2019). Therefore, the numerical methods used to accelerate neural differential equations can also be used here.
A.2 SIMILARITY AND DIFFERENCE
Several baselines are similar to ours in some ways. The first one is KNN (Sun et al. 2022). which does a KNN search in the feature space. However, KNN ignores the possibility that an OOD input may have similar feature an InD data, and all methods that only the final feature have this problem, too. In addition, using KNN in the input space directly is also invalid, because of data sparsity and irrelevant information interference.
Another similar approach data generation and augmentation. These methods retrain the discrim- inator, but our generation and discriminator models are trained separately. What's 's more, existing methods use generation models to generate OOD data (Marek et al. (021) to help the discriminator models to know the capability boundary. Our methods use the generation models to do interpolation between the new input data and the training data, which do not need to carefully classify the training data and design new loss functions. Some methods use data augmentation help the training process of discriminator models. Although classical data augmentation can enhance the richness of data and keep the data realistic, some new methods start to add complex and unreal augmentation (Hendrycks) et al. 2022), which increases the burden on the models and lacks clear motivation.
Many methods combine generation model and a discriminator model to do OOD detection, such as Ge et al. (2017); Neal et al. (2018): Lee et al.] (2017); ; Du et al. (2022). However, we find these papers all try to use generation models to generate OOD data, which needs to design new object functions and combine the training process of the generation and discriminator models. Our method use diffusion models to do interpolation and detect the dynamic change under DDP, which is simple and effective.
more papers using generation and dis- criminator models
A.3 INVERTIBLE DIFFUSION MODEL
Here, we introduce more details of the diffusion model used in this paper. In addition, we analyze how to make the iteration of diffusion models more invertible.
Score-based generation model Song et al. (2021b) show that the diffussin-denoisingg process can also be treated as two differential equations:
dx = (V1 - 3(1) - (1) + BB(jdw
(5)
dx = (V1 3(t) - 1)x(()) ((0)))(((()).) dt.
13

---

Under review as a conference paper at ICLR 2023
This is called probability flows (PFs). The noise co of DMs and the gradient of logic likelihood sø
are equivalent Bao et al. (2022). More specifically, we have that sy (z, t) = -- (0 t).
Pseudo numerical method
Liu et al. (0022 provide pseudo numerical methods for diffusion
models (PNDMs) accelerate DDIMs. PNDMs define Equation (3) with 02 0 as transfer function:
Vot-6 Vāl
(ot 5 (x))
δ -,,,t-8)
(6)
(1 Or- )ok + v(1 at )t)
PNDMs combine this transfer function with the noise estimated by classical numerical methods, like the linear multistep method, to get the new denoising equations:
{
E = मे (55e; 59c, +8 + 376 96(4-s)
Iz-5 = cl,,,,,t, 8).
Here, = c0( t). Both PFs and PNDMs accelerate the denoising process without loss of quality.
Invertibility We show the test results in Figure 9 For DDIMs,
the error occurs the beginning, and the error accumulates with the increase of the total generation step. For PFs, the initial error is not huge, but the cumulative error occurs when the number of the total generation steps is bigger than 500. DDIMs are first-
DDIM PF+ PNDM
M
1000
order methods, and other methods are high-order methods. We can say higher convergent order can increase the invertibility. PFs numerical methods of adaptive step size, and PNDMs use
methods of fixed step size. Therefore, we think that fixed step size can also benefit invertibility. To verify this, we replace the methods of adaptive step size used by PFs with the methods of fixed step size and call probability flows plus (PFs+). We find that the errors decrease immediately. The reason is that fixed step size maintains consistency between the sampling locations of the forward and reverse processes, which benefits the invertibility. Combining the above analysis, we have the following property:
200 400
600
800
1000
Figure 9: The reconstruction er- ror under different iteration inter- val [0, t] and fixed step size 20.
High convergent order and fixed iteration step size can improve the invertibility of DMs under fixed total iteration steps.
A.4 INTERPOLATION
In Algorithm 5 and 6 we introduce two types of interpolation using diffusion models. The positions of zo and 21 are symmetric in the original interpolation and asymmetric for the asymmetric one. According to our experiment, 2o is more important in asymmetric interpolation. The visualization of these interpolations can be found in Fig 3
Algorithm Symmetric interpolation
Algorithm Asymmetric interpolation
Input: Images x/ -g generative gap 6, interpo-
Input: Images z/.- 2g. generative gap 8, interpo- lation timestep
lation rate
1: xt, . Φ (x) 0, T,), + (27.
1: x = Φ (xū 0, T, 5)
2: inter Slerp( x 2,:0)
2: of = v T4.00 + v1 - d2
3: inter (aripier 0,
3: adtlerr & (z inter t, 4)
4: return inter
4: return xn
A.5 THE PROOF OF TOY EXAMPLE
To make this claim strict, we need to make some definitions. Each image can be represented as a function [0, 1 and the value of r(x, y) is the RGB value position (x, y). And the images continuous in most positions. Therefore, we simplify the input space to C(O, 112) the continuous
function on (0, 1 and then one-dimensional C((D, 1)) for simplicity. The mask operator is
14

---

Under review as a conference paper at ICLR 2023
restriction operator (r) = 210==00.026jj/0.5.000 and |r| is the max absolute value of r on [0, 1.. The bad cases form a set that satisfies A. when we choose proper }
here. The InD is just fr E C((O. 11) | o}
(rløs(r) (0) = 0, || > J.. We can prove that A. (6s)
A straightforward solution is that for each E C(R)) let {9(x); = r(x + a)ji E #0.25) ). which represents the moving operator. The proof is that we can use go to get the information about r on S and use $20.25| to get the remaining on [0, 1]/S. Then r mush satisfies |r(x)] < o, Va E [0. 1|.
There also exist other kinds of solutions. For example, let (g(2)), = -ー S r(x)dr a, E [0. 1]}. which represents the mixing operator. addition, the mask operator is not necessary and we can extend it to any restriction operator.
Here we prove that {g(2); f(x (x+ (0,25)))) and (g(x),.b = S2 f())do [0, solve the problem:
Given the input space C( 1J), the restriction operator &s and fixed 6, how can we design additional operator set {gi} to minimize the annihilator set As (gghs))))
For {g(x); = f(x + i) E +0.25)) 9ols = fls and equals to zero means that f is zero on S. And then we also know that 90.25 =f --0.25 and g-0.25| fls+0.25 equals zero, which means that f is zero on 0.25 US + 0.25 = [0, 11/5. Then the only choice of f zero and the annihilator set is empty.
For {g(2)a,b Nia Sh f(z)dzla,be e [o, 1]. If 3zo
[0, 1], f(zo) 6, then Ee s.t. Vo E (xo
f.. > d because f continuous. Then we have that g(z) -o- = 1 S f(z)dr and ((())a-rrattt Is > 8, too. This does not satisfy our condition, so the annihilator set empty.
A.6 OOD DETECTION RESULT
We add experiments about adversarial OOD examples in 2 We use 5000 images of CIFAR1O to generate 5000 adversarial OOD examples. We use PGD and OnePixel methods, which use Loc and Lo norms, respectively. We adversarial examples. For PGD, we use eps=8/255, alpha=2/255 and steps=10. For Onepixel, we use pixels=10, steps=10, popsize=10 and inf b2tc We find that our method can perform pretty well on the adversarial OOD examples.
open library
to generate the
We also test the computation cost of our method. We record the total time to compute the detection scores of 5000 images using an RTX3090. We use about 0.06s to process an image. To be honest, it is much slower compared to existing methods. However, just like the success story of diffusion model acceleration, we believe that our new method can be faster in the future.
Table 2:
The AUROC results different methods. We train the models on the training data 160k epochs and test the results the test data. We use the conditional version method and the guidance weight is 2. We set the disturbance degree = 300, the repeat size r and use the logic space as detection space. The higher results are better and the bold results are the best in each case.
ci
tin
pgd
onepixel
avg
time
ODIN
77.76
79.65
34.52
65.23
64.29
0.22
EBO
86.19
88.61
33.13
71.52
69.8625
0.05
ReAct
86.37
88.91
57.92
73.28
76.62
0.04
MLS
86.14
88.53
33.56
71.55
69.945
0.03
VIM
87.19
88.86
91.96
78.98
86.7475
0.09
KNN
89.62
91.48
87.39
80.09
87.
0.25
G-ODIN
88.75
90.7
97.79
80.99
89.5575
2.31
CSI
87.36
89.64
83,54
80.38
85.23
28.1
CutMix
85.72
87.99
82.92
73.17
82.45
2,05
PixMix
90.62
92.6
93.77
57.87
83.715
2.51
Diff (ours)
90.53
92.85
93.92
82.21
89.8775
340
15


Reviews:
[{"review_id": "clwSClz8z8F", "reviewer": "Reviewer_sRvP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces diffusion models (DMs), a kind of powerful generative model, into OOD detection and finds that the denoising process of DMs also functions as a novel form of asymmetric interpolation. This property establishes a diffusion-based neighborhood for each input data. Then, the authors perform discriminator-based OOD detection based on the diffusion-based neighborhood instead of isolated data.", "summary_of_the_review": "This paper introduces diffusion models (DMs) to generate outliers for OOD detection. The method is valid to me but further clarifications and visualizations are needed to understand the approach better. Will consider changing my score after I see some of the other reviewers' opinions and the authors' responses.", "strength_and_weaknesses": "strength:\n1. the paper uses diffusion models to replace the requirement of the real outliers in the ood detection literature. the idea makes sense to me.\n2. the results seem to be effective from the main tables on one benchmark.\n\nweakness:\n1. the paper is really hard to grasp. some of the descriptions are quite vague. What does the author mean by saying \"reduces the information loss of feature extraction.\"?\n2. there are some obvious mistakes in the paper, such as the algorithmic blocks such that It is heavy to grasp the flow of the method. For example, in Algorithm 2 and 3, the x_noise is generated but it is never used in the algorithm, x_neighbor does not have a closed-form relationship with x_noise.\n3. the training time is suspicious, the author trained the classification and also the diffusion model for 160k epochs, which is not scalable in practice. Also, no experiments show its effectiveness on large-scale benchmarks, such as Imagenet1k as the in-distribution data.\n4. There are no visualizations provided on the generated neighboring data points so that the readers might be confused why the proposed approach works.\n5. the paper ignores some existing literature on using generation for OOD detection, such as [1-4]. Some additional discussion are favorable for readers to get the context.\n\n[1] Zongyuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. In British Machine Vision Conference 2017. British Machine Vision Association and Society for Pattern Recognition, 2017.\n\n[2] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In International Conference on Learning Representations, 2018.\n\n[3] Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision – ECCV 2018\n\n[4] Xuefeng Du, Zhaoning Wang, Mu Cai, and Yixuan Li. Vos: Learning what you don’t know by virtual outlier synthesis. In International Conference on Learning Representations, 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WXb0S_Fcml", "reviewer": "Reviewer_ZBTr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new OOD detection framework that combines a typical discriminator approach with a diffusion-based neighborhood definition. Motivated by a synthetic scenario, the de-noising process of invertible diffusion models is regarded as a kind of asymmetric interpolation and is utilized to find the diffusion-based neighbors of a given input. Then the OOD samples are detected based on the features of the neighborhood. The method is validated on OOD detection benchmarks and some analyses on the model component and sensitivity on the hyperparameters are also given. ", "summary_of_the_review": "The proposed idea seems novel and could inspire the readers. However, its performance is less impressive yet, and to be officially presented at the conference, the manuscript must be strengthened in its readability, and the authors need to provide enough details to guarantee its reproducibility. ", "strength_and_weaknesses": "[+] Introducing the (noise) interpolation scheme to the OOD problem via the diffusion process has a strong mathematical background and is claimed to be more interpretable. \n\n[+] Combining generative and discriminative models adds another proxy for distinguishing between ID and OOD. This enables the coordination of generator-sensitive and discriminator-sensitive features. \n\n[+] The Open-OOD benchmarks allowed for a fair evaluation. \n\n[-] The presented experimental results (in Table 1) are not very impressive compared to the previous methods.  \n\n[-] Overall, the clarity and readability of the manuscript have been compromised, and there is a crucial need for extensive revisions. \n\n[-] Insufficient methodological clarification was given: it seems impracticable for readers to get a high-level overview of the diffusion models; rather, it is required to focus on how to organize and develop the diffusion-based neighborhood for identifying IOD and OOD. (especially how the ‘interpolation’ is managed to develop the classifier)\n\n[-] It is not straightforward to grasp the context of Figure 2 and Figure 4. A more detailed explanation should be presented in the caption and the main text. \n\n[-] The descriptions in Algorithms 1-3 need to be improved. For example, the input and output of each line of codes do not match with those in other lines and some notations are used without introduction. \n\n[-] Figure 7 illustrates that the performance of the proposed strategy appeared to be hyper-parameter-sensitive. Nonetheless, Table 1 lacks a criterion for its selection. \n\n[-] We encourage the authors to provide more implementation details and the source code to the public to guarantee reproducibility.\n\n[Q1] How does the model function when neighbors are obtained from random noises rather than images? \n ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TRqn_Jv2eXs", "reviewer": "Reviewer_H7mg", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "OOD detection is an important task for reliable AI. This paper proposes a new framework for OOD detection which combines the generative and discriminative models together. Specifically, they use the diffusion model for modifying the data under control and use the changes in the feature space for OOD detection. They choose ten representative methods to compare with the proposed methods on several datasets.", "summary_of_the_review": "Overall I like the idea of applying diffusion models to do out-of-distribution detection, and this paper showed good performance compared with existing baselines. This paper can be greatly improved in terms of: more realistic and meaningful motivation examples; more detailed demonstration of the proposed method, etc.", "strength_and_weaknesses": "Strength\n- It provides a novel and promising strategy for OOD detection. Although using the generative model together with the discriminative model for OOD detection has been proposed in [1], the intrinsic idea behind how to use the generative model is radically different.\n- The paper is well written, easy to follow, and with plenty of experiments to verify the effectiveness of their method.\n\n[1] Oodgan: Generative adversarial network for out-of-domain data generation\n\n\nWeaknesses:\n-   The toy example in section 3.1 and section 3.3 are too disjoint with the practice, which makes the motivation not very convincing. More specifically, the discriminator considered in section 3.1 is a mask operator, which will hardly be used in practice as a discriminator. Adding some more realistic and convincing examples would be helpful.\n-   Figure 4 demonstrates the dynamic of confidence over time has a different pattern for InD input and OOD input. However, their proposed algorithm does not rely on the dynamic of the confidence under the perturbance of DDP. And Figure 4 itself is also very confusing, with the differences between the three cases hard to understand. It is suggested that the author have a discussion of the performance by using the dynamic information.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "MU4fddrLrd", "reviewer": "Reviewer_ziL9", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper applies a denoising function (in the form of a diffusion model) to an image before doing OOD detection. The claim is that the new method outperforms other OOD methods on two of the datasets CIFAR10 and CIFAR100.", "summary_of_the_review": "The paper left me unconvinced of the approach or the claims. The paper could use improvements on the intuition behind the approach, presentation of the approach, and empirical results. \n\nUpdate after the rebuttal: \nI reviewed the changes made. I still find the writing quality to be a significant issue. For instance, I made the following comment in my original review as an example of poor writing:  \n\n*\"Let us assume that x0 is an image and epsilon is Gaussian noise, which  is the reverse of image x1, and we ...\". One has to figure out what which refers to. The noise seems wrong.*\n\nI don't see that addressed. The concerns with empirical results remain as well. The adversarial examples need to be adaptive to the authors' detection pipeline and designed with those in mind. I would recommend using AutoAttack as well.\n\nI do like the water analogy. But, the toy example still seems a bit contrived to me. \n\n", "strength_and_weaknesses": "Strengths: \n-- OOD detection is an important problem. The paper motivates their work with the point that discriminator-based methods can be susceptible to adversarial examples and bad cases. \n\n-- The authors propose a denoising technique that diffuses an image and then does OOD classification on it. \n\nCons: \n -- The paper is difficult to follow due to poor writing. For instance: \"Let us assume that x0 is an image and epsilon is Gaussian noise, which  is the reverse of image x1, and we ...\". One has to figure out what which refers to. The noise seems wrong. \n -- Figure 4 is not clear\n --The  mathematical model could be clearer.\n -- The Toy example seems  contrived and very unlikely to occur in practice.\n -- The empirical results are not convincing. It seems that the results are mixed. \n -- It is not clear how efficient the OOD detection scheme is.  Some discussion of the efficiency of OOD detection versus other techniques may be useful.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject