Diffusion-based  Layer-wise Semantic Reconstruction for Unsupervised Out-of-Distribution Detection

Authors: ying  yang
  de cheng
chaowei  fang
yubiao  wang
changzhe  jiao
lechao  cheng
nannan  wang
xinbo  gao
de  cheng
•  •••• ••••••
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-30 10:28:30

Abstract: Unsupervised out-of-distribution (OOD) detection aims to identify out-of-domain data by learning only from unlabeled In-Distribution (ID) training samples, which is crucial for developing a safe real-world machine learning system. Current reconstruction-based method provides a good alternative approach, by measuring the reconstruction error between the input and its corresponding generative counterpart in the pixel/feature space. However, such generative methods face the key dilemma, $i.e.$, improving the reconstruction power of the generative model, while keeping compact representation of the ID data. To address this issue, we propose the diffusion-based layer-wise semantic reconstruction approach for unsupervised OOD detection. The innovation of our approach is that we leverage the diffusion model's intrinsic data reconstruction ability to distinguish ID samples from OOD samples in the latent feature space. Moreover, to set up a comprehensive and discriminative feature representation, we devise a multi-layer semantic feature extraction strategy. Through distorting the extracted features with Gaussian noises and applying the diffusion model for feature reconstruction, the separation of ID and OOD samples is implemented according to the reconstruction errors. Extensive experimental results on multiple benchmarks built upon various datasets demonstrate that our method achieves state-of-the-art performance in terms of detection accuracy and speed.

Main paper: 
 introduction: 
 Unsupervised Out-of-Distribution (OOD) detection aims to identify whether a data point belongs to the In-Distribution (ID) or OOD dataset, by learning only from unlabeled in-distribution training samples. OOD detection plays a vital role in developing a safe real-world machine learning system, which ensures that the model is only performed on data drawn from the same distribution as its training data. If the test data does not follow the training distribution, the model could unintentionally produce nonsensical predictions, resulting in some misleading conclusions. Naturally, OOD detection is one of the key techniques for ensuring the model's robustness and safety.Existing research studies the OOD detection mainly under two settings, i.e., supervised and unsupervised. The supervised OOD detection methods usually deem this task as a binary classification problem, which relies on training with data labeled as OOD from disjoint categories or adversaries [Hendrycks et al., 2018] , [Ming et al., 2022] . However, in many practical applications, it is almost impossible to access representative OOD samples, as the OOD data usually can be highly diverse and unpredictable. Therefore, we prefer to study the more challenging while practical unsupervised OOD detection problem. We will build an OOD detector trained solely on unlabeled ID data, as large amounts of unlabeled data are readily available and widely utilized due to their ease of acquisition.Current reconstruction-based methods provide a good alternative approach for OOD detection, by measuring the reconstruction error between the input and its corresponding generative counterpart in the pixel/feature space. Obviously, the generative models and metric learning evaluation strategies are the main research directions. However, such methods of the generative models always face the following key dilemma: The projected in-distribution latent feature space should be compressed sufficiently to capture the exclusive characteristics of ID images, while it should also provide sufficient reconstruction power for the large-scale ID images of various categories. Existing generative-based methods (e.g., auto-encoder (AE), variational AE [Kingma and Welling, 2013] and Generative Adversarial Network(GAN)) [Goodfellow et al., 2014] , can not always fulfill these two requirements simultaneously, and a good balance between them is required. Besides, recent OOD detection methods based on diffusion models such as [Graham et al., 2023] , [Gao et al., 2023] and [Liu et al., 2023] often involve the pixel-level reconstruction of distorted images, which consume much training/inference time and computation resources.To address the above-mentioned issues, and inspired by the latent space noise addition mechanism in Latent Diffusion Models (LDM) Rombach et al. [2022] , we propose the diffusion-based layerwise semantic reconstruction approach for unsupervised OOD detection. Specifically, the proposed method makes full use of the diffusion model's intrinsic data reconstruction ability, to distinguish in-distribution samples from OOD samples in the latent feature space. In the diffusion denoising probabilistic models (DDPM) [Ho et al., 2020] , the model is trained to incrementally remove noise from the noised inputs of different levels. Clearly, we can see that, instead of faithfully reconstructing inputs from the distribution it was trained on as previous VAE Kingma and Welling [2013] or GAN Goodfellow et al. [2014] , the diffusion-based model shows more powerful reconstruction capabilities. Practically, our model involves reconstructing an input image feature from multiple values of the time step, this allows a single trained model to handle large amount of noise applied to the input, obviating the need for any dataset-specific fine-tuning.Moreover, to set up a comprehensive and discriminative feature representation, we devise a multilayer semantic feature extraction strategy. Performing feature reconstruction on top of the multi-layer semantic features, encourages to restrict the in-distribution latent features distributed more compactly within a certain space, so as to better rebuild in-distribution samples while not reconstructing OOD comparatively. Overall, by distorting the extracted multi-layer features with Gaussian noises and applying the diffusion model for feature reconstruction, the separation of ID and OOD samples is implemented according to the reconstruction errors. Note that, the proposed Latent Feature Diffusion Network (LFDN) is built on top of the feature level instead of the traditional pixel level, which could significantly improve the computation efficiency and achieve effective OOD detection. The other potential strength of such a strategy is that it avoids the reconstruction of minor characteristics unrelated to image understanding. In summary, the contributions of this work are as follows:• We propose a diffusion-based layer-wise semantic reconstruction framework to tackle OOD detection, based on multi-layer semantic feature distortion and reconstruction. Meanwhile, We are the first to successfully incorporate generative modeling of features within the framework of OOD detection in image classification tasks. • The layer-wise semantic feature reconstruction encourages restricting the in-distribution latent features to be more compactly distributed within a certain space, enabling better reconstruction of ID samples while limiting the reconstruction of OOD samples. • Extensive experiments on multiple benchmarks across various datasets show that our method achieves state-of-the-art detection accuracy and speed.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Yes Justification: The main claims in the abstract and introduction accurately reflect our contributions. We propose a diffusion-based layer-wise semantic reconstruction method for unsupervised out-of-distribution (OOD) detection. Our method demonstrates superior performance in detecting OOD samples, as detailed in Section 3 and Section 4of our paper.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 Existing researches study the OOD detection mainly under two settings: supervised and unsupervised. The Supervised method is generally based on classification. The method usually uses the maximum softmax probability [Hendrycks and Gimpel, 2016] from the final fully connected (FC) layer as the score to judge the ID sample. But the classification-based OOD detection methods often encounter issues with assigning high softmax probability to OOD samples. Recent works [Liu et al., 2020] , [Sun and Li, 2022] , [Djurisic et al., 2022] , [Zhao et al., 2024] , attempt to alleviate this issue. The unsupervised OOD detection can be roughly categorized as the distance-based metric evaluation and the generative-based reconstruction methods.Distance-based methods assume that OOD data lies far from ID class centroids. [Ren et al., 2021] improved OOD detection by separating image foregrounds from backgrounds and computing the Mahalanobis distance for each, then combining them. [Sun et al., 2022] used a non-parametric nearest neighbor distance for OOD detection. [Techapanurak et al., 2020] and [Chen et al., 2020] used cosine similarity to measure distances between test data features of in-distribution data to identify OOD data. [Huang et al., 2020] applied Euclidean distance, while [Gomes et al., 2022] used Geodesic distance for OOD detection. These methods often fail to capture sample distribution accurately.Among the generative-based methods, the Likelihood-based methods can be traced back to as early as [Bishop, 1994] . This method assumes that the generative model assigns high likelihood to ID data, while the likelihood for OOD data tends to be lower. Recently, several deep generative models have supported the computation of likelihood, such as VAE [Kingma and Welling, 2013] , PixelCNN++ [Salimans et al., 2017], and Glow [Kingma and Dhariwal, 2018] . However, some studies ( [Nalisnick et al., 2018] ; [Choi et al., 2018] ; [Kirichenko et al., 2020] ) have found that probabilistic generative models might also assign high likelihood to OOD data.A series of studies have attempted to mitigate this issue. [Serrà et al., 2019] explored the relationship between image complexity and likelihood values, which adjusted likelihoods based on the size of image compression. [Ren et al., 2019] enhanced OOD detection by comparing likelihood values derived from different models. Another closely related approach highlights that these indicators are not well suited for VAEs. [Xiao et al., 2020] proposed a specialized metric known as likelihood regret for OOD detection in VAEs. [Cai and Li, 2023] suggested to leverage the high-frequency information of images to improve the model's ability to recognize OOD data. Additionally, a range of studies [Nalisnick et al., 2019] , [Wang et al., 2020] , [Bergamin et al., 2022] , [Osada et al., 2023] , have proposed typicality tests, estimating layer activation distributions and other statistical measures on training data, which are then evaluated through hypothesis testing or density estimation.Another type of OOD detection methods leverage the idea that generative networks produce different reconstruction errors for ID and OOD data. Some methods such as [Sakurada and Yairi, 2014] , [Zong et al., 2018] , and [Zhou and Paffenroth, 2017] , used auto-encoders to analyze reconstruction errors. GAN-based methods [Schlegl et al., 2017] , [Zenati et al., 2018], and [Madzia-Madzou and Kuijf, 2022] utilized reconstruction errors and discriminator confidence to detect anomalies. Recent works [Graham et al., 2023] , [Gao et al., 2023] , and [Liu et al., 2023] applied diffusion models to model the pixel-level distribution of images, using errors from multiple reconstructions for OOD detection. Different from previous methods, we propose to leverage diffusion models to perform multi-layer semantic reconstruction in the latent feature space, not only for their stability in generation but also for significantly reducing training and inference time costs.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Yes Justification: The limitations of our work are discussed in detail in Section 5Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper. , but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 
method: 
 Unsupervised OOD detection leverages intrinsic information from an unlabeled ID dataset D to train a detector. Suppose D contains N images, namely D = {x i } N i=1 , where x i denotes the i-th image.The target is to learn an OOD detector denoted as S(•), which can effectively evaluate an OOD score for each input image. The judgment of whether the input image belongs to ID or OOD is implemented by thresholding the OOD score. For example, given a testing image x, it is recognized as an ID sample if the OOD score S(x) is lower than the pre-defined threshold λ; otherwise, it is recognized as an OOD sample.In this paper, we propose a diffusion-based layer-wise semantic reconstruction framework to accomplish the OOD detection task. Specifically, as illustrated in Figure 1 , the proposed framework consists of the following three components: the multi-layer semantic feature extraction module, the latent feature diffusion stage, and the OOD detection head.The proposed semantic reconstruction-based method achieves OOD detection by measuring the reconstruction error between the input and its generative counterpart in the feature space. Specifically, we devise a multi-layer semantic feature extraction strategy, to set up a comprehensive and discriminative feature representation for each input image. Such multi-layer features could better rebuild the samples and encourage the ID semantic features distributed more compactly within a certain space from different semantic layers.Specifically, given an image x ∈ R 3×w×h with w and h being the width and height of the input image, passing through an image encoder E(•), (e.g., EfficientNet [Tan and Le, 2019] ), we can extract its feature maps from different layers (i.e., low-level to high-level semantic blocks). The multi-layer intermediate feature map from the m-th block can be defined asF m ∈ R cm×wm×hm , m ∈ {1, ..., M },where c m , w m and h m are the number of channels, width and height of the feature map F m , and M is the total number of intermediate feature maps. Then, each feature map F m undergoes the global average pooling, obtaining the one-dimensional feature vector f m ∈ R cm . Afterward, Zscore normalization [Al Shalabi et al., 2006 ] is applied to each feature vector f m , resulting inf m = f m -µ f m √ Var(f m )+δfor the m-th intermediate feature vector f m of the input image x, where Var(f m )is the variance of f m along the channel elements, and δ is a small constant value. Finally, we obtain the overall multi-layer feature vector for the input image x as:z 0 = H(x) = [f 1 , . . . , f m , . . . , f M ] ∈ R cby concatenating all the intermediate feature vectors, where c = M m=1 c m , and H(x) denotes the whole feature extraction process. Fitting the semantic feature distribution of ID samples is crucial for identifying whether the input is an ID or OOD sample. However, it is difficult to explicitly model the semantic feature space which has moderate complexity. Existing generative-based models [Zhou, 2022] , [Cai and Li, 2023] address the modeling of complex data/feature space by transferring the original data/features into an implicit bottleneck space and learning a generator capable of recovering ID samples from the bottleneck space. Since the generator can not generalize well in recovering unseen OOD samples, it can be used as the OOD detector.Inspired by this, we set up a diffusion-based feature distortion and reconstruction framework, considering the strength of diffusion models in data reconstruction. Our framework is innovative in the introduction of diffusion models in modeling semantic features, while previous works [Graham et al., 2023] , [Liu et al., 2023] , [Gao et al., 2023] focus on applying diffusion models for straightforward pixel-level distortion and reconstruction.Semantic Feature Distortion.The semantic feature distortion process can be conceptualized as transforming the semantic features into distorted counterparts with different levels of noise. For each step t belonging to [1, . . . , T ], the generation of data point z t follows the formula:z t = ennoise(z 0 , t) = √ α t × z 0 + √ 1 -α t × ϵ, ϵ ∼ N (0 c , I c×c ) (1)where ϵ ∈ R c represents a Gaussian noise vector; N (•, •) denotes the Gaussian distribution; 0 c and I c×c denote the c-dimensional zero vector and the c × c identity matrix, respectively. α t is a predefined noise level that controls the amount of noise added at each step.Semantic Feature Reconstruction. For reconstructing the semantic features from their distorted counterparts, we build up a Latent Feature Diffusion Network (LFDN) constituted by 16 residual blocks (ResBlock), as shown in Fig. 1 .The structure of ResBlock is illustrated in Fig. 2 . Its residual branch is formed with two groups of Groupnorm [Wu and He, 2018] , SiLU, and linear layers, as well as a MLP used for absorbing in the time embedding.Following the calculation process of the denoising diffusion implicit model [Song et al., 2020] , we employ LFDN to remove the noises injected into the semantic features with skipping step stride denoted as s. The detailed noise-removing process for z t is described as follows. s is set to a value randomly selected from {1, 2, • • • , t}.1) We first input z t and the time embedding of t into LFDN, generating an initial reconstruction state denoted as zt . The calculation formulation can be summarized as: zt = LFDN(z t , t), where LFDN(•) denotes the feed-forward process of LFDN. 2) Afterwards, we estimate the noise correction vector for z t denoted as εt as follows,EQUATIONwhere α t is the predefined noise level of the t-th feature distortion step. 3) Then, we sample the input (z t ′ ) for implementing the t ′ -th step's feature reconstruction where t ′ = max(t -s, 0) as:EQUATIONwhere σ 2 t represents the variance of the additional noise at step t. Regarding zt ′ and time embedding of t ′ as inputs, LFDN predicts reconstruction results of the t ′ -th step as zt ′ = LFDN(z t ′ , t ′ ). 4) Repeating steps 2 and 3 until t ′ = 0, yields the final reconstructed semantic features z0 . We summarize the above process as z0 = denoise(z t , t). This framework ensures that z0 is not solely derived from the LFDN output but is continuously refined by DDIM, integrating detailed corrections to achieve high accuracy in reconstructing the original data from its noisy observations. Objective Function. For optimizing the network parameters of LFDN, the mean square error is used as the loss function for pulling close the outputs of LFDN with the original semantic features. The calculation formulation is as follows:L = 1 N x∈D ∥z 0 -LFDN(z t , t)∥ 2 2 (4)During training, t is randomly selected from {1, 2, • • • , T }. The detail is illustrated in Algorithm 1.Our approach can be integrated with three metrics to detect OOD data. Firstly, we utilize the Mean Squared Error (MSE) to measure the feature reconstruction error. Secondly, we use the Likelihood Regret metric (LR = MSE initial -MSE final ) [Xiao et al., 2020] , which quantifies the change in MSE from the initial epoch to the final epoch. This metric reflects the model's evolving certainty during training. Generally, the reconstruction errors for ID samples decrease as the model becomes more familiar with these samples, whereas the errors for OOD samples remain relatively stable.Lastly, we employ the Multi-layer Semantic Feature Similarity (MFsim), i.e., the cosine similarity.We assesses the cosine similarity between the original features z0 = [f 1 , . . . , f m , . . . , f M ] and the reconstructed features z0 = [ f 1 , . . . , f m , . . . , f M ] at various layers: Sim(f m , f m ) = f m • f m ∥f m ∥•∥ f m ∥ .The OOD detection score MFsim, is then computed as the negative average of these similarities: MFsim= -1 M M m=1 Sim(f m , f m ), where M is the number of feature maps. A higher MFsim score indicates a greater likelihood of the data being OOD. Algorithm 2 details the MFsim calculation. The flows for MSE and LR calculations are provided in Appendix A.Algorithm 1 Training Algorithm 1: Input: Train image x ∈ R 3×h×w 2: z 0 = H(x) = [f 1 , . . . , f m , . . . , f M ] ∈ R c 3: repeat 4: Draw t ∼ Uniform{1, . . . , T } 5: Draw ϵ ∼ N (0, I) 6:Compute z t and L 7:z t = √ α t z 0 + √ 1 -α t ϵ 8: L = 1 N x∈D ∥z 0 -LFDN(z t , t)∥ 2 2 9:Update the parameters via the AdamW optimizer. 10: until convergence Algorithm 2 Testing Algorithm1: Input: An image x ∈ R 3×h×w 2: Output: OOD score 3: z 0 = H(x) = [f 1 , . . . , f m , . . . , f M ] ∈ R c 4: z t ← ennoise(z 0 , t) 5: z0 ← denoise(z t , t) 6: [ f 1 , . . . , f m , . . . , f M ] ← z0 7: for m = 1 to M do 8: S m ← Sim(f m , f m ) 9: end for 10: MFsim ← - M m=1 S m /M 11: return MFsim 4 Experiments 
datasets and evaluation metrics: 
 Datasets: We train the OOD detection model on three in-distribution (ID) datasets: CIFAR-10 [ Krizhevsky et al., 2009] , CIFAR-100, and CelebA [Liu et al., 2015] . When testing models learned on a specific ID dataset, we select several datasets from SVHN [Netzer et al., 2011] , SUN [Xiao et al., 2010] , LSUN-c [Yu et al., 2015] , LSUN-r, iSUN [Xu et al., 2015] , iNaturalist [Van Horn et al., 2018] , Textures [Cimpoi et al., 2014] , Places365 [Zhou et al., 2017] , MNIST [Deng, 2012] , FMNIST, KMNIST [Clanuwat et al., 2018] , Omniglot [Lake et al., 2015] , and NotMNIST as OOD data.Evaluation Metrics: We employed the area under the receiver operating characteristic (AUROC) and the false positive rate at 95% true positive rate (FPR95) as evaluation metrics. Results in FPR95 metric are provided in Appendix C.1.We utilize EfficientNet-b4 [Tan and Le, 2019] or ResNet50 [He et al., 2016] pre-trained on ImageNet [Deng et al., 2009] as our encoder. The main text presents results using EfficientNet-b4, while results using ResNet50 are detailed in Appendix C.2. For EfficientNet-b4, we select feature maps from the first to fifth stages (M = 5) to construct the multi-layer semantic features, resulting in a feature dimension (c) of 720. The LFDN is consisting of 16 residual blocks. Inside each residual block, the number of groups in Groupnorm and the intermediate feature dimension of the residual branch are set to 1 and 1440, respectively. We employ the AdamW optimizer with a weight decay of 10 -4 . Our method is trained on NVIDIA Geforce 4090 GPU for 150 epochs, with a batch size of 128 and a constant learning rate of 10 -4 throughout the training phase.Compared Generative-based Methods: In Table 1 , regarding CIFAR-10 as the ID dataset, we compare our method against pixel-level generative-based methods including GLOW [Serrà et al., 2019] , PixelCNN++ [Serrà et al., 2019] , VAE [Xiao et al., 2020], and DDPM [Graham et al., 2023] . To validate the effectiveness of LFDN, we implement a variant of our method through replacing LFDN with AutoEncoder in which MFsim is used for estimating the OOD score. In comparison with the best pixel-level method, VAE, our method achieves a 9.1% improvement in average AUROC when using MFsim for OOD score estimation. Compared to DDPM, our method variants show a significantly improvement in average AUROC. For example, when integrated with MSE, our method achieves 20.4% higher AUROC than DDPM. This indirectly indicates that performing OOD detection at the pixel level is much worse than performing OOD detection at the feature level. Generating pixels may reconstruct more content unrelated to the image's semantics, which may interfere the identification of OOD samples. Making the model focus on the reconstruction of compactly distributed semantic features benefits in separating ID and OOD samples. In terms of testing speed, our method is nearly 100 times faster than DDPM, significantly enhancing performance while reducing detection costs. Moreover, the final version of our method built upon LFDN improves average AUROC by 18.5% compared to the variant basd on AutoEncoder, as the diffusion model captures data distribution more effectively.In Table 2 , we compare our method with VAE, DDPM and AutoEncoder, using CelebA as the ID dataset. Our method integrated with MFsim achieves state-of-the-art performances, with an AUROC improvement of 19.89% compared to DDPM, and the performance of the remaining two metrics also far exceeds the baseline, demonstrating the generalizability of our approach.Compared Classification-based and Distance-based Methods: In Table 3 , we compare our method with classification-based methods including MSP [Hendrycks and Gimpel, 2016] , EBO [Liu et al., 2020] , DICE [Sun and Li, 2022] , and ASH-S [Djurisic et al., 2022] , as well as distance-based methods including 'SimCLR+Mahalanobis Distance' [Xiao et al., 2021] and 'SimCLR+KNN' [Sun et al., 2022] . All methods are evaluated using EfficientNet-b4 as the backbone. Compared to classification-based and distance-based methods, our approach consistently shows a clear advantage. Specifically, for CIFAR-100 as the in-distribution dataset, our method integrated with MFsim achieves an average AUROC of 13.84% higher than the classification-based method DICE. Moreover, unlike classification-based methods, our approach does not require labeled data.The inference speed of our method based on MSE or MFsim is faster than that of distance-based methods SimCLR+Maha and SimCLR+KNN, because the computation of covariance matrix or K nearest neighbors occupies part of time. Our method is also comparable to classifier-based methods including MSP, EBO, DICE and ASH-S. This demonstrates the effectiveness of leveraging the strong ability of diffusion models to reconstruct original distributions from different noise levels for reconstructing low-dimensional features and performing OOD detection. Performance variations across different sampling time steps: Figure 4 illustrates the variations in average AUROC and FPR95 values for different evaluation metrics at various sampling time steps, using CIFAR-10 as the ID data, with the final time step T = 100. It is observed that all metrics perform poorly at t = 1 primarily due to minimal noise added, making z t too similar to z 0 and thus, limiting the denoising capability of LFDN; both ID and OOD data are well reconstructed. As t increases to about 3-10 steps, the appropriate amount of noise allows MSE, LR, and MFsim to reach optimal performances. However, as t continues to increase, the difference between z t and the original z 0 enlarges, with z t gradually approaching random noise, thereby worsening the reconstruction differences between z0 and z 0 for both ID and OOD samples. Ablation study on LFDN network parameters. We conducted ablation experiments on two groups of parameters within the LFDN network: the dimension of the linear layers and the number of ResBlocks. For each experiment, we reduced one of these parameters to half of its original size while keeping all other parameters unchanged.  
conclusion and limitation: 
 In this paper, we propose a diffusion-based layer-wise semantic reconstruction framework for unsupervised OOD detection. We leverage the diffusion model's intrinsic data reconstruction ability to distinguish in-distribution and OOD samples in the latent feature space. Specially, the diffusion-based feature generation is built on top of the devised multi-layer semantic feature extraction strategy, which sets up a comprehensive and discriminative feature representation benefiting the generative OOD detection methods. Finally, we hope our proposed OOD detection method could make contributions to develop a safe real-world machine learning system. Additionally, it needs to point out that the performance of our method also relies on the quality of features extracted by the encoder. Therefore, selecting an encoder with strong feature extraction capabilities is crucial for achieving good performances. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Yes Justification:We have provided open access to our data and code, along with detailed instructions for reproducing the main experimental results. These resources are described in the supplemental material and can be accessed via the provided links.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Yes Justification: Our paper specifies all necessary training and test details, including data splits, hyperparameters, and optimizer settings. These details are provided in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
broader impacts: 
 Positive Societal Impacts: The proposed diffusion-based layer-wise semantic reconstruction method for unsupervised out-of-distribution (OOD) detection can significantly enhance the security and safety of machine learning systems. By effectively identifying OOD data, the system can prevent incorrect or potentially harmful decisions, making AI applications more reliable in critical areas such as healthcare, autonomous driving, and financial systems. This method increases the robustness of AI systems by ensuring they can handle unexpected inputs gracefully. This contributes to the overall stability and trustworthiness of AI deployments in various industries, thereby promoting wider acceptance and integration of AI technologies. Negative Societal Impacts: As with any advanced detection method, there is a risk that the technology could be misused. For instance, surveillance applications, it could be employed to monitor individuals without their consent, leading to privacy violations and ethical concerns.The MSE calculation, as shown in Algorithm 3, computes the mean squared error between the original and reconstructed latent features. It serves as a basic measure of reconstruction error for detecting OOD samples.The LR calculation, detailed in Algorithm 4, measures the reduction in reconstruction error by comparing the MSE values at the initial and final epochs of training. This metric reflects how well the model has adapted to the ID data over time, with a higher reduction indicating better adaptation.Algorithm 3 Testing Algorithm for MSE Calculation Table 1 : CIFAR-10 Dataset The CIFAR-10 test set consisted of 10,000 images. The SVHN dataset contained 26,032 images, LSUN-r had 10,000 images, and Fashion-MNIST, MNIST, and KMNIST each comprised 10,000 images. Omniglot included 13,180 images, and notMNIST had 18,724 images, totaling 97,936 OOD samples. The testing of the MFsim metric took a total of 98 seconds, with an average speed of 999.3 images per second.1: Input: An image x 2: Output: MSE score 3: z 0 = H(x) 4: z t ← ennoise(z 0 ) 5: z0 = denoise(z t , t) 6: MSE ← 1 N N i=1 (z 0 [i] -z0 [i]) 2 ▷ iTable 2 : CelebA Dataset The CelebA test set comprised 60,780 images, SUN included 10,000 images, iNaturalist had 100,000 images, Textures consisted of 1,678 images, and Places365 had 1,002 images, making up a total of 112,680 OOD samples. Testing the MFsim metric took a total of 109 seconds, processing an average of 1033.8 images per second.Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Yes Justification: We have reported error bars for our main experimental results, calculated as the mean and standard deviation over three runs. Details on the calculation of error bars and the factors of variability considered (such as train/test split and random initialization) are provided in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Yes Justification: The paper provides detailed information on the compute resources used for the experiments, including the type of compute workers (GPU), memory, and execution time. These details are specified in the experimental setup section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Yes Justification: We have thoroughly reviewed the NeurIPS Code of Ethics and confirm that our research conforms to these guidelines in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] YesJustification: We discuss the potential positive and negative societal impacts of our work in Section 7. Specifically, we address how our method could improve unsupervised out-ofdistribution detection, as well as the potential risks associated with misuse in surveillance applications.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] N/A Justification: Our paper does not involve the release of data or models that have a high risk for misuse. Therefore, this question is not applicable.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Both CIFAR-10 and CelebA datasets were trained for 200 epochs using the VAE model. The GLOW model was trained for 150 epochs with a learning rate of 5 × 10 -4 , and PixelCNN+ was trained for 150 epochs at the same learning rate. Under the DDPM model, both datasets were trained for 350 epochs, following the experimental setups and code provided in the original papers. We used LFDN without time-step embeddings as our autoencoder, used MFsim metrics, and kept all remaining training details consistent with our approach.We conducted tests to evaluate the FPR95 (False Positive Rate at 95% True Positive Rate) values using CIFAR10 and CIFAR100 datasets as in-distribution data while treating the remaining six datasets as out-of-distribution datasets. The specific FPR95 values are summarized in Table 5 . As shown in Table 5 , our method demonstrates a significant advantage in terms of FPR95 values compared to other classification-based and distance-based approaches. Specifically, when using CIFAR100 as in-distribution data, our method achieves an average reduction of 36.93% in FPR95 values compared to the state-of-the-art classification-based approach, ASH-S.Besides using EfficientNet-b4 as the encoder, we also employed the commonly used ResNet50 to extract multi-layer semantic features. For ResNet50, feature maps from stages 1 to 3 were selected, with channel counts of 256, 512, and 1024, respectively. These feature maps were concatenated to form a 1792-dimensional vector, which was then used as input for the LFDN. The results for three OOD detection metrics are presented in Table 6 and Table 7 . Both tables compare our method with classification-based and distance-based methods.As shown in Table 6 and Table 7 , when using ResNet50 as the backbone, our method still achieves the best performance. Specifically, with CIFAR-10 as the in-distribution dataset, the average AUROC and MFsim values are 98.30% and 8.89%, respectively, outperforming the classification-based SOTA method DICE by 6.17% in AUROC and reducing the FPR95 by 24.43%.Figures 6 and 7 illustrate the differences in the MFsim score distributions for various datasets, with ResNet50 as the encoder and CIFAR10 as the in-distribution dataset, across the first and last epochs. As shown in Table 8 , when CIFAR-10 is used as the ID dataset and CIFAR-100 as the OOD dataset, our method consistently achieves the best performance across different evaluation metrics, including AUROC and FPR95. Compared to classification-based methods, the improvement is not significantly large, but our approach still shows a consistent edge, particularly in feature-generative-based models, demonstrating the robustness of our method. The comparison between our method and DDPM [Graham et al., 2023] can be referred to Table 1 and Table 2 . Our method outperforms DDPM consistently on benchmarks using CIFAR10 or CelebA as ID data.The comparison between our method and Diffuard [Gao et al., 2023] is provided in Table 9 . Results of Diffuard are taken from its original paper. Here, CIFAR10 is regarded as ID data, while CIFAR100 or TinyImagenet is regarded as OOD data. Our method based on MFsim achieves overall better performance than 'Diffuard+Deep Ens', with 1.55 higher AUROC and 21.77 lower FPR95.The comparison between our method and LMD [Liu et al., 2023] is shown in Table 10 . The evaluation metric is AUROC. The average AUROC of our method based on MFsim is 6.94 higher than that of LMD.Table 9 : The AUROC and FPR95 values compared to DiffGuard [Gao et al., 2023] C.5 Comparison against other methods using the multi-scale feature encodings as the input.In Table 11 , we have made comparison of our method against AE and VAE using the multi-layer feature encodings as inputs. For AE (AutoEncoder), we use the LFDN network without the timestep embedding, i.e., a 16-layer linear network. For VAE, we use a 5-layer linear network as the encoder and an 8-layer linear network as the decoder. Compared to AE and VAE, the diffusion model has significant advantages when modeling complex multidimensional distributions.C.6 Comparisons with pixel-level denoising approaches.We provide the distribution differences of the MSE score and MFsim score at two levels after training, with CIFAR-10 as ID dataset and other datasets as OOD; The results are shown in Figures 8 and 9 .It can be observed that at the pixel level(DDPM), the reconstruction error distributions of ID and OOD samples are very similar. The mixed MSE scores make it very hard to distinguish ID samples from OOD samples. However, at the feature level, the reconstruction score distribution of ID samples shows a clear distinction from that of OOD samples. The reason is that, our feature-level diffusion-based generative model makes the projected in-distribution latent space not only be compressed sufficiently to capture the exclusive characteristics of ID images, but also provide sufficient reconstruction power for the large-scale ID images of various categories. In other words, the pretrained encoder has inherent generalization capabilities, and the multi-layer features it extracts are more discriminative than the high-dimensional pixels of the images themselves. In Table 12 , our method using MSE outperforms the classification-based SOTA method DICE, achieving an improvement of 3.91% in AUROC when ImageNet100 is used as the ID dataset and various datasets such as SUN, iNaturalist, Textures, and Places365 are used as OOD datasets. The significant improvements in performance metrics demonstrate that our generative-based approach can effectively model the in-distribution characteristics, leading to better OOD detection capabilities. This indicates that our proposed method is particularly suitable for more complex datasets like ImageNet100, where capturing detailed features is crucial for accurate OOD detection. The MSE Score NeurIPS Paper Checklist 


Decision: Dear authors,

This draft has received 1 Accept, 2 Weak Accept, and 1 borderline. After carefully reviewing the comments, the draft is being accepted for publication. Authors are encouraged to update the draft according to the suggestions and concerns mentioned by the reviewers.

Congratulations.

regards
AC

Reviews: review: paper_summary: paper_summary: This paper proposes a diffusion-based layer-wise semantic reconstruction strategy for unsupervised Out of Domain (OOD) detection. Specifically, they leverage the intrinsic data reconstruction ability of the diffusion model to differentiate between the In-distribution (ID) and OOD samples in the feature space. The features/data reconstruction at the multi-layer feature spaces helps the generative OOD detection.  The experiments suggest the superiority of the proposed approach compared to existing approaches.
strength_weakness: strength_weakness: 1- The paper needs a writeup revision, for example the contributions and idea of the paper is repeated multiple times unnecessarily.

2- The third step of the proposed approach, I.e., OOD detection step, is not clear. The authors defines three metrics for OOD detection based on some threshold, however, the paper never mentioned those values nor provided any experimental  study, that how were they selected.

3- The comparison with recent generative methods based OOD is not provided. Instead only VAE is results are compared, despite there are some approaches available ([Graham et al., 2023], [Gao et al., 2023] and [Liu  et al., 20]). 

4- A comparative analysis between the pixel level and feature level denoising is not performed, which is the base of the proposed approach. 

5-The authors needs to clearly indicate that how the proposed approach is better than the existing pixel level approaches, both theoretically and experimentally. 

6- The proposed approach, and most of the methods cited in this paper experimented with classification problems. However, they build  their motivation and need based on the natural scenarios. In natural scenarios, the input images may be more complex and the task may be more difficult than classification only. A study is required to evaluate the proposed approach on complex scenes, object detection and semantic segmentation datasets, having more information instead of having a single primary object.
questions: questions: The questions are there in the weaknesses section, specifically, experimental evidence of points 2-6 in weaknesses section will significantly improve the completeness.
limitations: limitations: The authors mentioned about the limitations related to backbone architecture. However, the large models pre-trained on large datasets are more generalised. So, how this generalisation will effect the models performance?
paper_summary: paper_summary: The paper addresses the problem identification of out of distribution models in an unsupervised manner. The idea is that OOD samples have the largest reconstruction error. The method is evaluated on a plethora of setups including SVHN [Netzer et al., 2011], LSUN(and variants) iNaturalist, Textures, Places365.
strength_weakness: strength_weakness: - at the core, the idea is that reconstruction error points to OOD. This is new, but is in a broader family (including AE, contrastive leraning), thus some limited
- the limitation sections points only to the strength of the encoder. However the assumption that OOD samples lead to large reconstruction error is also a prior assumption (although a strong one given the evaluation), but one may identify cases where is not true.
questions: questions: None. Questions I had (e.g. other models as encoder) have been answered in extended material.
limitations: limitations: not the case
paper_summary: paper_summary: The authors introduce a method for unsupervised out-of-distribution (OoD) detection for image classification tasks. Their approach is based on the semantic reconstruction of latent features in multiple layers of an image encoder using a diffusion model and does not require any OoD data for training. Unlike existing methods, this approach operates on the feature level instead of the pixel level, leading to more efficient OoD detection due to the lower dimensionality of feature encodings. Experiments on multiple established benchmarks, such as CIFAR10 vs. SVHN, LSUN, FMNIST, Omniglot, and CelebA vs. SUN, Textures, and Places365, demonstrate the effectiveness of the OoD detection approach with respect to the evaluation metrics AUROC and FPR95.
strength_weakness: strength_weakness: - The comparisons in the experiments could be extended to ensure fairness. The proposed diffusion and feature-based approach is compared against other types of generative models, such as GLOW and VAE. However, these generative models are only applied at pixel level and rely on image reconstruction error as the OoD score. In my opinion, a fair comparison would involve applying all generative models to the same input, i.e., the multi-scale feature encodings. Different generative models come with their own advantages and drawbacks, and a fair comparison could better show why the diffusion model is the best choice.
- The speed comparison is incomplete. As mentioned, the other generative models are only applied at pixel level, so the time comparison is based on different inputs. Additionally, it would be interesting to see the speed gap between the proposed method and classification or distance-based methods.
- The feature-based approach relies on the feature encodings of the image encoder. From the comparison of EfficientNet-b4 and ResNet50 as encoders, it appears that OoD detection performance benefits from stronger encoders (although this is not explicitly validated). From this perspective, it is unclear what the underlying model is for the classification or distance methods. For a fair comparison, these approaches should also use EfficientNet-b4 and ResNet50 as backbones.
- There are no images to show some qualitative results. Particularly, examples of failure cases would be interesting to see.
questions: questions: - In line 142ff: “Such multi-layer features could better rebuild the samples and encourage the ID semantic features distributed more compactly within a certain space from different semantic layers.” To me it is unclear what is meant by semantic features being encouraged to be distributed more compactly. As far as I understand, the features of the encoders are only extracted and not modified during the training of the OoD detection model.
- Another insightful metric to measure OoD detection performance could be the AUPRC, in particular if there are significant less OoD examples than ID examples. The numbers for AUROC seem saturated for couple of datasets.
- The OoD detection task with the given datasets seems rather simple. It would be interesting to see the proposed method in other settings, when the original task is more challenging. Are there limitations in terms scalability?
limitations: limitations: The authors have discussed limitations. Potential negative social impact have not been mentioned.
paper_summary: paper_summary: This paper focuses on out of distribution detection. The detection is based on the reconstruction error. The authors use a diffusion model as their generative model and focus on the feature space instead of the original images. The authors test their proposed methods on several different datasets.
strength_weakness: strength_weakness: There are several concerns for this paper.

1) The novelty of this paper is very limited. It looks like it simply replaces a generative model with the diffusion model.

2) The current version only presents the "what"---for example, the model includes three components (multi-layer semantic feature extraction, diffusion model for feature reconstruction, and OOD detection head)---but not the "why". More explanations and theoretical analysis may need.

3) The evaluation metrics may be problematic. Due to the natural of the OOD problem, the dataset would be highly imbalanced. In addition to the AUROC and FPR95, other metrics that could capture the imbalance characteristics should be used.

4) The current ID and OOD are very different (from different datasets), what would happen if the difference between ID and OOD is not so significant? For example, both ID and OOD come from the same dataset (e.g., CIFAR10) but OOD is curated by adding some distortions.
questions: questions: Basically, my major concerns include: 1) why it works? 2) the experiments. Please see [Weaknesses] for more details.
limitations: limitations: yes.

Comments: title: Queries addressed
Concerns addressed
comment: Thanks so much for your time on the rebuttal. I have carefully read it. Based on the rebuttal and other available reviews, I think most of my previous concerns have been addressed. I have raised my score accordingly.
I have revised my rating accordingly.
Thank you for your thoughtful and detailed answer!
Dear Reviewer Jgzd,

Thank you very much for reviewing our paper and providing valuable feedback. We sincerely hope that our responses can address all the remaining concerns. Thank you again for your great help and many good questions and suggestions, which largely help improve the quality of our paper. We would like to clarify if you have further concerns. 

We would like to clarify if you have further concerns. Thanks very much.
Dear Reviewer Jgzd,

Thank you very much for reviewing our paper and providing valuable feedback. We have made the best effort to address all your questions and comments. In particular, we have further clarified the rationale behind our proposed method and provided additional points to highlight its novelty. We have also introduced several metrics that can capture the characteristics of imbalanced datasets. Additionally, we demonstrated the detection performance of our method in scenarios where the ID and OOD datasets are similar, showcasing the scalability of our approach.

We sincerely hope that our responses can address all your concerns. Is there anything that needs us to further clarify for the given concerns?     

Thank you again for your hard work and thoughtful review.
Thank you for the detailed response.
Thank you for the detailed response. I have raised my score accordingly.