Fake It Until You Make It : Towards Accurate Near-Distribution Novelty Detection
Authors: Hossein Mirzaei
Mohammadreza Salehi
Sajjad Shahabi
Efstratios Gavves
Cees G. M. Snoek
Mohammad Sabokrou
Mohammad Hossein Rohban
Venue: ICLR 2023 poster (2023)

Abstract: We aim for image-based novelty detection. Despite considerable progress, existing models either fail or face dramatic drop under the so-called ``near-distribution" setup, where the differences between normal and anomalous samples are subtle. We first demonstrate existing methods could experience up to 20\% decrease in their AUCs in the near-distribution setting. Next, we propose to exploit a score-based generative model to produce synthetic near-distribution anomalous data. Our model is then fine-tuned to distinguish such data from the normal samples. We make quantitative as well as qualitative evaluation of this strategy, and compare the results with a variety of GAN-based models.  Effectiveness of our method for both near-distribution and standard novelty detection is assessed through extensive experiments on datasets in diverse applications such as medical images, object classification, and quality control. This reveals that our method significantly improves upon existing models, and consistently decreases the gap between the near-distribution and standard novelty detection AUCs by a considerable amount.

FAKE IT TILL YOU MAKE IT: TOWARDS ACCURATE NEAR-DISTRIN DORR-DISTRIBUTIN NOVELTY DETECTION
Hossein Mirzae', Mohammadreza Saijaa Shahabi .EEstratios Gavves
Cees G. M. Snoek?, Mohammad Sabokrou 1, Mohammad Hossein Rohban
1 Sharif University of Technology University of Amsterdam
3 Institute for Research in Fundamental Sciences (IPM)
4 Okinawa Institute of Science and Technology
ABSTRACT
We aim for image- based novelty detection. Despite considerable progress, existing models either fail or face a dramatic drop under the so-called "near-distribution" setting, where the differences between normal and anomalous samples are subtle. We first demonstrate existing methods experience up to 20% decrease in perfor- mance the near-distribution setting. Next, we propose to exploit a score-based generative model to produce synthetic ccar-distribution anomalous data. Our model is then fine-tuned to distinguish such data from the normal samples. We provide quantitative as well as qualitative evaluation of this strategy, and compare the results with a variety of GAN-based models. Effectiveness of our method for both the near-distribution and standard novelty detection is assessed through ex- tensive experiments on datasets in diverse applications such as medical images, object classification. and quality control. This reveals that our method considerably improves over existing models, and consistently decreases the gap between the near-distribution and standard novelty detection performance. The code repository is available at
INTRODUCTION
In novelty detection [ the goal is to learn to identify test-time samples that unlikely to come from the training distribution, without having access to any class label data of the training set (40). Such samples are called anomalous, while the training set is referred to normal. One has access to only normal data during training in ND. Recently, PANDA (34) and CSI (44) have considerably pushed ofate-of-the-art and achieved more than 90% the area under the receiver operating characteristics (AUROC) on the CIFAR- 10 dataset (23) in the ND task, where one class is assumed to be normal and the rest are considered anomalous. However, to achieve a similar performance in situations where outliers are semantically close to the normal distribution, e.g. instead of distinguishing dog vs. car, which is the regular ND, one desires to distinguish dog vs. fox in such settings. That is, they experience a performance drop when faced with such near-anomalous inputs. In this paper, our focus on such scenarios, which we call near novelty detection (near-ND). We note that Near-ND is a more challenging task and has been explored to smaller extent. Near novelty detection has found several important practical applications in diverse areas such as medical imaging, and face liveness detection (3I)
will show empirically, these methods struggle
Our first contribution which consists of ND problems whose normal and anomaly classes are either naturally semantically close, or else synthetically forced to be close. Fig. [ compares the performance of PANDA (34) and CSI (44) in instance of the near-ND, and the standard ND setups, which shows roughly 20% AUROC drop in near-ND compared with the ND. Furthermore, while MHRot (16) performs relatively comparable PANDA and CSI in ND, it is considerably worse in near-ND, highlighting the need for near novelty detection benchmarking.
benchmark eight recent novelty detection methods in the near-ND setting,
In the literature novelty detection and anomaly detection are used interchangeably. We use the term novelty detection (ND) throughout this paper.

---

A similar problem setup has recently been investigated in the out-of-distribution (OOD) detection domain, known as "near dfit-o--distibttioon" detection (9) where the in-distribution and out-of- distribution samples semantically similar. OOD detection and ND are closely related problems with the primary difference being that unlike OOD detection, the labels for sub-classes of the normal data are not accessible during training in ND, i.e. if the normal class is car, the type of car is given for each normal sample during training in OOD detection, while being unknown in ND. This makes anomaly detection more challenging problem than the OOD detection, as this side information turns out to be extremely helpful in uncertainty quantification (55). To cope with the challenges in the near-OOD detection, (9) (16), and (36) employ outlier exposure techniques, i.e., exposing the model to the real outliers, available on the internet, during training. Alternatively, some approaches 21 k2) utilized GANs to generate outliers. Such real or synthetic outliers are used in addition to the normal data to train the OOD detection, and boost its accuracy in the near-OOD setup.
In spite of all these efforts, the issue of nearly anomalous samples has not been studied in the context of ND tasks (i.e., the unsupervised setting). Furthermore, the solutions to the near-OOD problem are not directly extendable to the near-ND, as the sub-class information of the normal data is not available in the ND setup (37 40). Furthermore, the challenge in the case of ND is that in most cases, the normal data constitutes less conceptual diversity compared with the OOD detection setup, making the uncertainty estimation challenging. especially for the nearly abnormal inputs. One has note that some explicit or implicit form of uncertainty estimation is required for ND and out-of outto--distiibution detection. This makes near-ND an even more difficult task than near- near-OOD detection.
Apart from these obstacles in extending near-OOD solutions to the near-ND problem, we note that elements of these solutions, which are outlier exposure, and generating anomalous samples adaptively through GANs are both less effective for near-ND. It is well known that the performance of outlier exposure (OE) techniques significantly depends on the diversity and distribution shift of the outlier dataset that is used for the training. This makes it difficult for OE to be used in the domains such as medical imaging, where it is hard to find accessible real outliers. In addition, unfortunately, most GAN models suffer from (1) instability in the training phase, (2) poor performance on high-resolution images, and (3) low diversity of generated samples in the context of ND (39). These challenges have prevented their effective use in ND. To address the mentioned challenges, we propose to use a "non- adversarial" diffusion- based anomaly data generation method, which can estimate the true normal distribution accurately and smoothly over the training time. Here, our contribution is to shed light on the capabilities of the recently proposed diffusion models (43), in making near- distriiitiittoonn synthetic anomalies to be leveraged in the training of ND models. By providing comprehensive experiments and visualizations, we show that prematurely trained SDE-based model can generate diverse, high quality, and non-noisy outrio,, which considerably beat samples that are generated by GANs or obtained from the available datasets for tuning the novelty detection task. The importance of artifact- and foise-rree anomalous samples in fine tuning due to the fact that deep models tend to learn such artifacts as shortcuts, preventing them from generalization to the true essence of the anomalous samples. Finally, our last contribution is to show that fine-tuning simple baseline ND methods with the generated samples to distinguish them from the normal data leads to performance boost for both ND and near-ND. We use nine benchmark datasets that span a wide variety of applications and anomaly granularity. Our method achieves state-of state-of-the-arr results in the ND setting, and is especially effective for the near-ND setting, where we improve over existing work by a large margin of up to 8% in AUROC.
2 PROPOSED NEAR-NOVELTY DETECTION METHOD
We introduce a two-step training approach, which even can be employed to boost the performance of most of the existing state-of-the-as (SOTA) models. Following the current trend in the field, we start with a pre-trained feature extractor as 4 34) have shown their effectiveness. We use ViT ( backbone since (9) has demonstrated its superiority on the near ditt-f--distiibution detection. In the first step, fake dataset of anomalies is generated by a SDE-based diffusion model. We quantitatively and qualitatively show that the generated fake outliers are high-quality, diverse, and yet show semantic differences compared to the normal inputs. In the second step, the pre-trained backbone is fine-tuned by the generated dataset and given normal training samples through optimizing binary classification loss. Finally, all the normal training samples are passed to the fine-tuned feature extractor, and their embeddings are stored in memory, which is further used to obtain the k-NN distance of each test
2

---

Near hovelty Detect
Howertyy Datectio
AUC Drep
Training
100
9.1%
Sumples:
NO Model
90
18.2%
Testing
Anomaly
Normal
Your Anemedy
Far Ansezaly
60
12.8%
30
ND
Predicions
30
Assealy
DeepSVDD
MHRot
CSI
PANDA
Ours
Figure 1: Sensitivity of state-of-the-art anomaly detectors when differences between normal and anomalous samples are subtle. For each model, its performance in detecting far (Blue) and near (Orange) outliers has been reported. A single class of CIFAR-10 10 is considered as normal, the rest of the classes are considered far anomalies, and the semantically nearest class of CIFAR-100 to the normal class is the source of near anomalies. This is the class that corresponds to the lowest test AUROC in anomaly detection. considerable performance drop happens for the most recent methods, which means the learned normal boundaries are not tight enough and need to be adapted. For instance, while some might be able to distinguish cat image from a car, they are still unable to detect tiger or lion images anomalies.
input sample. This distance could serve as an anomaly score. The method is summarized visually in Fig. 2
2.1 PROPOSED PIPELINE
Step 1 : Near Anomaly Generation In order to generate near anomalous samples, basic diffusion model, called Stochastic Differential Equation (SDE) (43) adopted. the diffusion models, the gradient of the logarithmic probability density of the data with respect to the input is called the score. SDE model considers continuum distributions that develop over time accordance with diffusion process, corrupt sample and turn it into pure noise. This is called the forward process. The model trained to learn how to gradually transform the noisy image back to its original form through the score fucntion (backward process). Running the backward process allows to seamlessly transform a random noise into the data that could be used for sample generation as follows:
-n-1 2, [f g² (t)s t)]dt + g(t)du, .T (0,
(1)
where nn is the data sample at time n. f O and = of are called the drift and diffusion coefficients, respectively. sp t) is called the score function, a Brownian motion. The score function so (xx, t) is usually modeled by a U-Net, whose weights are 6, in x is an image. The loss function train this network is typically set enforce estimate V, log (x)):
expected to estimate V, lo (x). à represents
min (..)
(x))sss(x. () Va, log px
where A(t) is a positive weighting function to account for change in scale of the loss across different values of t. One has to note that the conditional likelihood pa (Ira 220) takes the form of Gaussian distribution, as a result accumulation of normal noises over time the forward process. For more details, (43)
3

---

Training
Testing
Near ceneeeeeeeet cooeesseenn..................... Stornd Treinag Sexples
Fiature Fine-Tuning
Feature C Collection
Test Samplees
Fao kaii Fosas Extucno
Tridine Set
Trocde trpit
Pastur Potmectr
Nacal
Worres Neightr Soarch
Extr Sugped
Reo Fustur Exnuar
Easuced Focaacce
Brasyy Clambioe
BCR
Nociaa Arnnaal
[Calletinng)
Trining Set Pesture
Figure 2: Overview of our framework for near distribution novelty detection. In the first step, the SDE diffusion model generates semantically close outliers, after its training early-stopped. As is shown, they have very subtle yet semantic differences with the normal data distribution (down- left). After that, using a linear layer, a pre-trained feature extractor fine-tuned to solve binary classification between the normal and abnormal inputs. This modifies the normal boundaries toward including more distinctive features. Finally, normal embeddings are stored and used compute the k-NN distance at the test time as the novelty score.
The main benefit of using score-based generative model is that in the backward process (Eq. D the score function denoises the input gradually that results in a relatively smooth decline in the Fréchet Inception Distance (FID) (17) across training epochs. This enables
stably produce
near anomalous samples based on the FID score of the generated outputs, which could be achieved by stopping the training process earlier than the stage where the model achieves its maximum performance. This is empirically assessed in Fig. left. Note that totally different training trajectory obtained GANs, where the FID oscillates and premature training does necessarily produce high-quality anomalous samples. That is why methods like OpenGAN rely on a validation outlier dataset determine where stop. In addition, Fig. 3 right, shows that the quality and closeness of the samples that are generated by the diffusion model are visually more appealing that the SOTA GAN-based models.
Step Feature Fine-tuning and Collection Having generated a high-quality fake dataset, a lightweight projection head is added to the feature extractor, and the whole network is trained to solve a binary classification task between the given normal and generated abnormal inputs. This way, normal class boundaries are tightened and adjusted according the abnormal inputs. After the training, all the normal embeddings are stored in the memory M that is used at the test time for assigning abnormality score given input. Note that due to the feature fine-tuning phase of our approach, the borders of the normal samples become normal semantic features.
thus better representing the
Step 3
Testing Phase At the test time, for each input x, its nearest neighbours are found in the M, which are denoted by m.. m² m m. Finally, the novelty score is found follows:
Novelty Score(æ) =
Σ 1 m
(3)
n=l
4

---

2.2 DIFFUSION MODELS VS. GANS
Here, we provide some rationale on the preference of diffusion models over GANs for the the task of near-ND. Although diffusion models showed great promise in image-based generative models in recent years, some flavors of GANs, like StyleGAN-XL (42), still outperform diffusion models even in small datasets such as CIFAR-10. Therefore, is not immediately clear, which class of generative models should be applied for the near-ND task.
We note that in diffusion models, such DDPM, the model trained by optimizing the Evidence Lower Bound (ELBO) (18). That is, the objective function is to minimize an upper bound on the negative evidence:
E Bg log po
where and po are the original and estimated distribution the data, respectively. Here, stands for the denoiser parameters in the DDPM. But, note that maximizing the evidence is equivalent with minimizing the Kullback-Leibe (KL) divergence between q and ps:
(o
} + H1 D(g po) ((x)
E Eg log po (x)) Eawg
log
(5)
Therefore, if the gap in the ELBO bound is small, minimizing the DDPM loss is equivalent with making the estimated generative distribution pe close to the original data distribution Hence, expect the estimated generative distribution pe ends up sufficiently close if the training is prematurely stopped, and therefore generates "near-distribution" patterns. It should be noted that variational autoencoders (VAEs) also follow the same logic, but because of their limited capacity, which stems from their low dimensional latent space, fail to make the KL-divergence D(g pe small enough (see Table for the comparisons). One may also wonder why GANs do exhibit the
Figure Left: Unlike GAN-based models, the diffusion SDE model performs much better in consistently improving the fidelity and diversity of the generated samples along the training. The metries "density" and "coverage" (27) are used to reliably measure these concepts. Thus, obtaining high-quality and diverse anomalous samples is feasible by early stopping in the SDE model. Right: different datasets as well as their corresponding normal samples shown. As be seen SDE samples, in the third row, are authentic (differ from the normal class), high-quality, and have minimal differences to the normal class, while other methods produce low quality samples that are far away from the normal distribution.
Some generated samples by various methods
5

---

same property. It is also well-known that if one uses optimal discriminator for given generator, the generator loss becomes proportional to the Jensen-Shannon Shannon divergence of the original and estimated distributions (13). However, this does not happen in GANs, due to practical comsiderations, as the discriminator and generator are alternatively updated through SGD. Hence the discriminator not necessarily optimal in each step. Therefore, early stopping does not necessarily lead to near-distribution generators. Our empirical results also confirm that various quality metrics, such as coverage and density 28), which were previously proposed evaluate synthetic images' fidelity and diversity, where larger values that are close to 1.0 indicate better fidelity and diversity, respectively. The detailed information can be found in the Appendix 7.1 Theses metrics decline or fluctuate significantly for GANs along the training. However, such metrics exhibit steady and consistent increase till the estimated distribution po gets sufficiently close to the true distribution q (see Fig. left).
EXPERIMENTAL RESULTS
In this section, we compare our method against the SOTA in the ND and near-ND setups. All the experiments performed 10 times and the average result is reported. Training details and the full dataset descriptions are provided the Appendix 7 We compare our approach with current top sulf-super and pre-trained feature adaptation methods in the literature (44 34 B3 7 16). Results already reported in the original papers were copied. Also, if the results were not reported in the original papers, we the experiments whenever possible.
Our experiments are organized into three categories and are presented separately in sub-tables. The following subsections will provide overview of each category.
Additionally, we considered novelty detection multi-class unlabeled settings, which you can see in Appendix 9.3
3.1 REGULAR ND BENCIIMARK
Following previous works, we evaluate the methods on the standard ND datasets. According to the standard ND protocol, multi-class datasets are first converted into an ND task by setting a class as normal and all other classes as anomalies. This is performed for all classes, in practice turning single
dataset with C' classes into datasets 33 34) The datasets that are tested this setting include CIFAR-10 (23). CIFAR-100 (23), MVTecAD (3), Birds (46), Flowers Table aa) provides the
results of the case, which indicates our superiority.
3.2 FINE-GRAINED DATASETS FOR NEAR-ND BENCHMARK
To evaluate the performance on near-ND benchmark, we use 4 different datasets that naturally show slight differences between their class distributions. These datasets are not commonly used in ND but used une-ggrined image classification segmentation, containing hard-to -initt--dis object classes. Thus they are suitable for benchmarking near-ND). The datasets that are tested in this setting include FGVC-Aircraft (25), Stanford-Cars (22), White Blood Cell (WBC) (49), and Weather (I0). Table (b) provides the results in this category, which again shows that our method outperforms SOTA by roughly 8 percentage points of AUROC the mentioned Near-ND setup.
3.3 PROPOSED DATASETS FOR NEAR-ND BENCHMARK
To further show the effectiveness of approach, propose the evaluation procedure represented in the [.). This setup includes CIFAR-IO-FSDE, CIFAR-IO-SSGAN, and CIvAR- which are described in more details below.
3.3.1 CIFAR- 1 00 BENCHMARK
In this setup, each class of CIFAR- 10 (C! and its closest class in CIFAR-100 (C)00) are selected, to represent normal and abnormal classes, respectively, where and j are class numbers that are less than 10 and 100, respectively. Then, the model is trained the training samples of (C!") and

---

tested against the aggregation of (C) and (C;") test sets. As is shown, almost all the methods considerably lag behind our method by large margin.
3.3.2 CIFAR-I0-FSDE BENCHMARK.
To show the quality of the proposed generated dataset, we evaluate the performance of some of the most outstanding ND methods when the generated dataset is considered as anomalous at the test time. We call the dataset CIFAR-IO-FSDE In this setting, one class of CIFAR-10 is selected as normal and the test set is made using the corresponding test samples of the normal class and random subset of the fake dataset, as abnormal samples, with the same size as the normal class.
3.3.3 RIFAR-1O-FSGLA BENCHMARK.
The anomalous data that generated by StyleGAN-ADA contain lots of artifact-full and distorted samples that practical novelty detector is expected to detect. Table (c) compares our method with other SOTA methods in detecting such samples. Due learning more distinguishing features, we pass the SOTA by 12 percentage points in the AUROC score, showing the applicability of our approach in detecting wide range of anomalies.
Table I:
The performance of novelty detection methods (AUROC %) in near-ND and ND on various datasets. Our method outperforms SOTA by significant margins on all settings. Note that except CSI, all the other SOTA approaches employ pre-trained architecture, particularly Transformaly (7), that utilizes a ViT-backbone.
(a) Comparison of AUROC of SOTA proposed method in the regular ND setting.
Datasets
From Scratch
Pre-trained
CSI
PatchCorc
PANDA (ResNet-152)
MSAD (ResNet-152)
MSAD
Transformaly (VIT-R_16)
Ours (ViT-B 16)
(ResNet-18)
(WideResNet50)
(ViT-B_16)
Birds
52.4
58.1
95.3
96.7
93.3
97.8
98.5
CIFAR-10
94.3
67.2
96,2
97.2
94.1
98.3
99.1
CIFAR-100
89.6
64.1
94.1
96.4
93.0
97.3
98.1
Flowers
60.8
74.8
94.1
96.5
98.6
99.9
99.9
MvTecAD
63.6
99.6
87.2
85.5
87.9
86.4
For greater comparabilityy, MSAD was also reimplemented with VTT-backbone.
(b) Comparison of AUROC of SOTA to our proposed method in the natural near-ND setting.
Datasets
From Scratch
Pre-trained
CSI (ResNet-18)
PANDA (ResNet-152)
MSAD (ResNet-152)
MSAD" (VTT-B 16)
Transformaly (VTT-B 16)
Ours (VTT-B 16)
FGVC-Aircraft
64.6
77.7
79.8
81.3
84.0
88.7
Stanford-Cars
87.1
85.7
86.7
WBC
50.4
87.4
87.0
83.6
85.1
91.2
Weather
91.5
81.5
92.4
85,9
94.3
97.0
For greater comparability, MSAD was also reimplemented with VIT-backbone.
(c) The performance of the ND methods compared to ours in AUROC % in the proposed near-ND setting.
Setting
Dataset
From Sertch
Pre-trained
DeepSAD
Opesslas-Opestii
Transfomaaly
CIFAR- 10vs100 HIPAR-10-FSDD.
50,0
62.6
63.1
76.1
50.0
60.7
58.5
76.8
79.5
82.3
90.0
nea--ND
53.5
60.5
64.5
87.4
38.1
$2.0
61.7
57.7
64.9
75.0
96.4
CIFAR- 10.RR-10PSSGAAA 10-PSGAN
61.8
70.4
72,8
83.1
64.4
55.4
71.3
66.1
69.4
82.2
95.1
Require Exposure Dutuets.
Ohe-clas-OPenGANN represents the OpenGAN wdspted for our setting.
While Transformaly, PANDA, and CSI achieve very close results in the ND setup (see Table ] (a)), they work significantly worse in the near-ND setting. To be specific, most of the methods significantly fail to detect generated fake samples, which is particularly the case for PANDA (34) with 40 percentage points performance drop. Surprisingly, DeepSAD with roughly 30 percentage points performance decrease does not work decently despite being exposed to real outliers during

---

the training process. To further validate method, we adapt OpenGAN, SOTA OOD detection, to the ND setting and report its near-ND results. Surprisingly, even by employing fake abnormal samples generated by GAN and being exposed to real outliers, it is not able to achieve decent performance in the near-ND setting (Table (e)), which again supports the benefits of our proposed approach for generating near distribution abnormal samples. All of these results highlight the need of benchmarking previous ND methods in the near-ND setup.
4 ABLATION STUDIES
Sensitivity to Backbone Replacements. Table 2 shows our method sensitivity the backbone replacements. All the backbones are obtained and used from (9). The results are provided for both the ND and near-ND evaluation setups. For each backbone, the performance with and without the feature fine-tuning phase based on the generated data is reported (denoted as "Pre-trained" and "Fine-tuned" in the table), showing a consistent improvement across different architectures. The performance boost is surprisingly large for some backbones such as ResNet- 152, ViT-B 16, and R50+ViT-B roughly 16%, 25%, and 36% boosts in AUROCs in the near-ND setting. Moreover, ViT-B 16 and R50+ VIT-B 16 perform roughly 8% and 16% better after the fine-tuning in the ND setup, showing the generality of our approach regardless of the setting or backbone.
with
Table 2: The effectiveness of our approach over different backbones (in AUROC %) in the ND and near-ND settings. The results show consistent improvements regardless of the backbone setting that is used.
Setting
Dataset
Training
Models
ResNet-152 152
VTT-B_
VTT-B_ 32'
R50+ VIT-B_ 16'
ConvNeXt-B'
Pre-trained
92.5
91.0
95.3
82.2
97.1
ND
CIFAR-10
Fine-tuned
95.3
99.1
98.3
98.8
97.8
Pro-trained
58.5
65.5
66.3
50.0
71.5
near-ND
IIFAR-10vs100
Fine-tuned
74.7
90.0
78.9
85.9
81.1
Pretrained on ImageNet K1
Pretrained on ImageNet 2 and fine-tuned on ImageNet IK.
SDE vs. Others. In this experiment, the effect of using different generative models is explored. All the experiments are done in the one-vs-all setting. Table 3 shows clear superiority of using a SDE-based model compared to the well-known generative models (20 42 5 (4) with 2% 20% better AUROCS based on the chosen dataset. Evidently, employing our fake samples generated by the SDE is almost always beneficial as opposed the ones generated by the other models. Interestingly. other models generated samples are harmful to the performance in Weather and MvTecAD datasets. Having artifacts, mode-collapse, and undiversified generated samples could be the reasons behind this observation, which are shown in AppendixTT This highlights that not all the generative models are beneficial for the ND and near-ND tasks, and they need be carefully selected based on the constraints defined in such domains.
Table 3: Comparison of our model performance (in AUROC %) upon using different generative models for each class of the CIFAR-10. Weather, WBC and MvTecAD dataset. The performance of the base backbone is also included to measure the effectiveness of the training process. Diffusion- based data generation considerably passes the GAN-based SOTAs in the novelty detection task. Coverage& Density metrics are calculated according to the dataset and generative model. These metries measure the diversity and fidelity of the samples that are generated by generative models.
Dataset
Matrie
Buse BackDone
Generative Model
totheut tile daza
Baseline- VAE
Baseline Gan
BigGIAN
SeyleGAN-ADA
StyleGAN-XL
Demsellow
SDB
CIFARIO
AUROC
91.0
93.8
04.5
96.9
97.8
96.5
96.4
99.1
Desdly MensityCooerag
(0.53740473)
(0.6600155))
(075211.76)
(0.81400885)
(0.766/1668)
(0. 7040066635)
(8.72219944
Weather
AUROC
86.5
69.6
$3.7
$5.3
32.4
76,8
$1.9
97,0
(0.5830.463 453)
(0.642/0661)
(0640001435)
(6.7820.765)
(165581.514)
(0.6280.741)
(0.2590.796)
WBC
AUROC
$4.6
$2.0
91,2
Desiiy&Coazage
(0.00200364)
(0.7 (991990443)
(0.5350) (84)
(0.8100)261)
(1.6320)5775)
(0.7540882))
(68860.873)
McTesAD
AUROC
65.8
80.5
78.3
$6.4
Dendty&Cossmgs
00.908404403 4007)
(0.641/81675)
0468841555 587)
(0.8260.641)
(0.42301.644)
(0.761/0555)
(8.98811833)
8

---

Sensitivity to k-NN and Stopping Point. Table 4 shows the performance stability with respect to the number of nearest neighbours for both the ND and near-ND setups. Clearly, the method is barely sensitive this parameter. DN2 (2) has provided extensive experiments of applying k-NN to the pre-trained features for the AD task, claiming k = 2 is the best choice. We also observe that the same trend happens in our experiments. Similarly, the method is robust against the stopping point with at most 2% variation for reasonable FID scores, making it practical for the real-world applications.
the effectiveness
Table The sensitivity of the proposed method the k-NN parameter and training stopping point based on the FID score.
Setting
Dataset
KNN
Stopping Point
k=1
k=2
k=5
k=10
k=50
300<FID
100<FID<200
30<FID<50
FID<20
ND
CIFAR-IE 10
99.0
99.1
98.9
98.9
98.7
96.8
97.7
99.1
92.6
ncar-ND
CIFAR-IOvs100
89.8
90.0
90.0
90.0
89.7
82.7
87.1
90.0
68.2
RELATED WORK
Outlier Exposure Based Approaches Utilizing the fake data for the novelty detection task has previously been considered. The general idea is to employ synthetic images, which may be generated by GANs, augment the training set (6) 26 50 29 D. In the case of open-set recognition, (21) proposed OpenGAN, which adverserially generates fake open-set images. The discriminator is then utilized at the test time for the novelty detection. The key point in OpenGAN that tiny additional dataset, containing both in- and outliers, is used as a validation set for the model selection. This, also known as outlier exposure, is necessary due to the GAN unstable training. Even though these models can work well on simple datasets, they cannot handle more complex datasets and cannot detect hard cinddiittiibttin anomalics.
Self-supervised learning Many studies have shown that self-supervised methods can extract meaningful features that could be exploited in the anomaly detection tasks such as MHRot (16). GT (12) uses geometric transformations including flip, translation, and rotation to learn normal features for anomaly detection. Alternatively, puzzle solving (38), and Cut-Paste (24) are proposed in the context of anomaly detection. has recently been shown that contrastive learning can also improve anomaly detection (44), which encourages the model to learn normal features by contrasting positive and negative samples.
Pre-trained Methods Several works used pre-trained networks anomaly detectors. Intuitively, abnormal and normal data do not overlap in the feature space because many features model high-level and semantic image properties. So, in the pre-trained feature space, one may classify normal vs. anomaly. (2) used the k-nearest neighbors distance between the test input and training set features as an anomaly score. (48) trained GMM on the normal sample features, which could then identify anomalous samples as low probability areas. PANDA (34) attempts to project the pre-trained features of the normal distribution to another compact feature space employing the DSVDD (35) objective function. In 7 4 41). the pre- pre-trained model is a teacher and the student is trained to mimic the teacher's behavior solely on the normal samples.
CONCLUSION
In this paper, we revealed a key weakness of existing SOTA novelty detection methods, where they fail to identify anomalies that are slightly different from the normal data. We proposed to take non-adaptive generative modeling that allows controllable generation of anomalous data. turns out that SDEs fit this specification much better compared to other SOTA generative models such as GANs. The generated data levels up pre-traines distance-basee novelty detection methods not only based on their AUROCs in the standard ND setting but also in the near-ND setting. The improvements that are made by our method are consistent across wide variety of datasets, and choices of the backbone model.

---

REFERENCES
[1] Mohammad Azizmalayeri, Arshia Soltani Moakhar, Arman Zarei, Reihanch Zohrabi, Moham- mad Taghi Manzuri, and Mohammad Hossein Rohban. Your out-of-distribution detection method is not robust! arXiv preprint MxXi::229.112444 2022.
[2] Liron Bergman, Niv Cohen, and Yedid Hoshen. Deep nearest neighbor anomaly detection. arXiv preprint arXiv:2002.10445, 2020.
[3] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Mvtee ad-a comprehen- sive real-world dataset for unsupervised anomaly detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9592-9600, 2019.
[4] Paul Bergmann, Michael Fauser, David Sattlegger, and Carsten Steger. Uninformed students: Student-teacher anomaly detection with discriminative latent embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4183441344 4192, 2020.
[5] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 190999999111,,,,.111 2018
[6] Pierrick Chatillon and Coloma Ballester. History-based anomaly detector: An adversarial approach to anomaly detection. In Proceedings of SAI Intelligent Systems Conference, pp. 761-776. Springer, 2020.
[7] Matan Jacob Cohen and Shai Avidan. Transformaly-two (feature spaces) are better than one. arXiv preprint arXiv: /2.04188, 2021.
[8] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19299 2020.
[9] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of duiti---ist detection. Advances in Neural Information Processing Systems, 34, 2021.
[10] A Gbeminiyi. Multi-class weather dataset for image classification. Mendeley Data, 2018.
[11] Zong Yuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi-class open set classification. arXiv preprint arXiv: 170.00777,, 2017.
[12] Izhak Golan and Ran Yaniv.. Deep anomaly detection using geometric transformations. Advances in neural information processing systems, 31, 2018.
[13] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, (3(11):139-144 2020.
[14] Matej Greié, Ivan Grubišić, and Siniša Šegvić. Densely connected normalizing flows. Advances in Neural Information Processing Systems, 01223688-23988 2021.
[15] Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of-distributtin examples neural networks. arXiv preprint 16X0v0:6100.00 2016.
[16] Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-supervised learning can improve model robustness and uncertainty. Advances in Neural Information Processing Systems, 32, 2019.
[17] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems. 30, 2017.
[18] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
10

---

[19] Minguk Kang, Joonghyuk Shin, and Jaesik Park. Studiogan: A taxonomy and benchmark of gans for image synthesis. arXiv preprint arXiv:2206.09479 2022.
[20] Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in Neural Information Processing Systems, 1111214--2214 2020.
[21] Shu Kong and Deva Ramanan. Opengan: Open-set recognition via open data generation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 813-822, 2021.
[22] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
[23] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[24] Chun-Liang Li, Kihyuk Sohn, Jinsung Yoon, and Tomas Pfister. Cutpaste: Self-supervised learning for anomaly detection and localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9664-9674, 2021.
[25] S. Maji, Kannala, E. Rahtu, M. Blaschko, and Vedaldi. Fine-grained visual classification of aircraft. Technical report, 2013.
[26] Hironori Murase and Kenji Fukumizu. Algan: Anomaly detection by generating pseudo anomalous data via latent variables. arXiv preprint arXiv:2202.10281, 2022.
[27] Muhammad Ferjad Nacem, Scong Joon Oh, Youngjung Uh. Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metrics for generative models. CoRR, abs/2002.09797, 2020.
URL httttt::///arttttt http:///aaxxiv.ooraa///////// 20wp. 09797
[28] Muhammad Ferjad Naeem, Scong Joon Oh, Youngjung Uh, Yunjey Choi, and Jaejun Yoo. Reliable fidelity and diversity metries for generative models, In International Conference on Machine Learning, pp. 7176-7185 PMLR, 2020.
[29] Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng- Keeg-Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 613-628, 2018
[30] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008.
[31] Daniel Pérez-Cabo, David Jiménez-Cabello, Artur Costa-Pazo, and Roberto J López-Sastre. Deep anomaly detection for generalized face anti-spoofing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
[32] Masoud Pourreza, Bahram Mohammadi, Mostafa Khaki, Samir Bouindour, Hichem Snoussi. and Mohammad Sabokrou. G2d: generate to detect anomaly. In Proceedings of the IEEF/CVF Winter Conference Applications of Computer Vision, pp. 2003-2012, 2021.
[33] Reiss and Yedid Hoshen. Mean-shifted contrastive loss for anomaly detection. arXiv preprint arXiv:2106. 200.:2200000844, 2021.
[34] Tal Reiss, Niv Cohen, Liron Bergman, and Yedid Hoshen. Panda: Adapting pretrained features for anomaly detection and segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2880-28144 2021.
[35] Lukas Ruff, Robert Vandermeulen, Nico Goernitz, Lucas Deecke, Shoaib Ahmed Siddiqui, Alexander Binder, Emmanuel Müller, and Marius Kloft. Deep one-class classification. In International conference machine learning, pp. 4393 4402.. PMLR, 2018.

---

[36] Lukas Ruff, Robert A Vandermeulen, Nico Gömmizz, Alexander Binder, Emmanuel Müller, Klaus-Robert Müller, and Marius Kloft. Deep semi-supervised anomaly detection. arXiv preprint arXiv: 190.0099960009944 02694, 2019.
[37] Lukas Ruff, Jacob R Kauffmann, Robert A Vandermeulen, Grégoire Montavon, Wojciech Samek, Marius Kloft, Thomas G Dietterich, and Klaus-Robert Müller. A unifying review of deep and shallow anomaly detection. Proceedings of the IEEE, 109(5):756-75)) 2021.
[38]
Mohammadreza Salehi, Ainaz Eftekhar, Niousha Sadjadi, Mohammad Hossein Rohban, and Hamid R Rabiee. Puzzle-ae: Novelty detection in images through solving puzzles, arXiv preprint arXiv:2008. 22959, 2020.
[39] Mohammadreza Salehi, Atrin Arya, Barbod Pajoum, Mohammad Otoofi, Amirreza Shaeiri, Mohammad Hossein Rohban, and Hamid R Rabiee. Arae: Adversarially robust training of autoencoders improves novelty detection. Neural Networks, 144:726-736, 2021.
[40] Mohammadreza Salchi, Hossein Mirzaei, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. A unified survey on anomaly, novelty, open-set, and out-of-distribution detection: Solutions and future challenges. arXiv preprint arXiv:2110.1405:: 14051, 2021.
[41] Mohammadreza Salchi, Niousha Sadjadi, Soroosh Baselizadeh, Mohammad Rohban, and Hamid R. Rabiee. Multiresolution knowledge distillation for anomaly detection. In Proceedings of the /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14902- 14912, June 2021.
[42] Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 Conference Proceedings, pp. 1-10, 2022.
[43] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
[44] Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning on distributionally shifted instances. Advances in neural information processing systems, 13:1999-11882 2020.
[45] Sagar Vaze, Kai Han, Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. CoRR, abs/21 10.06207, 2021. URL. httph:tt/p/aaatt lorg/bbs/ 2110. 06207
[46] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd
dataset. 2011.
[47] Jim Winkens, Rudy Bunel, Abhijit Guha Roy, Robert Stanforth, Vivek Natarajan, Joseph R Led- sam, Patricia MacWilliams, Pushmeet Kohli, Alan Karthikesalingam, Simon Kohl, et al. Con- trastive training for improved out-of-distributi detection. arXiv preprint arXiv:2007.05560 2020.
[48] Zhisheng Xiao, Qing Yan, and Yali Amit. Do really need to learn representations from in-domain data for outlier detection? arXiv preprint arXiv:2105. 09270, 2021.
[49] Xin Zheng, Yong Wang. Guoyou Wang, and Jianguo Liu, Fast and robust segmentation of white blood cell images by self-super learning. Micron, 107:55-71, 2018.
[50] DeiWei Zhou, Han-Jia Ye. and De-Chuan Zhan. Learning placeholders for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2021.
12

---

Appendix
EXPERIMENTAL SETTINGS
We use a ViT-B_ 16 as the feature extractor (pretrained on ImageNet 21k), learning rate = 4e-4, weight decay Se-5, batch size = 16, optimizer= SGD, and linear head with 2 output neurons. We freeze the first six layers of the network, and the rest fine-tune tine-tuned till convergence for all the experiments. For the data generation phase, setup default Syper-parameters e.g. we used predictor-correctt as a sampler. The inputs are resized to 224 224, and rule of thumb, the data generation is stopped FID 40 and FID = 200 for low and high-resolution datasets. All the results our tables are either reported from the reference papers or run by using their official repositories.
the SDE-based model explained in (43) with exactly the same
7.1 DENSITY & COVERAGE
The density and coverage metrics can be used to assess the diversity and fidelity of generative models, respectively. The method measures the distance between the generated and real images using manifold estimation procedure code (19). In order to exploit feature representations for these metrics, features prior to the final classification layer mathematical expression for these metrics:
pre-trained model are used. The following is
Density (X,, X,,
(fas B (f.., (F..). (6)
RM
N N
Coverage (X,, X,, F,k)
Σ (Bjst. Jog E B (Sa,, NN,
(7)
where a feature extractor, f a collection features from F.
X. (Ie.1s..... N,,N) denotes real images, X; (.r,,....... Dt,M) denotes generated images, B(f,r) r) is the n-dimensional sphere in which f = F(x) E R" is the center and r is the radius, fF. f. k) is the distance from to the k-th nearest embedding in F, and I(-) is indicator function. We use the standard Inception V3 features, which are also used to compute the FID. The
measures computed using the official code 28
7.2 OOD vs. ND
The common objective of OOD detection and novelty detection is to detect anomalies. Since they share the same goal, they may be considered the same. However, OOD detection an extra piece of information is present, which is the class information of each normal training sample. This extra information makes the techniques for the OOD detection and novelty detection quite different. For instance, could achieve satisfactory detection rates in OOD, by simply training model to classify normal classes, and threshold the maximum softmax probability (13). This simple remedy is unfortunately impossible for novelty detection, where the training of the base classifier is impossible due to non-existence of this extra class information. Therefore, OOD detection methods cannot simply be used to solve the novelty detection problem. In light of this, the literature has categorized these two problems as being different.
7.3 ANALYSIS OF STOPPING POINT
The stopping point of the SDE determined based on the ideal FID, i.e., once the SDE has converged on the training data, the FID is computed. We then set a stopping point equal factor (5) of the ideal FID as a rule of thumb. E.g.. in the CIFAR- 10, SDE achieves FID as 8. Consequently, we set the stopping point at FID zs 40 for low-resolution. The reason that FID is higher than the previously
13

---

reported FID (i.e. 2.2) is that we only have class, resulting in smaller training Also, set up an ablation study on stopping point sensitivity in table indicating that the method relatively robust against limited variations in stopping points and the results not fragile. It is important to note that the synthesized images were not an effective anomaly in the first training procedure hecause they extremely noisy. Furthermore, learning artifacts from their input, instead of paying attention to the general semantics of the input. In addition, once the model converges to the real distribution, the images generated become similar to the normal distribution, which disallows using these images as anomaly samples. Therefore, it seems reasonable that performance results in the first and last phases of the training procedure would be poor. figure illustraaees the mentioned trend (37
has to note that current deep models are highly capable of
AUROD%
50
15
AUROC%
300
250
200
FID
100
75
Figure 1: Spectrum of AUCs, based on Stopping points for WBC dataset and CIFAR1O-vs setting, as proxies for low- and high-resolution data, respectively.
8 NEAR NOVELTY DETECTION DEFINITION
To provide standard benchmark for the near-ND task there is need to define a one-class distribution closeness score. Suppose K K-class training dataset given from which a normal class C, randomly sampled and used to train the model M. The closest abnormal distribution with respect to the selected normal class for M can be defined as C, # C, that minimizes the test time performance when considered as the abnormal distribution. We call the performance in the mentioned scenario the bottom-I score of class C, with the backbone M.. However, this score depends on the choice of M, making it specific for every problem setup and method. Therefore, inspired by the CLP criterion introduced in (47) as a measurement of dataset distance, we could use CLP as an alternative closeness score for the novelty detection task.
Given a K -lalass training dataset, one category is randomly selected as the normal distribution. Then, supervised classifier P trained the rest K abnormal categories. Assume be training
14

---

sample of the selected normal class. The closeness score of each abnormal class with respect the normal class obtained as follows:
Closeness score;
P(g = i|.r).
(8)
NNNommaal Class
The higher the closeness of abnormal category, the similar it is the normal class. The same situation also holds when abnormal categories are selected from another dataset. Fig. in the Appendix indicates a decent correlation between the bottom-1 score and closeness score, implying that our proposed criterion can be considered as proxy for the ideal score.
NEAR-ND
9.1 COMPARISON BETWEEN BOTTOM-1 AND CLP
section [] we proposed the closeness score (CLP) to find the closest abnormal class for each class of the normal dataset. In experiments, used the ViT-B 16 (pretrained on ImageNet-21 as the backbone used for extracting the closest abnormal classes based on this criterion. The bottom- abnormal class is the class that has the lowest novelty detection performance during the testing phase. Note that the abnormal classes chosen based on the CL. criterion do not necessarily have the worst novelty detection performance; therefore, it expected for the novelty detection methods to perform better on the CLP criterion. This section aims to investigate how well these two criteria match each other. In the Table 5 for each novelty detection model and every class of the CIFAR-I0 dataset, the bottom-1 1 class is shown. Furthermore, the last row indicates the closest abnormal class selected based on the CLP criterion. As shown the Table the extracted classes based on the CLP criterion are the same conceptually similar to the respected bottom-1 classes. Figure 2 also shows decent correlation between the bottom-1 score and the CLP score, which means both these criteria could be used to extract near-distribution abnormal classes, and that the CL. could be used as a proxy to the bottom-1 I criterion. Also, The Pearson correlation coefficient between these two results is 0.843, indicating strong positive relationship.
AUROC
The performance of novelty detection methods (AUROC %) in the near-ND and ND settings. In the near-ND setting, results reported according to both criteria, i.e. CLP and bottom-1.
Figure 2:
9.2 BOTTOM-i CLASSES AS THE ABNORMAL DISTRIBUTION
The Bottom-i means averaging the AUROCs for the abnormal classes that have the lowest AUROCs, e.g. bottom- 100 in the case of CIFAR-100 denotes averaging the AUROC results for all 100 classes.
15

---

Table 5: For each model and every class of the CIFAR-IO, the anomalous class that minimizes the anomaly detection performance among 100 classes of the CIFAR- 100 dataset is reported. The last row is the CIFAR- 100 class that achieves the highest CLP score. Note that the latter does not depend on the anomaly detection method.
Arglone
Figg
Pees5V130
Plain
Plats
Plain
Cloud
Rountain
Plain
Flain
Plain
P1ain
G7
Mountate
Pickog_Truck
C1me1
Pox
Klephant
ReaE
Groondsle
Flephant
Fratn
Pickup_Teuck
Plecup.teack
P-ain
Plais
Plain
Verstoo
Pozeat
Delephone
Aquasion Fisa
Zisin
C3I
Tauck
Packsp_7tuck
Fex
CaLSiN
Dinoasss
Frohup_Treck
Packup_teuck
DN2
PANDA
Belphin
P1 Piskopp Truck
Fanganco
Pex
Kangsron
Raceson
Drcendste
Daccle
kriege
Plecop_Teict
MSAD
Cloud
Paucsp_7eucss
Kangacoo
Fex
Rangusoo
Beavec
Cassie
S64
Plccup_teack
Trandiemally
Cloud
Piskop_ Trucs
Iazand
Cattle
Plecup, Terok
Okn
Zank
Truss
Carma
M812
Ranqueoo
LAbazd
Cazai
Pickup_Tesck
piskog
Lengard
kridge
Table 6: The performance of novelty detection methods (AUROC %) in the near-ND setting, For each method and every normal class, both results have been reported according to the CLP and bottom-1 metrics.
Method
Metric
Airllane
Automobile
Bird
Cat
Deer
Dog
Frig
Horse
Ship
Truck
Mean
bottom-1
17.2
24.0
22.1
34.0
21.5
25.1
25.8
27.1
32.7
20.4
50.0
DeepSVDD
CLP
52.1
$2.3
53.9
64.7
53.0
39.9
57.5
55.6
61.7
54.5
54.5
GT
bottom-1
39.2
61.8
61.0
$4,2
60.1
65.1
71.0
70.4
80.5
62.9
62.6
CLP
85.6
61.8
81.4
58.7
61.6
80.2
78.8
76.1
80.6
62.9
72.8
bottom-1
34.1
62.4
65.2
$7.6
61.9
69.3
77.8
73.1
75.0
54.3
63.1
MHRot
CLP
77.4
62.4
81.5
60.4
61.9
82.1
82.3
73.3
79.9
64.7
72.6
one_elass OpenGan
bottom-1
11.4
10.0
29.8
16.9
20.9
40.4
6.2
50.0
CLP
66.3
92.2
94.9
64.7
51.7
63.7
93.3
66.5
85.7
85.0
76.4
bottom-1
73.4
68.4
81.3
65.0
71.8
78.1
88.3
85.9
91.1
57.5
76.1
CLP
94.2
68.4
93.7
79.9
71.9
90.6
91.5
87.1
91.0
74.6
84.3
DN2
bottom-1
57.5
61.8
43.8
54.0
51.5
66.2
59.4
74.6
54.9
61.0
58.5
CLP
87.5
61.8
61.7
66.5
84.4
83.5
74.3
74.6
77.0
69.8
74.1
PANDA
bottom-1
89.8
70.0
81.7
50.0
77.0
74.8
83.8
80.3
88.7
72.1
76.8
CLP
94.5
89.1
66.7
91.7
88.8
86.5
80.3
88.7
75.3
83.2
bottom-1
90.4
72.0
87.5
52.9
81.1
80.5
85.3
83.0
83.8
78.3
79.5
MSAD
CLP
94.0
72.0
89.8
75.3
87.3
94.5
90.2
83.0
88.6
79.0
85.4
Transformaly
bottom-1 CLP
88.3
78.4
89.8
85.3
93.1
84.0
73.5
82.3
90.1
67.5
89.0
79.1
90.1
93.6
85.3
93.1
90.2
74.7
85.3
bottom-1 1 CLP
95.6
83.5
93.4
86.9
92.3
84.5
94.8
97.8
93.7
77.0
90.0
Ours
96.3
83.5
99.2
89.4
97.0
95.8
94.8
98.0
93.8
79.9
92.8
AUROC(%)
OpenGAN
Dh2
MSAD
Tranebrraly
Ours
Bottom-i
Figure 3: By increasing i. the AUROCs of the models improve due to the presence of more anomalies that are further from the boundary. As expected, in most cases, the gap between the performance of models has become smaller. Furthermore, in the opge of i = 1, in which anomalies constitute the near distribution ones, the models' performances vary greatly. In this case, our proposed method achieves SOTA results by a large margin.

---

9.3 UNLABELED MULTI- CLASS
In this setup, we assume that normal samples are from specific multi-class dataset (i.e. CIFAR-10) without labels, testing various external datasets abnormal samples(e.g. CIFAR-100). We copied the CSI results from the original paper.
Table 7: AUROC (%) of methods trained on unlabeled CIFAR-10
SVHN
LSUN
CIFAR-100
Ours
99.9
99.4
95.7
CST
99.8
97.5
89.2
9.4 PER-CLASS RESULTS
In this section, provide model's performance for every class of the CIFAR-10 and CIFAR-100 datasets.
Table 8: Performance of our method in AUROC (%) for each class of the CIFAR-10 10 dataset in the one-vs-all setting.
Method
2
3
5
6
7
8
Mean
Ours
99.2
99.4
99.2
98.1
99.5
98.1
99.8
99.5
99.2
98.8
99.1
Table 9: Performance of our method in AUROC (%) for each class of the coarse-grained CIFAR-100 dataset in the one-vs-all setting.
Method
2
10
13
16
19
Meun
Ouss 98.0
29.0
99.1
98.2
98.2
97.7
98.8
98.7
99.0
96.6
96.5
98.1
98.
96.3
98.1
97.1
98.5
98.8
98.8
97.9
98.1
9.5 CIFAR-10 vs. CIFAR-100 100
In this section, compare images of the CIFAR- 10 dataset, images of respective classes selected from the CIFAR-100 dataset based on the CLP criterion, and the corresponding anomalies generated by our SDE model.
17

---

Figure 4: For each class of CIFAR-10, nearest class from CIFAR- 100 has been provided, according the closeness score. In each 3-row panel, the first row is assumed as normal, the second one assumed the nearest abnormal class, and the third row indicates the corresponding generated fake images using the SDE model.
18

---

CIARIIO
Dxg
CIEARIO
Gasoalee Anamayy
CIFARO Frog
CIFARIO) Hond
Gearraeed Anmmal
CIFARIO Home
CIFARIID
Conarate
CIEARIO Slip
CIEAROO Deidge
Genartte
Anumay
CIEARD
CIFARIO
Conrate
Figure 5: For each class of CIFAR-10, nearest class from CIFAR- has been provided, according to the closeness score. In each 3-row panel, the first row is assumed as normal, the second one is assumed the nearest abnormal class, and the third row indicates the corresponding generated fake images using the SDE model.
19

---

10
THE SDE-BASED GENERATIVE MODEL
10.1 SDE vs. OTHER GENERATIVE MODELS
In this section, according to our main setting, have replaced the SDE with other generative models, and sampled the images generated by these models before convergence. Because in our setup, only one class (the normal class) is available for the training of the generative model. As the sample size becomes limited in such setting, model convergence and generation of high quality real samples become a challenge. Even under such circumstances, the SDE-based models converge. Furthermore, such models eventually generate high quality real images. However, other generative models do not converge properly, and the images that are sampled using the early stopped model are often noisy and contain artifacts. As opposed to other generative methods, the proposed premature training of the SDE yields authentic, near-distribution, diverse, and artifact-free images. This makes it better choice for the near-ND setup.
Original
(Our))
SDE
SyiGANS
Syl-GNZ
ads
VQ-VA22
VAE Baseins
GAN Badiins
Figure Generated anomalous samples by different generative models the FGVC-Aircraft dataset.
20

---

Oniginal
SDE
(Ours)
StyleGAN3
StyleGAN2
ada
Baselines
VAE
Baselnns
GAN
Figure 7: Generated anomalous samples by different generative models on the WBC dataset.
21

---

10.2 CIFAR- 10-FSDE
These images are randomly plotted for each of the classes in the proposed dataset, CIFAR-IO-FSDE generated through early stopped SDE. As can be seen, the images are clearly different semantically from the normal class. Despite this semantic difference, most novelty detection models have a poor performance in the detection of these images as anomalous.
Bird Automoblle Airplans
Cat
Deer
Dog
Frog
Horse
Ship
Trukk
Figure 8: An overview images in the synthetic CIFARIO-FSHD dataset.
22

---

10.3 OVERVIEW OF GENERATED ANOMALY IMAGES ON VARIOUS DATASETS
Each row is a class on which the SDE model is trained. The SDE model is trained on higher number of iterations in moving from the left to the right columns. The rightmost column contains real images.
Figure 9: Generated anomaly images on 102 category of the Flowers dataset. The images highlighted in green are normal samples.
23

---

Figure 10: Generated anomaly images on the StanfordCars dataset. The images highlighted in green normal samples.
24

---

Figure 11: Generated anomaly images on the Caltech- Birds 200 dataset. The images high- lighted in green are normal samples.
25

---

Figure 12: Generated anomaly images on the MVTecAD dataset. The images highlighted in green are normal samples.
26

---

11 DATASET DESCRIPTIONS
The setting reported from 55
Standard Datasets We evaluate our method on of commonly used datasets: CIFAR-I0 consists of RGB images of 10 object classes. CIFAR-100 100: we use the coarse-grained version that consists of classes.
Small datasets To further extend our results, compared the methods number of small datasets from different domains: 102 Category Flowers and Caltech-UCSD Birds 200. For each of these datasets, evaluated the methods using only each of the first 20 classes normal, and using the entire test set for the evaluation. For FGVC-Aircratt, due to the extreme similarity of the classes each other, randomly selected subset of ten classes from the entire dataset, such that no two classes have the same Manufacturer. Following are the selected classes: [91, 96, 59, 19, 37, 45 ,C, 68, 74, 89).
MvTecAD: This dataset contains 15 different industrial products, with normal images of the proper products for training and to types of manufacturing errors anomalies. The anomalies in MvTecAD are in-class, i.e. the anomalous images come from the same class of the normal images with subtle variations.
Symmetric datasets We evaluated our method datasets that contain symmetries, such images that have no preferred angle (microscopy, aerial images): WBC : we used the big classes "Dataset 1" of the microscopy images of white blood cells, with 80%/20% train-test split.
27


Reviews:
[{"review_id": "yNhNGyxq3fP", "reviewer": "Reviewer_xzvz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper is focused on a challenging task: near-distribution novelty detection (near-ND). The authors propose to exploit a diffusion generative model to produce synthetic near-distribution anomalous data(i.e., “fake\"), then the  pre-trained feature extractor is  fine-tuned to distinguish generated anomalies from the normal samples.\n", "summary_of_the_review": "Overall, this paper sheds some insights that diffusion- based data generation considerably passes the GAN-based SOTAs in the novelty detection task, especially for the neat-ND setting.  I recommend to accept this paper. ", "strength_and_weaknesses": "The idea of this paper is very natural: try to generate near-distribution outliers to augment the original training dataset which only contains normal data (semi-supervised learning), such that we can fine-tune the pre-trained feature extractor by training a binary classifier. \n\nStrengths:\n(1) Using generated outliers to improve the accuracy of novelty detection is not new. There have been some similar works which use GANs instead of SDE.  The main contribution of this paper is that they replace the GANs with the diffusion model (which is increasingly popular recently).  The motivation here is that the outliers generated by SDE are “closer“ to the nominal distribution, and then using these \"boarder\" points  to deal with the near-ND setting.  Even though the pipeline and SDE are all from existing works, the novel combination is meaningful considering the theoretically and experimental results.  \n\n(2) The experimental results show that the proposed method considerably improves over existing models (especially for the near-ND setting).  It is not surprising that SDE over-perform GANs for the near-ND setting, but I am surprised that the improvement is so obvious.  It will be great if the code can be provided.\n\nWeakness:\n(1) In Section 2.2 (Diffusion models vs. GANs), they try to explain, for near-ND, why choosing diffusion models instead of other generative modes (GAN and VAE), it would be better to include flow-based models in this part. Also,  VAE-related results are not included in figure3.\n\n(2) In Figure3,  density, coverage and FID are used as the metrics to measure the fidelity and diversity.  Could you add a brief introductions/definitions for these metrics? Also, I am confused how to get the \"density\" from GAN-based models as they are not tractable models.\n\n(3)About Stopping point: In section 4, \"the method is robust against the stopping point with at most 2% variation for the reasonable FID scores\". However,  from table4,  it is difficult to get this conclusion. what is \"reasonable\" FID? Actually, the stopping point decide how similar the fake points and the real points. How to decide when to stop ?\n\nOther minor comments:\nThe \"normal distribution\" used in this paper sometimes are confusing, which means the distribution of nominal data instead of Gaussian distribution. It would be better to use other notations.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ftRvpnrttXi", "reviewer": "Reviewer_wBQG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a novelty detection method that utilizes synthesize outliers during training. The synthetic outliers are generated from a score-based generative model trained on in-distribution data.", "summary_of_the_review": "Even though the problem that the paper tackles is relevant to the community, the paper has limited novelty and clarity.", "strength_and_weaknesses": "* There is a large room for improvement in clarity.\n    * The paper should be clearer on how exactly the generative model is trained and used. The most important fact, in my opinion, is that the score-based generative model is trained on in-distribution data and terminated prematurely. However, the fact is only briefly mentioned in Page 2, and then mentioned in Figure 2. Given the importance of this information, this point should be described in detail in Section 2.1. Also, other details and design choices, for example, how early the training of the score-based model should be stopped, needs to be elaborated.\n    * The experimental setup described in Section 3.2 is difficult to follow. There are several comparison settings but are not well organized.\n* Discussion on the differences among diffusion models, GANs, and VAEs in Section 2.2 are not very persuasive. It does not have concrete examples or quantitative arguments. Plus, the argument of Section 3.2 is mostly about generative modeling itself and not about its application to generating auxiliary data in novelty detection.\n* When performing CIFAR-10 vs CIFAR-100 experiment, it is more standard and more challenging to incorporate all the classes, instead of using only one class from each dataset. Overall, I am not convinced that the experiment settings used in the paper are standard.\n* Replacing a GAN with a diffusion model in the data generator for novelty detection itself is not a very novel contribution to the community.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Mx_7O5Zwp3", "reviewer": "Reviewer_ahZi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper tackles the problem of __near__ novelty detection (near ND), with __near__ referring to\nthe case in which the novelty (OOD) classes derive are semantically similar to those classes\ncontained in the in-distribution (ID) set used for training (a novelty class under this regime, for\nexample, being 'fox' when the ID set contains 'dog'). The authors begin by showing that\nrecently-proposed ND algorithms exhibit a significant performance gap with respect to the standard (far)\nND vs. near ND setup to motivate the problem. To solve this, the authors propose to use\nearly-stopped diffusion models, trained on the ID data, to generate a dataset of anomalous samples, with\nfeatures for the ND scoring mechanism can be learned by framing the task as one of binary\ndiscrimination (real data vs. diffusion-model-generated data), solved with a linear classifier\nhead. In order to score samples, a simple k-NN-based algorithm is employed, acting on the aforementioned\nfeatures extracted from the training set. The authors show that this simple algorithm succeeds in reducing the gap between near ND and ND performances while also improving upon previous methods in the far ND setting. It is reasoned and shown empirically GANs (as in OpenGAN) lack some the necessary properties that diffusion models that are crucial to this success (e.g. image-quality metrics do not show a monotonic improvement in generative ability as training progresses).\n", "summary_of_the_review": "The paper is well-motivated and proposes a simple but apparently strong method for tackling the\nproblem of near novelty-detection, a setting in which the novel (OOD) classes are semantically\nclose to classes in the training set. The explanation of the method is easy-to-follow and the\nproblem and the solution are soundly established. The results look promising and have been compiled across a good range of datasets and baselines. While I worry that the sensitivity of the\nmethod to the stopping criterion, as well as the assumptions\nthat are made in the distributions of the ID/OOD datasets, might limit practical applicability, I nonetheless think the\noverall the paper is strong enough to meet the threshold for acceptance based on the aforementioned merits.\n", "strength_and_weaknesses": "### Strengths\n\n- The paper is, for the most part, well-written, easy-to-follow, and does a good job of\ncontextualising and motivating the work.\n- The proposed method is simple and intuitive, and seems to perform well empirically\ncompared with the baselines in both the regular ND setting and the near ND setting, showing a\nsignificant reduction in the performance gap of going from the former to the latter.\nthe NR to near-NR setting.\n- The figures are well-chosen and well-put-together: they convey well the essential points of the\n  paper, such as the shortcomings of existing methods and the feasibility of using GANs vs diffusion\n  models for creating the synthetic data.\n- Evaluation performed using a good range and selection of baseline methods and benchmark datasets.\n- Results are aggregated over a good number (10) of replicates.\n\n### Weaknesses\n- Despite the authors claiming otherwise, the method is quite sensitive to the stopping criterion in\n  the near ND setting with an empirical range of [68.2, 90.0]; in practice, different datasets may\n  require wildly different FID thresholds and there is no reliable way of validating this given the\n  nature of the problem. Also, I wonder if the number of sampling steps and the choice of sampler have a\n  significant effect on the results -- there doesn't seem to be any mention of this.\n  Taking this a step further, might it be feasible to train the diffusion model fully and then rely\n  upon early-stopping the reverse diffusion process according to the FID score?\n- The need to train, and indeed sample, from a diffusion model can be computationally burdensome for\n  higher-resolution datasets.\n- The description of the SDE model is a little overdone; a much briefer description with a pointing of the\n  reader to the original paper would be sufficient. This is especially the case given that, as\n  I understand it, the proposed method should work with any diffusion based model (Imagen, Parti,\n  DALLE-2, etc.) or are there theoretically some constraints on the form said model and the\n  associated sampler should take?\n- While this applies to several methods of the same class, the method seems to make certain\nassumptions about the distribution of the ID and OOD data that might not hold in the wild.\n- While results are aggregated over a good number (10) of replicates no measure of dispersion\n  (standard deviation/error) seems to be reported alongside the mean in each case.\n- Some typographic and syntactical errors (e.g. 'that unlikely come' (line 1 of the introduction)).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AxoSMem0dt", "reviewer": "Reviewer_mA9p", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the near anomalies detection and points a significant decrease in performance of state of the art approaches on near anomalies cases. A new solution is proposed: 1) using SDE models trained to generate data from the training distribution. 2) generate a dataset from the trained SDE. 3) train a binary classifier on top of a feature extractor to distinguish between the generated data and real data. 4) use nearest neighbour for detecting anomalies.\nExtensive experiments and empirical analysis are carried on. The method shows good results. \n", "summary_of_the_review": "A new method for detecting near anomalies based on SDE, binary classification training and then KNN.\nGood empirical results. \nNot super clear how design choices were made.", "strength_and_weaknesses": "Strengths\n1) Near anomalies are a very important case that should be tackled more.\n2) Good insights and good results.\nWeaknesses\n1) I am not quite sure about the made distinction between Near ND and Near OOD. The authors stated the difference, however, without using any reference.\n2) Design choices are not well explained, e.g., why a binary classifier is used, why NN is deployed for ND and not the binary classifier itself.\n3) How about training complexity, hyper-parameter selection?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster