Efficient Out-of-Distribution Detection via CVAE data Generation
Authors: Mengyu Wang
Yijia Shao
Haowei Lin
Wenpeng Hu
Bing Liu
Venue: ICLR 2022 Submitted (2022)
Keywords: Out-of-distribution Detection

Abstract: Recently, contrastive loss with data augmentation and pseudo class creation has been shown to produce markedly better results for out-of-distribution (OOD) detection than previous methods. However, a major shortcoming of this approach is that it is extremely slow due to significant increase in the data size and the number of classes and the quadratic complexity of pairwise similarity computation. This paper proposes a novel and simple method that can build an effective data generator using Conditional Variational Auto-Encoder (CVAE) to generate pseudo OOD samples. Based on the generated pseudo OOD data, a flexible and efficient OOD detection method is proposed through fine-tuning, which achieves results comparable to the state-of-the-art OOD detection techniques, but the execution speed is at least 10 times faster. Also importantly, the proposed approach is in fact a general framework that can be applied to many existing OOD methods and improve them via the proposed fine-tuning. We have combined it with the best baseline OOD models in our experiments to produce new state-of-the-art results.

Under review as a conference paper at ICLR 2022
EFFICIENT OUT-OF-DISTRIBUTION DETECTION VIA CVAE DATA GENERATION
Anonymous authors Paper under double-blind review
ABSTRACT
Recently, contrastive loss with data augmentation and pseudo class creation has been shown to produce markedly better results for out-of-distribution (OOD) de- tection than previous methods. However, major shortcoming of this approach is that it is extremely slow due to significant increase in the data size and the number of classes and the quadratic complexity of pairwise similarity computation. This paper proposes a novel and simple method that can build an effective data genera- tor using Conditional Variational Auto-Encoode (CVAF) to generate pseudo OOD samples. Based on the generated pseudo OOD data, a flexible and efficient OOD detection method is proposed through fine-tuning, which achieves results compa- rable to the state-of-the-art OOD detection techniques, but the execution speed is at least 10 times faster. Also importantly, the proposed approach is in fact gen- eral framework that can be applied many existing OOD methods and improve them via the proposed fine-tuning. We have combined it with the best baseline OOD models in our experiments to produce new state-of-the-art results.
INTRODUCTION
With the development of deep learning, a well-trained neural network model is able to obtain very high accuracy on its testing data. However, when exposed samples or data instances drawn from a distribution that is far from the training distribution (called In-distribution (IND)), the model may make arbitrary predictions under the known framework (Nguyen al.] 2015 Recht et al. (019). This limits the model's application in broad range of applications, including secure authentica- tion (Sharif et al., 2016), autonomous driving (Nitsch et al. 2020) and medical diagnosis (Caruana al.) 2015) quently. Therefore, OOD detection (Hodge & Austin (004), which aims to detect abnormal or novel data that are very different from the training data, is an important research task.
these applications novel out-of distribution (OOD) data instances occur fre-
Many approaches have been proposed to solve this problem, from distance- based methods (Bendale & Bould 2015 2016)] Gunther et al. [2017] Junior et al. (2017). to generative models (Ge al. 2017
Neal et al |2018] ||Oza za & Patel| 2019 Nalisnick et |018 and self-supervised learning (Bergman & Hoshen] 2020 Golan & EI- Yaniv 2018 Hendrycks et 2019). Recently, contrastive learning has been shown highly effective in many applications (Hjelm et al. 2018 Oord et al., 20018) Chen et al.)
2020b a Falcon & Cho 2020). Applying supervised contrastive learning and data augmentation. the
recent CSI system has produced state-of-the-an (SOTA) OOD detection results (Tack et al. 2020).
contrastive learning also has some drawbacks. First, designing data augmentation functions and deciding how to use various types of augmented data in contrastive learning involve a great deal of trial and error and manual work. That is, there a large number of transformations (or augmentations) such as cropping, rotation and gray-scaling that can be exerted on images but not all of them may benefit the end tasks. In SimCLR (Chen et al., (020a), system- atic experiments have been reported to explore the augmentations' influence on classification tasks. Based on SimCLR's 's results, CSI Tack et al. (2020) chooses several operations for OOD detection.
However, data augmentation-bant
Second, contrastive learning with data augmentation is very time-consuming to run and resource- hungry due to a large amount of augmented data and quadratic pairwise similarity computation during training. For example, CSI creates shifted instances for each original image sample and the images are then subjected to an additional augmentation operation. Each image is finally expanded to 8 images or samples. Furthermore, every 2 samples in the augmented batch is treated

---

Under review as a conference paper at ICLR 2022
as a pair to calculate contrastive loss. The performance is negatively affected if the batch size is not large enough. Therefore, it is necessary to use a large batch size, which demands a huge amount of memory and takes a very long time to run. It is thus not suitable for applications on hardware devices that do not have the required resources such as edge devices.
In this paper, we propose a general and much more efficient solution, called CGA (CVAE-based Generative data Augmentation for OOD detection). CGA consists of two parts. The first part gen- erates pseudo OOD data and the second part makes use of the pseudo OOD data to train an OOD detection model. We discuss the first part first. OOD detection is basically classification problem but the challenge is that there no OOD data to use training. This paper proposes novel method to generate pscudo OOD data. This method works in the latent space of Conditional Variational Auto-Encoder (CVAE) and uses CVAF's decoder to generate pseudo OOD data. CVAE able to generate instances from the training distribution on the basis of latent representations consisting of conditional information and variables sampled from prior distribution of CVAE, normally the Gaussian distribution. If the latent space features or representations are created with some abnormal conditional information, the CVAE will generate "bad" instances but such instances can serve as ef- fective pseudo OOD data. The second part of CGA is fine-tuning framework that can make use of the generated pseudo OOD data to fine-tune any classification model built using only the IND data. Applying the framework to simple IND classification model, we can already produce comparable results to existing SOTA contrastive learning models but much more efficient. Applying it to the existing SOTA methods, we can improve their results too.
Our contributions can be summarized as follows: (1) We propose to employ a CVAE structure to generate pseudo OOD samples by providing some synthetic conditional information, which, to our knowledge, has not been done before.
(2) We design a two-stage framework to train an OOD de- tection model by leveraging the generated pseudo OOD data. The first stage simply builds a normal classification model using only the IND data. The second stage fine-tunes the model using the gen- erated OOD data to produce an effective OOD detection model. Extensive experiments show that this approach achieves comparable performance to the state-of-the-art (SOTA) contrastive learning methods in OOD detection, while consuming only one-tenth of the execution time. (3) Equally importantly, the proposed framework can be applied to existing SOTA OOD detection models to improve them to produce new SOTA results.
RELATED WORK
nuttof-distribution Detection. It is well-known that the discriminative neural networks can pro- duce overconfident predictions on out-of-distrbutiion (OOD) inputs (Hendrycks & Gimpell 2016 Lakshminarayanal et al. 2016). The early idea to solve this problem focuses on modifying soft- max scores to obtain calibrated confidence for OOD detection (Bendale & Bouli 2016 Guo al.] 2017). In addition, many other score functions have been proposed, such as likelihood ratio (Ren jet al. 2019), input complexity (Serrà et al., 2019) and typicality (Nalisnick et al., 2019). recent work utilizes Gram matrices to characterize activity patterns and identify anomalies in Gram matrix values to do OOD detection (Sastry& & Oore, 2020). Some methods found that auxiliary anomalous data significantly improve detection performance (Hendrycks et al. 2018; Mohseni et al. 2020) and thus generative models are adopted to anticipate the distribution of novel samples. In some of these methods, generated data are treated OOD samples to optimize the decision boundary and calibrate the confidence (Ge et al.) 2017 Vernekar et al. (2019). In some other works, generative models such as auto-encoders (Zong al. 2018| Pidhorskyi al., 2018) and generative adversarial networks (GAN) (Deecke et al. 2018 Perera et al.] 2019) are used to reconstruct the training data. During the training of GAN model, low quality samples acquired by the generator can also work as OOD data directly (Pourreza et al.) 2021). Owing to the fact that the model can hardly be generalized to unknown data, the reconstruction loss can help detect OOD data. There are also works using auxiliary OOD data to fine-tune the model (Liu et al.) (020). Fort et al.) (202T) showed that using pre-trained representations and taking few-shot outlier exposure can improve the results. Recently, self-supervised techniques have been applied to OOD detection. It focuses
acquiring rich rep- resentations through training with some pre-defined tasks (Gidaris et al.) 2018 Kolesnikov et al.) 2019). Self-supervised models show outstanding performance on OOD detection tasks (Kolesnikov ct al. 2019; Bergman & Hoshen 2020). CSI (Tack et al.) (020) is representative method (see more below), which uses contrastive learning and data augmentation to improve the features of all labeled
2

---

Under review as a conference paper at ICLR 2022
IND data and produce state-of-the-arr (SOTA) results. Some researchers also tried to improve con- trastive learning based methods (Schwag et al. 2021) and proposed distance- iistance-based methods (Miller et al. 202T). However, based our experiments. CSI outperforms them. Our method falls into the generative approach. Unlike existing methods that use perturbations to anticipate OOD data, our method uses synthetic conditions and CVAE to obtain effective and diverse pseudo OOD data.
Contrastive Learning. Contrastive learning learns representations by contrasting positive pairs against negative pairs (Hadsell et al. 2006). It has been applied various domains (Oord et al.) 2018). Recently, new method called SimCLR (Chen et al. 2020a) was proposed to create sample pairs via data augmentation. It effective but also very time and resource consuming. SimCLR also shows that contrastive learning benefits more from larger batch sizes and longer training time. CSI (Tack et al. 2020) proposes that augmentation can not only help construct positive pairs but also negative pairs and makes use of them to detect OOD samples with supervised contrastive learning. obtains the SOTA OOD detection results with labeled IND data. However, due to contrastive learning, it is extremely slow and memory demanding. Our proposed method generates pscudo OOD using CVAE, avoiding the use of contrastive loss, and is much more efficient than CSI.
Auto-Encoder Auto-Encoder (AE) is a family of unsupervised neural networks (Rumelhart et al.) 1986 Baldi & Hormik 1989). A basic AE consists an encoder and a decoder. The encoder en- codes the input data into fow-dimensional hidden representation and the decoder transforms the representation back to the reconstructed input data (Vincent et al. 2008; CCen et al., 2012 Hin- ton et al., 2006). Variational auto-encoder is a special kind of AE (Kingma & Welling 2013)- It encodes the input as a given probability distribution (usually Gaussian) and the decoder recon- structs data instances according variables sampled from that distribution. CVAE is an extension of VAE (Kingma et al. 2014). It encodes the label or conditional information into the latent rep- resentation so that a CV can generate new samples from specified class labels. CVAE makes possible to control the generating process, i.e., to generate samples with features of specified classes. We make use of this property of CVAE to generate high quality pseudo OOD data.
PROPOSED CGA METHOD
In tasks related to formulated as a classification problem. The main challenge is that an important class, OOD data, is not available. Therefore, to effectively train OOD detection model, an intuitive idea is generate pseudo OOD data and use them together with the IND data to train the model. As we mentioned earlier, data augmentation and contrastive learning have been shown especially effective for this purpose. However, this approach is extremely inefficient. We propose to use Conditional Variational Auto-Encoder (CVAE) to generate pseudo OOD data and present a new fine-tuning framework to leverage the generated pseudo OOD data to train an OOD detection model.
(OOD) detection, the problem of recognition is commonly
3.1 CONDITIONAL VARIATIONAL AUTO-ENCODER
Conditional Variational Encoder (CVAE) is derived from Variational auto-encoder (VAE). We first introduce VAE which is a conditional directed graphical model consisting of three main parts, an encoder (-) with parameters Φ, decoder pa(') with parameters 0 and loss function C(x; 0, Φ). where x represents an input sample. The loss function is as follows:
C(x; 0, d) = (zx)) [log (x|z)] + K [Lag(
(1)
where qu (z)x) is a proposal distribution to approximate the prior distribution (z)z). (x|z) is the likelihood of the input x with a given latent representation z, and K L-) is the function to calcu- late Kullback-Leebler divergence. To (z x) is the encoder and po (x|z) is the decoder. In [q. the expected negative log-likelihood term encourages the decoder to learn to reconstruct the data with samples from the latent distribution. The KL-divergence term forces the latent distribution to form to a specific prior distribution such as the Gaussian distribution. After training, a VAE can generate data using the decoder po (x(z)) with set of latent variables z sampled from the prior dis- tribution po (z). Commonly, the prior distribution is the centered isotropic multivariate Gaussian (z( N (zz 0. I).
However, VAE does not consider the class label information which is available in classification datasets and thus has difficulty generating data of particular class. Conditional variational Auto-
3

---

Under review as a conference paper at ICLR 2022
Encoder (CVAE) was introduced to extend VAE to address this problem. It improves the generative process by adding a conditional input information into latent variables so that a CVAE can generate samples with some specific characteristies from certain classes. We use to denote the prior class information. The loss function for CVAE can be written as follows:
[xx; 0, Ø) = - 2x (x/3) (o c)] + (L c)) pe
(2)
One implementation of CVAE uses one-hot vector represent class label ye: and a weight matrix is multiplied to it to turn the one-hot vector to class embedding y.. Then a variable z, generated from the prior distribution pe (z(z). is concatenated with y. to construct the whole latent variable. Finally, the generated instance po (x|z, Iz. c) of class c is produced. We can formulate the process as:
pe (x|2, po ()).)))
3.2 GENERATING PSEUDO OOD DATA
The ability of CVAE to control the generating process using the conditional information (e.g. class label in our case) inspired to design a method to generate possible OOD samples. This is done by the conditional decoder using atypical prior information c in P& (x c). As introduced before, OOD data need to be different from in-distribution data but also resemble them. The continuity property of CVAE, which means that two close points in the latent space should not give two completely different contents once decoded (Cemgil et al. 2019), ensures that we can manipulate CVAE latent space features to generate high quality pseudo OOD data. Since we have no information of the future OOD data, we have to make use of the existing training data (i.e., in-distribution data) to construct pseudo OOD data. We can provide it with pseudo label information to generate pseudo OOD data.
Specifically, we propose to construct pseudo class embedding by combining the embeddings of two existing classes in the in-distribution training data. The formulation is follows:
po (x k, (g, (x Y.,
k) * yey)²)) z))
where k is a vector generated from Bernoulli distribution B(0.5) with the same length as the class label embedding. k is basically for the system to randomly select the vector components of the two class embeddings with equal probability. Such a generated sample po (x|z, k, G; (y) will not likely to be an instance of either class ej ej but still keep some of their characteristics, which meets the need for pseudo OOD data. Furthermore, the pseudo class embedding has a great variety, owing to the diverse choices of classes and the vector k. To generate pseudo OOD samples, we also need to sample z from the encoder. In the training of CVAE, we ensure that z fits the Gaussian distribution another flatter Gaussian distribution A0, 0² I), where
A( I). To sample to make the generated samples highly diverse.
Z,
3.3 TRAINING PROCESS FOR OOD DETECTION
With the generated pseudo OOD samples and the original in an-distribution (IND) training data, train- ing an OOD detection model consists of two stages.
Stage (IND classifier building and CVAE training): Only the original IND data used train a classification model C. The classification model can be decomposed into two functions f and h, where f is the final linear classifier and h is the feature extractor. f(h(x)) the classification output. A separate CVAE model is also trained for generating pseudo OOD data.
Stage 2 (fine-tuning with pseudo OOD data): We keep the trained feature extractor h fixed (or frozen) and fine-tune only the classification layer f for OOD detection.
The proposed CGA approach two-stage training process. The framework is very flexible as the classifier in the first stage can use any model. Stage 2 is also flexible and can use many approaches. Here we introduce two specific methods. They produce results comparable to the state-of-the-ane OOD detection models, but very simple and very efficient. In fact, as we will see in the experiments, fine-tuning the existing state-of-the-an (SOTA) OOD detection models can also improve them to produce new SOTA results.
fact framework, which is illustrated in
with the
CGA-soffmax. In the fine-tuning stage, we simply add additional class (let us call it the OOD class) in the classification layer to accept the pseudo OOD data. If the IND data has N classes,
4

---

Under review as a conference paper at ICLR 2022
Predicted Class
Reconstructed Data
OOD Loss
optimize
Decoder
Pseudo OOD Data
Classifier
Classifier
Decoder
Feature
Encoder
Feature Extractor
Data
IND Data
k>y,+(2-k)yy,
First Stage
Second Stage
Figure 1: The CGA framework and its training process. The OOD loss can be the cross entropy loss CGA-softmax, cross-nntropy+energy loss CGA-energy, or other possible losses.
add parameters to the classifier to make it output N + 1 logits. These added parameters related to (N + 1)th OOD class is randomly initialized. We then train the model by only fine-tuning the classification layer using the cross entropy loss with feature extractor trained in stage fixed. Finally, we use the softmax score of the (N + 1)th class as the OOD score.
CGA-energy. In this approach, add energy loss to the cross entropy loss (Cent + XCenergy) to fine-tune the classification layer using the IND data and the pseudo OOD data to produce an OOD score for each test instance. In this case, OOD class is added. The energy loss follows,
(5) where in, denotes the IND training data, Dout denotes generated pseudo OOD data, and min and Mout are margin hyper-parameters values for all N logits so that they will not be favored by any N IND data classes. Here N is the number of classes of the IND data. As the loss function shows, the OOD data is necessary. This loss was used in (Liu et al. 2020), which has to employ OOD data but such OOD data is often not available in practice. This loss cannot be used by other OOD methods since they have no OOD data available (Bendale & Boult 2016 Khosla al., 2020 Tack al.) (2020). However, this is not an issue for us as we have pseudo data to replace real OOD training data.
Cenergy DD,, (max(0, E(x..) m)n))² Exaue>Donce (max(0,, Mout Exxout)))
The idea of this loss is to make the OOD data get similar
Stage 2 produces an energy score calculated from a classification model for OOD detection:
f(h)) = - log Σ fi(h())/T
(6)
where Exx; (((())) denotes the energy of instance x with the classification model f(h(-)), which maps x N logits, where N is the number classes the IND data, f(((x) is the i-th logit and T is the temperature parameter.
4 EXPERIMENTS
We construct OOD detection tasks using benchmark datasets and compare our proposed technique CGA with the state-of-the-art existing methods. The code of CGA has been submitted.
4.1 EXPERIMENT SETTINGS AND DATA PREPARATION
We use two experimental setups to evaluate our system.
Setting 1 OOD Detection the Same Dataset: In this setting, IND (in-distribution) and OOD instances are from different classes of the same dataset. This setting is often called open-set detec- tion. We the following 4 popular datasets for experiments in this setting.
(1) MNIST (LeCun et al. 2010y: handwritten digit classification dataset of 10 classes. The dataset has 70,000 examples/instanes, with the splitting of 60,000 for training and 10,000 for testing.
5

---

Under review as a conference paper at ICLR 2022
(2) CIFAR-10 (Krizhevsky & Hinton 2010): An image classification dataset consisting of 60,000 32x32 color images of 10 classes with the splitting of 50,000 for training and 10,000 for testing.
(3) SVHN Netzer et al. 2011): A colorful street view house number classification dataset of 10 classes. It contains 99289 instances with the splitting of 73257 for training and 26032 for testing.
(4) TinyImageNet (Le & Yang 2015;:
classification dataset of 200 classes. Each class contains 500 training samples and 50 testing samples. The resolution of the images is 64x64.
We follow the data processing method in (Sun et al., 2020; Zhou et al. 2021 to split known and unknown classes. For each dataset, we conduct experiments using different splits of known (IND) and unknown (OOD) classes. These same 5 splits used by all baselines and lowing (Sun et al., 2020), for MNIST, CIFAR-10 and SVHN, 6 classes are chosen as IND classes, and the other 4 classes are regarded as OOD classes. The following fixed sets of IND classes, 0-5, 1-6, 2-7, 3-8, and 4-9, are used and they are called partition 1, 2, 3, 4, and 5, respectively. The rest 4 classes in each case serve as the OOD classes. For TinyImageNet, each set of IND data contains 20 classes and the sets of IND classes in the 5 experiments are 0-19, 40-59, 80-99, 120-139, and 160-189 respectively. The rest 180 classes are regarded as the OOD classes.
system. Fol-
Setting 2 OOD Detection on Different Datasets: The IND data and OOD data come from dif- ferent datasets. Following (Tack et al. 2020), we use CIFAR-10 as the IND dataset and each of the following datasets as the OOD dataset.
(I) SVHN Nettzer et al.
2011 See above. All 26032 testing samples are used as OOD data.
(2) LSUN Yu et al.
2015 This is large-scale scene understanding dataset with a testing set of 10,000 images from 10 different scenes. Images are resized to 32x32 in our experiment.
(3) LSUN-FIX (Tack et al.
2020): To avoid artificial noises brought by general resizing operation, this dataset generated by using a fixed resizing operation on LSUN to change the images to 32x32.
(4) TinyImageNet (Le & Yang 2015): See above. All 10000 testing samples are used as OOD data.
(5) ImageNet-FIX (Le & Yang 2015;: 10,000 images are randomly selected from the training set of ImageNet-30, excluding "airliner", "ambulance", "parkingmeter", and "schooner" classes to avoid overlapping with CIFAR-10. 10. fixed resizing operation is applied transform the images to 32x32.
(6) CIFAR100 (Krizhevsky et al. 2009): An image classification dataset consisting of 60,000 32x32 color images of 100 classes. Its 10,000 test samples are used as the OOD data.
4.2 BASELINES
We compare with the following state-of-the-att baselines using the official code from their authors.
(I) Softmax: This is the popular classification score model. The highest softmax probability is used as the confidence score for OOD detection.
(2) OpenMax (Bendale & Boult 2016): This method combines the softmax score with the distance between the test sample and IND class centers to detect OOD data.
(3) ODIN Liang et al.,
2017): This method improves the OOD detection performance of a pre- trained neural network by using temperature scaling and adding small perturbations to the input.
(4) Maha (Lee et al. 2018): This method uses Mahalanobis distance to evaluate the probability that an instance belongs to OOD.
(5) CCC (Lee et al., pseudo OOD generator for OOD detection.
2017 This is a GAN- method, jointly training the classification and
2018): This method also uses GAN to generate pseudo instances and further improves the model to predict novelty (OOD) examples.
(6) OSRCI (Neal et al.
(7) CAC (Miller et al.
2021h: This is a distance-based method, using the Class Anchor Clustering
loss cluster IND samples tightly around the anchored centres for OOD detection.
(8) SupCLR (Khosla et al. 2020): This a contrastive learning based method. It extends contrastive learning to fully-supervised setting improve the quality of features for classification.

---

Under review as a conference paper at ICLR 2022
This is also supervised contrastive learning method. It uses extensive data augmentations to generate shifted data instances, also has a score function that benefits from the augmented instances for OOD detection.
(9) CSI (Tack al. 2020):
For Softmax, OpenMax and OSRCI, we use OSRCI's implementation") For SupCLR and CSI, we CSI's code" For ODIN, Maha, CCC and CAC, their original code PRRM We also use their
default paperrparammters.
4.3 IMPLEMENTATION DETAILS
For MNIST, use 9-layer CNN the encoder (feature extractor) and 2-layer MLP as the projection head. CVAE includes 2-layer CNN as the encoder and a 2-layer deconvolution net- work (Zeiler et al. 2011) the decoder, well two 1-layer MLPs turn features into means and variations. For the other datasets, the encoder is a ResNet18 (He et al. 2016) and the projection head is a 2-layer MLP. CVAE also ResNet18 the encoder, and 2 residual blocks and 3-layer deconvolution network as the decoder. The mean and variation projection are completed by two 1-layer MLPs. During the first stage of training, we use Adam optimizer (Kingma & Ba 2014) with 31 = 0.9, 82- = 0.999 and learning rate of 0.001. We train both the classification model and CVAE model for 200 epochs with batch size 512. In the second stage, the learning rate is set to 0.0001 and the fine-tuning process with the generated pseudo data are run for 10 epochs. The number of generated pseudo OOD data is the same as the IND data (we will study this further shortly). Each batch has 128 IND samples and 128 generated OOD samples. There is no special hyper-parameter for CGA-softmax in stage 2. For CGA-energy, two special hyper-parameters of the energy loss man and Mout decided at the beginning of stage by IND and pseudo data. We calculate the energy of all training IND data and generated pscudo data. Then min and Mout are chosen to make 80% of IND data's is energy larger than min and 80% of pseudo data's s energy smaller than Mout- This ensures that 80% of data get non-zero loss. We use the same WVIDIA-GeFFocc--R" Fo.--o------ GPU for the experiments of evaluating the running speed of different methods.
4.4 RESULTS AND DISCUSSIONS
Table shows the results of the two OOD detection settings on different datasets. Due to the large image size, a large number of IND classes and large batch size requirement, we were unable to run SupCLR and CSI using TinylmageNet on our hardware and thus do not have their results in Setting 1. On average, CGA model can achieve the best results in Setting 1. In Setting 2, benefiting from strong features learned using contrastive loss, CSI performs the best on average and our CGA-e (CGA-energy) is slightly weaker than CSI. But from Table we learn that CGA is much more efficient than the contrastive learning methods. With comparable overall performances on OOD detection, CGA spends only about 10% of CSI's training time. We also notice that our CGA-s (CGA-softmax) is slightly weaker than CGA-e, which shows that the energy function is effective.
Equally importantly, Table 2 demonstrates that CGA's fine-tuning (stage can improve the best performing baselines in Table [ i.e., GAN-based OSRCI and contrastive learning based SupCLR and CSI. Here after each baseline finishes its training, we apply fine-tuning of CGA's stage fine- tune the trained model using CGA-energy. Although CSI already produces the best OOD detection result, it is improved further by our CGA framework to produce a new SOTA result.
4.5 ABLATION STUDY
We now perform the ablation study with various options of our system and report the AUC scores on 5 partitions of the CIFAR10 dataset in Setting OOD detection on the same dataset.
hnapss//gitubb.co//aaalbiiSsS htnps://gihbb.cmmmfaccccoceeeeee
hhaps///tthnb.cm//kkapppaale Mahalanobis detector
classifier
7

---

Under review as a conference paper at ICLR 2022
Table 1: AUC (Area Under the ROC curve) (%) on detecting IND and OOD samples 2 settings. Results are averaged over the 5 partitions on setting 1. CGA-s is the CGA-softmax model and CGA-e the CGA-energy model. Every experiment was run 5 times.
Duiseets
Sofima
OpenMax
ODIN
Maha
Ccc
OSRCI
CAC
SupCLR
CSI
CGA-s
CGA
Setting 1- OOD Detection the Same Dataset
MINIT
97.5200.7
98.1099
9711002
96.31.022
CIFAR-10
6,.5:00.5
6.49-0.4
1.
73.4222
74.0114
67.51088
739::077
800205
84.10033
86.31:12
85.6:00.6
SVHN
90.3-+0.5
9..7=0.4
89.4+22.
91.5+0066
64.44223
91.74022
938+02
93840022
9394-0.1
91,8+-055
92.1=004
Tax Emageet
57,5200.7
07
1.
56.3119
51.0::12
58.11044
7194007
72.24:0.5
7.,7::0.6
Average
77.8
78.4
71.0
Setting 2-00D Detection on Difforent
Datasess (CIFAR-10 IND)
SVIIN
80.221.8
82.7
$3.2
97.511.6
833308
502118
67.3146
91.100.1
97.910.11
95.1:066
26.2:225
LSLN
70.1+:2.5
72.2-1.8
79994118
96.84-1.4
LSUN-FIX
76.7.20.8
75.6=1.22 1.2
8..1::17
77.8:12.11
16.6.116
782110.5
1..7
91661:15
93.5::0.44
94.1:10.9
93.7=00.4
Timp Imt
62.52:3.6
65.2=13.1
68.7:222
56.8:12111
83.2:118
1.00:::7
86.11:466
914112
97.6.0033
94.8:1
95.2:227
ImageNet FIX
75.9:24.6
75.6=0.7
7488=066
79.0++3.1
1317+1111
78.1-033
85.6+-03
90.4-005
94.01-0.1
89.74-0.3
92,9-1.2
CIFAR10
0..5::0.4
74.5:00.8
6144009
7144004
8394002
87.91:0.4
Average 73.3
74.5
77.9
72.3
$1.1
77.3
863
92.0
95.5
93.2
94.2
Table 2: AUC (Area Under the ROC curve) (%) results of the original model (denoted by original) and the model plus fine-tuning using CGA-energy (denoted by +CGA-e). Almost every +CGA-e version of the baselines outperforms the original model. Every experiment was run 5 times.
Datasets
original
OSRCI
+CGA-
orgginal
SuPCLR
+DGAv
original
CSI
+GGA
1-OD
MINEST
98.3 109
9.110.
91.1.0022
98.6:1022
97.21:033
99.310..1
CIEAR-10
67.5:1:0.8
72.3:10.66
$0.0:10..5
88.9:10.5
84.710.3
89.8:10.66
SVHN
91.74-022
92.14-0.1
93.8-50.22
9634003
9394-0.1
96.74-0.2
TinyImaceNet
4-.14:044
5:091:0.3
Setting 2- -OD Detection om Different Datasset (CIPAR- 10 as IND)
SVHN
812
793-1-233
97.3:50011
1..0:1:.3
97.99:0.1
97.8
LSUN
92.11:0.6
97.7.10.6
97.71:0.4
99.2:1:.11
LSUN-FIX
78:24-0.5
81:24-1100
91.6:1..
94.1:-0.3
93.5++0.4
96.24-0.3
Tiny EnygeeegeNt
70.04-1.7
83.2-11.77
9..4:1122
96.3-1-088
97.6-5-0.3
98.7 14.33
ImageNes-FIXX
78.14:0.3
78.5-:0.2
90.5:20.5
92.920.3
94.0-20.1
95.7
CIFARIOO
7744-04
7744-066
38.6++0.22
90.3--022
92.24-0.11
92.0--0.2
Average
27.3
82.0
92.0
94.1
95.5
96.6
Table 3: Execution time (min) of each method spent in running the whole experiment on benchmark datasets for setting
Datasets MNIST
Soltma
OpenMax
ODIN
Maha
CCC
OSRCI
CAC
SupCLR
CSI
CGA- (ours)
1280
1728
24
CIFAR
20
20
61
56
111
70
1110
1428
144
SVIIN
20
20
142
140
196
71
37
1770
2471
249
Trny ImngeNet
22
22
64
54
46
79
131
CGA Stage 2. To verify the effect of different options of stage 2, we compare the results of CGA-e model with (1) without stage i.e. directly use energy score of stage 1 to compute AUC, (2) stage 2 without using pseudo OOD data, and (3) full stage 2. Figure (a) shows that without stage stage produces poor results. Stage 2 without the generated pseudo OOD data only improves the performance slightly. The full stage 2 with the generated pseudo data greatly improves the performance of OOD detection. The experiments prove the necessity of stage and the effectiveness of the generated pseudo OOD data. We do not vary stage as our contribution is in stage 2.
Amount of Pseudo OOD Data. We run experiments of stage 2 with different numbers of generated pseudo OOD samples to further analyze their effectiveness. Figure 2 demonstrates that the model can benefit from only few pseudo samples significantly. Though the amount is only 10% of the IND data, the pscudo data can improve the results markedly, which indicates that pseudo samples highly effective. The results are similar when pscudo samples are more than half of the IND training samples. We use the same number of pseudo samples the IND samples in all our experiments.
Pseudo OOD Data Distribution. The CVAE generator is trained to make the latent variables or features conform to the Gaussian distribution (0,0 I) (see Section 3.2). To make pseudo data diverse and different from the training data, we sample the latent variables z from a pseudo data sampling distribution A a² I). We conduct experiments to study the effect of the distribution. First,
8

---

Under review as a conference paper at ICLR 2022
tood stger
partoon
pettoon
pettord
0.9
1.aap 2 aittsssaaaa
0.55
0.9
AUCin
0.85
0.
9.75
6.75
parthoond
parttton2
partiiond
parthioné partitons
0.5
1.5
partitions CIFARIO
The Psoudo training Data
(a) Different options of CGA Stage
(b) Ratio of pseudo OOD data IND data.
Figure 2: Ablation study (a) CGA Stage 2 and (b) amount of generated pseudo OOD data.
study the influence of a. Note that a is training. With larger a values, the sampled values will be more likely to be far from (which is the mean) to make the latent features different from those seen in training. Figure a) shows the results, which indicate the necessity of using o > 1. Results similar when
5 and use 5 in experiments.
Intuitively, we may only keep latent features z that are far from the Gaussian distribution mean by filtering values that are close 0 (or the mean). We out the sampled z whose component values are within the range [-t, tj. Experimental results in Figure bb) allow us make the following observations. When a improves slightly. But comparing with the results in Figure 3)). we see that a larger improves the performance more. Figure B(c) tells us that when o 5. the effect of filtering diminishes. For simplicity and efficiency, all our experiments employed o = 5 without filtration.
filtering threshold t to filter
grows, the performance
0.2
AUC.nn 90
0.9
W
0
10
05
15
25
05
15
25
of the Distribution
Filter Thresholdd
Riter Threshole
(a) Influence of
(b) Filter under
(c) Filter under
Figure 3: Ablation study on different options of generating pseudo OOD data. Figure 3(a) shows AUC results of different values of the sampling distribution. Figure 3(b) filters values near the center of the Gaussian distribution with a 1. Figure 3(c) filters values near the center of the Gaussian Distribution with = 5.
CONCLUSION
has been shown recently that contrastive learning with extensive data augmentations can pro- duce the state-of-the-art out-of-distribution detection results. Our experiments also confirmed that. However, such algorithms extremely inefficient and resource hungry due to a large amount of augmented data and quadratic pairwise similarity computation, which makes such algorithms un- suitable for applications that do not have the required computing resources, e.g., edge devices. In this paper, we proposed a simple alternative based on data generation using CVAE which delivers similar accuracy results is much more efficient. What is also interesting is that our proposed approach improve the results of those state-of-the-art contrastive learning based methods too. Thus, our future work will focus on improving the detection accuracy while maintaining or fur- ther reducing the computing resources requirements. We also plan to explore pre-trained feature extractors and more advanced variants of CVAE to improve our method.
On broader impacts, we believe that the proposed approach has great potential and may also be applicable to one-class learning, positive and unlabeled (PU) learning, and continual learning. for these problems, the ability generate pseudo OOD data is useful.

---

Under review as a conference paper at ICLR 2022
ETHICS STATEMENT
We believe that our work has no ethic issues as we propose general algorithm for out-of-distribution detection, which not for any specific application. Our experimental datasets are public domain datasets.
REPRODUCIBILITY STATEMENT
We have submitted our code in the supplementary material.
REFERENCES
Pierre Baldi and Kurt Hornik. Neural networks and principal component analysis: Learning from examples without local minima. Neural networks, 2(1):53-58, 1989.
Abhijit Bendale and Terrance Boult. Towards open world recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1893-1902, 2015.
Abhijit Bendale and Terrance E Boult. Towards open set deep networks. In Proceedings the IEEE conference on computer vision and paltern recognition, pp. 1563-1572, 2016.
Liron Bergman and Yedid Hoshen. Classification-baseed anomaly detection for general data. arXiv preprint arXiv:2005.0235 2020.
Rich Caruana, Yin Lou, Johannes Gehrke, Paul Koch, Mare Sturm, and Noemic Elhadad. Intel- ligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 2/th ACM SIGKDD international conference on knowledge discovery and data mining, pp. 1721-1730. 2015.
Taylan Cemgil, Sumedh Ghaisas, Krishnamurthy Dj Dvijotham, and Pushmeet Kohli. Adversarially robust representations with smooth encoders. In International Conference Learning Represen- tations, 2019.
Minmin Chen, Zhixiang Xu, Kilian Weinberger, and Fei Sha. Marginalized denoising autoencoders for domain adaptation. arXiv preprint arXiv: 4683, 2012.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. International conference on machine learning, pp. 1597-1607. PMLR, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self- supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006. 10029, 2020b.
Lucas Deecke, Robert Vandermeulen, Lukas Ruff, Stephan Mandt, and Marius Kloft. Image anomaly detection with generative adversarial networks. In Joint european conference on ma- chine learning and knowledge discovery in databases, pp. 3-17. Springer, 2018.
William Falcon and Kyunghyun Cho. A framework for contrastive self-supervised learning and designing new approach. arXiv preprint 20X0::000.00/044 2020.
Stanislay Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out-of-distribution detection, arXiv preprint arXiv:2 2021.
Zong Yuan Ge, Sergey Demyanov, Zetao Chen, and Rahil Garnavi. Generative openmax for multi- class open set classification. arXiv preprint
2017.
Spyros Gidaris. Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv: 18X3:18003.0728. 2018.
Izhak Golan and Ran El- Yaniv. Deep anomaly detection using geometric transformations. arXiv preprint arXiv: 1805. 1091.1001517777 2018.
10

---

Under review as a conference paper at ICLR 2022
Manuel Gunther, Steve Cruz, Ethan M Rudd, and Terrance E Boult. Toward open-set face recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 71-80, 2017.
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning, pp. 1321-1330. PMLR, 2017.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out- distiiibibtttion examples in neural networks. arXiv preprint 1610.02 2016.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. arXiv preprini 18/2. 2018.
Dan Hendrycks, Mantas Mazeika, Saurav Kadavath, and Dawn Song. Using self-superriieeeeeeeee learning can improve model robustness and uncertainty. arXiv preprint arXiv: 12340, 2019.
Geoffrey E Hinton, Simon Osindero, and Yee-Whi Whye Teh. A fast learning algorithm for deep belief nets. Neural computation,
2006.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchilee Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. arXiv preprint arXiv: 066700 2018.
Victoria Hodge and Jim Austin. A survey of outlier detection methodologies. Artificial intelligence review, 22(2):85-126, 2004.
Pedro R Mendes Júnior, Roberto M De Souza, Rafael de O Werneck, Bernardo V Stein, Daniel V Pazinato, Waldir R de Almeida, Otávio AB Penatti, Ricardo da S Torres, and Anderson Rocha. Nearest neighbors distance ratio open-set classifier. Machine Learning, 106(3):35--386, 2017.
Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. arXiv preprint 1rX1v 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 1412 (9802,6668.0000 2014.
Diederik Kingma and Max Welling. arXiv:1312.61 14, 2013.
Auto-encoding variational bayes. arXiv preprint
Diederik P Kingma, Danilo J Rezende, Shakir Mohamed, and Max Welling. Semi-supervised learn- ing with deep generative models. arXiv preprint arXiv: 140v:5500.5295 2014.
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual represen- tation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1920-1929, 2019.
Alex Krizhevsky and Geoff Hinton. Convolutional deep belief networks on cifar-10. 10. Unpublished manuscript, 40(7):1-9, 2010.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. arXiv preprint 1,Xiv:1662..0147,, 2016.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7:7, 2015.

---

Under review as a conference paper at ICLR 2022
Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. 2010.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training www.fiddenne-aalbrrtttto classifiers
for detecting out- dfdstiibbbttnn samples, arXiv preprint arXiv: 2017.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. simple unified framework for detecting out-of-distribution samples and adversarial attacks. Advances in neural information processing systems, 31, 2018.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. arXiv preprint 17X.v:7700.026900 2017.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy-based out-of-distribution detection. arXiv preprint arXiv:2010.03759, 2020.
Dimity Miller, Niko Sunderhauf, Michael Milford, and Feras Dayoub. Class anchor clustering: A loss for distance-based open set recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3570-3578, 2021.
Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable distriiiitiiiioonnnn detection. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 5216-5223, 2020.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan, Do deep generative models know what they don't 't know? arXiv preprint arXiv:1110.09136, 2018.
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, and Balaji Lakshminarayanan. Detecting out-of-distribution inputs to deep generative models using a test for typicality. arXiv preprint arXiv: 029994, 5:5, 2019.
Lawrence Neal, Matthew Olson, Xiaoli Fern, Weng- Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 613-628, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 427-4444444 426.436, 2015.
Julia Nitsch, Masha Itkina, Ransalu Senanayake, Juan Nieto, Max Schmidt, Roland Siegwart, Mykel Kochenderfer, and Cesar Cadena. Out-of-distribution detection for automotive percep- tion. arXiv preprint arXiv:2011.01413. 2020.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predic- tive coding. arXiv preprint 18X 2018
Poojan Oza and Vishal M Patel. C2ae: Class conditioned auto-encoder for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 2307-2316, 2019.
Pramuditha Perera, Ramesh Nallapati, and Bing Xiang. Ocgan: One-class novelty detection using gans with constrained latent representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2898-2906, 2019.
Stanislav Pidhorskyi, Ranya Almohsen, Donald Adjeroh, and Gianfranco Doretto. Generative probabilistic novelty detection with adversarial autoencoders. arXiv preprint 2018.
02588,
Masoud Pourreza, Bahram Mohammadi, Mostafa Khaki, Samir Bouindour, Hichem Snoussi, and Mohammad Sabokrou. G2d: Generate to detect anomaly. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2003-2012, 2021.
12

---

Under review as a conference paper at ICLR 2022
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pp. 5389-5400.
PMLR, 2019.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark DePristo, Joshua V Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out-of-distribution detection. arXiv preprini arXiv: /etv:/906.0000 02845, 2019.
David E Rumelhart, Geoffrey Hinton, and Ronald J Williams. Learning representations by back- propagating errors. nature, 363(6888 536-536, 1986.
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with gram matrices. In International Conference on Machine Learning, pp. 8491-8501. PMLR, 2020.
Vikash Sehwag, Mung Chiang, and Prateek Mittal. Ssd: A unified framework for self-supe outlier detection. arXiv preprint arXiv:2103. 2021.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. In- put complexity and out-of-distribution detection with tikelihood-based generative models. arXiv preprint 1r0iv 1119999.11484.. 1480, 2019.
Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, and Michael K Reiter. Accessorize to crime: Real and stealthy attacks on state-of-the-art face recognition. Proceedings of the 2016 acm sigsac conference computer and communications security, pp. 1528-1540, 2016.
Xin Sun, Zhenning Yang, Chi Zhang, Keck- Voon Ling, and Guohao Peng. Conditional gaussian distribution learning for open set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13489, 2020.
Jihoon Tack, Sangwoo Mo, Jongheon Jeong, and Jinwoo Shin. Csi: Novelty detection via contrastive learning distributionally shifted instances. arXiv preprint rxXiv:2077008176. 2020.
Sachin Vernekar, Ashish Gaurav, Vahdat Abdelzad, Taylor Denouden, Rick Salay, and Krzysztof Czarnecki. Out-of-distribution detection in classifiers via generation. arXiv preprint arXiv:1910.04241, 2019.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Picrre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pp. 1096-1103, 2008.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint 033603366000, 2015.
Matthew D Zeiler, Graham W Taylor, and Rob Fergus. Adaptive deconvolutional networks for mid and high level feature learning. In 2011 International Conference Computer Vision, pp. 2018 2025. IEEE, 2011.
Da- Wei Zhou, Han-Jia Ye, and De-Chuan Zhan. Learning placeholders for open-set recognition. arXiv preprint 1rXiv:2103. 2021.
Bo Zong, Qi Song, Martin Renqiang Min, Wei Cheng, Cristian Lumezanu, Daeki Cho, and Haifeng Chen. Deep autoencoding gaussian mixture model for unsupervised anomaly detection. In Inter- national Conference on Learning Representations, 2018.
A IMPLEMENTATION DETAILS OF BASELINES
For OOD detection on the same dataset, since the partitions of in-distribution (IND) and out-of- distribution (OOD) classes from dataset affect the results significantly, show the detailed results of individual partitions in the next section. As for OOD detection on different datasets, we copy SupCLR and CST's results from the CSI paper and produce own results for the other baselines.
13

---

Under review as a conference paper at ICLR 2022
Table 4: AUC (Area Under the ROC curve) (%) on detecting OOD samples from the dataset. Results of the 5 partitions of each dataset are listed in the table.
Partitions
Softmax
OpenMax
OSRCI
SupCLR
CSI
CGA-s
CGA-e
MNIST
Partition
97.9
97.6
97.3
97.5
96.8
98.0
99.0
Partition2
95.2
96.7
98.8
96.8
95.4
98.4
98.9
Partition3
98.3
98.6
98.0
96.6
97.6
98.4
99.1
Partition4
98.4
98.6
98.3
97.9
98.4
97.6
98.7
Partition5
98.0
98.8
99.1
96.5
97.7
98.9
99.3
Range
3.2
2.1
1.8
1.4
3.0
1.3
0.6
CIFAR-10
Partition1
62.4
62.2
64.2
77.3
81.8
81.6
79.7
Partition2
61.8
62.2
65.7
75.3
76.3
84.8
80.6
Partition3
67.3
70.4
70.8
88.7
93.5
91.9
91.5
Partition4
64.5
66.1
67.7
77.3
85.2
89.7
89.9
Partition5
69.9
71.4
72.3
81.2
86.6
83.3
86.3
Range
8.1
9.2
8.1
13.4
17.2
10.3
11.8
SVHN
Partitionl
88.4
89.3
89.9
92.7
93.3
91.3
90.5
Partition2
90.0
90.4
91.7
92.1
91.9
90.4
91.3
Partition3
90.8
90.3
92.1
94.2
95.3
92.1
92.3
Partition4
90.8
91.4
92.4
95.2
94.6
92.7
93.2
Partition5
91.6
91.8
92.5
95.2
94.3
92.5
93.3
Range
3.3
2,4
2.6
3.2
3,4
2.3
2.8
Imayema
Partition
57.2
57.9
58.9
73.2
75.0
Partition2
56.4
56.3
56.9
73.0
74.0
Partition3
61.9
62.3
62.9
71.2
73.0
Partition4
60.3
60.3
60.6
74.4
75.8
Partition5
51.6
52.8
51.2
69.3
70.8
Range
10.3
9.5
11.7
5.1
5.0
B DETAILED EXPERIMENT RESULTS
We found that when conducting OOD detection experiments using the same dataset (i.c., some classes of the dataset used as the IND classes and the rest the OOD classes), the choices of IND and OOD classes can affect the results greatly. Some partitions of IND and OOD classes very hard while some easy. Therefore, comparison between different methods fair. Table 3 shows the average AUC score for each of the partitions of each dataset. From Table 4 we learn that the ranges (or differences) of the results for different IND and OOD partitions of the datasets are very large. On the CIFAR-10 dataset, the difference is large as 17.2%.
fix 5 sets of IND and OOD classes make the
C VISUALIZATION OF DATA DISTRIBUTION
We visualize the latent representations of different data learn their distributional property. The visualization is done on partition 1 of MNIST dataset. Figure shows the distribution of the gen- erated data after dimensionality reduction by t-distributed Stochastie Neighbor Embedding (t-SNE) method. The left figure shows 6 clusters of IND data of 6 classes. Red points in the middle figure represent our generated pseudo data. This figure demonstrates that a large number of pseudo sam- ples are around the boundaries of the clusters (representing the known classes). These the most effective OOD-like samples. At the same time, our generated data distribute widely, which shows that the pseudo data is of great diversity. Pink points in the right figure represent real OOD data during testing. Comparing red points and pink points, we can learn that the diverse pseudo data can cover nearly all areas where there true 0OD instances.
14

---

Under review as a conference paper at ICLR 2022
5
5
2
2
2
Figure 4: Distribution of IND data and the generated pseudo OOD data. The grey points represent the IND data and the numbers are their class labels. The red points in the middle figure represent the generated pseudo OOD data and the pink points in the right figure represent the true OOD data used testing. The left figure contains only the IND data of 6 classes
D VISUALIZATION OF PSEUDO OOD IMAGES
Figure shows the generated pseudo OOD images from our CVAE. For each group, first employ the standard class embedding and sample latent variables to generate 2 IND images. Then we use these class embeddings to compute our pseudo class embedding (see the paper) and then generate pseudo OOD images with variables or features from different sampling distributions. NO, a² * I). pseudo OOD images still resemble the IND images great deal. But as grows, the difference between the IND images and pseudo OOD images gets greater and the diversity also increases. That also explains why changing can improve the final performance.
easy to see some characteristics of IND images from pseudo images. When
15

---

Under review as a conference paper at ICLR 2022
SCE
PCE & a
PCE & a 2
PCE & a 3
0
Group 1
5
B
7
Group
2
3
4
3
3
3
Group 3
3
4
5
Figure 5: Generated data of different settings. SCE means standard class embedding and this col- umn shows the generated data from CVAE using the standard class embedding. PCE means pseudo class embedding computed using the class embeddings of the classes in each group (e.g., 0 and 5 in Group 1). For each setting of PCE & a, we generate 4 images by sampling different z values. is the parameter of the sampling distribution for 2, ( o² I).
16


Reviews:
[{"review_id": "o3GgGCR6OcM", "reviewer": "Reviewer_RyWJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n1.This paper claims that they propose a general and much more efficient out-of-distribution detection model. \n2.This paper shows that a simple CVAE model can generate pseudo OOD data to assist the training phase.\n\nWeaknesses\n1.It is obvious that this paper applies CVAE to the OOD data detection. The question is why to select CVAE as the efficient model to generate the OOD data. What is the motivation?\n2.This paper claims that we can already produce comparable results to existing SOTA contrastive learning models but much more efficient. Why? The detailed explanation is necessary.\n3.The contribution is mainly the metrics. ", "summary_of_the_paper": "This paper proposes to employ a CVAE structure to generate pseudo OOD samples by providing some synthetic conditional information. Besides, they design a two-stage framework to train an OOD detection model by leveraging the generated pseudo OOD data. They show that their approach outperforms other SOTA methods in the task of out-of-distribution detection.", "summary_of_the_review": "Overall, the paper does not have enough interesting results for acceptance.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ld9PqRSdCDs", "reviewer": "Reviewer_BUDw", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n* I'd say the main strength in this paper is the empirical results - particularly the combination with CSI seems to help, sometimes significantly.\n* In the same vain as the above strength, the hyper-parameter ablations are quite thorough - and seem to cover the hyperparameters that I find interesting.\n* The paper is quite clear and easy to follow.\n\nWeaknesses:\n\n* Novelty: using generative models has been extensively done in anomaly detection in many ways including synthesis of anomalies (e.g. [1]). The basic ideas are similar to this paper (but as there has been a lot of previous work, there are also some more nuanced ideas). \n* CVAE are only fast for small images - larger images are very hard to generate reliably, particularly with VAE. Conveniently, the datasets evaluated here are small-scale but I find it hard to imagine the sa,e results would follow for ImageNet or another suitable large-scale dataset.\n* The choice of CVAE does not seem to have been seriously ablated. There are many CVAEs possible as well as GAN-based and more recent models. This comparison could have significantly helped the significance of this work.\n* The advantage of training time is interesting, but this (and better accuracy) can easily be achieved by using ImageNet-pretrained representations. E.g. [2] and earlier references. Although these methods do not apply for tabular data, but I suspect that CVAE will also not work well there.\n\n[1] Pourreza et al., G2D: Generate to Detect Anomaly, WACV'21\n[2] Fort et al.,  \"Exploring the Limits of Out-of-Distribution Detection, NeurIPS'21.\n", "summary_of_the_paper": "This paper suggests to model outliers using a CVAE. It then suggests simple scoring functions for incorporating these simulated outlier into a classifier and uses a confidence based criterion. A numerical comparison shows this method is able to outperform CSI on a couple of datasets (while it underperforms on most other ones) . It also shows that synthetic anomalies can be combined with existing methods and obtain gains in performance.", "summary_of_the_review": "Although the empirical results are encouraging, I believe this paper still does not make the bar in two aspects: i) conceptually - the ideas of synthesizing anomalies or using generative models for this purpose are not new, and the techniques used here are not new either - not has there been a very extensive analysis of them ii) practically - better results can be obtained using other methods such as ImageNet-pretraining, so in terms of practical utility to the engineer - this might not be the first choice.\n\n############################\n\nI thank the authors for the response. I do not believe it addresses my main concern i.e. lack of convincing ablation. I therefore kept my borderline reject rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bXaLL-a8jPI", "reviewer": "Reviewer_eP8b", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1) Use of the class-conditioned generative models to generate pseudo-OODs seems novel and interesting to me.\n2) The paper is well-organized and easy to read.\n3) Result of the paper is competitive against the SOTA method, CSI, while the paper benefits from a 10x decrease in computation time.\n\nWeakness\n1) The proposed strategy requires knowledge of the labels, while CSI can be operated without labels via utilizing data augmentation. Following the analogy, I think adding more baselines that use knowledge from the classifiers can be beneficial. (e.g. [1])\n\nComments.\n1) As mentioned in the previous section, I suggest adding more baselines that perform OOD detection from pre-trained classifiers.\n2) I'm curious to see the ablation study that does not freeze the feature extractor in the 2nd phase of training CGA.\n3) As CGA utilizes label information for training the CVAE, I am curious about the robustness of CGA in the case of corrupted, or noisy labels. For example, when no labels are given, we can generate the pseudo-label from clustering and apply the labels to the pipeline.\n4) How's the result in the CIFAR-100 dataset? while I see the results on the tinyImageNet dataset, I think the result would be strengthened by adding CIFAR-100 results.\n\n\n***********References**********\n[1] Detecting Out-of-Distribution Examples with Gram Matrices", "summary_of_the_paper": "UPDATE2\n\nI acknowledge that I've read the author responses as well as the other reviews\n\nI think the authors gave a good explanation of where their proposed method outperforms the conventional methods. Therefore, I increase the score to 6 weak accept. \n\n=========================================================================\n\nUPDATE: \nI acknowledge that I‘ve read the author responses as well as the other reviews. \nI initially gave 6 weak accept to the paper. However, after reading the author response and other reviewers' comments, I think the proposed work lose its merit due to several reasons.\n\n1. The proposed model utilizes label information and underperforms against conventional methods that utilize label information. (see the comments). As such methods do not train an extra model, the proposed method's efficiency becomes also vague.\n\n2. As Reviewer BUDw mentioned, I also agree that CVAE possibly becomes inefficient when the image size becomes larger.\n\n3. Furthermore, as the performance and efficiency become vague, I'm not sure about the novelty, or how \"surprising\" the proposed method can be. The application of the generative models to OOD detection has a long history and better generative models will give better results.\n\nDue to the following reasons, I am leaning towards rejection of the paper and downgrading the score to 5. I hope the authors clarify the issues.\n\n================================================================================================\n\n\nThe paper proposes CGA (CVAE-based generative data augmentation for out-of-distribution (OOD) detection) to improve the OOD detection performance when the class information is given. The module consists of a classification module (classifier and feature extractor) and a generation module (encoder and decoder). In the first phase, only in-distribution data are used to train two modules. In the second phase, CGA utilizes the generated pseudo-OOD data to fine-tune the classifier. CGA is compared against SOTA baselines in various setups. In the majority of setups, CGA shows competitive performance against the conventional OOD detection methods with the advantage of reduced computation times. Various ablation supports the design of CGA.", "summary_of_the_review": "Overall, I liked the proposed algorithm and think that CGA can be a competitive baseline for supervised OOD detection. Therefore I'm leaning towards accepting the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9lNcluZqBp", "reviewer": "Reviewer_N6Uq", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well written, covers the related works, and importantly, is well motivated.\nThe usage of 'known OOD' samples in order to improve OOD detection is very common, and while in some times there is abundance of such `known OOD` data, that is often not the case.\n\nThe proposed method is rather simple, and is implemented in two baselines, creating surprisingly good results.\n\nThe reason the results are surprising, in the CIFAR-10 experiments (in my opinion),\nis that extrapolating on the latent space will often results with samples that in high probability will not resemble any thing the model will see in the future (OOD or ID).\nIn addition, the results of a VAE reconstruction are often blurry, unless explicit efforts has been made to prevent that, which none were done.\n\nTo summarize, I would expect the model to learn how to classify the generated output of the VAE as OOD, while 'real' images, such as CIFAR-100 (which was used as OOD set) still get classified ad inset.\n\nIf the authors could elaborate - what was done to prevent the above?\n\nAn interesting ablation would be to use `Known OOD` data in the same baselines, to measure how well the generated samples are compared to real OOD data.\n\nIn addition, results on more difficult datasets could contribute - Cifar100 as IND, or even the entire TinyImageNet as IND. The current partitioning of TinyImageNet leaves only 20 classes IND in each partition, which is not 'difficult' enough.", "summary_of_the_paper": "The authors proposed to use a Conditional VAE in order to generate pseudo OOD data, which can be used to improve OOD detection.\nThe authored demonstrated the method in several benchmarks, with good results.", "summary_of_the_review": "The authors proposed what seems like a simple solution (which is great in my opinion) for an important problem, however they have overcome several challenging problems and achieved good results.\nThe paper itself is well written, easy to follow, and is very detailed regarding the implementation and experiment settings.\n\n====================================\n\nFollowing the discussion with the other reviewers, I have decided to keep my current score. As other reviewers have pointed out, the selection of the CVAE is not justified enough, and a serious ablation study should be added to justify it and not other generative methods. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper adopts CVAE to generate OOD samples for training an outliner detector. It consists of two phases that train an OOD detector by leveraging the generated OOD data and shows it outperform other methods. According to reviewers’ discussion, there is a concern from the discussion: why CVAE works but other variants or cGAN doesn’t. The paper needs more motivation or evidence or ablations to support the generality of the work.
Decision: Reject