Efficient Out-of-Distribution Detection via CVAE data Generation

1 INTRODUCTION . With the development of deep learning , a well-trained neural network model is able to obtain very high accuracy on its testing data . However , when exposed to samples or data instances drawn from a distribution that is far from the training distribution ( called In-distribution ( IND ) ) , the model may make arbitrary predictions under the known framework ( Nguyen et al. , 2015 ; Recht et al. , 2019 ) . This limits the model ’ s application in a broad range of applications , including secure authentication ( Sharif et al. , 2016 ) , autonomous driving ( Nitsch et al. , 2020 ) and medical diagnosis ( Caruana et al. , 2015 ) as in these applications novel or out-of distribution ( OOD ) data instances occur frequently . Therefore , OOD detection ( Hodge & Austin , 2004 ) , which aims to detect abnormal or novel data that are very different from the training data , is an important research task . Many approaches have been proposed to solve this problem , from distance-based methods ( Bendale & Boult , 2015 ; 2016 ; Gunther et al. , 2017 ; Júnior et al. , 2017 ) , to generative models ( Ge et al. , 2017 ; Neal et al. , 2018 ; Oza & Patel , 2019 ; Nalisnick et al. , 2018 ) and self-supervised learning ( Bergman & Hoshen , 2020 ; Golan & El-Yaniv , 2018 ; Hendrycks et al. , 2019 ) . Recently , contrastive learning has been shown highly effective in many applications ( Hjelm et al. , 2018 ; Oord et al. , 2018 ; Chen et al. , 2020b ; a ; Falcon & Cho , 2020 ) . Applying supervised contrastive learning and data augmentation , the recent CSI system has produced state-of-the-art ( SOTA ) OOD detection results ( Tack et al. , 2020 ) . However , data augmentation-based contrastive learning also has some drawbacks . First , designing data augmentation functions and deciding how to use various types of augmented data in contrastive learning involve a great deal of trial and error and manual work . That is , there is a large number of transformations ( or augmentations ) such as cropping , rotation and gray-scaling that can be exerted on images but not all of them may benefit the end tasks . In SimCLR ( Chen et al. , 2020a ) , systematic experiments have been reported to explore the augmentations ’ influence on classification tasks . Based on SimCLR ’ s results , CSI ( Tack et al. , 2020 ) chooses several operations for OOD detection . Second , contrastive learning with data augmentation is very time-consuming to run and resourcehungry due to a large amount of augmented data and quadratic pairwise similarity computation during training . For example , CSI creates 3 shifted instances for each original image sample and the 4 images are then subjected to an additional augmentation operation . Each image is finally expanded to 8 images or samples . Furthermore , every 2 samples in the augmented batch is treated as a pair to calculate contrastive loss . The performance is negatively affected if the batch size is not large enough . Therefore , it is necessary to use a large batch size , which demands a huge amount of memory and takes a very long time to run . It is thus not suitable for applications on hardware devices that do not have the required resources such as edge devices . In this paper , we propose a general and much more efficient solution , called CGA ( CVAE-based Generative data Augmentation for OOD detection ) . CGA consists of two parts . The first part generates pseudo OOD data and the second part makes use of the pseudo OOD data to train an OOD detection model . We discuss the first part first . OOD detection is basically a classification problem but the challenge is that there is no OOD data to use in training . This paper proposes a novel method to generate pseudo OOD data . This method works in the latent space of a Conditional Variational Auto-Encoder ( CVAE ) and uses CVAE ’ s decoder to generate pseudo OOD data . CVAE is able to generate instances from the training distribution on the basis of latent representations consisting of conditional information and variables sampled from a prior distribution of CVAE , normally the Gaussian distribution . If the latent space features or representations are created with some abnormal conditional information , the CVAE will generate “ bad ” instances but such instances can serve as effective pseudo OOD data . The second part of CGA is a fine-tuning framework that can make use of the generated pseudo OOD data to fine-tune any classification model built using only the IND data . Applying the framework to a simple IND classification model , we can already produce comparable results to existing SOTA contrastive learning models but much more efficient . Applying it to the existing SOTA methods , we can improve their results too . Our contributions can be summarized as follows : ( 1 ) We propose to employ a CVAE structure to generate pseudo OOD samples by providing some synthetic conditional information , which , to our knowledge , has not been done before . ( 2 ) We design a two-stage framework to train an OOD detection model by leveraging the generated pseudo OOD data . The first stage simply builds a normal classification model using only the IND data . The second stage fine-tunes the model using the generated OOD data to produce an effective OOD detection model . Extensive experiments show that this approach achieves comparable performance to the state-of-the-art ( SOTA ) contrastive learning methods in OOD detection , while consuming only one-tenth of the execution time . ( 3 ) Equally importantly , the proposed framework can be applied to existing SOTA OOD detection models to improve them to produce new SOTA results . 2 RELATED WORK . Out-of-distribution Detection . It is well-known that the discriminative neural networks can produce overconfident predictions on out-of-distribution ( OOD ) inputs ( Hendrycks & Gimpel , 2016 ; Lakshminarayanan et al. , 2016 ) . The early idea to solve this problem focuses on modifying softmax scores to obtain calibrated confidence for OOD detection ( Bendale & Boult , 2016 ; Guo et al. , 2017 ) . In addition , many other score functions have been proposed , such as likelihood ratio ( Ren et al. , 2019 ) , input complexity ( Serrà et al. , 2019 ) and typicality ( Nalisnick et al. , 2019 ) . A recent work utilizes Gram matrices to characterize activity patterns and identify anomalies in Gram matrix values to do OOD detection ( Sastry & Oore , 2020 ) . Some methods found that auxiliary anomalous data significantly improve detection performance ( Hendrycks et al. , 2018 ; Mohseni et al. , 2020 ) and thus generative models are adopted to anticipate the distribution of novel samples . In some of these methods , generated data are treated as OOD samples to optimize the decision boundary and calibrate the confidence ( Ge et al. , 2017 ; Vernekar et al. , 2019 ) . In some other works , generative models such as auto-encoders ( Zong et al. , 2018 ; Pidhorskyi et al. , 2018 ) and generative adversarial networks ( GAN ) ( Deecke et al. , 2018 ; Perera et al. , 2019 ) are used to reconstruct the training data . During the training of a GAN model , low quality samples acquired by the generator can also work as OOD data directly ( Pourreza et al. , 2021 ) . Owing to the fact that the model can hardly be generalized to unknown data , the reconstruction loss can help detect OOD data . There are also works using auxiliary OOD data to fine-tune the model ( Liu et al. , 2020 ) . Fort et al . ( 2021 ) showed that using pre-trained representations and taking few-shot outlier exposure can improve the results . Recently , self-supervised techniques have been applied to OOD detection . It focuses on acquiring rich representations through training with some pre-defined tasks ( Gidaris et al. , 2018 ; Kolesnikov et al. , 2019 ) . Self-supervised models show outstanding performance on OOD detection tasks ( Kolesnikov et al. , 2019 ; Bergman & Hoshen , 2020 ) . CSI ( Tack et al. , 2020 ) is a representative method ( see more below ) , which uses contrastive learning and data augmentation to improve the features of all labeled IND data and produce state-of-the-art ( SOTA ) results . Some researchers also tried to improve contrastive learning based methods ( Sehwag et al. , 2021 ) and proposed distance-based methods ( Miller et al. , 2021 ) . However , based on our experiments , CSI outperforms them . Our method falls into the generative approach . Unlike existing methods that use perturbations to anticipate OOD data , our method uses synthetic conditions and CVAE to obtain effective and diverse pseudo OOD data . Contrastive Learning . Contrastive learning learns representations by contrasting positive pairs against negative pairs ( Hadsell et al. , 2006 ) . It has been applied to various domains ( Oord et al. , 2018 ) . Recently , a new method called SimCLR ( Chen et al. , 2020a ) was proposed to create sample pairs via data augmentation . It is effective but also very time and resource consuming . SimCLR also shows that contrastive learning benefits more from larger batch sizes and longer training time . CSI ( Tack et al. , 2020 ) proposes that augmentation can not only help construct positive pairs but also negative pairs and makes use of them to detect OOD samples with supervised contrastive learning . It obtains the SOTA OOD detection results with labeled IND data . However , due to contrastive learning , it is extremely slow and memory demanding . Our proposed method generates pseudo OOD using CVAE , avoiding the use of contrastive loss , and is much more efficient than CSI . Auto-Encoder Auto-Encoder ( AE ) is a family of unsupervised neural networks ( Rumelhart et al. , 1986 ; Baldi & Hornik , 1989 ) . A basic AE consists of an encoder and a decoder . The encoder encodes the input data into a low-dimensional hidden representation and the decoder transforms the representation back to the reconstructed input data ( Vincent et al. , 2008 ; Chen et al. , 2012 ; Hinton et al. , 2006 ) . Variational auto-encoder is a special kind of AE ( Kingma & Welling , 2013 ) . It encodes the input as a given probability distribution ( usually Gaussian ) and the decoder reconstructs data instances according to variables sampled from that distribution . CVAE is an extension of VAE ( Kingma et al. , 2014 ) . It encodes the label or conditional information into the latent representation so that a CVAE can generate new samples from specified class labels . CVAE makes it possible to control the generating process , i.e. , to generate samples with features of specified classes . We make use of this property of CVAE to generate high quality pseudo OOD data . 3 PROPOSED CGA METHOD . In tasks related to out-of-distribution ( OOD ) detection , the problem of recognition is commonly formulated as a classification problem . The main challenge is that an important class , OOD data , is not available . Therefore , to effectively train an OOD detection model , an intuitive idea is to generate pseudo OOD data and use them together with the IND data to train the model . As we mentioned earlier , data augmentation and contrastive learning have been shown especially effective for this purpose . However , this approach is extremely inefficient . We propose to use Conditional Variational Auto-Encoder ( CVAE ) to generate pseudo OOD data and present a new fine-tuning framework to leverage the generated pseudo OOD data to train an OOD detection model .

Summary: UPDATE2  I acknowledge that I've read the author responses as well as the other reviews  I think the authors gave a good explanation of where their proposed method outperforms the conventional methods. Therefore, I increase the score to 6 weak accept.   =========================================================================  UPDATE:  I acknowledge that I‘ve read the author responses as well as the other reviews.  I initially gave 6 weak accept to the paper. However, after reading the author response and other reviewers' comments, I think the proposed work lose its merit due to several reasons.  1. The proposed model utilizes label information and underperforms against conventional methods that utilize label information. (see the comments). As such methods do not train an extra model, the proposed method's efficiency becomes also vague.  2. As Reviewer BUDw mentioned, I also agree that CVAE possibly becomes inefficient when the image size becomes larger.  3. Furthermore, as the performance and efficiency become vague, I'm not sure about the novelty, or how "surprising" the proposed method can be. The application of the generative models to OOD detection has a long history and better generative models will give better results.  Due to the following reasons, I am leaning towards rejection of the paper and downgrading the score to 5. I hope the authors clarify the issues.  ================================================================================================   The paper proposes CGA (CVAE-based generative data augmentation for out-of-distribution (OOD) detection) to improve the OOD detection performance when the class information is given. The module consists of a classification module (classifier and feature extractor) and a generation module (encoder and decoder). In the first phase, only in-distribution data are used to train two modules. In the second phase, CGA utilizes the generated pseudo-OOD data to fine-tune the classifier. CGA is compared against SOTA baselines in various setups. In the majority of setups, CGA shows competitive performance against the conventional OOD detection methods with the advantage of reduced computation times. Various ablation supports the design of CGA.