Efficient Out-of-Distribution Detection via CVAE data Generation

1 INTRODUCTION . With the development of deep learning , a well-trained neural network model is able to obtain very high accuracy on its testing data . However , when exposed to samples or data instances drawn from a distribution that is far from the training distribution ( called In-distribution ( IND ) ) , the model may make arbitrary predictions under the known framework ( Nguyen et al. , 2015 ; Recht et al. , 2019 ) . This limits the model ’ s application in a broad range of applications , including secure authentication ( Sharif et al. , 2016 ) , autonomous driving ( Nitsch et al. , 2020 ) and medical diagnosis ( Caruana et al. , 2015 ) as in these applications novel or out-of distribution ( OOD ) data instances occur frequently . Therefore , OOD detection ( Hodge & Austin , 2004 ) , which aims to detect abnormal or novel data that are very different from the training data , is an important research task . Many approaches have been proposed to solve this problem , from distance-based methods ( Bendale & Boult , 2015 ; 2016 ; Gunther et al. , 2017 ; Júnior et al. , 2017 ) , to generative models ( Ge et al. , 2017 ; Neal et al. , 2018 ; Oza & Patel , 2019 ; Nalisnick et al. , 2018 ) and self-supervised learning ( Bergman & Hoshen , 2020 ; Golan & El-Yaniv , 2018 ; Hendrycks et al. , 2019 ) . Recently , contrastive learning has been shown highly effective in many applications ( Hjelm et al. , 2018 ; Oord et al. , 2018 ; Chen et al. , 2020b ; a ; Falcon & Cho , 2020 ) . Applying supervised contrastive learning and data augmentation , the recent CSI system has produced state-of-the-art ( SOTA ) OOD detection results ( Tack et al. , 2020 ) . However , data augmentation-based contrastive learning also has some drawbacks . First , designing data augmentation functions and deciding how to use various types of augmented data in contrastive learning involve a great deal of trial and error and manual work . That is , there is a large number of transformations ( or augmentations ) such as cropping , rotation and gray-scaling that can be exerted on images but not all of them may benefit the end tasks . In SimCLR ( Chen et al. , 2020a ) , systematic experiments have been reported to explore the augmentations ’ influence on classification tasks . Based on SimCLR ’ s results , CSI ( Tack et al. , 2020 ) chooses several operations for OOD detection . Second , contrastive learning with data augmentation is very time-consuming to run and resourcehungry due to a large amount of augmented data and quadratic pairwise similarity computation during training . For example , CSI creates 3 shifted instances for each original image sample and the 4 images are then subjected to an additional augmentation operation . Each image is finally expanded to 8 images or samples . Furthermore , every 2 samples in the augmented batch is treated as a pair to calculate contrastive loss . The performance is negatively affected if the batch size is not large enough . Therefore , it is necessary to use a large batch size , which demands a huge amount of memory and takes a very long time to run . It is thus not suitable for applications on hardware devices that do not have the required resources such as edge devices . In this paper , we propose a general and much more efficient solution , called CGA ( CVAE-based Generative data Augmentation for OOD detection ) . CGA consists of two parts . The first part generates pseudo OOD data and the second part makes use of the pseudo OOD data to train an OOD detection model . We discuss the first part first . OOD detection is basically a classification problem but the challenge is that there is no OOD data to use in training . This paper proposes a novel method to generate pseudo OOD data . This method works in the latent space of a Conditional Variational Auto-Encoder ( CVAE ) and uses CVAE ’ s decoder to generate pseudo OOD data . CVAE is able to generate instances from the training distribution on the basis of latent representations consisting of conditional information and variables sampled from a prior distribution of CVAE , normally the Gaussian distribution . If the latent space features or representations are created with some abnormal conditional information , the CVAE will generate “ bad ” instances but such instances can serve as effective pseudo OOD data . The second part of CGA is a fine-tuning framework that can make use of the generated pseudo OOD data to fine-tune any classification model built using only the IND data . Applying the framework to a simple IND classification model , we can already produce comparable results to existing SOTA contrastive learning models but much more efficient . Applying it to the existing SOTA methods , we can improve their results too . Our contributions can be summarized as follows : ( 1 ) We propose to employ a CVAE structure to generate pseudo OOD samples by providing some synthetic conditional information , which , to our knowledge , has not been done before . ( 2 ) We design a two-stage framework to train an OOD detection model by leveraging the generated pseudo OOD data . The first stage simply builds a normal classification model using only the IND data . The second stage fine-tunes the model using the generated OOD data to produce an effective OOD detection model . Extensive experiments show that this approach achieves comparable performance to the state-of-the-art ( SOTA ) contrastive learning methods in OOD detection , while consuming only one-tenth of the execution time . ( 3 ) Equally importantly , the proposed framework can be applied to existing SOTA OOD detection models to improve them to produce new SOTA results . 2 RELATED WORK . Out-of-distribution Detection . It is well-known that the discriminative neural networks can produce overconfident predictions on out-of-distribution ( OOD ) inputs ( Hendrycks & Gimpel , 2016 ; Lakshminarayanan et al. , 2016 ) . The early idea to solve this problem focuses on modifying softmax scores to obtain calibrated confidence for OOD detection ( Bendale & Boult , 2016 ; Guo et al. , 2017 ) . In addition , many other score functions have been proposed , such as likelihood ratio ( Ren et al. , 2019 ) , input complexity ( Serrà et al. , 2019 ) and typicality ( Nalisnick et al. , 2019 ) . A recent work utilizes Gram matrices to characterize activity patterns and identify anomalies in Gram matrix values to do OOD detection ( Sastry & Oore , 2020 ) . Some methods found that auxiliary anomalous data significantly improve detection performance ( Hendrycks et al. , 2018 ; Mohseni et al. , 2020 ) and thus generative models are adopted to anticipate the distribution of novel samples . In some of these methods , generated data are treated as OOD samples to optimize the decision boundary and calibrate the confidence ( Ge et al. , 2017 ; Vernekar et al. , 2019 ) . In some other works , generative models such as auto-encoders ( Zong et al. , 2018 ; Pidhorskyi et al. , 2018 ) and generative adversarial networks ( GAN ) ( Deecke et al. , 2018 ; Perera et al. , 2019 ) are used to reconstruct the training data . During the training of a GAN model , low quality samples acquired by the generator can also work as OOD data directly ( Pourreza et al. , 2021 ) . Owing to the fact that the model can hardly be generalized to unknown data , the reconstruction loss can help detect OOD data . There are also works using auxiliary OOD data to fine-tune the model ( Liu et al. , 2020 ) . Fort et al . ( 2021 ) showed that using pre-trained representations and taking few-shot outlier exposure can improve the results . Recently , self-supervised techniques have been applied to OOD detection . It focuses on acquiring rich representations through training with some pre-defined tasks ( Gidaris et al. , 2018 ; Kolesnikov et al. , 2019 ) . Self-supervised models show outstanding performance on OOD detection tasks ( Kolesnikov et al. , 2019 ; Bergman & Hoshen , 2020 ) . CSI ( Tack et al. , 2020 ) is a representative method ( see more below ) , which uses contrastive learning and data augmentation to improve the features of all labeled IND data and produce state-of-the-art ( SOTA ) results . Some researchers also tried to improve contrastive learning based methods ( Sehwag et al. , 2021 ) and proposed distance-based methods ( Miller et al. , 2021 ) . However , based on our experiments , CSI outperforms them . Our method falls into the generative approach . Unlike existing methods that use perturbations to anticipate OOD data , our method uses synthetic conditions and CVAE to obtain effective and diverse pseudo OOD data . Contrastive Learning . Contrastive learning learns representations by contrasting positive pairs against negative pairs ( Hadsell et al. , 2006 ) . It has been applied to various domains ( Oord et al. , 2018 ) . Recently , a new method called SimCLR ( Chen et al. , 2020a ) was proposed to create sample pairs via data augmentation . It is effective but also very time and resource consuming . SimCLR also shows that contrastive learning benefits more from larger batch sizes and longer training time . CSI ( Tack et al. , 2020 ) proposes that augmentation can not only help construct positive pairs but also negative pairs and makes use of them to detect OOD samples with supervised contrastive learning . It obtains the SOTA OOD detection results with labeled IND data . However , due to contrastive learning , it is extremely slow and memory demanding . Our proposed method generates pseudo OOD using CVAE , avoiding the use of contrastive loss , and is much more efficient than CSI . Auto-Encoder Auto-Encoder ( AE ) is a family of unsupervised neural networks ( Rumelhart et al. , 1986 ; Baldi & Hornik , 1989 ) . A basic AE consists of an encoder and a decoder . The encoder encodes the input data into a low-dimensional hidden representation and the decoder transforms the representation back to the reconstructed input data ( Vincent et al. , 2008 ; Chen et al. , 2012 ; Hinton et al. , 2006 ) . Variational auto-encoder is a special kind of AE ( Kingma & Welling , 2013 ) . It encodes the input as a given probability distribution ( usually Gaussian ) and the decoder reconstructs data instances according to variables sampled from that distribution . CVAE is an extension of VAE ( Kingma et al. , 2014 ) . It encodes the label or conditional information into the latent representation so that a CVAE can generate new samples from specified class labels . CVAE makes it possible to control the generating process , i.e. , to generate samples with features of specified classes . We make use of this property of CVAE to generate high quality pseudo OOD data . 3 PROPOSED CGA METHOD . In tasks related to out-of-distribution ( OOD ) detection , the problem of recognition is commonly formulated as a classification problem . The main challenge is that an important class , OOD data , is not available . Therefore , to effectively train an OOD detection model , an intuitive idea is to generate pseudo OOD data and use them together with the IND data to train the model . As we mentioned earlier , data augmentation and contrastive learning have been shown especially effective for this purpose . However , this approach is extremely inefficient . We propose to use Conditional Variational Auto-Encoder ( CVAE ) to generate pseudo OOD data and present a new fine-tuning framework to leverage the generated pseudo OOD data to train an OOD detection model .

Summary: This paper suggests to model outliers using a CVAE. It then suggests simple scoring functions for incorporating these simulated outlier into a classifier and uses a confidence based criterion. A numerical comparison shows this method is able to outperform CSI on a couple of datasets (while it underperforms on most other ones) . It also shows that synthetic anomalies can be combined with existing methods and obtain gains in performance.