An Algorithm for Out-Of-Distribution Attack to Neural Network Encoder 

Deep neural networks ( DNNs ) , especially convolutional neural networks , have achieved superior performance on image classification tasks . However , such performance is only guaranteed if the input to a trained model is similar to the training samples , i.e. , the input follows the probability distribution of the training set . Out-Of-Distribution ( OOD ) samples do not follow the distribution of training set , and therefore the predicted class labels on OOD samples become meaningless . Classification-based methods have been proposed for OOD detection ; however , in this study we show that this type of method has no theoretical guarantee and is practically breakable by our OOD Attack algorithm because of dimensionality reduction in the DNN models . We also show that Glow likelihood-based OOD detection is breakable as well . 1 INTRODUCTION . Deep neural networks ( DNNs ) , especially convolutional neural networks ( CNNs ) , have become the method of choice for image classification . Under the i.i.d . ( independent and identically distributed ) assumption , a high-performance DNN model can correctly-classify an input sample as long as the sample is “ generated ” from the distribution of training data . If an input sample is not from this distribution , which is called Out-Of-Distribution ( OOD ) , then the predicted class label from the model is meaningless . It would be great if the model has the ability to distinguish OOD samples from in-distribution samples . OOD detection is needed especially when applying DNN models in life-critical applications , e.g. , vision-based self-driving or image-based medical diagnosis . It was shown by Nguyen et al . ( 2015 ) ( Nguyen et al. , 2015 ) that DNN classifiers can be easily fooled by OOD data , and an evolutionarily algorithm was used to generate OOD samples such that DNN classifiers had high output confidence on these samples . Since then , many methods have been proposed for OOD detection using classifiers or encoders ( Hendrycks & Gimpel , 2017 ; Hendrycks et al. , 2019 ; Liang et al. , 2018 ; Lee et al. , 2018b ; a ; Alemi et al. , 2018 ; Hendrycks & Gimpel , 2017 ) . For instance , Hendrycks et al . ( Hendrycks & Gimpel , 2017 ) show that a classifier ’ s prediction probabilities of OOD examples tend to be more uniform , and therefore the maximum predicted class probability from the softmax layer was used for OOD detection . Regardless of the details of these methods , every method needs a classifier or an encoder , which takes an image x as input and compresses it into a vector z in the laten space ; after some further transform , z is converted to an OOD detection score τ . This computing process can be expressed as : z = f ( x ) and τ = d ( z ) . To perform OOD detection , a detection threshold needs to be specified , and then x is OOD if τ is smaller/larger than the threshold . For the evaluation of OOD detection methods , ( Hendrycks & Gimpel , 2017 ) , an OOD detector is usually trained on a dataset ( e.g . Fashion-MNIST as indistribution ) and then it is tested on another dataset ( e.g . MNIST as OOD ) . As will be shown in this study , the above mentioned classification-based OOD detection methods are practically breakable . As an example ( more details in Section 3 ) , we used the Resnet-18 model ( He et al. , 2016 ) pre-trained on the ImageNet dataset . Let xin denote a 224×224×3 image ( indistribution sample ) in ImageNet and xout denote an OOD sample which could be any kinds of images ( even random noises ) not belonging to any category in ImageNet . Let z denote the 512- dimensional feature vector in Resnet-18 , which is the input to the last fully-connected linear layer before the softmax operation . Thus , we have zin = f ( xin ) and zout = f ( xout ) . In Fig . 1 , xin is the image of Santa Claus , and xout could be a chest x-ray image or a random-noise image , and “ surprisingly ” , zout ∼= zin which renders OOD detection score to be useless : d ( zout ) ∼= d ( zin ) . In Section 2 , we will introduce an algorithm to generate OOD samples such that zout ∼= zin . In Section 3 , we will show the evaluation results on publicly available datasets , including ImageNet subset , GTSRB , OCT , and COVID-19 CT . Since some generative models ( e.g . Glow ( Kingma & Dhariwal , 2018 ) ) can approximate the distribution of training samples ( i.e . p ( xin ) ) , likelihood-based generative models were utilized for OOD detection ( Nalisnick et al. , 2019 ) . It has been shown that likelihoods derived from generative models may not distinguish between OOD and training samples ( Nalisnick et al. , 2019 ; Ren et al. , 2019 ; Choi et al. , 2018 ) , and a fix to the problem could be using likelihood ratio instead of raw likelihood score ( Serrà et al. , 2019 ) . Although not the main focus of this study , we will show that the OOD sample ’ s likelihood score from the Glow model ( Kingma & Dhariwal , 2018 ; Serrà et al. , 2019 ) can be arbitrarily manipulated by our algorithm ( Section 2.1 ) such that the output probability p ( xin ) ∼= p ( xout ) , which further diminishes the effectiveness of any Glow likelihood-based detection methods . 2 METHODOLOGY . 2.1 OOD ATTACK ON DNN ENCODER . We introduce an algorithm to perform OOD attack on a DNN encoder z = f ( x ) which takes an image x as input and transforms it into a feature vector z in a latent space . Preprocessing on x can be considered as the very first layer inside of the model f ( x ) . The algorithm needs a weak assumption that f ( x ) is sub-differentiable . A CNN classifier can be considered a composition of a feature encoder z = f ( x ) and a feature classifier p = g ( z ) where p is the softmax probability distribution over multiple classes . Let ’ s consider an in-distribution sample xin and an OOD sample x′out , and apply the model : zin = f ( xin ) and z′out = f ( x ′ out ) . Usually , z ′ out 6= zin . However , if we add a relatively small amount of noise δ to x′out , then it could be possible that f ( x ′ out + δ ) = zin and x ′ out + δ is still OOD . This idea is realized in Algorithm 1 , OOD Attack on DNN Encoder . The clip operation in Algorithm 1 is very important : it will limit the difference between xout and x′out so that xout may be OOD . The algorithm is inspired by the method called projected gradient descent ( PGD ) ( Kurakin et al. , 2016 ; Madry et al. , 2018 ) which is used for adversarial attacks . We note that the term “ adversarial attack ” usually refers to adding a small perturbation to a clean sample x in a dataset such that a classifier will incorrectly-classify the noisy sample while being Algorithm 1 OOD Attack on DNN Encoder Input : An in-distribution sample xin in a dataset . An OOD sample x′out not similar to any sample in the dataset . f , the neural network feature encoder . , the maximum perturbation measured by Lp norm . N , the total number of iterations . α the learning rate of the optimizer . Output : an OOD sample xout s.t . f ( xout ) ∼= f ( xin ) Process : 1 : Generate a random noise ξ with ||ξ|| ≤ 2 : Initialize xout = x′out + ξ 3 : Setup loss J ( xout ) = ||f ( xout ) − f ( xin ) ||2 ( L2 norm ) 4 : for n from 1 to N do 5 : xout ← clip ( xout − α · h ( J ′ ( xout ) ) ) , where J ′ ( x ) = ∂J/∂x . 6 : end for Note : The clip operation ensures that ||xout − x′out||p ≤ . The clip operation also ensures that pixel values stay within the feasible range ( e.g . 0 to 1 ) . If L-inf norm is used , h ( J ′ ) is the sign function ; and if L2 norm is used , h ( J ′ ) is a function that normalizes J ′ by its L2 norm . Adamax optimizer is used in the implementation able to correctly-classify the original clean sample x . Thus , OOD attack and adversarial attack are completely different things . In practice , the Algorithm 1 can be repeated many times to find the best solution . Random initialization is performed in line-1 and line-2 of the algorithm process . By adding initial random noise ξ to x′out , the algorithm will have a better chance to avoid local minima caused by a bad initialization . 2.2 DIMENSIONALITY REDUCTION AND OOD ATTACK . Recall that in a classification-based OOD Detection approach , a DNN encoder transforms the input to a feature vector , i.e. , z = f ( x ) , and an OOD detection score is computed by another transform on z , i.e. , and τ = d ( z ) . If zout ∼= zin , then d ( zout ) ∼= d ( zin ) which breaks the OOD detector regardless of the transform d. Usually , a DNN encoder makes dimensionality reduction : the dimension of z is significantly smaller than the dimension of x . In the example shown in Fig . 1 , z is a 512-dimensional feature vector ( dim ( z ) = 512 ) in Resnet-18 , and the dimension of x is 150528 ( 224× 224× 3 ) . Dimensionality reduction in an encoder provides the opportunity for the existence of the mapping of OOD and in-distribution samples to the same locations in the latent space . This is simply because the vectors in a lower-dimensional space can not represent all of the vectors/objects in a higher-dimensional space , which is the Pigeonhole Principle . Let ’ s do an analysis on the Resnet-18 example in Fig . 1 . A pixel of the color image x has 8-bits . In the 150528-dimension discrete input space , there are 256224×224×3 different images/vectors , which defines the size of the input space . float32 data type is usually used in computation , a float32 variable can roughly represent 232 unique real numbers . Thus , in the 512-dimensional latent space , there are 232×512 unique vectors/objects , which defines the size of the latent space . The ratio is ( 232×512 256224×224×3 ) 1 , and it shows that the latent space is significantly smaller than the input space . Thus , for some sample x in the dataset , we can find another sample x′ such that f ( x′ ) = f ( x ) as long as dim ( z ) < dim ( x ) . A question arises : will the x′ be in-distribution or OOD ? To answer this question , let ’ s partition the input discrete space Ω into two disjoint regions ( Ω = Ωin ∪ Ωout ) , Ωin of in-distribution samples and Ωout of OOD samples . |Ω| denotes the size of Ω . Usually , the training set is only a subset of Ωin , and the size of Ωout is significantly larger than the size of Ωin . For example , if Ωin is ImageNet , then Ωout contains medical images , noise images , and other weird images . If Ωin contains human face images , then Ωout contains non-face images and then |Ωin| |Ωout| . The latent space ( z-space ) is denoted by F and partitioned into two subspaces : F = Fin ∪Fout . An encoder is applied such that Ωin → Fin and Ωout → Fout . If there is overlap Fin∩Fout 6= ∅ , then the encoder is vulnerable to OOD attack . Usually , the encoder is a part of a classifier trained to classify in-distribution samples into different classes , and therefore the encoder can not guarantee that there is no overlap between Fin andFout . What is the size ofFin∩Fout or what is the probability P ( |Fin ∩ Fout| ≥ a ) ? While it is hard to calculate it for an arbitrary encoder and dataset , we can do a worst-case-scenario analy- sis . Assuming that every OOD sample is i.i.d . mapped to the latent space with a uniform distribution over a number of |F| spots , then the probability of OOD samples covering the entire latent space is P ( Fout = F ) = |F| ! ×Stirling ( |Ωout| , |F| ) / |F||Ωout| → 1 as |F| / |Ωout| → 0 , where Stirling is the Stirling number of the second kind . Noting that |F| / |Ωout| = 2 32×512 256224×224×3−1.4×107 ≈ 0 and 1.4 × 107 being the number of samples in ImageNet , then it could be true that almost ( with probability close to 1 ) the entire latent space of Resnet-18 is covered by the z vectors of OOD samples . Next , we discuss how to construct OOD samples to fool neural networks . First , let ’ s take a look at one-layer linear network : z = Wx , and make notations : an in-distribution input x ∈ RM , latent code z ∈ RK and K M . W is a K ×M matrix , and rank ( W ) ≤ K. The null space of W is Ωnull = { η ; Wη = 0 } . Now , let ’ s take out the basis vectors of this space , η1 , η2 , . . . , ηM−K , and compute x′ = ∑ i λiηi + x where λi is a non-zero scalar . Obviously , z ′ = Wx′ = z . We can set the magnitude of the “ noise ” ∑ i λiηi to be arbitrarily large such that x ′ will look like garbage and become OOD , which is another explanation of the existence of OOD samples . Then , we can try to apply this attack method to multi-layer neural network . If the neural network only uses ReLU activation , then the input-output relationship can be exactly expressed as a piecewise-linear mapping ( Ding et al. , 2020 ) , a similar approach can be applied layer by layer . If ReLU is not used , a new method is needed . We note that the filter bank of a convolution layer can be converted to a weight matrix . We have examined the state-of-the-art CNN models that are pre-trained on ImageNet and available in Pytorch , and dimensionality reduction is performed in most of the layers ( except 1 or 2 layers near the input ) , i.e . |F| ≤ |Ωin| |Ωout| . Instead of constructing an OOD sample by adding perturbations to an in-distribution sample , in Algorithm-1 , we construct an OOD sample paired with an in-distribution sample by starting from an initial sample that is OOD . Could an encoder be made robust to the OOD attack by including OOD samples in training set for supervised binary classification : in vs out ? Usually |Ωin| |Ωout| and we will have to collect and label “ enough ” samples in Ωout , which is infeasible considering the large size of Ωout ≈ Ω . As a comparison , to enhance DNN classifier robustness against adversarial noises , it is very effective to include noisy samples in the training set , i.e . Ωin = Ωin clean∪Ωin noisy . It is known as adversarial training ( Goodfellow et al. , 2018 ) and computationally feasible as |Ωin noisy| |Ωout| .

Summary: The paper presents a method that attacks existing out-of-distribution (OOD) detection methods. Most of the existing OOD detection methods perform detection using a latent representation. Main motivation of the paper is that the size of the latent representation is much smaller than the input images which results mapping both OOD and in-distribution images to the same place in the latent space and diminishing OOD detection performance. With this motivation, the proposed method perturbs input images to obtain an image whose latent representation is similar to the latent representation of an in-distribution image. Since such perturbations can be obtained for any OOD image, existing OOD detection algorithms fails distinguishing such OOD samples. The paper contains experiments on multiple dataset to demonstrate that the proposed method obtains a latent representation similar to the representation of an in-distribution image.