An Algorithm for Out-Of-Distribution Attack to Neural Network Encoder 
Authors: Liang Liang
Linhai Ma
Linchen Qian
Jiasong Chen
Venue:  (2021)
Keywords: Out-Of-Distribution
DNN
image classification

Abstract: Deep neural networks (DNNs), especially convolutional neural networks, have achieved superior performance on image classification tasks. However, such performance is only guaranteed if the input to a trained model is similar to the training samples, i.e., the input follows the probability distribution of the training set. Out-Of-Distribution (OOD) samples do not follow the distribution of training set, and therefore the predicted class labels on OOD samples become meaningless. Classiﬁcation-based methods have been proposed for OOD detection; however, in this study we show that this type of method has no theoretical guarantee and is practically breakable by our OOD Attack algorithm because of dimensionality reduction in the DNN models. We also show that Glow likelihood-based OOD detection is breakable as well. 

Under review as a conference paper at ICLR 2021
AN ALGORITHM FOR OUT-OF-DISTRIBUTION ATTACK TO NEURAL NETWORK ENCODER
Anonymous authors Paper under double-blind review
ABSTRACT
Deep neural networks (DNNs), especially convolutional neural networks, have achieved superior performance image classification tasks. However, such per- formance is only guaranteed if the input to a trained model is similar to the train- ing samples, i.e., the input follows the probability distribution of the training set. Out-Of-Distributiin (OOD) samples do not follow the distribution of training set, and therefore the predicted class labels on OOD samples become meaningless. classification-baseee methods have been proposed for OOD detection; however, in this study we show that this type of method has no theoretical guarantee and is practically breakable by our OOD Attack algorithm because of dimensionality reduction the DNN models. We also show that Glow likelihood- based OOD detection is breakable as well.
INTRODUCTION
Deep neural networks (DNNs), especially convolutional neural networks (CNNs), have become the method of choice for image classification. Under the i.i.d. (independent and identically distributed) assumption, a high-performance DNN model can correctly-classffy an input sample as long as the sample is "generated" from the distribution of training data. If an input sample is not from this distribution. which is called model is meaningless. It would be great if the model has the ability distinguish OOD samples from in-distribution samples. OOD detection is needed especially when applying DNN models in life-critical applications, e.g., vision-based self-driving or image-based medical diagnosis.
(OOD), then the predicted class label from the
It was shown by Nguyen et al. (2015) (Nguyen et al. 2015) that DNN classifiers can be easily fooled by OOD data, and an evolutionarily algorithm was used to generate OOD samples such that DNN classifiers had high output confidence on these samples. Since then, many methods have been proposed for OOD detection using classifiers or encoders (Hendryeks & Gimpel| 2017 Hendrycks
et al. 2019 Liang et al. 2018 Lee et al. 2018b Alemi et al. 2018 Hendrycks & Gimpel (011)) For instance, Hendrycks et al. (Hendrycks & Gimpel| (2017) show that classifier's prediction probabilities of OOD examples tend to be more uniform, and therefore the maximum predicted class probability from the softmax layer was used for OOD detection. Regardless of the details of these methods, every method needs a classifier or an encoder, which takes an image x as input and compresses it into a vector in the laten space; after some further transform, is converted to an OOD detection score т. This computing process can be expressed as: = f(x) and To perform OOD detection, a detection threshold needs to be specified, and then æ is OOD if T smaller/larger than the threshold. For the evaluation of OOD detection methods, (Hendrycks) & Gimpell 2017). an OOD detector is usually trained on dataset (e.g. Fashion-MNIST as in- distribution) and then it tested on another dataset (e.g. MNIST as OOD).
d(z).
As will be shown in this study, the above mentioned classification-based OOD detection methods are practically breakable. As an example (more details in Section 3), we used the Resnet-18 model (He et al. 2016) pre-trained on the ImageNet dataset. Let zin denote 224x224x3 image (in- distribution sample) in ImageNet and Tout denote an OOD sample which could be any kinds of images (even random noises) not belonging any category in ImageNet. Let 2 denote the 512- dimensional feature vector in Resnet-18, 18, which is the input to the last fully-connected linear layer before the softmax operation. Thus, have Zin = f(rmn) and Zout [x In Fig. Zin

---

Under review as a conference paper at ICLR 2021
(a)
(b)
(c)
Figure I: The 1st column shows the image of Santa Claus Vin and the scatter plot of ain using blue dots. The 2nd column shows chest x-ray image Tout and the scatter plot of Fout (red circles) and Zin (blue). The 3rd column shows a random image Zout, and the scatter plot of zout (red) and 2in (bluc).
is the image of Santa Claus, and Tout could be chest x-ray image or random-noise image, and "surprisingly", Zout 2 Zin which renders OOD detection score be useless: d(zoat) M d(zan).
In Section 2, we will introduce an algorithm to generate OOD samples such that Zout 2 In Section 3, we will show the evaluation results on publicly available datasets, including ImageNet subset. GTSRB, OCT, and COVID-19 CT. Since some generative models (e.g. Glow (Kingma & Dhariwal 2018)) can approximate the distribution of training samples (i.e. p( likelihood-based generative models were utilized for OOD detection (Nalisnick et al. 2019). It has been shown that likelihoods derived from generative models may not distinguish between OOD and training samples Nalisnick et al.) 2019 Ren et al. 2019 Choi al. (2018), and fix to the problem could be using likelihood ratio instead of raw likelihood score (Serrà et al. 2019). Although not the main focus of this study, we will show that the OOD sample's likelihood score from the Glow model (Kingmal & Dhariwal] (2018; (Será et al.] 2019) can be arbitrarily manipulated by our algorithm (Section 2.1) such that the output probability p(rin) = p(Fout), which further diminishes the effectiveness. of any Glow likelihood- based detection methods.
METHODOLOGY
2.1 OOD ATTACK ON DNN ENCODER
We introduce an algorithm to perform OOD attack on DNN encoder = f(x) which takes an image as input and transforms into feature vector in a latent space. Preprocessing on can be considered as the very first layer inside of the model f(z). The algorithm needs a weak assumption that f(x) sub-differentiable. A CNN classifier can be considered a composition of a feature encoder z f(x) and a feature classifier p g(2) where is the softmax probability distribution over multiple classes.
Let's consider an in-distribution sample and an OOD sample d'our> and apply the model: z [a and ='ut = f(='bu)). Usually, 2/out * Zin- However, we add a relatively small amount of noise 8 to I'ur. then it could be possible that f (Iout + 6) 2in and I'.ut + is still OOD. This idea is realized in Algorithm 1, OOD Attack on DNN Encoder.
The clip operation in Algorithm 1 is very important: it will limit the difference between Tout and T'ut so that Fout may be OOD. The algorithm is inspired by the method called projected gradient descent (PGD) (Krrkkin et al. 2016 Madry al.) 2018) which is used for adversarial attacks. We note that the term "adversarial attack" usually refers to adding a small perturbation to a clean sample in dataset such that classifier will incorectly-classify the noisy sample while being
2

---

Under review as a conference paper at ICLR 2021
Algorithm 1 OOD Attack on DNN Encoder
Input: An in-distribution sample Iin in a dataset. An OOD sample door not similar to any sample in the dataset. f. the neural network feature encoder. the maximum perturbation measured by Lp norm. N. the total number of iterations. a the learning rate of the optimizer.
Output: OOD sample Tout s.t. (Tout) = f
Process:
1: Generate random noise with [IEl S
2: Initialize Tout = Lout +
3: Setup loss (Fout) (If())) f( 112 (1.2 norm)
4: for from 1 to N do
5: Tout < cippp aaa-aaaaaaaaaaaaaaaaa (Foutt)) where J'(x) - (()))=JJ/00 0J/Oz.
6: end for
Note: The clip operation ensures that ||Fout d'oeelle I c. The clip operation also ensures
that pixel values stay within the feasible range (e.g. to 1). If L-inf norm is used, h (J") is the sign function; and if L2 norm is used, h (J") is a function that normalizes by its L.2 norm.
Adamax optimizer is used in the implementation
able to correctly- claysssfy the original clean sample x. Thus, OOD attack and adversarial attack are completely different things.
In practice, the Algorithm 1 can be repeated many times to find the best solution. Random initial- ization is performed line-1 and line-2 of the algorithm process. By adding initial random noise to Toua the algorithm will have better chance to avoid local minima caused by bad initialization.
2.2 DIMENSIONALITY REDUCTION AND OOD ATTACK
Recall that in a classification- OOD Detection approach, a DNN encoder transforms the input to feature vector, i.e., = f(x), and an OOD detection score is computed by another transform on 2, i.e., and r d(=). If Zout M Zins then d(zout) d(2.n) which breaks the OOD detector regardless of the transform d. Usually, DNN encoder makes dimensionality reduction: the dimen- sion of is significantly smaller than the dimension of x. In the example shown in Fig. 1, z is a 5 512-dimensional feature vector (dim(z) 512) in Resnet-18, and the dimension of is 150528 (224 x 224x x 3).
Dimensionality reduction in an encoder provides the opportunity for the existence of the map- ping of OOD and in-distribution samples to the same locations in the latent space. This is sim- ply because the vectors in lower-dimensional space cannot represent all of the vectors/objects in higher- digher-dimensional space, which is the Pigeonhole Principle. Let's do an analysis on the Resnet- 18 example in Fig. 1. A pixel of the color image has 8-bits. In the 150528-dimension discrete input space, there are 256221x224x3 different images/vectors, which defines the size of the input space. float32 data type usually used in computation, float32 variable can roughly represent 232 unique real numbers. Thus, in the 112-dimensional latent space, there are 232x512 unique vectors/objects,
232x012 256-244224X3
which defines the size of the latent space. The ratio is
<< 1, and it shows that the
latent space is significantly smaller than the input space. Thus, for some sample in the dataset, we can find another sample x' such that f (x' ) = (x) as long as dim(z) < dim((x. A question arises: will the x' be in-distribution or OOD? To answer this question, let's partition the input dis- crete space Ω into two disjoint regions (N = Ω4, U Uouu). Qin of in-distribution samples and Rout of OOD samples. [S2] denotes the size of Q. Usually, the training set is only subset of Qan, and the size of Pout is significantly larger than the size of Ω.n. For example, if Qin is ImageNet, then Dout contains medical images, noise images, and other weird images. If 2in contains human face images, then Sout contains non-face images and then [aaml & 1Rout The latent space (z-space) is denoted by F and partitioned into two subspaces: F == Fu U An encoder is applied such that Din + F... and Pout + Fout. If there is overlap NF..... out + 0, then the encoder is vulnerable to OOD attack. Usually, the encoder a part of a classifier trained to classify in-distribution samples into different classes, and therefore the encoder cannot guarantee that there is no overlap between F.,. and Fout. What is the size Fin, n..ut or what is the probability P ([F... Fouel 2 a)? While it is hard to calculate it for arbitrary encoder and dataset, we can do worst-case-scenatt analy-
3

---

Under review as a conference paper at ICLR 2021
sis. Assuming that every OOD sample is i..dd mapped to the latent space with uniform distribution over a number of [F] spots, then the probability of OOD samples covering the entire latent space is
P (Fout = F) |F|! x Strrling([Soua) (FF)// | [Rowsl - 1 as
[FI [S - O. where Stürling
the Stirling number of the second kind. Noting that |
256 1.4 Rs 0 and
232x512
1.4 x 10 being the number of samples in ImageNet, then it could be true that almost (with proba- bility close to 1) the entire latent space of Resnet-18 covered by the 2 vectors of OOD samples.
Next, we discuss how to construct OOD samples to fool neural networks. First, let's take a look at one-layer linear network: = Wx, and make notations: an in-distribution input x E RMM latent code E R and K < M. W is K x M matrix, and rank (W) I K. The null space of W is Smalt = {ıg Wn= = 0}. Now, let's take out the basis vectors of this space, 7/1 T2. and compute x' = E; Am + x where X is a non-zero scalar. Obviously, 2' = WW = We can set the magnitude of the "noise" " Σ, A to be arbitrarily large such that x' will look like garbage and become OOD, which is another explanation of the existence of OOD samples. Then, we can try to apply this attack method to multi-layer neural network. If the neural network only uses ReI.U activation, then the input-output relationship can be exactly expressed a piecewise-linen mapping (Ding et al.) 2020), similar approach can be applied layer by layer. If ReLU is not used, a new method is needed. We note that the filter bank of a convolution layer can be converted to a weight matrix. We have examined the state-of-the-an CNN models that are pre-trained on ImageNet and available in Pytorch, and dimensionality reduction is performed in most of the layers (except layers near the input), i.e. |F] < |Sin| << [Raux] | Instead of constructing an OOD sample by adding perturbations to an in-distribution sample, in Algorithm-1, we construct an OOD sample paired with an in-distribution sample by starting from an initial sample that is OOD.
7M-K- KK
Could an encoder be made robust to the OOD attack by including OOD samples in training set for supervised binary classification: in vs out? Usually |92m. << [oouel and we will have to collect and label "enough" samples in Dout. which is infeasible considering the large size of Pout & S2 As a comparison, enhance DNN classifier robustness against adversarial noises, it is very effective to include noisy samples in the training set, i.e. Pin = Qin clean Ul4n noisy. is known as adversarial training (Goodfellow al. 2018) and computationally feasible as [0en moisyl < [Pouel-
2.3 PROBLEM OF GL.W LIKEL OOD DETECTION
Generative models have been developed to approximate the training data distribution. Glow (Kingma & Dhariwal 2018) is one of these models, and it has very special property: it bijective and the latent space dimension is the same as the input space dimension, i.e., no dimensionality reduction, which is the reason that we studied this model.
Several studies have found the problem of Glow-based OOD detection: Glow may not distinguish between OOD and training samples (Ren et al., 2019 Choi et al., 2018). and a possible fix to the issue could be using likelihood ratio (Serrà et al.) 2019). In this study, we further show that negative tog-likelihood (NLL) from the Glow model can be arbitrarily manipulated by our algorithm in which (z) denotes NLL. The results on CelebA face image dataset are in Section 3. We think the major reason causing Glow's vulnerability to OOD attack is that we do not have enough training data in high dimensional space. Glow is a mapping: Vin - Zin - p(2tn) > p(aii). ). the probability of Tin. For an OOD sample Tout, the mapping is Tout + Zont - p(zoue) + p. Since the number of training samples is significantly smaller than the size of the space, there are a huge number of "holes" in the latent space (i.e., regions that no training samples are mapped to), and it easy to put zout in one of these "holes" close to Zin such that p(Zout) S p(zm))
likelihoods derived from
2.4 RECONSTRUCTION-BANE OOD DETECTION
Auto-encoder style OOD detection has been developed for anomaly detection (Chalapathy & Chawla 2019 Cohen et al. 2019) based on reconstruction error. The data flow of an auto-encoder is + & where ã is the reconstruction of x. The OOD detection score can be the difference between x and &, e.g., the Lp distance ||æ - Ellp or Mahalanobis Distance. This type of method has two known issues. The first issue is that auto-encoder may well reconstruct OOD samples. i.e., Dont 83 Lout- Thus, one needs to make sure it has large reconstruction errors on OOD samples, which can be done by limiting the capacity of auto-encoder or saturating it with in-distribution samples.
4

---

Under review as a conference paper at ICLR 2021
The second issue is that pixel-to-pixe distance is not good measurement of image dissimilarity, especially for medical images. For example, I could be a CT image of a heart and could be the image of the same heart that deforms little bit, but the pixel-to-pixe pixel distance between r and @ can be very large. Thus, a robust image similarity measurement is needed.
Interestingly, the proposed OOD attack algorithm has effect on this type of method. Let's con- sider the data flow: Zin + Zin + Ein and Tout - Sout - &out- If Zout Zin- then eowt 6in Then, it is easy find out that Tout is OOD because || -ootlll = || iin lp which is very large. Ironically, in this case, the attack algorithm helps to identify the OOD sample. In future work, we will evaluate the effectiveness of combining the proposed algorithm and auto-encoder for OOD detection.
EXPERIMENTS
We applied the proposed algorithm to attack state-of-the-art DNN models on image datasets. For each in-distribution sample Tin, an OOD sample Zoua is generated by the algorithm. To measure attack strength, mean absolute percentage error is calculated by MAPE((zot)) = mean([zout zin (a.l)/aaz(( (mn)). Here, 2out = (Tout) and 2in f ( / Zin| is an error vector, and mean(]zout 2 is the average error. max([2mml) is the maximum absolute value in the vector Zin- We also applied the algorithm to attack the Glow model on CelebA dataset. In all of the evaluations, L.2 norm was used in the proposed algorithm. Pytorch was used to implement the algorithm. Nvidia Titan V GPUs were used for model training and testing.
3.1 EVALUATION ON A SUBSET OF IMAGENET
ILSVRC2012 ImageNet has over 1 million images 1000 classes. Given the limited computing power, it is impractical to test the algorithm on the whole dataset. Instead, we used a subset of 1000 images in 200 categories. The size of each image 224x224x3. Two CNN models pretrained on the ImageNet were evaluated, which are Resnet-18 and 121 available in Pytorch.
Resnet-18 latent space has 512 dimensions. Since ImageNet covers a variety of natural and artificial objects, we choose medical images and random-noise images to make sure that d'ous is indeed OOD. Using each of the three initial OOD samples (chest x-ray, lung-CT and random noise to be I'ma) > generated 1000 OOD samples paired with the 1000 (in-distribution) samples in the dataset and calculated MAPE values The three MAPE histograms are shown in Fig. 3. Most of the MAPE values are less than 0.1%.
We also evaluated another CNN, named Densenet- 121, and obtained similar results. The latent space has 1024 dimensions. Again, using each of the three initial OOD samples, 1000 OOD samples are generated for the samples in the dataset, and then MAPE values are calculated. The three MAPE histograms are shown in Fig. Most of the MAPE values are less than 0.1%, indicating strong OOD attack.
From the results in Fig.2 to Fig. it can be seen that each of the two CNN models mapped sig- nificantly different OOD samples and in-distribution samples to almost the same locations in the latent space. Dimensionality reduction leads to the existence of such mapping, and our algorithm can find such OOD samples out. In other words, the mapping from input space to the latent space is many-to-one, not bijective. And therefore, it is almost guaranteed that such OOD samples exist and they can break any OOD detector d that computes detection score d(z) only from the latent space (z-space). We tested a classical OOD detection method using the maximum of softmax output as detection score (Hendrycks & Gimpel 2017). The results shown in Table- 1, and the AUROC scores are close to 0.5, showing that the method is unable to tell the difference between the 1000 OOD samples and 1000 in-distribution samples.
5

---

Under review as a conference paper at ICLR 2021
(b)
(c)
(d)
The 1st column shows three in-distribution samples (e.g. Santa Claus) Tins and the corresponding scatter plots of Zin (blue dots). The 2nd column shows OOD samples Tout generated from CT image 2'uts and the corresponding scatter plots of Zout (red) and Zin. (blue). The 3rd column shows OOD samples Fout generated from a random image z'mee> and the corresponding scatter plots of znut (red) and zin (blue). The 4th column shows OOD samples Font generated from a x-ray image d'uer and the corresponding scatter plots Zont (red) and Zin (blue). MAPE values are embedded in these scatter-plots. Please zoom-in for better visualization.
Figure 2:
Figure 3: Left: the MAPE histogram using chest x-ray as the initial OOD sample. Middle: the MAPE histogram using lung CT image the initial OOD sample. Right: the MAPE histogram using a random-noise image as the initial OOD sample. The results are from Resnet-18.

---

Under review as a conference paper at ICLR 2021
Figure 4: Left: MAPE histogram using a chest x-ray as the imitial OOD sample. Middle: MAPE histogram using lung CT image as the initial OOD sample. Right: MAPE histogram using random-nowe noise image as the initial OOD sample. The results are from Densenet-12 121.
Table I: AUROC of two networks under OOD attack with each of x-ray, CT and noise as the initial OOD sample
X-ray
CT
random-
Resnet-18
0.643
0.633
0,500
Densenet- 121
0.638
0.651
0.500
3.2 EVALUATION ON CELEBA DATASET
We tested the algorithm and the Glow model (Kingma Dhariwal, 2018) on the CelebA dataset (human face images). The size of each image is 64x64x3. After traiiing the model was able to generate realistic face images. The model also outputs the negative log-likelihood (NLL) of the input sample, i.e., NNL (x) = -log (p (x)). By setting (x) make ff (Xout) to be close to 0 or very large to match any f (Iin). which renders NLL score useless for OOD detection. To demonstrate the effectiveness of our algorithm, we randomly selected 160 iin-iitribution) samples in the dataset. We used a color spiral image as the initial OOD sample 3.5268. The distributions of NLL(f from 160 in-distribution samples and N oLL( from 160 corresponding OOD samples, well OOD sample images are shown Fig. 5. The two distributions are almost identical. More examples of OOD samples are shown in Fig. 6. In each row of Fig. 6, although the images have different NL. scores, they look like each other.
NNL(2), our algorithm
r oout and NNL (x'cne) N
Histegra Samples
sungge Court
Histogram Samples
Sarnighl COUnt
0.7
NLL=0.00
M.L=0.20
NLL=0.00
NLL=0.60
NEL:=0.50 0.00
NLL=1.00
Figure 5: Top: NLL histogram (blue bars) of Figure 6: Top: OOD samples generated by us- the in-distribution; samples Middle: NLL his- ing one 2x2 checkerbox image for initializa- togram (red bars) of OOD samples; Bottom: tion; Bottom: OOD samples generated by using some OOD samples with NLL from to 1. The one 8x8 checkerbox image for initialization. initial OOD sample is spiral image.

---

Under review as a conference paper at ICLR 2021
We have done more evaluations of our algorithm and OOD detection methods, please read the appendices.
4 DISCUSSION
We hypothesized that dimensionality reduction in an encoder provides the opportunity for the cxis- tence of the mapping of OOD and in-distribution samples the same locations in the latent space. We applied the OOD Attack algorithm to DNN classifiers on various datasets (see Appendices A and B), and the results (i.e. low MAPE values) confirmed our hypothesis. The results imply that classifier/encoder -based OOD detection methods may be vulnerable to the OOD attack.
By using our OOD Attack algorithm, we evaluated nine OOD detection methods (see Appendices C to J). The AUROC scores of these methods are close to 0.5 in our experiments, which means these methods could not distinguish between the in-distribution samples (e.g. CIFAR10) and the OOD samples generated by our algorithms. Our algorithm was unable to break recent method named Certified Certain Uncertainty (Meinke & Hein 2020), because this method utilizes Gaussian mixture models (GMMs) in the input space (note: no dimensionality reduction in GMMs). However, it is well known that GMMs have convergence issues for high dimensional data (e.g. medical images).
Compared to adversarial attacks and defenses, it is much more difficult to defend against OOD at- tacks. Adversarial attacks and OOD attacks are doing completely different things to neural networks, although the attack algorithms may use similar optimization techniques. For image classification ap- plications, an adversarial attack will add a small amount of noise to the input (clean) image, and the resulting noisy image is still recognzabe. Therefore, the magnitudes of adversarial noises are constrained. For example, noisy image of a panda is still an image of the panda. By the judg- ment of humans, the noisy image and the clean image are the images of the same object, and the two images should be classified into the same class. Compared to adversarial samples, OOD samples, which can be generated by our OOD Attack algorithm, have much more freedom (e.g. they can be random noises), as long as they do not look like in-distribution samples. Thus, OOD detection is very challenging.
We would like to point out that it is difficult to evaluate an OOD detector to "prove" that it can detect, say 90% of the OOD samples by experimentally testing it on Pont because Pout is too large to be tested on: [S...l << |Mouel R |S2]. For example, if MaSSi is used as in-distribution, then MINST and Omniglot are usually as OOD, which the "standard" approach in the literature. Clearly, MINST and Omniglot cannot cover Pout the space of OOD samples. If the image size is larger, then [oovel becomes much larger. Could we design an evaluation method (experimental or analytical) that does not rely OOD samples?
Before the OOD detection issue is fully resolved, for life-critical applications, any machine learning system that uses DNN classifiers should not make decisions independently and can only serve as- sistants to humans. The OOD Attack algorithm and the experimental results can serve as a reference for the evaluation of new OOD detection methods.
We will release the code on GitHub when the paper is accepted. All figures are in high-resolution, please zoom in.
REFERENCES
Alexander A Alemi, Ian Fischer, and Joshua V Dillon. Uncertainty in the variational information bottleneck. UAI 2018 Uncertainty in Deep Learning Workshop, 2018.
Alvaro Arcos-Gareia, Juan Alvarez-Gareia, and Luis M Soria-Morillo. Deep neural network for traffic sign recognition systems: An analysis of spatial transformers and stochastic optimisation methods. Neural Networks, 99:158- 99:158-165. 2018.
Raghavendra Chalapathy and Sanjay Chawla. Deep learning for anomaly detection: A survey. arXiv preprint arXiv:1901.034077 2019.
Hyunsun Choi, Eric Jang, and Alexander A Alemi. Waic, but why? generative ensembles for robust anomaly detection. arXiv preprint arXiv: 1810.0139, 2018.
8

---

Under review as a conference paper at ICLR 2021
Joseph Paul Cohen, Paul Bertin, and Vincent Frappier. Chester: A web delivered locally computed chest x-ray disease prediction system. arXiv preprint arXiv:1901.112I1 2019.
Gavin Weiguang Ding. Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Max-margin ad- versarial (mma) training: Direct input space margin maximization through adversarial training. International Conference Learning Representations, 2020.
Yilun Du and Igor Mordatch. Implicit generation and generalization in energy-based models. Ad- vances in Neural Information Processing Systems, 2019.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. Robust physical-worke attacks on deep learning visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1625-1634, 2018.
Ian Goodfellow, Patrick McDaniel, and Nicolas Papernot. Making machine learning robust against adversarial inputs. Communications of the ACM, 61(7):56-66 66, 2018.
Will Grathwohl, Kuan-Chieh Wang, Jörn-Henrte Jacobsen, David Duvenaud, Mohammad Norouzi, and Kevin Swersky. Your classifier is secretly an energy based model and you should treat it like one. International Conference on Learning Reprexentations, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. A baseline for detecting misclassified and out-of- distrribbittn examples in neural networks. International Conference Learning Representations, 2017.
Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. International Conference on Learning Representations, 2019.
Daniel S Kermany, Michael Goldbaum, Wenjia Cai, Carolina CS Valentim, Huiying Liang, Sally Baxter, Alex McKeown, Ge Yang, Xiaokang Wu, Fangbing Yan, et al. Identifying medical diag- noses and treatable diseases by image-based deep learning. Cell, 172(5): (223):11222111 1131, 2018.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. In Advances in neural information processing systems, pp. 10215-10224, 2018.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world.
arXiv preprint drXiv:1607.02533 2016.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predic-
tive uncertainty estimation using deep ensembles. In Advances in neural information processing systems, pp. 6402-6413 2017.
Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibrated classifiers for detecting duitt--distiiuution samples. International Conference on Learning Representations, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In Advances in Neural Information Process- ing Systems, pp- 7167-7177, 2018b.
Shiyu Liang, Yixuan Li, and Rayadurgam Srikant. Enhancing the reliability of out-of--isttiiutiinn image detection in neural networks. International Conference on Learning Representations, 2018 88.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't t know. International Conference Learning Representations, 2020.

---

Under review as a conference paper at ICLR 2021
Eric Nalisnick, Akihiro Matsukawa, Yee Whye Teh, Dilan Gorur, and Balaji Lakshminarayanan. Do deep generative models know what they don't 't know? International Conference on Learning Representations, 2019.
Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confi- dence predictions for unrecognizable images. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 427-436, 2015.
Jie Ren, Peter J Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark Depristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for out- out-of-distribution detection. In Advances in Neural Information Processing Systems, pp. 14707-14718, 2019.
Chandramouli Shama Sastry and Sageev Oore. Detecting out-of-distribution examples with in- distribution examples and gram matrices. NeurlPS 2019 Workshop Safety and Robustness in Decision Making, 2019.
Joan Serrà, David Álvarez, Vicenç Gómez, Olga Slizovskaia, José F Núñez, and Jordi Luque. Input complexity and out-of-distribttion detection with likelihood-based generative models. Interna- tional Conference on Learning Representations 2020, 2019.
Feng Shi, Jun Wang, Jun Shi, Ziyan Wu, Qian Wang, Zhenyu Tang, Kelei He, Yinghuan Shi, and Dinggang Shen. Review of artificial intelligence techniques in imaging data acquisition, segmen- tation and diagnosis for covid- 19. IEEE reviews in biomedical engineering, 2020.
Eduardo Soares, Plamen Angelov, Sarah Biaso, Michele Higa Froes, and Daniel Kanda Abe. Sars- cov-2 ct-scan dataset: A large dataset of real patients ct scans for sars-cov-2 identification. medRxiv, 2020.
Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19, 2018.
A APPENDIX
The parameters in each evaluation are listed.
1. Parameters for evaluation on ImageNet subset Attacking resnet-18: e = 5, N = 1e4, a e/100 with x-ray and CT:
20, N 1e4, a = E/100 with random noise. Attacking densnet-1211 e = 5, N le4, / 100 with x-ray and CT; 30, N = le4, s/100 with random noise.
We inspected the OOD images from random noise: they are not recognizable to human vision.
2. Parameters for evaluation OCT dataset = 10, N = 1e4, = E/ /00 with retinal fundus photography image E = 20, N = 1e4, a = e/100 with random noise.
3. Parameters for evaluation on COVID-19 CT dataset e 20, N = 1e4, e/100
Parameters for evaluation on GTSRB dataset ε = 10, = le4, a = E/100
5. Parameters for Evaluation on CelebA Dataset & = 10, N = 1e4, a e/ 1000
B APPENDIX
B. EVALUATION ON OCT DATASET
We tested our algorithm and Resnet-18 on a retinal optical coherence tomography (OCT) dataset Kermany et al. 2018), which has four classes. Each image is resized to 224x224. 1000 samples per class were randomly selected to obtain a training set of 4000 samples. The test set has 968 images. We modified Resnet-18 for this four-class classification task. The latent space has 512 dimensions. After training, the Resnet- 18 model achieved a classification accuracy > 95% on the test set.
We used two references images as the initial OOD sample 2/stt- The first reference image is grayscale retinal image converted from an RGB color retinal fundus photography image. Compared to this retinal fundus photography image, the OCT images have unique patterns of horizontal "white
10

---

Under review as a conference paper at ICLR 2021
(a)
(b)
Figure 7: The 1st column shows in-distribution samples Iún in the OCT dataset, and the scatter plots of Zin (blue dots). The 2nd column shows OOD samples Tout generated from a retinal fundus photography image z'utt and the scatter plots of 2out (red) and Zin (blue). The 3rd column shows OOD samples Zout generated from a random image a'.a. and the scatter plots of 2out (red) and zin (blue). MAPE values are embedded in these scatter-plots. Please zoom-in for better visualization.
bands". We selected this OOD image by purpose: there may be chance that both types of images are needed for retinal diagnosis. The second reference image is generated from random noises. Examples are shown in Fig. 7, and the two MAPE histograms confirm that the algorithm can generate OOD samples (968) which are mapped by the DNN model the locations of the in-distribution samples (968) in the latent space, i.e., Zout & Zin-
shown in Fig. 8. The results

---

Under review as a conference paper at ICLR 2021
21
MAF
HARE
Figure 8: Left: MAPE histogram using retinal fundus photography image as the initial OOD sample. Right: MAPE histogram using a random-noise image as the initial OOD sample. Please zoom-in for better visualization.
B.2 EVALUATION ON COVID-19 CT DATASET
We also tested our algorithm and Resnet-18 on a public COVID-19 lung CT (2D) image dataset (Soares al. 2020). It contains 1252 CT scans (2D images) that positive for COVID-11 in- fection and 1230 CT scans (2D images) for patients non-infected by COVID-19, 2482 CT scans in total. From infected cases, we randomly selected 200 samples for testing, 30 for validation, and 1022 for training. From the uninfected cases, we randomly selected 200 for testing, 30 for validation and 1000 for training. Each image resized 224x224.
We modified the last layer of Resnet- 18 for this binary classification task, infected vs uninfected. We also replaced batch normalization with instance normalization because known that batch normalization is not stable for small batch-size (Wu & He| 2018). The latent space still has 512 dimensions. We set batch-size to 32, the number of training epochs to 100, and used Adam optimizer with the default parameters. After training, the model achieved classification accuracy 95% on test set.
We used two reference images as the initial OOD sample x'uus chest x-ray image, and a random- noise image. The two MAPE histograms are shown in Fig. 9 that most of the MAPE values are less than 0.1%. The results also confirm that the algorithm can generate OOD samples (400) which are mapped by the DNN model to the locations of the in-distribution samples (400) the latent space, i.e,, Zout Zin-
Examples are shown in Fig. 10. As reported in the previous studies (hi et al. 2020). infected regions in the images have unique pattern called ground-glass opacity. The CT images in the 1st and 3rd rows show COVID-19 infections with ground-glass opacity on the upper-left area. The CT image in the 5th row does not show any signs of infection. It be seen that the random-noise images and the COVID-19 CT images have the same feature vectors in the latent space, which is astonishing.
12
200
Figure 9: Left: MAPE histogram using chest x-ray image the initial OOD sample. Right: MAPE histogram using a random-noii noise image as the initial OOD sample. Please zoom-in for better visualization.
12

---

Under review as a conference paper at ICLR 2021
(a)
Figure 10: The 1st column shows in-distribution samples in the COVID-19 19 dataset, and the scatter plots of Zin (blue dots). The 2nd column shows OOD samples Tour generated from a chest x-ray image 2'mts and the scatter plots of Zout (red) and žin (blue). The 3rd column shows OOD samples Fout generated from a random image z'muss and the scatter plots of Zout (red) and 2in (blue). MAPE values are embedded in these scatter-plots. Please zoom-in for better visualization.
B.3 EVALUATION ON GTSRB TRAFFIC SIGN DATASET
We tested our algorithm and state-of-the-arr traffic sign classifier on the GTSRB dataset. The classifier similar to the one in (Arcos-Garcaa et al. 2018). which has -pattallt network. The size of each image is 32x32x3. The latent space has 128 dimensions. After training, the classifier achieved over 99% accuracy on the test set. We used a random-nois noise image as the initial OOD sample x to generate 12630 OOD samples paired with the 12630 in in-distribution samples in the test set. The MAPE histogram shown in Fig. 11, in which most of the MAPE values are less than 0.1 1%. Examples shown in Fig. 12.
13

---

Under review as a conference paper at ICLR 2021
It can be seen that Sout of random-noise noise images almost the same 2in of the stop sign, the speed limit sign, and the turning signs. Not only the classifier cannot tell the difference between a real traffic sign and generated noise image, but also any detectors based on 2in for OOD detection will fail. We note that adversarial robustness of traffic sign classifiers has been studied (Eykholi al. 2018), and after adding adversarial noises to the traffic sign images, the noisy images are still recognizable. OOD noises and adversarial noises are very different (discussed in Sections 2.1 and 2.2). Thus, it would be wise to disable any vision-based auto-pilot in your self-driving cars today until this issue is resolved.
200
240
6.0
Figure 11: Left: MAPE histogram. Right: zoom-in view of the histogram
STOP
50
(a)
(b)
(c)
(d)
Figure 12: The 1st row shows four traffic sign images. The 3rd row shows the generated OOD im- ages. The 2nd row shows the scatter-plots of Sout (red) and Zin (blue). MAPE values are embedded in these scatter-plots. Please zoom-in for better visualization.
C APPENDIX
We applied our OOD Attack algorithm to test the OOD detection method named ODIN (Liang et al.) 2018
C.I SUMMARY OF THE ODIN METHOD
The method does temperature scaling the logits (input softmax) by logits/T, which is the Eq. in the ODIN paper. The temperature T could be in the range of to 1000. The ODIN method also does input preprocessing, which is the (2) in the ODIN paper. For preprocessing. the perturbation
14

---

Under review as a conference paper at ICLR 2021
magnitude (PM) could be in the range of 0 0.004. The OOD score is defined to be the maximum of the softmax outputs from a neural network, given the preprocessed input. An OOD sample is expected to have low OOD score.
C.2 EVALUATION ON CIFAR10
Wide residual network with depth 28 and widen factor 10 is used in the ODIN paper. After training for 200 epochs, the model achieved the classification accuracy of 94.71 on CIFARIO set.
In our algorithm, we set f (x:) to be the logits output from the model, given the preprocessed input. For the CIFARIO 10 dataset, the logits output contains 10 elements, which significant dimensionality reduction compared to the size of an input color image: 32 x 32 3. In our algorithm, the parameters are E 10, a = s/100, N = 100. The initial OOD sample a random noise image. For every sample in the CIFARIO test set, the algorithm generated an OOD sample to match the logits output. The generated OOD samples look like random noises. The OOD scores of these samples were calculated by the ODIN method.
The results are reported in Table Fig. 13 shows the OOD score histograms of the in-distribution and OOD samples when T=1000 and PM=000011. When T and PM 0, ODIN becomes the Baseline method (Hendrycks & Gimpel 2017).
Table 2: AUROC scores of ODIN
CIFAR10 10 OOD
T=1
T=10
T=100
T=1000
PM=0
0.500
0.500
0.500
0.500
PM=0.001
0.500
0.500
0.500
0.500
PM=0.002
0.500
0.500
0.500
0,500
PM=0.004
0.500
0.500
0.500
0.500
OOD score histogram of the in-distribution samples
400
100
histogram genereted samples
400
200
0.1006
0.1008
0.2010
0.1012
0.3014
0.1016
0.1618
0.1020
Figure 13: the OOD score histograms of the in-distribution (blue) and OOD (red) samples when T=1000 and PM=0.001
D APPENDIX
We applied our OOD Attack algorithm to test the OOD detection method named Mahalanobis Leee al. 2018b).
15

---

Under review as a conference paper at ICLR 2021
D.1 SUMMARY OF THE MAHALANOBIS METHOD
The method extracts the feature maps from multiple layers of neural network and applies average pooling per channel to reduce each feature map into 1D feature vector. Then, Mahalanobis distance calculated between feature vector and the corresponding mean vector. The distance values from all of the feature vectors are linearly combined together to produce a single distance, i.e., the OOD score. The OOD performance, the method does input preprocessing with given perturbation magnitude (PM), and then OOD score of the preprocessed input is obtained. The weights combine the Mahalanobis distances from multiple layers could be determined on validation set of OOD samples. In practice, it is impossible to obtain such validation set. In our evaluation, we simply take the average of the distance values, which gives the OOD score.
of OOD sample is expected to be large. To further improve the
Although the feature maps from multiple layers are utilized, the method still does dimensionality reduction to those feature maps (e.g. averaging). Therefore, the method is breakable by our OOD Attack algorithm.
D.2 EVALUATION ON CIFARIO AND CIFARI00
The neural network model is a residual network named Resnet34 in the Mahalanobis paper, and by changing the number of outputs, it can be used for CIFARIO and CIFARI00. We used the pre-trained models that are available online at http://ggihub..omoooa/ppp layers used for feature extraction, are exactly the same as those the source code of the method.
Mahalanobis detector/. The
In our algorithm, we have two different settings for f (a:): (1) it can be the OOD score, and (2) it can be the concatenation the feature vectors, given the original (not preprocessed) input. We did experiments with the two settings. In our algorithm, the parameters are & 10, = e/100, N 1000 for all experiments. The initial OOD sample a random noise image. For every sample the test set, the algorithm generated an OOD sample to match the corresponding output. The generated OOD samples look like random noises. The OOD scores of these samples were calculated by the Mahalanobis method.
The results on the two datasets are reported in Table 3 and Table 4. Fig. 14 shows the OOD score histograms of the in-distribution and OOD samples, when the in-distribution dataset is CIFARIO, PM=0.01, and f (xc) = OOD score.
Table 3: AUROC scores Mahalanobis on CIFARIO vs OOD
f f(x) OOD score of æ
f (x)
feature concatenation
PM=0 PM=0.01
0,500
0.467
0.500
0.179
Table 4: AUROC scores of Mahalanobis on CIFAR100 OOD
f f(x)
OOD score of æ
f
feature concatenation
PM=0
0.500
0.604
PM=0.01
0.500
0.377
16

---

Under review as a conference paper at ICLR 2021
000 score histogram of the in-distribution samples
1000
800
OOD score histogram of the genereted 0OD samples
600
600
200
200
400
800
1000
1200 1400
Figure 14: The OOD score histograms of the in-distribution (blue) and OOD (red) samples. The in-distribution dataset CIFARIO, PM=0.01, and (x) = OOD score æ.
Fig. 15 shows the OOD score histograms the in-distribution and OOD samples, when the in- distribution dataset is CIFAR10, 100 PM=0.01, and (a) feature concatenation. It can be seen that the OOD samples have smaller distances, which caused by input preprocessing.
score histogram indistribution samples
200
OOD score histogram of the genereted 000 samples
600
200
40c
800
800
1000
1200
1400
Figure 15: The OOD score histograms of the in-distribution (blue) and OOD (red) samples. The in-distribution dataset CIFARI0, PM=0.01, and (x:) feature concatenation.
E
APPENDIX
We applied our OOD Attack algorithm to test the OOD detection method named Outlier Exposure (Hendrycks al. 2019)
E. SUMMARY OF THE OUTLIER EXPOSURE METHOD
The method trains neural network not only the standard training set (i.e. in distribution) but also an auxiliary dataset outliers (i.e. OOD samples). the paper, it states that the OOD score is defined to be the cross-entropy between uniform distribution and the softmax-output distribution.
17

---

Under review as a conference paper at ICLR 2021
In the actual implementation (i.e. source code of the method), the OOD score is defined to be the average of the logits minus the logsumexp the logits. In the evaluation, we used the actual implementation in the souree code.
E.2 EVALUATION SVHN, CIFARIO AND CIFARIO0
Wide residual networks are used in the Outlier Exposure paper. We downloaded the source code and pre-trained weights from https://github.. https//gthib..om////nnkokoooooos wwwphh/rrccks/uuirreeppppooooo from scratch using Outlier Exposure, and they were named "oe scratch" by the authors.
The models were trained
In our algorithm, we set f (ar) to be the logits (input to softmax) from each model. The parameters are c=10, a c/100, N=le4 for all experiments. The initial OOD sample is random noise image. For every sample the test set, the algorithm generated an OOD sample to match the logits output. The generated OOD samples look like random noises. The OOD scores of these samples were calculated by the Outlier Exposure method.
The results reported in Table 5. samples are shown in Fig. 16, where the in-distribution dataset is CIFARIO.
The OOD score histograms of the in-distribution and OOD
Table 5: AUROC of Outlier Exposure on three datasets
SVHN OOD
CIFARIO OOD CIFAR OOD
0.500
0,500
0.500
histogram in-distributiin samples
0OD Score histogram of the genereted DOD samples
-12
-10
-5
-6
-4
Figure 16: The OOD score histograms of the in-distribution (blue) and OOD (red) samples. The in-distribution dataset CIFARIO.
APPENDIX
We applied our OOD Attack algorithm to the OOD detection method named Deep Ensemble Lakshminarayanan et al. 2017).
F.1 SUMMARY OF THE DEEP ENSEMBLE METHOD
Deep Ensemble is collection of neural network models working together for a classification task. The output of Deep Ensemble is probability distribution across the classes, which is the average of the probability/softmax outputs of individual models. the experiments, the number of models is in Deep Ensemble. To further improve performance, adversarial training is applied to the
18

---

Under review as a conference paper at ICLR 2021
models. The OOD score is defined to be the entropy of the probability distribution from the Deep Ensemble. The entropy is expected to be large for an OOD sample.
F.2 EVALUATE ON CIFAR10
The authors of the Deep Ensemble method did not provide source code and trained models. There- fore, we used pre-trained models from a recent work on adversarial robustness (Ding et al. 2020), which presented state-of-the-art adversarial training method. Six pre-trained models were down- loaded from htpps//gthuh..com/Brreaiiiiii htwpa//wwww..ooooooooooooooo.nne traimmingurreemmtttettteeeddd models. The names of the models are cifar/0---2-MM--100--- (aarl00-00M---.0--oole cifar (-aiarr00--------2222- ------- cifar (oarnoo-OO-OA-----
cifarl00-L2MMMAA2200000
eifar 10-L2-0MMA-I 1.0-sdo,
The models were trained CIFARIO to be robust against adversarial noises in large range. Classification accuracy of the ensemble on test set is 89.85%.
In our algorithm, we set f (z) be the concatenation of the logits from each of the six models. The parameters are s=10, a= e/100, N=le4 for all experiments. The imitial OOD sample is a random noise image. For every sample in the test set, the algorithm generated an OOD sample to match the logits output. The generated OOD samples look like random noises. The OOD scores of these samples were calculated by the Deep Ensemble method.
The AUROC of the Deep Ensemble method is 0.500 CIFAR1O 10 vs OOD, The OOD score his- tograms of the in-distribution and OOD samples are shown Fig. 17.
OOD score histogram of the in-distribution sampiis
1500
1250
1000
750
250
OOD histogram of the genereted samples
1750
1500
1250
1000
0.50
0.25
0.50
6.75
1.00
1:25
1:50
1.75
2.00
Figure 17: The OOD score histograms of the in-distribution (blue) and OOD (red) samples.
G APPENDIX
We applied our OOD Attack algorithm to test the OOD detection method that builds Confidence- Calibrated Classifiers (Lee et al. 2018a).
G.1 SUMMARY OF THE OOD DETECTION METHOD
The method jointly trains a classification network and a generative neural network (i.e. GAN) that generates OOD samples for training the classification network. Given input, the OOD score is defined to the maximum of the softmax outputs from the classification network. The OOD score expected to be low for OOD sample.
19

---

Under review as a conference paper at ICLR 2021
G.2 EVALUATION ON SVHN AND CIFARI0
The neural network model VGG13, and the source code of the method is provided by the authors We downloaded the code and trained VGGI3 model with a GAN on SVHN and another VGG13 model with a GAN on CIFARIO by using the parameters in the source code. VGG13 has feature module and classifier module.
at
In our algorithm, we set f (a) to be the vector input to the classifier module VGG13. The param- eters E=10, a= =/100, 1e4 for all experiments. The initial OOD sample is a random noise image. For every sample in the test set, the algorithm generated an OOD sample to match the vector input to the classifier module. The generated OOD samples look like random noises. The OOD scores of these samples were calculated by the OOD detection method.
The results reported in Table 6 The OOD histograms of the in-distribution and OOD samples are shown in Fig. 18, where the in-distribution dataset is CIFAR10.
Table 6: AUROC of the method on two datasets
SVHN OOD CIFAR1O OOD 0.501
0.576
OOD score histogram of the in-distribution samples
8000
6000
4000
2000
histogram the genereted samples
8000
6000
4000
0.2
0.4
066
0.8
1.0
Figure 18: The OOD score histograms of the in-distribution (blue) and OOD (red) samples. The in-distribution dataset CIFARIO.
H APPENDIX
We applied OOD) Attack algorithm to test the OOD detection method named Gram (Sastry & Oore 2019).
H.1 SUMMARY OF THE GRAM METHOD
The method extracts the feature map Fi from every layer of a network and then computes the p-th order Gram matrix Gf (FF FP") 1/p Gram matrices with different p values from different layers are then used to compute the OOD score, which is named Total Deviation of the input sample. An OOD sample is expected have high OOD score.
20

---

Under review as a conference paper at ICLR 2021
H.2 RESOLVING A NUMERICAL PROBLEM
The formula of the p-th order Gram matrix be written A = (B) The Gram matrices caused gradients to be inf or nan during back-propagation in the OOD attack algorithm. To resolve this problem, we tried three tricks:
(a) use double precision (float64)
(b) rewrite A exp( + eps)) where eps=le-40
(c) use the equation in (b) to generate images during OOD attack and use the original equation
= (B) to compute OOD scores.
The above tricks work for in the range of to 5. For larger p. still get numerical problems (inf or nan). As shown in Fig. 2 of the Gram paper, the method has already achieved better performance compared to the Mahalanobis method when the max value of is 5. Thus, set the max value of p to 5 in our experiments.
H.3 EVALUATION ON CIFARIO AND CIFARI00
The source code and pre-trained Resnet models provided by the authors https://iihub..om/eer/r/oo Vectefnsinneeggga-oooo-eeeeed
Due the unique process of the method, it is very difficult do parallel computing with mini- batches, and we have to set batch_size =1. The computing process is very time-consuming, and therefore we selected the first 500 samples CIFAR10 test set and the first 500 samples in CI- FARI00 test set in our experiments.
In our algorithm, we set f (x) to be the OOD score x, and the parameters are ==10, a= e/100, N=100. The initial OOD sample is random noise image. For every in in-distribution sample, the algorithm generated an OOD sample match the OOD score. The generated OOD samples look like random noises. The OOD scores of these samples were calculated by the Gram method.
The results are reported in Table 7. The OOD score histograms of the in-distribution and OOD samples are shown in Fig. 19, where the in-distribution dataset is CIFAR10. The results show that the OOD score from the Gram method can be arbitrarily manipulated by our algorithm.
Table 7: AUROC of Gram two datasets.
CIFARTO vs OOD CIFAR 100 vs OOD 0.500
0.500
21

---

Under review as a conference paper at ICLR 2021
OOD score histogram of the in-distribution samples
40
20
histogram genereted samples
40
40
200
400
600
800
1000
Figure 19: The OOD score histograms of the in-distribution (blue) and OOD (red) samples. The
in-distribution dataset CIFARIO.
I APPENDIX
We applied OOD Attack algorithm test the OOD detection method based Glow (Serrà et al. 2019).
I.1 SUMMARY OF THE OOD DETECTION METHOD
The method combines Glow negative log-likelihood (NLL) and input-complexity. PNG compression used to compress the input image. The input-complexity, L,, is measured by bits per dimension, where the "bits" refers the number of bits of the compressed image, and the dimension is the total number of pixels per image. The OOD score is NLL- -
1.2 EVALUATE ON CELEBA
The source code the Glow model is downloaded from https://github. com/roosialliyllllwww pyttoch,, and we trained it from scratch on CelbaA dataset, in which the size of each face image is 64643. After training, model was able to generate realistic face images. For method evaluation, we randomly selected 160 samples in the dataset because the computation cost is very high.
In our algorithm, we set f (x) to be the OOD score x, and the parameters are e=10, o= e/100, N=le4. The initial OOD sample is color spiral image. For every in-distribution sample, the algorithm generated an OOD sample to match the OOD score. The generated OOD samples look like color spirals, i.e., not face images. The OOD scores of these samples were calculated by the OOD detection method.
AUROC of the method is 0.500 CelbaA OOD. The OOD histograms of the in- distribution and OOD samples are shown in Fig. 20, The result indicates that NLL combined with input complexity still be arbitrarily manipulated.
22

---

Under review as a conference paper at ICLR 2021
OOD Score Histogram of In-distribution Samples
sample count
15
OOD Score Histogram of OOD Samples
sample count
20
10
-5.00
-5.25
-5.00
-4.75
-4.50
-4.25
-4.00
-3.75
-3.50
OOD Score
Figure 20: The OOD score histograms of the in-distribution (blue) and OOD (red) samples.
APPENDIX
We applied our OOD Attack algorithm to test the OOD detection method using energy-based model named JEM (Grathwohl et al. 2020).
J.1 SUMMARY OF THE OOD DETECTION METHOD USING JEM
In the JEM paper, it was shown that standard classifier can be trained to be an energy-based model (EBM). Using the EBM, three types of OOD scores can be obtained for an input sample, which are: (1) Log Likelihood logp(x)), (2) the maximum of softmax classification output, i.e. mazyp (y lr),
and (3) -1| alogp(a)
2..
An OOD sample is expected to have low OOD score.
8z
J.2 EVALUATION CIFARIO
wide residual network pretrained on CIFAR10 is available at
In algorithm, set f (x:) to be the logits (i.e. input to softmax for classification), and the parameters are 8=10, o= e/100, N=le3. The initial OOD sample a color spiral image. For every OOD sample match the logits output. The generated OOD samples look very weird, not in any of the 10 classes of CIFAR1O. The OOD scores of these samples were calculated by the OOD detection method.
in-distribution sample, the algorithm generated
We note that we tried to use random noises as initial OOD samples, but many generated images look like images CIFARIO dataset, and therefore, we used color spiral image the initial OOD sample.
The results are reported in Table 8. The OOD score histograms of the in-distribution and OOD samples are shown in Fig. 21, Fig. 22, and Fig. 23, We note that when using -|| ölogp(#) OOD score, the AUROC is 0.203. One may think if we flip the sign of the OOD score, then AUROC will increase to 0.797. we do so, then AUROC scores the last row of Table the JEM paper will be close to 0 for the OOD detection experiments done by the authors.
öz ||z as the
23

---

Under review as a conference paper at ICLR 2021
Table 8: AUROC of the OOD detection method CIFARIO 10
OOD
OOD score AUROC
logp(z)
maryP (y|z)
[logp( x)
0.559
0.513
0.203
OOD score (1) histogram of the in-distribution samples
80
60
40-
20-
OOD histogram of the genereted OOD samples
80-
40
20
-3
Figure 21: The OOD score (logp(z)) histograms of the in-distribution (blue) and OOD (red) sam- ples.
OOD score (2) histogram of the in-distribution samples
4000
3000
2000
1050
(2) histogram the genereted samples
4000
2000
1000
0.85
0.96
037
0.58
3.99
1.00
Figure 22: The OOD score (max, P (y|z)) histograms of the in-distribution (blue) and OOD (red) samples.
24

---

Under review as a conference paper at ICLR 2021
OOD score (3) histogram of the in-distribution samples
600
OOD histogram genereted samples
500
200
100
0.5
-.4
0.3
6.2
0.
Figure 23: The OOD score (-11 @logp(a) samples.
|12) histograms of the in-distribution (blue) and OOD (red)
8x
Fig. 24 shows an example of the loss curve over 1000 iteration in the OOD attack algorithm.
1.0
0.8
0.6
0.4
0.2
0.0
0
200
400
600
800
1000
Figure 24: An example of the loss curve from the OOD Attack algorithm.
Fig. 25 shows some of the generated images, which look like the images of Frankenstein's 's monsters: randomly put some parts of objects together, twist/deform them. and then pour some paint them. may be difficult for neural networks to learn what is an object (e.g. airplane) just from images and class labels.
25

---

Under review as a conference paper at ICLR 2021
Figure 25; Examples of the generated images.
Energy-based models (EBMs), such JEM, can generate OOD samples during training, which may explain why the OOD attack failed when the initial OOD sample was random noise. If we take closer look at the sampling procedure (e.g- Langevin dynamics) and the objective function, it is easy to find out EBM training algorithm is trying to pull down the energy scores of positive (in- distribution) samples and pull up the energy scores of negative (OOD) samples (Du & Mordatch, 2019), which is similar to the basic idea of adversarial training. From this perspective, OOD de- tection using EBMs could be promising direction if the computation cost is acceptable, and the challenge is how to train neural network to learn what is object?
26


Reviews:
[{"review_id": "vHjLHm-HFQT", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n--------------\nThe paper presents a method that attacks existing out-of-distribution (OOD) detection methods. Most of the existing OOD detection methods perform detection using a latent representation. Main motivation of the paper is that the size of the latent representation is much smaller than the input images which results mapping both OOD and in-distribution images to the same place in the latent space and diminishing OOD detection performance. With this motivation, the proposed method perturbs input images to obtain an image whose latent representation is similar to the latent representation of an in-distribution image. Since such perturbations can be obtained for any OOD image, existing OOD detection algorithms fails distinguishing such OOD samples. The paper contains experiments on multiple dataset to demonstrate that the proposed method obtains a latent representation similar to the representation of an in-distribution image.\n\nComments:\n----------------\n1 - From the abstract, I infer that the paper has roughly 3 contributions: 1) the paper shows existing OOD detection methods are practically breakable, 2) Glow likelihood-based OOD detection is ineffective, and 3) present a simple theoretical solution with guaranteed performance for OOD detection. However, the 3rd contribution is never mentioned/introduced in the paper until Appendix A where they briefly presents 2 different implementations of an OOD detection method idea. I would not consider this as a contribution since there is no experimental evaluation showing the OOD detection performance of the idea. Also, since this contribution is mentioned in the abstract, I would expect seeing its description and the results in the main paper rather than the Appendix.\n\n2 - Until the end of the Introduction, it is not very clear that the main contribution of the paper is a method that attacks OOD detection methods. I liked the motivating example in Figure 1, but the contribution can be given in a more clear way.\n\n3 - In the last paragraph of page 2, it is mentioned that the clip operator can ensure x_out to be OOD after a small modification to x'_out. I found this statement quite vague. How this operation \"ensures\" that x_out to be OOD after the modification?\n\n4 - In the last paragraph of Section 2.1, it is mentioned that adding initial random noise helps to avoid local minimum caused by a bad initialization. Is this something that you observed empirically? The contribution of the noise is not very clear to me and I think showing results with and without the noise would be very useful to demonstrate how the noise helps to avoid local minima. Also, it would be interesting to show how this loss evolves during the optimization.\n\n5 - In Sec 2.2 - Each pixel in an 8-bit image can take 256 different values. So, for an image with size 224x224x3, there are 256^(224x224x3) possible images that can be generated; not 8^(224x224x3).\n\n6 - I didn't quite understand the message in the last paragraph of Sec. 2.2. Why \\Omega_in is split into \\Omega_{in_clean} and \\Omega_{in_noisy}? How this is used in the proposed method?\n\n7 - I think that experimental evaluations are not comprehensive enough, \n7.1. It is mentioned that implementing OOD detection methods are not necessary since Alg.1 already produces z_out ~= z_in. I don't agree with this since most of the SoTA methods do not rely on only a single representation but multiple representations at different layers [ref1, ref2, ref3]. Therefore, it is crucial to show the performance loss of these methods due to the proposed method.\n\n[ref1] Erdil et al., Unsupervised out-of-distribution detection using kernel density estimation\n[ref2] Lee et al., A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks\n[ref3] Sastry et al., Detecting Out-of-Distribution Examples with Gram Matrices\n\n7.2. Benchmark datasets that have been used in OOD detection paper are usually different than the ones used in the paper. Please see the references above. I wonder why the datasets choice are different than the standard benchmarks?\n\n7.3. As mention in paragraph 3 of Sec. 4., there are other generative models that show promising results. However, I didn't quite understand comparison with these methods are \"out of the scope\" while Glow is relevant.\n\n8. The parameters used in different experiments are different than each other. How are these parameters determined? How sensitive is the proposed method to parameter choice?\n\nMinor comments:\n-------------------------\n1 - In Introduction paragraph 2, there is a typo: laten -> latent\n2 - In Sec 2.1, paragraph 3: the algorithm is referred as \"above algorithm\" but it appears \"below\" in the paper.\n3 - In Sec 2.2, paragraph 3, there is a typo: state-of-art -> state-of-the-art\n4 - In Sec 2.2, paragraph 3, there is a typo: laten -> latent\n\nOverall:\n-----------\nI found the idea of attacking existing OOD detection methods interesting. However, I believe, this paper needs improvement in terms of clarity of the presentation, experimental evaluations and the presentation of the contributions. Therefore, my initial rating is reject for this paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "oQJRCSOMGfC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper defines an out-of-distribution attack, a process which drives an out-of-distribution (OOD) input to have the same latent representation to an inlier. The paper also analyzes that an encoder is inevitably vulnerable to out-of-distribution attack when its latent dimensionality is smaller than the dimensionality of the input.\n\nDecision:\n\nReject\n\nStrength:\n\nThe paper addresses an important vulnerability of classifier-based OOD detection. As classifier-based method is one of the currently dominating approaches for OOD detection, investigating its weakness is a significant contribution to the research community.\n\nWeakness:\n\nThe proposed attack algorithm is significantly similar to the previously known adversarial attack algorithms and therefore seems trivial.\n\nThe main quantitative result, Table 1, is not very convincing. I suggest the authors  provide AUC scores computed before and after OOD attack, so that the difference clearly shows that the proposed attack causes a decrease in OOD detection performance.\n\nThe paper should benchmark the proposed attack algorithm against state-of-the-art OOD detection methods. Currently, only a relatively simple method of Hendrycks and Gimpel, 2016 is used. The method should include at least [1,2,3] to show the effectiveness of the proposed attack. At least some of these OOD detection methods may be able to resist the proposed attack. For example, multiple hidden layer representations from a classifier are used in [1], and therefore it can still detect OOD even if a specific latent representation is under attack.\n\nThe organization of the paper needs to be improved. In the last sentence of the abstract, \"a simple theoretical solution\" is mentioned but is only addressed in Appendix. If it is a contribution that is important enough to be mentioned in the abstract, it should be covered in depth in the main manuscript instead of Appendix.\n\nMinor comments:\n- The visibility of figures are poor. The axis titles and MAPE values in Figure 2, 3, 4, 5 should be larger.\n- In Section 2.2, 8^{224*224*3} should 256^{224*224*3}. An 8-bit integer can represent 256 values. \n- The captions of Figure 3 and Figure 4 are the same.\n- The captions of Figure 5 and Figure 6 are too close.\n\nTypos:\nlaten → latent (Section 1 paragraph 2 line 10)\nDicussion → Discussion (Section 4 title)\nDifficulty → difficult (Section 4 paragraph 4 first line)\n\n[1] Lee, Kimin, et al. \"A simple unified framework for detecting out-of-distribution samples and adversarial attacks.\" Advances in Neural Information Processing Systems. 2018.  \n[2] Liang, Shiyu, Yixuan Li, and Rayadurgam Srikant. \"Enhancing the reliability of out-of-distribution image detection in neural networks.\" arXiv preprint arXiv:1706.02690 (2017).  \n[3] Grathwohl, Will, et al. \"Your classifier is secretly an energy based model and you should treat it like one.\" arXiv preprint arXiv:1912.03263 (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "XYwWwg2DSwl", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n**UPDATE**\n\nI acknowledge that I have read the author responses as well as the other reviews. Overall, I appreciate the clarifications and added experiments given by the authors. \n\nMy concerns about the low novelty of the presented algorithm and findings remain, however, as I find the OOD attack to be only a slight modification of existing adversarial attacks.\n\nI also appreciate that the defense solution claim has been weakened and moved to the appendix, yet these promises are still left to be validated.\n\nLastly, I find all the many added experiments positive, but these have significantly changed the content of the initial submission at this point, which is somewhat out of scope of the ICLR rebuttal phase (see Q4 in the FAQ of the Reviewer Guide: https://iclr.cc/Conferences/2021/ReviewerGuide).\n\nFor these reasons, I would keep my recommendation to reject this work for ICLR 2021, but I encourage the authors to further improve and re-submit the now extended work to some future venue.\n\n#####\n\n\n**Summary**\n\nThis paper presents an algorithm for out-of-distribution (OOD) attacks on neural networks. Given a target network and an initial OOD input, the proposed algorithm performs projected gradient descent (PGD) w.r.t. the input to obtain an output that is close to the embedding of some desired in-distribution sample. The paper makes the case that dimensionality reduction is one key reason that makes standard deep networks vulnerable to OOD attacks, due to the non-bijective nature of such mappings. An experimental evaluation on two ImageNet- pretrained classifiers (ResNet-18 and DenseNet-121) is presented, where a subset of ImageNet serves as in-distribution and chest x-ray, lung-CT, and noise images serve as OOD samples, that demonstrates empirically that these networks can be attacked with the proposed algorithm. Another experiment on the likelihood-based normalizing flow model Glow is carried out which demonstrates that such bijective, dimensionality-preserving deep generative models are also breakable by the proposed attack. Finally, a theoretical sketch for a solution of the problem is described.\n\n\n**Pros**\n+ The paper presents a simple OOD attack algorithm and demonstrates empirically that OOD samples can be perturbed such that they map to the embedding of some arbitrary in-distribution example.\n+ The paper makes a plausible argument that dimensionality reduction enables OOD attacks. Inversely, it is argued that reconstruction methods, which map back to the original space, are favorable for OOD detection for which the proposed attack is also ineffective.\n+ Attacking the dimensionality-preserving, likelihood-based Glow model is an interesting experiment to consider in the context of the presented dimensionality reduction argument.\n+ The paper is structured well and overall well-placed into existing literature.\n\n**Cons**\n- I find the novelty of the presented algorithm and experimental findings to be rather low.\n- The experimental evaluation of the proposed attack is limited to standard classifiers and does not include deep networks that have been trained to increase OOD robustness [1, 2, 4, 3].\n- The defense solution is only a sketch and makes promises that are left to be validated.\n- There are many errors in language and grammar, e.g. wrong use of tense, missing articles, etc. (see minor comments below)\n\n\n**Recommendation**\n\nI think the current paper is ok but not good enough (score: 4) due to (i) low novelty, (ii) a limited experimental evaluation, and (iii) solution claims that are left to be validated.\n\n(i) Though I see and agree that the OOD attack setting is slightly different to adversarial attacks, I find the brittleness of standard classifiers in this regard not surprising. In particular, the OOD attack has a greater degree of freedom since any arbitrary OOD input can be used as a starting point for perturbation (pure noise is also OOD, as remarked in the paper), i.e. there is no similarity constraint on the input as there is for adversarial attacks. Moreover, I find the algorithmic novelty to be low as well, since the proposed algorithm essentially is a slight adaptation of previously introduced projected gradient descent (PGD) attacks.\n\n(ii) Following (i), I think the current experimental evaluation is limited and the findings are not surprising for the two standard, pre-trained classifiers (ResNet-18 and DenseNet-121). There exist many approaches that have shown to improve OOD robustness [1, 2, 4, 3], which should be included in the analysis. It would be interesting to see how these approaches perform and compare, which could be insightful for improving OOD robustness.\n\n(iii) Proposing an attack begs the question what possible defenses could be. Currently, the main paper is only phenomenological, i.e. demonstrates that OOD attacks are an open issue, but the description of a possible defense at the end of the paper and in the appendix makes only a solution claim which is left to be validated. I don’t say or think such a solution would be necessary for an interesting and valid contribution, as OOD detection poses a hard problem which likely lacks a simple solution, but the current solution is a mere sketch making promises with questions left open. For example, how can a sufficient space saturation be achieved with a finite sample in practice? Which measure to use?\n\n\n**Additional feedback and ideas for improvement**\n\nI think the OOD detection problem, both from an attack and defense perspective is relevant and of great interest to the community, which is why I encourage the authors to build upon and extend the current manuscript. Some ideas:\n\n- Including methods that have shown to improve OOD robustness [1, 2, 4, 3] would greatly improve the value of the analysis. This could be insightful to further improve OOD robustness and see pros/cons of existing approaches.\n- Does adversarial training, as mentioned in Section 2.2, also help OOD robustness?\n- Implementing and validating the presented OOD detection solution would be interesting.\n- Further investigate the reason why the dimensionality-preserving Glow model can be attacked as well, as this suggests dimensionality reduction is only a part of the issue.\n- ‘Could we design an evaluation method (experimental or analytical) that does not rely on OOD samples?’ I think this is a worthwhile question to ask and research.\n- Add missing related work [5].\n\n\n**Minor Comments**\n\n1. ‘The algorithm needs a weak assumption that $f(x)$ is differentiable’ I think sub-differentiable would be sufficient, correct?\n2. ‘If the neural network only uses ReLU activation, then the input-output relationship can be exactly expressed as a linear mapping’ Only piecewise linear mapping, right?\n3. ‘[...], the above mentioned classification-based OOD detection is theoretically almost ineffective [...]’; ‘[...], then it is highly possible that the entire latent space is crawling with the shadows of OOD samples.’; ‘[...], there are a huge number of “holes” in the space, [...]’ Rather avoid such vague formulations.\n4. Published works should be included as such in the references, not by their preprint reference (e.g., [2] appeared at ICLR).\n5. A list of citations in a sentence should be concatenated within single parentheses/brackets, separated by commas (not individual parentheses/brackets per citation).\n6. When referring to a specific section, section should be capitalized, e.g. Section 3.\n7. Abstract: ‘Deep Neural *Networks (DNNs)*, especially convolutional neural *networks (CNNs)*, *have* become ...’ It is a class of models, hence plural.\n8. Algorithm 1: ‘[...] not similar to *any sample* in the dataset.’; ‘$\\alpha$ the learning rate of *the* optimizer’; $h$ used in the Algorithm is not defined; I would use $\\nabla_x J$ instead of $J'$ for the gradient to stress the derivative is w.r.t. the input.\n9. Many figure labels are tiny and hard to read.\n10. The captions of Figure 3 + 4 are (almost) identical. Add a label indicating the network.\n11. Section 1: ‘It was shown by *Nguyen et al. (2015)* [...], and *an* evolutionary algorithm was used [...]’\n12. Section 1: ‘Since then, many methods *have been* proposed for OOD detection [...]’\n13. Section 1: ‘For instance, *Hendrycks et al. (2016) show* that a classifier’s prediction *probabilities* of OOD examples tend to be *more uniform* [...]’ ‘lower’ is ambiguous as the softmax probabilites always add up to 1, right?\n14. Section 1: ‘For *the evaluation of OOD detection methods*, an OOD detector is *usually* trained [...]’\n15. Section 1: ‘[...] pre-trained on *the* ImageNet dataset.’\n16. Section 1: ‘[...] which could be any kind of *image* (even random *noise*) [...]’.\n17. Section 1: ‘[...], which is the input to the last fully-connected linear layer before *the* softmax operation.’\n18. Section 1: ‘[...], and *one* fix to the problem could be using likelihood ratio [...]’\n19. Section 1: ‘[...], we will show that *the* OOD sample’s likelihood score from the Glow model [...]’\n20. Section 2: ‘[...] projected gradient descent (PGD) which is used for adversarial *attacks*.’\n21. Section 2: ‘In practice, *Algorithm 1 can be repeated* many times [...]’\n22. Section 2: ‘[...], the algorithm will have a better chance to avoid *a local minimum* [...]’\n23. Section 2: ‘This *is* simply because the vectors in a lower-dimensional space [...]’\n24. Section 2: ‘[...], which is the Pigeonhole Principle.’ Citation?\n25. Section 2: ‘Usually, *the* training set is only a subset’\n26. ‘3 EXPERIMENT’ Do not capitalize and ‘3 Experimental Evaluation’?\n27. Section 3: ‘*A* Nvidia Titan V GPU was used [...]’\n28. Section 4: ‘We would like to point out that it is *difficult* to evaluate [...]’\n\n\n#####\n\n**References**\n\n[1] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, pages 6402–6413, 2017.\n\n[2] K. Lee, H. Lee, K. Lee, and J. Shin. Training confidence-calibrated classifiers for detecting out-of-distribution samples. In ICLR, 2018.\n\n[3] K. Lee, K. Lee, H. Lee, and J. Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS, pages 7167–7177, 2018.\n\n[4] S. Liang, Y. Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In ICLR, 2018.\n\n[5] A. Meinke and M. Hein. Towards neural networks that provably know when they don’t know. In ICLR, 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Op4ckaT5qbw", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary of paper: this work shows that adversarial perturbations can make any OOD image, map to the same latent code as an in-distribution - creating an attack on confidence-based or flow-based ODD detection methods. Results are shown on a few datasets with some attempts at evaluation.\n\nNovelty: Unfortunately, I don't think there is much new here. Adversarial attacks are of course well known - I am not sure that attacks on intermediate latent codes present novelty either. Adversarial attacks against anomaly detection methods have also been investigated before (e.g. [1] [2], although their setting is a little different) and there is nothing in the proposed method that is particularly tailored to OOD.\n\nEvaluation: the evaluation is not extensive - only one adversarial attack is investigated and no reasonable baselines have been selected. I am not sure that MAPE is an appropriate metric - it really depends on the allowed perturbation. Ss the allowed perturbation small enough? do the perturbed images look realistic - if I understood Fig.12, they don't - but the caption there is not clear.\n\nClarity - the paper is not particularly clearly written - although the idea is simple enough. E.g. I don't see the scatter plots clearly explained, the analysis in Sec. 2.2 is very dense for a fairly simple idea. \n\nOverall: ultimately, this is conceptually repeating the same thing as any other adversarial examples work, perturbations can make a network confident that any image has the label of another image - and this obviously would overcome confidence based OOD detection methods. I therefore do not see a strong contribution by this work. As there is also very limited methodological novelty, I do not think it should be accepted.\n\n###############################################################################\n\nI understand the distinction the authors are trying to draw between adversarial examples for anomaly detection and fooling OOD to think that images are in distribution where in fact they are OOD. I still don't think that technically or conceptually, there is much difference. The authors presented many fresh results during the rebuttal (which might have been better presented just as a table in the manuscript, rather than on this thread). The experiments can form a part of a resubmission of this work, that will incorporate the extensive comments presented by the current reviews.\n\n[1] Rigaki, Maria. \"Adversarial deep learning against intrusion detection classifiers.\" (2017).\n[2] Bergman and Hoshen, Classification-based anomaly detection for general data, ICLR'20", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: **Problem significance** This paper proposes an attack mechanism in the latent space of a neural network f(x), which produces out-of-distribution examples. The AC agrees reviewers on the significance of the OOD detection problem, particularly addressing the vulnerability aspect is relevant and of great interest to the community. 

**Technical contribution** The AC shares the concern with several reviewers on the limited technical novelty as well as the problem formulation. While the authors have clarified the difference between adversarial attack vs. OOD attack, the underlying attack mechanism is not new to the community (except for allowing for a larger degree of search space without constrained by the visual imperceptibility). In some sense, the search is made easier than the standard adversarial attack by removing the similarity constraint. Given the unrealisticness of the created OOD examples (largely noisy patches), the AC thinks perhaps a more interesting problem is to look at naturally occurring OOD examples that would lead to the similar latent encoding w.r.t in-distribution data, or adversarial robustness w.r.t the OOD detector.  This to me, would steer the community in the right direction. 

From a problem formulation perspective, the AC thinks it's useful to differentiate three highly related attacks (that are distinct but can cause confusions):

- adversarial attack w.r.t the classifier
- OOD attack w.r.t the classifier 
- adversarial attack w.r.t the OOD detector (see recent works [1][2][3] which considered the robustness aspect of OOD detector)


**Rebuttal feedback** The AC recognizes the effort made by the authors to address the concerns and comments raised by reviewers. The AC agrees with R1/R2/R3 that the additional experiments are valuable, however, the changes to the manuscript are substantial enough to deem another round of review in the future venue. The paper can improve with better organization and presentation, moving the results in the appendix to the main paper. 

**Recommendation** The AC recommends rejection. 

References

[1] Sehwag et al. Analyzing the robustness of open-world machine learning. 2019

[2] Hein et al. Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem. 2019

[3] Chen et al. Informative Outlier Matters: Robustifying Out-of-distribution Detection Using Outlier Mining. arXiv:2006.15207




Decision: Reject