Your Out-of-Distribution Detection Method is Not Robust!
Authors: Mohammad Azizmalayeri
Arshia Soltani Moakar
Arman Zarei
Reihaneh Zohrabi
Mohammad Taghi Manzuri
Mohammad Hossein Rohban
Venue: NeurIPS 2022 Accept (2022)
Keywords: Out-of-distribution Detection
Adversarial Robustness
Attack

Abstract: Out-of-distribution (OOD) detection has recently gained substantial attention due to the importance of identifying out-of-domain samples in reliability and safety. Although OOD detection methods have advanced by a great deal, they are still susceptible to adversarial examples, which is a violation of their purpose. To mitigate this issue, several defenses have recently been proposed. Nevertheless, these efforts remained ineffective, as their evaluations are based on either small perturbation sizes, or weak attacks. In this work, we re-examine these defenses against an end-to-end PGD attack on in/out data with larger perturbation sizes, e.g. up to commonly used $\epsilon=8/255$ for the CIFAR-10 dataset. Surprisingly, almost all of these defenses perform worse than a random detection under the adversarial setting. Next, we aim to provide a robust OOD detection method. In an ideal defense, the training should expose the model to almost all possible adversarial perturbations, which can be achieved through adversarial training. That is, such training perturbations should based on both in- and out-of-distribution samples. Therefore, unlike OOD detection in the standard setting, access to OOD, as well as in-distribution, samples sounds necessary in the adversarial training setup. These tips lead us to adopt generative OOD detection methods, such as OpenGAN, as a baseline. We subsequently propose the Adversarially Trained Discriminator (ATD), which utilizes a pre-trained robust model to extract robust features, and a generator model to create OOD samples. We noted that, for the sake of training stability, in the adversarial training of the discriminator, one should attack real in-distribution as well as real outliers, but not generated outliers. Using ATD with CIFAR-10 and CIFAR-100 as the in-distribution data, we could significantly outperform all previous methods in the robust AUROC while maintaining high standard AUROC and classification accuracy. The code repository is available at https://github.com/rohban-lab/ATD.

Your Out-of-Distriuution Detection Method is Not Robust!
Mohammad Azizmalayeri, Arshia Soltani Moakhar, Arman Zarei, Mohammad Taghi Manzuri, Mohammad Hossein Rohban
Reihaneh Zohrabi,
Department of Computer Engineering Sharif University of Technology
{m. azizmalayeri, arahia. arohia.soltani,
arman zarei, zohrabi, manzuri, rohban][shariis
Abstract
cut-of-distributtio (OOD) detection has recently gained substantial attention due to the importance of identifying out-of-domain samples reliability and safety. Although OOD detection methods have advanced by great deal, they are still susceptible to adversarial examples, which a violation of their purpose. To mitigate this issue, several defenses have recently been proposed. Nevertheless, these efforts remained ineffective, as their evaluations are based on either small perturbation sizes, weak attacks. In this work, re-examine these defenses against an end-to-end PGD attack on in/out data with larger perturbation sizes, e.g. up commonly used 3/555 for the CIFAR-10 dataset. Surprisingly, almost all of these defenses perform worse than a random detection under the adversarial setting. Next, we aim to provide a robust OOD detection method. In an ideal defense, the training should expose the model to almost all possible adversarial perturbations, which can be achieved through adversarial training. That is, such training perturbations should based on both in- and out-of-distribution samples. Therefore, unlike OOD detection in the standard setting, access to OOD, as well in-istributtion, samples sounds necessary in the adversarial training setup. These tips lead us to adopt generative OOD detection methods, such as OpenGAN, as baseline. We subsequently propose the Adversarially Trained Discriminator (ATD), which utilizes a pre-trained robust model to extract robust features, and a generator model create OOD samples. We noted that, for the sake of training stability, in the adversarial training of the discriminator, one should attack real in mn-disrrbution well as real outliers, but not generated outliers. Using ATD with CIFAR-10 and CIFAR-I06 100 as the in-distribution data, we could significantly outperform all previous methods in the robust AUROC while maintaining high standard AUROC and classification accuracy. The code repository is available at https:///gttaei http/gttttuu....m how.a//.ooooooooaaaaaa////
Introduction
Advances in the deep neural networks have led to their widespread use in real-world applications, such as object detection and image classification 2 These models have a high degree of generalizability to the extent that they assign an arbitrarily high probability even to the samples not belonging to the training set 3 This phenomenon causes problems safety-critical applications like medical diagnosis or autonomous driving that should treat anomalous data differently.
The problem of identifying out-of-distribution data has been widely explored in different categories such as Novelty Detection, Open-Set Recognition and Oft-of-uissee Distribution (OOD) detection 4 5 6
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Afrrct in data
Africk dafn
Affirn in and nata
8-
AIL DANES
AID B0x00
AIL LOann
OpendADi
OpenGAl
OpenGAN
USAD
OSAD
ALOZ
Figure 1: OOD detection scores for several models against the perturbation bound. The perturbations designed using end-to-end & PGD attack. CIFAR-10 10 is used as the in-distribution dataset. a) Only the in-distribution dataset is attacked in the evaluation. b) Only out-distribution datasets (e.g.. MNIST, Places, etc.) attacked in the evaluation. c) Both in- and out-distribution datasets attacked. ATD (Our method) outperforms others by significant margin.
These categories are almost similar since they all consider two disjoint sets called the closed (or normal) and open (or anomalous) sets, and the model should discriminate them [7]. So far, outstanding models have been proposed in this research field 8 9 10 11]. Still, these models need to be examined
from other aspects, specifically robustness against adversarial perturbations added to the input data.
Deep networks are vulnerable against adversarial examples. This phenomenon was first noticed in the image classification, but it also extends to other domains [12 are the inputs that are slightly perturbed with a bounded perturbation (e.g. through bounding the (p-nomm) to cause high prediction error in the model. Several defenses have been proposed make the image classification robust, none of which is as effective as adversarial training and its variants
13 [4]. Adversarial examples
[15 [6 17 18]. Similar to other learning domains, OOD detection methods also suffer from the existence of adversarial examples. A small perturbation can cause a sample from the closed set to be classified anomaly and vice-versa [9 20
Robust OOD detection the connection between these two safety-critical issues. Since sample does not change semantically under appropriately bounded perturbation in the adversarial attacks, the OOD detector model is expected to be able to label them as open/closed correctly. As a first thought, one might envision solving the issue by using the image classification defenses. Despite advances in these defenses, they have no intuition of the OOD data and not seen them in the training, making them less efficient against end-to-end attacks the OOD detection method.
Two main approaches are introduced in previous works for the robust OOD detection. The first one uses some OOD) data, uniformly sampled from the open set classes and disjoint from the closed set, and employs adversarial training to encourage the model to assign uniform label, as opposed to one-hot labels, to anomalous samples even under adversarial perturbations [21 22 23 The second approach conducts adversarial training only on the closed set, but encourages the model to learn more semantic features using representation learning techniques such as auto-encoders, self-supervised learning, and denoising methods 2 25). Despite the benefits of these methods, they still have fundamental issues such as:
They did evaluate their method with an end-to-end attack on the detection method, e.g. they attack the classification model, and not the detector, to evaluate the detection method 24 25 We show that this is not an effective and strong attack the detector.
They consider small perturbation sizes (e.g. 255 in the CIFAR-10 dataset [23] or not attacking the in-distribution images do not change perceptually with larger perturbation sizes (e.g. 255 in the CIFAR-10 dataset). Thus, these methods should be trained and evaluated against larger perturbations.
I9 22 to protect the standard classification accuracy, while we show that
The second approach is still unaware of OOD samples during training. Furthermore, the first approach also cannot cover various aspects of OOD data since they can only use small number of OOD training samples.
In this work, we first evaluate previous defenses as well as standard OOD detection methods against an end-to-end PGD attack [55. shown in Fig. [ almost all of the earlier methods perform worse than random in the binary OOD detection with e 255
$ in CIFAR- 10. As result, the robust

---

OOD detection method is still far from being solved at present. In spite of this, it is possible to design robust model by addressing the drawbacks of previous methods. To this end, we propose Adversarialy Trained Discriminator (ATD) that uses a GAN-based method to generate OOD samples and discriminate them from the closed set. An ideal generator naturally deceives the discriminator, and therefore produces adversarial outputs from the open set [26] 27 28). This leads to the implicit robustness of the discriminator against adversarial attacks on the open set. In addition, we also conduct adversarial training on the closed set and the tiny real open set (known outlier exposure) to achieve robustness on them as well. Our results (e.g., Fig. D show that ATD outperforms previous methods by large margin. Therefore, this article has taken a significant step in this area by revealing the vulnerability of all previous methods and providing a solution.
Background
2.1 Out-of-Distribution Detection
Probabilities: A simple but effective method for anomaly detection the Maximum Softmax Proba- bility (MSP) [29 [30] This method is applied a K-cllas classifier and returns m fa(x)
as the closed-set membership score of the sample x. Currently, it is shown that using the MSP with Vision Transformers (ViT) [3]] leads to SOTA results in cross- cross-dataset open set recognition [TT 32).
Also, OpenMax 33 replaces the softmax layer with a layer that calibrates the logits by fitting a class-wise probability model such the Weibull distribution 34
The anomaly sample can detected by its distance to the class conditional distributions. Mahalanobis distance (MD) (35) and Relative MD (RMD) (36) are the main methods in this regard. For an in-distribution with K classes, these methods class conditional Gaussian distribution (uR, Σ) to the pre-logit features The mean vector and covariance matrix are calculated as:
Distances:
1 N
Σ 2
Σ
1 N
Σ
Σ
(z, (=)(=--
k = 1,,, K.
(1)
k=
In addition, to use RMD, one has fit a N(uo 20) to the whole in-distribution. Next, the distances and anomaly score for the input x' with pre-logits z' are computed as:
MDx(2²) = (2' -mx= t (k)) RMDx(=)) MDx(2') -DD((2), scoremD(x') = min (iDD SCOFeRMD(E') min min((RMA(()))
(2)
(3)
Discriminators: Previous categories define the score function based on K-way classifier, but one can directly train a binary discriminator for this purpose. Outlier Exposure (OE) [37] exploits some outlier data to learn a binary discriminator for the open set discrimination. GOpenMax (38], OSRCI (39] and Confident Classifier [40] augment the closed set with fake images through a GAN-based generator. Also, OpenGAN [28] selects the best discriminator model with an open validation set during training due to unstable training of GANs.
2.2 Adversarial Attacks
For an input x with the ground-truth label an adversarial example x is crafted by adding small noise such that the predictor model loss J(x", y) is maximized. The lp norm of the adversarial noise should be less than specified value i.e. ||æ jar-z" ==ppp < c, to ensure that the image does not change semantically. Fast Gradient Sign Method (FGSM) [13] maximizes the loss function with single step toward the sign of the gradient of [zz y) with respect to the z:
+ e. (igg(Va.J(z,))
where the noise meets the fox norm bound c. Moreover, this method can be conducted iteratively 41 with smaller step size
= x. = = x + a sign 2=,
(5)
where the noise should be projected to the ball with radius at each step, which is called Projected Gradient Descent (PGD) attack [15). There are number of other attacks for evaluating robustness of the models 42 43 44 but PGD is regarded as standard and powerful attack.
3

---

170
142
140
123
1930
080
Figure 2: OOD score distribution shift after an end-to-end PGD attack with as the in-distribution set. The first and second row correspond to the ALOE and OSAD methods, respectively. In each row, the left column represents the standard score distributions of in- and out-distributions. In each the next columns, one the in or out sets is attacked. The plots show that the standard distribution scores have drastically changed after the attack.
using CIFAR-10
2.3 Adversarial Defenses
The most effective defenses for image classification and OOD detection variants of adversarial training. which are described in the following.
Adversarial Training (AT) [15]: Through the use of adversarial examples during training, the model can learn robust features withstand adversarial perturbations. This method is called AT and can be used to optimize model fe as:
J(x ; fo). arg migmin B(r,a)DD... 1lo
where the inner maximization can be approximated with PGD and outer minimization with SGD, AT naturally reduces the standard accuracy, i.c. the accuracy unperturbed samples (45). which can harm the anomaly detection score [[I]. Therefore, we also propose Helper-Base -Bsee Adversarial Training (HAT) [46 as baseline that achieves better trade-off between accuracy and robustness.
AT lacks information about the outlier data. Therefore, it is only robust on the closed set and not on outliers. To mitigate this issue, ALOE includes some outliers in the AT similar OE (37]. Outliers are used with uniform label U and attacked during training to obtain model that is robust on both in- and out-distribution. The objective function ALOE is summarized as:
Adversarial Learning with inliner and Outlier Exposure (ALOE) 23]:
arg min ((..))D
max we
J( M:: So) + X. E.E.D
max
Hx² Uk; so).
(7)
An identical objective function is also used in RATIO (21] with a la perturbation bound. Alternatively, one could consider using clean outliers samples in the training. addition to ALOE, we evaluate this alternative method as baseline, and refer to it as Adversarial OE (AOE).
Instead of including outlier data in the AT, OSAD tries to learn more semantic features in the training. To this end, OSAD adds dual-attentive denoising layers the model architecture, and combines the AT loss with an auto-encoder loss, whose aim is to reconstruct the clean image from its adversarial version. A self-supervision loss is also added by applying transformations on the input image. They claim that this combination improves OSR robustness, but they did not evaluate the model against an end-to-end attack.
Open-Set Adversarial Defense (OSAD) 24 25
3 Method
OOD detection method should robust against adversarial perturbations that are added to the closed or open datasets. Despite previous efforts to provide robust OOD detection methods, the

---

Real
Perturbation 8/255)
Pretrained Robust Model
Discriminator
5
Real Out
Perturbation (e 8/255)
In/Out
(NoO)v
Generator (Fake Out)
Figure ATD schematic architecture. Generator, discriminator, and robust feature extractor are represented with yellow, pink, and green colors, respectively.
evaluations have yielded a false sense of robustness due to the weak evaluation attacks. In Fig. we have evaluated ALOE and OSAD methods using an end-to-end PGD attack with c 255 According to the results, both the in and out detection score distributions remarkably change with this attack. So, these methods are not sufficiently robust. In the following, we aim to provide more robust solution by addressing the drawbacks of previous works. We then use toy example to provide more insights into the problem and our solution.
3.1
ATD: Adversarially Trained Discriminator
A robust OOD detection method must consider several factors to achieve robustness on both open and closed sets. There has been an extensive study of closed set robustness, which shows that variants of AT are the most effective defenses. Still, pure AT cannot provide robustness in OOD detection for two primary reasons. First, an AT model achieves lower accuracy due to the trade-off between accuracy and robustness [6 45). Note that accuracy plays important role in the OOD detection [л] Second, the AT does not consider samples from the open set during training. ALOE tried to mitigate this issue using open datasets in training. However, this strategy works only when the open training data a close proxy for the OOD data. Unfortunately, this not true in most cases.
To resolve these issues, we consider OpenGAN as our baseline. Here, the discriminator performs the in/out binary classification. In this setting, generator crafts OOD images to deceive discriminator by images similar to the closed set. An ideal generator would cover a broader range of OOD distributions, which is perfect setting for our problem. Furthermore, as the generator trained to deceive the discriminator, the generated samples are naturally adversarial examples to the discriminator, which meets the needs the AT. Hence no adversarial attack need on such samples. Thus, only need to perturb the closed set adversarially train the discriminator. We call this method Adverarially Trained Discriminator (ATD), which can be summarized as:
min max Bsn
log min
D(x") + EZEN log(1 - D(G(=))).
where the and G are the discriminator and generator, respectively. Moreover, the inner minimization approximated with the PGD attack and the outer minimization and maximization with SGD. Using adversarial perturbations to train GANs makes them more robust to adversarial examples, just like other architectures [47 48). On the other hand, adding perturbations the closed set makes difficult to optimize the discriminator. As a result, the generator would not be trained to cover a broad range since it could easily fool the discriminator in the min-max game for training GANs.
To get closer to the ideal generator that can cover broad range of OOD distributions, unlike previously proposed robust GANs [47], make the generator craft features instead of images. is
5

---

evident that generating and discriminating the low- -ww-dmensional features is an easier problem than the high-dimensional images, which brings the generator closer the ideal case. a result, we also need a model that can extract robust features from the closed set. For this purpose, we use a pre-trained model with the HAT [46] method that achieves satisfactory accuracy along with robustness on the closed set, considering the trade-off between accuracy and robustness [45]. The last layer, which could be thought of as a linear classifier, is excluded to obtain features instead of logits.
Finally, similar to the OpenGAN and ALOE, we utilize an open dataset in the training to stabilize the training. This data is also attacked during training since it is neither originally adversarial, nor is crafted by the generator. With these in mind, the objective function of the ATD can be reformulated as follows:
min max E. log min
D(fo(z")) +
(9)
a. F.E log(1
max 1l
D(Jo(x)))) + (1 (1- EZEN log(1 - D(G()))),
where fo is the pre-trained robust feature extractor and a controls the weight of real open dataset in the optimization. Fig. shows a schematic representation of ATD.
3.2
Toy Example
Here, we provide an insightful visualization of the ATD using 2D toy example. To this end, we represent open and closed sets with disjoint distributions in 2D space. Also, some samples selected randomly in this space as the generated data. Fig. 4(a) shows the distributions used for this purpose, which the blue, orange, and green samples are closed, open, and generated data, respectively. Next, we train a multi-layer feed-forware neural network as the discriminator to perform the binary classification of in/out on these data in different cases. It should be noted that the open and closed data are fixed during the training, but the generated data is resampled in every epoch to simulate ATD training. Finally, the discriminator output is displayed all over the space with a blue (in) and orange (out) background in each case.
Genembor
Ou
Cal
[a]
[b)
[c]
Desrador
(d)
[e]
Figure 4: Two-dimensional example to illustrate the effect of ATD method. a) The in- and out- distributions samples that are considered in this example, which are represented with blue and orange colors, respecitvely. In and out samples are fixed, while the generated data colored in green is resampled at each epoch. b to f) Using these distributions, OOD classification has been trained with different attack settings and the classifier output is represented with blue (in) and orange (out) background all over the feature space. Above each plot, green ticks and red crosses indicate which data distributions are attacked.

---

In our analysis, different conditions are taken into account. First, standard training is conducted on the data without attacking any samples. The results in Fig. Hbb]show that the model has learned the classification decision boundaries very well. In each of the next three cases, one of the closed, open and generated sets is attacked during training with e = Fig. [ shows that attacking the closed set leads to a larger margin in the decision boundaries around them. Having this margin ensures robustness against attacks that seek to misclassify the in-distribution samples. Additionally, note that this margin is smaller in the right side that an open set is present, which shows the impact of using open sets during training. In Fig. Fra] only the open set is attacked during the training. This has caused tighter decision boundaries around the closed set where the open set is present. Therefore, using attacked open set in the training can slightly improve the robustness on OOD data, but it can simultaneously reduce the robustness the closed set. As result, in Eq. 9 that controls the optimization weight of open set should be selected carefully. In contrast with the ATD method, we also attack the generated data see how it affects the result. According Fig [ this does not cause robustness in the OOD data in contrast to Fig [(d) which is another reason that ATD do not attack the generated data during training. It is noteworthy that the main cause for this observation
may be the instability of the generated samples during the training, which is inevitable and makes the adversarial optimization goal infeasible.
Eventually, we attack both the open and closed sets during training as in ATD. The results in Fig. 4(1) show that the margin around the closed set increases in the directions that open set is not present to provide robustness on both in and out samples, which can be considered as a mixture of results in Fig. [[C) and [(d] Also, this plot demonstrates that the robustness of the open and closed sets may be at odds with each other.
4 Experiments
In this section, we perform extensive experiments to evaluate existing OOD detection methods, including the standard and adversarially trained ones, and our ATD method against an end-to- end-to-end PGD attack. To this end, we first give details about the setting of the experiments. Next, we compare all the methods, which shows that ATD significantly outperforms the other methods. Toward the end, we conduct some additional experiments to investigate some aspects of our solution.
4.1 Setup
Detection Methods: Apart from the methods that use discriminators for the detection, other models evaluated using MSP, OpenMax, MD and RMD as probability and distance based methods.
In-distribution Datasets: CIFAR-10 and CIFAR- 100 44 are used as the in-distribution datasets.
The images pixel values are normalized to be in the range of 0 to 1.
nut-of-distribution Datasets: Following the setting in earlier works 23 37 we use eight different datasets that are disjoint from the in-distribution sets, including MNIST (50 ImageNteeNet [5I],
Places365 (52) LSUN (53), iSUN (54), Birds (55). Flowers [56), and COIL-100 (57 as the OOD test
sets. The results are averaged over these datasets to peforrm a comprehensive evaluation on different OOD datasets. Also, the SVHN 58 dataset is used as the OOD validation set select the best discriminator during the training, and Food- 101 [59] is used as the open training set.
Baselines: Various methods have been considered in comparisons. The ViT architecture and OpenGAN are used as the SOTA methods in the standard OOD detection. AT and HAT are considered as the effective defenses in the image classification, and AOE, ALOE, and OSAD as effective defenses in the OOD detection. Note that we are the first work that uses HAT for the OOD detection. All the defenses are trained with 255 to have the best results against attack with this perturbation budget.
These baseline methods are trained based on the guidelines in their original work. It should be noted that OpenGAN evaluates the model in the training mode. This causes a kind of information leakage among the batch of images used to predict the OOD scores. Nevertheless, the evaluations of OpenGAN is done according to their guidelines. This problem has been solved in implementations for ATD method and ATD is evaluated in the test mode.
ATD Hy Hyperparameters: A simple DCGAN 660 is used for the generator and discriminator architec- ture in the ATD. Furthermore, ATD is trained for 20 epochs with a = 0.5 using Adam 61] optimizer with learning rate of le- 4. Details of the ATD method is available in section 3.1
7

---

Table OOD detection AUROC under attack with 255 for various methods trained with CIFAR- 10 or CIFAR- 100 as the closed set. A clean evaluation is one where no attack is made on the data, whereas an in/out evaluation means that the corresponding data is attacked. The best and second-best results are distinguished with bold and underlined text for each column.
In- Di-Ditriibutin Dataset
Method
CIFAR-10
CIFAR-100
Clean
In
Out
In and Out
Clean
In
Out
In and Out
OpenGAN-fea
0.971
0.473
0.425
0.266
0.958
0.198
0.324
0.088
OpenGAN-pixe
0.818
0.000
0.008
0.000
0.767
0.000
0.004
0.000
VIT (MSP)
0.975
0.448
0.172
0.002
0.879
0.269
0.129
0.002
VIT (MD)
0.995
0.136
0.495
0.000
0.951
0.053
0.279
0.000
VIT (RMD)
0.951
0.427
0.446
0.025
0.915
0.365
0.361
0.037
ViT (OpenMax)
0.984
0.346
0.291
0.004
0.907
0.086
0.166
0.001
AT (MSP)
0.735
0.462
0.442
0.174
0.603
0.324
0.250
0.085
(MD)
0.771
0.429
0.527
0.232
0.649
0.278
0.357
0.108
AT (RMD)
0.836
0.436
0.523
0.151
0.700
0.366
0.363
0.136
(OpenMax)
0.805
0.468
0.508
0.208
0.650
0.319
0.350
0.132
HAT (MSP)
0.770
0.560
0.548
0.325
0.612
0.393
0.335
0.176
HAT (MD)
0.789
0.572
0.586
0.369
0.810
0.587
0.603
0.363
HAT (RMD)
0.878
0.602
0.606
0.258
0.730
0.443
0,416
0.191
HAT (OpenMax)
0.821
0.613
0.648
0.415
0.703
0.462
0.462
0.263
OSAD (MSP)
0.698
0.411
0.407
0.154
0.557
0.285
0.194
0.055
OSAD (MD)
0.626
0.375
0.432
0.231
0.615
0.368
0.416
0.216
OSAD (RMD)
0.776
0.421
0.456
0.123
0.680
0.369
0.353
0.140
OSAD (OpenMax)
0.827
0.544
0.554
0.251
0.647
0.325
0.330
0.123
AOE (MSP)
0.780
0.544
0.527
0.285
0.566
0.332
0.324
0.157
AOE (MD)
0.709
0.361
0.484
0.215
0.743
0.406
0.539
0,255
AOE (RMD)
0.780
0.382
0.421
0.075
0.682
0.355
0.313
0.121
AOE (OpenMax)
0.797
0.528
0.586
0.298
0.591
0.282
0.356
0.143
ALOB (MSP)
0.843
0.664
0.538
0.287
0.701
0.438
0.317
0.127
ALOE (MD)
0.827
0.369
0.479
0.132
0.793
0.516
0.543
0.264
ALOE (RMD)
0.815
0.293
0.364
0,022
0.632
0.275
0.283
0.078
ALOE (OpenMax)
0.868
0.584
0.606
0.261
0.731
0.399
0.389
0.125
ATD (Ours)
0.943
0.837
0.862
0.693
0.877
0.734
0.739
0.553
All the models are evaluated against an end-to-end PGD attack with 205 For the baseline methods, we only use a 10-step attack, but ATI is evaluated with 100 steps to ensure its robustness. We show that this perturbation budget does not change the images semantically and 100 steps is more than enough to ensure the robustness of ATD in Appendix Cland and E Also, the attack is performed with a single random restart and the random initialization in the range (-e, Moreover, the attack step size is selected as 2.5 N
Evaluation Attack:
Evaluation Metric: We use AUROC
well-known classification criterion. The AUROC value is in the range [0, 1], and the closer it is to 1, the better the classifier performance.
4.2 Results
OOD detection under adversarial attack: To perform a comprehensive study, AUROC is computed in four different settings for each method. First, the standard OOD detection without any attack is conducted (Clean). Next, either in- or out-datasets are attacked (In/Out). Finally, both the in- and out-sets are attacked (In and Out). Note that resistance against the attack to both in- and out-sets is much harder than other cases since the perturbation budget has effectively been doubled. Based on the results in Table I VDT+MD and OpenGAN-fea have the best performance in the standard OOD

---

Table 2: ATD is trained on CIFAR-10 and is evaluated with the transferred attack on in/out data generated by baseline methods (columns). Results show that ATD sufficiently robust under the transferred black-box attacks.
Attack from OpenGAN-fea VIT (MD)
AT (MD)
HAT (OM)
OSAD (OM)
AOE (OM)
ALOE (OM)
0.930
0,927
0.923
0.894
0,907
0,907
0,921
Out In and Out
0,940
0.940
0.933
0.914
0.927
0,925
0,933
0.928
0.925
0.920
0.865
0.895
0.892
0.917
Table 3: Ablation study our method. Other choices for the feature extractor, training method, and generated data attacking are tested on CIFAR-10, 10, but none them is as effective as the setting used in the ATD.
Config
Attack
Feature Extractor
Adversarial Train
Attack Generator
Clean
In
Out
In and Out
Standard Trained
0.706
0.263
0.147
0.029
HAT
X
0.947
0.741
0.720
0.511
Not Used
V
0.916
0.776
0.758
0.546
Not Used
0.923
0.789
0.771
0.560
HAT
X
0.943
0.837
0.862
0.693
detection, but they completely fail under the adversarial setting. Early defenses improved adversarial performance the cost of decrease in the clean detection rate. Among them, HAT+OpenMax and HAT+MD are the most effective defenses in CIFAR-10 and CIFAR-100, respectively. Still, they are not as effective as our method ATD, which significantly outperforms them in both datasets while preserving the clean detection performance.
Black-box evaluation: To evaluate ATD under the black-box setting, we generate adversarial perturbations by attacking the introduced baselines as the source models and transferring them to the ATD as the target model. This test can also be considered obfuscation in the model [62]. Since the single-step attack enjoys better transferability than the multi-step ones [11),
sanity check for detecting gradient
FGSM for attacking the source models. Based on the results in Table 2 ATD sufficiently robust against all the transferred attacks.
Ablation study: ATD uses HAT as the feature extractor and performs adversarial training on open and closed sets to robustify the discriminator. As an ablation study using CIFAR-10 as the in-distribution data, we replace HAT with a standard trained model to check its effectiveness. Also, in another experiment, we replace the discriminator adversarial training with the standard training. The results in Table 3 demonstrate that these settings are not as effective as ATD in achieving robust detection model. As another ablation, we consider removing the feature extractor and generating images instead of features. Based the results, the trained discriminator with this method is also not robust as the ATD. Moreover, we check the effect of attacking the generated images, which leads to a lower AUROC score. This confirms that attacking generated data is not helpful in the training.
Classification attack instead of end-to-end attack: OSAD method is evaluated against an attack on the classification, and not the OOD detection, in their original work. Here, we show that this is not strong attack on the detection method. This is done by checking whether an attack on classification or detection method is effective on the other or not. Results in Table for the OSAD and ATD methods demonstrate that attacking the classification and detection is significantly more effective on their self than the other. Thus, end-to-end attack is a better basis for the evaluation of the classification or detection defenses.
Comparison with ATOM: Our baselines include all the previous methods that consider robustness on both in- and of-distibbution data to make a fair and comprehensive evaluation of our methods. data,
ATOM method [19] is defense that considers robustness only on the out-of-distributie Therefore, this method would not be robust against attacks on the in-distribution data. In addition, we have compared our results against ALOE,, which can be regarded as an extension of ATOM that accounts for both in- and out-of-distribution robustness. Still, comparison with the ATOM method is performed in the Table 5 using PGD-100 with 253
8 that supports our arguments.

---

Table 4: Classification accuracy and OOD detection AUROC under attack to classification and detection methods for OSAD and ATD methods trained on CIFAR-10 dataset. end-to-end attack is more effective in both classification and detection.
OSAD
ATD (Ours)
Evaluation
Attack Classification Attack Detection
Attack Classification Attack Detection
Classification Accuracy
0.419
0.777
0.622
0.847
Detection AUROC
0.813
0.544
0.918
0.837
Table 5: Comparison of AUROC score for OOD detection with ATD and ATOM methods under PGD-100 attack with e = 255 CIFAR-10 and CIFAR-I00 datasets are used as the in-distribution dataset. A clean evaluation means that the corresponding data is attacked.
where attack is made the data, whereas in/out evaluation
In- Disttiiiuutio Dataset
CIFAR-10
CIFAR-100
Method
Clean
In
Out
In and Out
Clean
In
Out
In and Out
ATOM
0.983
056
0.447
0.067
0.925
0.089
0.728
0.085
ATD (Ours)
0.943
0.837
0.862
0.693
0.877
0.734
0.739
0.553
Conclusion
The existing OOD detection methods far from being robust against strong attacks, contrary to what has been claimed previously for methods such as ALOE and OSAD. To mitigate this issue, we proposed ATD that uses an adversarially trained discriminator to classify in and samples. Moreover, ATD utilizes HAT to extract robust features from the real samples, and a generator to craft broad range of OOD data features. This method could significantly outperform earlier methods against an end-to-end PGD attack. Also, it is sufficiently robust against black-box attacks. The primary advantage of ATD is that it preserves the standard OOD detection AUROC along with the robustness, which is needed in real-world situations.
Broader Impact
Detecting out-of-distribution inputs safety issue in many machine learning models. In spite of advances in this area, the existing models susceptible adversarial examples another important safety problem. This work aims at overcoming this issue by detecting OOD inputs even when they have been perturbed by adversarial threat models. We believe that this is important in most machine learning systems with safety-critical concerns. For instance, it is required to diagnose an if
unseen disease in health care systems an adversary has perturbed their inputs. Therefore, this work can benefit a wide range of machine learning researchers. Also, do expect efforts to have any negative consequences.
detect anomalous patterns in financial services
Acknowledgments
We thank Mahdi Amiri, Hossein Mirzaei, Zeinab Golgooni, and the anonymous reviewers for their helpful discussions and feedbacks on this work.
10

---

References
[1] Igbal H. Sarker. Machine learning: Algorithms, real-world applications and research directions. SN Computer Science, 2(3):1-21, 2021.
[2] Zhong- Qiu Zhao, Peng Zheng, Shou-Tao Xu, and Xindong Wu. Object detection with deep learning: review. IEEE Transactions on Neural Networks and Learning Systems, 30(1 10(11) 11:111232222222 3232, 2019.
Xiumiigg Zhang, Zhoutong Zhang, Chengkai Zhang. Josh Tenenbaum, Bill Freeman, and Jiajun Wu. Learning to reconstruct shapes from unseen classes. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 31.
Curran Associates, Inc., 2018.
[4] Mohammadreza Salehi, Atrin Arya, Barbod Pajoum, Mohammad Otoofi, Amirreza Shaeiri, Moham- mad Hosscin Rohban, and Hamid R Rabice. Arac: Adversarially robust training of autoencoders improves novelty detection. Neural Networks, 144:726-736, 2021.
[5] Weitang Liu, Xiaoyun Wang. John Owens, and Yixuan Li. Energy-based out-of -i--of--ittiiuttonn detection. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21464-21475. Curran Associates, Inc., 2020.
[6] Da-Wei Zhou, Han-Jia Jia Ye, and De-Chuan Zhan. Learning placcholders for open-set recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2021.
Mohammadreza Salehi, Hossein Mirzaci, Dan Hendrycks, Yixuan Li, Mohammad Hossein Rohban, and Mohammad Sabokrou. unified survey on anomaly, novelty, open-set, and uuttof-distribution detection: Solutions and future challenges. arXiv preprint arXiv:2/ 10. 2021.
[8] Yifei Ming. Ying Fan, and Yixuan Li. Poem: Out-of -idstiibiitiitoo detection with posterior sampling. In International Conference on Machine Learning, pages 1.660-156655. PMLR, 2022.
[9] Hongjie Zhang, Ang Li, Jie Guo, and Yanwen Guo. Hybrid models for open set recognition. In European Conference on Computer Vision, pages (02-117. Springer, 2020.
[10] Guangyao Chen, Limeng Qiao, Yemin Shi, Peixi Peng. Jia Li, Tiejun Huang, Shiliang Pu, and Yonghong Tian. Learning open set network with discriminative reciprocal points. In European Conference on Computer Vision, pages 507-522. Springer, 2020.
[11] Sagar Vaze, Kai Han. Andrea Vedaldi, and Andrew Zisserman. Open-set recognition: A good closed-set classifier is all you need. In International Conference on Learning Representations, 2022.
[12] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing propertics of neural networks. In International Conference Learning Representations, 2014.
[13] lan Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In Intermational Conference Learning Representations, 2015.
[14] Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. Access, 6:14410 14430, 2018.
[15] Aleksander Madry, Aleksandar Makelov. Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
[16] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent Ghaoui, and Michael Jordan. Theo- retically principled trade-off between robustness and accuracy. In International conference on machine learning, pages 7472-7482, PMLR, 2019.
[17] Eric Wong, Leslic Rice, and J. Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020.
[18] Mohammad Azizmalayeri and Mohammad Hossein Rohban. Lagrangian objective function leads to
improved unforcseen attack generalization in adversarial training. arXiv preprint arXiv:2 15385, 2021.
[19] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying -iittii-ittitttionn detection using outlier mining. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 430 445, Springer, 2021.

---

[20] Stanislav Fort. Adversarial vulnerability of powerful near out-of-distribution detection. arXiv preprint urXiv:2201.00012, 2022.
[21] Maximilian Augustin, Alexander Meinke, and Matchias Hein. Adversarial robustness on in-and out- distribution improves explainability. In European Conference on Computer Vision, pages 228-245. Springer, 2020.
[22] Matthias Hein, Maksym Andriushchenko, and Julian Bitterwolf. Why relu networks yield high high-confidence predictions far away from the training data and how mitigate the problem. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 41 50, 2019.
[23] Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Robust -ittr--iistituttoon detection for neural networks. In The AAAI-22 Workshop on Adversarial Machine Learning and Beyond, 2022.
[24] Rui Shao, Pramuditha Perera, Pong Yuen, and Vishal M Patel. Open-ser set adversarial defense. European Conference on Computer Vision, pages 682-698. Springer, 2020.
[25] Rui Shao, Pramuditha Perera, Pong C Yuen, and Vishal M Patel. Open-set adversarial defense with clean-adversarial mutual learning. International Journal of Computer Vision, 130(4):0070-108, 2022.
[26] Thomas Schlegl, Philipp Sechöck, Schastian M Waldstein, Ursula Schmidt- and Georg Langs. Unsupervised anomaly detection with generative adversarial networks to guide marker discovery. In International conference on information processing in medical imaging, pages 146-157. Springer, 2017.
[27] Yezheng Liu, Zhe Li, Chong Zhou, Yuanchun Jiang, Jianshan Sun, Meng Wang, and Xiangnan He. Generative adversarial active learning for unsupervised outlier detection. IEEE Transactions on Knowledge and Data Engineering, 1517)11517-1288 2019.
[28] Shu Kong and Deva Ramanan. OpenGAN:
Open-set recognition via open data generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 813-822, 2021.
[29] Dan Hendrycks and Kevin Cimpel. A baseline for detecting misclassified and outtof-distribution examples in neural networks. Pruceedings of International Conference on Learning Representations, 2017.
[30] Shiyu Liang. Yixuan Li, and R. Srikant. Enhancing the reliability of cat-of-distributtio image detection in neural networks. In International Conference on Learning Representations, 2018.
[31] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Hcigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[32] Mohammad Azizmalayeri and Mohammad Hossein Rohban. OOD augmentation may be at odds with open-set recognition. arXiv preprint arXiv:2206.04244 2022.
[33] Abhijit Bendale and Terrance E Boult. Towards open deep networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1563-1572, 2016.
[34] Walter J Scheirer, Anderson Rocha, Ross J Michcals, and Terrance E Boult. Meta-rocognition: The theory and practice of recognition score analysis. IEEE transactions on pattern analysis and machine intelligence, 3389)168991165, 2011.
[35] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out- of-distribution samples and adversarial attacks. Advances neural information processing systems, 31, 2018.
[36] Ren, Stanislav Fort, Jeremiah Liu, Abhijit Guha Roy, Shreyas Padhy, and Balaji Lakshminarayanan. simple fix to mahalanobis distance for improving near-ood detection. arXiv preprint wrXiv:2106.09002 2021.
[37] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. In International Conference on Learning Representations, 2019.
[38] Zongyuan Ge, Sergey Demyanov, and Rahil Garnavi. Generative openmax for multi-class open set classification. In British Machine Vision Conference (BMVC). 2017.
[39] Lawrence Neal, Matthew Olson, Xiaoli Femn Weng- Keen Wong, and Fuxin Li. Open set learning with counterfactual images. In Proceedings of the European Conference on Computer Vision (ECCV), pages 613-628, 2018
12

---

[40] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calibeee classifiers for detecting out-of-distribution samples. In International Conference on Learning Representations, 2018.
[41] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. In Artificial intelligence safety and security, pages 99 112. Chapman and Hall/CRC, 2018.
[42] Seyed-Mohsen Moosavi- -ezzoo,,,,,,, Alhussein Fawzi, and Pascal Frossard. Deepfool: simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2574-2582, 2016.
[43] Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp), pages 59. IEEE, 2017.
[44] Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficient black-box adversarial attack via random search. In European Conference Computer Vision, pages 184-501. Springer, 2020.
[45] Dimitris Tsipras, Shihani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robust- ness may be at odds with accuracy. International Conference on Learning Representations, 2019.
[46] Rahul Rade and Seyed-Mohsen Moosavi-Dezfooli. Reducing excessive margin achieve a better accuracy vs. robustness trade-off. In International Conference on Learning Representations, 2022.
[47] Xuanqing Liu and Cho-Jui Hsich. Rob-gan: Generator, discriminator, and adversarial attacker. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11234- 11243, 2019.
[48] Desheng Wang. Weidong Jin, Yunpu Wu, and Aamir Khan. Improving global adversarial robustness generalization with adversarially trained gan. arXiv preprint arXiv:2103.cm 04513,,,33,3 2021.
[49] Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
[50] Yann LeCun and Corinna Cortes. Mnist handwritten digit database. http:://yane lecun. wwt.co/.cc/ce/ee exdb/ mnist/ 2010.
[51] Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEFE conference on computer vision and pattern recognition, pages 248-255 Iece, 2009.
[52] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition, IEEE transactions pattern analysis and machine intelligence, 40(6): 1452-1464, 2017.
[53] Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Con- struction large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv: 1506.03365. 2015.
[54] Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R Kulkarni, and Jianxiong Xiao. Turkergaze: Crowdsourcing saliency with webcam based eye tracking. arXiv preprint arXiv:1504.0000 150.006555 2015.
[55] Peter Welinder, Steve Branson, Takeshi Mita, Catherine Wah, Florian Schroff, Serge Belongie, and Pietro Perona. Caltech-uesd birds 200. 2010.
[56] Maria-Eles Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics Image Processing, pages 722-729. IEEF, 2008.
[57] Sameer A Nene, Shree K Nayar, Hiroshi Murase, et al. Columbia object image library (coil-100), 1996.
[58] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. 2011.
[59] Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. food-101-mining discriminative components with random forests. In European conference computer vision, pages 446-461. Sprigee, 2014.
[60] Alec Radford, Luke Metz. and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 1.X64:151,.06444, 2015.
13

---

[61] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint urXiv:1412.6980. 2014.
[62] Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses adversarial examples. In International conference on machine learning, pages 283. PMLR, 2018.
[63] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. In Thirty-fitth Conference on Neural Information Processing Systems Datasets and Benchmarks
Track, 2021.
[64] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, 2020.
[65] Jeremy Cohen, Elan Rosenfeld, and Zico Kolter. Certified adversarial robustness via randomized smoothing.
In International Conference on Machine Learning. pages 1310-155 10-13200. PMLR, 2019.
[66] Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint 1rXiv:1810. 12715, 2018.
[67] Julian Bitterwolf, Alexander Meinke, and Matthias Hein. Certifiably adversarially robust detection of out-of-distribution data, Advances in Neural Information Processing Systems, 33:16085 13:16085-16009 16095, 2020.
[68] Alexander Meinke, Julian Bitterwolf, and Matthias Hein. Provably robust detection of out-of of-distribution data (almost) for free. arXiv preprint arXiv:2106.0422 2021.
[69] Hossein Mirzaei, Mohammadreza Salehi, Sajjad Shahabi, Efstratios Gavves, Cees GM Snoek, Mohammad Sabokrou, and Mohammad Hossein Rohban. Fake it till you make it: Near-distribution novelty detection by score-based generative models. arXiv preprint arXiv:2005 14297, 2022.
[70] Stanislav Fort, Jie Ren, and Balaji Lakshminarayanan. Exploring the limits of out- out-of-distribution detection. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances Neural Information Processing Systems, 2021.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen- tal material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4.1
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No] Error bars were too small have any visual impact. (d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] We have used single RTX 2060 Super GPU.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you eite the creators? [Yes]
(b) Did you mention the license of the assets? [No] Data are publicly available, which have been cited.
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you's 'rure using/curating? [N/A]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "jfat5lplidV", "reviewer": "Reviewer_1FKs", "summary": "This paper analyzes the robustness of outlier detection, and proposes to adversarially train the discriminator in order to get better robustness. Experiments demonstrate the effectiveness of the method", "questions": "The Atom paper is cited as \"Similar to other learning domains, OOD detection methods also suffer from the\n44 existence of adversarial examples. A small perturbation can cause a sample from the closed set to be\n45 classified as anomaly and vise-versa [17, 18].\"; so it seems that we only talk about the attack part of that paper, is there a particular reason why we don't consider the defense there? (in particular, that paper consider adversarial training of the discriminator)", "limitations": "As I mentioned above, as far as I understand, the idea in this paper is not new and has already been explored in previous work (not one work)", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "I will start with the most important weakness I saw: The idea of adversarially training the discriminator has already been explored in:\n\nJiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Atom: Robustifying out-of-distribution\n337 detection using outlier mining. In Joint European Conference on Machine Learning and Knowledge\n338 Discovery in Databases, pages 430–445. Springer, 2021.\n\nAnd also the recent ICML paper (pushing this even further):\n\nhttps://arxiv.org/abs/2206.13687 (ICML 2022 (Long Talk))\n\nThis paper cited the first paper, but never did comparison in experiments (not to mention the second paper), which I find to be a main weakness.\n\nOn the strengths side, this paper does a good job in explaining the ideas and the results are good\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "h-r4DRt73Jm", "reviewer": "Reviewer_2bsy", "summary": "This paper presents an OOD detection method that combines robust feature extraction, OOD feature generation and adversarial training. This Adversarially Trained Discriminator aims to provide a robust OOD detection result that can adapt to larger perturbation sizes. The experiment is conducted on different detector-extractor pairs with respect to applying attack on different data, and the result shows the proposed method consistently outperforms the SoTA methods.\n", "questions": "> Since It’s a key point for the performance improvement, there should be more detailed description about how to use HAT to extract features.\n\n> I don't see any analysis for the different OOD types. Ad the authors stated in Sec. 5, “Moreover, ATD utilizes HAT to extract robust features from the real samples, and a generator to craft a broad range of OOD data features”. However, there is no experiment setup relating to different OOD types, which may not prove ATD being robust enough.", "limitations": "As the authors have mentioned, this method significantly outperforms other SOTA OOD detection methods when end-to-end PGD attack is applied. However, further justification should be applied to evaluate the effectiveness/contribution of each detector-extractor pair on different OoD types.\n", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "++ This paper specifically pays attention to generating OOD samples and applying Adversarial Training on it to improve the OOD detection performance.\n\n++The paper is well written and easy to follow.\n\n-- Table 3 is not self-contained. It does not mention which dataset are used.\n\n-- In the setup of the experiments, the In-Distribution datasets only contain CIFAR-10 and CIFAR-100 but do not set any combinations with the other data sets like these shown in Out-of-Distribution datasets. Experiments with the different combinations of the ID / OOD dataset could be more convincing.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bvA3MPKFOgH", "reviewer": "Reviewer_47ku", "summary": "he authors of this paper claim to be showing that the current state-of-the art models for the adversarially robust detection of out-of-distribution samples are not effective when evaluated on sufficiently strong threat models. They then go on to propose a method that they claim remedies this issue by incorporating GAN training and manages to simultaneously achieve high clean OOD detection performance, adversarially robust OOD detection and high accuracy. ", "questions": "Minor comments:\n- They refer to OOD detectors that use the softmax score as \"likelihood\"-based. Likelihood-based methods generally refer to models that fit the density of the data distribution $p(x)$.\n- The authors forgot to delete the instructions part of the checklist.", "limitations": "I don't think the authors adequately discuss limitations of their approach. For example, they do not acknowledge that similar to AT, their model's accuracy is significantly reduced. They should also openly discuss how computationally expensive their method is.\nI agree with the authors' self-assessment in the checklist that their work is unlikely to have negative ethical implications.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper generally tackles an interesting problem of adversarially attacking out-of-distribution samples and they choose a fairly standard threat model. Unfortunately, the novelty of their approach is quite limited because they omit a lot of prior benchmarks. For example, they write \"The first [approach] uses some OOD data, uniformly sampled from the open set classes and disjoint from the closed set, and employs adversarial training to encourage the model to assign a uniform label, as opposed to one-hot labels, to anomalous samples even under adversarial perturbations [19].\" This describes the method ACET [Hein19], but instead they cite ATOM [19] (which uses an out-class) and which they nonetheless don't compare to for some reason. Both methods should be used as baselines. They furthermore use ALOE as a baseline which according to Eq. 7 is identical to the work RATIO [Augustin20] which should at the very least be cited in this context. On top of this, there even exist several works that tackle the issue of unreliable evaluations of adversarial OOD robustness by providing provable guarantees [Bitterwolf20, Meinke21]. Clearly, these are relevant baselines as well and should at the very least be cited. Additionally, even the idea of using GANs in order to produce adversarial OOD samples has previously been proposed in the well-known paper by [Lee18].\n\nThe authors point out that prior works supposedly only studied \"small perturbation sizes (e.g. $\\epsilon=\\frac{1}{255}$ in the CIFAR-10 dataset)\". It probably makes sense to explicitly cite exactly which papers they are referring to here. A similar comment applies to a statement such as \"The existing OOD detection methods are far from being robust against strong attacks, contrary to what has been claimed previously.\"\n\nThe paper uses 8 out-distributions for each in-distribution which is good. Unfortunately, they do not list the individual results for each out-distribution which makes it very hard to check or compare their results. These need to be listed individually in the appendix. Also, some of the out-distributions clearly overlap with the classes of the in-distributions, for example birds and the bird class from CIFAR10. This needs to be discussed somewhere in the paper, or alternatively different out-distributions should be used.\n\nThe authors' method seems to work without relying on a difficult training out-distribution. While they do use SVHN for this, it is quite impressive that their training apparently generalizes to out-distributions that are very different from this. \n\nThe authors should provide slightly more details on their evaluation attack, i.e. initialization, restarts and step size, potentially in the appendix. \n\nOverall, however, the paper's results seem encouraging. If their model does not get broken by a stronger attack, the fact that they attain high adversarially robust OOD detection performance and clean OOD detection performance at the same time is significant.\nThey attempt to account for potential gradient obfuscation in the paper via transfer attacks which is good.\nThey also provide both their code as well as pre-trained models which is also good.\n\nThe authors' claim that their method achieves \"high accuracy\" seems misleading. The backbone feature extractor that they use has a clean accuracy of 91.47% on CIFAR10, which of course is far from \"high\" compared to SOTA CIFAR10 models. As far as I can tell, this is not pointed out anywhere in the paper. \n\nI believe the paper cannot be accepted in its current form, because it a) does not individually list the performance with respect to different out-distributions and b) does not adequately discuss how its method relates to and compares to prior work that is substantially similar. If these points are adequately addressed I would be willing to raise my score, because the results are indeed promising.\n\n[Hein19] Why relu networks yield high-confidence predictions far away from the training data and how to mitigate the problem, Hein, Andriushchenko, Bitterwolf, @CVPR'19\n\n[Augustin20] Adversarial robustness on in-and out-distribution improves explainability, Augustin, Meinke, Hein @ECCV'20\n\n[Bitterwolf20] Certifiably adversarially robust detection of out-of-distribution data, Bitterwolf, Meinke, Hein @NeurIPS'20\n\n[Lee18] Training confidence-calibrated classifiers for detecting out-of-distribution samples, Lee, Lee, Lee, Shin, @ICLR'18\n\n[Meinke21] Provably Robust Detection of Out-of-distribution Data (almost) for free, Meinke, Bitterwolf, Hein @UDL workshop, ICML'21", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "KERmoDgNFh", "reviewer": "Reviewer_enTM", "summary": "This paper uncovers the vulnerability of the existing OOD detection methods to adversarial inputs. It then goes on to introduce a robust OOD detection method called Adversarially Trained Discriminator (ADT). ADT combines adversarial training with generative models to improve the detection robustness. In particular, the GAN model in this approach allows generating OOD samples beyond the available open sets during training and therefore boosts the performance of the detection method as a whole. The idea is simple and makes use of recent related work including HAT. While none of the components of the proposed model are novel, the particular system is a novel combination of these components. Results show substantial improvement in robustness beyond other baselines considered. \n\n\nPOST-REBUTTAL UPDATE: \nI thank the authors for responding to my comments and appreciate the new experiments in support of their arguments. I am increasing my score by 1 to recommend acceptance. ", "questions": "The text was clear and I do not have any additional questions. ", "limitations": "While CIFAR10 and CIFAR100 are good starting points, many of the past methods have been shown to not scale well to larger, more sophisticated datasets. It would great to see some experiments on how this model would scale to other datasets like tinyimagenet. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**strengths** \n- The method is simple and intuitively makes sense. \n- The text is well written and a helpful toy example is included to help the reader better understand the effect of the different components of the model. \n- The results section involves comparison across many different baselines. \n\n**weaknesses** \nAdversarial robustness has been a constantly evolving field in the past years and the frequent breakage of SOTA defenses by new adversarial attacks has made everyone working in this field very cautious about new proposed defenses. Much of recent work in the AT field points to the insufficiency of PGD as a reliable attack method and many defense methods in recent years have successfully \"fooled\" this attack without providing a \"robust\" form of robustness. AutoAttack has become a more reliable substitute to PGD and have proven to be much more difficult to be fooled. In the light of this, I would like to see how the proposed method stands against this stronger attack to be more sure about the reliability of the present results. \n\ntypo: line 45 vise-versa \n\nFig2. OpenMax score is not defined which makes it hard to judge what this figure is telling. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept