RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Out-of-Distribution Detection
Probabilities: A simple but effective method for anomaly detection is the Maximum Softmax Probability (MSP) hendrycks17baseline ; liang2018enhancing . This method is applied on a K -class classifier and returns \max_{c\in\{1,2,\ldots,k\}}f_{c}(x) as the closed-set membership score of the sample x . Currently, it is shown that using the MSP with Vision Transformers (ViT) dosovitskiy2021an leads to SOTA results in cross-dataset open set recognition vaze2022openset ; azizmalayeri2022ood .
Also, OpenMax bendale2016towards replaces the softmax layer with a layer that calibrates the logits by fitting a class-wise probability model such as the Weibull distribution scheirer2011meta .

Distances: The anomaly sample can be detected by its distance to the class conditional distributions. Mahalanobis distance (MD) lee2018simple and Relative
MD (RMD) ren2021simple are the main methods in this regard. For an in-distribution with K classes, these methods fit a class conditional Gaussian distribution \mathcal{N}(\mu_{k},\,\Sigma) to the pre-logit features z . The mean vector and covariance matrix are calculated as:

\mu_{k}=\frac{1}{N}\sum_{i:y_{i}=k}z_{i},\quad\Sigma=\frac{1}{N}\sum_{k=1}^{K}\sum_{i:y_{i}=k}(z_{i}-\mu_{k})(z_{i}-\mu_{k})^{T},\quad k=1,2,...,K

In addition, to use RMD, one has to fit a \mathcal{N}(\mu_{0},\,\Sigma_{0}) to the whole in-distribution. Next, the distances and anomaly score for the input x^{\prime} with pre-logits z^{\prime} are computed as:

MD_{k}(z^{\prime})=(z^{\prime}-\mu_{k})^{T}\Sigma^{-1}(z^{\prime}-\mu_{k}),\quad RMD_{k}(z^{\prime})=MD_{k}(z^{\prime})-MD_{0}(z^{\prime}),\quad (2)

score_{MD}(x^{\prime})=-\min_{k}\{MD_{k}(z^{\prime})\},\quad score_{RMD}(x^{\prime})=-\min_{k}\{RMD_{k}(z^{\prime})\}

Discriminators: Previous categories define the score function based on a K -way classifier, but one can directly train a binary discriminator for this purpose. Outlier Exposure (OE) hendrycks2018deep exploits some outlier data to learn a binary discriminator for the open set discrimination. GOpenMax Ge2017 , OSRCI neal2018open , and Confident Classifier lee2018training augment the closed set with fake images through a GAN-based generator. Also, OpenGAN kong2021opengan selects the best discriminator model with an open validation set during training due to unstable training of GANs.
Adversarial Attacks
For an input x with the ground-truth label y , an adversarial example x^{*} is crafted by adding small noise to x such that the predictor model loss J(x^{*},y) is maximized. The \ell_{p} norm of the adversarial noise should be less than a specified value \epsilon , i.e. \|x-x^{*}\|_{p}\leq\epsilon , to ensure that the image does not change semantically. Fast Gradient Sign Method (FGSM) Goodfellow2015 maximizes the loss function with a single step toward the sign of the gradient of J(x,y) with respect to the x :

x^{*}=x+\epsilon\ .\ sign( abla_{x}J(x,y)), (4)

where the noise meets the \ell_{\infty} norm bound \epsilon . Moreover, this method can be conducted iteratively kurakin2018adversarial with a smaller step size \alpha :

x^{*}_{0}=x,\quad x^{*}_{t+1}=x^{*}_{t}+\alpha\ .\ sign( abla_{x}J(x^{*}_{t},y)), (5)

where the noise should be projected to the \ell_{\infty} -ball with \epsilon radius at each step, which is called Projected Gradient Descent (PGD) attack madry2018towards . There are a number of other attacks for evaluating robustness of the models moosavi2016deepfool ; carlini2017towards ; andriushchenko2020square , but PGD is regarded as a standard and powerful attack.
Adversarial Defenses
The most effective defenses for image classification and OOD detection are variants of adversarial training, which are described in the following.

Adversarial Training (AT) madry2018towards : Through the use of adversarial examples during training, the model can learn robust features to withstand adversarial perturbations. This method is called AT and can be used to optimize model f_{\theta} as:

\operatorname*{arg\,min}_{\theta}\ \mathbb{E}_{(x,y)\in D_{in}}\ \max_{\|x-x^{*}\|_{\infty}\leq\epsilon}J(x^{*},y;f_{\theta}), (6)

where the inner maximization can be approximated with PGD and the outer minimization with SGD. AT naturally reduces the standard accuracy, i.e. the accuracy on unperturbed samples tsipras2018robustness , which can harm the anomaly detection score vaze2022openset .
Therefore, we also propose Helper-Based Adversarial Training (HAT) rade2022reducing as a baseline that achieves a better trade-off between accuracy and robustness.

Adversarial Learning with inliner and
Outlier Exposure (ALOE) chen2022robust : AT lacks information about the outlier data. Therefore, it is only robust on the closed set and not on outliers. To mitigate this issue, ALOE includes some outliers in the AT similar to OE hendrycks2018deep . Outliers are used with uniform label \pazocal{U}_{K} and attacked during training to obtain a model that is robust on both in- and out-distribution. The objective function of ALOE is summarized as:

\operatorname*{arg\,min}_{\theta}\ \mathbb{E}_{(x,y)\in D_{in}}\ \max_{\|x-x^{*}\|_{\infty}\leq\epsilon}J(x^{*},y;f_{\theta})\ +\ \lambda\ .\ \mathbb{E}_{x\in D_{out}}\ \max_{\|x-x^{*}\|_{\infty}\leq\epsilon}J(x^{*},\ \pazocal{U}_{K};f_{\theta})

An identical objective function is also used in RATIO augustin2020adversarial with a \ell_{2} perturbation bound. Alternatively, one could consider using clean outliers samples in the training. In addition to ALOE, we evaluate this alternative method as a baseline, and refer to it as Adversarial OE (AOE).

Open-Set Adversarial Defense (OSAD) shao2020open ; shao2022open : Instead of including outlier data in the AT, OSAD tries to learn more semantic features in the training. To this end, OSAD adds dual-attentive denoising layers to the model architecture, and combines the AT loss with an auto-encoder loss, whose aim is to reconstruct the clean image from its adversarial version. A self-supervision loss is also added by applying transformations on the input image. They claim that this combination improves OSR robustness, but they did not evaluate the model against an end-to-end attack.
Method
An OOD detection method should be robust against adversarial perturbations that are added to the closed or open datasets. Despite previous efforts to provide robust OOD detection methods, the evaluations have yielded a false sense of robustness due to the weak evaluation attacks. In Fig. 2 , we have evaluated ALOE and OSAD methods using an end-to-end PGD attack with \epsilon=\frac{8}{255} . According to the results, both the in and out detection score distributions remarkably change with this attack. So, these methods are not sufficiently robust. In the following, we aim to provide a more robust solution by addressing the drawbacks of previous works. We then use a toy example to provide more insights into the problem and our solution.
Toy Example
Here, we provide an insightful visualization of the ATD using a 2D toy example. To this end, we represent open and closed sets with disjoint distributions in a 2D space. Also, some samples are selected randomly in this space as the generated data. Fig. 4(a) shows the distributions used for this purpose, which the blue, orange, and green samples are closed, open, and generated data, respectively. Next, we train a multi-layer feed-forward neural network as the discriminator to perform the binary classification of in/out on these data in different cases. It should be noted that the open and closed data are fixed during the training, but the generated data is resampled in every epoch to simulate ATD training. Finally, the discriminator output is displayed all over the space with a blue (in) and orange (out) background in each case.

In our analysis, different conditions are taken into account. First, standard training is conducted on the data without attacking any samples. The results in Fig. 4(b) show that the model has learned the classification decision boundaries very well. In each of the next three cases, one of the closed, open and generated sets is attacked during training with \epsilon=1 . Fig. 4(c) shows that attacking the closed set leads to a larger margin in the decision boundaries around them. Having this margin ensures robustness against attacks that seek to misclassify the in-distribution samples. Additionally, note that this margin is smaller in the right side that an open set is present, which shows the impact of using open sets during training. In Fig. 4(d) , only the open set is attacked during the training. This has caused a tighter decision boundaries around the closed set where the open set is present. Therefore, using attacked open set in the training can slightly improve the robustness on OOD data, but it can simultaneously reduce the robustness on the closed set. As a result, \alpha in Eq. 9 that controls the optimization weight of open set should be selected carefully. In contrast with the ATD method, we also attack the generated data to see how it affects the result. According to Fig 4(e) , this does not cause robustness in the OOD data in contrast to Fig 4(d) , which is another reason that ATD do not attack the generated data during training. It is noteworthy that the main cause for this observation may be the instability of the generated samples during the training, which is inevitable and makes the adversarial optimization goal infeasible.

Eventually, we attack both the open and closed sets during training as in ATD. The results in Fig. 4(f) show that the margin around the closed set increases in the directions that an open set is not present to provide robustness on both in and out samples, which can be considered as a mixture of results in Fig. 4(c) and 4(d) . Also, this plot demonstrates that the robustness of the open and closed sets may be at odds with each other.
Experiments
In this section, we perform extensive experiments to evaluate existing OOD detection methods, including the standard and adversarially trained ones, and our ATD method against an end-to-end PGD attack. To this end, we first give details about the setting of the experiments. Next, we compare all the methods, which shows that ATD significantly outperforms the other methods. Toward the end, we conduct some additional experiments to investigate some aspects of our solution.
Setup
Detection Methods: Apart from the methods that use discriminators for the detection, other models are evaluated using MSP, OpenMax, MD and RMD as probability and distance based methods.

In-distribution Datasets: CIFAR-10 and CIFAR-100 krizhevsky2009learning are used as the in-distribution datasets. The images pixel values are normalized to be in the range of 0 to 1.

Out-of-distribution Datasets: Following the setting in earlier works chen2022robust ; hendrycks2018deep , we use eight different datasets that are disjoint from the in-distribution sets, including MNIST MNIST , TinyImageNet deng2009imagenet , Places365 zhou2017places , LSUN yu2015lsun , iSUN xu2015turkergaze , Birds welinder2010caltech , Flowers nilsback2008automated , and COIL-100 nene1996columbia as the OOD test sets. The results are averaged over these datasets to peform a comprehensive evaluation on different OOD datasets. Also, the SVHN netzer2011reading dataset is used as the OOD validation set to select the best discriminator during the training, and Food-101 bossard2014food is used as the open training set.

Baselines: Various methods have been considered in our comparisons. The ViT architecture and OpenGAN are used as the SOTA methods in the standard OOD detection. AT and HAT are considered as the effective defenses in the image classification, and AOE, ALOE, and OSAD as effective defenses in the OOD detection. Note that we are the first work that uses HAT for the OOD detection. All the defenses are trained with \epsilon=\frac{8}{255} to have the best results against attack with this perturbation budget.

These baseline methods are trained based on the guidelines in their original work.
It should be noted that OpenGAN evaluates the model in the training mode. This causes a kind of information leakage among the batch of images used to predict the OOD scores. Nevertheless, the evaluations of OpenGAN is done according to their guidelines.
This problem has been solved in our implementations for ATD method and ATD is evaluated in the test mode.

ATD Hyperparameters: A simple DCGAN radford2015unsupervised is used for the generator and discriminator architecture in the ATD.
Furthermore, ATD is trained for 20 epochs with \alpha=0.5 using Adam kingma2014adam optimizer with learning rate of 1e-4 . Details of the ATD method is available in section 3.1 .

Evaluation Attack: All the models are evaluated against an end-to-end PGD attack with \epsilon=\frac{8}{255} . For the baseline methods, we only use a 10-step attack, but ATD is evaluated with 100 steps to ensure its robustness. We show that this perturbation budget does not change the images semantically and 100 steps is more than enough to ensure the robustness of ATD in Appendix C and E . Also, the attack is performed with a single random restart and the random initialization in the range (-\epsilon,\epsilon) . Moreover, the attack step size is selected as \alpha=2.5\times\frac{\epsilon}{N} .

Evaluation Metric: We use AUROC as a well-known classification criterion. The AUROC value is in the range [0,1] , and the closer it is to 1 , the better the classifier performance.
Conclusion
The existing OOD detection methods are far from being robust against strong attacks, contrary to what has been claimed previously for methods such as ALOE and OSAD. To mitigate this issue, we proposed ATD that uses an adversarially trained discriminator to classify in and out samples. Moreover, ATD utilizes HAT to extract robust features from the real samples, and a generator to craft a broad range of OOD data features. This method could significantly outperform earlier methods against an end-to-end PGD attack. Also, it is sufficiently robust against black-box attacks. The primary advantage of ATD is that it preserves the standard OOD detection AUROC along with the robustness, which is needed in real-world situations.
Broader Impact
Detecting out-of-distribution inputs is a safety issue in many machine learning models. In spite of advances in this area, the existing models are susceptible to adversarial examples as another important safety problem. This work aims at overcoming this issue by detecting OOD inputs even when they have been perturbed by adversarial threat models. We believe that this is important in most machine learning systems with safety-critical concerns. For instance, it is required to diagnose an unseen disease in health care systems or to detect anomalous patterns in financial services even if an adversary has perturbed their inputs. Therefore, this work can benefit a wide range of machine learning researchers. Also, we do not expect our efforts to have any negative consequences.
Acknowledgments
We thank Mahdi Amiri, Hossein Mirzaei, Zeinab Golgooni, and the anonymous reviewers for their helpful discussions and feedbacks on this work.
Figure 1: OOD detection scores for several models against the perturbation bound. The perturbations are designed using an end-to-end \ell_{\infty} PGD attack. CIFAR-10 is used as the in-distribution dataset. a) Only the in-distribution dataset is attacked in the evaluation. b) Only out-distribution datasets (e.g., MNIST, Places, etc.) are attacked in the evaluation. c) Both in- and out-distribution datasets are attacked. ATD (Our method) outperforms others by a significant margin.; OOD detection scores for several models against the perturbation bound. The perturbations are designed using an end-to-end \ell_{\infty} PGD attack. CIFAR-10 is used as the in-distribution dataset. a) Only the in-distribution dataset is attacked in the evaluation. b) Only out-distribution datasets (e.g., MNIST, Places, etc.) are attacked in the evaluation. c) Both in- and out-distribution datasets are attacked. ATD (Our method) outperforms others by a significant margin.
Figure 2: OOD score distribution shift after an end-to-end PGD attack with \epsilon=\frac{8}{255} using CIFAR-10 as the in-distribution set. The first and second row correspond to the ALOE and OSAD methods, respectively. In each row, the left column represents the standard score distributions of in- and out-distributions. In each of the next columns, one of the in or out sets is attacked. The plots show that the standard distribution scores have drastically changed after the attack.; OOD score distribution shift after an end-to-end PGD attack with \epsilon=\frac{8}{255} using CIFAR-10 as the in-distribution set. The first and second row correspond to the ALOE and OSAD methods, respectively. In each row, the left column represents the standard score distributions of in- and out-distributions. In each of the next columns, one of the in or out sets is attacked. The plots show that the standard distribution scores have drastically changed after the attack.
Figure 3: ATD schematic architecture. Generator, discriminator, and robust feature extractor are represented with yellow, pink, and green colors, respectively.; ATD schematic architecture. Generator, discriminator, and robust feature extractor are represented with yellow, pink, and green colors, respectively.
Figure 4: Two-dimensional example to illustrate the effect of ATD method. a) The in- and out-distributions samples that are considered in this example, which are represented with blue and orange colors, respecitvely. In and out samples are fixed, while the generated data colored in green is resampled at each epoch. b to f) Using these distributions, OOD classification has been trained with different attack settings and the classifier output is represented with blue (in) and orange (out) background all over the feature space. Above each plot, green ticks and red crosses indicate which data distributions are attacked.; Two-dimensional example to illustrate the effect of ATD method. a) The in- and out-distributions samples that are considered in this example, which are represented with blue and orange colors, respecitvely. In and out samples are fixed, while the generated data colored in green is resampled at each epoch. b to f) Using these distributions, OOD classification has been trained with different attack settings and the classifier output is represented with blue (in) and orange (out) background all over the feature space. Above each plot, green ticks and red crosses indicate which data distributions are attacked.
Figure 5: Images from the open and closed sets before and after the attack with \epsilon=\frac{8}{255} . HAT+MSP is used as the base model in attacking the images. The images are perceptually similar before and after the attack. Also, the OOD detection score for each image is displayed above it, which demonstrates the effectiveness of the attack.; Images from the open and closed sets before and after the attack with \epsilon=\frac{8}{255} . HAT+MSP is used as the base model in attacking the images. The images are perceptually similar before and after the attack. Also, the OOD detection score for each image is displayed above it, which demonstrates the effectiveness of the attack.
Figure 6: Effect of attacking generated data along with the in and out sets.; Effect of attacking generated data along with the in and out sets.
Figure 7: OOD detection with an adversarially trained model using MSP as the detection method. Each of the rectangular distributions are regarded as a separate class, and adversarial training is conducted on them. Next, OOD scores are extracted with the MSP method all over the space, and thresholded to achieve the in/out decision boundaries. The blue and orange backgrounds are regarded as the in- and out-distributions, respectively.; OOD detection with an adversarially trained model using MSP as the detection method. Each of the rectangular distributions are regarded as a separate class, and adversarial training is conducted on them. Next, OOD scores are extracted with the MSP method all over the space, and thresholded to achieve the in/out decision boundaries. The blue and orange backgrounds are regarded as the in- and out-distributions, respectively.
Figure 8: Effect of the number of PGD attack steps on the detection AUROC in HAT, OSAD, and ATD methods trained with CIFAR-10 as the closed set. Three attacking scenarios are considered: attacking only the in data, attacking only the out data, and attacking both in and out data. Attacks are conducted with 1, 5, 10, 25, 50, and 100 steps. The AUROC reaches a stable value after 50 steps for all the methods and scenarios.; Effect of the number of PGD attack steps on the detection AUROC in HAT, OSAD, and ATD methods trained with CIFAR-10 as the closed set. Three attacking scenarios are considered: attacking only the in data, attacking only the out data, and attacking both in and out data. Attacks are conducted with 1, 5, 10, 25, 50, and 100 steps. The AUROC reaches a stable value after 50 steps for all the methods and scenarios.
OOD detection AUROC under attack with \epsilon=\frac{8}{255} for various methods trained with CIFAR-10 or CIFAR-100 as the closed set. A clean evaluation is one where no attack is made on the data, whereas an in/out evaluation means that the corresponding data is attacked. The best and second-best results are distinguished with bold and underlined text for each column.
ATD is trained on CIFAR-10 and is evaluated with the transferred attack on in/out data generated by baseline methods (columns). Results show that ATD is sufficiently robust under the transferred black-box attacks.
Ablation study on our method. Other choices for the feature extractor, training method, and generated data attacking are tested on CIFAR-10, but none of them is as effective as the setting used in the ATD.
Classification accuracy and OOD detection AUROC under attack to classification and detection methods for OSAD and ATD methods trained on CIFAR-10 dataset. An end-to-end attack is more effective in both classification and detection.
Comparison of AUROC score for OOD detection with ATD and ATOM methods under PGD-100 attack with \epsilon=\frac{8}{255} . CIFAR-10 and CIFAR-100 datasets are used as the in-distribution dataset.
A clean evaluation is one where no attack is made on the data, whereas an in/out evaluation means that the corresponding data is attacked.
OOD detection AUROC under PGD attack with \epsilon=\frac{4}{255} for various methods trained with CIFAR-10 or CIFAR-100 as the closed set. A clean evaluation is one where no attack is made on the data, whereas an in/out evaluation means that the corresponding data is attacked. The best and second-best results are distinguished with bold and underlined text for each column.
Comparing OOD detection methods robustness. For each of MSP, MD, RMD, and OpenMax, the AUROC is averaged over six different baselines in Table 1 that use these detection methods. A clean evaluation is the one where no attack is made on the data, whereas an in/out evaluation means that the corresponding data is attacked.
Comparison of AUROC score for OOD detection with ATD method under PGD-100 attack and AutoAttack with \epsilon=\frac{8}{255} . CIFAR-10 and CIFAR-100 datasets are used as the in-distribution dataset. A clean evaluation is one where no attack is made on the data, whereas an in/out evaluation means that the corresponding data is attacked.
OOD detection AUROC under PGD attack with \epsilon=\frac{4}{255} for ViT, AT, HAT, ALOE, and ATD trained with TinyImageNet as the closed set. A clean evaluation is one where no attack is made on the data, whereas an in/out evaluation means that the corresponding data is attacked.
OOD detection AUROC for each of OOD datasets under PGD attack with \epsilon=\frac{8}{255} for various methods trained with CIFAR-10 as the closed set. A clean evaluation is one where no attack is made on the data, whereas an in/out evaluation means that the corresponding data is attacked.