\def\year{2022}\relax
\documentclass[letterpaper]{article} %
\usepackage{aaai22}  %
\usepackage{times}  %
\usepackage{helvet}  %
\usepackage{courier}  %
\usepackage[hyphens]{url}  %
\usepackage{graphicx} %
\urlstyle{rm} %
\def\UrlFont{\rm}  %
\usepackage{natbib}  %
\usepackage{caption} %
\DeclareCaptionStyle{ruled}{labelfont=normalfont,labelsep=colon,strut=off} %
\frenchspacing  %
\setlength{\pdfpagewidth}{8.5in}  %
\setlength{\pdfpageheight}{11in}  %
\usepackage{algorithm}
\usepackage{algorithmic}

\usepackage{newfloat}
\usepackage{listings}
\lstset{%
	basicstyle={\footnotesize\ttfamily},%
	numbers=left,numberstyle=\footnotesize,xleftmargin=2em,%
	aboveskip=0pt,belowskip=0pt,%
	showstringspaces=false,tabsize=2,breaklines=true}
\floatstyle{ruled}
\newfloat{listing}{tb}{lst}{}
\floatname{listing}{Listing}
\pdfinfo{
/Title (AAAI Press Formatting Instructions for Authors Using LaTeX -- A Guide)
/Author (AAAI Press Staff, Pater Patel Schneider, Sunil Issar, J. Scott Penberthy, George Ferguson, Hans Guesgen, Francisco Cruz, Marc Pujol-Gonzalez)
/TemplateVersion (2022.1)
}
\setcounter{secnumdepth}{0} %
\title{Robust Out-of-distribution Detection for Neural Networks}
\author {
    Jiefeng Chen, \textsuperscript{\rm 1}
	Yixuan Li, \textsuperscript{\rm 1}
	Xi Wu, \textsuperscript{\rm 2}
	Yingyu Liang, \textsuperscript{\rm 1}
	Somesh Jha \textsuperscript{\rm 1}
}
\affiliations {
    \textsuperscript{\rm 1} University of Wisconsin-Madison \\
    \textsuperscript{\rm 2} Google \\
    \{jiefeng; sharonli\}@cs.wisc.edu, wu.andrew.xi@gmail.com, \{yliang; jha\}@cs.wisc.edu
}
\usepackage{paper}

\newcommand\SL[1]{\textcolor{blue}{[Sharon: #1]}}
\newcommand\yingyu[1]{\textcolor{red}{[Yingyu: #1]}}

\begin{document}
\maketitle

\begin{abstract}
Detecting out-of-distribution (OOD) inputs is critical for safely deploying deep learning models in the real world. Existing approaches for detecting OOD examples work well when evaluated on benign in-distribution and OOD samples. However, in this paper, we show that existing detection mechanisms can be extremely brittle when evaluating on in-distribution and OOD inputs with minimal adversarial perturbations which don't change their semantics. Formally, we extensively study the problem of {\em Robust Out-of-Distribution Detection} on common OOD detection approaches, and show that state-of-the-art OOD detectors can be easily fooled by adding small perturbations to the in-distribution and OOD inputs. To counteract these threats, we propose an effective algorithm called ALOE, which performs robust training by exposing the model to both adversarially crafted inlier and outlier examples. Our method can be flexibly combined with, and render existing methods robust. On common benchmark datasets, we show that ALOE substantially improves the robustness of state-of-the-art OOD detection, with 58.4\% AUROC improvement on CIFAR-10 and 46.59\% improvement on CIFAR-100. 

 
\end{abstract}

\section{Introduction}
\label{sec:intro}


Out-of-distribution (OOD) detection has become an indispensable part of building reliable open-world machine learning models~\cite{BendaleB15}. An OOD detector is used to determine whether an input is from the training data distribution (in-distribution examples), or from a different distribution (OOD examples). Previous OOD detection methods are usually evaluated on benign in-distribution and OOD inputs~\citep{HsuSJK20,HuangL21,lee2018simple,liang2017enhancing,LiuWOL20}. Recently, some works have shown the existence of adversarial OOD examples, which are generated by slightly perturbing the clean OOD inputs to make the OOD detectors fail to detect them as OOD examples, and have proposed some robust OOD detection methods to address the issue of adversarial OOD examples~\citep{sehwag2019analyzing,hein2019relu,meinke2019towards,BitterwolfM020,ChenLWLJ21}. 

In this paper, we also consider the problem of robust OOD detection. Different from previous works, we not only consider adversarial OOD examples, but also consider adversarial in-distribution examples, which are generated by slightly perturbing the clean in-distribution inputs and cause the OOD detectors to falsely reject them. We argue that both adversarial in-distribution examples and adversarial OOD examples can cause severe consequences if the OOD detectors fail to detect them, as illustrated in Figure~\ref{fig:adversarial-ood-example}. 
\begin{figure*}[t]
	\centering
	\includegraphics[width=0.8\linewidth]{figures/adversarial-ood-example.pdf}
	\caption{\small When deploying OOD detector $G(x)$ in the real world, there can be two types of attacks: outlier attack and inlier attack on $G(x)$. To perform outlier attack, we add small perturbation to an OOD input (e.g. mailbox) which causes the OOD detector to misclassify them as in-distribution example. The downstream classifier $f(x)$ will then classify this example into one of the known classes (e.g. stop sign), and trigger wrong action. To perform inlier attack, we  add small perturbation to an in-distribution sample (e.g. stop sign) which causes the OOD detector to misclassify them as out-of-distribution example and reject it without taking the correct action (e.g. stop sign). Solid lines indicate the actual computation flow.}
	\label{fig:adversarial-ood-example}
\end{figure*}
Formally, we study the problem of {\em
robust out-of-distribution detection} and reveal the lack of robustness of common OOD detection methods. We show that existing OOD detection algorithms can be easily attacked to produce mistaken OOD prediction
under small adversarial perturbations~\citep{papernot2016limitations,goodfellow2014explaining,biggio2013evasion,szegedy2013intriguing}. Specifically, we construct {\em adversarial in-distribution examples} by adding small perturbations to the in-distribution inputs such that the OOD detectors will falsely reject them; whereas {\em adversarial OOD
examples} are generated by adding small perturbations to the OOD inputs such that the OOD detectors will fail to reject them. Different from the common notion, the adversarial examples in our work are meant to fool the OOD detectors $G(x)$, rather than the original image classification model $f(x)$. 
It is also worth noting that the perturbation is
sufficiently small so that the visual semantics as well as true
distributional membership remain the same. Yet worryingly,
state-of-the-art OOD detectors can fail to distinguish between adversarial in-distribution examples and adversarial OOD examples. Although there are some works trying to make OOD detection robust to adversarial OOD examples, scant attention has been paid to making the OOD detectors robust against both the adversarial in-distribution examples and adversarial OOD examples. To the best of our knowledge, we are the first to consider the issue of adversarial in-distribution examples.  
To address the challenge , we propose an effective method, ALOE, that
improves the robust OOD detection performance. Specifically, we
perform robust training by exposing the model to two types of
perturbed adversarial examples. For in-distribution training data, we create
a perturbed example by searching in its $\epsilon$-ball that maximizes the
negative log likelihood. In addition, we also utilize an auxiliary
unlabaled dataset as in ~\cite{hendrycks2018deep}, and create
corresponding perturbed outlier example by searching in its $\epsilon$-ball that
maximizes the KL-divergence between model output and a uniform
distribution. The overall training objective of ALOE can be viewed as
an adversarial min-max game.  We show that on several benchmark
datasets, ALOE can improve the robust OOD detection performance by up
to  58.4\% compared to previous state-of-the-art method. Our approach can be complemented by techniques such as ODIN~\citep{liang2017enhancing}, and further boost the
performance.
Our main contributions are as follows:
\begin{itemize}
	\item We extensively examine the robust OOD detection
	problem on common OOD detection approaches by considering both adversarial in-distribution examples and adversarial OOD examples. We show that state-of-the-art OOD detectors can fail to distinguish between in-distribution examples and OOD examples under
	small adversarial perturbations; 
        \item We propose an effective
	algorithm, ALOE, that substantially improves the robustness
	of OOD detectors;
	\item We empirically analyze why common adversarial examples targeting the classifier with small perturbations should be regarded as in-distribution rather than OOD.
	\item We release a code base that integrates the most common OOD detection baselines, and our robust OOD detection methods at: \url{https://github.com/jfc43/robust-ood-detection}. We hope this can ensure reproducibility of all methods, and make it easy for the community to conduct future research on this topic. 
\end{itemize}


\section{Related Work}
\label{sec:related}

 \paragraph{OOD Detection.} \citeauthor{hendrycks2016baseline} introduced a baseline for OOD detection using the maximum softmax probability from a pre-trained network. Subsequent works improve the OOD detection by using deep ensembles~\citep{lakshminarayanan2017simple}, the calibrated softmax score~\citep{liang2017enhancing}, the Mahalanobis distance-based confidence score~\citep{lee2018simple}, and the energy score~\citep{LiuWOL20}. Some methods also modify the neural networks by re-training or fine-tuning on some auxiliary anomalous data that are or realistic~\citep{hendrycks2018deep, mohseni2020self} or artificially generated by GANs~\citep{lee2017training}. Many other works \citep{subramanya2017confidence,malinin2018predictive,bevandic2018discriminative} also regularize the model to have lower confidence on anomalous examples. Recent works have also studied the computational efficiency aspect of OOD detection~\citep{LinRL21} and large-scale OOD detection on ImageNet~\citep{HuangL21}. 
\paragraph{Robustness of OOD detection. } Worst-case aspects of OOD detection have previously been studied in \citep{sehwag2019analyzing,hein2019relu,meinke2019towards,BitterwolfM020,ChenLWLJ21}. However, these papers are primarily concerned with adversarial OOD examples. We are the first to present a unified framework to study both adversarial in-distribution examples and adversarial OOD examples. 
 
 \paragraph{Adversarial Robustness.}  A well-known phenomenon of adversarial examples \citep{biggio2013evasion,goodfellow2014explaining,papernot2016limitations,szegedy2013intriguing} has received great attention in recent years. Many defense methods have been proposed to address this problem. One of the most effective methods is adversarial training \citep{madry2017towards} which uses robust optimization techniques to render deep learning models resistant to adversarial attacks. In this paper, we show that the OOD detectors built from deep models are also very brittle under small perturbations, and propose a method to mitigate this issue using techniques from robust optimization.

\section{Traditional OOD Detection}
\label{sec:preliminaries}

Traditional OOD detection can be formulated as a canonical binary classification problem. Suppose we have an \textbf{in-distribution} $P_{\bm{X}}$ defined on an input space $\mathcal{X}\subset \mathbb{R}^n$. An OOD classifier $G:\mathcal{X}\mapsto \{0,1\}$ is built to distinguish whether an input $x$ is from $P_{\bm{X}}$ (give it label $1$) or not (give it label $0$). 

In testing, the detector  $G$ is evaluated on inputs drawn from a mixture distribution ${\mathcal{M}}_{\bm{X}\times Z}$ defined on $\mathcal{X}\times\{0,1\}$, where the conditional probability distributions ${\mathcal{M}_{\bm{X}|Z=1}=P_{\bm{X}}}$ and ${\mathcal{M}}_{\bm{X}|Z=0}=Q_{\bm{X}}$. We assume that $Z$ is drawn uniformly from $\{0,1\}$. $Q_{\bm{X}}$ is also a distribution defined on $\mathcal{X}$ which we refer to it as \textbf{out-distribution}. Following previous work~\citep{BendaleB16,sehwag2019analyzing}, we assume that $P_{\bm{X}}$ and $Q_{\bm{X}}$ are sufficiently different and $Q_{\bm{X}}$ has a label set that is disjoint from that of $P_{\bm{X}}$. We denote by $\mathcal{D}_{\text{in}}^{\text{test}}$  an in-distribution test set drawn from $P_{\bm{X}}$, and $\mathcal{D}_{\text{out}}^{\text{test}}$ an out-of-distribution test set drawn from $Q_{\bm{\bm{X}}}$. The {\em detection error} of $G(x)$ evaluated under in-distribution $P_{\bm{X}}$ and out-distribution $Q_{\bm{X}}$ is defined by 
\begin{align}
L(P_{\bm{X}}, Q_{\bm{X}}; G) & = \frac{1}{2}(\mathbb{E}_{x\sim P_{\bm{X}}} \mathbb{I}[G(x)=0] \\ \nonumber
  &+ \mathbb{E}_{x\sim Q_{\bm{X}}}  \mathbb{I}[G(x)=1])
\end{align}


\section{Robust Out-of-Distribution Detection}
\label{sec:problem-statement}
Traditional OOD detection methods are shown to work well when evaluated on natural in-distribution and OOD samples. However, in this section, we show that existing OOD detectors are extremely brittle and can fail when we add minimal semantic-preserving perturbations to the inputs. We start by formally describing the problem of {\em robust out-of-distribution detection}. 

\paragraph{Problem Statement.} We define $\Omega(x)$ to be a set of {semantic-preserving perturbations} on an input $x$. For $\delta \in \Omega(x)$, $x+\delta$ has the same semantic label as $x$. This also means that $x$ and $x+\delta$  have  the same distributional membership (i.e. $x$ and $x+\delta$ both belong to in-distribution $P_{\bm{X}}$, or out-distribution $Q_{\bm{X}}$). %
A robust OOD classifier $G:\mathcal{X}\mapsto \{0,1\}$ is built to distinguish whether a perturbed input $x+\delta$ is from $P_{\bm{X}}$ or not. In testing, the detector $G$ is evaluated on perturbed inputs drawn from a mixture distribution ${\mathcal{M}}_{\bm{X}\times Z}$ defined on $\mathcal{X}\times\{0,1\}$, where the conditional probability distributions ${\mathcal{M}_{\bm{X}|Z=1}=P_{\bm{X}}}$ and ${\mathcal{M}}_{\bm{X}|Z=0}=Q_{\bm{X}}$. We assume that $Z$ is drawn uniformly from $\{0,1\}$. The {\em detection error} of $G$ evaluated under in-distribution $P_{\bm{X}}$ and out-distribution $Q_{\bm{X}}$ is now defined by 

\begin{align}
L(P_{\bm{X}}, Q_{\bm{X}}; G, \Omega) & = \frac{1}{2}(\mathbb{E}_{x\sim P_{\bm{X}}} \max_{\delta \in \Omega(x)} \mathbb{I}[G(x+\delta)=0] \nonumber \\ 
 & + \mathbb{E}_{x\sim Q_{\bm{X}}} \max_{\delta \in \Omega(x)}  \mathbb{I}[G(x+\delta)=1])
\label{robust-detection-error}
\end{align}

In practice, it can be intractable to directly minimize $L(P_{\bm{X}}, Q_{\bm{X}}; G, \Omega )$ due to lack of prior knowledge on $Q_{\bm{X}}$.  In some cases we assume having access to auxiliary data sampled from a distribution $U_{\bm{X}}$ which is different from both $P_{\bm{X}}$ and $Q_{\bm{X}}$. 
\paragraph{Adversarial Attacks on OOD Detection.}
In the appendix, we describe a few common OOD detection methods such as MSP~\citep{hendrycks2016baseline}, ODIN~\citep{liang2017enhancing} and Mahalanobis~\citep{lee2018simple}. We then propose adversarial attack algorithms that can show the vulnerability of these OOD detection approaches. Computing the exact value of detection error defined in equation (\ref{robust-detection-error}) requires enumerating all possible perturbations. This can be practically intractable given the large space of $\Omega(x) \subset \mathbb{R}^n$. To this end, we propose adversarial attack algorithms that can find the perturbations in $\Omega(x)$ to compute a lower bound. 

Specifically, we consider image data and small $L_\infty$ norm-bounded perturbations on $x$ since it is commonly used in adversarial machine learning research~\citep{madry2017towards,athalye2018obfuscated}. %
For data point $x \in \mathbb{R}^{n}$, a set of adversarial perturbations is defined as 
\begin{align}
B(x, \epsilon) = \{\delta \in \mathbb{R}^{n}  \bigm| \| \delta \|_\infty \leq \epsilon \land x+\delta \text{ is valid} \},
\end{align} 
where $\epsilon$ is the size of small perturbation, which is also called adversarial budget. $x+\delta$ is considered valid if the values of $x+\delta$ are in the image pixel value range. 

For the OOD detection methods based on softmax confidence score (e.g. MSP, ODIN and OE~\citep{hendrycks2018deep}), we describe the attack mechanism in Algorithm ~\ref{alg:softmax-confidence-attack}. Specifically, we construct adversarial test examples by adding small perturbations in $B(x,\epsilon)$ so to change the prediction confidence in the reverse direction. To generate {\em adversarial in-distribution examples}, the model is induced to output probability distribution that is close to uniform; whereas {\em adversarial OOD examples} are constructed to induce the model produce high confidence score. We note here that the adversarial examples here are  constructed to fool the OOD detectors $G(x)$, rather than the image classification model $f(x)$. 
\begin{algorithm}[!htb] 
	\caption{Adversarial attack on OOD detectors based on softmax confidence score.}
	\label{alg:softmax-confidence-attack}
	\begin{algorithmic}
\INPUT $x$, $F$, $\epsilon$, $m$, $\xi$
\OUTPUT $\delta$
\STATE $\delta \leftarrow$ randomly choose a vector from $B(x,\epsilon)$
\FOR{$t=1, 2, \cdots, m$}
\STATE $x' \leftarrow x+\delta$
\IF{$x$ is in-distribution}
\STATE $\ell(x') \leftarrow L_{\text{CE}}({F}(x'), \mathcal{U}_K)$
\ELSE
\STATE $\ell(x') \leftarrow  - \sum_{i=1}^K F_i(x') \log F_i(x')$
\ENDIF
\STATE $\delta' \leftarrow \delta-\xi \cdot \text{sign}(\nabla_x \ell(x'))$ 
\STATE $\delta \leftarrow \prod_{B(x, \epsilon)} \delta'$  \hfill   \text{$\triangleright$ projecting $\delta'$ to $B(x, \epsilon)$}
\ENDFOR
	\end{algorithmic}
\end{algorithm}

For the OOD detection methods using Mahalanobis distance based confidence score, we propose an  attack algorithm detailed in Algorithm ~\ref{alg:mahalanobis-attack}. Specifically, we construct adversarial test examples by adding small perturbations in $B(x,\epsilon)$ to make the logistic regression detector predict wrongly. Note that in our attack algorithm, we don't perform input pre-processing to compute the Mahalanobis distance based confidence score. %

\begin{algorithm}[!htb] 
	\caption{Adversarial attack on OOD detector using Mahalanobis distance based confidence score.}
	\label{alg:mahalanobis-attack}
	\begin{algorithmic}
\INPUT $x$, $M_\ell (\cdot)$, $\{\alpha_\ell\}$, $b$, $\epsilon$, $m$, $\xi$
\OUTPUT $\delta$
\STATE $\delta \leftarrow$ randomly choose a vector from $B(x,\epsilon)$
\FOR{$t=1, 2, \cdots, m$}
\STATE $x' \leftarrow x+\delta$
\STATE $p(x') \leftarrow \frac{1}{1+e^{-(\sum_\ell \alpha_\ell M_\ell (x')+b)}}$
\IF{$x$ is in-distribution}
\STATE $\ell(x') \leftarrow -\log p(x')$
\ELSE
\STATE $\ell(x') \leftarrow -\log (1-p(x')) $
\ENDIF
\STATE $\delta' \leftarrow \delta + \xi \cdot \text{sign}(\nabla_x \ell(x'))$ 
\STATE $\delta \leftarrow \prod_{B(x, \epsilon)} \delta'$   \hfill   \text{$\triangleright$ projecting $\delta'$ to $B(x, \epsilon)$}
\ENDFOR
	\end{algorithmic}
\end{algorithm}

Our attack algorithms assume having access to the model parameters, thus they are white-box attacks. We find that using our attack algorithms, even with very minimal attack strength ($\epsilon=1/255$ and $m=10$), classic OOD detection methods (e.g. MSP, ODIN, Mahalanobis, OE, and OE+ODIN) can fail miserably. For example, the false positive rate of OE method can increase by 95.52\% under such attack when evaluated on CIFAR-10 as in-distribution dataset. %


\section{ALOE: Adversarial Learning with inliner and Outlier Exposure}
\label{sec:method}
In this section, we introduce a novel method called {\em Adversarial Learning with inliner and Outlier Exposure (ALOE)} to improve the robustness of the OOD detector $G(\cdot)$ built on top of the neural network $f(\cdot)$ against input perturbations.
\paragraph{Training Objective.} We train our model ALOE against two types of perturbed examples. For in-distribution inputs $x\in P_{\bm{X}}$, ALOE creates {\em adversarial inlier} within the $\epsilon$-ball that maximize the negative log likelihood. Training with perturbed examples from the in-distribution helps calibrate the error on inliers, and make the model more invariant to the additive noise. In addition, our method leverages an auxiliary unlabeled dataset $\mathcal{D}_{\text{out}}^{\text{OE}}$ drawn from $U_{\bm X}$ as used in~\cite{hendrycks2018deep}, but in a different objective. While OE directly uses the original images $x\in \mathcal{D}_{\text{out}}^{\text{OE}}$ as outliers, ALOE creates {\em adversarial outliers} by searching within the $\epsilon$-ball that maximize the KL-divergence between model output and a uniform distribution. The overall training objective of $F_\text{ALOE}$ can be formulated as a min-max game given by 

\begin{align}
\minimize_\theta & \mathbb{E}_{(x,y)\sim \mathcal{D}_{\text{in}}^{\text{train}}} \max_{\delta \in B(x,\epsilon)} [-\log {F_\theta}(x+\delta)_y] \nonumber \\ 
 + & \lambda \cdot \mathbb{E}_{x \sim \mathcal{D}_{\text{out}}^{\text{OE}}} \max_{\delta \in B(x,\epsilon)}  [L_{\text{CE}}({F_\theta}(x+\delta), \mathcal{U}_K)] 
\end{align}

where $F_\theta(x)$ is the softmax output of the neural network.  

To solve the inner max of these objectives, we use the Projected Gradient Descent (PGD) method \citep{madry2017towards}, which is the standard method for large-scale constrained optimization. The hyper-parameters of PGD used in the training will be provided in the experiments. 

Once the model $F_\text{ALOE}$ is trained, it can be used for downstream OOD detection by combining with approaches such as MSP and ODIN. The corresponding detectors can be constructed as $G_{\text{MSP}}(x; \gamma, F_{\text{ALOE}})$, and $G_{\text{ODIN}}(x; T, \eta, \gamma, F_{\text{ALOE}})$, respectively.
\paragraph{Possible Variants.} We also derive two other variants of robust training objective for OOD detection. The first one performs adversarial training {\em only} on the inliers. We denote this method as ADV, which is equivalent to the objective used in~\cite{madry2017towards}. The training objective for ADV is: 
 \begin{align*}
 \minimize_\theta & \quad \mathbb{E}_{(x,y)\sim \mathcal{D}_{\text{in}}^{\text{train}}} \max_{\delta \in B(x,\epsilon)} [-\log {F_\theta}(x+\delta)_y] 
 \end{align*}
 
 Alternatively, we also considered performing
 adversarial training on inlier examples while simultaneously performing outlier exposure as in~\cite{hendrycks2018deep}. We refer to this variant as AOE (adversarial learning with outlier exposure). The training objective for AOE is: 
\begin{align*}
\minimize_\theta & \quad \mathbb{E}_{(x,y)\sim \mathcal{D}_{\text{in}}^{\text{train}}} \max_{\delta \in B(x,\epsilon)} [-\log {F_\theta}(x+\delta)_y] \\
+ & \lambda \cdot \mathbb{E}_{x \sim \mathcal{D}_{\text{out}}^{\text{OE}}}  [L_{\text{CE}}({F_\theta}(x), \mathcal{U}_K)] 
\end{align*}

We provide ablation studies comparing these variants with ALOE in the next section.


\section{Experiments}
\label{sec:experiment}

In this section we perform extensive experiments to evaluate previous OOD detection methods and our ALOE method under adversarial attacks on in-distribution and OOD inputs. Our main findings are summarized as follows:
\begin{itemize}
	\item[{\bf (1)}] Classic OOD detection methods such as ODIN, Mahalanobis, and OE fail drastically under our adversarial attacks even with a very small perturbation budget.
	\item[{\bf (2)}] Our method ALOE can significantly improve the performance of OOD detection under our adversarial attacks compared to the classic OOD detection methods. Also, we observe that the performance of its variants ADV and AOE is worse than it in this task. And if we combine ALOE with other OOD detection approaches such as ODIN, we can further improve its performance. What's more, ALOE improves model robustness while maintaining almost the same classification accuracy on the clean test inputs (the results are in the appendix).  
	\item[{\bf (3)}] Common adversarial examples targeting the image classifier $f(x)$ with small perturbations should be regarded as in-distribution rather than OOD. 
\end{itemize}

Next we provide more details. 

\subsection{Setup}
\label{sec:setup}

\paragraph{In-distribution Datasets.} we use GTSRB~\citep{stallkamp2012man}, CIFAR-10 and CIFAR-100 datasets~\citep{krizhevsky2009learning} as in-distribution datasets. The pixel values of all the images are normalized to be in the range [0,1]. 

\paragraph{Out-of-distribution Datasets.} For auxiliary outlier dataset, we use 80 Million Tiny Images \citep{torralba200880}, which is a large-scale, diverse dataset scraped from the web. We follow the same  deduplication procedure as in \cite{hendrycks2018deep} and remove all examples in this dataset that appear in CIFAR-10 and CIFAR-100 to ensure that $\mathcal{D}_{\text{out}}^{\text{OE}}$ and $\mathcal{D}_{\text{out}}^{\text{test}}$ are disjoint.
For OOD test dataset, we follow the settings in \cite{liang2017enhancing,hendrycks2018deep}. For CIFAR-10 and CIFAR-100, we use six different natural image datasets: \texttt{SVHN}, \texttt{Textures}, \texttt{Places365}, \texttt{LSUN (crop)}, \texttt{LSUN (resize)}, and \texttt{iSUN}. For GTSRB, we use the following six datasets that are sufficiently different from it: \texttt{CIFAR-10}, \texttt{Textures}, \texttt{Places365}, \texttt{LSUN (crop)}, \texttt{LSUN (resize)}, and \texttt{iSUN}. Again, the pixel values of all the images are normalized to be in the range [0,1]. The details of these datasets can be found in the appendix.

\paragraph{Architectures and Training Configurations.}  We use the state-of-the-art neural network architecture DenseNet \citep{huang2017densely}. We follow the same setup as in \cite{huang2017densely}, with depth $L=100$, growth rate $k=12$ (Dense-BC) and dropout rate $0$. All neural networks are trained with stochastic gradient descent with Nesterov momentum \citep{duchi2011adaptive,kingma2014adam}. Specifically, we train Dense-BC with momentum $0.9$ and $\ell_2$ weight decay with a coefficient of $10^{-4}$. For GTSRB, we train it for 10 epochs; for CIFAR-10 and CIFAR-100, we train it for 100 epochs. For in-distribution dataset, we use batch size 64; For outlier exposure with $\mathcal{D}_{\text{out}}^{\text{OE}}$, we use batch size 128. The initial learning rate of $0.1$ decays following a cosine learning rate schedule \citep{loshchilov2016sgdr}.

\paragraph{Hyperparameters.} For ODIN~\citep{liang2017enhancing}, we choose temperature scaling parameter $T$ and perturbation magnitude $\eta$ by validating on a random noise data, which does not depend on prior knowledge of out-of-distribution datasets in test. In all of our experiments, we set $T=1000$. We set $\eta=0.0004$ for  GTSRB, $\eta=0.0014$ for CIFAR-10, and $\eta=0.0028$ for CIFAR-100. For Mahalanobis \citep{lee2018simple}, we randomly select 1,000 examples from $\mathcal{D}_{\text{in}}^{\text{train}}$ and 1,000 examples from $\mathcal{D}_{\text{out}}^{\text{OE}}$ to train the Logistic Regression model and tune $\eta$, where $\eta$ is chosen from 21 evenly spaced numbers starting from 0 and ending at 0.004, and the optimal parameters are chosen to
minimize the FPR at TPR 95\%. For OE, AOE and ALOE methods, we fix the regularization parameter $\lambda$ to be 0.5. In PGD that solves the inner max of ADV, AOE and ALOE, we use step size  $1/255$, number of steps $\lfloor 255\epsilon+1 \rfloor$, and  random start. For our attack algorithm, we set $\xi=1/255$ and $m=10$ in our experiments. The adversarial budget $\epsilon$ by default is set to $1/255$, however we perform ablation studies by varying the value (see the results in the appendix). 
More experiment settings can be found in the appendix. 

\subsection{Evaluation Metrics}
We report main results using three metrics described below. 

\paragraph{FPR at 95\% TPR.} This metric calculates the false positive rate (FPR) on out-of-distribution examples when the true positive rate (TPR) is 95\%. %

\paragraph{Detection Error.} This metric corresponds to the minimum mis-detection probability over all possible thresholds $\gamma$, which is  $\min_{\gamma} L(P_X, Q_X; G(x;\gamma))$. 

\paragraph{AUROC.} Area Under the Receiver Operating Characteristic curve is a threshold-independent metric \citep{davis2006relationship}. It can be interpreted as the probability that a positive example is assigned a higher detection score than a negative example \citep{fawcett2006introduction}. A perfect detector corresponds to an AUROC score of 100\%.

\subsection{Results}

\begin{table*}[t]
	\begin{adjustbox}{width=2\columnwidth,center}
\begin{tabular}{l|l|ccc|ccc}
\toprule
\multirow{4}{0.08\linewidth}{$\mathcal{D}_{\text{in}}^{\text{test}}$} &  \multirow{4}{0.06\linewidth}{\textbf{Method}}  &\bf{FPR}  & \bf{Detection}  & {\bf AUROC}  & {\bf FPR}  &   {\bf Detection} & {\bf AUROC}  \\
& & $\textbf{(95\% TPR)}$ & $\textbf{Error}$ & $\textbf{}$ & $\textbf{(95\% TPR)}$ & $\textbf{Error}$ & $\textbf{}$  \\
& & $\downarrow$ & $\downarrow$ & $\uparrow$ & $\downarrow$ & $\downarrow$ & $\uparrow$ \\ \cline{3-8}
& & \multicolumn{3}{c|}{\textbf{without attack}} & \multicolumn{3}{c}{\textbf{with attack ($\epsilon=1/255$, $m=10$)}} \\  \hline 
\multirow{9}{0.06\linewidth}{{{\bf GTSRB}}}  
& MSP \citep{hendrycks2016baseline} & 1.13 & 2.42 & 98.45 & 97.59 & 26.02 & 73.27 \\
& ODIN \citep{liang2017enhancing} & 1.42 & 2.10 & 98.81 & 75.94 & 24.87 & 75.41 \\
& Mahalanobis \citep{lee2018simple} & 1.31 & 2.87 & 98.29 & 100.00 & 29.80 & 70.45  \\
& OE \citep{hendrycks2018deep} & 0.02 & {\bf 0.34} & {\bf 99.92} & 25.85 & 5.90 & 96.09 \\
& OE+ODIN & 0.02 & 0.36 & 99.92 & 14.14 & 5.59 & 97.18 \\ 
& ADV \citep{madry2017towards} & 1.45 & 2.88 & 98.66 & 17.96 & 6.95 & 94.83 \\
& AOE & 0.00 & 0.62 & 99.86 & 1.49 & 2.55 & 98.35 \\
& ALOE (ours) & {\bf 0.00} & 0.44 & 99.76 & {\bf 0.66} & 1.80 & 98.95  \\
& ALOE+ODIN (ours) & 0.01 & 0.45 & 99.76 & 0.69 & {\bf 1.80} & {\bf 98.98} \\ \hline
\multirow{9}{0.06\linewidth}{{{\bf CIFAR-10}}}  
& MSP \citep{hendrycks2016baseline} & 51.67 & 14.06 & 91.61 & 99.98 & 50.00 & 10.34 \\
& ODIN \citep{liang2017enhancing} & 25.76 & 11.51 & 93.92 & 93.45 & 46.73 & 28.45  \\
& Mahalanobis \citep{lee2018simple} & 31.01 & 15.72 & 88.53 & 89.75 & 44.30 & 32.54  \\
& OE \citep{hendrycks2018deep} & 4.47 & 4.50 & 98.54 & 99.99 & 50.00 & 25.13\\
& OE+ODIN & {\bf 4.17} & {\bf 4.31} & {\bf 98.55} & 99.02 & 47.84 & 34.29  \\ 
& ADV \citep{madry2017towards} & 66.99 & 19.22 & 87.23 & 98.44 & 31.72 & 66.73 \\
& AOE & 10.46 & 6.58 & 97.76 & 88.91 & 26.02 & 78.39  \\
& ALOE (ours) &  5.47 & 5.13 & 98.34 & 53.99 & 14.19 & 91.26 \\
& ALOE+ODIN (ours) & 4.48 & 4.66 & 98.55 & {\bf 41.59} & {\bf 12.73} & {\bf 92.69} \\ \hline 
\multirow{9}{0.06\linewidth}{{\bf CIFAR-100}} 
& MSP \citep{hendrycks2016baseline} & 81.72 & 33.46 & 71.89 & 100.00 & 50.00 & 2.39 \\
& ODIN \citep{liang2017enhancing} & 58.84 & 22.94 & 83.63 & 98.87 & 49.87 & 21.02 \\
& Mahalanobis \cite{lee2018simple} & 53.75 & 27.63 & 70.85 & 95.79 & 47.53 & 17.92  \\
& OE \citep{hendrycks2018deep} & 56.49 & 19.38 & 87.73 & 100.00 & 50.00 & 2.94 \\
& OE+ODIN &  {\bf 47.59} & {\bf 17.39} & {\bf 90.14} & 99.49 & 50.00 & 20.02 \\  
& ADV \citep{madry2017towards} & 85.47 & 33.17 & 71.77 & 99.64 & 44.86 & 41.34 \\
& AOE &  60.00 & 23.03 & 84.57 & 95.79 & 43.07 & 53.80 \\
& ALOE (ours) &  61.99 & 23.56 & 83.72 & 92.01 & 40.09 & 61.20 \\
& ALOE+ODIN (ours) & 58.48 & 21.38 & 85.75 & {\bf 88.50} & {\bf 36.20} & {\bf 66.61} \\
\bottomrule
\end{tabular}
	\end{adjustbox}
	\caption[]{\small Distinguishing in- and out-of-distribution test set data for image classification. We contrast performance on clean images (without attack) and PGD attacked images. $\uparrow$ indicates larger value is better, and $\downarrow$ indicates lower value is better. All values are percentages and are averaged over six OOD test datasets.  }
	\label{tab:main-results}
\end{table*}

\begin{figure}[t]
	\centering
	\begin{subfigure}{0.48\linewidth}
\centering
\includegraphics[width=\linewidth]{figures/OE-without-attack-SVHN.pdf}
\caption{}
	\end{subfigure}
	\begin{subfigure}{0.48\linewidth}
\centering
\includegraphics[width=\linewidth]{figures/OE-with-attack-SVHN.pdf}
\caption{}
	\end{subfigure} 
	\begin{subfigure}{0.48\linewidth}
\centering
\includegraphics[width=\linewidth]{figures/ALOE-with-attack-SVHN.pdf}
\caption{}
	\end{subfigure}
	\begin{subfigure}{0.48\linewidth}
\centering
\includegraphics[width=\linewidth]{figures/ALOE+ODIN-with-attack-SVHN.pdf}
\caption{}
	\end{subfigure}
	\caption{\small Confidence score distribution produced by different methods. For illustration purposes, we use CIFAR-10 as in-distribution and SVHN as out-of-distribution. (a) and (b) compare the score distribution for Outlier Exposure~\citep{hendrycks2018deep}, evaluated on clean images and PGD attacked images, respectively. The distribution overall shift toward the opposite direction under our attack, which causes the method to fail. Our method ALOE can mitigate the distribution shift as shown in (c). When combined with ODIN~\citep{liang2017enhancing}, the score distributions can be further separable between in- and out-distributions, as shown in (d). }
	\label{fig:score-distribution}
\end{figure}

\begin{table}[!bth]
\begin{adjustbox}{width=\columnwidth,center}
\begin{tabular}{l|l|c}
\toprule
\multirow{2}{0.12\linewidth}{$\mathcal{D}_{\text{in}}^{\text{test}}$} &  \multirow{2}{0.06\linewidth}{\textbf{Method}}  &\bf{1-FPR}   \\
& & $\textbf{(95\% TPR)}$ \\
 \hline
\multirow{9}{0.12\linewidth}{{{\bf CIFAR-10}}}  
& MSP \citep{hendrycks2016baseline} & 10.75 \\
& ODIN \citep{liang2017enhancing} & 4.02 \\
& Mahalanobis \citep{lee2018simple} &  7.13 \\
& OE \citep{hendrycks2018deep} & 12.22  \\
& OE+ODIN &  12.95 \\ 
& ADV \citep{madry2017towards} & 7.69 \\
& AOE &  11.18 \\
& ALOE (ours) & 8.85 \\
& ALOE+ODIN (ours) & 8.71  \\ \hline 
\multirow{9}{0.12\linewidth}{{\bf CIFAR-100}} 
& MSP \citep{hendrycks2016baseline} & 0.06 \\
& ODIN \citep{liang2017enhancing} & 0.74 \\
& Mahalanobis \cite{lee2018simple} & 4.29 \\
& OE \citep{hendrycks2018deep} & 4.36 \\
& OE+ODIN & 5.21 \\  
& ADV \citep{madry2017towards} & 3.14 \\
& AOE &  8.08\\
& ALOE (ours) & 7.32  \\
& ALOE+ODIN (ours) & 7.06 \\
\bottomrule
\end{tabular}
	\end{adjustbox}
	\caption[]{\small Distinguishing adversarial examples generated by PGD attack on the image classifier $f(x)$. 1-FPR indicates the rate of misclassifying adversarial examples as out-of-distribution examples. For PGD attack, we choose $\epsilon$ as $1/255$ and the number of attack steps as $10$.  All values are percentages. }
	\label{tab:adv-results}
\end{table}
All the values reported in this section are averaged over {\em six} OOD test datasets. %

\paragraph{Classic OOD detection methods fail under our attack.} 
As shown in Table \ref{tab:main-results}, although classic OOD detection methods (e.g. MSP, ODIN, Mahalanobis, OE and OE+ODIN) could perform quite well on detecting natural OOD samples, their performance drops substantially under the attack (even with very minimal attack budget $\epsilon=1/255$ and $m=10$). For the best-performing OOD detection method (i.e., OE+ODIN), the FPR at 95\% TPR increases drastically from 4.17\% (without attack) to 99.02\% (with attack) when evaluated on the CIFAR-10 dataset.

\paragraph{ALOE improves robust OOD detection performance.} As shown in Table \ref{tab:main-results}, 
our method ALOE could significantly improve the OOD detection performance under the adversarial attack. For example, ALOE can substantially improve the AUROC from  34.29\% (state-of-the-art: OE+ODIN) to 92.69\% evaluated on the CIFAR-10 dataset. The performance can be further improved when combining ALOE with ODIN. We observe this trend holds consistently on other benchmark datasets GTSRB and CIFAR-100 as in-distribution training data. We also find that adversarial training (ADV) or combining adversarial training with outlier exposure (AOE) yield slightly less competitive results. 
To better understand our method, we analyze the distribution of confidence scores produced by the OOD detectors on SVHN (out-distribution) and CIFAR-10 (in-distribution). As shown in Figure~\ref{fig:score-distribution}, OE could distinguish in-distribution and out-of-distribution samples quite well since the confidence scores are well separated. However, under our attack, the confidence scores of in-distribution samples move towards 0 and the scores of out-of-distribution samples move towards 1.0, which renders the detector fail to distinguish in- and out-of-distribution samples. Using our method, the confidence scores (under attack) become separable and shift toward the right direction. If we further combine ALOE with ODIN, the scores produced by the detector are even more separated.
\paragraph{Evaluating on common adversarial examples targeting the classifier $f(x)$.}  Our work is primarily concerned with adversarial examples targeting OOD detectors $G(x)$. This is very different from the common notion of adversarial examples that are constructed to fool the image classifier $f(x)$. 
Based on our robust definition of OOD detection, adversarial examples constructed from in-distribution data with small perturbations to fool the image classifier $f(x)$ should be regarded as in-distribution. To validate this point, we generate PGD attacked images w.r.t the original classification model $f(x)$ trained on CIFAR-10 and CIFAR-100 respectively using a small perturbation budget of $1/255$. We measure the performance of OOD detectors $G(x)$ by reporting  1-FPR (at TPR 95\%), which indicates the rate of misclassifying adversarial examples as out-of-distribution examples. As shown in Table~\ref{tab:adv-results}, the metric in general is low for both classic and robust OOD detection methods, which suggests that common adversarial examples with small perturbations are closer to in-distribution rather than OOD. 

\section{Conclusion}
\label{sec:conclusion}
In this paper, we study the problem, Robust Out-of-Distribution Detection, and propose adversarial attack algorithms which reveal the lack of robustness of a wide range of OOD detection methods. We show that state-of-the-art OOD detection methods can fail catastrophically under both adversarial in-distribution and  out-of-distribution attacks. To counteract these threats, we propose a new method called ALOE, which substantially improves the robustness of state-of-the-art OOD detection. We empirically analyze our method under different parameter settings and optimization objectives, and provide theoretical insights behind our approach. Future work involves exploring alternative semantic-preserving perturbations beyond adversarial attacks. 



\begin{quote}
\begin{small}
\bibliography{paper}
\end{small}
\end{quote}

\appendix

\begin{center}
	\textbf{\LARGE Appendix}
\end{center}

\section{Existing Approaches}
\label{sec:ood-techs}
Recently, several approaches propose to detect OOD examples based on different notions of confidence scores from a neural network $f(\cdot)$, which is trained on a dataset $\mathcal{D}_{\text{in}}^{\text{train}}$ drawn from a data distribution $P_{\bm{X},Y}$ defined on $\mathcal{X} \times \mathcal{Y}$ with $\mathcal{Y}=\{1,2,\cdots,K \}$. Note that $P_{\bm{X}}$ is the marginal distribution of $P_{\bm{X},Y}$. Based on this notion, we describe a few common methods below.

\paragraph{Maximum Softmax Probability (MSP).} Maximum Softmax Probability method is as a common baseline for OOD detection \citep{hendrycks2016baseline}. Given an input image $x$ and a pre-trained neural network $f(\cdot)$, the softmax output of the classifier is computed by 
$F(x)=\frac{e^{f_i(x)}}{\sum_{j=1}^{K} e^{f_j(x)}}.$

A threshold-based detector  $G(x)$ relies on the confidence score $S(x;f) = \max_i F_i(x)$ to make prediction as follows
\begin{align}
G_{\text{MSP}}(x; \gamma, f) = 
\begin{cases} 
0 & \quad \text{if } S(x;f) \leq \gamma \\
1 & \quad \text{if } S(x;f) > \gamma 
\end{cases}
\end{align} 
where $\gamma$ is the confidence threshold. 

\paragraph{ODIN.}  The original softmax confidence scores used in \cite{hendrycks2016baseline} can be over-confident.  ODIN~\citep{liang2017enhancing} leverages this insight and improves the MSP baseline using the calibrated confidence score instead~\citep{guo2017calibration}. Specifically, the calibrated confidence score is computed by 
$S(x;T,f)=\max_i \frac{e^{f_i(x)/T}}{\sum_{j=1}^{K} e^{f_j(x)/T}},$
where $T \in \mathbb{R}^+$ is a temperature scaling parameter. In addition, ODIN applies small noise perturbation to the inputs 
\begin{equation}
\label{eq:perturbation}\tilde{{x}}={{x}}-\eta \cdot \text{sign}(-\nabla_{{{x}}}\log S({{x}};T, f)),
\end{equation}
where the parameter $\eta$ is the perturbation magnitude. 

By combining the two components together, ODIN detector $G_{\text{ODIN}}$ is given by
\begin{align}
G_{\text{ODIN}}(x; T, \eta, \gamma, f) = 
\begin{cases} 
0 & \quad \text{if } S(\tilde{x};T,f)  \leq \gamma \\
1 & \quad \text{if } S(\tilde{x};T,f) > \gamma 
\end{cases}
\end{align} 
In real applications, it may be difficult to know the out-of-distribution samples one will encounter in advance. The hyperparameters of $T$ and $\eta$ can be tuned instead on a random noise data such as Gaussian or uniform distribution, without requiring prior knowledge of OOD dataset. 

\paragraph{Mahalanobis.}  \citeauthor{lee2018simple} model the features of training data as class-conditional Gaussian
distribution, where its parameters are chosen as empirical class means and empirical
covariance of training samples. Specifically, for a given sample $x$, the confidence score from the $\ell$-th feature layer is defined using the Mahalanobis distance with respect to the closest class-conditional distribution:
\begin{align}
	M_\ell(x) = \max_c -(f_\ell(x)-\hat{\mu}_{\ell,c})^T \hat{\Sigma}_\ell^{-1} (f_\ell(x)-\hat{\mu}_{\ell,c}),
\end{align}
where $f_\ell(x)$ is the $\ell$-th hidden features of DNNs, and $\hat{\mu}_{\ell,c}$ and $\hat{\Sigma}_\ell$ are the empirical class means and covariances computed from the training data respectively.
In addition, they use two techniques (1) input pre-processing and (2) feature ensemble. Specifically, for each test sample $x$, they first calculate the pre-processed sample $\tilde{x}_\ell$ by adding the small perturbations as  in~\cite{liang2017enhancing}: 
$\tilde{x}_\ell = x+\eta \cdot \text{sign}(\nabla_x M_\ell(x)),$
where $\eta$ is a magnitude of noise, which can be tuned on the validation data. 

The confidence scores from all layers are integrated through a weighted averaging: $\sum_\ell \alpha_\ell M_\ell (\tilde{x}_\ell)$. The weight of each layer $\alpha_\ell$ is learned through a logistic regression model, which predicts 1 for in-distribution and 0 for OOD examples. The overall Mahalanobis distance based confidence score  is 
\begin{align}
M(x) = \frac{1}{1+e^{-(\sum_\ell \alpha_\ell M_\ell (\tilde{x}_\ell)+b)}},
\end{align}
where $b$ is the bias of the logistic regression model. Putting it all together, the final Mahalanobis detector $G_{\text{Mahalanobis}}$ is given by
\begin{align}
G_{\text{Mahalanobis}}(x; \eta, \gamma, \{\alpha_\ell\}, b, f) = 
\begin{cases} 
0 & \quad \text{if } M(x)  \leq \gamma \\
1 & \quad \text{if }  M(x) > \gamma
\end{cases}
\end{align}
\section{Experimental Details}
\label{sec:experimental-details}

\subsection{Setup}
\label{sec:detail-experiment-setup}
\paragraph{Software and Hardware.} We run all experiments with PyTorch and NVDIA GeForce RTX 2080Ti GPUs. 

\paragraph{Number of Evaluation Runs.} We run all experiments once with fixed random seeds. 

\paragraph{In-distribution Dataset. } We provide the details of in-distribution datasets below:
\begin{enumerate}
	\item \textbf{CIFAR-10 and CIFAR-100.} The CIFAR-10 and CIFAR-100~\citep{krizhevsky2009learning} have 10 and 100 classes respectively. Both datasets consist of 50,000 training images and 10,000 test images. 
	\item \textbf{GTSRB.} The German Traffic Sign Recognition Benchmark (GTSRB)~\citep{stallkamp2012man} is a dataset of color
	images depicting 43 different traffic signs. The images are not of a fixed dimensions and have rich
	background and varying light conditions as would be expected of photographed images of traffic
	signs. There are about 34,799 training images, 4,410 validation images and 12,630 test images.
	We resize each image to $32 \times 32$. The dataset has a
	large imbalance in the number of sample occurrences across classes. We use data augmentation
	techniques to enlarge the training data and make the number of samples in each class balanced.
	We construct a class preserving data augmentation pipeline consisting of rotation, translation, and
	projection transforms and apply this pipeline to images in the training set until each class contained
	10,000 training examples. This new augmented dataset containing 430,000 samples
	in total is used as $\mathcal{D}_{\text{in}}^{\text{train}}$. We randomly select 10,000 images from original test images as $\mathcal{D}_{\text{in}}^{\text{test}}$. 
\end{enumerate}

\paragraph{OOD Test Dataset.} We provide the details of OOD test datasets below:
\begin{enumerate}
	\item \textbf{SVHN.} The SVHN dataset \cite{netzer2011reading} contains $32 \times 32$ color images of house numbers. There are ten classes comprised of the digits 0-9. The original test set has 26,032 images. We randomly select 1,000 images for each class from the test set to form a new test dataset containing 10,000 images for our evaluation. 
	\item \textbf{Textures.} The Describable Textures Dataset (DTD) \cite{cimpoi14describing} contains textural images in the wild. We include the entire collection of 5640 images in DTD and downsample each image to size $32\times 32$. 
	\item \textbf{Places365.} The Places365 dataset \cite{zhou2017places} contains large-scale photographs of scenes with 365 scene categories. There are 900 images per category in the test set. We randomly sample 10,000 images from the test set for evaluation and downsample each image to size $32\times 32$.   
	\item \textbf{LSUN (crop) and LSUN (resize).} The Large-scale Scene UNderstanding dataset (LSUN) has a testing set of 10,000 images of 10 different scenes \cite{yu2015lsun}. We construct two datasets, \texttt{LSUN-C} and \texttt{LSUN-R}, by randomly cropping image patches of size $32 \times 32$ and downsampling each image to size $32 \times 32$, respectively. 
	\item \textbf{iSUN.} The iSUN \cite{xu2015turkergaze} consists of a subset of SUN images. We include the entire collection of 8925 images in iSUN and downsample each image to size $32\times 32$. 
	\item \textbf{CIFAR-10.} We use the 10,000 test images of CIFAR-10 as OOD test set for GTSRB. 
\end{enumerate}

\subsection{Additional Results}
\label{sec:additional-results}

\begin{table*}[t]
	\begin{adjustbox}{width=2\columnwidth,center}
\centering
\begin{tabular}{l|l|ccc|ccc|ccc}
\toprule
\multirow{5}{0.08\linewidth}{$\mathcal{D}_{\text{in}}^{\text{test}}$} & \multirow{5}{0.06\linewidth}{\textbf{Method}}  &\bf{FPR}  & \bf{Detection}  & {\bf AUROC}  & {\bf FPR}  &   {\bf Detection} & {\bf AUROC} & {\bf FPR}  &   {\bf Detection} & {\bf AUROC} \\
& & $\textbf{(95\% TPR)}$ & $\textbf{Error}$ & $\textbf{}$ & $\textbf{(95\% TPR)}$ & $\textbf{Error}$ & $\textbf{}$ & $\textbf{(95\% TPR)}$ & $\textbf{Error}$ & $\textbf{}$ \\
& & $\downarrow$ & $\downarrow$ & $\uparrow$ & $\downarrow$ & $\downarrow$ & $\uparrow$ & $\downarrow$ & $\downarrow$ & $\uparrow$ \\ \cline{3-11}
& & \multicolumn{3}{c|}{\textbf{with attack}} & \multicolumn{3}{c|}{\textbf{with attack}} & \multicolumn{3}{c}{\textbf{with attack}} \\
& & \multicolumn{3}{c|}{($\epsilon=2/255$, $m=10$)} & \multicolumn{3}{c|}{($\epsilon=3/255$, $m=10$)} & \multicolumn{3}{c}{($\epsilon=4/255$, $m=10$)} \\  \hline 
\multirow{9}{0.08\linewidth}{\textbf{GTSRB}} & MSP \citep{hendrycks2016baseline} & 99.88 & 50.00 & 26.11 & 99.99 & 50.00 & 6.79 & 99.99 & 50.00 & 6.39 \\
& ODIN \citep{liang2017enhancing}& 99.23 & 49.97 & 27.38 & 99.83 & 50.00 & 6.94 & 99.84 & 50.00 & 6.52 \\
& Mahalanobis \cite{lee2018simple} & 100.00 & 49.97 & 26.37 & 100.00 & 50.00 & 8.27 & 100.00 & 50.00 & 7.82 \\
& OE \citep{hendrycks2018deep} & 96.79 & 16.09 & 83.06 & 99.91 & 25.36 & 68.62 & 99.97 & 26.37 & 66.91 \\
& OE+ODIN & 89.88 & 15.78 & 84.56 & 99.25 & 24.70 & 69.71 & 99.45 & 25.67 & 68.02 \\ 
& ADV \citep{madry2017towards} & 92.17 & 11.51 & 89.92 & 99.65 & 18.59 & 80.85 & 99.49 & 18.68 & 81.17 \\
& AOE & 7.94 & 5.36 & 94.82 & 16.16 & 10.38 & 88.72 & 38.05 & 17.95 & 83.84 \\
& ALOE (ours) & 4.03 & 4.19 & {\bf 95.90} & 10.82 & 7.64 & {\bf 91.21} & 16.10 & 10.10 & {\bf 89.52} \\
& ALOE+ODIN (ours) & {\bf 3.95} & {\bf 4.15} & 95.72 & {\bf 9.56} & {\bf 6.91} & 91.08 & {\bf 13.85} & {\bf 9.22} & 89.44 \\ \hline
\multirow{9}{0.08\linewidth}{\textbf{CIFAR-10}} & MSP \citep{hendrycks2016baseline} & 100.00 & 50.00 & 1.16 & 100.00 & 50.00 & 0.13 & 100.00 & 50.00 & 0.12 \\
& ODIN \citep{liang2017enhancing}& 99.73 & 49.99 & 5.67 & 99.98 & 50.00 & 1.14 & 99.99 & 50.00 & 1.06 \\
& Mahalanobis \cite{lee2018simple} & 100.00 & 50.00 & 5.90 & 100.00 & 50.00 & 1.27 & 100.00 & 50.00 & 1.05 \\
& OE \citep{hendrycks2018deep} & 100.00 & 50.00 & 5.99 & 100.00 & 50.00 & 1.52 & 100.00 & 50.00 & 1.48\\
& OE+ODIN &100.00 & 50.00 & 8.89 & 100.00 & 50.00 & 2.76 & 100.00 & 50.00 & 2.69 \\ 
& ADV \citep{madry2017towards} & 99.94 & 36.57 & 56.01 & 99.89 & 39.64 & 49.88 & 99.96 & 40.57 & 48.02 \\
& AOE &  91.79 & 35.08 & 66.92 & 99.96 & 39.53 & 54.43 & 98.40 & 37.37 & 59.16 \\
& ALOE (ours) & 75.90 & 23.36 & 83.26 & 83.14 & 31.54 & 73.46 & 82.53 & 29.92 & 75.52  \\
& ALOE+ODIN (ours) & {\bf 68.80} & {\bf 20.31} & {\bf 85.92} & {\bf 79.19} & {\bf 28.04} & {\bf 77.88} & {\bf 78.46} & {\bf 27.55} & {\bf 78.83} \\
\bottomrule
\end{tabular}
	\end{adjustbox}
	\caption[]{\small Distinguishing in- and out-of-distribution test set data for image classification. $\uparrow$ indicates larger value is better, and $\downarrow$ indicates lower value is better. All values are percentages. The in-distribution datasets are GRSRB and CIFAR-10. All the values reported are averaged over six OOD test datasets. }
	\label{tab:blation-epsilon}
\end{table*}

\begin{table}[t]
\small
	\centering
	\begin{tabular}{l|l|c|c}
\toprule
$\mathcal{D}_{\text{in}}^{\text{test}}$ &  \textbf{Method}  &\bf{Classifcation}  & \bf{Robustness}   \\  
&&\bf{Accuracy}  & \bf{w.r.t image classifer} \\ \hline 
\multirow{6}{0.2\linewidth}{{\bf GTSRB}}  
& Original & 99.33\% & 88.47\% \\
& OE & 99.38\% & 83.99\% \\
& ADV  & 99.23\% & 97.13\% \\
& AOE & 98.82\% & 94.14\% \\
& ALOE & 98.91\% & 94.58\% \\  \hline 
\multirow{6}{0.2\linewidth}{{\bf CIFAR-10}}  
& Original & 94.08\% & 25.38\% \\
& OE& 94.59\% & 28.94\% \\
& ADV  & 92.97\% & 84.81\% \\
& AOE & 93.35\% & 78.60\% \\
& ALOE & 93.89\% & 84.02\% \\  \hline 
\multirow{6}{0.2\linewidth}{{\bf CIFAR-100}} 
& Original & 75.26\% & 7.29\% \\
& OE & 74.45\% & 7.84\% \\
& ADV& 70.58\% & 54.58\% \\
& AOE & 72.56\% & 52.96\% \\
& ALOE & 71.62\% & 55.97\% \\
\bottomrule
	\end{tabular}
	\caption[]{\small The image classification accuracy and robustness of different models on original tasks (GTSRB, CIFAR-10 and CIFAR-100).  \textit{Robustness} measures the accuracy under PGD attack w.r.t the original classification model.}
	\label{tab:classification-performance}
\end{table}
\paragraph{Effect of adversarial budget $\epsilon$.} We further perform ablation study on the adversarial budget $\epsilon$ and analyze how this  affects  performance. On GTSRB and CIFAR-10 dataset, we perform comparison by varying $\epsilon=1/255, 2/255, 3/255, 4/255$. The results are reported in Table \ref{tab:blation-epsilon}. We observe that as we increase $\epsilon$, the performance on classic OOD detection methods (e.g. MSP, ODIN, Mahalanobis, OE, OE+ODIN) drops significantly under our attack: the FPR at 95\% TPR reaches almost 100\% for all those methods. We also observe that our methods ALOE (and ALOE+ODIN) consistently improves the results under our attack compared to those classic methods. 

\paragraph{Classification performance of image classifier $f(x)$.} In addition to OOD detection, we also verify the accuracy and robustness on the original classification task. The results are presented in table \ref{tab:classification-performance}. \textit{Robustness} measures the accuracy under PGD attack w.r.t the original classification model. We use adversarial budget $\epsilon$ of $1/255$ and number of attack steps of 10. \textit{Original} refers to  the vanilla model  trained with standard cross entropy loss on the dataset. 
On  both GTSRB and CIFAR-10, ALOE improves the model robustness, while maintaining almost the same classification accuracy on the clean inputs. On CIFAR-100, ALOE improves robustness from 7.29\% to 55.97\%, albeit dropping the classification accuracy slightly (3.64\%). Overall our method achieves good trade-off between the accuracy and robustness due to adversarial perturbations. 
 


\end{document}


Peer reviews:

content: confidence: 3: The reviewer is fairly confident that the evaluation is correct
rating: 5: Marginally below acceptance threshold
review: **Summary**: 

The paper talks about how adversarial perturbations can affect the performance of OOD detectors. They present a unified framework to study both adversarial attacks on in-distribution as well as OOD inputs (ie attacks which aim to increase both FP and FN). An algorithm called ALOE is presented that trains the model with perturbed inputs and improves robustness towards such attacks. 


**PROS**

1. OOD detection is an important problem due to adversarial attacks and data drift in the real world, hence the problem is pertinent. 
2. Evaluation is detailed and compares against well-known methods. 


**CONS**

1. To me, the paper lacks innovation and novelty. Reduced to the basic version, the proposed approach is simply adversarial training with some minor tweaks. Exposing the model to perturbed inputs to improve robustness has been done previously, see [1] and [2] for examples. The only difference is the domain; while previous experiments were on classifiers, these are on OOD detectors (which are also fundamentally, classifiers). 
2. The authors claim that one of the contributions is that they 'show that state-of-the-art OOD detectors can fail to distinguish between in-distribution examples and OOD examples under small adversarial perturbations'; however, this is not surprising, see [3].
3. Finally, the attacks presented are _white box_ which seems improbable. OOD detectors are generally not exposed as an endpoint, and most users do not even perceive that there is an OOD detector at work here. Therefore, I believe that white box attacks against OOD detectors are not of practical signifiance.


**References**

[1] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples." arXiv preprint arXiv:1412.6572 (2014).

[2] Wang, Y., Ma, X., Bailey, J., Yi, J., Zhou, B., & Gu, Q. (2019, June). On the Convergence and Robustness of Adversarial Training. In ICML (Vol. 1, p. 2).

[3] Sehwag, V., Bhagoji, A. N., Song, L., Sitawarin, C., Cullina, D., Chiang, M., & Mittal, P. (2019, November). Analyzing the robustness of open-world machine learning. In Proceedings of the 12th ACM Workshop on Artificial Intelligence and Security (pp. 105-116).
title: Lacks Novelty

content: confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct
rating: 7: Good paper, accept
review: **Summary of the paper:**

This paper extensively studies the problem of robust OOD detection on common OOD detection approaches and shows that existing OOD detection algorithms can be easily attacked to produce mistaken OOD prediction by adding small perturbations to the in-distribution and OOD inputs. To address the challenge, the authors propose an effective method to improve the robust OOD detection performance. Moreover, the authors show that the proposed method can improve the robust OOD detection performance by up to 58.4% compared to the previous state-of-the-art method on several benchmark datasets.

**Detailed comments:**

a.) This paper studies the problem of _Robust Out-of-Distribution Detection_ and shows that state-of-the-art OOD detectors can be easily fooled by adding small perturbations to the in-distribution and OOD inputs. The related analysis and conclusion are interesting and valuable if there is no similar analysis in the past.

b.) The proposed method performs robust training by exposing the model to both adversarially crafted inlier and outlier examples, which is simple yet effective and works well for a wide range of datasets.


title: Review