Provably Robust Detection of Out-of-distribution Data (almost) for free
Authors: Alexander Meinke
Julian Bitterwolf
Matthias Hein
Venue: ICLR 2022 Submitted (2022)
Keywords: out-of-distribution detection
adversarial noise
provable robustness
guarantees

Abstract: The application of machine learning in safety-critical systems requires a reliable assessment of uncertainy. However, deep neural networks are known to produce highly overconfident predictions on out-of-distribution (OOD) data. Even if trained to be non-confident on OOD data one can still adversarially manipulate OOD data so that the classifier again assigns high confidence to the manipulated samples. In this paper we propose a novel method that combines a certifiable OOD detector with a standard classifier from first principles into an OOD aware classifier. This way we achieve the best of two worlds: certifiably adversarially robust OOD detection, even for OOD samples close to the in-distribution, without loss in either prediction accuracy or detection performance for non-manipulated OOD data. Moreover, due to the particular construction our classifier provably avoids the asymptotic overconfidence problem of standard neural networks.

Under review as a conference paper at ICLR 2022
PROVABLY ROBUST DETECTION OF OUT-OF- DISTRIBUTION DATA (ALMOST) FOR FREE
Anonymous authors Paper under double-blind review
ABSTRACT
The application of machine learning in safety-critical systems requires reliable assessment of uncertainy. However, deep neural networks are known to produce highly overconfident predictions on out-of-distribution (OOD) data. Even if trained to be non-confident on OOD data one can still adversarially manipulate OOD data that the classifier again assigns high confidence the manipulated samples. In this paper we propose novel method that combines a certifiable OOD detector with standard classifier from first principles into OOD aware classifier. This way we achieve the best of two worlds: certifiably adversarially robust OOD detection, even for OOD samples close the in-distribution, without loss in either prediction accuracy or detection performance for non-manipulated OOD data. Moreover, due to the particular construction our classifier provably avoids the asymptotic overconfidence problem of standard neural networks.
INTRODUCTION
Deep neural networks have achieved state-of-the-art performance many application domains. However, the widespread usage of deep neural networks in safety-critical applications, e.g. in healthcare, autonomous driving/aviation, manufacturing, raises concerns as deep neural networks have problematic deficiencies. Among these deficiencies are overconfident predictions on non-task related inputs (Nguyen et al. 2015 Hendrycks & Gimpel| 2017)which has recently attracted a lot of interest. ReLU networks have even been shown be provably overconfident far away from the training data (Hein et al. 2011)) However, reliable confidences of the classifier on the classification task )inddistrbbttonn (Guo al. 2017) as well as on the out-distribution (Hendrycks & Gimpell 2017 [Hein et al. (2019) are important to be able to detect when the deep neural network is working outside of its specification, which can then be used to either involve human operator or to fall back into a "safe state". Thus, solving this problem is of high importance for trustworthy ML systems. Many approaches have been proposed for OOD detection, (Hendrycks & Gimpel, 2017 Liang et al.
2018
Lee et al.] [0188 bj Hendrycks et al. 2019 Ren et al.) 2019| Hein et al. [209 Meinke &
Hein
2020; Chen et al. 2020; Papadopoulos et al. 2021 Macèdo & Ludermir 2021] Macèdo et al.
2021 and one of the currently best performing methods enforces low confidence during training
("outlier exposure" (OE)) on a large and diverse set of out-distribution images (Hendrycks et al.
2019) which leads to strong separation of in- and out-distribution based on the confidence of the classifier. Crucially, this also generalizes to novel test out-distributions.
However, current OOD detection methods are vulnerable to adversarial manipulations, i.e. small adversarial modifications of OOD inputs lead to large confidence of the classifier on the manipulated samples (Nguyen al. 2015: Hein et al. 2019 Sehwag et al.) 2019). While different methods for adversarilly robust OOD detection have been proposed (Hein et al., 2019 Schwag et al.) 2019
Meinke & Hein 2020 Chen al. 2020 Bitterwolf et al. 2020) there is little work on provably
adversarially robust OOD detection Meinke & Hein 2020;
Bitterwolf et al. 2020; Kopetzki et al.)
2020 Berrada et al. 2021). In CCU (Meinke & Hein 2020) they append density estimators based on Gaussian mixture models for in- and out-distribution to the softmax layer. By also enforcing low confidence on a training aut-distribution, they achieve similar OOD detection performance to (Hendrycks et al., 2019) but guarantee that the classifier shows decreasing confidence one moves away from the training data. However, for close nn-distribution inputs this approach yields no guarantee as the Gaussian mixture models are not powerful enough for complex image classification

---

Under review as a conference paper at ICLR 2022
Table ProoD combines desirable properties of existing (adversarially robust) OOD detection methods. It has high test accuracy and standard OOD detection performance (as (Hendrycks et al. 2019)) and has worst-case guarantees if the oit-iistrinttion samples are adversarially perturbed in an Toc neighborhood to maximize the confidence (see Section 4.2). Similar to CCU (Meinke & Hein 2020 it avoids the problem of asymptotic overconfidence far away from the training data.
OE
CCU
ACET/ATOM
GOOD
ProoD
High accuracy
High clean OOD detection performance
Adv. OOD loo robustness
Adv. OOD (sw -cetiificates
Provably not asympt. overconfident
tasks. In (Kisstadii et al. 2020a b) similar asymptotic guarantees are derived for Bayesian neural networks but without any robustness guarantees. In (Kopetzki et al. (200) they apply randomized smoothing obtain guarantees wrt. lg-perturbations for Dirchlet-based models (Malinin & Gales 2018: 2019; Sensoy et al.] 2018) which already show quite some gap in terms of AUC-ROC to
SOTA OOD detection methods even without attacks. Interval bound propagation (IBP) (Gowal et al. 2018 Mirman et al.] 2018 Zhang et al., 2020) Jovanović et al.) 2021 y has been shown to be one the most effective techniques certified adversarial robustness on the in-distribution when applied during training. In GOOD (Bitterwolf et al. 2020) they use IBP to compute upper bounds on the confidence in an loe -eighboohooo of the input and minimize these upper bounds on a training out- nut-distribution. This yields classifiers with pointwise guarantees for adversarially robust OOD detection even for "close" out-distribution inputs which generalize to novel OOD test distributions. However, the employed architectures of the neural network are restricted to rather shallow networks as otherwise the bounds of IBP are loose. Thus, they obtain low classification accuracy which is far from the state-of-the-arr, e.g. 91% CIFAR- 10, and their approach does not scale to more complex tasks like ImageNet. In particular, despite its low accuracy the employed network architecture is quite large and has higher memory consumption than ResNet50. Moreover, one does not get any guarantees on the asymptotic behavior the classifier as one moves away from the training data.
The authors of (Berrada al. 202T) use SOTA verification techniques (Dathathri al., 2020) and
get guarantees for OOD detection wrt. -erttueturaations for ACET models (Hein et al. 2019) that were not specifically trained to be verifiable but the guarantees obtained by training the models via IBP in (Bitterwolf et al. 2020) are significantly better.
In this paper we propose ProoD which merges certified binary discriminator for in-versus out- distribution with classifier for the in-distribution task in a principled fashion into joint classifier. This combines the advantages of CCU (Meinke & Hein 2020) and GOOD (Bitterwolf et al. (0200 without suffering from their downsides. In particular, ProoD simultaneously achieves the following:
Guaranteed adversarially robust OOD detection via confidence upper bounds on lox -ball around OOD samples.
Additionally, it provably prevents the asymptotic overconfidence of deep neural networks.
It can be used with arbitrary architectures and has no loss in prediction performance and standard OOD detection performance.
Thus, we get provable guarantees for adversarially robust OOD detection, fix the asymptotic over- confidence (almost) for free as we have (almost) loss in prediction and standard OOD detection performance. We qualitatively compare the properties of our model to prior approaches in Table [
2 PROVABLY ROBUST DETECTION OF OUT-OF-DISTRIBUTION DATA
In the following we consider feedforward networks for classification, ʃ Rd R with K classes defined with z (0) x as
の = _ (WW)z(-) + ((0) 1= 1,,,... L 1-1, f(x) w(1) (0)))
2

---

Under review as a conference paper at ICLR 2022
High confidence
Classifier p("dog"(a: (2, 99.9%
on in-Distribution
In-Distribution R.mgNer
pl dood (e) 98.4%
Certified
Discriminator
p(taa) 98.3%
vs.
Guaranteed confidence adversanal COD
Worst-Case Out-Distributin B
MAX mazt PLU x')
55-47%
Not R. R.tggNer
Certitied
Discriminator Out
TAX
p(l
Figure I: ProoD's Architecture: Our method produces high confidence on the in-distribution sample Imggmttt) using Eq. (2). It also provides an upper bound on the worst-case confidence over all images within neighborhood of the shown OOD sample (Openlmages) using Eq. B3). Bounding the confidence on this infinite set of OOD samples possible without any loss in accuracy.
where E N is the number of layers, W(( and ((0) are weights and biases and σ() is either the ReLU leaky ReLU activation function of layer 1. We refer to the output of f as the logits and get
efy(*)
a probability distribution over the classes via p (x) Σ ctat for y = 1,...,K, K. We define the
confidence as (fn(([(x)) maxy=1...... p(ylar).
2.1 JOINT MODEL FOR OOD DETECTION AND CLASSIFICATION
In our joint model assume that there exists an in- and out-distribution where the out-distribution samples are unrelated to the in-distribution task. Thus, we can formally write the conditional distribution the input
p(y|>) = p(y| i) ()p(i)z) o)p(oo))
where p(ilæ) is the conditional distribution that sample r belongs to the in-distribution and Oyy| the conditional distribution for the in-distribution. We assume that OOD samples unrelated and thus maximally un-informative to the in-distribution task, i.e. we fix p(ylæ, o) = K- (Meinke
Hein 2020 they further decomposed p(i|z) p(z))))((3) and used Gaussian mixture models to p(a)
estimate p(a with fixed p(i) p(o) - As the properties of the Gaussian mixture model can be directly controlled in comparison to a more powerful generative model such as VAEs, this allowed (Meinke & Hein 2020) to prove that p(y/(z) (z) becomes uniform over the classes if is far away from the training data. However, the downside of the Gaussian mixture models that this approach yields guarantees for close out-distribution samples. Instead in this paper directly learn ((il which results in a binary classification problem and we train this binary classifier in a certified robust fashion wrt. an loo -threat model that even adversarially manipulated OOD samples are detected. order to avoid confusion with the multi-class classifier, we will refer to p(i]ær as a binary discriminator. In an -bal of radius around Rd and for all y we have the upper bound (also see App.
1 p()l=') 1 -
K -
max p(y))') max ||ا-=||| Se
K p(ilz')) (1-0(i)))
max
p(i)x')
(3)
J| ||'->|| =
so we can defer the certification "work" to the binary discriminator. Using particular constraint on the weights of the binary discriminator, we get similar asymptotic properties as in (Meinke & Hein 2020) but additionally get certified adversarial robustness for close out-distribution samples as in Bitterwolf al., 2020). In contrast (Bitterwolf et al., 2020) this comes without loss in test accuracy or non-adversarial OOD detection performance in our model the neural network used for the in-distribution classification task p(yl.a ar, i) independent the binary discriminator. Thus, we have the advantage that the classifier can arbitrary deep neural networks and is not constrained to certifiable networks. We call our approach Provable out-of-Dins of-Distributton detector (ProoD) and visualize its components in Figure
Certifiably Robust Binary Discrimination of In- versus Duttristribttinn The first goal is to get certifiably adversarially robust OOD detector /(il.:). We train this binary discriminator independent

---

Under review as a conference paper at ICLR 2022
of the overall classifier the training schedules for certified robustness are incompatible with the standard training schedules of normal classifiers. For this binary classification problem we use a logistic model p(ilar) I+c stoy, where 9 Ed IR are logits of neural network (we denote the weights and biases of g by Wa and b, in order discriminate it from the classifier f introduced in the next paragraph). Let (z+, yr) be our un-distribution training data (we use the class encoding +1 for the in-distribution and - for out-distribution) and (z+) be our training out-distribution data. Then the optimization problem associated to the binary classification problem becomes:
M
1 N
log 1 1 e -a(aa)) (1
log + ed(zu)),
min
+
M
W w
<0
where we minimize over the parameters of the neural network under the constraint that the weights of the output layer W/(4a) are componentwise negative and g(2) max (2,e) g(u) is upper bound the output of g around OOD samples for given Lp 1p-threat model (2(2,e) [0, 1 ||u zllp S eJ. In this paper always use an -thrett model. This upper bound could, in principle, be computed using any certification technique but we will use interval bound propagation simple, fast and has been shown to produce SOTA results (Gowal et al. 2018). Note that this is not standard adversarial training for binary classification problem here we have an asymmetric situation: we want to be (certifiably) robust to adversarial manipulation on the out-distribution data but not the in-distribution and thus the upper bound is only used for dut-distribution samples. The negativity of the output layer's weights W is enforced by using the parameterization (Wof-))) - componentwise and optimizing over hy- In Section we show how the negativity of W; (L)))) allows us to control the asymptotic behavior of the joint classifier.
{u e
(IBP) since
For the reader's 's convenience we quickly present the upper ((4) and lower z(()) bounds on the output of layer/ in a feedforward neural network produced by IBP:
7 ( ((O (1-1) + bb)), 1 = (w!) (1-1) ((')((--))) + bb)): (5)
where W+ (0, W) and W_ = min (0, W) (min/max used componentwise). For an L0 threat model one starts with the upper and lower bounds for the input layer xz()) = + and r = x
and then iteratively computes the layerwise upper and lower bounds I(0), x() which fulfill
(1)
S
min ar -
(x')
max || Se Se
(x)) S +
(6)
While in (Bitterwolf et al., 2020) they also used IBP to upper bound the confidence of the classifier this resulted in bound that took into account all O(K2) logit differences between all classes. In contrast, our loss in Eq. 4 is significantly simpler as we just have a binary classification problem and therefore only need single bound. Thus, our approach easily scales to tasks with large number of classes and training the binary discriminator with IBP turns out to be significantly more stable than the approach in (Bitterwolf al. 2020)) and does not require many additional tricks.
(Semi)-Joint Training of the final Classifier Given the certifiably robust model p(i]z) for the binary classification task between in- and out-distribution, we need to determine the final predictive distribution p(y|2) in Eq. (2). On top of the provable OOD performance that we get from Eq. (3). we also want to achieve SOTA performance on unperturbed OOD data. In principle we could independently train model for the predictive in-distribution task p (z, i), e.g. using outlier exposure (OE) (Hendrycks et al., 2019) or any other state-of-the-an OOD detection method and simply combine it with our p(i(z). While this does lead to models with high OOD performance that also have guarantees, it completely ignores the interaction between p(i]æ: and p(yl<,, i) during training. Instead we propose to train p(v) j. i) by optimizing our final predictive distribution (z) Note that in order retain the guarantees of p(( we only train the parameters of the neural network f: Rd + K and need to keep p(i]z) resp. g fixed. Because g stays fixed we call this semi-joint training. We use OE (Hendrycks et al. 2019 for training p (z) with the cross-entropy loss and use
4

---

The loss in Eq. 44) implicitly weighs the in-distribution and worst-case aut-distribution equally, which amounts the assumption p(t) = 2 = p(o). This highly conservative choice simplifies training the binary discriminator but may not reflect the expected frequency of OOD samples at test time and in effect means that (x)) tends to be quite low. This typically yields good guaranteed AUCs but can have a negative impact the standard out-distribution performance. In order better explore the Under review as a conference paper at ICLR 2022
the softmax-funm ffunction in order to obtain the predictive distribution (
Ev efets) from
N
M
K
1
min
log (p(u-lzr))
log (p(112)))
N
M
K
==1
1=1
1
min
log
+ K
(1 -(ilor))
M
log 1/(11..01pp1133 + [ B(ilza))
M
where the first term is the standard cross-entropy loss on the in-distribution but now for our joint model for P(yl z) and the second term enforces uniform confidence on out out-distribution samples. In App. we show that semi-joint training leads stronger guarantees than separate training.
trade-off of guaranteed and standard OOD detection, we repeat the above semi-joint training with different shifts of the offset parameter in the output layer
6' b(L +
where A 2 0 leads to increasing p(i]z)) This shift has a direct interpretation terms of the probabilities p(1) and p(o). Under the assumption that our binary discriminator y is perfect, that is
p(x(i)p(i)
p((xx) = p(z)))(()) p(z(o)p(O) 1 1+
PEOPO p(a
1 --
then it holds that e9(x) p(ali)(())
p(z)o)p(o)* change of the prior probabilities p(1) and p(o) without
changing p(zlt) and p(x(o) then corresponds to a novel classifier
@(x) p())ii)p(i) p(=l))(()) p(o)p(i) p(z(o)p(o) p(zlo)p(O) p())(())
p(o))(i)) p())(O))
ea(4), e
with A= log
(10)
Note that p(1) > p(i) corresponds to positive shifts. In a practical setting, this parameter can be chosen based the priors for the particular application. Since such priors are available in case we determine a suitable shift by evaluating on the training out-distribution, see Section 4.2 for details. Please note that we explicitly do not train the shift parameter since this way the guarantees would get lost as the classifier implicitly learns a large A in order to maximize the confidence on the in-distribution, thus converging to a normal outlier exposure-tyre type classifier without any guarantees.
GUARANTEES ON ASYMPTOTIC CONFIDENCE
In this section we show that our specific construction provably avoids the issue of asymptotic overconfidence that was pointed out in (Hein et al., 2019). Note that the resulting guarantee (as stated in Theorem I is different from and in addition to the robustness guarantees discussed in the previous section (see Eq. (3). The previous section dealt with providing confidence upper bounds on neighborhoods around OOD samples whereas this section deals with ensuring that a classifier's moves away from all training data.
confidence decreases asymptotically
We note that a ReLU neural network f IRd + g as defined Eq. L using ReLU or leaky ReLU as activation functions, potential max-or average pooling and skip connection yields a piece-wise affine function (Arora et al. 2018 Hein et al. 2019). ie. there exists a finite set of polytopes Q, C Rd with 1, ... R such that U Q, R and f restricted to each of the polytopes is affine function. Since there are only finitely many polytopes some of them have to extend to infinity and on these the neural network is essentially affine classifier. This fact has been used in (Hein et al.
5

---

Under review as a conference paper at ICLR 2022
2019) to show that ReLU networks are almost always asymptotically overconfident in the sense that one moves to infinity the confidence of the classifier approaches 1 (instead of converging to 1/ as in these regions the classifier has never seen any data). The following result taken from (Hein et al.) 2019) basically says that as one moves to infinity by upscaling a vector one eventually ends up in polytope which extends to infinity.
Lemma 1 (Heii et al. 2019)). Let {Qr) be the set of convex polytopes on which ReLU- network f: 1 - IRA is an affine function, that is for every k E (1,.... R} and x E Qs there exists V E Rkxd and c² E RRK such that f() v + c². For any x E R with o 0 there exists E R and t E {I,...,... RJ such that Bx E frr all 3 2 a.
The following theorem now shows that, opposite to standard ReLU networks (Hein et al. 2019). our proposed joint classifier gets provably less confident in its decisions as one moves away from the training data which is a desired property of any reasonable classifier.
Theorem 1. Let x E Rd with x + O and let g R. + R be the ReLU-network of the binary discriminator and denote by {Qr R the finite set of polytopes such that g is affine these polytopes which exists by Lemmal ! Denote by Qi the polytope such that Ba E for all B 2 a and let s.ll ())))1(2)) = U2 + d with U IR_-- and d TR_ be the output of the pre-logit layer of g for 2 E Qr U' # O, then
1 K
lim p(y]3x) 8
The proof is in App. In App. A we that the condition UR & 0 is not restrictive, this property holds in all cases where we checked it for our joint classifier. The negativity condition on the weights Wr(4-)) of the output layer of the in-vs. out-distribution discriminator is crucial for the proof. This condition may seem restrictive, but we did not encounter any negative influence of this constraint on test accuracy, guaranteed or standard OOD detection performance. Thus the asymptotic guarantees come essentially for free. In (Meinke & Hein 2020) they derived non-asymptotic guarantees and it would be relatively easy to also achieve this for the joint classifier via a decay factor for p(i|z) that depends on the distance to the training data but we prefer not to enforce this explicitly.
4 EXPERIMENTS
4.1 TRAINING OF PROOD
We provide experiments on CIFARIO, CIFARI00 (Krzhevsky & Hinton 2009) and Restricted Imagenet ImgNet) (Tsipras al. 2018). J. The latter consists of images from the ILSVRC2012
subset of ImageNet (Deng et al. 2009 Russakovsky et al. 2015) belonging to 9 types of animals.
Training the Binary Discriminator We train the binary discriminator between in-and out- distribution using the loss in Eq. (4) with the bounds over an ball of radius e 0.01 for the out-distribution following (Bitterwol et al.) 2020). We use relatively shallow CNNs with only 5 layers plus pooling layers, see App. D For the training out-distribution, we could follow previous work and use 80M Tiny Images (Torralaa et al. 2008) for CIFAR10 and CIFAR100. However, there have been concerns over the use this dataset Birhanee & Prabhu 2021 because of offensive class labels. Although we do not use any of the class labels, we choose to use OpenImages (Kuznetsova et al.) 2020) as training OOD instead. In order to ensure a fair comparison with prior work we also present results that were obtained using 80M Tiny Images in App. D For ImgNeet we use the ILSVRC2012 train images that do not belong to ImggNet as training out-distribution (NotR.ImgNet)
Semi-Joint Training For the classifier we use ResNet18 8 architecture on CIFAR and ResNer50 on ImgNet. Note that the architecture of our binary discriminator is over an order of magnitude smaller than the one in (Bitterwolf et al., 2020) (1 LM instead of 135MB) and thus the memory overhead for the binary discriminator is less than a third of that of the classifier. All schedules, hardware and hyperparameters are described in App- D. As discussed in Section 2.1 when training the binary discriminator one implicitly assumes that in- and (worst-case) out-distribution samples equally likely. seems very unlikely that one would be presented with such a large number of OOD samples in practice but as discussed in Section 2.1 can adjust the weight of the losses after training

---

Under review as a conference paper at ICLR 2022
0.99
0.94
0.98
0.97
DE
Proo
0.3
DE
ProoD
0.92
DE
ProeD
0.96
GOODB0
GOODB0
GOODBO
0.90
0.95
0.88
0.0
2 Bias
Shift
Figure 2: (Provable) OOD performance depends on the bias: Using CIFARIO as the in- distribution and the test set of OpenImages as OOD we plot the test accuracy, AUC and GAUC (see Section 422 as function of the bias shift A (see Eq. (8). For small A the AUCs tend to be worse than for the OE model, but small bias shifts also provide stronger guarantees so some trade-off exists.
the discriminator (but before training the classifier) by shifting the bias bg (f.v) in the output layer of the binary discriminator. We train several ProoD models for binary shifts in {0, 2, 4, 4,5, 6) and then evaluate the AUC and guaranteed AUC (see 4.2) on subset of the training out-distribution OpenImages (resp. NotR, ImgNe). For all bias shifts we use the same fixed provably trained binary discriminator and only train the classifier part. As our goal is to have provable guarantees with minimal or no loss on the standard OOD detection task, among all solutions which have better AUC than outlier exposure (OE) (Hendrycks et al., 2019) we choose the one with the highest guaranteed AUC on OpenImages (on TO/CIARR 100) resp. IogRe (on ImgNet.. If none of the solutions has better AUC than OE on the training out-distribution we take the one with the highest AUC. We show the trade-off curves for the example of CIFARIO in Figure The corresponding figures for CIFAR 100 and I.IgNeet can be found in App. D
4.2 EVALUATION
Setup For OOD evaluation for CIFAR10/100 we use the test sets from CIFAR100/10, SVHN NNet zer et al. 2011), the classroom category of downscaled LSUN (Yu et al. 2015) (LSUN_ CR) as well as smooth noise suggested in (Hein et al., 2019) and described in App. D For I.IgNeet we use
Flowers Nilsback & Zisserman 2008). FGVC Aircraft (Maji et al. 2013), ) Stanford Cars (Krause
et al. 2013) and smooth noise test out- nut-distributions. Since the computation of adversarial ACCs (next paragraph) requires computationally expensive adversarial attacks, we restrict the evaluation on the out-distribution fixed subset of 1000 images (300 in the case of L-SUN_ [RN for the CIFAR experiments and 400 for the R. ImgNet models. We still use the entire test set for the in-distribution.
Guaranteed and Adversarial AUC We use the confidence of the classifier as the feature to discriminate between in- and out-distribution samples. While in standard OOD detection uses the area under the receiver-operator characteristic (AUC) to measure discrimination of in- from out-distribution, BBtterwolf et al. (220)) introduced the worst-case AUC (WCAUC) which is defined as the minimal AUC one can achieve if all out-distribution samples are allowed to be perturbed to reach maximal confidence within a certain threat model, which in our case an loc -bal of radius e. The AUC and WAUC of a feature h Rd + R is defined as:
AUCA(PL,
E. [IAG]S)())] WCAUC
E x-
1
1
max
M(a')
(11)
|====
20-22
where P1. P2 in-resp. out-distribution and with slight abuse of notation the indicator function 1 returns 1 if the expression in its argument is true and 0 otherwise. For all but one of our baselines, the OOD detecting feature h is the confidence of the classifier. Since the exact evaluation of the WCAUC is computationally infeasible, we compute an upper bound and lower bound on the WCAUC by
finding ((z) s max h(2²) < h(z). We find the upper bound on the WCAUC, the adversarial
| Se
AUC (AAUC), by maximizing the confidence using adversarial attack inside the l ball (i.e. finding an /)) and we compute lower bound on the WCAUC, the guaranteed AUC (GAUC), by using upper on the confidence inside the log ball via IBP (i.e. N). For non-provable methods, non-trivial
7

---

Under review as a conference paper at ICLR 2022
upper bound h < oc is available so their GAUCs are always 0. Note that our threat model is different from adversarial robustness on the in-distribution which neither our method nor the baselines pursue.
Gradient obfuscation (Papernot et al., 2017 Athalye et al.) 2018) poses significant challenge for the evaluation of AAUCs (Bitterwolf et al. (0200) so we employ an ensemble of different versions of projected gradient descent (PGD) (Madry al., 2018) as well as Square Aquare/Attckk (Andriushchenko eet al. 2020 with 5000 queries. We use APGD (Croce & Hein 2020) (except on RimgNet, due to memory leak) with 500 iterations and 5 random restarts. We also use 200-step PGD attack with momentum of 0.9 and backtracking that starts with a step size of 0.1 which is halved every time a gradient step does not increase the confidence and gets multiplied by 1.1 otherwise. This PGD is applied to different starting points: i) a decontrasted version of the image, i.c. the point that minimizes the loc -diittannce to the grey image 0.5 I within the threat model, ii) 3 uniformly drawn samples from the threat model and iii) 3 versions of the original image perturbed by Gaussian noise with 0 = 10-4 and then clipped to the threat model. We always clip to the box [0. 1 at each step of the attack. For all attacks and all models we directly optimize the final score that is used for OOD detection. Using different types of starting points is crucial for strong attacks on these OOD points, as some models have precisely zero gradient on many OOD samples.
Baselines We compare to a normally trained baseline (Plain) and outlier exposure (OE), both trained using the same architecture and hyperparameters as the classifier in ProoD. For both ATOM and ACET we found the models' OOD detection to be much less adversarially robust than claimed in (Chen et al. 2020) (see App. E) we retrained their models using the same architecture, threat Running these adversarial training procedures on ImageNet resolution is infeasibly expensive. For GOOD also retrain using Openlmages as training OOD with the code from (Bitterwolf et al., 2020) (comparisons with their pre-trained models can be found in App. D Since they are only available on CIFARIO. I0, we attempted to train models on CIFAR 100 using their code and the same hyperparameters and schedules as they used for CIFARIO. This only lead to models with accuracy below 25%. so we do not include these models in our evaluation. Since CCU was already shown to not provide benefits over OE on OOD data that not very far from the in-distribution (e.g. uniform noise) (Meinke & Hein 2020) Bitterwolf et al. 2020) we do not include it as a baseline. We also evaluate the of the provable binary discriminator (ProoD-Dise) that we trained for ProoD, Note that this is not a classifier and so it is included simply for reference. All results are shown in Table 2
model and traninn out-distribution with their original code (for CIFAR1O/100
Results Prool) achieves non-trivial GAUCs on all datasets. As was also observed in (Bitterwolf et al., 2020). this shows that the IBP guarantees not only generalize to unseen samples but even to unseen distributions. In general the gap between our GAUCs and AAUCs is extremely small. This shows that the seemingly simple IBP bounds can be remarkably tight, as has been observed in other works (Gowal et al. 2018 Jovanović et al. 2021). It also shows that there would be very little benefit in applying stronger verification techniques like (Cheng et al. 2017 Katz et al., 2017 Dathathri et al. 2020) in ProoD. The bounds are also much tighter than for GOOD, which is likely due to the fact that for GOOD the confidence is much harder to optimize during an attack because it involves maximizing the confidence in an essentially random class.
For CIFAR10, on 3 out of 4 out-distributions ProoD's GAUCs are higher than ATOM's and ACET's AUUCs, i.e. our model's provable adversarial robustness exceeds the SOTA methods' empirical adversarial robustness in these cases. Note that this is not due to our retraining, because the authors' pre-trained models perform even more poorly (as shown in App. D. On CIFARION ProoD's guarantees are weaker and ATOM produces strong AAUCs. However, we observe that training both ACET and ATOM can produce inconsistent results, i.c. sometimes almost no robustness is achieved. For the successfully trained robust ATOM model on CIFAR 100 we observe drastically reduced accuracy. Due to the difficulty in attacking these models, is not unlikely that a more sophisticated attack could produce even lower AAUCs. Combined with the fact that both ACET and ATOM rely on expensive adversarial training procedures we argue that using ProoD is preferable in practice.
On CIFARIO we see that ProoD's GAUCs are comparable to, if slightly worse than the ones of both GOOD and GOOD100- Note that although the presented GOOD models are retrained, the same observations hold true when comparing to the pre-trained models (see App. E). However, point out that ProoD achieves this while retaining both high accuracy and OOD performance, both of which are lacking for GOOD. It is also noteworthy that the GOOD models" memory footprints
want to
8

---

Under review as a conference paper at ICLR 2022
Table OOD performance: For all models we report accuracy on the of the in-distribution and AUCs, guaranteed AUCs (GAUC), adversarial AUCs (AAUC) for different test out-distributions. The radius of the -blll for the adversarial manipulations of the OOD data is 0.01 for all datasets. The bias shift A that was used for ProoD is shown for each in-distribution. The AAUCs and GAUCs for ProoD tend to very close, indicating remarkably tight certification bounds.
In: CIFARIO
CIFARI00
SVHN
LSUN CR
Smooth
Acc
AUC GAUC
GAUC AAUC|
AUC
GAUC AAUC AUC
GAUC AAUC
Plain
95.01
90.0
0.0
0.7
93.8
0.0
0.3
93.1
0.0
0.5
98.0
0.0
0.7
OE
94.91
91.1
0.0
0.9
97.3
0.0
0.0
100.0
0.0
2.7
99.9
0.0
1.5
ATOM
93.63
78.3
0.0
21.7
94.4
0.0
24.1
79.8
0.0
20.1
99.5
0.0
73.2
ACET
93.43
86.0
0.0
4.0
99.3
0.0
4.6
89.2
0.0
3.7
99.9
0.0
40.2
GOOD 80
87.39
76.7
47.1
57.1
90.8
43.4
76.8
97.4
70.6
93.6
96.2
72.9
89.9
GOOD
86.96
67.8
48.1
49.7
62.6
34.9
36.3
84.9
74.6
75.6
87.0
76.1
78.1
PrsoD-Disc
62.9
57.1
57.8
72.6
65.6
66.4
78.1
71.5
72.3
59.2
49.7
50.4
ProoD 3
94.99
89.8
46.1
46.8
98.3
53.3
54.1
100.0
58.3
59.7
99.9
38.2
38.8
In: CIFAR100 110
CIFAR10
SVHN
LSUN CR
Smooth
Ace
AUC GAUC AAUC|
AUC
AUC
GAUC
AAUC|
AUC GAUC AAUC AUC GAUC AAUC
Plain
77.38
77.7
0.0
0.4
81.9
0.0
0.2
76.4
0.0
0.3
86.6
0.0
0.4
OE
77.25
77.4
0.0
0.2
92.3
0.0
0.0
100.0
0.0
0.7
99.5
0.0
0.5
ATOM
68.32
78.3
0.0
50.3
91.1
0.0
67.0
95.9
0.0
75.6
98.2
0.0
80.7
ACET
73.02
73.0
0.0
1.4
97.8
0.0
0.7
75.8
0.0
2.6
99.9
0.0
12.8
ProoD-Disc
56.1
52.1
52.3
61.0
58.2
58.4
70.4
66.9
67.1
29.6
26.4
26.5
ProoD 5
77.16
76.6
17.3
17.4
91.5
19.7
19.8
100.0
22.5
23.1
98.9
9.0
9.0
In: ImgNeet
Flowers AUC GAUC AAUC
FGVC
Cars
Smooth GAUC AAUC
Acc
AUC GAUC
AUC
AAUC
AUC
GAUC AAUC AUC
Plain
96.34
92.3
0.0
0.5
92.6
0.0
0.0
92.7
0.0
0.1
98.9
0.0
8.6
OE
97.10
96.9
0.0
0.2
99.7
0.0
0.4
99.9
0.0
1.8
98.0
0.0
1.9
ProoD-Disc
81.5
76.8
77.3
92.8
89.3
89.6
90.7
86.9
87.3
81.0
74.0
74.8
ProoD A 4
97.25 96.9
57.5
58.0
99.8
67.4
67.9
99.9
65.7
66.2
98.6
52.7
53.5
*Uses different architecture classifier, see "Baselines" in Section 4.2
over twice large ProoD's. Generally, for ProoD the accuracy comparable OE and the OOD performance is similar or marginally worse. Thus ProoD shows that it is possible to achieve certifiable adversarial robustness on the out-distribution while keeping very good prediction and OOD detection performance. Note that all methods struggle on separating CIFARIO and CIFAR100 when using OpenImages as training OOD (as compared 80M Tiny Images in App.
To the best of our knowledge with ImgNeet we provide the first worst case OOD guarantees on high-resolution images. The GAUCs are higher than on CIFAR which indicates that meaningful certification on higher resolution is more achievable on this task than one might expect. FGVC and Cars may seem simple to separate from the animals in ImgNett but the same cannot be said for Flowers which are difficult to provably distinguish from images of insects with flowers in them.
5 CONCLUSION
We have demonstrated how to combine a provably robust binary discriminator between in- and out-distribution with a standard classifier in order to simultaneously achieve high accuracy, high OOD detection performance as well as worst-case OOD guarantees that are comparable to previous works. Thus, we have combined the best properties of previous work with only small increase in total model size and only single hyperparameter. This suggests that certifiable adversarial robustness on the out-distribution (as opposed to the n--distiibttio)) is indeed possible without losing accuracy. We further showed how in model simply enforcing negativity in the final weights of the discriminator fixes the problem of asymptotic overconfidence in ReLU classifiers. Training ProoD models is simple and stable and thus ProoD provides OOD guarantees that come (almost) for free.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY
All details necessary for reproducing our experiments are given in Section and App- D including hyperparameters and hyperparameter selection. In order to further aid reproducibility we inllude the source code in the supplemental material (github link in final version), together with instructions for how to use We will also provide pre-trained models for public download in the final version.
ETHICS
The only potential ethical concern that we see in this work is our use of the retracted dataset 80M Tiny Images (Torralba al. 2008). However, we clearly explain that the only reason we use the dataset is providing fair comparison to prior work. We firmly believe that our work helps the community move away from the use of this dataset because we provide extensive experiments on different training out-distribution, going as far as retraining several previous baselines on it.
REFERENCES
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficient black-box adversarial attack via random search. In ECCV, 2020.
R. Arora, A. Basuy, P. Mianjyz, and A. Mukherjee. Understanding deep neural networks with rectified linear unit. In ICLR, 2018.
A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security: Circum- venting defenses adversarial examples. In ICML, 2018.
Leonard Berrada, Sumanth Dathathri, Robert Stanforth, Rudy Bunel, Jonathan Uesato, Sven Gowal, M Pawan Kumar, al. Verifying probabilistic specifications with functional lagrangians. arXiv:2102.09477 2021.
Abeba Birhane and Vinay Uday Prabhu. Large image datasets: A pyrrhic win for computer vision? In WACV, 2021.
Julian Bitterwolf, Alexander Meinke, and Matthias Hein. Certifiably adversarially robust detection of out-of-distribution data. NeurIPS, 2020.
Jiefeng Chen, Yixuan Li, Xi Wu, Yingyu Liang, and Somesh Jha. Informative outlier matters: Robustifying disttiibiiiiitoo detection using outlier mining. preprint, arXiv:2006. 15207, 2020.
Chih-Hong Cheng, Georg Nührenberg, and Harald Ruess. Maximum resilience artificial neural networks. In International Symposium on Automated Technology for Verification and Analysis, 2017.
M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. on Computer Vision and Puttern Recognition (CVPR), 2014.
Croce and M. Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In ICML, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment:
Learning augmentation strategies from data. In CVPR, 2019.
Sumanth Dathathri, Krishnamurthy Dvijotham, Alexey Kurakin, Aditi Raghunathan, Jonathan Uesato, Rudy Bunel, Shreya Shankar, Jacob Steinhardt, lan Goodfellow, Percy Liang, et al. Enabling certification of verification cerficattionaanossiic agnostic networks via memory-effiiiett semidefinite programming. In NeurIPS, 2020.
Deng. W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
10

---

Under review as a conference paper at ICLR 2022
Sven Gowal, Krishnamurthy Dvijotham, Robert Stanforth, Rudy Bunel, Chongli Qin, Jonathan Uesato, Relja Arandjelovic, Timothy Mann, and Pushmeet Kohli. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv:1810.12711 2018.
C. Guo, G. Pleiss, Y. Sun, and Weinberger. On calibration of modern neural networks. In ICML, 2017.
M. Hein, M. Andriushchenko, and J. Bitterwolf. Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In CVPR, 2019.
D. Hendrycks and K. Gimpel. A baseline for detecting misclassified and out-of-distribution examples in neural networks. In ICLR, 2017.
D. Hendrycks, M. Mazcika, and T. Dietterich. Deep anomaly detection with outlier exposure. In ICLR, 2019.
Nikola Jovanović, Mislav Balunović, Maximilian Baader, and Martin Vechev. Certified defenses: Why tighter relaxations may hurt training? arXiv:2102.06700, 2021.
G. Katz, C. Barrett, D. Dill, K. Julian, and M. Kochenderfer. Reluplex: An efficient smnt solver for verifying deep neural networks. In CAV, 2017.
Anna-Kathrin Kopetzki, Bertrand Charpentier, Daniel Zügner, Sandhya Giri, and Stephan Günnemann. Evaluating robustness of predictive uncertainty estimation: Are dirichlet-based models reliable? arXiv:2010. 14986, 2020.
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In ICCV vision workshop, 2013.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks. ICML, 2020a.
Agustinus Kristiadi, Matthias Hein, and Philipp Hennig. Fixing asymptotic uncertainty of bayesian neural networks with infinite relu features. arXiv:2010002709, 2020b.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tusse Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Vision, 2020.
K. Lee, H. Lee, K. Lee, and J. Shin. Training confidence-calibrated classifiers for detecting out-of- distribution samples. In ICLR, 2018a.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detecting out-of-distribution samples and adversarial attacks. In NeurIPS, 2018b.
Shiyu Liang. Yixuan Li, and R Srikant. Enhancing the reliability of out-of-distribution image detection neural networks. In ICLR, 2018.
Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- out-or-distribution detection. Advances in Neural Information Processing Systems, 2020.
David Macêdo and Teresa Ludermir. Enhanced isotropy maximization loss: Seamless and high- performance out-of-distribution detection simply replacing the softmax loss. arXiv:2 /4399, 2021.
David Macèdo, Tsang Ing Ren, Cleber Zanchettin, Adriano LI Oliveira, and Teresa Ludermir. Entropic out-of-distribution detection: Seamless detection of unknown examples. IEFE Transactions on Neural Networks and Learning Systems, 2021.
A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Valdu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018.
11

---

Under review as a conference paper at ICLR 2022
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine- Fine-graine grained visual classification of aircraft. arXiv:1306.5151, 2013.
Andrey Malinin and Mark Gales. Predictive uncertainty estimation via prior networks. In NeurlPS, 2018.
Andrey Malinin and Mark Gales. Reverse kl-divergence training of prior networks: Improved uncertainty and adversarial robustness. In NewrIPS, 2019.
Alexander Meinke and Matthias Hein. Towards neural networks that provably know when they don't know. In ICLR, 2020.
M. Mirman, T. Gehr, and M. Vechev. Differentiable abstract interpretation for provably robust neural networks. In ICML, 2018.
Y. Netzer, T. Wang. A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NeurlPS Workshop on Deep Learning and Unsupervised Feature Learning. 2011.
A. Nguyen, J. Yosinski, and J. Clune. Deep neural networks are easily fooled: High confidence predictions for unrecognizable images. In CVPR, 2015.
Maria-Elens Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008.
Aristotelis-Angelos Papadopoulos, Mohammad Reza Rajati, Nazim Shaikh, and Jiamian Wang. Outlier exposure with confidence control for outto--distribution detection. Neurocomputing, 2021.
Nicolas Papernot, Patrick McDanicl, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In ACM ASIACCS, 2017.
Jie Ren, Peter J. Liu, Emily Fertig, Jasper Snoek, Ryan Poplin, Mark A. DePristo, Joshua Dillon, and Balaji Lakshminarayanan. Likelihood ratios for dfttt--distribution detection. In NeurIPS. 2019.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015.
Vikash Sehwag, Arjun Nitin Bhagoji, Liwei Song, Chawin Sitawarin, Daniel Cullina, Mung Chiang, and Prateek Mittal. Better the devil you know: An analysis of evasion attacks using out-of- distribution adversarial examples. preprint, 19Xi::900..01266. 2019.
Murat Sensoy, Lance Kaplan, and Melih Kandemir. Evidential deep learning to quantify classification uncertainty. In NeurlPS, 2018
Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nonparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence, 30(1 1958-199977 2008
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. In ICLR, 2018
Pingmei Xu, Krista A Ehinger, Yinda Zhang, Adam Finkelstein, Sanjeev R. Kulkarni, and Jianx- iong Xiao. Rich feature hierarchies for accurate object detection and semantic segmentation. 104.v:1504006755, 2015.
Fisher Yu, Yinda Zhang, Shuran Song, Ari Seff, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. CoRR, abs/ 503655. 2015.
12

---

Under review as a conference paper at ICLR 2022
Huan Zhang, Hongge Chen, Chaowei Xiao, Sven Gowal, Robert Stanforth, Bo Li, Duane Boning, and Cho-Jui Hsich. Towards stable and efficient training of verifiably robust neural networks. In ICLR, 2020.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6))1452-1444 2017.
13

---

Under review as a conference paper at ICLR 2022
1.0
0.8
Plain
Confidence
Confidence
0.6
0.6
ACET
GOOD80
0.4
0.4
ProeD
0.2
0.2
109 10*
102 alpha
103
10a
109 10a 102
10a
10a
alpha
Figure 3: Left, Asymptotic confidence: We plot the mean confidence in the predicted in-distribution class for different models one moves away from CIFARI00 samples along the trajectories + where n E [-0.5, 0.5) and a 2 0. Only GOOD and ProoD converge to uniform confidence. Right, Adversarial asymptotic confidence: We try to find adversarial directions in which ProoD remains at a constant high confidence, as opposed to converging to low confidence. We plot the maximum of p(i|z:) across 100 adversarially chosen directions of a. Note that p((la)) >> O implies p(y(æ) K
moves further in these directions by factors
A ADVERSARIAL ASYMPTOTIC OVERCONFIDENCE
According the authors of (Hein et al. 2019), under mild conditions, should expect to find asymptotic overconfidence in all ReLU networks and almost all directions. In order to empirically evaluate this, we take different models that were trained on CIFARIO and evaluate their confidence on different CIFARI00 samples. For each sample I we track the confidence, max pkk(2z) along trajectory in uniform noise direction confidence across 100 such trajectories is shown on the left side of Figure 3 Even the models that produce low confidences on the original OOD sample asympotically converge to maximal confidence far away. The only exceptions here are GOOD and ProoD and only ProoD can guarantee that the confidence cannot converge to
an, where
[-0.5, 0.5) and a 2 The mean
However, even though the architecture provably prevents arbitrarily overconfident predictions and Theorem ensures that most directions will indeed converge to uniform, is, in principle, possible to find directions where the confidence ş(i|a:) remains constant if the condition Uz * 0 in Theorem I not satisfied. We attempted to find such directions by running the following type of attack. We start from random point & E [-0.5, 0.5) that project onto sphere of radius 100. We gradient descent (for 20000 steps), maximizing g(x) while projecting onto the sphere at each step (unnormalized gradients with step size 0.1 for the first 10000 steps and 0.01 for the last 10000 steps). We then increase the radius to 1000 and run an additional 20000 steps with step size 0.1. We rescale the resulting direction vector down
4oc -ball of norm 1 and compute the confidence p(ilær) function of the scaling in the adversarial directions. We show the resulting scale-wise maximun over 100 adversarial directions in Figure Note that even the worst-case over 100 adversarially found directions decays to asymptotically. thus empirically confirming the practical utility of Theorem T Note that the value of ((ilar) converging to 0 implies that the confidence of the ProoD model p(yla converges to 10%.
In Figure 3 GOOD also stands out as having low confidence in all directions that we studied. This is because in all the asymptotic regions that we looked at, the pre-activations of the penultimate layer are all negative. directions far away from the data, the confidence does, in fact, remain low. Unfortunately, also leads to gradients that are precisely zero, which is why the same attack can not be applied here. However, there is guarantee that GOOD does not also get in some direction asymptotically overconfident.
one moves outward and these pre-activations only get more negative in all
14

---

Under review as a conference paper at ICLR 2022
Table Architecture: The architectures that are used for the binary discriminators. Each convolu- tional layer is directly followed by ReLU.
CIFAR
I.ImgNet
Conv2d(3, 128)
Conv2d(3, 128)
Conv2d 256)
AvgPool(2)
Conv2d(256, 256)
Cony2d(128, 256),=2
AvgPool(2)
AvgPool(2)
FC(16384, 128)
Conv2d(256, 256)
FC(128, 1)
AvgPool(2)
FC(50176, 128)
FC(128, 1)
B SEPARATE TRAINING FOR PROOD
In Section 2.1
describe semi-joint training of p(ylæ). However, as pointed out in that section, it possible to separately train a certifiable binary discriminator p(ilz) and an OOD aware classifier p(y|ær, i) and to then simply combine them via Eq. (2). We call this method separate training ProoD-S and evaluate it by using an OE trained model for ply i). We show the results in Table 4 where we repeat the results for OE and ProoD for the reader's convenience. Note that OE and ProoD-S must always have the same accuracy on the in-distribution since they use the same model for classification (note that (2) preserves the ranking of p(yl i)).
We see that the AUCs of ProoD-S are almost identical to those of OE. Without almost any loss in performance ProoD-S manages provide non-trivial GAUCs. However, as one would expect, the semi-jointly trained ProoD provides stronger guarantees at similar clean performance. Nonetheless, this post-hoe method of adding some amount certifiability to an existing system may be interesting in applications where retraining deployed model from scratch is infeasible.
Table 4: Separate Training: Addendum to Table 2 showing the AUCs, GAUCs and AAUCs of ProoD-S all datasets. The accuracy must always be identical that of OE and the clean AUCs are also very similar to those of OE. The guarantees are strictly weaker than those provided by the semi-jointly trained ProoD.
In: CIFARIO
CIFARI00 Acc AUC GAUC AAUC|AUC GAUC AAUC|
SVHN
LSUN CR
Smooth
AUC
GAUC AAUCIAUC GAUC AAUC
OE
94.91 91.1
0.0
0.9
197.3
0.0
0.0
100.0
0.0
2.7
199.9
0.0
1.5
ProoD-S A
94.91
89.3
44.7
45.3
97.3
51.8
52.6
100.0
56.7
57.7
99.9
36.7
37.6
ProoD
A A
94.99
89.8
46.1
46.8
98.3
53.3
54.1
100.0
58.3
59.7
99.9
38.2
38.8
In: CIFAR100
CIFARIO AUC GAUC AAUC|AUC GAUC AAUC|
SVHN
LSUN CR GAUC AAUC|AUC GAUC AAUC
Smooth
Ace
AUC
OE
77.25
77.4
0.0
0.2
19..3
0.0
0.0
100.0
0.0
0.7
99.5
0.0
0.5
ProoD-S A
77.25
77.4
17.2
17.3
92.3
19.5
19.6
100.0
22.4
22.6
99.5
9.0
9.1
ProoD A 5
77.16
76.6
17.3
17.4
91.5
19.7
19.8
100.0
22.5
23.1
98.9
9.0
9.0
In: ImgNet
Flowers
FGVC
Smooth GAUC AAUC|AUC GAUC AAUC
Cars
Ace
AUC GAUC AAUC|AUC, GAUC AAUC|
AUC
OF
97.10 96.9
0.0
0.2
199.7
0.0
0.4
99.9
0.0
1.8
198.0
0.0
1.9
ProoD-S A
97.10 96.9
50.1
50.7
99.7
59.7
60.6
99.9
57.9
58.9
98.0
40.8
42.3
ProoD A =4
97.25 96.9
57.5
58.0
99.8
67.4
67.9
99.9
65.7
66.2
98.6
52.7
53.5
C PROOF OF THEOREM 1
Theorem 1. Let x E Idd with x + O and let g Rd - R be the ReLU-network of the binary R the finite set of polytopes such that g is affine these discriminator and denote by R
15

---

Under review as a conference paper at ICLR 2022
polytopes which exists by Lemma 7 let xetxol111 (2))))(2) = Uz + d with U E R 1xd and E R be the output of the pre-logit layer of f Qr U # 0, then
Denote by Qi the polytope such that 3x Qu all 3 2 and
1
lim p(y]3r) 8-x0
K
Proof. We note that with similar argument as in the derivation of 3 it holds
B(yl8x) & B(] + K (1 -((((xx) = K
1
K (())3x +
(12)
K
For all B 820 a it holds that Bz E Q2 so that
p(132)) = 1 + e g(ax)
(wht') "yy/"a,,Bxrrr (BBzd))+b tb2 "
1 1
(x) 2 0 for all a E g it has to hold (BUx + d), 2 0 for all 3 2 and = 1,..., LLL1- This implies that (Ux);; 2 for all I,..,, Pz-1 and since Ux f 0 there has to exist at least one component i' such that (Ux)); > O. Moreover, wr(do) has strictly negative components and thus for all a holds
As Iz (x)
g(3x)= = (WrLk)) U3x d) + B((a) 3 (W.(.), + (wolho, .d) + bbllu)
As (WiL))) g
Ux) < 0 we get lim ((x) = -o and thus
lim p()]5x) = 0. 8
Plugging this into 12 yields the result.
D EXPERIMENTAL DETAILSS
Datasets We use CIFARIO and CIFARI00 (Kizhevssy & Hinton 2009) (MIT license),
SVHN (Netzer al. 2011) (free for non-commercial use), LSUN (Yu al. 2015) (no license),
the ILS22C2012 split of ImageNet (Deng et al.) 2009
Russakovsky et al. 2015) (free for non-
commercial use), FGrC-Air (Maji al., 2013)
(free for non-commercial use), Stanford
Cars Krause et al.) 2013) (free for non con-commereial use), OpenImages v4 (Kuznetsova et al. 2020) (images have a CC BY 2.0 license), Oxford 102 Flower (Nilsback & Zisserman, (0088) (no license) as well as 80 million tiny images (Torral et al. 2008) (no license given, see also App. E). For the
train/test splits we use the standard splits, except on 80M Tiny Images where we treat a random but fixed subset of 1000 images in the first 1, 1,000,000 as our test set. For all datasets that get used as test out-distribution we use a random but fixed subset of 1000 images.
Following (Bitterwolf et al.) (220), the smooth noise that is used is generated as follows. Uniform noise is generated and then smoothed using Gaussian filter with width that is drawn uniformly at random in [1, 2.5). Each datapoint is then shifted and scaled linearly to ensure full range in [0, 1), i.c. x-min(x)
max (x) [m]
Binary Training The architecture that we use for the binary discriminator is relatively shallow (5 linear layers). The architecture shown in Table 3 Our results are fairly robust to the exact choice of architecture and significantly larger models do not necessarily lead to better results as we show in App. D Similarly to (Zhang et al., 2020 Bitterwolf et al. (020). we use long training schedules, running Adam for 1000 epochs, with an initial learning rate of Te 4 that we decrease by a factor of 5 on epochs 500, 750 and 850 and with a batch size of 128 from the in-distribution and 128 from the out-distribution (for R. ImgNet: 50 epochs with drops at 25, 35, 45, batch sizes 32). In order to avoid large losses we also use a simple ramp up schedule for the e used in IBP and downweight the out-distribution loss during the initial phase of training by scalar k. Both and r are increased linearly from 0 to their final values (0.01 and 1, respectively) over the first 300 epochs (for R. Img over the first 25 epochs). Compared to the training of (Bitterwolf al. 2020
16

---

Under review as a conference paper at ICLR 2022
0.79
0.996
CE
DE
0.30
0.78
0.992
0.20
Accuracy
0.77
0,990
0.76
0.10
0.988
ProcD
0.05
0.75
0.906
0.00
Blas Shift
Blas
Blas Shift
0.990
0.6
0.905
OE
OE
0.5
0,980
0.4
Accuracy
0.975
GAlIC
0.3
0,970
0.2
0.965
0.060
ProoD]
0.955
Bias Shift
Bias Shift
Blas Shift
Figure Bias selection for CIFAR100 and RImgNet: Using CIFARI00 (top) and I.ImNeet (bottom) as the in-distribution and the test set of Openlmages (or NotR.Img. respectively) as OOD we plot the test accuracy, AUC and GAUC as function of the bias shift (see Eq.
which sometimes fails, we found that training of the binary discriminator is very stable and even 100 epochs CIFAR would be sufficient, but found that longer training lead slightly better results. Weight decay is set to 5 10-1 but is disabled for the weights in the final layer. As data augmentation we use AutoAugment (Cubuk al., 2019) for CIFAR and simple 4 pixel crops and reflections on I.Igneet. The strict negativity of the weights leads to a negative bias of which can cause problems initialization. All binary
early stage if the (La) is initialized discriminators were trained on single 2080Ti GPUs, managed on a SLURM cluster. Overall, the training of provable discriminator takes around 16h on CIFAR and 44h on I.ImgNet (wall clock time including evaluations and logging on each epoch).
and thus choose
Semi-Joint Training On CIFAR train for 100 epochs using SGD with momentum of 0.9 and learning rate of 0.1 that drops by factor of 10 on epochs 50, 75 and 90 (on R. ImgNet 75 epochs with drops at 30 and 60). For all datasets train using batch size of 128 (plus 128 out-distribution samples in the case of OE). The CIFAR experiments were run on single 2080Ti GPUs. This takes about 4h20min wall clock time. In order fit batches of 128 in-distribution samples and 128 out- ant-distribution samples on R. ImgNet we had to train using 4 V100 GPUs in parallel. Because of batch normalization in multi-GPU training is important not simply stack the batches but to interlace in- and out-distribution samples. The wall clock time was around 15h for the semi-joint training on R. Imgt For selecting the bias we use the procedure described in Section 4 The trade-off curves for the AUC and GAUC on CIFAR100 and I.IgNet given in Figure
E 80M TINY IMAGES AS TRAINING OUT-DISTRIBUTION
The 80M Tiny Images dataset has been retracted by the authors due to concerns over offensive class labels (Birhane & Prabhu (02T). We support the decision of the community to move away from the use of 80M Tiny Images, of OpenImages v4 (Kuznetsova et al. 2020) as a training out-distribution However, since all prior work used this dataset, we present results 80M Tiny Images here in order compare ProoD's performance to prior baselines. We encourage the community to use the results in Table for future comparisons.
choose to train CIFAR models using downscaled version
Our results are shown in Table Apart from the models shown in Table 2 we add here the pre-trained energy-based OOD detector EB from (Liu et al. 2020 additional baseline for clean OOD
17

---

Under review as a conference paper at ICLR 2022
detection. As EB is not trained robustly, expected EB has low AAUCs. In terms of clean OOD detection it performs similar to OE but with worse results on the more difficult OOD detection task CIFARIO 10 vs CIFAR 100 and vice versa. For GOOD
the pre-trained models from (Bitterwoli et al. 2020). For ATOM and ACET we use the pre-trained models from (Chen et al., 2020 Note that these the densenet architecture and were actually trained to withstand attacks in the much stronger threat model of e 255 In the original paper, the authors claim near perfect AAUCs on this task but we can show that this case of gross overestimation of robustness. Even on the easier threat model <= 0.01 that we test in Table their robustness is almost non-existent for both ATOM models and the CIFARI00 ACET model. The fact that their adversarial attacks were unable to find these samples clearly demonstrates that evaluating models adversarially is very difficult and potentially unreliable. Because of this we believe that our guarantees are a valuable contribution to the community.
Table 5: Training with 80M Tiny Images: We repeat the evaluation from Table 2 for models that were trained using 80M Tiny Images as out-distribution instead of OpenImages. Plain is identical before and just repeated for the reader's convenience. Note that the conclusions from the main paper still hold, which indicates that our method is robust to changes in the choice of training out-distribution. For ATOM and ACET we compare to pre-trained models from (Chen et al. 2020). Note that these models show almost no robustness on CIFAR100 despite the far stronger claims in (Chen et al. 2020).
In: CIFARIO
CIFARIO0
SVHN
LSUN CRR
Smooth
Acc
AUC GAUC AAUC|AUC GAUC AAUC|AUC GAUC AAUC|AUC GAUC AAUC
Plain
95.01
0.0
0.6
93.8
0.0
0.1
93.1
0.0
0.5
99822
0.0
0.6
OE
95.53 96.1
0.0
6.0
99.2
0.0
0.4
99.5
0.0
15.2
99.0
0.0
11.3
EB*
95.22 93.8
0.0
2.8
99.3
0.0
0.0
99.5
0.0
6.0
99.4
0.0
3.5
ATOM
95.20 93.7
0.0
14.4
99.6
0.0
8.6
99.7
0.0
40.0
99.6
0.0
18.8
ACETT
91.48
91.2
0.0
80.5
95.3
0.0
87.6
98.9
0.0
95.0
99.9
0.0
98.3
GOOD
90.13
87.2
42.5
63.9
94.2
37.5
67.4
93.3
55.2
83.6
95.3
57.3
88.5
GOOD 100 *
90.14
70.7
54.5
55.0
74.9
56.3
56.6
75.2
61.0
61.6
81.4
66.6
67.5
ProoD-Dre Disc
67.4
61.0
61.7
73.2
65.5
66.4
78.0
72.2
72.7
82.3
71.5
72.9
ProoD A= 3
95.47
96.0
41.9
43.9
99.5
48.8
49.4
99.6
47.6
53.1
99.7
55.8
57.0
In: CIFAR100
CIFARIO
SVHN
LSUN CR AUC GAUC AAUC|AUC GAUC AAUC
Smooth
Acc
AUC GAUC AAUC|AUC GAUC AAUC
Plain
77.7
77.7
0.0
0.3
81.9
0.0
0.2
76.4
0.0
0.3
88,8
0.0
0.5
OE
77.28
83,9
0.0
0.8
92.8
0.0
0.1
97.4
0.0
4.6
97.6
0.0
0.9
EB*
75.70
77.4
0.0
0.8
96.5
0.0
0.0
96.7
0.0
5.9
98.9
0.0
4.3
ATOM
75.06
64.3
0.0
0.2
93.6
0.0
0.2
97.5
0.0
9.3
98.5
0.0
15.0
ACET!
74.43
79.8
0.0
0.2
90.2
0.0
0.0
96.0
0.0
2.1
92.9
0.0
0.3
ProoD-Disc
53.8
50.3
50.4
73.1
69.8
69.9
68.1
63.8
64.0
67.2
63.8
63.9
ProoD =
76.79
80.5
23.1
23.2
93.7
33.9
34.0
97.2
29.6
30.4
98.9
29.7
31.3
Pre-trained WideResnet from (Liu et al.
2020)
pre-trained with *CNN architecture using pre-trained models from [Bitterwolf et al. 2020).
Densenet architecture, using models from (Chen et al.
2020)
F FALSE POSITIVE RATES
Since in a practical setting a threshold for OOD detection ultimately has to be chosen, it can be interesting to study the false positive rate at fixed threshold. is relatively standard to pick the false positive rate at 95% true positive rate (called FPR in Table where low values are desirable. We show the results for all methods and datasets in Table 6 Although ProoD has similarly good performance as OE on this task, fails to give non trivial guarantees. Therefore achieving stronger bounds on the worst-case FPR is interesting task for future work.
18

---

Under review as a conference paper at ICLR 2022
Table 6: False Positive Rates: For all models we report accuracy on the test of the in-distribution and the false positive rate at 95% true positive rate (FPR) (smaller is better). We also show the adversarial FPR (AFPR) and the guaranteed FPR (GFPR) for different test out-distributions. The radius of the -bll for the adversarial manipulations of the OOD data is e 0.01 for all datasets. The bias shift that was used for ProoD is shown for each in-distribution. ProoD struggles to give non-trivial guarantees for the FPR @ 95% on most datasets. However, different from GOOD or ProoD-Dise the
clean performance is generally as good as that of OE
In: CIFARIO
CIFAR 100
SVHN
LSUN_CH
Smooth
Acc
FPR GFPR AFPR
FPR
GFPR
AFPR
FPR
GFPR
AFPR
FPR
GFPR
AFPR
Plain
95.01 56.3 100.0
100.0
40.7 100.0 100.0
46.7
100.0
100.0
10.6
100.0
100.0
OE
94.91
52.2 100.0
99.9
15.4 100.0
100.0
0.0
100.0
99.0
0.0
100.0
99.0
ATOM
93.63 73.4 100.0
98.8
33.9
100.0
100.0
85.3
100.0
100.0
0.0
100.0
86.1
ACET
93.43 65.4 100.0
99.5
3.0 100.0
99.8
62.7
100.0
100.0
0.0
100.0
89.0
GOOD 80
87.39
65.1 100.0
84.8
26.8
100.0
48.8
6.0
100.0
24.7
19.6
100.0
52.8
GOOD 100
86.96 84.8 100.0
99.3
87.9
100.0
99.7
66.0
100.0
99.0
68.6
100.0
98.2
ProoD- D
83.9
87.5
87.2
76.9
84.9
84.2
76.7
85.3
84.7
96.6
98.4 98.4
ProoD A 3
94.99
48.0
99.9
99.7
9.1
100.0
100.0
0.0
100.0
98.7
0.0
100.0 100.0
In: CIFAR100
CIFARIO
SVHN
LSUN CR
Smooth
Acc
FPR
GFPR AFPR|
FPR
GFPR
AFPR
FPR
GFPR
AFPR
FPR
GFPR
AFPR
Plain
77.38
80.1 100.0 100.0
77.3 100.0 100.0
79.0 100.0 100.0
70.0
100.0
100.0
OE
77.25
81.8 100.0 100.0
37.7 100.0 100.0
0.0 100.0
99.7
0.0
100.0
100.0
ATOM
68.32
81.3 100.0 99.6
51.0 100.0 97.4
26.3
100.0
94.7
10.0
100.0
86.0
ACET
73.02
87.9 100.0
100.0
8.2 100.0 100.0
87.0
100.0
100.0
0.0
100.0 98.2
ProoD-Disc
95.9
97.5 97.5
91.3
92.8 92.7
91.7
95.3
95.0
100.0
100.0
100.0
ProoD A =5 5
77.16 82.2
100.0 100.0
37.6
100.0 100.0
0.0
100.0
99.3
3.4
100.0 100.0
In: Img
Flowers FPR
FGVC
Cars
Smooth
Acc
GFPR AFPR
FPR
GFPR
AFPR
FPR
GFPR
AFPR|
FPR
GFPR
AFPR
Plain OE
96.34 55.2 100.0 100.0
48.2
100.0 100.0
75.2 100.0 100.0)
0.0
100.0
100.0
97.10 18.2 100.0 100.0
0.2 100.0
100.0
0.0
100.0
100.0
0.0
100.0
100.0
ProoD-Disc
59.2
65.2 65.0
51.0 67.8 66.5
51.7
63.7
62.3
100.0
100.0
100.0
ProoD A 4
97.25
18.5
100.0
100.0
0.5
100.0
100.0
0.0
100.0
100.0
0.0
100.0
100.0
G ADDITIONAL DATASETS
In addition to the results shown in Table is interesting to study how ProoD performs on additional datasets as well as the test set of the out-distribution it was trained on. For the CIFAR datasets,
we report LSUN crops, LSUN resize, Places365 (Zhou et al. 2017 iSUN (Xu al. 2015).
Textures (Cimpoi et al. 2)14), 80M Tiny Images and uniform noise in Table On RlmgNet we
report uniform noise and the training out-distribution in Table
As in Table 2 the clean performance of ProoD is comparable to that of OE, but it achieves non-trivial GAUC. On CIFAR1O, GOOD100 achieves almost perfect GAUC against uniform noise, which comes at the price of significantly worse clean AUCs on all other out-distributions, see Table Almost all methods achieve very high AAUCs on uniform noise, but is not clear a sufficiently powerful attack could lower those scores significantly. The unusually large gap between the GAUC and AAUC of ProoD would seem to indicate that that might be the case. Surprisingly, the robustness characteristics of both ATOM and ACET vary wildly between the different datasets, sometimes appearing to be perfectly robust and on other datasets displaying no robustness at all. Especially surprising are the relatively low AAUCS the test set of the training out-distribution.
19

---

Under review as a conference paper at ICLR 2022
Table 7: Additional Datasets: We show the AUC, AAUC and GAUC for all models on uniform noise and on the test set of the train out-distribution.
In: CIFARIO
LSUN
LSUN resize
Places365
iSUN
Acc
AUC
GAUC AAUC
AUC
GAUC AAUC|AUC GAUC AAUC)
AUC
GAUC
AAUC
Plain
95.01
95.9
0.0
1.3
95.0
0.0
8.7
89.5
0.0
0.3
94.5
0.0
10.4
OE
94.91
98.7
0.0
0.9
97.4
0.0
6.1
99.9
0.0
2.5
97.6
0.0
9.4
ATOM
93.63
77.3
0.0
12.1
100.0
0.0
98.4
82.6
0.0
22.0
100.0
0.0
98.8
ACET
93.43
89.2
0.0
2.5
100.0
0.0
91.3
88.0
0.0
3.4
100.0
0.0
92.1
GOOD80
87.39
96.5
68.4
89.7
87.4
61.7
69.5
96.8
58.8
90.2
87.1
58.9
70.3
GOOD100
86.96
95.0
86.0
86.7
81.1
67.6
67.9
74.4
59.7
60.9
77.5
63.3
65.2
ProoD-Disc
95.8
94.1
94.2
76.4
70.3
71.5
76.6
71.1
71.5
74.9
69.0
70.3
ProoD A 3
94.99
99.2
82.2
82.3
97.1
57.7
59.2
99.9
58.7
59.6
97.1
56.4
58.1
In: CIFARIO
Uniform
Textures
80M Tiny Images GAUC AAUC|AUC GAUC AAUC)
OpenImages AUC GAUC AAUC
Acc
AUC
GAUC AAUC
AUC
Plain
95.01
97.9
0.0
83.1
92.0
0.0
8.6
91.1
0.0
0.8
84.0
0.0
0.4
OE
94.91
99.9
0.0
98.2
99.9
0.0
13.8
93.6
0.0
0.6
99.7
0.0
2.5
ATOM
93.63
100.0
0.0
100.0
97.5
0.0
74.7
82.8
0.0
26.7
73.4
0.0
22.3
ACET
93.43
100.0
0.0
99.9
98.0
0.0
41.6
89.0
0.0
7.8
81.7
0.0
5.6
GOODS0
87.39
99.1
83.9
98.1
96.1
54.7
89.9
84.7
50.5
66.8
93.6
53.7
85.6
GOOD100
86.96
94.6
86.1
89.3
71.0
49.9
57.1
74.5
56.7
58.2
69.5
54.2
55.6
ProoD-Disc
53.8
46.7
46.7
75.3
69.9
70.5
75.7
69.8
70.9
64.8
59.0
59.6
ProoD A =3
94.99
99.9
35.0
94.7
99.9
57.6
63.1
93.8
57.5
59.0
99.8
47.7
49.7
In: CIFARI00
LSUN
LSUN resize
Places365
iSUN
Acc
AUC
GAUC
AAUC
AUC GAUC AAUC|AUC GAUC AAUC
AUC
GAUC
AAUC
Plain
77.38
84.5
0.0
0.9
78.9
0.0
3.9
75.7
0.0
0.6
79.4
0.0
5.6
OE
77.25
94.5
0.0
1.3
88.2
0.0
2.1
99.7
0.0
1.9
88.3
0.0
3.1
ATOM
68.32
83.4
0.0
66.7
92.1
0.0
71.3
87.0
0.0
62.1
91.7
0.0
73.7
ACET
73.02
81.3
0.0
4.3
100.0
0.0
77.0
74.8
0.0
4.3
100.0
0.0
78.2
ProoD-Disc
81.4
78.8
79.3
70.1
66.7
67.1
71.7
68.2
68.5
69.3
65.6
66.0
ProoD A 5
77.16
94.8
28.5
28.7
86.3
22.6
22.8
99.7
23.6
23.8
87.2
22.1
22.4
In: CIFARI00
Uniform
Textures
80M Tiny Images
OpenImages AUC GAUC AAUC
Acc
AUC
GAUC AAUC
AUC
GAUC
AAUC
AUC GAUC AAUC
Plain
77.38
82.7
0.0
53.0
77.5
0.0
5.3
79.7
0.0
1.2
75.5
0.0
0.7
OE
77.25
99.3
0.0
91.4
99.0
0.0
7.5
80.2
0.0
1.5
99.6
0.0
1.0
ATOM
68.32
100.0
0.0
100.0
90.0
0.0
66.6
90.1
0.0
70.0
83.7
0.0
58.9
ACET
73.02
100.0
0.0
99.9
91.5
0.0
21.0
77.0
0.0
6.8
74.5
0.0
3.0
ProoD-Disc
40.9
37.0
37.4
53.6
50.2
48.1
59.7
56.7
56.9
58.3
54.6
54.8
ProoD 5
77.16
99.7
12.1
87.0
99.1
16.8
22.6
80.5
19.5
19.7
99.7
18.6
20.1
Table 8: Additional Datasets: For RImgNet, we show the AUC, AAUC and GAUC for all models on uniform noise and on the test set the train out-distribution, NotRImgNet..
In: ImgNet
Uniform GAUC
NotR.ImgNet GAUC
Acc
AUC
AAUC
AUC
AAUC
Plain OE ProoD-Disc
96.34
99.3
0.0
74.9
91.7
0.0
0.2
97.10
99.6
0.0
84.6
98.7
0.0
1,2
99.7
99.2
99.3
73.6
69.9
69.9
ProoD A =
97.25
99.8
79.7
95.2
98.6
50.1
51.3
20

---

Under review as a conference paper at ICLR 2022
H
DERIVATION OF EQ. (3)
We restate the derivation in Eq.
in a slightly more verbose form. For all y we have
max
p(yla')
max
plulx', ())(((²))((((()))))))))))))))))) + -b(ala)))
1 K
(13)
x
1| we
p(i|z') + 1 (1-p(i(zx)) p(ilx'))
(14)
|| - 20
K
K
p(il=') 1
(15)
K
K
The incquality is obtained by using the trivial bound p(y|a, 1) < 1. This allows us to compute guarantees the confidence of the entire model while treating the classifier itself entirely black-box. Because of this, there are no restrictions whatsoever on the architecture or training procedure that are used for fitting p (,,
I SIZE ABLATION FOR BINARY DISCRIMINATOR
Since larger models should typically lead to better performance, we investigated the impact that model size has on the performance of our binary discriminator. We retrained ProoD-Dise models with CIFARIO 10 as in-distribution and 80M Tiny images as the out-distribution (since our dataloader is faster than for Openlmages which speeds up training). Since longer schedules only slightly improve results also used shorter schedules with only 300 epochs where and к linearly increase from 0 to 0.01 and 1.0 respectively within the first 100 epochs and the learning rate drops occur at 150, 200 and 250 epochs. As architectures
10 different CNNs with different widths and depths ranging from 5 to 8 layers (for the precise architectures please refer to sizes {S. XL_b, XS, SR, SR2, CI, C3s, C3, C2, C4) in the file prvabll_classi fiers. seerspyyyr... py of the accompanying code). We present scatter plots of the models' their performance on CIFAR 100 and 80M Tiny Images in Figure 5 against their size. Clearly, there no correlation between model size and performance and most differences are rather small, justifying our choice of a fairly small architecture in the main paper.
CIFAR10D
80M Tiny Images
0.62
0.73
0.61
0.72
GAUC
GAUC
0.59
0.70
0.58
0.69
15
16
17
188
14
16
18
19
Log armmss
Lag #Params
Figure 5: Bigger models do not yield better guarantees: We show scatter plots of the GAUC of different architectures against the log of the number of trainable parameters in the model. The orange cross indicates the architecture that is used in the main paper. There is no clear dependence of performance on model size, so it is preferable to use fairly small models.
J GENERALIZATION TO LARGER THREAT MODEL
Since E = 0.01 is a relatively weak threat model we evaluate if ProoD's s guarantees actually generalize to the much stronger robustness. We use the exact same CIFAR models from Table show the results our evaluation in Table
0.031 that is standard in much of the literature adversarial
21

---

Under review as a conference paper at ICLR 2022
Table 9: Generalization to larger et We evaluate all CIFAR models in Table using an e and thus unseen threat model. The provable methods GOOD and ProoD generalize surprisingly well, while neither ATOM nor ACET display any generalization to the larger threat model.
255
In: CIFARIO
CIFARI00
SVHN
LSUN CR
Smooth
Acc
AUC GAUC
AUC
GAUC AAUC
AUC
GAUC AAUC AUC
GAUC AAUC
Plain
95.01
90.0
0.0
0.0
93.8
0.0
0.0
93.1
0.0
0.0
98.0
0.0
0.0
OE
94.91
91.1
0.0
0.1
97.3
0.0
0.0
100.0
0.0
0.1
99.9
0.0
0.0
ATOM
93.63
78.3
1.3
94.4
0.0
1.5
79.8
0.0
0.2
99.5
9.6
ACET
93.43
86.0
0.0
1.1
99.3
0.0
1.1
89.2
0.0
0.8
99.9
0.0
3.8
GOOD80
87.39
76.7
37.5
51.6
90.8
38.6
74.3
97.4
57.6
90.2
96.2
61.1
87.8
GOOD100#
86.96
67.8
39.4
43.5
62.6
29.0
30.9
84.9
67.6
70.7
87.0
63.3
69.2
ProoD-Disc
62.9
44.1
46.1
72.6
52.5
57.1
78.1
56.3
58.9
59.2
34.9
37.2
ProoD A 33
95.15
84.8
39.2
41.0
98.3
46.9
50.8
100.0
50.2
52.7
99.9
30.4
30.6
In: CIFARI00
CIFARIO
SVHN
LSUN CR
Smooth
Ace
AUC
GAUC AAUC|AUC
GAUC AAUC
AUC
GAUC AAUC AUC GAUC AAUC
Plain
77.38
77.7
0.0
0.4
81.9
0.0
0.2
76.4
0.0
0.3
86.6
0.0
0.3
OF
77.25
77.4
0.0
0.2
92.3
0.0
0.0
100.0
0.0
0.7
99.5
0.0
0.5
ATOM
68.32
78.3
0.0
10.4
91.1
0.0
15.2
95.9
0.0
23.0
98.2
0.0
23.5
ACET
73.02
73.0
0.0
1.4
97.8
0.0
0.7
75.8
0.0
2.6
99.9
0.0
3.8
ProoD-Disc
56.1
41.1
43.1
61.0
50.5
51.8
70.4
57.5
58.8
29.6
20.9
20.8
ProoD A =5 5
76.51
76.6
13.7
14.1
91.5
16.9
16.9
100.0
18.1
18.2
98.9
8.1
8.1
In: ImgNeet
Flowers
FGVC
Cars
Smooth
Ace
AUC GAUC AAUC|AUC GAUC AAUC|
AUC
GAUC AAUC AUC GAUC AAUC
Plain OE
96.34
92.3
0.0
0.0
92.6
0.0
0.0
92.7
0.0
0.0
98.9
0.0
0.0
97.10
96.9
0.0
0.2
99.7
0.0
0.0
99.9
0.0
0.0
98.0
0.0
0.0
ProoD-Disc
81.5
60.4
61.4
92.8
78.0
80.8
90.7
76.3
79.2
81.0
47.3
53.7
ProoD)
97.25
96.9
42.8
45.0
99.8
57.0
59.4
99.9
56.0
58.7
98.6
31.6
36.3
*Uses different architecture of classifier, see "Baselines" in Section 4.2
Perhaps surprisingly, ProoD's guarantees generalize remarkably well to the much larger radius on CIFARIO. The same holds for GOOD, as was already observed in (Bitterwoll et al. 2020 On the other hand ATOM and ACET do show any robustness this radius, despite having lower accuracy, lower clean OOD performance and being more expensive to train. On the other hand ProoD's guarantees on CIFAR 100 are quite weak at this radius, but given ATOM's and ACET's low AAUCs here and the fact that GOOD cannot be trained at all on CIFARIOG the results are not worse than for the competitors.
K ERROR BARS
In order be mindful of our resource consumption we restrict the computation of error bars to experiments on CIFARIO. Additionally, because the dataloader much faster we these experiments using 80M Tiny Images as an out-distribution as opposed to OpenImages. We reran experiments using the same hyperparameters times. We computed the mean and the standard deviations for our models for all metrics shown in Table The results are shown in T We see that the fluctuations across different runs indeed rather small. Furthermore, the clean performance of OE and ProoD show no significant discrepancies.
22

---

Under review as a conference paper at ICLR 2022
Table 10: Error Bars: We show the mean and standard deviation of all metries for our CIFARIO models across 5 runs. The tolerances for ProoD's clean performance are very small and yet the differences in clean performance between OE ProoD are not significant.
In: CIFARIO
CIFARI00
SVHN
LSUN CR
Smooth GAUC AAUC
Ace
AUC GAUC AAUC|AUC
GAUC AAUC AUC GAUC AAUC|AUC
Plain
94.91
90.0
0.0
0.6
99399
0.0
0.1
93.4
0.0
0.7
96.7
0.0
1.2
Plain
0.16
0.1
0.0
0.1
1.2
0.0
0.0
0.3
0.0
0.2
2.1
0.0
0.5
OE OE
99.55
96.1
0.0
7.6
199.4
0.0
0.4
99.6
0.0
16.7
99.6
0.0
4.3
0.04
0.1
0.0
1.5
0.1
0.0
0.2
0.1
0.0
3.5
0.3
0.0
3.7
ProoD-Disc ProoD-Disc
67.7
61.6
62.2
75.5
68.6 1.4
69.3
76.5
70.4
70.9
87.2
77.7
78.8
0.7
0.7
0.7
1.7
1.5
1.4
1.7
1.7
3.6
4.3
4.3
ProoD A 3 95.60
96.0
42.2
44.1
19..4
48.6
49.2
99.6
47.1
52.0
99.88
55.2
57.0
ProoD A 3 o
0.11
0.1
0.8
0.8
0.1
0.6
0.6
0.1
1.5
1.9
0.1
2.9
3.4
23


Reviews:
[{"review_id": "tJQimu_Jzlg", "reviewer": "Reviewer_urW9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. Combining OOD detection and classifiers is an interesting idea and the proposed \"OOD-aware\" training can be effective to improve the robustness of the classifiers.\n\n2. Authors provide a confidence bound for the classifier which is useful to develop certifiable robust OOD-aware models.\n\n3. Experiments on the benchmark datasets show that the proposed approach achieves comparatively high accuracy while maintaining a guarantee on the AUC score. \n\n\nWeaknesses:\n\n1. The draft lacks clarity in many aspects. \n\na. In table 1, metrics are not clear until section 3.\n\nb. Figure 1 cannot be interpreted from the caption. It is not clear how the equations in the figure map to equations in the draft. \n\nc. In the experiments section, \"semi-joint training\" requires more details. For example, how are the several models trained with binary shifts? Metrics in figure 2 are not clear from the associated text.\n\n2.  Authors claim to achieve better guarantees than the existing approaches (e.g., CCU, GOOD). However, it is not clear why Theorem 1 entails a tighter bound than Lemma 1 (Hein 2019).\n\n3. The bounds in Eq 5 appear trivial from the computations of W_{+}, and W_{-}. How do these bounds affect the tightness of the bounds in Eq 6.\n\n4. I would expect the joint training would improve the performance of both OOD detection and the classifier. However, as shown in table 2, outlier exposure, which is not the best state of the art, still performs better. What are the authors' comments on this. \n\n5. Authors are encouraged to compare with the more recent OOD detection approaches \n\na. Liu et al., \"Energy-based out-of-distribution detection\"\n\nb. Lee et al., \"Training confidence-calibrated classifiers for detecting out-of-distribution samples.\"", "summary_of_the_paper": "This paper presents an approach to combine OOD detection and classifier to develop a robust OOD detection framework with high classification accuracy. The authors provide confidence bounds for the noise-perturbed and adversarially attacked OOD samples. Experiments are conducted on benchmark datasets and the performance is compared with approaches that provide asymptotic guarantees on robustness. ", "summary_of_the_review": "The proposed approach of combining OOD detection and classifier through joint training is an interesting approach. However, some parts of the paper lack clarity and thus, it is hard to evaluate the contributions. Authors are encouraged to address the comments and I will be willing to reconsider my decision.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Z4o8yaLiIpO", "reviewer": "Reviewer_guNu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this paper has the following strengths:\n\n1. The authors derive the method ProoD in a principled way and prove that the method can prevent the asymptotic overconfidence of deep neural networks; \n\n2.  They perform extensive experiments to evaluate the proposed method and also compare it to existing baselines; \n\n3. The paper is well-written and easy to understand. \n\nHowever, I think this paper has the following weaknesses: \n\n1. The idea of training a discriminator independently via interval bound propagation (IBP) to achieve certified robustness on out-of-distribution samples is not very novel since IBP is an existing technique that can ensure certified robustness. Also, the theoretical result (Theorem 1) that the proposed joint classifier gets provably less confident in its decisions as one moves away from the training data is not entirely novel since it is mainly based on the previous theoretical results in the literature; \n\n2. Since the proposed method ProoD uses IBP in training to get the certified discriminator g, it is expected that it has better certified robustness than other methods that don't use IBP. However, it might be hard to train models using IBP with a large perturbation budget (e.g. use $\\epsilon=8/255$ that is common in previous OOD detection papers such as ATOM). The authors may need to acknowledge this limitation; \n\n3. When the authors retrain ATOM and ACET models, they might need to consider using stronger PGD attacks during training since they use very strong attacks with adaptive step size for the evaluation of AAUC. In ATOM paper, they only use PGD attack with 5 steps and fixed step size. Using stronger PGD attacks with more steps and adaptive step size (e.g. use backtracking) for training ATOM and ACET might lead to better results (or AAUC) under their attacks in evaluation. \n\n4. The performance of the proposed method ProoD is not stable across different in-distribution and OOD datasets. For example, in Table 2, the performance of ProoD on CIFAR-10 vs. Smooth is worse than that of ATOM and GOOD in terms of AAUC metric, and in terms of GAUC metric, the performance of ProoD is also worse than that of GOOD. The GAUC metric for ATOM and ACET seems meaningless since it is a lower bound and is equal to 0. On CIFAR-100 as the in-distribution dataset, it seems the performance of ProoD is usually worse than that of ATOM in terms of the AAUC metric. In Table 7 (in the appendix), on CIFAR-10 vs. Uniform or CIFAR-100 vs. Uniform, the performance of ProoD is also worse than that of existing methods like ATOM and GOOD. So it is unclear whether the performance of ProoD is better than that of existing methods or not. I think the authors should evaluate ProoD on more OOD datasets. In the ATOM paper, they also evaluate OOD detectors on OOD datasets like Textures, Places365, LSUN (resize), and iSUN. I suggest the authors report the performance of ProoD on these OOD datasets and compare it to existing methods. \n\n5. I think the authors should give more details about attacking ProoD. For example, what's the attack objective they use to attack ProoD when evaluating the GAUC and AAUC? Since ProoD combines the classifier and detector, the adaptive attacks should attack both the classifier and the detector. If they only attack the detector when evaluating GAUC and AAUC, the results may not be correct and they need to re-evaluate them. \n\n", "summary_of_the_paper": "In this paper, the authors propose ProoD which merges a certified binary classifier for in-versus out-distribution with a classifier for the in-distribution task in a principled fashion into a joint classifier. They show that ProoD simultaneously achieves three properties: 1) guaranteed OOD robustness via confidence upper bounds on $l_\\infty$-balls around OOD samples; 2) it provably prevents the asymptotic overconfidence of deep neural networks; 3) it can be used with arbitrary architectures and has no loss in prediction performance and standard OOD detection performance. They perform extensive experiments to show the promising performance of the proposed method ProoD.", "summary_of_the_review": "Although this paper proposes a principled method ProoD for robust OOD detection and has some interesting theoretical results, the experiments conducted are not enough to show the effectiveness of the proposed method. As I mentioned, the authors should be more careful in building the baselines and evaluating the methods. Also, they should give more details about the experiments like the attack objectives used. Thus, I think this paper is not ready for publication. \n\n\n***[Post Rebuttal]***\n\nI think the proposed method ProoD is not very novel since it simply combines previous techniques, and the performance of ProoD is not stable across different in-distribution and OOD datasets. Also, in practice, it might be hard to use ProoD since under the False Positive Rate at 95% true positive rate metric, ProoD doesn't have good performance (it might be hard to select a suitable threshold for ProoD). Thus, I keep my original score and think the paper is not ready for publication.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "OzbaXmgyYOT", "reviewer": "Reviewer_6Xmu", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "By far, our main concern regarding the proposed solution is the following: Does the proposed method present (certified or not) adversarial robustness on the **in-distribution data**? For example, which part of the solution prevents a malicious person from using an attack on an **in-distribution example** (i.e., making the classifier believe that a sample belongs to the wrong class) to mislead the primary (i.e., the multiclass) classifier? The proposed binary classifier does not appear to be able to prevent this situation. The below excerpts from the paper make us suspect that the proposed solution does not present adversarial robustness on the **in-distribution data**.\n\n1. _\"In contrast to (Bitterwolf et al., 2020) this comes without loss in test accuracy or non-adversarial OOD detection performance as in our model the neural network used for the in-distribution classification task is independent of the binary discriminator. Thus, we p(yjx; i) have the advantage that the classifier can use arbitrary deep neural networks and is **not constrained to certifiable networks**.\"_\n\n2. _\"Note that this is not standard adversarial training for a binary classification problem as here we have an asymmetric situation: we want to be (certifiably) robust to adversarial manipulation on the out-distribution data **but not on the in-distribution** and thus the upper bound is only used for out-distribution samples.\"_\n\n3. _\"While in (Bitterwolf et al., 2020) they also used IBP to upper bound the confidence of the classifier this resulted in a bound that took into account all O(K2) logit differences between all classes. In contrast, our loss in Eq. (4) is significantly simpler as **we just have a binary classification problem and therefore only need a single bound**.\"_\n\nPlease, notice that ACET [2] presents adversarial robustness on the **in-distribution data**. Moreover, GOOD [3] presents _certified_ adversarial robustness on the **in-distribution data**. If we lose this, making the solution certified adversarial robust on the out-distribution data does not matter anymore, as the attacker may manipulate the solution by simply attacking in-distribution data rather than out-of-distribution data. Considering we are indeed correct, and the proposed solution does not provide certified adversarial robustness on the in-distribution data, **this fact may perfectly explain why the solution, unlike competing approaches, almost does not present loss in prediction accuracy**.\n \nMaybe we are missing something. In such case, please, make the paper clearer regarding this point and show evidence that the proposed solution indeed presents (certified or not) adversarial robustness on the **in-distribution data**. For example, provide classification accuracy on adversarially manipulated in-distribution data (e.g., see [2]).\n\nWe have some additional less problematic concerns regarding the paper.\n\nA drawback of the proposed method is the need to design an ad hoc binary classifier. We do not know whether the proposed binary classifier will work adequately for different models and datasets. Moreover, we need to define a training procedure for it. The solution adds a hyperparameter called the bias shift. We believe the authors could be more explicit about these limitations.\n\nWe also recommend that the authors combine their approaches with the IsoMax loss [4,5] and the IsoMax+ loss [6] rather than SoftMax loss or even OE to start with an improved OOD detection baseline. At least, the mentioned IsoMax loss could be cited as related works. The authors write: \"we also want to achieve SOTA performance on unperturbed OOD data.\" However, IsoMax+ loss outperforms OE in some cases [6]. Hence, OE does not currently present SOTA performance.\n\nThe authors use the word \"robust\" in the paper to mean \"adversarially robust.\" Considering that many other types of robustness exist, we suggest that authors write \"adversarially robust\" rather than simply \"robust\". Additionally, the authors sometimes refer to the multiclass classifier simply as the classifier. Considering that the solution also has a binary classifier, we suggest referring to the leading network as the multiclass classifier to make things more precise.\n\nWe recognize that the differences in performance of the proposed approach to the competing are significant. However, we need always to keep in mind that the proposed approach does not appear to present adversarial robustness on the **in-distribution data**, which is a major problem. Regardless of anything, it would be great to have the mean and standard deviation of five runs in Table 2. We also believe it is essential to add a column to Table 2 showing the classification accuracy on adversarial attacked in-distribution data.\n\nFinally, we understand that the authors say that the approach is (almost) for free because it does not produce classification accuracy drop, and it has OOD detection performance similar to OE. However, many procedures need to be done to achieve this. Hence, we believe that \"(almost) for free\" may a bit be misleading.\n\n[1] Deep Anomaly Detection with Outlier Exposure: https://arxiv.org/abs/1812.04606\n\n[2] Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem: https://arxiv.org/abs/1812.05720\n\n[3] Certifiably Adversarially Robust Detection of Out-of-Distribution Data: https://arxiv.org/abs/2007.08473\n\n[4] IsoMax loss: https://arxiv.org/abs/1908.05569\n\n[5] IsoMax loss (journal): https://arxiv.org/abs/2006.04005\n\n[6] IsoMax+ loss: https://arxiv.org/abs/2105.14399\n\nDavid Macêdo\n\n########################################################################################### ########################################################################################### ########################################################################################### ###########################################################################################\n\n===== FINAL RECOMMENDATION POST-REBUTTAL ========\n\nBy considering that the proposed approach is somewhat novel and presents convincing results and that, after the rebuttal, the authors performed many runs for their method, recognized ACET is robust for in-distribution, added classification accuracy on attacked in-distribution data for all methods, improved clarity by adding terms proposed by the reviewer, make clear that the solution is not robust against attacks on in-distribution data, recognize that we need to design and train an ad-hoc binary discriminator; we are changing our recommendation for \"accept\".\n\n########################################################################################### ########################################################################################### ########################################################################################### ###########################################################################################\n\n", "summary_of_the_paper": "The paper proposes a method to provide certified adversarial robustness on the out-of-distribution. Moreover, almost no classification accuracy drop is observed. Furthermore, the detection performance on clean OOD is similar to Outlier Exposure (OE) [1] approach. Finally, the solution provably avoids the asymptotic overconfidence problem. The method is based on adding a binary classifier responsible for being certifiably robust to adversarial manipulation on the out-of-distribution data. Apparently, adversarial robust training is not applied to the multiclass classifier, preventing it from presenting classification accuracy drop. A semi-joint training is then applied.", "summary_of_the_review": "Currently, it appears to us that the proposed solution does not provide adversarial robustness on the **in-distribution data**. If this is the case, we believe that this drawback makes the method much less valuable than previously published competing approaches. Suppose the authors clarify the text and prove that the proposed solution, like ACET and GOOD, provides adversarial robustness on the **in-distribution data**. In that case, we may improve our score mainly if the other concerns are also addressed. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KBuNgoPJH6", "reviewer": "Reviewer_U32o", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength**\\\nThe proposed method is sensible and shows consistent improvements.\\\nThe paper provides experiments on several detection benchmarks.\n\n**Weakness**\\\n*Limited evaluation*\\\nDespite the fact that this scheme utilizes a certified classifier, the author evaluates with empirical robustness measures (to calculate the maximum perturbation). Rather than that, I suggest utilizing a certified robustness measure with $l_\\infty$ extension (Zhang et al., 2021).\nAdditionally, considering AutoAttack (Croce et al., 2020) to attack the confidence $\\max p(y|x)$ will be more convincing (as an empirical robustness measure): calculate the maximum confidence over the ensemble attacks in AutoAttack. \n\n*Discussion with Tramer et al., 2021*\\\nTramer et al., 2021 prove that detecting adversarial samples is hard as classification (Tramer et al., 2021), i.e., classifying $\\epsilon$ is almost the same as detecting $2*\\epsilon$ sample. Due to this paper, it was hard for me to believe that adversarial detection is (almost) free: as it is known to sacrifice the clean accuracy to obtain adversarial robustness (Zhang et al., 2019). Can the author rigorously discuss with the following paper (Tramer et al., 2021)?\n\n*Limited technical novelty*\n* The proposed method can be seen as a combination of two OOD methods (Hsu et al., 2020) and (Bitterwolf et al., 2020).\n* The main technical novelty of this paper is to (1) model predictive distribution with in-and-out conditional distribution (2) utilize IBP for modeling in-and-out distribution to model certification of OOD robustness.\n* However, I believe (1) can be found in (Hsu et al., 2020), and (2) corresponds to (Bitterwolf et al., 2020). \n\n*The AUC of ProoD-disc seems to be low*. This implies that the discriminator does not capture the in-and-out distribution probability well.\n\n(minor) *The writing and presentation can be improved*\n* For instance, a single paragraph contains many messages, making the readers confused about the main message. \n\n**Questions**\\\nIs there a reason for utilizing a small network for certified robustness? The robustness tends to increase by the network size (Xie et al., 2020).\\\nIs it possible to report the GOOD (Bitterwolf et al., 2020) result in CIFAR-100 and R.ImgNet? I believe it is the main baseline to consider.\n\n**References**\\\nZhang et al., 2019 “Theoretically Principled Trade-off between Robustness and Accuracy”\\\nBitterwolf et al., 2020, “Certifiably adversarially robust detection of out-of-distribution data”\\\nCroce et al., 2020, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks”\\\nHsu et al., 2020, “Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data”\\\nXie et al., 2020, “Intriguing properties of adversarial training at scale”\\\nTramer et al., 2021, “Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them”\\\nZhang et al., 2021, “Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons”", "summary_of_the_paper": "This paper aims to detect out-of-distribution data in an adversarially robust manner. To this end, the authors incorporate a certified (binary) classifier to model in-and-out distribution and jointly model predictive distribution $p(y|x)$ by conditioning with the binary classifier, i.e., $p(y|x, in)p(in|x) + p(y|x,out)p(out|x)$. The authors show that the proposed method is empirically strong under various detection scenarios.", "summary_of_the_review": "I recommend weak rejection for this review. I believe the evaluation is somewhat questionable and also needs some rigorous discussion with related works. I still believe the idea is sensible, so I carefully request the authors to respond to my weakness part during the rebuttal.\n\n-------\n**POST REBUTTAL:** After the response, I am slightly above the threshold as the proposed method seems to be a scalable work as it stabilizes the hardness of training robust out-of-distribution (OOD) detector.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JkQq_qSDcCh", "reviewer": "Reviewer_fH7P", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Strengths\n\n- The formulation of Prood is novel, sound, well-motivated, and reasonable. \n- The theoretical contribution (Theorem 1) seems valid.\n- The presented empirical results seem promising. \n\n\n# Weaknesses\n\n- There is a large room to improve in terms of the paper's clarity. There are multiple points in the paper that needs to be made clearer.\n    - In Table 1, \"High clean OOD\" is supposed to be \"High clean OOD detection performance\".\n    - In Eq. (3), $y$ should be clarified. Does it hold for all $y$'s? Also, the first inequality needs to be explained.\n- The paper can be made more self-contained by including some background knowledge. For example, clear definitions of GAUC and AAUC should be provided.\n\n# Questions\n\n- For me, it is actually surprising that this approach works. In principle, a supervised classifier trained to discriminate inliers and outliers in a specific dataset will not necessarily be able to successfully detect unseen outliers. Probably this is why the clean AUC of ProoD-Disc in Table 2 is somewhat low. How can OOD detection AUC be improved even if an under-performing component is incorporated into the model?\n- From Eq. (7) and the text nearby, $p(y|x,i)$ is not trained for robustness and the only component that is robustified is the binary classifier. (Please correct me if I'm wrong.) How can the whole model be robust if only a part of the model is robustified?\n- Adversarial attack can also be performed on inliers so that it is misclassified (as in conventional attacks). How does ProoD respond to such attacks?", "summary_of_the_paper": "A novel certifiable OOD detector is described in the paper. The proposed method, ProoD, merges a binary classifier and a multi-class classifier in a clever way to produce an OOD detector robust to adversarial perturbation. The binary classifier is trained to discriminate inliers and outliers, while the multi-class classifier is trained to predict class labels under outlier exposure. ProoD achieves good multi-class classification performance, good outlier detection performance, and robustness against adversarial perturbations on outliers.", "summary_of_the_review": "I vote to accept this paper because its contributions are clear, novel, and significant.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper aims for detecting not only clean OOD data, but also their adversarially manipulated ones. The authors propose a method for this goal, with no/marginal loss in clean test accuracy (say, Acc) and clean OOD detection accuracy (say, AUC), while existing methods for targeting the same goal suffers from low Acc and AUC. 3 reviewers are positive and 2 reviewers are negative. Reviewers and AC think that the proposed idea of merging a certified binary classifier for in-versus out-distribution with a classifier for the in-distribution task is interesting. However, AC thinks that experimental results are arguable as pointed out by reviewers. For example, in CIFAR-10, the proposed method outperforms the baseline (GOOD) with respect to Acc and AUC, but often significantly underperforms it with respect to GAUC (guaranteed AUC) or AAUC (adversarial AUC). Then, the question is which metric is more important? It is arguable to say whether Acc is more important than GAUC or AAUC. But, at least, AC thinks that AUC and AAUC (or GAUC) are equally important as adversarially manipulated OOD data is nothing but another OOD data made from the original clean OOD data. Hence, the superiority of the proposed method over the baseline is arguable in the experiments, and AC tends to suggest rejection. 

ps ... AC is also a bit skeptical on the motivation of this paper. What is the value of obtaining "guaranteed AUC"? It is not the "real/true" worst case OOD performance, as it varies with respect to the tested clean OOD data. Namely, it is the worst case OOD performance just in a certain "subset" of OOD data, i.e., adversarially manipulated OOD data made from a certain clean OOD data. Hence, AC is curious about what is the value of establishing such a "partial" lower bound (rather than "true" lower bound considering all possible OOD data). AC thinks that the problem setup studied in this paper (and some previous papers) looks interesting/reasonable at the first glance, but feels somewhat artificial after a deeper look.
Decision: Reject