Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach to combine OOD detection and classifier to develop a robust OOD detection framework with high classification accuracy. The authors provide confidence bounds for the noise-perturbed and adversarially attacked OOD samples. Experiments are conducted on benchmark datasets and the performance is compared with approaches that provide asymptotic guarantees on robustness. 
main_review: Strengths:

1. Combining OOD detection and classifiers is an interesting idea and the proposed "OOD-aware" training can be effective to improve the robustness of the classifiers.

2. Authors provide a confidence bound for the classifier which is useful to develop certifiable robust OOD-aware models.

3. Experiments on the benchmark datasets show that the proposed approach achieves comparatively high accuracy while maintaining a guarantee on the AUC score. 


Weaknesses:

1. The draft lacks clarity in many aspects. 

a. In table 1, metrics are not clear until section 3.

b. Figure 1 cannot be interpreted from the caption. It is not clear how the equations in the figure map to equations in the draft. 

c. In the experiments section, "semi-joint training" requires more details. For example, how are the several models trained with binary shifts? Metrics in figure 2 are not clear from the associated text.

2.  Authors claim to achieve better guarantees than the existing approaches (e.g., CCU, GOOD). However, it is not clear why Theorem 1 entails a tighter bound than Lemma 1 (Hein 2019).

3. The bounds in Eq 5 appear trivial from the computations of W_{+}, and W_{-}. How do these bounds affect the tightness of the bounds in Eq 6.

4. I would expect the joint training would improve the performance of both OOD detection and the classifier. However, as shown in table 2, outlier exposure, which is not the best state of the art, still performs better. What are the authors' comments on this. 

5. Authors are encouraged to compare with the more recent OOD detection approaches 

a. Liu et al., "Energy-based out-of-distribution detection"

b. Lee et al., "Training confidence-calibrated classifiers for detecting out-of-distribution samples."
summary_of_the_review: The proposed approach of combining OOD detection and classifier through joint training is an interesting approach. However, some parts of the paper lack clarity and thus, it is hard to evaluate the contributions. Authors are encouraged to address the comments and I will be willing to reconsider my decision.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper aims to detect out-of-distribution data in an adversarially robust manner. To this end, the authors incorporate a certified (binary) classifier to model in-and-out distribution and jointly model predictive distribution $p(y|x)$ by conditioning with the binary classifier, i.e., $p(y|x, in)p(in|x) + p(y|x,out)p(out|x)$. The authors show that the proposed method is empirically strong under various detection scenarios.
main_review: **Strength**\
The proposed method is sensible and shows consistent improvements.\
The paper provides experiments on several detection benchmarks.

**Weakness**\
*Limited evaluation*\
Despite the fact that this scheme utilizes a certified classifier, the author evaluates with empirical robustness measures (to calculate the maximum perturbation). Rather than that, I suggest utilizing a certified robustness measure with $l_\infty$ extension (Zhang et al., 2021).
Additionally, considering AutoAttack (Croce et al., 2020) to attack the confidence $\max p(y|x)$ will be more convincing (as an empirical robustness measure): calculate the maximum confidence over the ensemble attacks in AutoAttack. 

*Discussion with Tramer et al., 2021*\
Tramer et al., 2021 prove that detecting adversarial samples is hard as classification (Tramer et al., 2021), i.e., classifying $\epsilon$ is almost the same as detecting $2*\epsilon$ sample. Due to this paper, it was hard for me to believe that adversarial detection is (almost) free: as it is known to sacrifice the clean accuracy to obtain adversarial robustness (Zhang et al., 2019). Can the author rigorously discuss with the following paper (Tramer et al., 2021)?

*Limited technical novelty*
* The proposed method can be seen as a combination of two OOD methods (Hsu et al., 2020) and (Bitterwolf et al., 2020).
* The main technical novelty of this paper is to (1) model predictive distribution with in-and-out conditional distribution (2) utilize IBP for modeling in-and-out distribution to model certification of OOD robustness.
* However, I believe (1) can be found in (Hsu et al., 2020), and (2) corresponds to (Bitterwolf et al., 2020). 

*The AUC of ProoD-disc seems to be low*. This implies that the discriminator does not capture the in-and-out distribution probability well.

(minor) *The writing and presentation can be improved*
* For instance, a single paragraph contains many messages, making the readers confused about the main message. 

**Questions**\
Is there a reason for utilizing a small network for certified robustness? The robustness tends to increase by the network size (Xie et al., 2020).\
Is it possible to report the GOOD (Bitterwolf et al., 2020) result in CIFAR-100 and R.ImgNet? I believe it is the main baseline to consider.

**References**\
Zhang et al., 2019 “Theoretically Principled Trade-off between Robustness and Accuracy”\
Bitterwolf et al., 2020, “Certifiably adversarially robust detection of out-of-distribution data”\
Croce et al., 2020, “Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks”\
Hsu et al., 2020, “Generalized ODIN: Detecting Out-of-distribution Image without Learning from Out-of-distribution Data”\
Xie et al., 2020, “Intriguing properties of adversarial training at scale”\
Tramer et al., 2021, “Detecting Adversarial Examples Is (Nearly) As Hard As Classifying Them”\
Zhang et al., 2021, “Towards Certifying L-infinity Robustness using Neural Networks with L-inf-dist Neurons”
summary_of_the_review: I recommend weak rejection for this review. I believe the evaluation is somewhat questionable and also needs some rigorous discussion with related works. I still believe the idea is sensible, so I carefully request the authors to respond to my weakness part during the rebuttal.

-------
**POST REBUTTAL:** After the response, I am slightly above the threshold as the proposed method seems to be a scalable work as it stabilizes the hardness of training robust out-of-distribution (OOD) detector.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A novel certifiable OOD detector is described in the paper. The proposed method, ProoD, merges a binary classifier and a multi-class classifier in a clever way to produce an OOD detector robust to adversarial perturbation. The binary classifier is trained to discriminate inliers and outliers, while the multi-class classifier is trained to predict class labels under outlier exposure. ProoD achieves good multi-class classification performance, good outlier detection performance, and robustness against adversarial perturbations on outliers.
main_review: # Strengths

- The formulation of Prood is novel, sound, well-motivated, and reasonable. 
- The theoretical contribution (Theorem 1) seems valid.
- The presented empirical results seem promising. 


# Weaknesses

- There is a large room to improve in terms of the paper's clarity. There are multiple points in the paper that needs to be made clearer.
    - In Table 1, "High clean OOD" is supposed to be "High clean OOD detection performance".
    - In Eq. (3), $y$ should be clarified. Does it hold for all $y$'s? Also, the first inequality needs to be explained.
- The paper can be made more self-contained by including some background knowledge. For example, clear definitions of GAUC and AAUC should be provided.

# Questions

- For me, it is actually surprising that this approach works. In principle, a supervised classifier trained to discriminate inliers and outliers in a specific dataset will not necessarily be able to successfully detect unseen outliers. Probably this is why the clean AUC of ProoD-Disc in Table 2 is somewhat low. How can OOD detection AUC be improved even if an under-performing component is incorporated into the model?
- From Eq. (7) and the text nearby, $p(y|x,i)$ is not trained for robustness and the only component that is robustified is the binary classifier. (Please correct me if I'm wrong.) How can the whole model be robust if only a part of the model is robustified?
- Adversarial attack can also be performed on inliers so that it is misclassified (as in conventional attacks). How does ProoD respond to such attacks?
summary_of_the_review: I vote to accept this paper because its contributions are clear, novel, and significant.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose ProoD which merges a certified binary classifier for in-versus out-distribution with a classifier for the in-distribution task in a principled fashion into a joint classifier. They show that ProoD simultaneously achieves three properties: 1) guaranteed OOD robustness via confidence upper bounds on $l_\infty$-balls around OOD samples; 2) it provably prevents the asymptotic overconfidence of deep neural networks; 3) it can be used with arbitrary architectures and has no loss in prediction performance and standard OOD detection performance. They perform extensive experiments to show the promising performance of the proposed method ProoD.
main_review: I think this paper has the following strengths:

1. The authors derive the method ProoD in a principled way and prove that the method can prevent the asymptotic overconfidence of deep neural networks; 

2.  They perform extensive experiments to evaluate the proposed method and also compare it to existing baselines; 

3. The paper is well-written and easy to understand. 

However, I think this paper has the following weaknesses: 

1. The idea of training a discriminator independently via interval bound propagation (IBP) to achieve certified robustness on out-of-distribution samples is not very novel since IBP is an existing technique that can ensure certified robustness. Also, the theoretical result (Theorem 1) that the proposed joint classifier gets provably less confident in its decisions as one moves away from the training data is not entirely novel since it is mainly based on the previous theoretical results in the literature; 

2. Since the proposed method ProoD uses IBP in training to get the certified discriminator g, it is expected that it has better certified robustness than other methods that don't use IBP. However, it might be hard to train models using IBP with a large perturbation budget (e.g. use $\epsilon=8/255$ that is common in previous OOD detection papers such as ATOM). The authors may need to acknowledge this limitation; 

3. When the authors retrain ATOM and ACET models, they might need to consider using stronger PGD attacks during training since they use very strong attacks with adaptive step size for the evaluation of AAUC. In ATOM paper, they only use PGD attack with 5 steps and fixed step size. Using stronger PGD attacks with more steps and adaptive step size (e.g. use backtracking) for training ATOM and ACET might lead to better results (or AAUC) under their attacks in evaluation. 

4. The performance of the proposed method ProoD is not stable across different in-distribution and OOD datasets. For example, in Table 2, the performance of ProoD on CIFAR-10 vs. Smooth is worse than that of ATOM and GOOD in terms of AAUC metric, and in terms of GAUC metric, the performance of ProoD is also worse than that of GOOD. The GAUC metric for ATOM and ACET seems meaningless since it is a lower bound and is equal to 0. On CIFAR-100 as the in-distribution dataset, it seems the performance of ProoD is usually worse than that of ATOM in terms of the AAUC metric. In Table 7 (in the appendix), on CIFAR-10 vs. Uniform or CIFAR-100 vs. Uniform, the performance of ProoD is also worse than that of existing methods like ATOM and GOOD. So it is unclear whether the performance of ProoD is better than that of existing methods or not. I think the authors should evaluate ProoD on more OOD datasets. In the ATOM paper, they also evaluate OOD detectors on OOD datasets like Textures, Places365, LSUN (resize), and iSUN. I suggest the authors report the performance of ProoD on these OOD datasets and compare it to existing methods. 

5. I think the authors should give more details about attacking ProoD. For example, what's the attack objective they use to attack ProoD when evaluating the GAUC and AAUC? Since ProoD combines the classifier and detector, the adaptive attacks should attack both the classifier and the detector. If they only attack the detector when evaluating GAUC and AAUC, the results may not be correct and they need to re-evaluate them. 


summary_of_the_review: Although this paper proposes a principled method ProoD for robust OOD detection and has some interesting theoretical results, the experiments conducted are not enough to show the effectiveness of the proposed method. As I mentioned, the authors should be more careful in building the baselines and evaluating the methods. Also, they should give more details about the experiments like the attack objectives used. Thus, I think this paper is not ready for publication. 


***[Post Rebuttal]***

I think the proposed method ProoD is not very novel since it simply combines previous techniques, and the performance of ProoD is not stable across different in-distribution and OOD datasets. Also, in practice, it might be hard to use ProoD since under the False Positive Rate at 95% true positive rate metric, ProoD doesn't have good performance (it might be hard to select a suitable threshold for ProoD). Thus, I keep my original score and think the paper is not ready for publication.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a method to provide certified adversarial robustness on the out-of-distribution. Moreover, almost no classification accuracy drop is observed. Furthermore, the detection performance on clean OOD is similar to Outlier Exposure (OE) [1] approach. Finally, the solution provably avoids the asymptotic overconfidence problem. The method is based on adding a binary classifier responsible for being certifiably robust to adversarial manipulation on the out-of-distribution data. Apparently, adversarial robust training is not applied to the multiclass classifier, preventing it from presenting classification accuracy drop. A semi-joint training is then applied.
main_review: By far, our main concern regarding the proposed solution is the following: Does the proposed method present (certified or not) adversarial robustness on the **in-distribution data**? For example, which part of the solution prevents a malicious person from using an attack on an **in-distribution example** (i.e., making the classifier believe that a sample belongs to the wrong class) to mislead the primary (i.e., the multiclass) classifier? The proposed binary classifier does not appear to be able to prevent this situation. The below excerpts from the paper make us suspect that the proposed solution does not present adversarial robustness on the **in-distribution data**.

1. _"In contrast to (Bitterwolf et al., 2020) this comes without loss in test accuracy or non-adversarial OOD detection performance as in our model the neural network used for the in-distribution classification task is independent of the binary discriminator. Thus, we p(yjx; i) have the advantage that the classifier can use arbitrary deep neural networks and is **not constrained to certifiable networks**."_

2. _"Note that this is not standard adversarial training for a binary classification problem as here we have an asymmetric situation: we want to be (certifiably) robust to adversarial manipulation on the out-distribution data **but not on the in-distribution** and thus the upper bound is only used for out-distribution samples."_

3. _"While in (Bitterwolf et al., 2020) they also used IBP to upper bound the confidence of the classifier this resulted in a bound that took into account all O(K2) logit differences between all classes. In contrast, our loss in Eq. (4) is significantly simpler as **we just have a binary classification problem and therefore only need a single bound**."_

Please, notice that ACET [2] presents adversarial robustness on the **in-distribution data**. Moreover, GOOD [3] presents _certified_ adversarial robustness on the **in-distribution data**. If we lose this, making the solution certified adversarial robust on the out-distribution data does not matter anymore, as the attacker may manipulate the solution by simply attacking in-distribution data rather than out-of-distribution data. Considering we are indeed correct, and the proposed solution does not provide certified adversarial robustness on the in-distribution data, **this fact may perfectly explain why the solution, unlike competing approaches, almost does not present loss in prediction accuracy**.
 
Maybe we are missing something. In such case, please, make the paper clearer regarding this point and show evidence that the proposed solution indeed presents (certified or not) adversarial robustness on the **in-distribution data**. For example, provide classification accuracy on adversarially manipulated in-distribution data (e.g., see [2]).

We have some additional less problematic concerns regarding the paper.

A drawback of the proposed method is the need to design an ad hoc binary classifier. We do not know whether the proposed binary classifier will work adequately for different models and datasets. Moreover, we need to define a training procedure for it. The solution adds a hyperparameter called the bias shift. We believe the authors could be more explicit about these limitations.

We also recommend that the authors combine their approaches with the IsoMax loss [4,5] and the IsoMax+ loss [6] rather than SoftMax loss or even OE to start with an improved OOD detection baseline. At least, the mentioned IsoMax loss could be cited as related works. The authors write: "we also want to achieve SOTA performance on unperturbed OOD data." However, IsoMax+ loss outperforms OE in some cases [6]. Hence, OE does not currently present SOTA performance.

The authors use the word "robust" in the paper to mean "adversarially robust." Considering that many other types of robustness exist, we suggest that authors write "adversarially robust" rather than simply "robust". Additionally, the authors sometimes refer to the multiclass classifier simply as the classifier. Considering that the solution also has a binary classifier, we suggest referring to the leading network as the multiclass classifier to make things more precise.

We recognize that the differences in performance of the proposed approach to the competing are significant. However, we need always to keep in mind that the proposed approach does not appear to present adversarial robustness on the **in-distribution data**, which is a major problem. Regardless of anything, it would be great to have the mean and standard deviation of five runs in Table 2. We also believe it is essential to add a column to Table 2 showing the classification accuracy on adversarial attacked in-distribution data.

Finally, we understand that the authors say that the approach is (almost) for free because it does not produce classification accuracy drop, and it has OOD detection performance similar to OE. However, many procedures need to be done to achieve this. Hence, we believe that "(almost) for free" may a bit be misleading.

[1] Deep Anomaly Detection with Outlier Exposure: https://arxiv.org/abs/1812.04606

[2] Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem: https://arxiv.org/abs/1812.05720

[3] Certifiably Adversarially Robust Detection of Out-of-Distribution Data: https://arxiv.org/abs/2007.08473

[4] IsoMax loss: https://arxiv.org/abs/1908.05569

[5] IsoMax loss (journal): https://arxiv.org/abs/2006.04005

[6] IsoMax+ loss: https://arxiv.org/abs/2105.14399

David Macêdo

########################################################################################### ########################################################################################### ########################################################################################### ###########################################################################################

===== FINAL RECOMMENDATION POST-REBUTTAL ========

By considering that the proposed approach is somewhat novel and presents convincing results and that, after the rebuttal, the authors performed many runs for their method, recognized ACET is robust for in-distribution, added classification accuracy on attacked in-distribution data for all methods, improved clarity by adding terms proposed by the reviewer, make clear that the solution is not robust against attacks on in-distribution data, recognize that we need to design and train an ad-hoc binary discriminator; we are changing our recommendation for "accept".

########################################################################################### ########################################################################################### ########################################################################################### ###########################################################################################


summary_of_the_review: Currently, it appears to us that the proposed solution does not provide adversarial robustness on the **in-distribution data**. If this is the case, we believe that this drawback makes the method much less valuable than previously published competing approaches. Suppose the authors clarify the text and prove that the proposed solution, like ACET and GOOD, provides adversarial robustness on the **in-distribution data**. In that case, we may improve our score mainly if the other concerns are also addressed.
review_title: null

Meta-review: This paper aims for detecting not only clean OOD data, but also their adversarially manipulated ones. The authors propose a method for this goal, with no/marginal loss in clean test accuracy (say, Acc) and clean OOD detection accuracy (say, AUC), while existing methods for targeting the same goal suffers from low Acc and AUC. 3 reviewers are positive and 2 reviewers are negative. Reviewers and AC think that the proposed idea of merging a certified binary classifier for in-versus out-distribution with a classifier for the in-distribution task is interesting. However, AC thinks that experimental results are arguable as pointed out by reviewers. For example, in CIFAR-10, the proposed method outperforms the baseline (GOOD) with respect to Acc and AUC, but often significantly underperforms it with respect to GAUC (guaranteed AUC) or AAUC (adversarial AUC). Then, the question is which metric is more important? It is arguable to say whether Acc is more important than GAUC or AAUC. But, at least, AC thinks that AUC and AAUC (or GAUC) are equally important as adversarially manipulated OOD data is nothing but another OOD data made from the original clean OOD data. Hence, the superiority of the proposed method over the baseline is arguable in the experiments, and AC tends to suggest rejection. 

ps ... AC is also a bit skeptical on the motivation of this paper. What is the value of obtaining "guaranteed AUC"? It is not the "real/true" worst case OOD performance, as it varies with respect to the tested clean OOD data. Namely, it is the worst case OOD performance just in a certain "subset" of OOD data, i.e., adversarially manipulated OOD data made from a certain clean OOD data. Hence, AC is curious about what is the value of establishing such a "partial" lower bound (rather than "true" lower bound considering all possible OOD data). AC thinks that the problem setup studied in this paper (and some previous papers) looks interesting/reasonable at the first glance, but feels somewhat artificial after a deeper look.
Decision: Reject