Detecting Brittle Decisions for Free: Leveraging Margin Consistency in Deep Robust Classifiers

Authors: jonas  ngnawé
sabyasachi  sahoo
yann  pequignot
frédéric  precioso
christian  gagné
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 21:10:58

Abstract: Despite extensive research on adversarial training strategies to improve robustness, the decisions of even the most robust deep learning models can still be quite sensitive to imperceptible perturbations, creating serious risks when deploying them for high-stakes real-world applications. While detecting such cases may be critical, evaluating a model's vulnerability at a per-instance level using adversarial attacks is computationally too intensive and unsuitable for real-time deployment scenarios. The input space margin is the exact score to detect non-robust samples and is intractable for deep neural networks. This paper introduces the concept of margin consistency -- a property that links the input space margins and the logit margins in robust models -- for efficient detection of vulnerable samples. First, we establish that margin consistency is a necessary and sufficient condition to use a model's logit margin as a score for identifying non-robust samples. Next, through comprehensive empirical analysis of various robustly trained models on CIFAR10 and CIFAR100 datasets, we show that they indicate high margin consistency with a strong correlation between their input space margins and the logit margins. Then, we show that we can effectively use the logit margin to confidently detect brittle decisions with such models. Finally, we address cases where the model is not sufficiently margin-consistent by learning a pseudo-margin from the feature representation. Our findings highlight the potential of leveraging deep representations to efficiently assess adversarial vulnerability in deployment scenarios.

Main paper: 
 introduction: 
 Deep neural networks are known to be vulnerable to adversarial perturbations, visually insignificant changes in the input resulting in the so-called adversarial examples that alter the model's prediction (Biggio et al., 2013; Goodfellow et al., 2015) . They constitute actual threats in real-world scenarios (Evtimov et al., 2017; Gnanasambandam et al., 2021) , jeopardizing their deployment in sensitive and safety-critical systems such as autonomous driving, aeronautics, and health care. Research in the field has been intense and produced various adversarial training strategies to defend against the vulnerability to adversarial perturbations with bounded ℓ p norm (e.g., p = 2, p = ∞) through augmentation, regularization, and detection (Xu et al., 2017; Madry et al., 2018; Zhang et al., 2019; Carmon et al., 2019; Wang et al., 2020; Wu et al., 2020; Rice et al., 2020) , to cite a few. The empirical robustness (adversarial accuracy) of these adversarially trained models is still far behind their high performance in terms of accuracy. It is typically estimated by assessing the vulnerability of samples of a given test set using adversarial attacks (Carlini & Wagner, 2016; Madry et al., 2018) or an ensemble of attacks such as the standard AutoAttack (Croce & Hein, 2020b) . The objective of that evaluation is to determine if, for a given normal sample, an adversarial instance exists within a given ϵ-ball around it. Yet, this robustness evaluation over a specific test set provides a global property of the model but not a local property specific to a single instance (Seshia et al., 2018; Dreossi et al., 2019) . Beyond that specific test set, obtaining this information for each new sample would typically involve rerunning adversarial attacks or performing a formal robustness verification, which in certain contexts may be computationally prohibitive in terms of resources and time. Indeed, the computational cost makes it prohibitive to estimate robust accuracy at scale on large test sets and/or large models, for example, when using AutoAttack in standard mode. Moreover, in high-stakes deployment scenarios, knowing the vulnerability of single instances in real-time (i.e., their susceptibility to adversarial attacks) would be valuable, for example, to reduce risk, prioritize resources, or monitor operations. Therefore, there is a need for efficient and scalable ways to determine the vulnerability of a model's decision on a given sample.The input space margin (i.e., the distance of the sample to the model's decision boundary in the input space), or input margin in short, can be used as a score to determine whether the sample is non-robust and, as such, likely to be vulnerable to adversarial attacks. Computing the exact input margin is intractable for deep neural networks (Katz et al., 2017; Elsayed et al., 2018; Jordan & Dimakis, 2020) . These input margins may not be meaningful for fragile models with zero adversarial accuracies as all samples are vulnerable (close to the decision boundary). However, for robustly trained models, where only certain instances are vulnerable, the input margin is very useful for identifying the critical samples. Previous research studies have explored input margins of deep neural networks during training, focusing on their temporal evolution (Mickisch et al., 2020; Xu et al., 2023) , and their exploitation in improving adversarial robustness through instance-reweighting with approximations (Zhang et al., 2020; Liu et al., 2021) and margin maximization (Elsayed et al., 2018; Ding et al., 2020; Xu et al., 2023) . However, to the best of our knowledge, no previous research studies the relationship between the input space margin and the logit margin of robustly trained deep classifiers in the context of vulnerability detection.In this paper, we investigate how the deep representation of robust models can provide information about the vulnerability of any single sample to adversarial attacks. We specifically address whether the logit margin as an approximation of the distance to the decision boundary in the feature space of the deep neural network (penultimate layer) can reliably serve as a proxy of the input margin for vulnerability detection. When this holds, we will refer to the model as being margin-consistent. The margin consistency property implies that the model can directly identify instances where its robustness may be compromised simply from a simple forward pass using the logit margin. Fig. 1 illustrates this idea of margin consistency. The following contributions are presented in the paper:• We introduce the notion of margin consistency1 , a property to characterize robust models that allow the use of their logit margin as a proxy estimation for the input space margin in the context of non-robust sample detection. We prove that margin consistency is a necessary and sufficient condition to reliably use the logit margin for detecting non-robust samples. • Through an extensive empirical investigation of pre-trained models on CIFAR10 and CIFAR100 with various adversarial training strategies, mainly taken from RobustBench (Croce et al., 2021) , we provide evidence that almost all the investigated models display high margin consistency, i.e., there is a strong correlation between the input margin and the logit margin. • We confirm experimentally that models with high margin consistency perform well in detecting samples vulnerable to adversarial attacks based on their logit margin. In contrast, models with weaker margin consistency exhibit poorer performance. • For models where margin consistency does not hold, exhibiting a weak correlation between the input margins and the logit margins, we simulate margin consistency by learning to map the model's feature representation to a pseudo-margin with a better correlation through a simple learning scheme. Figure 1 : Illustration of the input space margin, margin in the feature space and margin consistency.The model preserves the relative position of samples to the decision boundary in the input space to the feature space. 
notation and preliminaries: 
 Notation We consider f θ : R n → R K a deep neural network classifier with weights θ trained on a dataset of samples drawn iid from a distribution D on a product space X × Y. Each sample x in the input space X ⊂ R n has a unique corresponding label y ∈ Y = {1, 2, . . . , K}. The prediction of x is given by ŷ(x) = arg max k∈Y f k θ (x), where f k θ (x) is the k-th component of f θ (x).We consider that a deep neural network is composed of a feature extractor h ψ : X → R m and a linear head with Klinear classifiers {w k , b k } such that f k θ (x) = w k ⊤ h ψ (x) + b k .The predictive distribution p θ (y|x) is obtained by taking the softmax of the output f θ (x). A perturbed sample x ′ can be obtained by adding a perturbation δ to x within an ϵ-ball B p (x, ϵ), an ℓ p -norm ball of radius ϵ > 0 centered at x, {x ′ : ∥x ′ -x∥ p = ∥δ∥ p < ϵ}. The distance ∥x ′ -x∥ p = ∥δ∥ p represents the perturbation size defined as( n i=1 |δ i | p ) 1 p .In this paper, we will focus on ℓ ∞ norm (∥x∥ ∞ = max i=1,...,n |x i |), which is the most commonly used norm in the literature.Local robustness Different notions of local robustness exist in the literature (Gourdeau et al., 2021; Zhong et al., 2021; Han et al., 2023) . In this paper, we equate local robustness to ℓ p -robustness, a standard notion corresponding to the invariance of the decision within the ℓ p ϵ-ball around the sample (Bastani et al., 2016; Fawzi et al., 2018) and formalized in terms of ϵ-robustness.Definition 1. A model f is ϵ-robust at point x if for any x ′ ∈ B p (x, ϵ) (x ′ in the ϵ-ball around x), we have ŷ(x ′ ) = ŷ(x).For a given robustness threshold ϵ, a data instance is said to be non-robust for the model if this model is not ϵ-robust on it. This means it is possible to construct an adversarial sample from that instance in its vicinity (i.e., within an ϵ-ball distance from the original instance). A vulnerable sample to adversarial attacks is necessarily non-robust. This notion of local robustness can be quantified in the worst-case or, on average, inside the ϵ-ball. We focus here on the worst-case measurement given by the input margin, also referred to as the minimum distortion or the robust radius (Szegedy et al., 2014; Carlini & Wagner, 2016; Weng, 2019) The input space margin is the distance to the decision boundary of f in the input space. It is the norm of a minimal perturbation required to change the model's decision at a test point x:d in (x) = inf{∥δ∥ p : δ ∈ R n s.t. ŷ(x) ̸ = ŷ(x + δ)} = sup{ϵ : f is ϵ-robust at x}.(1)An instance x is non-robust for a robustness threshold ϵ if d in (x) ≤ ϵ. Evaluating Eq. 1 for deep networks is known to be intractable in the general case. An upper bound approximation can be obtained using a point x ′ 0 , the closest adversarial counterpart of x in ℓ p norm by din (x) = ∥x -x ′ 0 ∥ p (see Fig. 1 ).The logit margin is the difference between the two largest logits. For a sample x classified as i = ŷ(x) = arg max j∈Y f j θ (x) the logit margin is defined asf i θ (x) -max j,j̸ =i f j θ (x) > 0.It is an approximation of the distance to the decision boundary of f θ in the feature space. The decision boundary in the feature space around z = h ψ (x), the feature representation of x, is composed of (K -1) linear decision boundaries (hyperplanes) DB ij = {z ′ ∈ R m : w ⊤ i z ′ + b i = w ⊤ j z ′ + b j } (j ̸ = i).The margin in the feature space is therefore the distance to the closest hyperplane minj,j̸ =i d(z, DB ij ),where the distance d(z, DB ij ) from z to a hyperplane DB ij has a closed-form expression:EQUATIONwhere ∥ • ∥ q is the dual norm of p, q = p p-1 for p > 1 (Moosavi-Dezfooli et al., 2016; Elsayed et al., 2018) .When the classifiers w j are equidistant, i.e. ∥w i -w j ∥ q = C whenever i ̸ = j, the margin becomes:EQUATION)Under the equidistance assumption, the logit margin is proportional (equal up to a scaling factor) to the margin in the feature space. We will denote the logit margin of x by d out (x):EQUATIONDefinition 2. A model is margin-consistent if there is a monotonic relationship between the input space margin and the logit margin, i.e.,d in (x 1 ) ≤ d in (x 2 ) ⇔ d out (x 1 ) ≤ d out (x 2 ), ∀x 1 , x 2 ∈ X .A margin-consistent model preserves the relative position of samples to the decision boundary from the input space to the feature space. A sample further from (closer to) the decision boundary in the input space remains further from (closer to) the decision boundary in the feature space with respect to other samples, as illustrated in Fig. 1 .We can evaluate margin consistency by computing the Kendall rank correlation (τ ∈ [-1, 1]) between the logit margins and the input margins over a test set. The Kendall rank correlation tests the existence and strength of a monotonic relationship between two variables. It makes no assumption on the distribution of the variables and is robust to outliers (Chattamvelli, 2024) . While a positive value of τ indicates samples are ranked similarly (or identically for τ = 1) according to logit margins and input margins, a negative value of τ indicates that one margin's ranking is roughly reversed. Perfect margin consistency corresponds to the situation τ = 1.Non-robust detection can be defined as a scored-based binary classification task where non-robust samples constitute the positive class, and the input margin d in induces a perfect discriminative function g for that:g(x; f θ ) = 1 [din(x)≤ϵ] = 1 if x is non-robust 0 if x is robust .If a model is margin-consistent, its logit margin can also be a discriminative score to detect nonrobust samples. The following theorem establishes that this is a necessary and sufficient condition. Therefore, the degree to which a model is margin-consistent should determine the discriminative power of the logit margin.Theorem 1. If a model is margin-consistent, then for any robustness threshold ϵ, there exists a threshold λ for the logit margin d out that separates perfectly non-robust samples and robust samples. Conversely, if for any robustness threshold ϵ, d out admits a threshold λ that separates perfectly non-robust samples from robust samples, then the model is margin-consistent. Proof sketch. Fig. 2 presents intuition behind the proof of Theorem 1. For the first part of the theorem (see Fig. 2a ), if there is a monotonic relationship between d in and d out (margin consistency), any point x with d in less than the threshold ϵ (non-robust) will also haved out less than λ 0 = d out (x 0 ) (with d in (x 0 ) = ϵ).For the second part (see Fig. 2b ), if there are two points x 1 and x 2 with nonconcordant d in and d out (no margin consistency), then for a threshold ϵ 0 between d in (x 1 ) and d out (x 2 ), they will both have different classes but no threshold of d out can classify them both correctly. The complete proof of Theorem 1 is deferred to Appendix A. Common metrics for detection include (Hendrycks & Gimpel, 2017; Corbière et al., 2019; Zhu et al., 2023) : the Area Under the Receiver Operating Curve (AUROC), which ensures the ability of a model to distinguish between the positive and negative classes across all possible thresholds; the Area Under the Precision-Recall Curve (AUPR), which evaluates the trade-off between precision and recall and is less sensitive to imbalance between positive and negative classes; and the False Positive Rate (FPR) at a 95% True Positive Rate (TPR) (FPR@95), that is crucial in systems where missing positive cases can have serious consequences, such as minimizing the number of vulnerable samples missed. The AUROC and AUPR of a perfect classifier is 1, while 0.5 for a random classifier. 
experimental setup: 
 Datasets and models We investigate various pre-trained models on CIFAR10 and CIFAR100 datasets (Krizhevsky, 2009) . The majority of models were loaded from the RobustBench model zoo2 (Croce et al., 2021) , with a few more models that are ResNet-18 (He et al., 2016) models we trained on CIFAR10 with Standard Adversarial Training (Madry et al., 2018) , TRADES (Zhang et al., 2019) , Logit Pairing (ALP and CLP, Kannan et al. ( 2018)), and MART (Wang et al., 2020) , using the experimental setup of Wang et al. (2020) .Input margin estimation This is done using FAB attack (Croce & Hein, 2020a) , which is an attack that minimally perturbs the initial instance. Xu et al. (2023) used it in their adversarial training strategy as a reliable way to compute the closest boundary point given enough iterations. We perform the untargeted FAB attack without restricting the distortion to find the boundary for all the samples in the test set instead of constraining the perturbation inside a given ϵ-ball when evaluating robustness. As a sanity check for the measured distances, we compare the ratio of correct samples x with estimated input margins greater than ϵ = 8/255 and the robust accuracy in ℓ ∞ norm measured with AutoAttack (Croce & Hein, 2020b) at ϵ = 8/255. Both quantities estimate the same thing, with a mean absolute difference over the models of 1.3 and 0.48 for CIFAR10 and CIFAR100, respectively, which are reasonable.The estimation of the input margins over the 10, 000 test samples allows us to create for a given threshold ϵ a pool of vulnerable samples that can be successfully attacked at threshold ϵ and non- (b) CIFAR100Figure 3 : Margin consistency of various models: there is a strong correlation between input space margin and logit margin for most ℓ ∞ robust models tested, the exceptions being DI0 and XU80 on CIFAR10. See Table 1 for the references on the models. The correlations are given with standard error for the y-axis values in each interval.vulnerable samples that were not able to be attacked. Training and distance estimations were run on an NVIDIA Titan Xp GPU (1x).Correlation analysis The results presented in Fig. 3 show that the logit margin has a strong correlation (up to 0.86) with the input margin, which means that they have a level of margin consistency for those models. The plots are given with standard error for the y-axis values in each interval. However, we also observe that two models (i.e., DI0 (Ding et al., 2020) and XU80 (Xu et al., 2023) WideResNets) have a weaker correlation. We show in Sec. 3.3 that we can learn to map the feature representation of these models to a pseudo-margin that reflects the distance to the decision boundary in the input space. Additional results on ImageNet with ℓ ∞ norm and on ℓ 2 -robust models on CIFAR10 are given in Table 5 of appendix E.Vulnerable samples detection We present the results for the robustness threshold ϵ = 8/255 in Table 1 . As expected with the strong correlations, the performance over the non-robust detection task is excellent. We can note that the metrics are lower for the two models with low correlations and particularly very high FPR@95. The performance remains quite good with different values of ϵ (cf. appendix B). Moreover, we show in appendix F.1 that the empirical robust accuracy of margin-consistent models can be accurately estimated by attacking only a small subset of the test set.A neural network f is said to be L-Lipschitz if ∥f (x 1 ) -f (x 2 )∥ ≤ L∥x 1 -x 2 ∥, ∀x 1 , x 2 .Lipschitz smoothness is important for adversarial robustness because a small Lipschitz constant L guarantees the network's output cannot change more than a factor L of the change in the input. There are strategies to directly constraint the Lipschitz constant to achieve 1-Lipschitz networks (Cisse et al., 2017; Li et al., 2019; Serrurier et al., 2021; Araujo et al., 2023) . Empirical adversarial training strategies only indirectly encourage Lipschitz's smoothness of the model. However, we note that Lipschitz's continuity of the feature extractor h ψ does not imply margin consistency of the model. Considering two points x 1 and x 2 with0 < d in (x 1 ) < d in (x 2 ), the L-Lipschitz condition implies that d out (x i ) ≤ Ld in (x i ) for i = 1, 2.However, as long as d out (x 1 ) > 0, it is possible a priori to have d out (x 2 ) < d out (x 1 ), thus violating the margin consistency condition, while still satisfying the previous relations. We also note that the strength of the correlation, i.e. the level of margin consistency, does not depend on the robust accuracy (see Fig. 4a and 4b ).Insight into when margin consistency may hold? We hypothesize that when the feature extractor h ψ behaves locally as an isometry (preserving distances, up to a scaling factor κ, at least for directions normal to the decision boundary), i.e., ∥x - Figure 4 : Distribution of the correlation between input margins and logit margins in ℓ ∞ with robust accuracy. The strength of the correlation, which indicates the level of margin consistency, does not depend on the robust accuracy. References on models are given in Table 1 .x ′ ∥ p = κ∥h ψ (x) -h ψ (x ′ )∥ p ,point to the representation of x in the feature space. In that case we will have ∥x -x ′ ∥ = κ∥z -z ′ ∥ which implies margin consistency. Experimentally, what we observe is that on the one hand, the input margin and the distance between the feature representations of x and x ′ (feature distance) correlate and on the other hand, the feature distance and the logit margin also correlate (Fig. 5a and Fig. 5b respectively).For the two models that are weakly margin-consistent, we are proposing to directly learn a mapping that maps the feature representation of a sample to a pseudo-margin that reflects the relative position of the samples to the decision boundary in the input space. We use a learning scheme similar to the one of Corbière et al. (2019) , with a small ad hoc neural network for learning the confidence of the instances. Given some samples with estimations of their input margins, the objective is to learn (b) Feature distances vs logit margins.Figure 5 : The correlations between the input margin, the distance between the feature representations of samples and their closest adversaries (feature distance -∥h ψ (x) -h ψ (x ′ )∥), and the logit margin may be due to the local isometry of the feature extractor. See Table 1 for the specific references on the model ID. The correlations are given with standard error for the y-axis values in each interval.to map their feature representation to a pseudo-margin that correlates with the input margins. This learning task can be seen as a learning-to-rank problem. We use a simple learning-to-rank algorithm for that purpose, which is a pointwise regression approach (He et al., 2008) relying on the mean squared error as a surrogate loss.For the experiment, we used a similar architecture and training protocol as (Corbière et al., 2019) with a fully connected network with five dense layers of 512 neurons, with ReLU activations for the hidden layers and a sigmoid activation at the output layer. We learn using 5000 examples sampled randomly from the training set, with 20% (1000 examples) held as a validation. Fig. 6 and Table 2 show the improved correlation on the learned score compared to the logit margin for both models.The correlations are given with standard error for the y-axis values in each interval. The network has learned to recover the relative positions of the samples from the feature representation. 
related work: 
 Detection tasks in machine learning are found to be of three main types:• Adversarial Detection The goal of adversarial detection (Xu et al., 2017; Carlini & Wagner, 2017) is to discriminate adversarial samples from clean and noisy samples. An adversarial example is a malicious example found by adversarially attacking a sample; it has a different class while being close to the original sample. A vulnerable (non-robust) sample is a normal sample that admits an adversarial example close to it. The two detection tasks are very distinct. Adversarial detection is a defence mechanism like adversarial training; Tramer (2022) has established that both tasks are equivalent problems with the same difficulty. • Out-of-Distribution (OOD) detection In OOD detection (Hendrycks & Gimpel, 2017; Peng et al., 2024; Yang et al., 2021) , the objective is to detect instances far from the distribution of the training data. These are often instances with different labels from the training labels or instances with the same label as training labels but with a covariate shift. For example, for a model trained on the CIFAR10 dataset, samples from the SVHN dataset (Netzer et al., 2011) or the corrupted version of CIFAR10-C (Hendrycks & Dietterich, 2019) are OOD samples for such a model. • Misclassification Detection (MisD) It consists in detecting whether the classifier's prediction is incorrect. This is also referred to as Failure Detection or Trustworthiness Detection (Corbière et al., 2019; Jiang et al., 2018; Luo et al., 2021; Granese et al., 2021; Zhu et al., 2023) . MisD is often used for selective classification (classification with a reject option) (Geifman & El-Yaniv, 2017) to abstain from predicting samples on which the model is likely to be wrong. A score for non-robust detection cannot tell if the sample is incorrect, as a vulnerable sample could be from any side of the decision boundary.Formal robustness verification aims at certifying whether a given sample is ϵ-robust or if it is not an adversarial counter-example can be provided (Brix et al., 2023b) . Some complete exact methods Table 2 : Comparison of the correlation and detection performance between the actual logit margin and the pseudo-margin learned. The models are initially weakly margin-consistent, but the pseudo-margin learned from feature representations simulates the margin consistency with higher correlation and better discriminative power.based on solving Satisfiability Modulo Theory problems (Katz et al., 2017; Carlini et al., 2017; Huang et al., 2017) or Mixed-Integer Linear Programming (Cheng et al., 2017; Lomuscio & Maganti, 2017; Fischetti & Jo, 2017) provide formal certification given enough time. However, in practice, they are tractable only up to 100,000 activations (Tjeng et al., 2019) . Incomplete but effective methods based on linear and convex relaxation methods and Branch-and-Bound methods (Zhang et al., 2018; Salman et al., 2019; Xu et al., 2020 Xu et al., , 2021;; Zhang et al., 2022; Shi et al., 2023) are faster but conservative, without guaranteed certifications even if given enough time. Scaling them to bigger architectures such as WideResNets and large Transformers is still challenging even with GPU accelartion (Brix et al., 2023a; König et al., 2024) . Weng et al. (2018) converts the problem of finding the robust radius (input margin) as a local Lipschitz constant estimation problem. Computing the Lipschitz constant of Deep Nets is NP-hard (Virmaux & Scaman, 2018) and Jordan & Dimakis (2020) proved that there is no efficient algorithm to compute the local Lipschitz constant. The estimation provided by Weng et al. (2018) requires random sampling and remains computationally expensive to obtain a good approximation. Vulnerability detection with margin-consistent models does not provide certificates but an empirical estimation of the robustness of a sample as evaluated by adversarial attacks. At scale, it can help filter the samples to undergo formal verification and a more thorough adversarial attack for resource prioritization. 
limitations and perspectives: 
 Vulnerability detection scope The scope of this work is ℓ p robustness measured by the input space margin; the minimum distortion that changes the model's decision while this does not give a full view of the ℓ p robustness. Samples may be at the same distance to the decision boundary and have unequal unsafe neighbourhoods given by an average estimation over the ϵ-neighbourhood considered.The average estimation of local robustness for a given ϵ-neighborhood remains an open problem, so whether it is possible to extract other notions of robustness from the feature representation efficiently could be a potential avenue for further exploration.The margin consistency property does not rely on attacks; however, its verification and the learning of a pseudo-margin with an attack-based estimation may not be possible if the model cannot be attacked on a sufficient number of samples. The assumption is that we can always successfully provide the closest point to the decision with a sufficient budget. This is a reasonable assumption since the studied models are not perfectly robust, and the empirical evidence so far with adaptive attacks is that no defence is foolproof, which justifies the need to detect the non-robust samples. It might occur that we need to combine with an attack such as CW-attack (Carlini & Wagner, 2016) to find the closest adversarial sample.The work of Papyan et al. (2020) shows that when deep neural network classifiers are trained beyond zero training error and beyond zero cross-entropy loss (aka terminal phase of training), they fall into a state known as neural collapse. Neural collapse is a state where the within-class variability of the feature representations collapses to their class means, the class means, and the classifiers become self-dual and converge to a specific geometric structure, an equiangular tight frame (ETF) simplex, and the network classifier converges to nearest train class center. This implies that we may lose the margin consistency property. While neural collapse predicts that all representations collapse on their class mean, in practice, perfect collapse is not quite achieved, and it is precisely the divergence of a representation from its class mean (or equivalently its w i ) which encodes the information we seek about the distance to the decision boundary in the input space.Exploring the impact of the neural collapse on margin consistency as models tend toward a collapsed state could provide valuable insights into generalization and adversarial robustness.In this paper, we study the margin consistency of models on their training distribution by reporting the Kendall rank correlation between the logit margin and the input margin on the test set. The study of this property on inputs from a different distribution or specifically crafted examples is left for future research. However, we observe that the adversarial examples used for the input margin estimation have significantly smaller logit margins than the detection thresholds (see Table 4 in appendix D). This indicates that these specific adversarial examples are indeed identified as non-robust instances, together with clean non-robust samples. 
conclusion: 
 This work addresses the question of efficiently estimating local robustness in the ℓ p sense at a per-instance level in robust deep neural classifiers in deployment scenarios. We introduce margin consistency as a necessary and sufficient condition to use the logit margin of a deep classifier as a reliable proxy estimation of the input margin for detecting non-robust samples. Our investigation of various robustly trained models shows that they have high margin consistency, which leads to a high performance of the logit margins in detecting vulnerable samples to adversarial attacks. We also find that margin consistency does not always hold, with some models having a weak correlation between the input margin and the logit margin. In such cases, we show that it is possible to learn to map the feature representation to a better-correlated pseudo-margin that simulates the margin consistency and performs better on vulnerability detection. Finally, we present some limitations of this work, mainly the scope of robustness, the attack-based verification, the impact of neural collapse in terminal phases of training, and vulnerability to adaptive attacks. Beyond its highly practical importance, we see this as a motivation to extend the analysis of robust models and the properties of their feature representations in the context of vulnerability detection. 
 
 Appendix: Theorem 1. If a model is margin-consistent, then for any robustness threshold ϵ, there exists a threshold λ for the logit margin d out that perfectly separates non-robust samples and robust samples. Conversely, if for any robustness threshold ϵ, d out admits a threshold λ that perfectly separates non-robust samples from robust samples, then the model is margin-consistent.Proof. Formally, for a finite sample S and nonegative values ϵ ≥ 0, λ ≥ 0, we define:A S ϵ := {x ∈ S : d in (x) ≤ ϵ} and B S λ := {x ∈ S : d out (x) ≤ λ}. We say that d out perfectly separates non-robust samples from robust samples if for any finite sample S ⊆ X and every ϵ ≥ 0 there exists λ ≥ 0 such that A S ϵ = B S λ . Necessity: We proceed by contraposition and assume that the model is not margin-consistent, i.e., there exist two samples x 1 and x 2 such thatd out (x 1 ) ≤ d out (x 2 ) and d in (x 1 ) > d in (x 2 ). By taking S = {x 1 , x 2 } and ϵ = d in (x 2 ) we have that A S ϵ = {x 2 }. However for any λ ≥ 0, if x 2 ∈ B S λ , then d out (x 1 ) ≤ d out (x 2 ) ≤ λ and so x 1 ∈ B Sλ . Therefore d out does not perfectly separates non-robust samples from robust samples.Sufficiency: Let's assume the model is margin-consistent. Let S be a finite sample and consider a threshold ϵ. Let x 0 be an element of the finite set A S ϵ such that d in (x 0 ) = max{d in (x) : x ∈ A S ϵ } and let λ = d out (x 0 ). Since the model is margin-consistent, then for every x ∈ S:x ∈ A S ϵ ⇔ d in (x) ≤ ϵ ⇔ d in (x) ≤ d in (x 0 ) ⇔ d out (x) ≤ d out (x 0 ) margin consistency ⇔ d out (x) ≤ λ ⇔ x ∈ B S λ .This means we have A S ϵ = B S λ0 , which shows that d out perfectly separates non-robust samples from robust samples.Our formulation of perfect separation using finite samples is not fundamentally necessary, however it avoids dealing with the intricacy of the continuum.We present in Fig. 7 the performance of the detection for various values of the robustness threshold. We can see that the high margin consistency allows the logit margin to be a good proxy for detection at various thresholds. Note that below ϵ = 2/255 and beyond ϵ = 16/255, the ratio of vulnerable points to non-vulnerable points becomes too imbalanced, with little to no positive instances beyond ϵ = 32/255.In Eq. 3 in Sec. 2.1, we show that we can approximate the exact feature margin by the logit margin when the classifiers w k are equidistant, i.e. ∥w i -w j ∥ = C whenever i ̸ = j. The results in Table 3 show that using the logit margin instead of the exact minimum feature margin has a negligible effect on the results. However, computing the exact feature margin requires computing the minimum over K -1 pairs of scaled logit differences. The approximation provided by the logit margin thereby circumvents the computational overhead of the minimum search, which can take a second instead of just microseconds for inference. This difference can add up to hours at scale, offering scalability when dealing with a large number of classes. We additionally provide boxplots for the K(K -1)/2 distances between pairs of classifiers for each model (45 for CIFAR10 and 4950 for CIFAR100) in Fig. 8 and 9 .Adversarial examples are perturbed samples that are close to the decision boundary. Therefore, we would expect these samples to have a very small logit margin for strongly margin-consistent models. Here, we study the adversarial examples we used to estimate the input margin. In Table 4 , we present the 99 th percentile of logit margins of adversarial examples and the detection threshold selected to obtain 95% True Positive Rate. We can observe that the values of the adversarial logit margins are significantly smaller than the detection threshold, so they would be detected as non-robust -just like clean samples that lie close to the decision boundary.E Results on ImageNet and ℓ 2 -robust models on CIFAR10Table 5 shows the results with ℓ ∞ -robust models on ImageNet (Deng et al., 2009) , a larger dataset, and results on ℓ 2 -robust models (only available on CIFAR10 in Robustbench). The results extend well in both situations. -Select uniformly at random a subset X n of n samples from X. 4: -Compute the estimations of the input margins on X n , D n = { din (x) : x ∈ X s } 5: -Create ground truth labels for vulnerability at threshold ϵ i.e. 1 [ din(x)≤ϵ] (x), for x ∈ X s .6: -Determine the threshold λ of d out that gives best approximation of robust accuracy on X s . 7: -A r = |{x ∈ X : d out (x) > λ and ŷ(x) = y}|/N the threshold λ at line 6 of Algorithm 1 would be to set it to the detection threshold at α = 95% TPR or α = 90% TPR, but the logit margin threshold could vary from one model to another; therefore a better way is to select it by tuning over values α ≥ 0.80 that gives the best approximation of the robust accuracy in terms of the absolute error on the small subset X s . The same idea allows estimating a model's vulnerability over a large dataset without the labels.We show that this leads to an accurate estimation of the robust accuracy of the investigated models evaluated with over 10, 000 by attacking only a random subset of size 500. Fig. 10 shows the absolute error of the estimation obtained using 500 samples. As expected, the estimation over the two weakly margin consistent models is not accurate while having a relatively small absolute difference on the strongly margin consistent models.Robust models often display robustness bias, namely a disparity of robustness across classes. Interestingly, in a strongly margin-consistent model (see Fig. Figure 10 : Estimations of the robust accuracy reported by Robustbench using logit margins with only 500 samples are quite accurate both on CIFAR10 and CIFAR100 for strongly margin-consistent models. The numbers indicate the absolute difference between the two values, averaged over ten subsets. See Table 1 for the specific references on the model ID. Figure 11 : Robustness bias analysis with the input margin (first column) vs the logit margin (second column) and correlation scores per class (third column). We have a margin-consistent model on the top row, and on the bottom row, we have a weakly margin-consistent model. On the first column (resp. second column), each boxplot represents the distribution of the input margin (resp. logit margin) for the corresponding CIFAR10 class. The dotted blue line indicates the threshold on the first two columns and is the correlation score computed over all classes. The threshold λ in the second column is the logit margin threshold at 95% TPR for detection at ϵ = 8/255. The robustness bias of the strongly margin consistent model can be detected using the logit margin, unlike the weakly margin consistent model. consistency remains strong for each class. However, for a weakly margin-consistent model (Fig. 11 , bottom row), significant disparities exist between the correlations across classes, making using logit margin as a proxy for input margin problematic.Code available at: https://github.com/ngnawejonas/margin-consistencyhttps://github.com/RobustBench/robustbench 


Decision: The reviewers agree that the paper tackles an important problem with sound results. The authors introduce the idea of margin consistency of a classifier to connect the input-space margin and feature-space margin (or logit margin). A classifier is margin consistent if there is a monotonic increasing relationship between the input margin and logit margin. They show that margin consistency is a necessary and sufficient condition in order for the logit margin to be used as a score to detect (vulnerable) samples from "robust" samples.

Reviews: review: paper_summary: paper_summary: The paper addresses the problem of efficiently detecting vulnerable inputs to a robust deep classifier at test time without the need for running adversarial attacks or formal verification. They introduce the idea of margin consistency of a classifier to connect the input-space margin and feature-space margin (or logit margin). A classifier is margin consistent if there is a monotonic increasing relationship between the input margin and logit margin. They show that margin consistency is a necessary and sufficient condition in order for the logit margin to be used as a score to detect (separate) "non-robust" (vulnerable) samples from "robust" samples. 

Using a number of robust models trained using various adversarial training methods, from the RobustBench, they empirically show that a vast majortiy of these models exhibit high margin consistency (measured via a Kendall Tau correlation). For these models with high margin consistency, they also evaluate the detection power of the logit margin in separating non-robust samples from robust samples.
strength_weakness: strength_weakness: 1. It's not fully clear why the paper does not use the feature-space margin directly instead of the logit margin. Taking the minimum of the distance to the decision boundaries (over classes $j \neq i$, where $i$ is the predicted class) in Equation 2 should give the feature-space margin. Why then is the logit margin, which is an approximation of this, needed? Because the equidistance assumption (line 123) may not hold in practice. 

1. The proof of Theorem 1 is not precise and needs some clarifications (details in the "Questions" section).
questions: questions: ### Proof of theorem 1
**Sufficiency**: It seems like the finite sample $S$ assumption is not needed. \
Suppose $f_\theta(x)$ is margin consistent, and $A^S_\epsilon$ is the set of non-robust samples from $S$ for a given $\epsilon > 0$ (as defined in the paper). 
Let 
$\epsilon_0 = \sup\\{ d_{in}(\mathbf{x}) : \mathbf{x} \in A^S_\epsilon \\}$ and $\lambda_0 = \sup\\{ d_{out}(\mathbf{x}) : \mathbf{x} \in A^S_\epsilon \\}$. 
For any non-robust sample $\mathbf{x} \in A^S_\epsilon$, its logit margin satisfies $d_{out}(\mathbf{x}) \leq \lambda_0$ due to the margin consistency. Therefore, $\lambda_0$ perfectly separates the non-robust samples from the robust samples based on the logit margin. 
(This was a little unclear in the paper and does not need a finite $S$).

**Necessity**: It is important to state that the proof is by contradiction. Suppose that for any robustness threshold $\epsilon$, $d_{out}$ admits a threshold $\lambda$ that perfectly separates the non-robust samples from the robust samples. Assume that $f_\theta$ is *not* margin consistent. It seems to me that a three point example shows the contradiction better, since with two points, it is always possible to separate the points with the inequality direction reversed. 

Since the model is not  margin consistent, there exist points $S = \\{ \mathbf{x}_1, \mathbf{x}_2, \mathbf{x}_3 \\}$ such that: i) in the input margin $d\_{in}(\mathbf{x}_1) < d\_{in}(\mathbf{x}_2) < d\_{in}(\mathbf{x}_3)$ and  ii) in the logit margin $d\_{out}(\mathbf{x}_1) < d\_{out}(\mathbf{x}_3) < d\_{out}(\mathbf{x}_2)$. Letting $\epsilon = d\_{in}(\mathbf{x}_2)$ (or slightly larger), we get the set of non-robust samples to be $A^S\_\epsilon = \\{ \mathbf{x}_1, \mathbf{x}_2 \\}$. 

However, in this case, there exists no threshold $\lambda$ for the logit margin that can cleanly separate the non-robust samples  $\\{ \mathbf{x}_1, \mathbf{x}_2 \\}$ from the robust samples $\\{ \mathbf{x}_3 \\}$. Since this leads to a contradiction, the model must be margin consistent. 
 

### Questions and Comments

1. In Eqn 2, $DB_j$ is not defined. I suppose it is $DB_j = \\{ \mathbf{z} \in R^m : (\mathbf{w}_j - \mathbf{w}_i)^T \mathbf{z} + b_j - b_i \geq 0 \\}$.

1. Referring to lines 123 -- 124, the equidistance assumption seems to be glossed over, and it is thereafter assumed that the robust deep classifiers satisfy the property? As mentioned under "Weaknesses", why can we not use the feature space margins?

1. Line 133: What does `output scores` refer to here? And clarify if it is the logit margin. 

1. Line 136: It may be worth mentioning here that negative values of Kendall Tau (upto $-1$) imply that the two rankings are anti-correlated or reversed. 

1. In section 2.3, the perfect discriminative function $g$ should be defined for a specific robustness threshold $\epsilon \geq 0$ and is a function of the classifier $f_\theta$. So it would be clearer to show this in the notation, e.g. as $g_\epsilon(\mathbf{x} ; f_\theta)$. Also the indicator function does not need the extra argument $\mathbf{x}$. That is, it can just be $\mathrm{1}\_{[d_{in}(\mathbf{x}) \leq \epsilon]}$

1. In Figure 3, it should be $x_2 \in A_{\epsilon_0}$, not  $x_2 \in A_{\epsilon}$. Also, on line 152, it should be $\lambda_0 = d_{out}(\mathbf{x}_0)$, to correspond to the figure. 

1. The caption for Figure 4 is not so clear.

1. Lines 191, 192: the statement `... margin consistency is a property orthogonal to the robust accuracy ...` seems like a strong statement to make. It is possible that there could be a non-zero correlation (mutual information) between the two. 

1. Figure 2 can be placed closer to the results section for readability.

1. Line 215: the word "boundary" is missing after "decision". 

1. Based on line 220, it seems like a standard pointwise regression using MSE (and not learning to rank) is used to learn the mapping from the feature representation to a pseudo-margin. Could you also clarify why the Sigmoid activation is used at this network's output, for either regression or L2R?

1. Does the property of margin consistency and the detection of vulnerable samples extend to other norms such as $\ell_2$ norm?  

1. Section 4, under OOD detection: the objective is usually to detect inputs that are from new classes not seen by the classifier during training, but it could also include detecting inputs that are from a different (shifted) marginal distribution, i.e., covariate-shifted OOD. It would be useful to cite a survey paper such as [Generalized out-of-distribution detection: A survey](https://arxiv.org/abs/2110.11334)

1. For mis-classification detection, a couple of relevant references are missing [2] and [3]. \
[2] https://papers.neurips.cc/paper_files/paper/2021/file/2cb6b10338a7fc4117a80da24b582060-Paper.pdf \
[3] https://openaccess.thecvf.com/content/CVPR2023/papers/Zhu_OpenMix_Exploring_Outlier_Samples_for_Misclassification_Detection_CVPR_2023_paper.pdf
limitations: limitations: Yes, there is discussion on the limitations and scope of this work in section 5.
paper_summary: paper_summary: It is currently difficult to determine how susceptible a given input to a model is to adversarial perturbations. The distance from the input to the model's decision boundary in the input space (input space margin) is a reasonable metric, but it is intractible to compute for many deep neural networks and not always meaningful. This paper investigates the use of logit margin as a proxy for input margin. The logit margin is defined to be the difference between the two largest logits. Furthermore, a model is said to be margin consistent if there is a monotonic relationship between input margin and logit margins. Theorem 1 justifies that, if a model is margin consistent, then logit margin can be used to detect non-robust samples. Experiments show that common deep learning models can display a high level of margin consistency.
strength_weakness: strength_weakness: While small logit margin may be associated with susceptibility to attacks, there may be other reasons we would not want to flag samples in this manner. For example, if a model exhibits differential performance on different subpopulations, certain subpopulations as a whole might be labeled as "brittle decisions." I think that an exploration and discussion of how pervasive brittle decisions are and how they are distributed in practice is warranted and would strengthen this paper.

Similarly, flagging brittle decisions may lead to a false sense of security. Adversarial examples have been shown to induce very high logit margins [1]. It is not clear to me that this method would be able to detect that an adversarially perturbed sample is close to a decision boundary. An enlightening experiment would be similar to figure 5a, except the samples have been adversarially perturbed to maximize logit margin.

[1] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. "Explaining and harnessing adversarial examples." arXiv preprint arXiv:1412.6572 (2014).
questions: questions: - How would using a different distance measure change your results (i.e. measuring distance with $l_2$ norm rather than $l_\infty$ norm)?
- How do you think the experimental results would change if you were using a higher dimensional/more difficult to classify dataset? Would you expect the correlations observed in these results to hold?
- Do you think this method and your results have any implications for non-robust classifiers?
limitations: limitations: Limitations are discussed in section 5.
paper_summary: paper_summary: The authors propose using the distance between the two max values of a model's output as a proxy for the input margin to efficiently identify samples vulnerable to adversarial attacks. This proposed margin consistency is formally defined and shown to work across many robust models on the CIFAR10 and CIFAR100 datasets. Additionally, a basic learned fix is proposed for the few models which did not have margin consistency.
strength_weakness: strength_weakness: (W1) Additional experiments. Including experiments for one or two non adversarially robust models would help show the limitations (or additional potential) of margin consistency. Additionally, evaluation of a robust model trained on ImageNet would help show the scalability of the method. Pretrained robust ImageNet models are available [5], so if initial compute is not a factor computation should not be a hinderance.

(W2) No additional analysis for why margin consistency fails for the 2 CIFAR10 models.

[5] https://github.com/RobustBench/robustbench
questions: questions: (Q1) What is the computational cost to build the dataset required for training in section 3.3? And in general, what is the computational cost required for the proposed method? Not as much of an issue due to adversarially robust models already requiring more compute to train and the proposed method taking minimal compute at inference.

(Q2) How successful would an adversarial attack maximizing the difference of the 2 max logits be in bypassing the proposed detection? Experiments for this may be slightly outside the scope of the paper.
limitations: limitations: yes
paper_summary: paper_summary: This paper discovers that many existing neural network classifiers share a strong correlation between input margin (distance to decision boundary) and output margin (difference between top-2 logits). The paper further proposes to use this property for local robustness estimation.
strength_weakness: strength_weakness: The weaknesses are mainly twofold.

First, the observation of the correlation, while interesting, is not strong enough in my opinion. Given that robust neural networks are known to be Lipschitz [1], the correlation between input and output margins is not very surprising. The main theoretical result (Theorem 1) is relatively straightforward, self-explanatory, and unsurprising.

Second, the authors did not consider the possibility of adaptive attacks. While the correlation between input and output margins is strong when the input itself is benign, it is unclear whether this correlation still holds when the input itself is adversarial. Is it possible to use attack methods to find inputs that break this correlation (i.e., output margin is large but input margin is small), leading to overestimation of the local robustness? Furthermore, the pseudo-margin prediction model adds even more potential vulnerabilities.

[1] Yang et al. A closer look at accuracy vs. robustness.
questions: questions: - Equation (3) relies on an *equidistant* assumption. Is this assumption satisfied by real-world models? I found some short discussions in Appendix C, but would prefer a more formal analysis. Furthermore, Line 640 says "the values vary only in a small range." What do the "values" refer to? It also seems to me that Figures 8 and 9 demonstrate non-trivial variance?

- In addition to observing the correlation between input and output margins, the paper also mentions the distances in the feature space. How is the property regarding feature-space distance used in practice? For context, comparing Figure 5a and 5b, it seems that the input-output margin correlation is stronger than the input-feature margin correlation.
limitations: limitations: See weaknesses.

Comments: title: Response to rebuttal
Thank you once more for the reply
Thank you for your response
Thank you for the feedback.
Thank you for the response
comment: Thank you for your careful responses and additional results. I have read the other reviews and author responses. The points raised by Reviewer WVo4 regarding the method's applicability to adversarial inputs, adaptive attacks, and Lipschitz smoothness are important. 
 
I would encourage the authors to include some discussion on these aspects in the revised paper. Particularly, the method's potential vulnerability to carefully crafted adversarial inputs should be acknowledged as a limitation. Some of the results on adversarial inputs (from the review discussions) could also be included in the paper's appendices. The results on estimating the robust accuracy using the logit margin from a small test sample are interesting. These applications of the method could be included as motivation in the introduction (if not done already - I don't recall). 

Overall, I think the paper makes a valuable contribution towards a light-weight method for detecting non-robust or vulnerable samples at test time for robust models that satisfy margin consistency. Therefore, I will maintain my current score of 7 in favor of acceptance.
Thank you for the diligent response, and the result looks promising. It makes sense that FAB and CW, which aim to find minimally disturbed examples, result in small margins, whereas fixed-radius attacks such as APGD and Square find adversarial examples with larger margins, with the adaptive PGD-MC getting the largest output margin.

Yes, AutoAttack stops early, but it is possible to modify it so that it doesn't. Nonetheless, I agree that this is too much to ask at this stage.

If a comprehensive study like this were performed on multiple models in a rigorous format, I would have strongly advocated for acceptance. However, since the current results are only preliminary, **I maintain a rating of 5, which is still above the acceptance threshold**, because I agree that the other motivation (quick estimation of overall robustness) is valid.
Thank you for your response.

* Regarding the deployment scenario, we agree that the model can receive all kinds of examples, including adversarial examples. For such instances, we confirm below on one of the strongly margin consistent models that adversarial examples crafted with standard attacks are indeed flagged as non-robust (in the same way as the non-robust clean examples) even if we cannot tell if they are adversarial or clean.

* The standard AutoAttack sequentially runs the untargeted APGD-CE, the targeted APGD-DLR, the targeted FAB and the Square Attack. For evaluation, it stops as soon as it finds adversarial examples below the $\epsilon$ threshold. The latter attacks might not be run on all samples or not at all.  In order to get adversarial examples for each of these attacks, we ran the evaluations one by one and collected the adversarial examples produced.  We have also run the Carlini-Wagner attack (CW, cleverhans pytorch implementation) and the tentative adaptive attack using PGD that maximises the logit margin (PGD-MC). The table contains the 99 percentile value of the adversarial logit margins and shows that they are all less or equal to the logit margin threshold found to detect 95%TPR for non-robust detection (1.20, from the previously provided table).
 
Model: AD10, $\epsilon=8/255$, 95TPR logit margin threshold=1.20
|  | APGD-CE | APGD-DLR | FAB | SQUARE | CW | PGD-MC |
|---|---|---|---|---|---|---|
| adversarial logit margin (99 percentile) | 0.928 | 0.779 | 0.004 | 0.204 | 0.004 | 1.230 |

Even if the tentative adaptive PGD-MC finds bigger logit margins than other attacks, the 99th percentile is still quite at the limit (with 95th percentile = 1.07). Although this by no means dismisses the possibility of creating more effective adaptive attacks, it suggests that the task may not be as straightforward as initially perceived.

* To find the minimally distorted adversarial for all samples, we used a FAB attack with a sufficient budget [1] and unbounded threshold, which does not stop the search as in the original AutoAttack.

[1]Xu, Yuancheng, et al. "Exploring and exploiting decision boundary dynamics for adversarial robustness." arXiv preprint arXiv:2302.03015 (2023).
Thank you for the response.

**Regarding the two applications:**

> Given a large dataset, the logit margin can provide a reasonable estimate of the empirical robust accuracy by only performing attacks on a small subset of the dataset.

***This makes sense. Hence, I have increased my rating to 5.*** Please add this result to the paper.

> In a real-time deployment scenario, you can determine in real-time, just from the forward pass, which samples are vulnerable to adversarial attacks without actually performing an attack.

In a deployment scenario, attacks would be performed by some attacker outside the control of the deployer. Hence, the deployer may only receive adversarial examples and not have access to clean examples. Since the proposed method is only proven to work on clean examples, I am not convinced that it would be effective in this scenario.

**Regarding FAB attack:**

Is it the case that FAB attack finds minimally perturbed adversarial examples, and stops as soon as it finds one? If this is the case, then it makes sense why the output margins of adversarial examples are tiny. Does a similar phenomenon still hold for other  attack algorithms that do not terminate early, such as untargeted and targeted PGD?

**Regarding adaptive attack:**

> While we have given it some thought, it is not clear to us at this point, how to generate such adversarial examples.

As mentioned above, PGD is an option. You can modify the attack objective, so that increasing output margin becomes part of the goal.

You may also try AutoAttack. However, since the original AutoAttack also stops as soon as it finds an adversarial example, some modifications are needed to increase output margin (see *Minimum-Margin AutoAttack* in Appendix B.1 of [1] for an example).

Regarding generating examples that lead to margin overestimation (but do not necessarily change the predicted class), one possibility is to run a PGD attack with an objective that maximizes output margin while restricting the distance from the nominal point (thereby restricting the input margin).

[1] Bai et al. MixedNUTS: Training-Free Accuracy-Robustness Balance via Nonlinearly Mixed Classifiers.
1/ We believe it is interesting in the scenarios where, indeed, you would like to know the adversarial robustness of a sample (1) but may not afford to perform heavy adversarial attacks or use an even more expensive formal verification method. We agree that our results only provide empirical evidence that the logit margin is a good measure of the adversarial robustness of a clean IID sample (2).  Here are two applications that we have in mind:
* Given a large dataset, the logit margin can provide a reasonable estimate of the empirical robust accuracy by only performing attacks on a small subset of the dataset. The tables below show the robust accuracy estimates when attacking only 500 samples and compare them with the accuracies reported on the benchmark. The estimation is made by finding the logit margin threshold at 95%TPR for non-robust detection at $\epsilon=8/255$ on the subsample, then using that threshold to predict over the whole 10k test samples.
* In a real-time deployment scenario, the use of logit margin can be particularly beneficial if you know that your model is margin-consistent. In such a case, you can determine in real-time, just from the forward pass, which samples are vulnerable to adversarial attacks without actually performing an attack. This capability could be used for monitoring or making decisions, with the disclaimer that local robustness does not indicate whether the sample is wrong or not, and that the detection is not perfect. For instance, this could be important when the uncertainty of the camera sensor is known. 


CIFAR10
| Model ID | Estimated Robust Accuracy | Reported Robust Accuracy |
|---|---|---|
| MD0 | 38.55 | 36.91 |
| MR0 | 39.61 | 39.12 |
| CL0 | 41.54 | 40.08 |
| AL0 | 40.94 | 40.21 |
| TR0 | 43.19 | 42.23 |
| EN0 | 50.91 | 49.25 |
| AD10 | 51.79 | 51.06 |
| AD20 | 53.42 | 52.48 |
| ZH0 | 53.87 | 53.08 |
| HE0 | 54.32 | 54.92 |
| SE10 | 56.00 | 55.54 |
| DS0 | 57.59 | 56.14 |
| DM0 | 59.77 | 57.27 |
| RE802 | 60.82 | 60.73 |
| PA20 | 63.04 | 61.04 |
| WA80 | 68.29 | 67.31 |
| CU80 | 69.01 | 67.73 |

CIFAR100
| Model ID | Estimated Robust Accuracy | Reported Robust Accuracy |
|---|---|---|
| RI1 | 20.59 | 18.95 |
| AD1 | 27.51 | 27.14 |
| AD21 | 27.89 | 27.67 |
| HE1 | 29.56 | 28.42 |
| RE11 | 29.11 | 28.50 |
| WU1 | 29.38 | 28.86 |
| RA11 | 29.64 | 28.88 |
| PA21 | 31.69 | 31.08 |
| CU41 | 31.61 | 31.65 |
| RE812 | 33.11 | 32.06 |
| DS1 | 33.44 | 32.19 |
| WA81 | 38.96 | 38.83 |
| CU81 | 40.32 | 39.18 |

2/In the tables, we have used the adversarial attacks generated by the FAB attack. We acknowledge without reserve that adaptive attacks or other types of attacks may produce adversarial examples that can actually have bigger logit margins while being close to the decision boundary. We apologize if we cannot provide more results on this matter. While we have given it some thought, it is not clear to us at this point, how to generate such adversarial examples.
Thank you for the response. After reviewing the comments and concerns of other reviewers and your responses, I have decided to maintain my current rating.
Thank you for the clarification, which cleared up my confusion on the two minor points. However, I would like to follow up on the possibility of adaptive attacks. I will raise my score if these questions can be sufficiently addressed.

> We aim at detecting (clean) samples on which the network’s decision is non-robust (susceptible to being attacked).

In this case, are the applications of the proposed method restricted to the case where 1) adversarial robustness is of interest; 2) clean examples are always provided?

Could you please provide a motivating scenario that satisfies this assumption?

> We observe that the logit margins on adversarial examples are almost always (much) smaller than that threshold.

This is where an adaptive attack is important. With a white-box adaptive attack, is it possible to find adversarial examples (which trick the model into mispredictions) with larger logit margins? Is it possible to find examples that do not necessarily lead to mispredictions, but induce large logit margins without actually being far from the decision boundaries (i.e., lead to robustness overestimation)?

Also, what types of attacks are used to generate the tables?

> We are not trying to differentiate between adversarial samples and clean samples that are close to the decision boundary. Instead, we both flag them as non-robust.

I agree that the proposed method can detect *clean* examples close to the decision boundary. However, for the above reasons, I am not fully convinced that adversarial examples can also be flagged.
Thank you for your kind response and for raising your score. Note that we have provided some results on the logit margin on adversarial examples in our response to reviewer WVo4.
Thank you for the feedback.
 
(Minor 1) We believe that we have provided enough evidence that Robust networks (or Lipschitz networks) are not necessarily margin-consistent. In particular, Lipschitz does not imply margin consistency. However, we agree that the interplay between these concepts is an interesting direction for future research.

(Major) We do not claim to solve adversarial examples detection. Instead, we aim at detecting (clean) samples on which the network’s decision is non-robust (susceptible of being attacked). When fixing the threshold on the logit margin to obtain a 95% True Positive Rate (for which the corresponding FPR are provided in Table 1), we observe that the logit margins on adversarial examples are almost always (much) smaller than that threshold. By doing this, we are not trying to differentiate between adversarial samples and clean samples which are close to the decision boundary. Instead, in this case, we both flag them as non-robust. As evidence of this, we provide the 99th percentile of the logit margin on adversarial examples together with the logit margin threshold at 95TPR in the table below. 

CIFAR10
| Model ID | Logit margin of Adversarial Examples (99 percentile) | Threshold of logit margin at 95% TPR for detection |
|---|---|---|
| AD1 | 0.01 | 1.42 |
| DS0 | 0.61 | 2.19 |
| DM0 | 1.64 | 3.01 |
| MR0 | 0.04 | 8.35 |
| MD0 | 0.06 | 11.04 |
| SE10 | 0.06 | 2.83 |
| PA20 | 0.14 | 1.69 |
| HE0 | 0.07 | 2.82 |
| TR0 | 0.07 | 4.11 |
| CL0 | 0.03 | 5.76 |
| EN0 | 0.13 | 5.06 |
| AL0 | 0.03 | 5.11 |
| AD20 | 0.03 | 1.33 |
| CU80 | 1.32 | 2.02 |
| RE802 | 0.13 | 1.87 |
| ZH0 | 0.17 | 2.87 |
| WA80 | 0.70 | 2.14 |
| AD10 | 0.01 | 1.20 |

CIFAR100
| Model ID | Logit margin of Adversarial Examples (99 percentile) | Threshold of logit margin at 95% TPR for detection |
|---|---|---|
| AD1 | 0.01 | 1.42 |
| AD21 | 0.02 | 1.44 |
| CU81 | 0.13 | 2.09 |
| CU41 | 0.04 | 1.95 |
| DS1 | 0.13 | 1.36 |
| HE1 | 0.06 | 1.81 |
| PA21 | 0.08 | 1.56 |
| RA11 | 0.08 | 1.77 |
| RE812 | 0.17 | 1.73 |
| RE11 | 0.07 | 1.40 |
| RI1 | 0.03 | 3.42 |
| WA81 | 0.09 | 1.59 |
| WU1 | 0.06 | 2.02 |

(Minor 2)  Figures 8 and 9 show, on a common y-axis, the distributions of distances between classifiers for each model. Comparing these ranges across models is not meaningful. We agree that these plots alone cannot provide evidence that equidistance holds in these models. In contrast, the results in Table 1 of the rebuttal PDF are evidence that the impact of the distance between classifiers is negligible for margin consistency; the logit margin and the feature margin lead to quite similar ranks. Thank you for pointing this out, we will modify section C of the supplementary material in this sense.
Thank you for the response. While the response cleared up my confusion on Q2, I still have some remaining questions.

- (Minor) I agree that Lipschitzness and margin consistency are different concepts and it is possible to find counter-examples that satisfy one but not the other. That being said, they are still intuitively and empirically connected. The authors found two models that are "likely Lipschitz" but not margin-consistent, but also found 17 models that are robust margin-consistent (Figure 2). Hence, the theoretical results, while valid, are unsurprising in my opinion.

- (Major) As mentioned in multiple places in the paper, this work focuses on detecting non-robust decisions with logit margins, which often manifest as adversarial examples. If it is possible to construct adversarial examples that can break the margin consistency and deceive the logit margin-based detection, then the purpose of the proposed method is defeated.

***Hence, I kindly disagree that studying the margin consistency on adversarial examples or designing adaptive attacks is out of the scope of this work, and believe that they must be carefully analyzed***. This is especially important because existing work has shown that adaptive attacks can significantly compromise adversary detectors [1].

- (Minor) I am still confused about Figures 8 and 9. The authors mentioned that the distances between the classifiers vary by a small range. However, the data points in Figures 8 and 9 seem all over the place.

Therefore, for now, I maintain a rating of 4.

[1] Carlini, N. and Wagner, D. Adversarial examples are not easily detected: Bypassing ten detection methods.
Thank you for your response and for the new experimental results. My concerns have largely been addressed. However, I still believe that an investigation of adversarial/adaptive attacks would help the reader understand in what situations margin consistency is expected to hold. I have raised my score to a 6.