Detecting Brittle Decisions for Free: Leveraging Margin Consistency in Deep Robust Classifiers
Authors: Jonas Ngnawe
Sabyasachi Sahoo
Yann Batiste Pequignot
Frederic Precioso
Christian Gagné
Venue: NeurIPS 2024 poster (2024)
Keywords: adversarial robustness
empirical robustness estimation
classification
vulnerability detection
Primary area: machine_vision
TL;DR: We introduce and use margin-consistency for robust deep classifiers the efficient detection of vulnerable instances to adversarial examples via the logit margin..

Abstract: Despite extensive research on adversarial training strategies to improve robustness, the decisions of even the most robust deep learning models can still be quite sensitive to imperceptible perturbations, creating serious risks when deploying them for high-stakes real-world applications. While detecting such cases may be critical, evaluating a model's vulnerability at a per-instance level using adversarial attacks is computationally too intensive and unsuitable for real-time deployment scenarios. The input space margin is the exact score to detect non-robust samples and is intractable for deep neural networks. This paper introduces the concept of margin consistency -- a property that links the input space margins and the logit margins in robust models -- for efficient detection of vulnerable samples. First, we establish that margin consistency is a necessary and sufficient condition to use a model's logit margin as a score for identifying non-robust samples. Next, through comprehensive empirical analysis of various robustly trained models on CIFAR10 and CIFAR100 datasets, we show that they indicate high margin consistency with a strong correlation between their input space margins and the logit margins. Then, we show that we can effectively use the logit margin to confidently detect brittle decisions with such models. Finally, we address cases where the model is not sufficiently margin-consistent by learning a pseudo-margin from the feature representation. Our findings highlight the potential of leveraging deep representations to efficiently assess adversarial vulnerability in deployment scenarios.

Detecting Brittle Decisions for Free: Leveraging Margin Consistency in Deep Robust Classifiers
Jonas Ngnawé
Sabyasachi Sahoo
IID-Université Laval and Mila j joass ngnawe. 10ulaval.ca
IID-Université Laval and Mila
sabyasachi. sahoo. 1@ulaval.c. ca
Yann Pequignot
Frédéric Precioso
IID-Université Laval
Université Côte d' Azur, CNRS, INRIA, 13S, Maasai
yann. pengingquinottii.d ulaval.ca ca
frederic, frederic.preecosseuunii wwwpcc.ooooeiiiiiioooeaauuu
Christian Gagné
IID-Université Laval, Mila and Canada CIFAR Al Chair christian. caristian.gagneggell ulaval.ca
Abstract
Despite extensive research on adversarial training strategies improve robustness, the decisions of even the most robust deep learning models can still be quite sensitive to imperceptible perturbations, creating serious risks when deploying them for bigh-stakes real-worke applications. While detecting such cases may be critical, evaluating a model's vulnerability at per-instance level using adversarial attacks is computationally too intensive and unsuitable for real-time deployment scenarios. The input space margin is the exact score to detect non-robust samples and is intractable for deep neural networks. This paper introduces the concept of margin consistency - property that links the input space margins and the logit margins in robust models for efficient detection of vulnerable samples. First, we establish that margin consistency is a necessary and sufficient condition to model's ' logit margin score for identifying non-robust samples. Next, through comprehensive empirical analysis various robustly trained models CIFAR10 and CIFAR100 datasets, show that they indicate high margin consistency with strong correlation between their input space margins and the logit margins. Then, we show that margin to detect brittle decisions with such models. Finally, we address cases where the model is not sufficiently margin-consistent by learning pseudo-margin from the feature representation. Our findings highlight the potential of leveraging deep representations to assess adversarial vulnerability in deployment scenarios efficiently.
effectively and confidently the logit
Introduction
Deep neural networks are known to be vulnerable to adversarial perturbations, visually insignificant changes in the input resulting in the so-called adversarial examples that alter the model's 's prediction (Biggio et al. 2013 Goodfellow et al. (2015). They constitute actual threats in real-world scenarios al. 2021), jeopardizing their deployment in sensitive and safety-critical systems such as autonomous driving, aeronautics, and health care. Research in the field has been intense and produced various adversarial training strategies to defend against the vulnerability to adversarial perturbations with bounded E, norm (e.g., = 2, = oc) through
(Evtimov et al. 2017; Gnanasambandam
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

augmentation, regularization. and detection (Xu et al. Carmon al. 2019. Wang et al., [020, Wu al. 2020 Rice et al., 2020). to cite a few. The empirical robustness (adversarial accuracy) of these adversarially trained models is still far behind their high
2017, Madry et al. 2018 Zhang et al. 2019
performance in terms of accuracy. It typically estimated by assessing the vulnerability of samples of a given test set using adversarial attacks (Carlini & Wagner | Madry et al. (0188) or an ensemble
of attacks such as the standard AutoArtack (Croce & Hein| 2020b). The objective of that evaluation is to determine if, for a given normal sample, an adversarial instance exists within given e-ball around it. Yet, this robustness evaluation over a specific test set provides a global property of the model but not a local property specific to a single instance (Seshia et al. 2018 Dreossi et al. 2019). Beyond that specific test set, obtaining this information for each new sample would typically involve rerunning adversarial attacks performing formal robustness verification, which in certain contexts may be computationally prohibitive in terms of resources and time. Indeed, the computational cost makes it prohibitive to estimate robust accuracy at scale on large test sets and/or large models, for example, when using AutoAttack in standard mode. Moreover, in high-stakes deployment scenarios, knowing the vulnerability of single instances in real-time (i.e., their susceptibility to adversarial attacks) would be valuable, for example, to reduce risk, prioritize resources, or monitor operations. Therefore, there need for efficient and scalable ways to determine the vulnerability of a model's '' decision on given sample.
The input space margin (i.e., the distance of the sample to the model's decision boundary in the input space), or input margin in short, can be used as a score to determine whether the sample is non-robust and, such, likely to be vulnerable to adversarial attacks. Computing the exact input margin is intractable for deep neural networks (Katz et al. 2017 Elsayed et al. 2018 Jordan & Dimakis 2020). These input margins may not be meaningful for fragile models with zero adversarial accuracies as all samples are vulnerable (close to the decision boundary). However, for robustly trained models, where only certain instances are vulnerable, the input margin is very useful for identifying the critical samples. Previous research studies have explored input margins of deep neural networks during training, focusing on their temporal evolution (Mickisch et al. 2020; Xu et al. 2023), and their exploitation in improving adversarial robustness through instance-reweighiee reweighting with approximations (Zhang et al. 2020 Liu et al. 202I) and margin maximization (Elsayed al. [0188] Ding et al.) 20200) Xu et al. 2023). However, to the best of our knowledge, no previous research studies the relationship between the input space margin and the logit margin of robustly trained deep classifiers the context
of vulnerability detection.
In this paper, we investigate how the deep representation of robust models can provide information about the vulnerability of any single sample to adversarial attacks. We specifically address whether the logit margin an approximation of the distance to the decision boundary in the feature space of the deep neural network (penultimate layer) can reliably serve as a proxy of the input margin for vulnerability detection. When this holds, we will refer to the model as being margin- The margin consistency property implies that the model can directly identify instances where its robustness may be compromised simply from a simple forward pass using the logit margin. Fig. !] illustrates this idea of margin consistency. The following contributions are presented in the paper:
We introduce the notion of margin consistency"] property to characterize robust models that allow the use of their logit margin as a proxy estimation for the input space margin in the context of non-robust sample detection. We prove that margin consistency is necessary and sufficient condition to reliably use the logit margin for detecting non-ro samples.
Through an extensive empirical investigation of pre-trained models on CIFARIO and CIFAR100 with various adversarial training strategies, mainly taken from RobustBench (Croce et al. 2021). we provide evidence that almost all the investigated models display high margin consistency, i.e., there is strong correlation between the input margin and the logit margin.
We confirm experimentally that models with high margin consistency perform well in detecting samples vulnerable to adversarial attacks based on their logit margin. In contrast, models with weaker margin consistency exhibit poorer performance.
For models where margin consistency does not hold, exhibiting weak correlation between the input margins and the logit margins, we simulate margin consistency by learning to map the model's feature representation pseudo- -arrinn with a better correlation through a simple learning scheme.
'Code available at: http:///hhhhhooooooo//////////as
2

---

Input Space
Feature Space
dx)(x,) & dm((x))
du (xx)
Margin Consistency
d=y(X;) Ann(X2) s dar(Xg)
2, = h(xx))
Robust
Yx2
-2
Non-robust
x adversarial sample
original sample
ball
Robust area with radius d(x) Model's Decision B Boundary
Figure I: Illustration of the input space margin, margin in the feature space and margin consistency. The model preserves the relative position of samples to the decision boundary in the input space to the feature space.
2 Methodology
2.1 Notation and Preliminaries
Notation We consider fo :R' + - Re a deep neural network classifier with weights 0 trained on dataset of samples drawn iid from distribution on product space X' x V. Each sample in the input space X' C R has unique corresponding label y E y given by (x) = arg max (x), where f (x) is the k-th component of fo(x). We consider that deep neural network is composed of feature extractor h2 X + R and linear head with K linear classifiers {Wk, bx such that f (x) Wk hy (x) + by. The predictive distribution po [x) is obtained by taking the softmax of the output fx(). A perturbed sample x' can be obtained by adding a perturbation 8 to x within an e-ball (x,x, e), an nn-nr ball of radius e > centered at x, {x' ||x - x = ||0||p < e). The distance ||x' - xlIp = ||ó1 represents the perturbation
}}. The prediction of x
size defined as (Σ |6 p In this paper, we will focus on foo norm (||x| max; I..... [x:]).
which is the most commonly used norm in the literature.
Local robustness
Different notions of local robustness exist in the literature (Gourdeau et al. 2021 Zhong et al. 2021 Han et al. 2023). In this paper, we equate local robustness lp robusttess,, a standard notoon corresponding to the invariance the decision within the bp e-ball around the sample (Bastani et al., 2016 Fawzi et al. 2018) and formalized in terms of c-robustness.
Definition 1. A model f is c-robust at point x for any x' E By(x, c) (x in the c-ball around x),, we have ú(x') = ix)).
For given robustness threshold c, data instance is said be non-robust for the model if this model is not c-robust on it. This means it is possible to construct an adversarial sample from that instance in its vicinity (i.e., within an c-ball distance from the original instance). A vulnerable sample to adversarial attacks is necessarily non-robust. This notion of local robustness can be quantified in the worst-case or, average, inside the e-ball. We focus here the worst-case measurement given by the input margin, also referred to as the minimum distortion or the robust radius (Szegedy et al. 2014 Carlini & Wagner 2016 Weng 2019)
The input space margin the distance to the decision boundary of f in the input space. is the norm of minimal perturbation required to change the model's decision at test point
d (x) inf (10)p 5 E R" s.t. ((x) + x(x 8)} = sup(e ʃ is e-robust at x}.
(1)
An instance x is non-robust for robustness threshold if d (x) < c. Evaluating Eq. | for deep networks known to be intractable the general case. An upper bound approximation can be
obtained using point x6- the closest adversarial counterpart of x in lp norm by d,, (x) = ||x x lpp
(see Fig. D..
3

---

The logit margin is the difference between the two largest logits. For a sample x classified as i = ((x) = arg maxjey f (x) the logit margin is defined as fa(() - max fz
0. is an approximation of the distance the decision boundary of fo in the feature space. The decision boundary in the feature space around z = hop (x), the feature representation of x, is composed of 11 linear decision boundaries (hyperplanes) DB:, = {z' E R : w z' +b; w x + by ( + i). The margin in the feature space is therefore the distance to the closest hyperplane min d(z, DB,,), aJ
J:
where the distance d(zz, D),,) from z hyperplane D has closed-form expression:
d(z, DB,;) = inf inf(lllll : E RM s.t. z ++ E DB.,) fx(x) fd (x)
(2)
w: wjllg
where || l|g is the dual norm of p. 9 - for p 1 MMosavi-Dezfooll et al. 2018).
2016 Elsayed et al.
When the classifiers w, are equidistant, i.e. ||w; - w,llo whenever + j. the margin becomes:
fx(()
1 (((x)) - f8(x3) = T f0(x) max fo (x) joZi
min
min
(3)
JJ
C
C
logit margin
Under the equidistance assumption, the logit margin is proportional (equal up to a scaling factor) to the margin in the feature space. We will denote the logit margin of x by doue
dous (x) =x((x)
max fØ(x) (x).
2.2 Margin Consistency
A model is margin-consistent if there is a monotonic relationship between the input space margin and the logit margin, i.e., din (x1) S din (X2) <> (x1) S doua(x2), Vx1, E X.
Definition 2.
margin-consisten model preserves the relative position of samples to the decision boundary from the input space to the feature space. A sample further from (closer to) the decision boundary in the input space remains further from (closer to) the decision boundary in the feature space with respect to other samples, illustrated in Fig.
We can evaluate margin consistency by computing the Kendall rank correlation (r E (-1,1) between the logit margins and the input margins over a test set. The Kendall rank correlation tests the existence and strength of monotonic relationship between two variables. makes no assumption on the distribution of the variables and is robust to outliers (Chattamvelli (024). While a positive value of indicates samples are ranked similarly (or identically for 1) according to logit margins and input margins, negative value of indicates that one margin's 's ranking is roughly reversed. Perfect margin consistency corresponds to the situation
2.3 Non-robust Samples Detection
Non-robust detection can be defined as a scored-based binary classification task where non-robust samples constitute the positive class, and the input margin d,, induces perfect discriminative function for that:
g(x; fo) (4... {1
if non-robust
if x robust
If model is margin-consistent, its logit margin can also be discriminative score to detect non- robust samples. The following theorem establishes that this is a necessary and sufficient condition. Therefore, the degree to which a model margin-consistent should determine the discriminative power of the logit margin.
Theorem If a model is margin-consistent then for any robustness threshold e, there exists threshold > for the logit margin dont that separates perfectly non-robust samples and robust samples. Conversely, if for any robustness threshold e, dont admits a threshold that separates perfectly non-robust samples from robust samples, then the model is margin- cangin-consistent.

---

(20
mmargn-consstteecy
Ag
dar/P2)
E An
Au
4,00%)
da(k))
dui00
M (3) s.s)
[A_ (x:dul)s)) cdl
(a) Margin consistency implies doet can perfectly separate non robust samples in A. from robust samples.
Without margin consistency, dent cannot be good discriminator for robust and non-robust samples.
Figure 2: Illustration of Theorem 1 proof.
2 presents intuition behind the proof of Theorem For the first part of the if there is a monotonic relationship between da and dout (margin consistency), any point with din less than the threshold (non-robust) will also have dout less than Åu dout (Xo) (with (Xo) = c). For the second part (see Fig. 2b), if there are two points X1 and X2 with non- concordant din and dout (no margin consistency), then for threshold co between (x1 and (x2). they will both have different classes but no threshold of dout can classify them both correctly. The complete proof of Theorem is deferred to Appendix A Common metrics for detection include (Hendrycks & Gimpel| 2017 Corbière et al. 2019 Zhu et al. 2023): the Area Under the Receiver Operating Curve (AUROC), which ensures the ability of model to distinguish between the positive and negative classes across all possible thresholds; the Area Under the Precision- Recall Curve (AUPR), which evaluates the trade-off between precision and recall and is less sensitive to imbalance between positive and negative classes; and the False Positive Rate (FPR) at 95% True Positive Rate (TPR) (PPR@95), that crucial in systems where missing positive cases can have serious consequences, such as minimizing the number of vulnerable samples missed. The AUROC and AUPR of perfect classifier is 1, while 0.5 for a random classifier.
Proof sketch. Fig.
theorem (see Fig. 2a
3 Evaluation
3.1 Experimental Setup
Datasets and models We investigate various pre-trained models CIFARIO and CIFARI00
datasets (Krizhevskyl 2009). The majority of models were loaded from the RobustBench model
zoo CCoce et al. 2021). with few more models that are ResNet- 18 (He et al. 2016) models
trained on CIFARIO with Standard Adversarial Training (Madry et al. 2018). TRADES (Zhang et al.)
2019), Logit Pairing (ALP and CLP, Kannan et al. (2018)) and MART (Wagg al. 2020), using the experimental setup of Wang et al. (222)).
Input margin estimation This is done using FAB attack (Croce & Hein 2020a), which is attack that minimally perturbs the initial instance. Xu et al.] (2023) used it in their adversarial training strategy reliable way compute the closest boundary point given enough iterations. We perform the untargeted FAB attack without restricting the distortion to find the boundary for all the samples in the test set instead of constraining the perturbation inside given e-ball when evaluating robustness. As a sanity check for the measured distances, we compare the ratio of correct samples x with estimated input margins greater than 8/255 and the robust accuracy in & norm measured with AutoAttack (Croce & Hein, 2020b) at e = 8/255. Both quantities estimate the same thing, with mean absolute difference over the models of 1.3 and 0.48 for CIFARIO and CIFAR 100, respectively, which are reasonable.
The estimation of the input margins over the 10, 000 test samples allows to create for given threshold e a pool of vulnerable samples that can be successfully attacked at threshold e and non-
http:://ggaabbooo/////bttttttttt
5

---

Logit Margin
15
21
401%
122
Inpurt Margin
Input Margin
(a) CIFARIO
(h) CIFARI00
Figure 3: Margin consistency of various models: there strong correlation between input space margin and logit margin for most foo robust models tested, the exceptions being DIO and XU80 on CIFARIO. See Table for the references on the models. The correlations are given with standard error for the y-axis values in each interval.
vulnerable samples that were not able to be attacked. Training and distance estimations were run on an NVIDIA Titan Xp GPU (Ix).
3.2 Results and Analysis
Correlation analysis The results presented in Fig. 3 show that the logit margin has strong correlation (up to 0.86) with the input margin, which means that they have level of margin consistency for those models. The plots are given with standard error for the y-axis values in each interval. However, we also observe that two models (i.e., DIO (Ding et al. 2020) and XU80 (Xu et al.. 2023) WideResNets) have weaker correlation. We show in Sec. 3.3 that we can learn to map the feature representation of these models to a pseudo-margin that reflects the distance to the decision boundary the input space. Additional results ImageNet with &oo norm and on f--robust models on CIFARIO are given Table of appendix
Vulnerable samples detection We present the results for the robustness threshold 8/255 in Table I As expected with the strong correlations, the performance over the non-robust detection task is excellent. We can note that the metrics are lower for the two models with low correlations and particularly very high FPR @95. The performance remains quite good with different values of (cf. appendix B). Moreover, we show in appendix F.I that the empirical robust accuracy of margin-consistent models can accurately estimated by attacking only small subset of the test set.
Margin Consistency and Lipschitz Smoothness A neural network f is said to be L-Lipschitz [xf - f(x2)1l S L||x| x2l.. X2. Lipschitz smoothness important for adversarial robustness because small Lipschitz constant L guarantees the network's 's output cannot change directly constraint the
more than a factor L of the change in the input. There are strategies
Lipschitz constant to achieve 1-Lipschitz networks (Cisse et al.) 2017 Li et al. 2019 [Serrurier al. 202T) Araujo al. (023). Empirical adversarial training strategies only indirectly encourage Lipschitz's s smoothness of the model. However, we note that Lipschitz's 's continuity of the feature extractor hy does not imply margin consistency of the model. Considering two points x1 and x2 with 1, 2.
0 < <xinxx)) < din (x2), the T--Lipschiz condition implies that dout(x) < Idlnnxx; for However, as long as dout dous(xi) > O, is possible a priori to have dowt (x2) < (x)t(x)), thus violating the margin consistency condition, while still satisfying the previous relations. We also note that the strength of the correlation, i.e. the level of margin consistency, does not depend on the robust accuracy (see Fig. 4a and 4b).
Insight into when margin consistency may hold? We hypothesize that when the feature extractor hy behaves locally as an isometry (preserving distances, up to scaling factor s, at least for directions normal to the decision boundary), i.e.. ||x x' x'I1, xx) - h (x') x>)' margin consistency will occur. Given an input sample x, by definition dowe = 1 z'|l where
hep ()) and z' the closest
point to z on the feature space decision boundary. The bijectivity of local isometry implies that we
have (x') z', i.c. the representation of the closest point to x in input space matches the closest

---

Model ID
Kendall r
AUROC
AUPR
FPR@95
Acc
Rob. Ace
Architecture
DIO
Wi
2020
0.28
67.49
70.91
82.56
84.36
41,44
WideResNet-28--
XU80
QU
2024
0.43
83.30
80.50
83.42
93.69
63.89
Wid:ReaNet-28-10
Wang
0.68
29.76
79.69
39.12
ResNet-18
DM0
Dehenedelli el
202.5
0.71
94.31
93.20
32.76
91.30
57.27
XCIT-M12
ALO
Kannan et nl.
0.72
24.93
40.21
ResNet-18
CU80
Cureri HUAS
0.73
96.87
94.42
17.90
92.16
67.73
WidcResNet-28-10
WA80
Wrang el 20223
0.74
96.82
94,33
17.60
92.44
67.31
WideResNet-28-10
SE10
Senway Ci
2024
0.74
96.03
94.66
19.13
84,59
55.54
ResNet-18
CIEART
ENO
Eagstrom et
2019
0.74
95.16
95.07
24.10
87.03
49.25
ResNet-50
TRO
2019
0.74
94.63
96.13
30.93
80.72
42.23
ResNet-18
DS0
Dehenerielti ot 202.5
0.75
80
95.08
24.65
90.06
56.14
XCIT-SI2
MD0
0.75
77.00
23.23
81.85
36.91
ResNet-18
ZHO
Ziang ot 2019
0.75
95.86
95.65
24.91
84.92
53.08
WideReaNeet34410
HEO
2019
0.76
20.01
87.11
54.92
Wid:ResNeet28-10
CLO
Kannan et al. 2018
0.77
95.93
96.98
20.01
81.12
40.08
ResNet-88
20
Paing mil 211
0.78
97.65
96.39
14.40
88.61
61.04
WideResNett28-11
AD20
Addepalli er
0.81
67
13.42
85.71
52.48
ResNet-18
AD10
Addcpalli
0.82
97.86
97.68
13.26
80.24
51.06
ResNet-18
HE1
Hendryeks et al
2019
0.74
94.43
97.39
30.40
$9.23
28.42
WideReaNet-28-11
WUI
0.78
23.34
28.86
WiddResNett34-110
RE81
Rehulli el mi 2021
0.80
96.87
98.30
18.06
62.41
32,06
WideResNet
2023
67.34
32.19
XCTT-S12
CIFARTOR
CU4T
LD ci al. 2023
Curera 2026
0.82
97.07
98.48
15.62
17.21
64.08
73.85
31.65
39.18
WideResNet-28-11
WideResNet 34-10
RII Rice Rice cral 2020
0.83
96.61
99.05
18.14
53.83
18.95
PreActResNet-18
PA21
Past al. 2030
0.83
97.66
98.82
13.83
63.66
31.08
WideResNet-28-10
WA81
Waing
0.83
97.51
98.28
14.96
72.58
38.83
WideResNet 28-10
AD21
Addeppali ala
0.84
97.46
98.92
16.00
65.45
27.67
ResNet-18
ADI
Audeepali
0.84
97.65
98.99
13.88
62.02
27.14
PreActResNet
RET
Kehurn et ai
Mi21
0.85
97.97
99.05
13.21
56.87
28.50
PreActResNet-1
RAII
Rade Moosavi-Dezoooli 202
0.85
99.08
12.36
61.50
28.88
PreActResNet-18
Table 1:
Correlations and vulnerable points detection performance at
8/255 on different
adversarially trained models.
68-
1100
D50
REBO"
WAB0 Uha
68-
cofl: HEL hes1
SE30
Kormaatt
XUBI
84-
644
accuracy
accuracy
(a) CIFARIO
(b) CIFAR100
Figure 4: Distribution of the correlation between input margins and logit margins in 6x0 with robust accuracy. The strength of the correlation, which indicates the level of margin consistency, does not depend on the robust accuracy. References on models are given in Table
point to the representation of in the feature space. In that which implies margin consistency. Experimentally, what we observe is that on the one hand, the input margin and the distance between the feature representations of x and x' (feature distance) correlate and on the other hand, the feature distance and the logit margin also correlate (Fig. 5aland and Fig. 5b respectively).
will have ||x -x'| 1z|z z'
3.3 Learning a Pseudo-Margin
For the two models that are weakly Langin-consistent. we proposing to directly learn mapping that maps feature representation of sample a pseudo-margin that reflects the relative position of the samples to the decision boundary in the input space. We use learning scheme similar to the one of Corbière al. (2019), with a small ad hoc neural network for learning the confidence of the instances. Given some samples with estimations of their input margins, the objective is to learn

---

Feature Spacee: 10000-10 1,
cux -210
-3- 4000
330
014 m-1
(ol.ved 29)
Legit Ma Margin
K4lmm330 25
639
624 3.53
024
[4-
(a) Input space margins vs feature distances.
(b) Feature distances logit margins.
Figure 5: The correlations between the input margin, the distance between the feature representations of samples and their closest adversaries (feature distance h (a) hy (x')))) and the logit margin may be due to the local isometry of the feature extractor. See Table I for the specific references on the model ID. The correlations are given with standard error for the y-axis values in each interval.
to map their feature representation to a pseudo-margin that correlates with the input margins. This learning task can be for that purpose, which is a pointwise regression approach (He et al. 2008) relying on the mean squared error surrogate loss.
learning-to- -orninngt--rakkkk problem. We use simple learning-to- to-rankkk algorithm
For the experiment, we used similar architecture and training protocol as (Corbière et al. 2019 with fully connected network with five dense layers of 512 neurons, with ReLU activations for the hidden layers and a sigmoid activation the output layer. We learn using 5000 examples sampled randomly from the training set, with 20% (1000 examples) held validation. Fig. 6 and Table show the improved correlation on the learned score compared to the logit margin for both models. The correlations are given with standard error for the y-axis values in each interval. The network has learned to recover the relative positions of the samples from the feature representation.
4 Related Work
Detection tasks in machine learning are found to be of three main types:
Adversarial Detection The goal of adversarial detection (Xu et al. 2017 Carlini & Wagner 2017) is to discriminate adversarial samples from clean and noisy samples. An adversarial example malicious example found by adversarially attacking sample; it has different class while being close to the original sample. A vulnerable (non-robust) sample is a normal sample that admits adversarial example close to it. The two detection tasks very distinct. Adversarial detection is a defence mechanism like adversarial training: Tramer (2022) has established that both tasks are equivalent problems with the same difficulty.
Out-of- Distribbuutinnni (OOD) detection In OOD detection (Hendrycks & Gimpel 2017 Peng ct al. 2024; Yang et al. 2021). the objective is to detect instances far from the distribution of the training data. These are often instances with different labels from the training labels or instances with the same label as training labels but with covariate shift. For example, for a model trained on the CIFAR 10 dataset, samples from the SVHN dataset (Netzer et al.) 2011 or the corrupted version of CIFAR10-C (Hendrycks & Dietterich] 2019) are OOD samples for such model.
Misclassification Detection (MisD) consists in detecting whether the classifier's prediction is incorrect. This is also referred to Failure Detection Trustworthiness Detection (Corbière
et al. 2019 Jiang et al., 2018 Luo et al. 2021 Granese et al. 2021 Zhu et al. 2023). ). MisD is often used for selective classification (cl (classification with reject option) (Geifman & al-Yaniv 2017) to abstain from predicting samples on which the model is likely to be wrong. A score for non-robust detection cannot tell if the sample is incorrect, as vulnerable sample could be from any side of the decision boundary.
Formal robustness verification aims certifying whether a given sample e-robust or if not an adversarial counter-example can be provided (Brix al., 2023b). Some complete exact methods
8

---

DI0
3.9
0.9
Logit Margin (r=0.28, FPR
9.8
Learned Pseudo Margin (T=0.57, FPRg95-51.13)
08
8.6-
0.6
Margin
0.5
Margin
0.5
9.3
0.3
0.2
0.
Logit Margin (t=0.43, FPR@95=85.37)
Learned Pseudo-Margin (T=0.62, FPRU895-33.00
8.0
9.04
0.06
0.10
0.14
0.0
0.02
0:66
0.38
Input Margin
Input Margin
Figure 6: Correlation improvement of the learned gpeudo-margin the logit margin for DIO et al. 2020) and XU80 (Xu et al. 2023).
DDig
Model ID
Margin
Kendall tau (?)
AUROC
AUPR
FPR@95 ())
Acc.
Rob. Acc
D10 Ding al. 2020
Logit margin
0.28
67.49
70.91
82.56
84.36
41.44
Learned pscudo-margin
0.57
88.49
89.04
51.13
XU80
Xu et 2023
Logit margin
0.43
83,30
80.50
83.42
93.69
63.89
Learned mecudo-margin
0.62
93.66
90.22
33,00
Table 2: Comparison of the correlation and detection performance between the actual logit margin and the geudo-marghe learned. The models are initially weakly margin-consiitent, but the pseudo-mane margin learned from feature representations simulates the margin consistency with higher correlation and better discriminative power.
based on solving Satisfiability Modulo Theory problems (Katz et al. 2017 Carlini et al., 2017 Huang al. (2017) or Mixed- Mixed-Integer Linear Programming (Cheng et al. 2011 Lomuscio & Maganti 2017 7
Fischetti & Jo 2017) provide formal certification given enough time. However, in practice they are tractable only up to 100,000 activations (Tenng et al. 2019). Incomplete but effective methods based methods (Zhang al.) 2018 Salman Xu et al.) 2020 2021 Zhang et al. 2022; Shi et al., (203) are faster but conservative, without guaranteed certifications even if given enough time. Scaling them to bigger architectures such as WideResNets and large Transformers is still challenging even with GPU accelartion (Brix al. (2023a König et al. Wengg et al. (018) converts the problem of finding the robust radius (input margin) as local Lipschitz constant estimation problem. Computing the Lipschitz constant of Deep Nets is NP-hard (Virmaux & Scaman 2018) and Jordan & Dimakis (2020) proved that there is no efficient algorithm to compute the local Lipschitz constant. The estimation provided by Weng et al. (2018) requires random sampling and remains computationally expensive to obtain good approximation. Vulnerability detection with margin-consistent models does not provide certificates but empirical estimation of the robustness of sample as evaluated by adversarial attacks. At scale, can help filter the samples to undergo formal verification and a more thorough adversarial attack for resource prioritization.
on linear and convex relaxation methods and and-nchhann-Baaee
al., 2019)
5 Limitations and Perspectives
The scope of this work is ép robustness measured by the input space margin; the minimum distortion that changes the model's decision while this does not give a full view of the l robustness. Samples may be at the same distance to the decision boundary and have unequal unsafe neighbourhoods given by an average estimation over the e-neighbourhood considered. The average estimation of local robustness for given c-neighborhood remains open problem, so whether it is possible to extract other notions robustness from the feature representation efficiently could be a potential avenue for further exploration.
Vulnerability detection scope
Attack-based verification The margin consistency property does not rely on attacks; however, its verification and the learning of pscudo-margin with attack-based estimation may not be

---

possible if the model cannot be attacked on a sufficient number of samples. The assumption is that we can always successfully provide the closest point the decision with sufficient budget. This is a reasonable assumption since the studied models are not perfectly robust, and the empirical evidence so far with adaptive attacks is that no defence foolproof, which justifies the need to detect the non-robust samples. It might occur that we need to combine with an attack such as CW-attack (Carlini & Wagner 2016) to find the closest adversarial sample.
Influence of terminal phase of training The work of Papyan et al. (2020) shows that when deep neural network classifiers are trained beyond zero training error and beyond zero cross-entropy loss (aka terminal phase of training), they fall into state known as neural collapse. Neural collapse is state where the within-class variability of the feature representations collapses to their class means, the class means, and the classifiers become self-dual and converge to a specific geometric structure, an equiangular tight frame (ETF) simplex, and the network classifier converges to nearest train class center. This implies that we may lose the margin consistency property. While neural collapse predicts that all representations collapse on their class mean, in practice, perfect collapse is not quite achieved, and it is precisely the divergence of representation from its class mean (or equivalently its wi) which encodes the information we seek about the distance to the decision boundary in the input space, Exploring the impact of the neural collapse margin consistency models tend toward a collapsed state could provide valuable insights into generalization and adversarial robustness.
Adaptaptive attacks and adversarial examples In this paper, we study the margin consistency of models on their training distribution by reporting the Kendall rank correlation between the logit margin and the input margin on the test set. The study of this property on inputs from different distribution or specifically crafted examples is left for future research. However, we observe that the adversarial examples used for the input margin estimation have significantly smaller logit margins than the detection thresholds (see Table in appendix (D). This indicates that these specific adversarial examples are indeed identified as non-robust instances, together with clean non-robust samples.
6 Conclusion
This work addresses the question of efficiently estimating local robustness in the lp sense at per-instance level in robust deep neural classifiers in deployment scenarios. We introduce margin consistency as a necessary and sufficient condition to use the logit margin of a deep classifier as a reliable proxy estimation of the input margin for detecting non-robust samples. Our investigation of various robustly trained models shows that they have high margin consistency, which leads to a high performance of the logit margins in detecting vulnerable samples to adversarial attacks. We also find that margin consistency does not always hold, with some models having a weak correlation between the input margin and the logit margin. In such cases, we show that it is possible to learn to map the feature representation to a better-correlated pseudo-margin that simulates the margin consistency and performs better on vulnerability detection. Finally, we present some limitations of this work, mainly the scope of robustness, the attack-based verification, the impact of neural collapse in terminal phases of training, and vulnerability to adaptive attacks. Beyond its highly practical importance, we see this motivation to extend the analysis of robust models and the properties of their feature representations in the context vulnerability detection.
Acknowledgements
This work is supported by the DEEL Project CRDPJ 537462-18 funded by the Natural Sciences and Engineering Research Council of Canada (NSERC) and the Consortium for Research and Innovation in Aerospace in Québec (CRIAQ), together with its industrial partners Thales Canada inc, Bell Textron Canada Limited, CAE inc and Bombardier inc
References
Addepalli, S., Jain, S., Sriramanan, G., Khare, S., and Radhakrishnan, V. B. Towards achieving adversarial robustness beyond perceptual limits. In ICML 2021 Workshop on Adversarial Machine Learning, 2021. URL http///p///ooeeeriie.....e http:arreweeeeeeeeooooooo/dadaaa 2555
htttps:/deee.queee
10

---

Addepalli, S., Jain, S., Sriramanan, G., and Venkatesh Babu, R. Scaling adversarial training to large perturbation bounds. In European Conference on Computer Vision, pp. 301-316. Springer, 2022.
Araujo, A., Havens, A. J., Delattre, B., Allauzen, A., and Hu, B. A unified algebraic perspective on lipschitz neural networks. In The Eleventh International Conference on Learning Representations, 2023.
net/forrum:d=kK1IGGGGGB
Bastani, O., Ioannou, Y., Lampropoulos, L., Vytiniotis, D., Nori, A., and Criminisi, A. Measuring neural net robustness with constraints. Advances in neural information processing systems, 29, 2016.
Biggio, B., Corona, I., Maiorca, D., Nelson, B., Šrndić, N., Laskov, P., Giacinto, G., and Roli, Evasion attacks against machine learning at test time. In Joint European conference on machine learning and knowledge discovery in databases, pp. 387-402. Springer, 2013.
Brix, C., Bak, S., Liu, C., and Johnson, T. T. The fourth international verification of neural networks competition (vnn-comp 2023): Summary and results. arXiv preprint arXiv:2312.16760, 2023a.
Brix, C., Müller, M. N., Bak, S., Johnson, T. T., and Liu, C. First three years of the international verification of neural networks competition (van-comp). International Journal on Software Tools for Technology Transfer, 25(3):329-339, 2023b.
Carlini, N. and Wagner, D. Adversarial examples are not easily detected: Bypassing ten detection methods. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp- 3-14, 2017.
Carlini, N. and Wagner, D. A. Towards evaluating the robustness of neural networks. 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57, 2016.
Carlini, N., Katz, G., Barrett, C., and Dill, L. Provably minimally-distorter adversarial examples. 2017.
arXiv preprint arXiv:
Carmon, Y., Raghunathan, A., Schmidt, L., Duchi, C., and Liang, S. Unlabeled data improves
adversarial robustness. Advances in neural information processing systems, 32, 2019.
Chattamvelli, R. Correlation in Engineering and the Applied Sciences: Applications in R. Springer Nature, 2024.
Cheng, C.-H., Nührenberg, G., and Ruess, H. Maximum resilience of artificial neural networks. In Automated Technology for Verification and Analysis: 15th International Symposium, ATVA 2017, Pune, India, October 3-6, 2017, Proceedings 15, pp. 251-268. Springer, 2017.
Cisse, M., Bojanowski, P., Grave, E., Dauphin, Y., and Usunier, N. Parseval networks: Improving robustness to adversarial examples. In International conference on machine learning, pp. 854-863.
PMLR, 2017.
Corbière. C., Thome, N., Bar-Hen, A., Cord, M., and Pérez, P. Addressing failure prediction by learning model confidence. Advances in Neural Information Processing Systems, 32, 2019.
Croce, and Hein, M. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning, pp. 2196-2205. PMLR, 2020a.
Croce, F. and Hein, M. Reliable evaluation of adversarial robustness with ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206-2216. PMLR, 2020b.
Croce, F., Andriushchenko, M., Schwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. Robustbench: standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021. URL https:// nett:/orum/xd--SS22PP
Cui, J., Tian, Z., Zhong, 7.,, Qi, X., Yu, B., and Zhang, Decoupled kullback-leible divergence loss. arXiv preprint arXiv:2305.139488 2023.

---

Debenedetti, E., Sehwag, V., and Mittal, P. A light recipe to train robust vision transformers, In 2023 IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), pp. 225-253, IEEE, 2023.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255.. Icce, 2009.
Ding, G. W., Sharma, Y. Lui, K. Y. C.,, and Huang, R. Mma training: Direct imput space margin max- imization through adversarial training. International Conference on Learning Representations. 2020. htt.l///ee/.oooeeeieeeee/////eee wtt..e/e/oorrn...ddteerraaaaaaa
Drcossi, T., Ghosh, S., Sangiovanni- Vincentelli, A., and Seshia, S, A. formalization of robustness for deep neural networks. arXiv preprint
2019.
Elsayed, G., Krishnan, D., Mobahi, H., Regan, K., and Bengio, S. Large margin deep networks for classification. Advances in neural information processing systems, 31, 2018
Engstrom, L.,, Ilyas, A., Salman, H., Santurkar, S., and Tsipras, D. Robustness (python library), 2019.
URL https:t/gittttttt http////aaaallllabboruuuusseeeee
Evtimov, I., Eykholt, K., Fernandes, E., Kohno, T., Li, B., Prakash, A., Rahmati, A., and Song, D. Robust physical-world attacks on machine learning models. arXiv preprint urXiv:1707.08944 (3):4, 2017.
Fawzi, A., Fawzi, H., and Fawzi, O. Adversarial vulnerability for any classifier, Advances in neural information processing systems, 31, 2018.
Fischetti, M. and Jo, Deep neural networks as 0-1 mixed integer linear programs: A feasibility study. arXiv preprint 17/2.0677 2017.
Geifman, Y. and Yaniv, Selective classification for deep neural networks. Advances in neural information processing systems, 30, 2017.
Gnanasambandam, A., Sherman, A. M., and Chan, S. H. Optical adversarial attack. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 92-101, 2021.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explaining and harnessing adversarial examples. In Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL
http:///axiv: org/ wwtg/ba///aae. htt/aaabs/11..1 6572
Gourdeau, P., Kanade, V., Kwiatkowska, M., and Worrell, J. On the hardness of robust classification. The Journal of Machine Learning Research, 22(1):12521-1255 2021.
Granese, F,, Romanelli, M., Gorla, D., Palamidessi, C., and Piantanida, P. Doctor: A simple method for detecting misclassification errors. Advances in Neural Information Processing Systems, 34: 5669-5681, 2021.
Han, T., Srinivas, S., and Lakkaraju, H. Efficient estimation of local robustness of machine learning models. In ICML 3rd Workshop on Interpretable Machine Learning in Healthcare (IMLH), 2023, URL
He, C., Wang, C., Zhong, Y.-X., and Li, R.-F. A survey on learning to rank. In 2008 International Conference on Machine Learning and Cybernetics, volume 3, pp. 1734-1739, Iece, 2008
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness common corruptions and perturbations. International Conference on Learning Representations, 2019. URI https //openreview nwt. wtt.te/ooorr.....ttesstiito....
12

---

Hendrycks, D. and Gimpel, K. A baseline for detecting misclassified and out -fitrribttibuttion examples in neural networks. In International Conference on Learning Representations, 2017. URL https://openrevieen http://pperreeeee.eeeforrrrrrree hit.oorruumnddlgggTT1....
Hendryeks, D., Lee, K., and Mazeika, M. Using pre-training can improve model robustness and
uncertainty. In International conference machine learning, pp. 2712-2721. PMLR, 2019.
Huang, X., Kwiatkowska, M., Wang. S., and Wu, M. Safety verification of deep neural networks. In
International conference computer aided verification, pp. 3-29. Springer, 2017.
Jiang, H., Kim, B., Guan, M., and Gupta, M. To trust or not to trust a classifier. Advances in neural information processing systems, 31, 2018.
Jordan, M. and Dimakis, A. G. Exactly computing the local lipschitz constant of relu networks. Advances in Neural Information Processing Systems, 33:7344-73555 2020.
Kannan, H., Kurakin, A., and Goodfellow, Adversarial logit pairing. arXiv preprint arXiv: 1803.063777 2018.
Katz, G., Barrett, C., Dill, D. I., Julian, K., and Kochenderfer, M. J. Reluplex: An efficient smt solver for verifying deep neural networks. In Computer Aided Verification: 29th International Conference, CAV 2017, Heidelberg, Germany, July 24-28, 2017, Proceedings, Part I 30, pp. 97-117. Springer, 2017.
König, M., Bosman, A. W. Hoos, H. H., and van Rijn, N. Critically assessing the state of the art in neural network verification. Journal of Machine Learning Research, 25(12):1-53, 2024.
Krizhevsky, A. Learning multiple layers of features from tiny images. Technical Report TR-2009, University of Toronto, 2009.
Li, Q., Haque, S., Anil, C., Lucas, J., Grosse, R. B., and Jacobsen, J.-H. Preventing gradient attenuation in lipschitz constrained convolutional networks. Advances in neural information processing systems, 32, 2019.
Liu, F,, Han, B., Liu, T., Gong, C., Niu, G., Zhou, M., Sugiyama, M., et al. Probabilistic margins for instance reweighting in adversarial training. Advances in Neural Information Processing Systems, 34:23258-2322 23269,, 2021.
Lomuscio, A. and Maganti, L. An approach to reachability analysis for feed-forward relu neural networks. arXiv preprint arXiv: (00.:0736,,,, 2017.
Luo, Y., Wong, Y., Kankanhalli, M. S., and Zhao, Q. Learning to predict trustworthiness with steep slope loss. Advances in Neural Information Processing Systems, 14:21533-21544 2021.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D., and Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. URL http:::/
Mickisch, D., Assion, F., Greßner, F,, Günther, W., and Motta, M. Understanding the decision boundary of deep neural networks: An empirical study. arXiv preprinl arXiv:2002.01810, 2020.
Moosavi-Dezfoolii, S.-M., Fawzi, A., and Frossard, P. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2574 2582, 2016.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Pang, T., Lin, M., Yang, X., Zhu, J., and Yan, S. Robustness and accuracy could be reconcilable by (proper) definition, International Conference on Machine Learning, pp. 17258-17277. PMLR, 2022.
13

---

Papyan, V., Han, X., and Donoho, D. L. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40)24452224465) 2020.
Peng. B., Luo,, Y., Zhang, Y., Li, Y., and Fang, Z. Conjnorm: Tractable density estimation for out-of- distribution detection. In Proceedings of the International Conference on Learning Representations, 2024.
Rade, R. and Moosavi-Dezfooll S.-M. Helper-based adversarial training: Reducing excessive margin to achieve a better aceuracy vs. robustmess trade-off. In JCML 2021 Workshop on Adversarial Machine Learning, 2021. URL https::/ B
Rebuffi, S.-A., Gowal, S., Calian, D. A., Stimberg. F., Wiles, O., and Mann, T. Fixing data augmentation to improve adversarial robustness. arXiv preprint arXiv:2103.0194: 2021.
Rice, L.. Wong, E., and Kolter, Z. Overfitting in adversarially robust deep learning. In International Conference on Machine Learning, pp. 8093-8104, PMLR, 2020.
Salman, H., Yang. G., Zhang, H., Hsieh, C.-., and Zhang, A convex relaxation barrier to tight robustness verification of neural networks. Advances in Neural Information Processing Systems, 32:9835-9846, 2019.
Sehwag, V., Mahloujifar, S., Handina, T., Dai, S., Xiang, C., Chiang, M., and Mittal, P. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? arXiv preprint arXiv:2104 09425,, 2021.
Serrurier, M., Mamalet, F., González-Sanz, A., Boissin, T., Loubes, J.-M., and Del Barrio, Achieving robustness in classification using optimal transport with hinge regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 505- 2021.
Seshia, S. A., Desai, A., Dreossi, T., Fremont, J., Ghosh, S., Kim, E.,, Shivakumar, S., Vazquez- Chanlatte, M., and Yue, X. Formal specification for deep neural networks. In International Symposium on Automated Technology for Verification and Analysis, pp. 20-34. Springer, 2018.
Shi, Z., Jin, Q.. Kolter, Z., Jana, S., Hsich, C.-J,, and Zhang, H. Formal verification for neural networks with general nonlincarities via sanch-andd-ound 2nd Workshop on Formal Verification of Machine Learning (WFVML 2023), 2023.
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. J., and Fergus, R. Intriguing properties of neural networks. In Bengio, Y. and LeCun, Y. (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. UtLtatt:://ant //triiv.//rii.///aa orw.oogg/ao. hogg/bbs/a 1312. 6199
Tjeng, V., Xiao, K. Y., and Tedrake, R. Evaluating robustness of neural networks with mixed integer programming. In International Conference on Learning Representations, 2019. URL
https::/opperevviee wtwparrvvieeeeee/oruun/eraa www.te/oorrui.d--HyIiiii.te..
Tramer, F. Detecting adversarial examples is (nearly) as hard as classifying them. In International Conference on Machine Learning, pp. 21692-21702. PMLR, 2022.
Virmaux, A. and Scaman, K. Lipschitz regularity of deep neural networks: analysis and efficient estimation. Advances in Neural Information Processing Systems, 31, 2018
Wang, Y., Zou, D., Yi, J., Bailey, J., Ma, X., and Gu, Q. Improving adversarial robustness requires revisiting misclassified examples. In International Conference on Learning Representations, 2020. URL http::: wwt.:e//oooommmmidrr........
Wang, Z., Pang, T. Du, C., Lin, M., Liu, W., and Yan, Better diffusion models further improve adversarial training. In International Conference on Machine Learning (ICML), 2023.
Weng, T.-.. Proven: Verifying robustness of neural networks with a probabilistic approach pow- erpoint presentation, ltttps:///cmmm.. htt.ee/e/eeelaeeeeeeee////1///1e ggwadbaalll11111-aaaa1 11 12- 15- 4739 www.-----7----eoeeeeeeee wrwp://--rrooeeeeeieii.....peee 2019. (Accessed on 05/23/2023).
14

---

Weng, T.-W., Zhang, H., Chen, P.-Y.. Yi, J., Su, D., Gao. Y., Hsieh, C.-J., and Daniel, L. Evaluating the robustness of neural networks: An extreme value theory approach. International Conference on Learning Representations, 2018. URL. httpttts///ppeeeeieieee
Wu, D., Xia, S.-T., and Wang, Y. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33:2958-29606 2020.
Xu, K., Shi, Z., Zhang, H., Wang, Y., Chang, K.- Huang, M., Kailkhura, B., Lin, X., and Hsieh, C.-J. Automatic perturbation analysis for scalable certified robustness and beyond. Advances in Neural Information Processing Systems, 33, 2020.
Xu, K., Zhang, H., Wang. S., Wang, Y., Jana, S., Lin, X., and Hsich, C.-J. Fast and Complete: Enabling complete neural network verification with rapid and massively parallel incomplete verifiers. In International Conference on Learning Representations, 2021. URL htl.htttss:/oppprvviiee net/ forum?id nVZtXBI6L.n
Xu, W., Evans, D., and Qi, Feature squeezing: Detecting adversarial examples in deep neural 2017.
networks. arXiv preprint
Xu, Y., Sun, Y., Goldblum, M., Goldstein, T., and Huang, F. Exploring and exploiting decision boundary dynamics for adversarial robustness. In The Eleventh International Conference on Learning Representations, 2023. httpp::://pparrrreee..eeeeeeeeee httppp/aaeeeeeeeeeeoooooaaaaaaaa
Yang, J., Zhou, K., Li, Y., and Liu, Z. Generalized out-o--distribution detection: A survey. arxiv. arXiv preprint 1 10. 1334, 2021.
Zhang. H., Weng. T.-W., Chen, P.-Y., Hsieh, C.-J,, and Daniel, L. Efficient neural network robustness certification with general activation functions. Advances in Neural Information Processing Systems, 2018 URL http:tttt////rrrs v. org/ pdf 1811. 00866.000...0... .ddf]
Zhang, H., Yu, Y., Jiao, J., Xing, E., El Ghaoui, L., and Jordan, M. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019.
Zhang, H., Wang, S., Xu, K., Wang, Y., Jana, S., Hsieh, C.-J,, and Kolter, Z. A branch and bound framework for stronger adversarial attacks of ReLU networks. In Proceedings of the 39th International Conference on Machine Learning, volume 162, pp. 26591-26604, 2022.
Zhang. J., Zhu, J., Niu, G., Han, B., Sugiyama, M., and Kankanhalli, M. Geometry-aware instance- reweighted adversarial training. arXiv preprint arXiv:2010.01736, 2020.
Zhong, 7., Tian, Y., and Ray, B. Understanding local robustness of deep neural networks under natural variations. International Conference on Fundamental Approaches Software Engineering, pp- 313-337. Springer, Cham, 2021.
Zhu, F., Cheng, Z., Zhang, X.-Y., and Liu, C.-L. Openmix: Exploring outlier samples for misclassifi- cation detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12074- 12083, 2023.
15

---

A Proof of Theorem 1
then for any robustness threshold 6, there exists threshold for the logit margin dout that perfectly separates non-robust samples and robust samples. Conversely, if for any robustness threshold e, dout admits a threshold X that perfectly separates non-robust samples from robust samples, then the model is margin-comsistemt
Theorem 1. If a model is
Proof. Formally, for finite sample S and nonegative values e 2 0, A 2 O, we define:
A = {I S (x) S < and == : {I
S dout(x) (x) X}.
We say that dout perfectly separates non-robust samples from robust samples if for any finite sample C X' and every 2 0 there exists À 2 0 such that A B
Necessity: We proceed by contraposition and assume that the model is not margin-consistent, i.e., there exist samples x1 and X2 such that (x1) I dout (xe) and (xx) > dip (x2). By taking S = {x1.X2) x23 and e = din (x2) we have that A {x2}. However for any À 2 0, if E B then dows (x1) S. (X2) I À and so X E B Therefore dout does not perfectly separates non-robust samples from robust samples.
Sufficiency: Let's assume the model is margin- consistent. Let S be a finite sample and consider a threshold Let Xy be element of the finite set A such that din (Xo) max din (z) x A
and let x = (xo)( Since the model is sargin-consistem. then for every x E S:
E A +> din dx)(x) S + din(x) S din (Xo) <> dowt (x) S dont (X0) +> dont (x) S S + B
margin consistency
This means we have A B which shows that dout perfectly separates non-robust samples from robust samples.
Our formulation of perfect separation using finite samples is not fundamentally necessary, however avoids dealing with the intricacy of the continuum.
B Detection Performance with Different Values of
We present in Fig. 7 the performance of the detection for various values of the robustness threshold. can see that the high margin consistency allows the logit margin to be a good proxy for detection at various thresholds. Note that below 2/255 and beyond 16/255, the ratio of vulnerable points to non-vulnerable points becomes too imbalanced, with little to no positive instances beyond 32/ 2/255
C Equidistance Assumption of the Linear Classifiers
In Eq. in Sec. 2.1 we show that we can approximate the exact feature margin by the logit margin when the classifiers Wi: are equidistant, i.e. ||w, - w C whenever * j. The results in Table show that using the logit margin instead of the exact minimum feature margin has negligible effect on the results. However, computing the exact feature margin requires computing the minimum over K pairs of scaled logit differences. The approximation provided by the logit margin thereby circumvents the computational overhead of the minimum search, which can take a second instead of just microseconds for inference. This difference can add up to hours at scale, offering scalability when dealing with a large number of classes. We additionally provide boxplots for the K(K 1)/2 distances between pairs of classifiers for each model (45 for CIFARIO and 4950 for CIFARIOO1 in Fig. 8 and
D Logit Margins of Adversarial Examples
Adversarial examples are perturbed samples that are close to the decision boundary. Therefore, we would expect these samples to have very small logit margin for strongly margin-consistent models.
16

---

CIFAR10 AD10(T: 0.82)
0.8
0.8
True Positive Rate
Precision
0.6
2-255 (AUC = 0.99)
2-2/255 (AUPR - 0.97)
#=//255 (AUC = 0.99)
E=//255 (AUPR = 0.97)
0.4
E-8/255 (AUC 0.98)
0.4
E-8/255 (AUPR 0.98).
==12/255 |AUC 0.97)
c=12255 (AUPR 0.98)
E-16/255 (AUC
E-16/255 (AUPR
e=32/255 (AUC
0.2
E=32/55 (AUPR
chance level [AUC 0.5)
chance level (AUC = 0.5)
0.0
perfect level (AUC 1.0)
0.0
perfect level (AUC 1.0)
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
False Positive Rate
Recall
(a) Model AD10 Addepalli et al.
2021 on CIFARIO
CIFAR100 RA11(T: 0.84)
1.0
1.0
True Positive Rate
0.6
Precision
0.6
c=21255 (AUC 0.99)
c=2/555 (AUPR 0.98)
E-4/255 (AUC
E-4/255 (AUPR 0.98)
E-8/255 (AUC 0.98)
E-8/255 (AUPR = 0.99).
e=12/255 |AUC 0.98)
c=12/255 [AUPR 0.99)
E=16/255 (AUC 0.97)
E=16/255 (AUPR 1.00)
0.2
E-32/255 (AUC 0.98)
0.2
E-32/255 (AUPR 1.00)
chance level (AUC 0.5)
chance level (AUC 0.5)
perfect level (AUC 1.0)
perfect level (AUC L.0)
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
False Positive Rate
Recall
(b) RAI1 Rade Moosavi-Dezfooli
2021
on CIFARI00
Figure 7: Variation of AUROC score for different threshold values
Here, we study the adversarial examples we used estimate the input margin. In Table we present the 99th percentile of logit margins of adversarial examples and the detection threshold selected to obtain 95% True Positive Rate. We can observe that the values of the adversarial logit margins are significantly smaller than the detection threshold, they would be detected as non-robust just like clean samples that close to the decision boundary.
E Results on ImageNet and ro-usttst models CIFAR10
Table 5 shows the results with foe -rbust models on ImageNet Deng et al. 2009 a larger dataset, and results on robbtt models (only available on CIFAR10 in Robustbench). The results extend well in both situations.
17

---

120
100
80
60
40
20
HB
HB
0
DIO
D50
XU80
DMO
MRO
MD0
SE10
PA20
HEO
TRO
CLO
ENO
ALO
AD20
CU80
RE80
ZHO
WA80
AD10
Figure 8: Equidistance of classifiers for CIFAR10 models. The boxplot reports the distances'
minimum value, lower quartile (QI), median, upper quartile (Q3), and maximum value.
50
40
30
20
HH
H
ADO
AD20
CU80
CU40
D50
HEO
PA20
RA10
RE80
RE10
RIO
WA80
WUO
Figure 9: Equidistance of classifiers for CIFAR100 models. The boxplot reports the distances'
minimum value, lower quartile (QI), median, upper quartile (Q3), and maximum value.
18

---

Model
Lm
Pm
AUROC
AUROC Fm
AUPR Lm
AUPR Fm
FPR95 Lm
FPR95 Fm
Acc
Rob,
Architecture
0.32
67.49
70.91
74.28
82.56
80.57
84.36
WilkResNes-844
50.50
82.31
WideRReNNer
0.68
0.70
92.95
94.92
29.76
27.43
79.09
39:12
ResNer-11
ALD
0.72
0.74
94-67
95.5 12
95.98
95.75
24.93
22.21
80.38
40.21
RenNet-18
CU80
0.73
0.75
95.87
97.52
94.42
95.47
17.90
14.77
92:16
57:73
WedestesNes
CIFARID
WA80
0.74
0.76
96.82
97. 50
94.35
95.43
17.60
14.42
92:44
67.31
WidbRo-Nes-2--00
SE10
0.74
0.71
96.03
96.39
94.66
95.52
19.15
16.85
84.59
55.54
Res 18
END
0.74
0.76
95.16
95.00
95.07
95.85
24.10
21. 11.22
67.03
49.25
ReaNar-Si
TR0
0.74
0.77
94.63
95.5
96.13
96.74
30.93
25.73
80.72
42.23
ResNet
DS0
0.75
08
24.65
90106
56.14
NCTES12
0.75
95.
23.23
23.75
36.91
RoSNe118
ZHO
0.75
0.:77
95.86
95.
24,91
21.74
84-92
53.08
ADII
0.82
0.84
97.86
98
97.68
98.09
13.26
11.50
80124
51.06
ResNer-18
HET
0.74
0.74
94.43
94.41
97.39
97.39
30,40
30,60
59.23
28.42
Wed 28:
WIll
0.78
0.79
95.81
95.83
98.00
98.01
23,
22.
661
28.86
WitzRoNNes-4400
RES12
0.80
0.80
96.87
96.91
96.30
98.32
18.05
17.49
62.41
32.06
WidkReeNet 28 10
DS1
0.81
0.81
95.78
96.78
98.30
95.30
19.15
18,46
67:34
$2.19
XCTIS12
C1441
0.82
0.83
97.07
97
08.48
98.49
17.21
17:35
64.00 64-08
31
Wilk 34-110
CIEARLOO
CU81
0.83
0.83
97.41
97.43
98.24
96:25
15.62
15.67
73.85
39.18
WideReeNet 10
0.83
0.83
18.14
17.70
53.83
18:95
PreActReeNet-188
0.833
13.85
13.80
63:06
WideReesNet2HH
0.81
0.83
97
98.27
14,96
14.86
72.58
38.83
WilbResNNst28-10
AD21
0.84
0.84
97.16
97.48
98.92
98.93
16.00
15.36
65.45
27.67
ResNett 18
ADD
0.84
1.884
97.63
97.67
99.99
99.59
13.88
13.34
62.00
27.14
REII
0.85
0.84
07.97
97. 88
99.05
99.01
13.21
13.36
$6.87
28. 33.50
Pre Ret-Re---
RAII
0.85
0.85
98.01
98.01
99.08
99.08
12.36
12.20
61.50
28.88
PreActResNet 18
Table 3:
Correlations between the input margin and the logit margin (Lm) the exact Feature margin
(Fm) and detection scores on CIFAR1O (Coe, c = 8/255).
F Additional Applications
F1 Sample Efficient Robust Accuracy Estimation
Margin consistency enables empirical robustness evaluation over an arbitrarily large test set by only attacking a small subset of test samples. For a robustness evaluation at threshold c (e.g., = 8/ 8/255 in foo norm on CIFAR10 and CIFAR100), we randomly sample small subset of the test set and determine the threshold À for the logit margin that best corresponds to c. We propose to use this threshold estimate the standard robust accuracy by evaluating the proportion of test samples which are correctly classified and whose logit margin is above À (see Algorithm T). A naive way to set
Algorithm 1 Sample Efficient Robustness Estimation with Margin Consistency
1: Input: Test Dataset (X, Y) (X' 2) Robustness threshold >> 0, Subset size & N.
2: Output: Robust Accuracy Estimation A,
3: Select uniformly at random subset X,, of samples from X.
4: Compute the estimations of the input margins on Xn, D, (x):
5: Create ground truth labels for vulnerability at threshold c i.e. 1 (x)ss) (x), for x E X,,
6: Determine the threshold À of dout that gives best approximation of robust accuracy on X..
7: A,, = [xx X doue(x) (x) > À and j(x) //N
the threshold À line 6 of Algorithm 1 would be to set it to the detection threshold or a = 90% TPR, but the logit margin threshold could vary from one model to another; therefore a better way is select it by tuning over values a Z 0.80 that gives the best approximation of the robust accuracy in terms of the absolute error on the small subset X,. The same idea allows estimating model's vulnerability over large dataset without the labels.
95% TPR
We show that this leads to an accurate estimation of the robust accuracy of the investigated models evaluated with over 10, 000 by attacking only a random subset of size 500. Fig. shows the absolute error of the estimation obtained using 500 samples. As expected, the estimation over the two weakly margin consistent models is not accurate while having relatively small absolute difference on the strongly margin consistent models.
F.2 Robustness Bias Analysis using the Logit Margin
Robust models often display robustness bias, namely disparity of robustness across classes. Interest- ingly, strongly margin- model (see Fig. [T top row), we show that these discrepancies across classes with respect to input margin are reflected in the logit margin. Additionally, the margin
19

---

Logit margin of Adversarial Examples (99th percentile)
Threshold of logit margin at 95% TPR for detection
Model ID
AD1
0.01
1.42
DSO
0.61
2.19
DM0
1.64
3,01
MRO
0.04
8.35
MDO
0.06
11.04
SE10
0.06
2.83
PA20
0.14
1.69
CIFARIO
HEO
0.07
2.82
TRO
0.07
4.11
CLO
0.03
5,76
ENO
0.13
5.06
ALO
0.03
5.11
AD20
0.03
1.33
CU80
1.32
2.02
RE802
0.13
1.87
ZHO
0.17
2.87
WA80
0.70
2.14
ADI0
0.01
1.20
ADI
0.01
1.42
AD21
0.02
1.44
CU81
0.13
2.09
CU41
0.04
1.95
DSI
0.13
1.36
CIFARI00
HEI
0.06
1,81
PA21
0.08
1.56
RA11
0.08
1.77
RE812
0.17
1.73
RE11
0.07
1.40
RII
0.03
3.42
WA81
0.09
1.59
WUI
0.06
2.02
Table 4: Logits of the adversarial examples are very small compared to the logit margin for the logit margin threshold for the detection at epsilon=8/255. See Table foo the specific references the model ID.
Model ID
Kendall tau
AUROC
AUPR
FPR95
Acc
Rob. Acc
Architecture
DI02
0.66
95.91
93.05
21.94
88.02
66.09
WideResNet
AU02
0.69
95.59
87.41
19.30
91.08
72.91
ResNet-50
SE102
0.72
97.07
90.99
13.25
89.76
74,41
ResNet-18
WA802
0.74
98.55
93.59
8.00
95.16
83.68
WideResNet-28-
RI02
0.74
96.82
92.86
13.98
88.67
67.68
PreActResNet-16
CIFARI0 (22)
EN02
0.75
96.92
92.78
13.25
90.83
69.24
ResNet-50
RO02
0.76
98.19
96.51
10.15
89.05
66.44
WideResNNet28-11
RE802
0.77
98.72
95.02
6.66
91.79
78.80
WideResNet-28-11
RE102
0.79
98.58
95.32
7.04
90.33
75.86
PreActResNNet188
RA102
0.80
99.00
96.78
5.53
90.57
76.15
PreActResNet-18
SSIO
0.63
90.95
90.06
36.47
72.56
48.08
VIT-S ConvStem
ImageNet (f.o)
ENI0
0.73
94.95
97.34
25.07
62.56
29.22
ResNet-50
SAII
0.74
95.00
96.50
24.88
64.02
34.96
ResNet-50
SAIO
0.77
95.16
97.99
29.31
$2.92
25.32
ResNet-18
WOI0
0.78
96.28
98.34
25.65
$5.62
26.24
ResNet-50
Table 5: Correlations between the input margin and the logit margin and detection scores on CIFARI0
(lz, 0.5) and ImageNet (loc, 4/255, 1000 samples). See Table for the specific references
on the model ID.
20

---

sebuot Moo
Mapno
Lop Magt
Robust Accuracy
Robustt Accurcyy
211
Modals
Models
(a) CIFARIO
(b) CIFARI00
Figure 10: Estimations the robust accuracy reported by Robustbench using logit margins with only 500 samples are quite accurate both CIFARIO and CIFAR 100 for strongly margin-consistent models. The numbers indicate the absolute difference between the two values, averaged over ten subsets. See Table for the specific references the model ID.
0.5-
2
Robustness bias analysis with the input margin (first column) vs the logit margin (second column) and correlation scores per class (third column). We have margin-consistent model on the top row, and on the bottom row, we have a weakly margin-consistent model. On the first column (resp. second column), each boxplot represents the distribution of the input margin (resp. logit margin) for the corresponding CIFAR 10 class. The dotted blue line indicates the threshold on the first two columns and is the correlation score computed over all classes. The threshold À in the second column the logit margin threshold at 95% TPR for detection at e = 8/255. The robustness bias of the strongly margin consistent model can be detected using the logit margin, unlike the weakly margin consistent model.
Figure 11:
consistency remains strong for each class. However, for weakly margin- model (Fig. bottom row), significant disparities exist between the correlations across classes, making using logit margin proxy for input margin problematic.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have presented clearly the main claims in the abstract and the introduction; in particular, the end of the introduction details the list of the contributions and the main results.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have created a specific section "Limitations".
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [Yes]
The paper's theorem, formulas, and proof are numbered and cross-referenced. We have provided the proof in the supplemental material, and we have provided short proof sketch to provide intuition in the main paper.
Justification:
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All our experiments involve CIFAR 10 and CIFAR100 datasets, and the majority of models were loaded from the RobustBench model (Croce et al., 2021) and a few models are ResNet-18 (He et al., 2016) models that we trained on CIFARIO with Standard Adversarial Training (Madry al., 2018), TRADES (Zhang al., 2019), Logit Pairing (ALP and CLP, Kannan et al. (2018)), and MART (Wang et al., 2019). The code is available https://gittiiiie
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All the experiments based standard datasets CIFAR10, CIFAR100 and models mainly provided on RobustBench model zoo (Croce et al., 2021), https://
gi github. http://///oooottttoo//oooootttna
The code is available at: https: /ttpt:
com cctp://paaanggoaassaaaaaiiioooas
6. Experimental Settin//ettalss
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: For the first part of the experiments, the parameters for estimating the input margins are explained. In the second part, learning pseudo-margin, we specified the architecture, dataset size, and training and validation splits and gave the reference for the choices in the training protocol.
7. Experiment Statistical Significance
22

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The results accompanied by error bars, confidence intervals. or statistical significance tests, at least for the experiments that support the main claims of the paper.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: The article details the computing resources used for the experiments.
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics httt::stttt///// httprr/pp..p/piicc//hccccceiiiia
Answer: [Yes]
Justification: [NA]
10. Broader Impacts
Question: Does the paper discuss potential positive and negative societal impacts of work performed?
Answer: [NA]
Justification: This work aims at detecting vulnerable unseen samples for robust trained models. There is no obivous negative societal impact per se in this task, but as for any predictive model, inexact predictions may impact downstream decisions and thus, model quality must taken into account to control such situations.
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The original papers that produced code package and the datasets used for eval- uation are carefully referenced / com/ http///ooossseBeccc/rrrrsteeebe
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code produced to evaluate our approach is documented and the information about training and limitations discussed in the paper.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
23

---

The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Justification:
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
24


Reviews:
[{"review_id": "DaUuZGkBNq", "reviewer": "Reviewer_4V5x", "summary": "The paper addresses the problem of efficiently detecting vulnerable inputs to a robust deep classifier at test time without the need for running adversarial attacks or formal verification. They introduce the idea of margin consistency of a classifier to connect the input-space margin and feature-space margin (or logit margin). A classifier is margin consistent if there is a monotonic increasing relationship between the input margin and logit margin. They show that margin consistency is a necessary and sufficient condition in order for the logit margin to be used as a score to detect (separate) \"non-robust\" (vulnerable) samples from \"robust\" samples. \n\nUsing a number of robust models trained using various adversarial training methods, from the RobustBench, they empirically show that a vast majortiy of these models exhibit high margin consistency (measured via a Kendall Tau correlation). For these models with high margin consistency, they also evaluate the detection power of the logit margin in separating non-robust samples from robust samples.", "questions": "### Proof of theorem 1\n**Sufficiency**: It seems like the finite sample $S$ assumption is not needed. \\\nSuppose $f_\\theta(x)$ is margin consistent, and $A^S_\\epsilon$ is the set of non-robust samples from $S$ for a given $\\epsilon > 0$ (as defined in the paper). \nLet \n$\\epsilon_0 = \\sup\\\\{ d_{in}(\\mathbf{x}) : \\mathbf{x} \\in A^S_\\epsilon \\\\}$ and $\\lambda_0 = \\sup\\\\{ d_{out}(\\mathbf{x}) : \\mathbf{x} \\in A^S_\\epsilon \\\\}$. \nFor any non-robust sample $\\mathbf{x} \\in A^S_\\epsilon$, its logit margin satisfies $d_{out}(\\mathbf{x}) \\leq \\lambda_0$ due to the margin consistency. Therefore, $\\lambda_0$ perfectly separates the non-robust samples from the robust samples based on the logit margin. \n(This was a little unclear in the paper and does not need a finite $S$).\n\n**Necessity**: It is important to state that the proof is by contradiction. Suppose that for any robustness threshold $\\epsilon$, $d_{out}$ admits a threshold $\\lambda$ that perfectly separates the non-robust samples from the robust samples. Assume that $f_\\theta$ is *not* margin consistent. It seems to me that a three point example shows the contradiction better, since with two points, it is always possible to separate the points with the inequality direction reversed. \n\nSince the model is not  margin consistent, there exist points $S = \\\\{ \\mathbf{x}_1, \\mathbf{x}_2, \\mathbf{x}_3 \\\\}$ such that: i) in the input margin $d\\_{in}(\\mathbf{x}_1) < d\\_{in}(\\mathbf{x}_2) < d\\_{in}(\\mathbf{x}_3)$ and  ii) in the logit margin $d\\_{out}(\\mathbf{x}_1) < d\\_{out}(\\mathbf{x}_3) < d\\_{out}(\\mathbf{x}_2)$. Letting $\\epsilon = d\\_{in}(\\mathbf{x}_2)$ (or slightly larger), we get the set of non-robust samples to be $A^S\\_\\epsilon = \\\\{ \\mathbf{x}_1, \\mathbf{x}_2 \\\\}$. \n\nHowever, in this case, there exists no threshold $\\lambda$ for the logit margin that can cleanly separate the non-robust samples  $\\\\{ \\mathbf{x}_1, \\mathbf{x}_2 \\\\}$ from the robust samples $\\\\{ \\mathbf{x}_3 \\\\}$. Since this leads to a contradiction, the model must be margin consistent. \n \n\n### Questions and Comments\n\n1. In Eqn 2, $DB_j$ is not defined. I suppose it is $DB_j = \\\\{ \\mathbf{z} \\in R^m : (\\mathbf{w}_j - \\mathbf{w}_i)^T \\mathbf{z} + b_j - b_i \\geq 0 \\\\}$.\n\n1. Referring to lines 123 -- 124, the equidistance assumption seems to be glossed over, and it is thereafter assumed that the robust deep classifiers satisfy the property? As mentioned under \"Weaknesses\", why can we not use the feature space margins?\n\n1. Line 133: What does `output scores` refer to here? And clarify if it is the logit margin. \n\n1. Line 136: It may be worth mentioning here that negative values of Kendall Tau (upto $-1$) imply that the two rankings are anti-correlated or reversed. \n\n1. In section 2.3, the perfect discriminative function $g$ should be defined for a specific robustness threshold $\\epsilon \\geq 0$ and is a function of the classifier $f_\\theta$. So it would be clearer to show this in the notation, e.g. as $g_\\epsilon(\\mathbf{x} ; f_\\theta)$. Also the indicator function does not need the extra argument $\\mathbf{x}$. That is, it can just be $\\mathrm{1}\\_{[d_{in}(\\mathbf{x}) \\leq \\epsilon]}$\n\n1. In Figure 3, it should be $x_2 \\in A_{\\epsilon_0}$, not  $x_2 \\in A_{\\epsilon}$. Also, on line 152, it should be $\\lambda_0 = d_{out}(\\mathbf{x}_0)$, to correspond to the figure. \n\n1. The caption for Figure 4 is not so clear.\n\n1. Lines 191, 192: the statement `... margin consistency is a property orthogonal to the robust accuracy ...` seems like a strong statement to make. It is possible that there could be a non-zero correlation (mutual information) between the two. \n\n1. Figure 2 can be placed closer to the results section for readability.\n\n1. Line 215: the word \"boundary\" is missing after \"decision\". \n\n1. Based on line 220, it seems like a standard pointwise regression using MSE (and not learning to rank) is used to learn the mapping from the feature representation to a pseudo-margin. Could you also clarify why the Sigmoid activation is used at this network's output, for either regression or L2R?\n\n1. Does the property of margin consistency and the detection of vulnerable samples extend to other norms such as $\\ell_2$ norm?  \n\n1. Section 4, under OOD detection: the objective is usually to detect inputs that are from new classes not seen by the classifier during training, but it could also include detecting inputs that are from a different (shifted) marginal distribution, i.e., covariate-shifted OOD. It would be useful to cite a survey paper such as [Generalized out-of-distribution detection: A survey](https://arxiv.org/abs/2110.11334)\n\n1. For mis-classification detection, a couple of relevant references are missing [2] and [3]. \\\n[2] https://papers.neurips.cc/paper_files/paper/2021/file/2cb6b10338a7fc4117a80da24b582060-Paper.pdf \\\n[3] https://openaccess.thecvf.com/content/CVPR2023/papers/Zhu_OpenMix_Exploring_Outlier_Samples_for_Misclassification_Detection_CVPR_2023_paper.pdf", "limitations": "Yes, there is discussion on the limitations and scope of this work in section 5.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. The paper address an important and practical problem of detecting vulnerable inputs to a robust deep classifier in an efficient way using the logit margin as a proxy for the input margin (when the model satisfies approximate margin consistency). \n\n1. The development of ideas and presentation (with figures) is mostly clear and easy to follow. \n\n1. The idea of connecting the input margin and logit (or feature space) margin via the margin consistency, and using it for detecting vulnerable (non-robust) inputs looks novel. \n\n1. The experiments are extensive and explore interesting questions.", "weaknesses": "1. It's not fully clear why the paper does not use the feature-space margin directly instead of the logit margin. Taking the minimum of the distance to the decision boundaries (over classes $j \\neq i$, where $i$ is the predicted class) in Equation 2 should give the feature-space margin. Why then is the logit margin, which is an approximation of this, needed? Because the equidistance assumption (line 123) may not hold in practice. \n\n1. The proof of Theorem 1 is not precise and needs some clarifications (details in the \"Questions\" section).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! We will take the corrections into account. Please see our comments in the general response about defining the margin consistency in terms of the logit margin and results on the $\\ell_2$ norm. \n \n**a/ The proof of Theorem 1.**\n\nThank you for reading the proof carefully; we agree that there was some clumsiness in the sufficiency part. In this sense, we suggest modifying line 625 by:\n\n*\"Let $x_0$ be an element of the finite set $A^S_\\epsilon$ such that $d_{in}(x_0)=\\max \\{d_{in}(x): x\\in A^S_\\epsilon\\}$ and let $\\lambda = d_{out}(x_0)$.\"*\n\nWe agree that formulating perfect separation for finite samples only is not fundamentally necessary. However, it appears to us that this opens the way for a simpler proof which avoids dealing with the intricacy of the continuum. We propose to add a comment in this sense in the paper.\n\nAs for the necessity part, we need to indicate the proof is by contradiction, thank you for noticing this. We agree that our notion of perfect separability is oriented'' (i.e. non-robust points are required to be the ones with small $d_{out}$) and that we use this property in the proof. We understand that \"it is always possible to separate the points with the inequality direction reversed\", but this happens if and only if the model is \"reversed margin consistent\" ($d_{in}(x)<d_{in}(y)$ iff $d_{out}(x)>d_{out}(y)$), which cannot happen for the logit margin which is always non-negative and takes the value zero on that lie on input space decision boundaries (i.e. $d_{in}(x)=0$ implies $d_{out}=0$). While the three points counter example is attractive, it seems to us that the existence of such an example does not follow from the negation of margin consistency. We, therefore, suggest keeping this proof as it is while adding some comments that address the points you rightfully raised.\n\n**b/** $DB_j$  In Equation 2 should be the boundary $DB_{ij}$ defined similarly in the paragraph above \n\n$$\\text{DB}_{ij} = \\\\{z'  \\in  \\mathbb{R}^m: (w_i-w_j)^\\top z'+(b_i-b_j)=0\\\\}$$\n\nIt is a good catch; we will correct it.\n\n**c/** Yes, in Line 133, we indeed refer to the logit margin as a score computed from the neural network's output (logits). We will clarify that.\n\n**d/ Lines 191, 192: the statement ... margin consistency is a property orthogonal to the robust accuracy ... seems like a strong statement to make.**\n\nWe recognize that it is not the correct way to convey our message here, which is that being robust does not imply margin consistency (cf. discussion on Lipschitz smoothness vs. margin consistency, the first point of the answer to Reviewer WVo4). There may be indeed more to explore about the connection between the two properties. We propose to remove that statement and add the discussion about Lipschitz's smoothness and margin consistency.\n\n**e/ Pseudo-margin learning and the Sigmoid activation**\n\nThe purpose of our study on pseudo-margin learning is to provide evidence that it is possible to learn to map the features  $h_\\psi(x)$ to a proxy $d_{out}$ that simulates the margin consistency. Our approach is inspired by the setup used for confidence learning [1] (ConfidNet): a target confidence $y \\in [0,1]$, Sigmoid as output activation with MSE loss. In our case, we use $y=\\frac{d_{in}(x)}{\\max \\\\{ d_{in}(z): z\\in S\\\\}} \\in [0,1]$ as a target, where S is the training set. In L2R, the earliest and simplest methods are point-wise methods, which are no different from standard regression with the MSE loss [2]. This worked worked well for our purpose.\n\n[1] Corbière, Charles, et al. \"Addressing failure prediction by learning model confidence.\" Advances in Neural Information Processing Systems 32 (2019).\n\n[2] He, Chuan, et al. \"A survey on learning to rank.\" International Conference on Machine Learning and Cybernetics. Vol. 3. Ieee, (2008).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "m57mnJ9ir9", "reviewer": "Reviewer_jLJt", "summary": "It is currently difficult to determine how susceptible a given input to a model is to adversarial perturbations. The distance from the input to the model's decision boundary in the input space (input space margin) is a reasonable metric, but it is intractible to compute for many deep neural networks and not always meaningful. This paper investigates the use of logit margin as a proxy for input margin. The logit margin is defined to be the difference between the two largest logits. Furthermore, a model is said to be margin consistent if there is a monotonic relationship between input margin and logit margins. Theorem 1 justifies that, if a model is margin consistent, then logit margin can be used to detect non-robust samples. Experiments show that common deep learning models can display a high level of margin consistency.", "questions": "- How would using a different distance measure change your results (i.e. measuring distance with $l_2$ norm rather than $l_\\infty$ norm)?\n- How do you think the experimental results would change if you were using a higher dimensional/more difficult to classify dataset? Would you expect the correlations observed in these results to hold?\n- Do you think this method and your results have any implications for non-robust classifiers?", "limitations": "Limitations are discussed in section 5.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This work represents an interesting new perspective on detecting adversarial attacks. Rather than detecting adversarial examples themselves, the method presented here focuses on detecting when example is at risk of being attacked. This approach may lead be useful in the future in the design of new defense systems against adversarial attacks.\n\nThe experimental section of this work is strong. In particular, the correlation between input margin and logic margin of robust models is striking in figure 2 and serves as support for the plausibility of this metric.\n\nThe authors present a compelling hypothesis for when margin consistency might hold in machine learning models, relating margin consistency to isometry. This may lead to an interesting line of future work. A thoughtful commentary on directions for future work is also provided in Section 5.", "weaknesses": "While small logit margin may be associated with susceptibility to attacks, there may be other reasons we would not want to flag samples in this manner. For example, if a model exhibits differential performance on different subpopulations, certain subpopulations as a whole might be labeled as \"brittle decisions.\" I think that an exploration and discussion of how pervasive brittle decisions are and how they are distributed in practice is warranted and would strengthen this paper.\n\nSimilarly, flagging brittle decisions may lead to a false sense of security. Adversarial examples have been shown to induce very high logit margins [1]. It is not clear to me that this method would be able to detect that an adversarially perturbed sample is close to a decision boundary. An enlightening experiment would be similar to figure 5a, except the samples have been adversarially perturbed to maximize logit margin.\n\n[1] Goodfellow, Ian J., Jonathon Shlens, and Christian Szegedy. \"Explaining and harnessing adversarial examples.\" arXiv preprint arXiv:1412.6572 (2014).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! Please see our comments in the general response for answers to your questions (Q1 and Q2 are addressed in the third point of the general response, and Q3 is addressed in the last point). Below are our responses to your other concerns.\n\n**a/ Bias across subpopulations**:\n\nLocal robustness bias exists indeed in models; thank you for bringing up this interesting comment. Interestingly, when margin consistency (approximately) holds, we observe that robustness discrepancies across classes in input margin can also be observed using the logit margin (see top row, Figure 1 of the PDF attached to the general response section). We also observe that margin consistency (approximately) holds for each class individually, so no bias seems present as far as margin consistency goes. For the weakly margin-consistent models (example shown in the bottom row of Figure 1, attached PDF), there are significant disparities between the correlations across classes, which is also why using the logit margin for such models would be problematic.\n\n**b/ False sense of security and detection of adversarial examples:**\n\nMargin consistency, like standard accuracy, measures a property of the model over test inputs sampled iid from the training data distribution. From the objective they optimize, adversarial examples are close to the decision boundary and, therefore, have very small input margins, and we also observe that they have very small logit margins when compared to clean examples. However, we agree that it could, in principle, be possible for specially crafted adversarial examples to bypass margin consistency. This could be interesting for future work. The detection of adversarial examples itself is a different task, considered a defence strategy, and [1] shows that it is an equivalent task to robust classification. We believe studying margin consistency on adversarial examples or designing adaptive attacks is out of the scope of this work.\n\n[1] Tramer, Florian. \"Detecting adversarial examples is (nearly) as hard as classifying them.\" International Conference on Machine Learning. PMLR, 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SWvlQxp8o3", "reviewer": "Reviewer_utdZ", "summary": "The authors propose using the distance between the two max values of a model's output as a proxy for the input margin to efficiently identify samples vulnerable to adversarial attacks. This proposed margin consistency is formally defined and shown to work across many robust models on the CIFAR10 and CIFAR100 datasets. Additionally, a basic learned fix is proposed for the few models which did not have margin consistency.", "questions": "(Q1) What is the computational cost to build the dataset required for training in section 3.3? And in general, what is the computational cost required for the proposed method? Not as much of an issue due to adversarially robust models already requiring more compute to train and the proposed method taking minimal compute at inference.\n\n(Q2) How successful would an adversarial attack maximizing the difference of the 2 max logits be in bypassing the proposed detection? Experiments for this may be slightly outside the scope of the paper.", "limitations": "yes", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "(S1) Proposed margin consistency for vulnerable sample detection in the context of robust models appears novel. Using logit statistics to detect adversarial examples is not new [1,3,4], and attacks have incorporated the difference between the two max logits [2]. However, this specific formulation (1), being on the defending side (2), and evaluating robust models (3) appear to be a novel combination.\n\n(S2) Experiments are included for a wide variety of models, and a variety of metrics are provided for balanced evaluation. Additionally, the basic modification proposed for weakly correlated models significantly improves results.\n\n(S3) Writing is clear, and mathematical formulation appears sound.\n\n[1] Wang, Yaopeng, et al. \"Model-agnostic adversarial example detection through logit distribution learning.\" 2021 IEEE International Conference on Image Processing (ICIP). IEEE, 2021.\n\n[2] Weng, Juanjuan, et al. \"Logit margin matters: Improving transferable targeted adversarial attack by logit calibration.\" IEEE Transactions on Information Forensics and Security 18 (2023): 3561-3574.\n\n[3] Aigrain, Jonathan, and Marcin Detyniecki. \"Detecting adversarial examples and other misclassifications in neural networks by introspection.\" arXiv preprint arXiv:1905.09186 (2019).\n\n[4] Ozbulak, Utku, Arnout Van Messem, and Wesley De Neve. \"Not all adversarial examples require a complex defense: Identifying over-optimized adversarial examples with IQR-based logit thresholding.\" 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 2019.", "weaknesses": "(W1) Additional experiments. Including experiments for one or two non adversarially robust models would help show the limitations (or additional potential) of margin consistency. Additionally, evaluation of a robust model trained on ImageNet would help show the scalability of the method. Pretrained robust ImageNet models are available [5], so if initial compute is not a factor computation should not be a hinderance.\n\n(W2) No additional analysis for why margin consistency fails for the 2 CIFAR10 models.\n\n[5] https://github.com/RobustBench/robustbench", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! Please see our comments in the general response about standardly trained models with zero adversarial accuracy and results on ImageNet. Below are our responses to your other concerns.\n\n**a/ (W2) No additional analysis for why margin consistency fails for the 2 CIFAR10 models.**\n\nWe agree that further analysis of these two models may be helpful. However, we believe it may be closely related to a broader question on the relationship between robustness and margin consistency, which can be an avenue for future exploration. We have added an analysis in Figure 1 (PDF attached to the general response section) that gives further insights. It shows a disparity of correlations across different classes, while it is almost uniform in margin-consistent models (top row).\n\n**b/ (Q1) Computational cost.**\n\nEstimating input margins requires running the FAB attack with a sufficient budget [1]. The average time to estimate the input margins over 1000 samples is the following (on 16GB GPU-Titan XP):\n\n- CIFAR10: ResNet-18 (6min), ResNet-50 (19min), WideResNet-28-10 (35min)\n- CIFAR100: ResNet-18 (58min), ResNet-50 (7h), WideResNet-28-10 (6h)\n\nFor ImageNet models, a ResNet-50 takes about 1 day 20hrs to 2 days on V100-32GB.\n\n**c/ (Q2) How successful would an adversarial attack maximizing the difference of the two max logits be in bypassing the proposed detection?**\n\nWe believe it is possible to attack a sample to bypass that detection, but we think this is beyond the scope of this work. Nevertheless, in our case, we checked (line 178) that our non-robust samples are almost similar to the ones found by the standard AutoAttack evaluation which includes APGD-DLR that maximizes the DLR loss (Difference in Logits Ratio), which is a rescaled difference between the logit of the true class and the biggest logit among the others. This would seem to indicate that the logit margin can also detect vulnerable samples to this sort of attack.\n\n[1] Xu, Yuancheng, et al. \"Exploring and exploiting decision boundary dynamics for adversarial robustness.\" arXiv preprint arXiv:2302.03015 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sxyKClbAYO", "reviewer": "Reviewer_WVo4", "summary": "This paper discovers that many existing neural network classifiers share a strong correlation between input margin (distance to decision boundary) and output margin (difference between top-2 logits). The paper further proposes to use this property for local robustness estimation.", "questions": "- Equation (3) relies on an *equidistant* assumption. Is this assumption satisfied by real-world models? I found some short discussions in Appendix C, but would prefer a more formal analysis. Furthermore, Line 640 says \"the values vary only in a small range.\" What do the \"values\" refer to? It also seems to me that Figures 8 and 9 demonstrate non-trivial variance?\n\n- In addition to observing the correlation between input and output margins, the paper also mentions the distances in the feature space. How is the property regarding feature-space distance used in practice? For context, comparing Figure 5a and 5b, it seems that the input-output margin correlation is stronger than the input-feature margin correlation.", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Efficient local robustness estimation is an important problem, and the proposed method offers a viable direction for this purpose. The strong correlation between input and output margins is an interesting discovery. Furthermore, the paper clearly formalizes and quantifies this correlation via Kendall rank correlation. The paper is well-written, typo-free, and clearly conveys the results.", "weaknesses": "The weaknesses are mainly twofold.\n\nFirst, the observation of the correlation, while interesting, is not strong enough in my opinion. Given that robust neural networks are known to be Lipschitz [1], the correlation between input and output margins is not very surprising. The main theoretical result (Theorem 1) is relatively straightforward, self-explanatory, and unsurprising.\n\nSecond, the authors did not consider the possibility of adaptive attacks. While the correlation between input and output margins is strong when the input itself is benign, it is unclear whether this correlation still holds when the input itself is adversarial. Is it possible to use attack methods to find inputs that break this correlation (i.e., output margin is large but input margin is small), leading to overestimation of the local robustness? Furthermore, the pseudo-margin prediction model adds even more potential vulnerabilities.\n\n[1] Yang et al. A closer look at accuracy vs. robustness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review! Please find below our responses to your concerns.\n\n**a/ Lipschtz Smoothness vs. Margin Consistency**\n\nThank you for raising this point. Lipschitz smoothness is an essential property of robust models since a small Lipschitz constant $L$ guarantees the network's output cannot vary more than a factor $L$ of the input variation. Empirical adversarial training strategies aim in a certain way to achieve Lipschitz's smoothness indirectly with various levels of Lipschitz constants, and it is also possible to directly constraint the Lipschitz constant to achieve $1$-Lipschitz networks ([1] and references therein). However, *Lipschitz smoothness does not imply margin consistency*. Indeed:\n\n*Let $f$ be an $L$-Lipschitz neural network i.e. $||f(x_1)-f(x_2)|| \\leq L||x_1-x_2||$, $\\forall x_1, x_2$. Let us consider two points $x_1$ and $x_2$ with $0<d_{in}(x_1)<d_{in}(x_2)$. Note that the $L$-Lipschitz condition implies that $d_{out}(x_i)\\leq L d_{in}(x_i)$ for $i=1,2$. However, as long as $d_{out}(x_1)>0$, it is clearly possible a priori to have $d_{out}(x_2)<d_{out}(x_1)$, thus violating the margin consistency condition, while still satisfying the previous relations.*\n\nThis theoretical possibility is supported by the empirical evidence that we found two robust (hence probably Lipschitz)  models which are not margin-consistent. The question of how both properties influence each other is out of the scope of this work and may be grounds for future work. We propose to add this clarification to the paper.\n\nThe theorem itself establishes that margin consistency (preservation in the feature space, of the order of the samples' input space distances to the decision boundary) is a necessary and sufficient condition for using the logit margin as a proxy score for the input margin.\n  \n\n**b/ Possibility of adaptive attacks**\n\nImplicitly, margin consistency, like standard accuracy, measures a property of the model over test samples iid with the training data. We leverage it to detect vulnerable iid test samples susceptible to being adversarially attacked. We concede that it may indeed be possible to craft examples that are non-robust ($d_{in} \\leq \\epsilon)$) but for which $d_{out}>\\lambda$, and therefore would bypass the proposed detector. We believe studying the margin consistency on adversarial examples or designing adaptive attacks to be out of the scope of this work.\n\n**c/ (Q1) Equidistance Assumption**\n\nIn Appendix C, by \"values,\" we refer to the distances between the classifiers $||w_i-w_j||_q$ and by a small range, we mean that the interquartile range is small compared to the mean value of the distance. The empirical evidence shows that using the logit margin instead of the exact feature margin (minimum over margins to other classes) marginally affects the results and avoids the computational overhead of searching the minimum (See the general response section and Table 1 of the attached PDF file).\n\n**d/  (Q2) The property regarding the feature space distance**\n\nIn Figure 5, by the feature space distance, we mean the distance $||h(x)-h(x’)||$ in the feature space between the representations of a sample $x$ and the representation of its adversarial example $x’$. Although the correlation is a bit stronger than with the logit margin, it is as hard to compute as the input margin ($||x-x’||$) since they both require finding the closest adversarial example in the input space.\n\n[1] Araujo, Alexandre, et al. \"A unified algebraic perspective on Lipschitz neural networks.\" arXiv preprint arXiv:2303.03169 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the reviewers for their appreciation and thoughtful comments! The points raised by the reviewers are certainly very useful for clarifying and improving the presentation of our work while also bringing interesting avenues for future exploration. Below are some general comments about the margin consistency that are relevant to all reviewers. We individually respond to other reviewers' concerns right after. 

1. Margin consistency is an order preservation property not implied by robustness. More precisely, Lipschitz's smoothness does not imply margin consistency—see details on that in the answer to reviewer WVo4. However, knowing how local robustness influences margin consistency and vice versa is an avenue for future exploration.

2. While equidistance may not be achieved perfectly in practice, we do not need perfect equidistance for the logit margin to be a practical approximation of the exact feature margin. Computing the exact feature margin requires computing the minimum over (K-1) pairs of scaled logit differences, where $K$ is the number of classes. The approximation circumvents the computational overhead of the minimum search, which can take a second instead of just microseconds for inference. This difference can add up to hours at scale, offering scalability when dealing with a large number of classes.  Table 1 in the attached PDF shows side-by-side the results when using the logit margin (Lm columns) and the exact feature margin (Fm columns). There is little to no difference between the two results.

3. Results for some ImageNet models in $\ell_\infty$ and $\ell_2$-robust models in Robustbench (Table 2, attached PDF file) show that these models are also strongly margin consistent. However, we do not exclude that exceptions may exist.

Note that measuring the $\ell_\infty$ norm for an $\ell_\infty$ robust model makes more sense than measuring another norm like $\ell_2$ because an $\ell_\infty$-robust model is not necessarily $\ell_2$-robust and vice-versa. However, the measured input margins in the $\ell_2$ norm and $\ell_{\infty}$ norm for the models we investigated are highly correlated, so the results do not change if we measure $\ell_2$ instead. We provide results in the cases where $\ell_2$ norm matters (i.e., for $\ell_2$-robust models, available only for CIFAR10 in Robustbench).

4. While we believe that margin consistency is far less attractive for standard models (for which almost all decisions are non-robust), we nonetheless provide Kendall $\tau$ correlations for a variety of standard models in the table below. Models can be margin-consistent without being robust and vice-versa. It is unclear at this point how we could leverage the margin consistency for non-robust models.

| Model ID              | Kendal tau | Accuracy | Architecture     | Dataset  |
|-----------------------|------------|----------|------------------|----------|
| STD\_M1               | 0.45       | 93.07    | ResNet18         | CIFAR10  |
| STD\_M2               | 0.69       | 92.60    | ResNet20         | CIFAR10  |
| STD\_M3               | 0.36       | 93.65    | ResNet-50        | CIFAR10  |
| STD\_M5 (Robustbench) | 0.50       | 94.78    | WideResNet-28-10 | CIFAR10  |
| STD\_M6               | 0.75       | 72.63    | ResNet-56        | CIFAR100 |
| STD\_M7 (Robustbench) | 0.70       | 76.52    | ResNet-50        | ImageNet |

Meta-review: The reviewers agree that the paper tackles an important problem with sound results. The authors introduce the idea of margin consistency of a classifier to connect the input-space margin and feature-space margin (or logit margin). A classifier is margin consistent if there is a monotonic increasing relationship between the input margin and logit margin. They show that margin consistency is a necessary and sufficient condition in order for the logit margin to be used as a score to detect (vulnerable) samples from "robust" samples.
Decision: Accept (poster)