Exploring Geometry of Blind Spots in Vision models
Authors: Sriram Balasubramanian
Gaurang Sriramanan
Vinu Sankar Sadasivan
Soheil Feizi
Venue: NeurIPS 2023 spotlight (2023)
Keywords: Neural networks
Vision models
blind spots
undersensitivity
invariance
level set geometry
input connectivity
TL;DR: We discover blind spots of common vision models by exploring the geometry of their level sets

Abstract: Despite the remarkable success of deep neural networks in a myriad of settings, several works have demonstrated their overwhelming sensitivity to near-imperceptible perturbations, known as adversarial attacks. On the other hand, prior works have also observed that deep networks can be under-sensitive, wherein large-magnitude perturbations in input space do not induce appreciable changes to network activations. In this work, we study in detail the phenomenon of under-sensitivity in vision models such as CNNs and Transformers, and present techniques to study the geometry and extent of “equi-confidence” level sets of such networks. We propose a Level Set Traversal algorithm that iteratively explores regions of high confidence with respect to the input space using orthogonal components of the local gradients. Given a source image, we use this algorithm to identify inputs that lie in the same equi-confidence level set as the source image despite being perceptually similar to arbitrary images from other classes. We further observe that the source image is linearly connected by a high-confidence path to these inputs, uncovering a star-like structure for level sets of deep networks. Furthermore, we attempt to identify and estimate the extent of these connected higher-dimensional regions over which the model maintains a high degree of confidence.

Exploring Geometry of Blind Spots in Vision Models
Sriram Balasubramanian" sriramb6cs. umd. edu
Gaurang Sriramanan" gaurangs@cs umd. edu
Vinu Sankar Sadasivan vinu@cs. umd. edu
Soheil Feizi sfeiziecs. umd. edu
Department of Computer Science University of Maryland, College Park
Abstract
Despite the remarkable success of deep neural networks in myriad of settings, sev- eral works have demonstrated their overwhelming sensitivity to near-imperceptible perturbations, known adversarial attacks. On the other hand, prior works have also observed that deep networks can be under-sensitive, wherein lagge-magnitdd perturbations in input space do not induce appreciable changes network activations. In this work, we study in detail the phenomenon of under-sensitivity in vision models such as CNNs and Transformers, and present techniques to study the geometry and extent of "equi- "oquii-cofidence" level sets of such networks. We propose a Level Set Traversal algorithm that iteratively explores regions of high confidence with respect to the input space using orthogonal components of the local gradients. Given source image, we use this algorithm identify inputs that lie in the same equi-confidence level set the source image despite being perceptually similar to arbitrary images from other classes. We further observe that the source image is linearly connected by high-confidence path to these inputs, uncovering a star-like structure for level sets of deep networks. Furthermore, we attempt to identify and estimate the extent of these connected higher-dimensional regions over which the model maintains a high degree of confidence. The code for this project is publicly available this URL
Introduction
Deep neural networks have demonstrated remarkable success in various diverse domains including computer vision and natural language processing, surpassing the performance of classical algorithms by significant margin. However, though they achieve state the art results on many computer vision tasks like image classification, sometimes even exceeding human-level performance (He et al.) 2015 2016), the overall visual processing conducted by such models can deviate significantly from that effectively observed in the human visual system. Perhaps differences lies in the overwhelming susceptibility of neural networks to near- seaa-mmpreepppbbee changes their inputs commonly called adversarial attacks (Szegedy et al. [0014] an extraordinary failure mode that highlights the over-sensitivity of such models. Indeed, an active area of research over the past few years has been focused towards analyzing adversarial perturbations under different threat models [Goodfellow et al.) 2015] Tramèr et al. 2018 Wong et al.) 2019 Laidlaw et al. 2021 Croce and Hein 2022 and in addressing adversarial vulnerabilities using robust training methodologies [Madry et al. 2018 /Zhang et al., 2019b, (iinlla and Feizi 2020 Wu et al. 2020
iconic and representative of these
Laidlaw et al. 2021 Levine and Feizi 2021].
"Equal Contribution, author names ordered alphabetically
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

On the other hand. it has been shown that such models may also be under-sensitive, wherein input images that are unmistakably disparate to human oracle induce near identical network activations
Jaaaaeeennnnnooennnnnneeee or predictions [Jacobsen et al. [0T8a]. To tractably analyze this phenomenon, Jacobsen et al. [201 8a utilize a special class of neural networks that are bijective functions, called fully Invertible RevNets (Jacobsen et al. 2018b, KKnggaa and Dhariwal 2018), to craft large-magnitude semantic perturbations in input space that are designed to leave its corresponding network representations unchanged. Furthermore, Tramèr et al. (2020) show that robust training with l, bounded adversaries can be source of excessive model mnver--nvreennce in itself, due to the poor approximation of the true imperceptible threat model of human oracles by ly norm- comn-oundedd balls in RGB- RGB-pixel space (Laidlaw et al., 2021] Indeed, the authors 'break' a provably robust defense on MNIST (Zhang et al. 2019a with a certified accuracy of 87%, by crafting perturbations within the certified lo radius of 0.4, that however cause model agreement with human oracles to diminish to 60%.
However, these methods either rely upon special invertible network architectures or the selection of the nearest training image of another class as a target followed by a sequence of complex alignment and spectral clustering techniques, so as to semantically alter the input in a conspicuous manner that induces a change in the human assigned oracle label, while leaving the model prediction unchanged. This leads us to our research question: Is it possible to analyze the phenomenon of under-sensittiiity of general vision models in a systematic manner on natural image datasets, and characterize the geometry and extent of "blind spots" of such networks? Indeed, empirically demonstrate the veracity of this claim - in this work, we present a novel Level Set Traversal algorithm to explore the "cqui-confidence" level sets of popular vision models. Given an arbitrary source and target image pair, our proposed algorithm successfully finds inputs that lie in the same level set as the source image, despite being near-identical perceptually the target image. Furthermore, the proposed algorithm identifies a connected path between the original source image and the "blind spot" input so generated, wherein high prediction confidence with respect the source class maintained throughout the path. In summary, we make the following contributions in this work:
We present a novel Level Set Traversal algorithm that iteratively uses orthogonal components of the local gradient to identify the "blind spots" of common vision models such as CNNs and ViTs on CIFAR- 10 and ImageNet.
We thereby show that there exist piecewise- connected paths in input space between images that human oracle would deem to extremely disparate, though vision models retain a ncar-uniform level of confidence on the same path.
Furthermore, we show that the linear interpolant path between these images also remarkably lies within the same level set; as we observe the consistent presence of this phenomenon across arbitrary souree-target image pairs, this unveils a star-like set substructure within these equi-confidence level sets.
We demonstrate that adversarially robust models tend to be under-sensitive over subsets of the input domain that lie well beyond its original threat model, and display level-sets of high-confidence that extend over a significant fraction of the triangular convex hull of a given source image and arbitrary pair of target images.
2 Preliminaries
Notation: In this paper, we primarily consider the setting of classification with access to a labelled dataset. Let x E X' denote d-dimensional input images, and let their corresponding labels be X' - [o, 1] denote the classification model considered, where f(x) (z)..... (x)))) (z)) represents the softmax predictions over the N-classes. Further, let (x)x be the argmax over the N N-dimensional softmax output, representing the class predicted by the model for input x. For a given data sample (æ: y). let the cross-entropy loss achieved by the model be denoted CF y). Given a prediction confidence value p [0, 1]. and class j (1...., N)
denoted as y E [1,....,]] N). Let
we define the Level Set Ly (p. and Superievel Set L (p.j) for the function f as follows:
(y(p.)) = {x E X f(x)= = pp} , LÝ (p: = (a E X: f((x)) p]
Given a pair of inputs æ; and 22, we define the linear interpolant path between them as a(); 22) = X æ + (1 A) æ2, for X E [0. A given set S is thus said to be convex if ((Aaar, E S, Van ma E and X E [0, 1]. Further, a set S is said to be star-like if there exists some So E S such that p(A:so,)) x) S, Va S and X E [0, 1].

---

2.1 Conjugate Nature of Adversarial and Confidence Preserving Perturbations
Given a classification model f and correctly classified benign input (æz y), an adversarial image is a specially crafted image v = x + E such that both a and T appear near -dantical to human oracle, but induces the network to misclassify. i.e. C + E) + Cz = y. To enforce imperceptibility in computationally tractable manner, several adversarial threat models have been proposed, with the l2 and &a norm constraint models being the most popular. Amongst the earliest adversarial attacks specific to the latter threat model was the Fast Gradient Sign Method (FGSM) attack, proposed by Goodfellow et al. [2015]. wherein the adversarial perturbation is found by single-step direct ascent along the local gradient with pixel-wise clipping. A stronger multi-step variant of this attack called (terated-FGSM (IFGSM) was later introduced by [Kurakin et al. 2017). wherein iterated gradient ascent is performed alternately with projection operations onto the constraint set. A popular variant, called the Projected Gradient Descent (PGD) attack was introduced by Madry et al. (2018) which incorporates a initial random perturbation to the clean image, which was observed to help mitigate gradient masking effects Kurakin et al.] [2016]. A large class of adversarial attacks [Croce and Hein 2020 Gowal et al., 2019 Carlini al. 2019] Sriramanan al., 2020) thus utilize perturbations parallel to the gradient, with appropriate projection operations to ensure constraint satisfaction.
In contrast, perturbations that leave the network prediction confidence unchanged, are locally orthog- onal to the gradient direction. Indeed, for any differentiable function g : Rd - I2 denote the level set as (c) for a given output E I.. Let 7(t) [0. (c) be any differentiable curve within the level set. Then, (()(t)) = c Vt E [0, 1]. Thus, & (e()(t))) 0 (Vg()(t),, (()). implying that (t) is orthogonal to Vg(r(t)), vt E [o. 1]. Since this is true for any curve contained in the level set, we conclude that the gradient vector is always perpendicular to the level set. Furthermore, we can additionally show that the level set L (c) is often a differentiable submanifold, with mild additional conditions on the gradient. Indeed, a level set Lg(c) is said to be regular Vg(æ)) + Vaz E Lg(c).
Lemma 1. If g: R2 + IR is continuously differentiable function, then each of its regular level sets is a (d - 1) dimensional submanifold of R
We present the proof of Lemma T in Section A of the Appendix. We thus observe that adversarial perturbations, largely parallel to the gradient, are locally orthogonal to confidence preserving direc- tions which correspond to the (d - 1) dimensional tangent space of the level set. We take inspiration from this observation to develop a general framework that applies to a broad class of differentiable neural networks, opposed to previous works (Jacobsen et al. 2018a| that require the network to be invertible to identify confidence preserving perturbations.
We also remark that adversarial attacks and level set preserving perturbations are complementary from another perspective as well, as noted by prior works: the former attempts to find inputs that change model predictions without modifying human oracle assignments, while the latter attempts to keep network predictions unchanged though human oracles would likely change their original label assignment. Thus, both classes of adversarial attacks and level set preserving perturbations induce misalignment between oracle and model predictions, and cast light onto independent means of evaluating the coherence of models towards human expectations.
3 Proposed Method: Level Set Traversal
We now describe our algorithm for traversing the level set, which we call the Level Set Traversal (LST) algorithm (Algorithm D. We try to find path from a source image to a target image such that all points on that path are classified by the model as the source class with high confidence. Given that these (d-1) 1) dimensional level set submanifolds can be potentially highly complex in their geometries, we use discretized approximation using small yet finite step sizes to tractably explore these regions. Let the source image be æ, with true label y.. and entropy loss of the model (f) prediction with respect to æ. The key idea is to get as close as possible to a target image I; from some image x by computing the projection of Ax = æ, onto the orthogonal complement of g. to obtain new image Fnew that leaves the model confidence unchanged. We can compute the projection of Ax on the orthogonal complement by subtracting the component
V (f ys) ) be the gradient of the cross
of Ar parallel to g, that is, 21ig = 9
(L.6). Then, using a scale factor 7). the vector update to x can be expressed as Ar = r)( (Ar - x ) (I.7), and therefore Feew = x + Ar. Starting from the source image x., we repeatedly perform this iteration to get reasonably close to the target image
Arr" g |
3

---

Algorithm 1 Level Set Traversal (LST)
1: Input: Source image o, with label y, target image æt,, model f, max iterations m, scale factor 1), stepsize c, confidence threshold 6
Ar 0/107
Fiew
2: Initialize 2 æs. 3||
3: for i to m do
4: Ar 3z
5:
= VaCE(f(2),))
6:
C// (g ///)//// /19||
Figure 1: Pictorial representation of the LST al- gorithm. The contour lines represent level sets of the model confidence f(x),, with blue representing high confidence (low loss) and red representing low confidence (high loss). each iteration, we obtain Tnew by adding two vectors, & 77( A c//9) (projection of Aæ onto the orthogonal complement of g) and Ii| (a small perturbation to increase the confidence and remain within the level set).
7:
A =n(Ar -1/9)
8:
I (x Eg. -,
9:
T new = x + A + 2||
10:
if f( ]5] - - ([new) ) then
11:
return
12: x new
13:
return
Step
Step: 10
Step:
Step: 40
Step 60
Step:
Shep: 120
Step: 160
Step 200
Steg: 300
Step: 400
Sance
Costdince Confoentet Sor5dine 0.202
Covidanoe
Corl
CARTORNCE
Figure 2: Intermediate images over the path traversed by the LST algorithm using source image of a "goose' and target image a 'Scottish terrier' (a dog) for normally trained ResNet-50 model. We observe that the model predicts the 'goose' class with very high confidence for all images over the path, though the target blindspot found by LST clearly appears as a 'dog' human oracles.
while carefully ensuring that the confidence of the model prediction at any given point does not drop below preset confidence threshold ō compared to the source image (L11)..
While the above method works well with a relatively large yj if the curvature of f is low, it risks a non-trivial drop in model confidence (or even a change in the model prediction) if the curvature of f at æ is high enough. Therefore, after each step, we add a small perturbation vector æjj in the direction of -g scaled by factor of e. This step decreases the cross entropy loss, and thus increases the confidence so as to offset any confidence drops incurred due to the addition of Ar Thus, we can maintain higher model confidence over the path. In order to ensure that the norm of this perturbation is not arbitrarily large, we bound the fo norm by projecting the vector an foo ball of radius e. Then, the step 2|| can be computed as clamp(eg, -c, <), where the function clamp( U, a, b) is the element-wise application of the function min mmax(-, a). on all elements of v. Thus, we modify L9 so that now Enew x + Aæ 1 T||. We present pictorial schematic of Algorithm ] in Fig I We further employ an exponential moving average of 2 in order to smoothen components that potentially undulate heavily during the iterative traversal. Thus, since the frequent changes in L are smoothened out, we find that the final output images are often linearly connected to the source image with high confidence over the linear interpolations (see Fig 6
We now apply this algorithm to explore the level sets of standard and robust ResNet-50 models. In Fig 2] present the path traversed by the LST algorithm, wherein the model predicts the source 'goose" class with very high confidence over the entire path, though the target blindspot found by LST clearly appears as 'dog' human oracles. To substantiate the efficacy of the LST algorithm, we randomly select five images from five arbitrary ImageNet classes ('goose', 'Scottish Terrier', 'meerkat', "academic gown', 'cleaver'), and compute LST blindspots for all possible source-target image pairs. We show the final images output by the LST algorithm in Fig as an image-grid. If we order the five selected images, the tth row and jth column of the image-grid the LSTT output obtained using the tt image as target and jth image as the source. Thus, each column represents the source image being transformed iteratively into the other target images. The confidence of the model prediction for source class (names top of each column) is displayed just below each image. For both the normally trained and adversarially trained model, these images are almost indistinguishable from the target while retaining high model confidence for the source class. Since ad adversarially robust models have

---

Cees Label Predicted by ResVeto0
Gass Label Predicted by Robust ResNer500
SADE
Scatran Norter
mnoohut
surdams goen
Ubseer
SINDA
Scaticah harker
manhhat
ssademis genn
320/3
00863
Class Label Percesed hymmman
Class Label Percessed Humanss
00000
04912
cams
CA99
3:4 LOWG
0444
10
3994
Figure 3: The images returned by LST for 5 random source and target images for normally trained (left) and adversarially trained (right) ResNet-50 models. The image in the 2thh row and jth column is the LST output obtained using the jt image as target and jth image as the source. The confidence of the model prediction for the source class (names on top of each column) is displayed just below each image. For example, all four LST output blindspots in the first column (highlighted), using the source "goose' image, are all predicted be of the 'goose' class with very high confidence. Diagonal images are unchanged, as source equals target. We observe that almost any source image can be iteratively modified using LST to resemble any target image very closely without any loss in confidence for both normal and adversarially trained ResNet-50 models.
perceptually aligned gradients, can sometimes visually notice a few traces of the source image in the final LST image; for example the 'meerkat' image the 3rd row, 2nd column in the right side of Fig has some traces of the source 'terrier' image, but differences are usually hard to perceive.
We also examine the model confidence over linear interpolations between the source image and LST outputs for all target pairs in Fig 6 Formally, consider source image I, with label y and let oop represent the LST output when applied toward a target image X,, Denote the difference vector as Au op .g. Then, we observe that x, aAv v is assigned high confidence with respect to class y by the model Va E [0, 1]. which represents the entire linear interpolant path between I, and Cop- Furthermore, observe that the path discovered by the Level Set Traversal algorithm enjoys two key properties: (1) Uniqueness (once the target image is fixed) and (2) Extremality:
(1) Uniqueness: Since the local tangent space of the level set is (d 1) dimensional, several independent directions are orthogonal to the local gradient, and apriori do not yield a unique path like gradient-flow. However, once fix target image, we use its difference vector with respect to the current iterate (L4) and compute its projection onto the local tangent space (L.7 of the level set, thereby generating a uniquely defined path.
(2) Extremality: Though this bowed path may be non-linear, we additionally discover that the final output-point of this flow is surprisingly linearly connected with high-confidence to the source image after we apply discretized approximations in practical settings for common vision models etc. Formally, the LST output Top is linearly extremal in the sense that æ, + (1 + +e)Au is rapidly assigned low-confidence by the model even for extremely small values of e > O. where Av = Top E..
Thus, using the LST algorithm, we find that the level sets of common models extend outwards in an expansive, connected manner include images of arbitrary classes, which human oracles would never state being similar. Since the linear path from any given source image to LST outputs for arbitrary target images retains high model confidence throughout, unveils remarkable star-like substructure for superlevel sets as shown in Fig where the number of "limbs" or linear protuberances of the star-like structure extraordinarily large, plausibly as large as the number of images in all other classes. Furthermore, to study the size of the level sets beyond the one-dimensional interpolant paths, we analyze the wvo-dimensional triangular convex hull between a given source image and two LST output blindspot images, using quantitative metrics in Section 6
5

---

to X: {2)) (our)(s)) - 63
Scottish Berier
Prediction Confidence
0.8-
0.6
goose
Leval Set Traversal
0.2
Targeted Feature Attack
largeted Attack
ecademic gown
0.7 <-- Linear interpolant Path Coefficient
0.1 02 a3
meerkat
Figure 5: Model Confidence over Linear In- terpolant Paths: We plot the median prediction confidence with respect to class y/2 over linear paths between the target benign sample (x2) and adversarial examples (such as 01 + E12 targeted towards input X22) in input space. While model confidence falls to near-zero using standard targeted adversarial attacks, the linear path between the LST output and source image has high confidence of almost 1.0 throughout.
Figure 4: Schematic of Star-like set substructure of Superlevel sets: The linear interpolant paths between the source image æ, and blindspots found using L.TT maintain high-confidence throughout for arbitrary target images of other classes
4 Disconnected Nature of Standard Adversarial Attacks
At first glance, the images output by the LST algorithm may seem similar those produced using targeted variants of standard adversarial attacks. In this section, we explore the connectivity of high-confidence paths induced by standard adversarial attacks, and show that adversarially attacked source images are not linearly connected with high-confidence paths to their target image. In detail, let 91) and (22, be any two data samples from different classes (y/ * y/2- targeted adversarial attack [Carlini and Wagner 2017] on input æ1 with respect to class y2 formulated as solving 12 arg mig (11) CE( (xx + E12),y2). A related variant, called feature-level targeted adversarial attack [Sabour al.) 2015] instead uses intermediate network activations to craft image perturbations. If f represents the network activations from hidden layer L for an input æ, then this attack attempts match features by optimizing 12 = arg min ap:llaall [l (x) + +112 - f (x22 )l. The hidden layer (I.) that is often selected cor- the final fully-connected layer, and thus often induces misclassification.
responds to pre-activations
Using this framework, we can thus analyze the connectivity of high-confidence paths with respect to class y2 between the target benign sample (xz)) and adversarial examples (such as 21 + E12 targeted towards æ22), using the linear interpolant path between the two. By doing so, we can analyze the level set with respect to class 1/2 using targeted adversarial examples alone, though the perturbation utilized is inherently norm-limited. In Fig 5 we plot the median confidence of normally trained ResNet-50 model over the linear interpolant paths for 1000 source-target pairs. We find that though the model confidence with respect to class y2 is high for both the target benign input #2 and the targeted adversarially attacked input æ + E1 (that is, at the end-points of the linear path), the model does not maintain high confidence over the linear interpolant path between the two, but sharply declines to valley of near zero-confidence over the path. For specific inputs, we often observe sharp, erratic diminutions in target-class prediction confidence over the convex combination, with a non-trivial region of near-zero confidence. This contrasts sharply with the existence of linear-commn connectvtnnetyvvity observed between source image and LST blindspot images. We thus crucially note that adversarial attacks are standalone insufficient to study the structure level sets of common vision models. However, incorporate them into our proposed Level Set Traversal algorithm in fruitful manner.
5 Theoretical Analysis for Common Models
To better understand the geometry of level set submanifolds for given prediction confidence thresh- old, we analyze the nature of confidence preserving perturbations in few simplified model settings.

---

1) Linear Functional: First, consider the classification model to be a linear functional f R + R. that is, faz + x2) = f(z1) + f(x2) and f(xz) )f()). Then, by the Riesz Representation theorem, there exists a unique vector wy [Rd such that f(z) (wy, 2) Vo E IRd. We thus observe that for any vector v orthogonal to wy, f(z + v) = (wy, x + v) = ( x) + (W,, v) = (w,, x). Thus, in this setting, the level sets of f are (d-1) 1) dimensional linear subspaces spanned by the set {o E R (w, 2) 0}. We observe that a similar argument can be extended to affine functions of the form f())) = (w x) + e, where c E R.
We remark that by applying first-order Taylor series approximation for general real-valued smooth functions, we observe near-affine behavior locally within a small neighborhood: f(x + ea) = f<)()((( a) +O(=l|aa) [[)))) Thus locally, observe that confidence preseevingn perturbations can arise from a (d - 1) dimensional plane orthogonal to the gradient. Indeed we incorporate this implicitly in our proposed Level Set Traversal algorithm, wherein the orthogonal projection Ax_ T with respect to the local gradient is iteratively computed, and its relative success can partly be attributed to the orthogonal hyperplane having large dimension, namely (d (d-1) 1).
Another related setting worthy of note that of neural networks that utilize ReLU activations, which induces a piece-wise linear structure over tessellated subsets of the input domain. Thus, given output neuron of a ReL.U network functionally has the form f(xr) = (wn, 2²) + e within a given tessellated region, and thus has constant gradient within the same region. Further, between two such adjacent regions, the two orthogonal (d - 1) dimensional hyperplanes typically intersect over affine space over dimension at least (d (d-2) 2). Thus if æ1, 22 are two inputs such that their linear interpolant path cuts across n distinct tessellated regions, the common intersection of these orthogonal hyperplanes will typically be (d n) dimensional, indicating that there exist perturbations which lie in the common null-space of the gradients as defined along each tessellated region that is crossed. Indeed in the following section, we demonstrate empirically for Residual Networks that though the exact iterative path followed by the Level Set Traversal algorithm is discretized and non-linear, the final outputs so found are often linearly connected through paths of high confidence to the source image, thereby lying within the level set of the original source class. This indicates that the overlap of different (d (d-1) 1) dimensional hyperplanes is non-trivial at non-local scale in image space, whereby we observe extended connected regions of high confidence.
2) Full- Linear Transformations: Let us now consider setting apart from classification, such as regression, wherein the complete vector representation of the output is of principal interest. Let the model be of the form f R + Rd, fzx) = Ax, where A R4 is a full-rank matrix. In this setting, we observe that if fz11) = f = Ax1 = A implying that (x1 x2) = 0, the zero- vector. But since A is full-rank, this implies that æ: Ø2 0, i.e, æ1 22. Thus, is bijective function and has a trivial null space. Thus in this setting, we necessarily have to relax the problem to be that of identifying perturbations of large magnitude to the input that minimally change the
function output: that is, we solve for min ||
Tol Indeed, let the Singular Value Decomposition
(SVD) of the full rank matrix be given by A = UEV*, where U, V are d d orthogonal matrices, and Σ is a diagonal matrix consisting of the positive singular values o; for 1 < i S d. in descending order without loss of generality. Then, ||Av|| = |CEVT = ||VVVV T v) = ||V vll since U
is an orthogonal matrix. Similarly, since V is orthogonal as well, let /
U. that | = ||w|]
||wlll = min Thus, min Tell
|x = min, || | that z²'= = 1. But since Σ is diagonal, -
min; || = - of under the constraint that || 1. It is then easy to observe that the minimum value attained is o = oii.. and is attained when the input vector to f is the right-singular vector corresponding to the minimum singular value of A.
In this setting, we remark that adversarial perturbations, in contrast, can be formulated in this ||lbll setting as max Top with the maximum value given by o2 o and attained by the right-singular vector corresponding to the maximum singular value of A, highlighting the complementary nature of the two problems as indicated previously in Section 2.1 Indeed, the condition number x of an invertible matrix A is defined as K = mmaz/ = | ||A- If condition number is large with A >> 1, the matrix A is said to be ill-conditioned, and induces an inescapable compromise: if Omaz= = ||A||| x 1 (as potentially expected in "robust" networks), then 1/0main = =|A--1 >> 1 is necessarily large, thereby inducing extreme under-sensittivyy along some dimensions; while if 1/ 1/Omin ||A-|| R 1, then Fmax = ||A| > 1 and the model is extremely over-sensitive, similar to the phenomenon of adversarial vulnerability.
7

---

Table 1: Quantitative image distance metrics between output of Level Set Traversal and target images.
Models
RMSE: : + o
foo dist: + o
SSIM: μ + o
LPIPS dist: + o
ResNet-50 (Normal)
0.008 = 0.001
0.046 = 0.020
0.990 + 0.021
0,002 + 0.004
ResNet-50 (AT)
0.029 = 0.008
0.746 = 0.124
0.915 + 0.041
0.057 + 0.037
DeïT-S (Normal)
0.011 + 0.002
0.116 = 0.030
0.973 + 0.024
0.024 - 0.017
DeiT-S (AT)
0.046 = 0.010
0.821 = 0.117
0.898 + 0.041
0.219 = 0.068
Quantifying Under-Sensitivity of Vision Models
In this paper, we primarily consider standard vision datasets such as ImageNet (Deng et al. 2009 and CI (Krizhevsky al. 2009] (latter in Section Of the Appendix). We thereby explore the "blind-spots" of popular vision models such ResNet [He et al., [2016] and Vision Transformers [Dosovitskiy al. 2020 Touvron et al. [022]. Furthermore, we explore the connectivity of such level sets on normally trained variants of such networks, as well as adversarially robust counterparts. For the latter, we analyze robust models trained adversarially against (4/255) 620 constrained adversaries, available on RobustBench [Croce et al. [021]] Specifically, we utilize robust ResNet-50 model
from (Salman et al. [2020] and robust DeiT model from [Singh al.] [2023). We also fix the hyperparameters of LST for all models for a fair comparison (detailed in Section E of the Appendix).
We now proceed quantitatively verify our observations in Section 3 First, to help quantify the deviation between target images and blindspot outputs generated by our proposed Level Set Traversal algorithm, we utilize a combination of classical image metrics such as RMSE, Ex and Structural Similarity Index (SSIM), and perceptual measures such as LPIPS distance using AlexNet [Zhang et al. 2018). For SSIM, a higher value indicates a closer match, while for all other metrics, a lower value indicates a closer match. To calculate these metries, we sample around 1000 source images from ImageNet, and select five other random target images of different classes for each source image. We present the image quality metrics for blindspots discovered by LST in Here the standard deviation is over the different randomly chosen source and target images.
Image Quality Metries:
Table T
Metries for Model Confidence: To evaluate the extent of the model invariance over the regions between the LST outputs and the source image, we evaluate the model confidence with respect to the source class over one-dimensional linear interpolant paths and over two-dimensional subspaces as well. For the latter, we evaluate the model confidence over the triangular convex hull obtained by linear interpolation over three reference points, namely the source image and the two target blindspot images produced using LST. For example, the input at the 'centroid' of the triangle formed by a source image and pair of target blindspots is the arithmetic mean of the three images. We visualize these in 6 wherein the prediction confidence (in the range [0. 11) assigned by the model with respect to the source class is mapped to a continuous colorbar, with high-confidence points (close to 1.0) appearing as bright yellow, and low-confidence points (close to 0.0) appearing as dark violet. Specifically, we use the following metrics: (a) Average Triangle (A) Confidence: the mean of the model's ''s source class confidence over the enclosed triangle, (b) Average Triangle (A) Fraction for various values of 6: the fraction of inputs in the triangular region for which the model confidence is greater than Pare - 8. averaged over all possible target blindspot pairs, where Pare is the confidence of the source image, (c) Average Path Confidence: the average model confidence over all linear paths from the source image to all LST blindspot images. The higher these metrics, the more confident, and thus invariant, the model is in this region. For computing these metrics, we use linear interpolations between the source images and the 5 LST outputs found previously for computing the distance metrics. We thus use ( = 10 triangles for each source image, and sample this triangular area in an equispaced manner to obtain 66 images for computation of the triangle (A) metrics. We present these metrics in Table 2 along with the mean (and standard deviation) of model confidence on the source class images (pare) for reference. Here, the standard deviation is over the different randomly chosen source images.
We can now quantitatively confirm many of the trends we qualitatively observed in Section 3 In [ we observe that the LST outputs found for the models are closer to the targets as compared to the adversarially trained models. The difference particularly stark when comparing the lxc distances, as fx is a particularly sensitive metric and is also the threat model against which the models were trained. However, for other distance metrics like RMSE or LPIPS, the difference between normally trained and adversarially trained 5e is not as high. In particular, LPIPS
8

---

Table 2: Quantitative confidence metrics over the triangular convex hull (A) of given source image and two target LST blindspot image-pair and over linear interpolant paths between source and blindspot images. (For reference, random classifier would have confidence of 0.001)
Models
Pur
Avg A Conf.
Avg Frac. (a + a) 8 - 0.2
Avg Path Cont.
(u g))
(j ()
5 0.0
0.1
8 0.3
(j d)
ResNet-50 (Normal)
0.99 + 0.00
0.56 + 0.10
0.13 +.13+00.1 0.15
0,51
+ 0.11
0,53 + 0.1
0.54 + +0110
0.96 00.55
ResNet-50 (AT)
0.88 0.11
0.83 0.09
0.49 =
0.79
0.13
0.85
+ 0.1
0.88 0.09
0.93 0.06
D (Normal)
0.85 + 0.06
0.68 + 0.05
0.54 + 0.11
0.67
+ 0.06
0.71
+ 0.06
0.73 0.06
0.94 0.02
DeiT-S (AT)
0.76
0.59 + 0.07
0.20 + 0.09
0.43
0.14
0.63 = 0.15
0.76 4: 0.12
0.73 0.06
Figure 6: Visualization of confidence of standard (top) and robust (bottom) ResNet-50 models over the triangular convex hull of source 'goose" image and two LST outputs for all pairs of target images from 4 other classes (same images as in Fig 3). In all source-target image pairs, the linear interpolant path maintains high confidence, implying that the source image is linearly connected the LST target output in star-like substructure within the level set. For adversarially trained models, we observe that significant fraction of the triangular hull lies in the superlevel sets of high-confidence. therehy indicating their under-sensitivity in regions far beyond their original threat model.
distances for both models are low, which indirectly implies that the perceived difference between the images for humans oracles is relatively low. This can also be confirmed by visually inspecting the images in Fig For the normally trained DeiT-S, the distance metries are very similar to that of ResNett55, with slightly higher RMSE and LPIPS distances. However, the adversarially trained (AT) variant is significantly less under-sensitive compared both its normal counterpart and the ResNet-50 (AT). Specifically, the LPIPS distance metric is much greater for DeïT-S (AT), which differences in the image. We can confirm this in 7 where the LST output for the adversarially trained DeiT-S model contains visible traces of the source image. However, the LST outputs are still clearly much closer to the target class as compared to the source, which indicates that there are still some significant blind spots in DeiT-S (AT).
implies there exist significant numan-percepilnn
However, when we measure the extent of model invariance the convex regions enclosed between the LST output and the target images, we find that adversarially trained ResNet-50 are overall more invariant (or under-sensitive) as compared the normally trained variant. For example, the average triangle confidence for ResNet-50 (AT) is higher than that of normally trained ResNet-50, even though its source confidence is much lower. We also find that much larger fraction of the triangular convex hull lies within the superlevel set for Pare - for ResNeet50 (AT) as compared to normal ResNet-50 for all values of 8. The average path confidence much closer to Pure for ResNet-50 (AT) as compared to normally trained ResNet-50. This quantitatively verifies the observation made in Fig O that adversarial training demonstrably exacerbates under-sensitivity. Interestingly, these robust models are sendrr-ee over subsets of the input domain that lie well beyond the original threat model used in its training. Moreover, between the normally trained DeiT-S and ResNet-50 models, the former appears to be more invariant with greater average confidence over the triangular convex hulls, despite having lower source image confidences Par. For the robust variant of DeiT-S however, the trend is less apparent due to the significantly lower average source image confidences Pare- However the average relative A fraction becomes higher for larger values of (such as 0.3), indicating that the superlevel sets are indeed expansive, albeit for lower confidence thresholds.

---

7 Discussion
While the existence of level sets alone is not very significant in itself, using the proposed LST algorithm, we find that the level set for common vision models is remarkably expansive - large enough to contain inputs that look identiall to arbitrary target images from other classes. Since the linear path from any given source image to LST blindspot outputs retain high model confidence throughout, the level sets have a star-like connected substructure, where the number of 'limbs' or linear protuberances of the star-like structure is extraordinarity large, plausibly as large as the number of images in all other classes. This is considerably noteworthy since it indicates the hitherto unknown and unappreciated scale and extent of under-sensitivity in common vision models. Moreover this hints at the potential degree of difficulty towards adequately mitigating this phenomenon in practical settings. For instance, if the level set for images of class 3/1 contained sizable protuberances towards only one other class y/ alone, the problem could perhaps be tackled by introducing a contrastive training objective that encourages the network to better discriminate between yı 1/2 image pairs by utilizing a denser sampling of related image augmentations, likely resulting in the diminution of these specific 'directed' protuberances (assuming reasonable train-test generalization). But since the star- like connected substructure uncovered by LST implies that such protuberances exist towards any generic image of any other class, such simple approaches will likely be ineffective and possibly computationally infeasible from a combinatorial perspective. Thus, based on the observations uncovered with LST, we hypothesize that addressing the pervasive issue of under-sensitivity in conventional vision models might present a significantly non-trivial challenge.
8 Related Work
The phenomenon of under-sensitivity in classification models was first pointed out by Jacobsen et al.) [2018a], wherein they utilize a class of invertible neural networks called fully Invertible RevNets (Jacobsen et al. 20186 Kingma and Dhariwal 2018] to specifically craft input images that do not affect network activations at a given layer. In contrast, our proposed algorithm is applicable to general network architectures since we solely utilize imput-gradients to perform the traversal over image space. [Tramrr et al. [2020] further demonstrated that due to the misalignment of lp norm bounded balls and the ideal set human- iimnaa--ippreettbllee perturbations, networks that are adversarially trained against such Ep bounded perturbations of relatively large radius are overly-smooth, and become excessively susceptible to invariance-based attacks within the same lp radius. To find such images that lie within a given l, threat model, but induce human oracles to change their label assignment, the authors propose to identify the training image from another class closest in image space and apply series of semantic- pressacviiieeeiiiggggg. transformations, and additionally use techniques such as realignment and spectral clustering. Given that these operations are fairly complex, the attack algorithm is slow, with alignment alone requiring an amortized time of minutes per input example. In contrast, our proposed method relies upon gradient-backpropagttee backpropagation steps which are efficiently technique is seen to be successful
parallelized across minibatch of input images. Furthermore, for arbitrary source-targe target image pairs, since we do not utilize near-neighbour training images as the target. On the theoretical front, Jayaram et al. 22020) analyze the problem of span-recovery of neural networks given only oracle access to the network predictions. They characterize the feasibility of span recovery, and thus approximations of the null space of networks in a provable setting, but remark that its success in practical settings is potentially limited to networks that are extremely thin.
Conclusions
In this work, we investigate the phenomenon of under-sensitivity of classification models, wherein maggitma semantic perturbations leave network activations unchanged. To identify such "blind spots" that occur within high-confidence level sets of common vision models, we develop a novel Level Set Traversal algorithm that iteratively perturbs given source image to appear visually similar to an arbitrary target image by utilizing orthogonal projections with respect the local input gradient. The proposed method is applicable to general neural networks, and helps uncover a star-like substructure for the level and superlevel sets of CNNs and ViTs on common datasets. We further observe that adversarially trained models retain high-degree of confidence over regions that lie far beyond its original threat model, with super-level sets that extend over a significant fraction of the triangular convex hull between a given source image and arbitrary pair of blindspot images.
10

---

10 Acknowledgements
This project was supported in part by grant from an NSF CAREER AWARD 1942230, ONR YIP award N00014-22-1-2271, ARO's Early Career Program Award 310902-00001, Meta grant 23010098, HR001 12090132 (DARPA/RED), HRO01 19900226 (DARPA/GARD), Army Grant No W91 INF21 20076, NIST 60NANB20D134, the NSF award CCF2212458, an Amazon Research Award and an award from Capital One.
References
A. Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give false sense of security: Cir- cumventing defenses to adversarial examples. In International Conference on Machine Learning (ICML), 2018.
N. Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (SP). IEEE, 2017.
N. Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, and A. Madry. On evaluating adversarial robustness. arXiv preprint arXiv: (rXiv::902.06705. 2019.
F. Croce and M. Hein. Minimally distorted adversarial examples with fast adaptive boundary attack.
In International Conference on Machine Learning (ICML), 2020.
Croce and M. Hein. Adversarial robustness against multiple and single 1p-threat models via quick fine-tuning of robust classifiers. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 4436-4454, PMLR, 17-23 Jul 2022. URL htttp:://proceedingg
mlr. waalr.prees//v62//rroee v162/ html
Croce, M. Andriushchenko, V. Sehwag, F. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein. Robustbench: standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
URL
JCt7B
J. Deng, W. Dong, R. Socher, I..J. Li, K. Li, and I.. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 148-255,722 2009. doi: D0.1109/CVPR22000 200.5506888..
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprini arXiv:2010.1 1929, 2020.
J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and hamessiing adversarial examples, 2015.
S. Gowal, J. Uesato, C. Qin, P.-S. Huang, T. Mann, and P. Kohli. An alternative surrogate loss for pgd-based adversarial testing. arXiv preprint arXiv:1910.00 1930.09338 2019.
K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level perfor- mance on imagenet classification. arXiv preprint arXiv: Arxiv- Arxiv-1502.01852. 2015.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
J. Jacobsen, J. Behrmann, R. S. Zemel, and M. Bethge. Excessive invariance causes adversarial vulnerability. CoRR, abs/188 his/1811/ 100401, 201 8a. http:/ttt://axxi www.aaiwiv..oogaaaaaaaa aa//aa////// httg//bb//811...... 01811.004011
J. Jacobsen, A. Smeulders, and E. Oyallon. i-revnet: Deep invertible networks. arXiv preprint 17X2::8002007888 2018b.
R. Jayaram, D. P. Woodruff, and Q. Zhang. Span recovery for deep neural networks with applications to input obfuscation, 2020.
D. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018

---

A. Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533, 2016.
A. Kurakin, I. Goodfellow, and S. Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations (ICLR), 2017.
C. Laidlaw, S. Singla, and S. Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In ICLR, 2021.
A. J. Levine and S. Feizi. Improved, deterministic smoothing for 1_ certified robustness. M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 6254-6264, PMLR, 18-24 Jul 2021. URL ttttps::/proceeddiin
mml pptpss/veeeeseeeeeeepeee w119//eevine211 html
A. Madry, A. Makelov, Schmidt, TT. Dimitris, and Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR), 2018.
A. Paszke, S. Gross, F. Massa, A. Lever, J. Bradbury, G. Chanan, T. Killeen, 7. Lin, N. Gimelshein, L.. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Vito,, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. J. Bai, and S. Chintala. Pytorch: An imperative style, nigh-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages Curran Associates, Inc., 2019. URL http::/paperr:: http::///ppppeee.eeiippspssspes neurips cc/ppprr
9015-pytorch- 1005--9forrd-----pprratvv-----ee am--ippratiie-sty/// hirp-eehhgh--pfformmaa deep- Lew.aanennede------innnen learning--nn... library. pdf
R. Rade and S.-M. Moosavi-Dezfooll. Reducing excessive margin achieve better accuracy
vs. robustmess trade-off. In International Conference on Learning Representations, 2022. URL
https:///penrrevie. wwt.:n//forrmm..cccea000004-----
Sabour, Cao, F, Faghri, and D. J. Fleet. Adversarial manipulation deep representations. arXiv 2015,
preprint arXiv:1511.055
H. Salman, Ilyas, L. Engstrom, A. Kapoor, and A. Madry. Do adversarially robust imagenet models 08489 transfer better? CoRR, abs/2007.08458 2020. URL /RLLaatpes/aaxivv.rrr/aaas/////
N. D. Singh, F. Croce, and M. Hein. Revisiting adversarial training for imagenet: Architectures, training and generalization across threat models. arXiv preprint arXiv:2303.01877 2023.
S. Singla and S. Feizi. Second-order provable defenses against adversarial attacks. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 8981-888888888 1-8911. PMLR, 13-18 Jul 2020. URL htttps::/proceeednngg mlr www.ppressvviie www.///iigaaaaaaaanoooo html
G. Sriramanan, S. Addepalli, A. Baburaj, and R. Venkatesh Babu. Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses. Advances in Neural Information Processing Systems (NeurIPS), 2020.
C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks, 2014.
H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-cfficient image transformers & distillation through attention. In International conference on machine learning, pages 10347-10357. PMLR, 2021.
F. Tramer, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations (ICLR), 2018.
Tramèr, J. Behrmann, N. Carlini, Papernot, and J.-H. Jacobsen. Fundamental tradeoff' between invariance and sensitivity adversarial perturbations, 2020.
12

---

E. Wong, F. Schmidt, and Z. Kolter. Wasserstein adversarial examples via projected sinkhorn iterations. In International Conference on Machine Learning. pages 6808-6817. PMLR, 2019.
D. Wu, S.-T. Xia, and Wang. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems, 33, 2020.
H. Zhang, H. Chen, C. Xiao, S. Gowal, R. Stanforth, B. Li, D. Boning, and C.-J. Hsieh. Towards stable and efficient training of verifiably robust neural networks. arXiv preprint 1rXiv:1906.06316, 2019a.
H. Zhang, Y. Yu, J. Jiao, E. Xing. L. El Ghaoui, and M. I. Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICML), 2019b.
R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as perceptual metric, 2018.
13

---

Appendix
A Theoretical Characterization of Level Sets
g IRd - R. be a differentiable function. For E IR, the level set T.g (c) is said to be regular if the gradient of g is non-vanishing for all points in the level set, that is. Vg(x) * Væ E Lc(c). We can then show that the regular level sets of continuously differentiable functions are (d 1) dimensional submanifolds of I2d (Lemmal) IT. using the Implicit Function Theorem. We restate the lemma here for the readers' convenience, and present its proof below.
Lemma 1. Ifg Rd - RR is continuously differentiable function, then each of its regular level sets a (d- 1) dimensional submanifold of R
Proof. Without loss of generality, let (1)(0) be regular level set of g. and let ε [0)(0) (if O,
can define a new function shifted by the constant -c, so it suffices to consider I.g (0)). By definition of regularity, we know that Vg(a) of and least one of its components must be non-zero. By renumbering the coordinates, without loss of generality, we can assume the first vector component is non-zero, i.c. g(a)= 0z1 g(a) + O. Thus we have a function such that g R x Rd-1 + R with
Ög dx
g(0,, ....0.,, =
(01, 02,-00,,.0..
ad) #
Since the Jacobian submatrix Implicit Function Theorem 9 a, which states that there exist open sets U, V such that U C R with a1 E U and V C Rd-1 with (A2,... aa) E V, and an "implicit" continuously differentiable function t: V U such that t(a....... ad) d1 with the property that for all (Ig...., Fa) V, have:
J2z ((a) is non-zero and thus invertible, we can thus apply the
.2,-..,,
Ta) =0
Further, we observe that the set W = U x C RR is open since U, U. are open. Now, (0) n W, f(x2. ma) Thus, consider da L, (0) n W + I with @ (x1, 22. ................................ rd) (g(1((22...... Ta); .., 52 Fd): ... va)
which is map with its first coordinate identically zero. Thus, if (I1, is the projection map from I2 to Rd-1 that omits the first coordinate, II_1 Oa is a continuously differentiable chart from (0) W to I containing a. Since this is true pointwise for each Lg (0), (0) a (d 1) dimensional submanifold of Rd.
x.) (x2, (. ma)
B
Implementation Details and Methodology
B.1 Details on Datasets and Pretrained Models
In this paper, we present results on vision datasets such as ImageNet (Deng al. 2009] and CIFAR- 10 (Krizhevsky et al. 2009), given that they have come to serve as benchmark datasets in the field. Image.ett Large Scale Visual Recognition Challenge (ILSVRC) consists of over million labelled RGB color images arising from a wide variety of objects, totalling to 1000 distinct classes. The images themselves are of relatively high resolution, with common vision models utilizing an input shape of 224 x 224 pixels. On the other hand, CIFAR-10 is smaller dataset, consisting of 32 x 32 pixel RGB images from ten classes: "airplane", "automobile", "bird", "cat", "deer", "dog", "frog" "horse", "ship" and "truck". In this paper, all training and experimental evaluations were performed using Pytorch Paszke et al. 2019].
The foo norm constrained threat model is the most commonly used setting for adversarial robustness in both these datasets. On ImageNet, we analyse robust models trained with adversaries constrained within a radius of 4 4/255. while on CIFAR-10, we consider robust models trained with adversaries over larger radius of 8/255. We use adversarially trained models as made available on RobustBench [Croce et al.) 2021 to analyze popular vision models. For the class of Residual Networks (He et al.) 2016), we analyze normally trained ResNet-50 model from the Pytorch Image Models (timm) repository, and the adversarially robust ResNet-50 model from Salman et al. (020). For the class of Vision transformers (Dosovitskly et al. 2020 Touvron et al. [2021]. we utilize robust DeiT' model
from Singh et al. 20033 For CIFAR- 10, we utilize adversarially trained WideResNet-344 34-10 models
14

---

09913
0337
3,99
CORAI
25842
0L778
3/277
TAVO
pARSS
DITMA
01873
8.71
Figure 7: The images returned by LST for the same 5 source and target images in for normally trained (left) and adversarially trained (right) DeiT-S models. We again observe the LST outputs are highly similar to the target images in this setting well.
from Wu al. 2020 Zhang al. 2019h] and Rade and Moosavi-Deeziooli [2022], alongside normally trained counterpart.
B.2 Algorithm and Hyperparameter Details
ImageNet In the main paper, we fix the parameters of the LST algorithm for all the visualizations (Fig 7 and Tables 2 the Main paper). The scale factor for the step perpendicular to gradient, y), is 10 2. The stepsize for the perturbation parallel to the gradient - VCC (C or is 10-3 The confidence threshold (d) is 0.2, which means that the confidence never drops below the confidence of the source image by more than 0.2. In practice. significant drops in the confidence during the level set traversal. The algorithm is run for m = 400 iterations. The number of iterations required is relatively high compared to standard adversarial attacks because of the much larger distance between the source and the target images. We also present results with different parameters in the subsequent sections.
rarely observe such
CIFAR-10 We fix the parameters of LST when applied on CIFAR-10 in similar manner. The scale factor for the step perpendicular the gradient, or 1). is 10-2 The stepsize for the perturbation parallel to the gradient - ((CE( y). or is 2 x 10-3, The confidence threshold (8) is 0.25, and we run the algorithm for 300 iterations obtain high fidelity output images. The adversarially trained model is observed to have low prediction confidence even on the source images, and thus again requires large number of iterations for good convergence.
C Results on CIFAR-10
In this section, we present qualitative and quantitative results obtained on the CIFAR-IO 10 dataset. Similar Section of the Main paper, we consider RMSE, foc. the Structural Similarity Index (SSIM), and LPIPS with AlexNet [Zhang et al. 2018). For SSIM, higher value indicates a closer match, while for all other metrics, lower value indicates closer match. To calculate these metrics, we sample at 1000 images from CIFAR-10 to serve as the souree images, and consider 9 fixed images from all the remaining classes as targets as shown Figure-8] By visual inspection, we see that any source image can be iteratively modified by LST to resemble any given target image, while retaining high confidence on the original class as indicated in each column the figure. Indeed, we quantify this in Table where observe that the final output images of the LST traversal algorithm are close
15

---

Figure 8: CIFARIO dataset- Images returned by the Level Set Traversal algorithm for a normally trained WideResNet-25 model. The image in the i-th row and j-th column is obtained by using the j-th image as the source image and i-th image as target. The images in the diagonal are the unchanged random images, as the source and target are identical. The confidence of the model prediction for the source class (names top of each column) is displayed just below each image. We observe that almost any source image can be iteratively changed to resemble any target image very closely without any loss in confidence, as indicated by images presented in any given column.
to the target over these different metrics; in particular the high SSIM similarity and small LPIPS distance implies high visual similarity for human oracles.
Similar to the Main paper, again evaluate the extent of the model invariance over the regions between the LST outputs and the source image, and compute various metrics which evaluate the model's 's confidence in the source class all triangular convex hulls enclosed by the source image and any two target images. We observe that the extent of convexity is greatly pronounced on CIFAR- 10, with high confidence maintained over significant fraction of the convex hull. Indeed, quantify these observations we use the following metrics: (a) Average Triangle (A) Confidence: the mean of the model's source class confidence over the enclosed triangle, (b) Average Triangle (A) Fraction for various values of 6: the fraction of images in the triangular region for which the model confidence greater than Pae 8, averaged over all possible target pairs. Here Psre the confidence of the source image, (c) Average Path Confidence: the average model confidence overall linear paths from the source image all target images. The higher these metrics, the more confident, and thus invariant, the model is in this region. For all of these metrics, we use ten linear interpolations between the source and each of the LST output images and around 65 images that cover the triangular region. We present these metrics in Table along with the mean (and standard deviation of) confidence of the model on the source class (pec) for reference. We thus observe that the average confidence over triangular convex hull is extremely high on average, and that the super-level sets corresponding to 5 0.1, 0.2, 0.3 cover a significant fraction of the triangular hull. Further, we observe that the linear interpolant paths maintain very high degree of confidence as well.
16

---

Figure 9: CIFARIO dataset: Images returned by the Level Set Traversal algorithm for an adversarially trained WideResNet-228 model.
Figure 10: Triangular convex hull enclosed by LST outputs and source images for CIFAR-IO for (top) normally trained WideResNet-28. (bottom) adversarially trained WideResNet-28. In all source-target image pairs, the linear interpolant path maintains high confidence, implying that the source image is linearly connected to the LST target output in star-like substructure within the level set.
D Limitations and Computational Complexity
We note that the proposed Level Set Traversal algorithm typically requires around 200 400 iterations to find images which are near-identical to the target image while still maintaining high confidence for the source class since both the semantic and pixel level changes required are of relatively large magnitude. If restricted to much smaller budget such as 10 steps, we observe that the final image is visually distinct from the target image. Thus, it non-trivial to design experiments which explicitly train new networks against outputs found by our proposed LST algorithm, akin to that performed
17

---

Table 3: Quantitative image distance metrics between output of Level Set Traversal and target images with WideResNet on CIFAR-10.
Models
RMSE dist: μ u
Ex dist: j
SSIM: μ p
LPIPS dist: μl
WideResNet (Normal)
0.020 0.009
0.071 +: 0.033
0.985 +: 0.03
0.001 + 0.004
Trades [Zhang et al. 2019b
0.109 = 0.044
0.599 1 0.123
0.742 0.132
0.074 + 0.037
Rade and Monsavi- Deztool 2022
0.113 0.033
0.652 + 0.117
0.717 0.111
0.079 + 0.043
Wü et al.
2020
0.127 0.032
0.653 + 0.12
0.695 0.105
0.09 = 0.042
Table 4: Quantitative measures of model confidence invariance over the triangular convex hull (A) of given source image and all possible target image pairs and over linear interpolant paths between all possible source-target image pairs, using WideResNet on CIFAR-10.
Poc
Avg (jct.n)
(w=
Avg (j- o)
G m)
0.1
4= 0.2
WideResNNe (Normall
0,994 + 0.033
0857 +: 0.097
0.364 0
0.81 0.1133
0.83 0.109
0.42 + 0.105
0:985 + 0.031
Trades Zhang et al 20196
0.745 1 0.2
0,888 1 0.093
0,31
1: 0.233
0.994 1: 0.021
0.999 1: 0.007
0,999 1: 0.004
0.878 1: 0.099
Rade Mooole 2022
0,745 1.0.1.1.11 0.18
0873 0.06
0.34 0.312
0.995 0.00
0,999 1 0.005
L. 0.002
0.862 1. 0.073
0.785 4
0.891
0.19
0.997 00077
1.00
within standard adversarial training Madry et al. 2018 Furthermore, our quantitative experiments could be potentially extended over higher-dimensional simplices as well, if the curse of dimensionality could be addressed effectively.
D.1 Best and Worst case scenarios for LST based exploration
Given that the LST algorithm depends primarily on the parallel and orthogonal components of the model gradient, LST is efficacious deep networks that do not suffer from gradient obfuscation or gradient masking issues, which encompasses nearly all commonly used vision models. On the other hand, for models with non-differentiable randomized inference-time components, the LST algorithm would have to be modified to utilize techniques such as Backward Pass Differentiable Approximation (BPDA) and Expectation over Transformation (EoT), proposed in past adversarial attack literature [Athalye et al. 2018), that helps overcome shattered, stochastic, exploding and vanishing gradients. We also note that these modifications also often require sizable increase in the maximum iteration count to achieve adequate convergence levels.
D.2 Wall Clock Time
We record wall clock time on a single RTXAS000 GPU with a ResNet-50 model on ImageNet, using a batchsize of 100, and report mean and standard deviation (μ = σ) statistics over 5 independent minibatches. The proposed LST algorithm is seen to require 101.8 seconds (# 0.4s) for 400 iterations, and 52.0 seconds (=0.3s) for 200 iterations. In comparison, a 100-step PGD attack of the same model takes 25.1 seconds 1.4s.. Thus, per iteration, the L.ST algorithm takes the same amount of time as PGD attack iteration (both about 0.25 seconds). In total, the LST algorithm of 200/400 iterations requires 2x/4x the computation time of a standard 100-step PGD adversarial attack.
We generally observe that for adversarially robust models, near-optimal convergence to the target image (wherein LST output looks essentially identical the target image) requires larger number of steps as compared to normally trained models. While we fix the max iterations to 400 across all ImageNet models for instance, possible to reduce this parameter for normally trained models while maintaining good visual quality.
E
Ablation and Sensitivity Analysis
Here we perform ablation studies on LST hyperparameters to study the impact of each one indepen- dently. We use default parameters - number of LST iterations m = 400, step size perpendicular to the gradient 10-2 step size parallel to the gradient 0.002, confidence threshold 5 = 0.2 unless mentioned otherwise. We evaluate our ablative analysis on set of 500 images from ImageNet using normally trained ResNet-50 model. In general, we observe trade-off' between optimizing for
18

---

visual similarity towards the target image of another class, and high- high-confidence metrics associated with the LST blindspot output so produced.
A1. Varying number of iterations m for which LST is run (see Tables 5 and 6p: As the number of iterations increases, the distance metrics between the output of LST and the final images decrease. We ideally desire these deviations between LST outputs and target images to be low, to ensure good visual quality. Hence, a higher m value is preferred for performing LST.
A2. Varying adversarial step size e parallel to the negative gradient for LST iterations m 100 (see Tables and 8): With fewer iterations for LST, larger adversarial step size is greatly beneficial. As shown in the Tables, a larger e results better confidence metrics over the triangular convex hulls.
A3. Varying step size v perpendicular to the gradient for LST iterations m 100 (see Tables and 10 From the Tables, we can see that as the step size 1j increases, LSTT output images become closer to the target images for smaller value of m 100. Also, with larger 7), we can achieve larger triangular confidence scores. However, the best confidence scores are achieved using larger m values.
A4. Varying step size perpendicular to the gradient (see Tables and 12): : As expected, the distance metrics between the LST output images and the target images are larger with lower r). 0.01, 0.05 achieve desirable small distance scores. Among these, our default parameter 0.05 achieves the best confidence scores, alongside adequate convergence with minimal distance metrics towards the target image.
AS. Varying step size parallel the gradient (see Tables 13 and 14 A larger results in LST output images that are closer to the final images in terms of our distance metrics. We also note that a larger adversarial perturbation helps in achieving higher triangular confidence scores. We observe that setting the adversarial step size to zero results in output images that are perceptually different compared the target image, observed with the larger image distance metrics such LPIPS distance. However, we also observe that even extremely small non-zero adversarial step-sizes are very effective in improving convergence towards the target images.
Table 5: A1. Varying number of iterations for which LST is run.
RMSE: / +
foo dist: +
SSIM: μ +: a
LPIPS dist: μ +
100
0.141 + 0.029
0.351 + 0.018
0.715 + 0.1
0.288 + 0.109
200
0.052 + 0.01
0.133 +
0.913 + 0.074
0.064 + 0.051
300
0.019 + 0.004
0.015
0.971 = 0.05
0.011 + 0.013
400
0.008 + 0.001
0.045
0.989 + 0.025
0.002 + 0.002
Table 6: A1. Varying number of iterations m for which LST is run.
Pue (j + d)
Avg Conf. o)
Avg A Frac. (p (t)
Avg Path Conf. (u
5 0.0
6=
d= 0.2
5 0.3
100
0.996 +: 0.0.0.0 0.022
0.111
0.333 0.291
0,876 0.134
0.892 1: 0.125
0.901
0.999 + 0.007
200
0.996
0. 0.022
0.702+0..00
0.204 -:
0.654 0.139
0.673
0.135
0.686 0.132
0.996 0.015
300
0.996
0.022
0.611:0.11
0.157
0.562 0.131
0,582 0.128
0.595 0.126
0.985 0.034
400
0.996
+: 0.022
0.564 0.119
0.138 + 0.1622
0.512 = 0.129
0.534 +: 0.126
0.547 + 0.124
0.96 + 0.062
19

---

Table 7: A2. Varying adversarial step size for LST iterations m 100.
RMSE: μt +
&o dist: μ +
SSIM: μ =
LPIPS dist: μl
0.002
0.141 + 0.029
0.351
0.715 0.1
0.288 + 0.109
0.004
0.141 + 0.029
0.351 0.018
0.715 - 0.1
0.288 + 0.109
0.008
0.141 + 0.029
0.351 0.02
0.715
0.288 + 0.109
0.012
0.141 + 0.029
0.352 + 0.022
0.715
0.289 + 0.109
Table 8: A2. Varying adversarial step size for LST iterations 100.
Punt +
Avg Conf. (jc (ctto))
Avg A Frac. (2:te) 8 0.1
Avg Puth Conf.
8:0
6 0.2
à 0.3
a))
0.002
0.996 + 0.022
0.91 + 0.111
0.333 = 0.291
0.876 + 0.134
0.892 + 0.125
0.901 + 0.119
0.999 0.007
0.004
0.996 + 0.022
0.914 0.107
0.353 1: 0.3
0.582 + 0.129
0.897 + 0.121
0.905 - 0.116
0.999
0. 0.006
0.996
0.022
0.918 0.104
0.377
0.309
0.887
0.901 + 0.118
0.909 - 0.112
0.999 0.005
0.012
0.996 0.022
0.92 0.102
0.393 0.314
0.89
0.122
0.904 + 0.115
0.912 0.11
0.999 0.004
Table 9: A3. Varying step size 7 for LST iterations 100.
7
RMSE: =
Lo0 dist: μ +
SSIM: to
LPIPS dist: μ u
0.01
0.141 + 0.029
0.351 + 0.018
0.715 0.1
0.288 + 0.109
0.02
0.051 + 0.011
0.132 + 0.015
0 0.074
0.063 + 0.052
0.03
0.019
0.006
0.062 0.02
0.972 2
0.01 1 0.019
0.04
0.008 0.012
0.046 + 0.034
0.988 + 0.035
0.004 + 0.026
Table 10: A3. Varying step size for LST iterations 100.
7
Pac +)
Avg Conf. (
(p Avg $ 0.1
Avg Conf.
= 0.0
6= 0.2
5 0.3
(p g)
0.01
0.996 + 0.022
0.91 #: 0.111
0.333 0.291
0.876 +: 0.4
0.892 + 0.125
0.901 # 0.119
0.999 +: 0.007
0.02
0.996 + 0.022
0.704 0.1122
0.18 4: 0.197
0.655 0.132
0.676
1:76::000 0.129
0.688 :: 0.127
0.997 + 0.011
0.996
0.114
0.113
0.04
0.996 + 0.022
0.622 + 0.102
0.13
0.574 0.108
0.594 0.108
0.607 : 0.106
0.993 + 0.017
20

---

Table 11: A4. Varying step size for LST.
RMSE: + o
foo dist: +
SSIM: + a
LPIPS dist: +
0.001
0.257 + 0.052
0.642 + 0.036
0,433 + 0.093
0.573 + 0.089
0.005
0.052 + 0.01
0.135 + 0.015
0.912 + 0.074
0.065 + 0.051
0.01
0.008 + 0.001
0.045 + 0.019
0.989 = 0.025
0.002 = 0.002
0.05
0.004 + 0.013
0.04 = 0.032
0.994 = 0.027
0.003 = 0.024
Table 12: A4. Varying step size 1) for LST.
Avg A Conf.
Avg A Frac. (μ + o)
Avg Path Conf.
(μ o)
(4 o)
0.0
0.1
0.2
5 = 0.3
(j
0.001
0.996 0.022
0.998 0.021
0.612
0.379
0.997 0.03
0.998 + 0.026
0.998 = 0.024
0.002
0.005
0.996 0.022
0.707 0.129
0.237
0.227
0.661
0.14
0.679 + 0.136
0.69 0.134
0.996 0.017
0.01
0.996 + 0.022
0.564 + 0.119
0.138
= 0.162
0.512 + 0.129
0.534 + 0.126
0.547 = 0.124
0.96 +: 0.002
0.05
0.996 0.022
0.61:00
0.131 0.154
0.561 = 0.108
0.582 = 0.104
0.595 = 0.103
0.985 +:
Table 13: AS, Varying adversarial stepsize for which LST is
RMSE: μ +
fxc dist: +
SSIM: μ + o
LPIPS dist: μ =
0.02
0.008 + 0.003
0.055 + 0.055
0.986 0.03
0.005 0.015
0.002
0.008 + 0.001
0.045 #
0.989 + 0.025
0.002 + 0.002
0.0002
0.007 + 0.001
0.04
0.99 0.025
0.002 0.002
0.0
0.093 + 0.063
0.234 + 0.147
0.83 + 0.129
0.147 + 0.118
Table 14: AS. Varying adversarial stepsize for which LST is
Pure +
Avg Conf. ta
Avg Frac. (u 6 0.1
Avg Path Conf. (jc a)
8 0.0
A= 0.2
0.02
0.996 + 0.022
0.596 + 0.114
0.193 : 0.195
0.551 = 0.122
0.57 : 0.119
0.581 0.111
0.981 0.012
0.002
0.996
0.022
0.564
+ 0.119
0.138 + 0.162
0.512 + 0.129
0.534
0.126
0.547 0.124
0.96 + 0.062
0.0002
0.996
0.527 0.125
0.097 0.129
0.135
0.491
0.507 0.131
4 0.088
0.0
0.996 0.022
0.741 + 0.104
0.148 + 0.177
0.656 + 0.126
0.715 + 0.12
0.731 + 0.114
0.97 + 0.032
21

---

F Additional Level Set Traversal Examples
Figures
1 and 13 show examples images output by our LST algorithm using ImageNet
images for normall ly trained ResNet-50 from ablation studies A1, A2, A3, A4, and A5, respectively.
Figures
19 and show their respective triangular convex hull.
block
Tixttan
dIfT dwclling
Tistan
bottle
cit Swdllig
5.807
00604
08630
12
23097
Thener anter
Tplr Sertl
TEM Jartee
63460
19607
50009
822
0.440
0.9900
LI94
30099
Figure 11: ImageNet images returned by the LST algorithm for normally trained ResNet-50. These are examples from ablation A1. Left: m = 100 and Right: m = 400.
black swen
Thette Teerer
daaling
blark cawn
Tibeton
Expe beatle
Sawilog
Mod Motmma
0/481
DAVA
1.0
C.wW
1.
51697
Homte lente
6.443
55634
03429
1990
35002
Tgp Sentee
ATT
00017
8.074
0.991
0.504
C390
33
3.009
Figure 12: ImageNet images returned by the LST algorithm for a normally trained ResNet-50. These are examples from ablation A2. Left: m = 100, c = 0.002 and Right: m 4000 <= 0.012.
22

---

Siper
dwelling
Swin
Tige
Swelling
6:271
E3007
2.009
0.004
0.5955
LG
C2000
0508
300004
Hoatee Mriter
63460
1900
50009
C12
05440
0.999
69770
9999
19944
0330
S
CI Ooe by
CUT
Figure 13: ImageNet images returned by the LST algorithm for normally trained ResNet-50. These are examples from ablation A3. Left: m 100, 0.01 and Right: m 400, 0.04.
39
3.9
100
10
1K
1.6
6.999
09900
1.999
3.99
Oig Share
11911
00WM
6.99
1999
1990
1954
CEP ovening
Figure 14: ImageNet images returned by the LST algorithm for normally trained ResNet-50. These are examples from ablation A4. Left: 77 = 0.001 and Right: 7 = 0.05.
23

---

Med man
Zapn
03071
1:9
16
Thmes leme
Sginr Marte
9.20
21441
DARS7
10
16
1994
1Y94
10
0423
5002
03EM
C250
OFF cowliin
BA237
Figure 15: ImageNet images returned by the LST algorithm for normally trained ResNet-50. These are examples from ablation A5. Left: e = 0.0 and Right: = 0.02.
Figure 16: Triangular convex hull enclosed by LST outputs and source images for ImageNet with normally trained ResNet-50. These are examples from ablation A1. Top: m = 100 and Bottom: m 400.
MANA
Figure 17: Triangular convex hull enclosed by LST outputs and source images for ImageNet with
normally trained ResNet-50. These are examples from ablation A2. Top: m = 100, = 0.002 and Bottom: m 400,- e 0.012. In the latter setting, the confidences are extremely high given that the final images as indicated in Fig. 12 are not extremely similar to the final target images.
24

---

Figure 18: Triangular convex hull enclosed by LST outputs and source images for ImageNet with normally trained ResNet-50. These are examples from ablation A3. Top: = 100, 17 = 0.01 and Bottom: m = 400, n = 0.04.
Figure 19: Triangular convex hull enclosed by LST outputs and source images for ImageNet with normally trained ResNet-50. These are examples from ablation A4. Top: v) = 0.001 and Bottom: ? 0.05. In this setting with l 0.001, the confidences are extremely high given that the final images as indicated in Fig-[44] are not sufficiently close to the final target images.
Figure 20: Triangular convex hull enclosed by LST outputs and source images for ImageNet with normally trained ResNet-50. These are examples from ablation A5. Top: 0.0 and Bottom: == 0.02.
25


Reviews:
[{"review_id": "ectjy3ozWr", "reviewer": "Reviewer_QPKJ", "summary": "The authors aim to systematically study specific shortcomings (called blind spots) of vision models caused by their under-sensitivity.\nFor this purpose, they devise an algorithm that given an arbitrary pair of images (called source and target) can produce inputs that result in the same prediction output as the source image despite being \"perceptually\" similar to the target image (as quantified using LPIPS).\nThe authors study the geometry of the generated inputs and compare it under different training criteria (e.g. standard vs. adversarially robust models).\n", "questions": "How sensitive is your analysis to the parameters of the level set algorithm, besides delta (e.g. max iterations, scale factor and stepsize)?", "limitations": "Partially discussed ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Interesting analysis that can potentially shed light into inherent shortcomings of vision models.\n- The authors provide their source code\n\nUpdate after reubttal\n------------------------\nI would like to thank the authors for their elaborate response to the review remarks. My remarks are largely addressed in these responses, and I am hence leaning for accepting this contribution. In that case, I urge the authors to highlight the significance of their work and its utility to diagnose the sensitivity issues in adversarial training upfront. The manuscript is heavy on technical jargon that, while important, could be simplified in the main text and presented in the appendix.\nAs an additional suggestion, the following workshop at NeurIPS '23 would be a great fit to present some of the theoretical parts / foundations of this work https://www.neurreps.org/", "weaknesses": "- The contribution is rather limited. It is unclear what actionable insights we can gain using the proposed level-set analysis. There were no sufficient findings that can inform their design and training of vision models.\n- The observations with respect to the \"geometry of blind spots\" are highly anecdotal \n- The novelty is rather limited. A wide variety of methods have been devised to minimally perturb an image in order to fool the model to make a specific prediction. In the interpretability domain, various methods have meaningfully utilized minimal perturbation for the purpose of visualizing which image areas are most relevant for the input. In fact, similar perturbations are used in Integrated Gradients (Sundararajan et al. 2017), however, for a tangible application (feature attribution). See also the work by Wagner et al:\n\"Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks\" (CVPR '19)\n\nMinor: There were frequent language issues. Below are the ones I noted:\n- sensitivty\n- dimnesional\n- upto human expectations => unclear (did you mean, compared with?)\n- phenonmenon\n- atttacks\n- near-neighbour training images => nearest-neighbour images?\n- simalar\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the detailed comments, and address the specific aspects raised below:\n\n**Contributions**\n- We present a novel Level Set Traversal (LST) algorithm that iteratively uses orthogonal components of the local gradient to identify the extent and geometry of equi-confidence sets or level sets of deep networks.\n- We emphasize that the proposed LST algorithm is applicable to general vision models such as CNNs and ViTs, unlike prior works such as Jacobsen et al. [2018a] that utilize a special class of bijective neural networks called fully Invertible RevNets, to understand and study the phenomenon of under-sensitivity of classification models.\n- Given a source image, we use LST to identify inputs (informally, \"blind spots\") that are very similar perceptually to arbitrary target images from other classes, while leaving the model prediction nearly unmodified.\n- We discover for the first time that such inputs (\"blind spots\") surprisingly are linearly connected to the original source image, that is, the network retains high confidence on the convex interpolant path as well. Thus given any source image, our work uncovers a remarkable star-like substructure within the equi-confidence level sets of common models on ImageNet and CIFAR-10, which was hitherto unknown.\n- We agree with the reviewer that at present, we cannot directly use the LST algorithm without further modifications to train vision models in order to mitigate under-sensitivity as mentioned in the Limitations section (Lines 295-300), since the images produced using a smaller budget (e.g. 10 steps) look visually distinct from the target image. We hope however that future works could tackle this problem in a computationally effective manner.\n- We draw a parallel to the field of robust defenses following the discovery of adversarial examples, which required several years to identify compute-effective techniques to mitigate the over-sensitivity of deep models, and is indeed an ongoing area of research even today. However, even the present version of the LST algorithm can help practitioners better evaluate different training methods to select an \"optimal model\" prior to deployment in real-world systems, even if it cannot be directly utilized during training.\n\n**Quantitative Characterisation of Under-Sensitivity**\n- To analyze the geometric structure of level sets, we demonstrate the existence of convex linear interpolant paths within the level set between the source image and the LST output quantitatively by reporting the average confidence over this linear path in Table-2, which is only marginally lower than the source image confidence itself. Since the linear interpolant path maintains high confidence for arbitrary source-target image pairs, the source image is linearly connected to the LST outputs of different target classes in a star-like substructure within the level set.\n- We then characterize the extent of the level sets by computing statistical measures which evaluate the model confidence over the triangular convex regions enclosed by the source image and any two target blindspot images. In particular, we report the Average Triangle Confidence, and the fraction of images in the triangular region for which the model confidence is greater than $p_{src}−\\delta$, representing different thresholds.\n- Using these statistical metrics, we further demonstrate that adversarially trained models display a greater extent of under-sensitivity quantitatively in Table-2, as compared to normally trained models of the same architecture.\n- Backed by these quantitative measures, we firmly submit that the observations so made are not just “highly anecdotal”.\n\n**Novelty and Comparisons to Interpretability Works**\n- By utilizing orthogonal components of local gradients iteratively, LST is the first effective algorithm in exploring under-sensitivity towards arbitrary target images for vision models of generic architectures. Thus, LST helps uncover “maximal perturbations” that leave model confidence unchanged, in sharp contrast to the wide variety of adversarial attacks that minimally perturb samples to induce a significant change in the network output.\n- Moreover, in Section-1 of the Supplementary, we demonstrate that standard adversarial examples crucially are standalone insufficient to study the structure of level sets of common vision models, since model confidence along the path between a target benign sample ($x_2$) and adversarial examples (such as $x_1 + \\delta_{12}$ targeted towards input $x_2$) sharply declines to a valley of near zero-confidence. This contrasts sharply with the existence of linear-connectivity observed between the source image and blind-spot inputs generated using LST.\n- To uncover important parts of a source image, the interpretability paper by Wagner et al. does engage in changing the image by computing a sparse mask formulated as a deletion/preservation game, while Integrated Gradients averages gradients along the interpolant path with respect to a “reference” all-zero or black image. In contrast however, LST finds a path with respect to a valid target image of any other class, and is meant to study the overall under-sensitivity and level set substructure in input space surrounding the source image, rather than highlighting parts of the image to interpret the network prediction at that specific input-point.\n\n\n\nWe report an extensive set of quantitative and qualitative experiments for ablation analysis and robustness tests for various hyperparameters in Section-4 of the Supplementary.\n\nTypos: We thank the reviewer for pointing these out, we will amend them in future versions. We apologize for the oversight.\n\n\nWe thank the reviewer again for their valuable comments and suggestions. We kindly ask if the reviewer would consider increasing their score to support acceptance of the paper if their concerns or questions have been addressed, and would be glad to engage further during the discussion period."}, {"review_id": "wf5lxrXyo2", "reviewer": "Reviewer_LWdq", "summary": "This paper introduces the idea of level sets for image classification models. Image classification models are said to be ‘under-sensitive’ when two visually distinct images produce the same output (class). The authors propose a method to compute ‘equi-cconfidence’ level sets such that two images belonging to this set produce the same output when passed through a classifier. \n\nThe Level Set Traversal (LST) algorithm takes a source image and target image (visually distinct from the source, different class) and produces an image visually similar to the target while maintaining the output prediction of the source image. This is done by iteratively updating the source image in the direction perpendicular to the gradient. The paper then goes on to show (theoretically) the behavior or level sets in some basic ML settings (linear classifier, rely neural networks etc).\n\nAdditionally, the authors also highlight the complimentary nature of adversarial examples and level sets: Adversarial examples try to change model output while keeping human predictions the same while levels sets try to keep model output the same when human predictions differ.\nExperiments are done on ResNet-50 and DeiT on ImageNet. Empirical results show adversarially trained models exacerbate the problem of under sensitivity compared to vanilla models.", "questions": "1. How are the images inside the convex hull in Fig 4 obtained? The images on the edges are calculated via linear interpolation but I am not sure about those inside the convex hull. \n2. How do you distinguish ‘under-sensitivity’ from just poor training? Suppose you chose an architecture/set of hyperparameters which is bad fit for the dataset. Wouldn’t this setting also be ‘under-sensitive’? Is there any way to tell them apart? ", "limitations": "1. (Relatively) Large computational budget required to calculate images close to the target via LST. (Mentioned by the authors)\n2. Level Sets are a property of target class in addition to source class (unlike adversarial examples)", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The LST algorithm is the main contribution of the paper and is a novel way to calculate equity-confidence sets given a source-target pair.\n2. The paper is well written and concepts are introduced in an easy to understand fashion.\n3. Significance: Understanding what causes under-sensitivity in a model is a major open problem.\n4. I really like the explanation of the under-sensitivity of adversarial models using the LST algorithm. I think that is the major insight from this paper (apart from the algorithm itself of course).\n\nOverall, I really like the idea and the execution of the paper so I'm conflicted whether to recommend acceptance  (see my comments below). I'm looking forward to hearing from the authors and seeing other reviewers' comments.", "weaknesses": "1. I think the main weakness of the paper is the lack of analysis around the level set found for a particular source-target pair. For example, in the adversarial example setting, adversarial examples don’t need to be w.r.t. some target, they just need to change the label with a perturbation. In this paper’s case, the level set itself depends on a target label. This isn’t necessarily an issue on it’s own, however, it is difficult to understand what showing the mere existence of a level set is supposed to show about the under-sensitivity of the model. Is there something about the level set we've found that tells us something about the under-sensitivity of the model? Other questions: How does choice of target affect the level set? Is there a ’size’ for a level set? Are level sets from source -> target and target -> source symmetric?\n2. Related to presentation: I think including a figure showing images along a path from source to the final image generated by the level set algorithm would be useful to understand what exactly is within a level set. This figure could potentially replace Figure 3 which, in my opinion, does not add anything new (it shows the same thing as Figure 2 except a change in architecture)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. We are glad that they found our contributions novel, significant, and insightful. We answer the questions raised below:\n\n**Analysis and implications of level set**\n- We would like to emphasize that the level set itself is primarily a property of the source image/source class - formally, given a prediction confidence value $p \\in [0,1]$, and a class $j \\in \\lbrace 1, \\dots, N\\rbrace$ the Level Set $L_f(p,j)$ for a function/model $f$, is defined as the set of all inputs that are assigned the confidence $p$ with respect to class $j$. Thus, from a theoretical standpoint, the level set itself does not depend explicitly on the choice of the target, but empirically we explore the level set using the LST algorithm which simply starts from the source image and explores in the direction of the target image. The true level set for any particular class is thus a superset of the union of the sets found by LST for all possible target images of other classes. We use random images from random target classes to empirically show that inputs very similar to the target image lie inside the level set containing the source image. \n- We agree that showing the mere existence of the level set is not very significant in itself. We however find that the level set for common vision models is remarkably expansive — large enough to contain inputs that look near-identical to target images from other classes. Further, LST helps uncover a remarkable star-like connected geometry for the level sets, wherein the linear convex interpolant paths between any source image and LST output “blindspot” image lies within the same level/super-level set. Furthermore, we show that adversarially trained models exhibit significant under-sensitivity over subsets of the input domain that lie well beyond the original threat model used in its training. The large extent of these level sets implies that we may need to think beyond adversarial training in order to solve the under-sensitivity issue.\n\n*Intermediate images along Path:* Thank you for the suggestion. We present these intermediary images found over different iterations by LST for both normal and robust ResNet-50 models in Figure-1 of the Rebuttal pdf. \n\n*Computation of images inside Convex Hull:*  In order to visualize the extent of level sets over a two-dimensional subset, we evaluate the model confidence over the triangular convex hull obtained by linearly interpolating over three reference points, namely the source image and the two target blindspot images. For example, the image at the ‘centroid’ of the triangle formed by the source image and any target image pair is the arithmetic mean of the three images. In Figure-4, the lower left vertex in each triangle is given by a source “goose” image, and the other two vertices are given by two random target blindspot images. Apart from the high-confidence assigned along the linear paths between the source and target images (represented by two sides of the triangle), we observe that the model also assigns high confidence to a sizable fraction of the interior of the triangular convex hull as well, indicating the extent of the level set.\n\n*Analyzing the 'Size' of Level Sets:*  To quantitatively characterize the extent/size of the level/super-level sets, we compute statistical measures which evaluate the model confidence on the source class over the triangular convex regions enclosed by the source image and any two target blindspot images (as shown in Figure 4 of the Main paper). In particular, we report the Average Triangle Confidence, and the fraction of images in the triangular region for which the model confidence is greater than $p_{src}−\\delta$, representing different threshold confidences in Table-2 of the Main paper. \t\n\n*Symmetry of Level Sets:* Given a source image $x_s$ with label $y_s$ and a target image $x_t$ of class $y_t$, the level set as defined for the source image/class is mathematically distinct from that of the level set corresponding to the target image/class and are not exactly symmetric. However, by empirically running the LST algorithm with source image $x_s$ and target image $x_t$, we obtain a new input $x’$ that is perceptually similar to $x_t$ but lies within the level set of the source image $x_s$, and even the linear convex path between $x’$ and $x_s$ lie within the same level/super-level set. Symmetrically, we can rerun the LST algorithm with $x_t$ as the source while targeting $x_s$, to obtain a new image $x^”$ perceptually similar to $x_s$, though $x^”$ and the the entire linear convex path between $x^”$ and $x_t$ lie within the level set of $x_t$. Thus, though mathematically distinct, we observe similar symmetric perceptual characteristics for the two level sets empirically on common vision models.\n\n*Under-sensitivity vs Poor training:* We agree that poorly trained models will also likely display a pronounced degree of under-sensitivity. However, we can discern between these two cases by considering the network prediction of the target image itself. If the target image itself is incorrectly predicted to be in the same class as that of the source image (and say lies in its level set), then the model itself is a bad fit for these images. To control for this in our experimental evaluations, we only use target images which are correctly classified with high confidence by the network with respect to the target class. Using the LST algorithm, we show that though standard vision models may be considered a “good fit” (e.g. ResNet-50 model on ImageNet), they also simultaneously display a surprising extent of excessive under-sensitivity.\n\n\nWe thank the reviewer again for their valuable comments and suggestions. We kindly ask if the reviewer would consider increasing their score to support acceptance of the paper if their concerns or questions have been addressed. We would also be glad to engage further during the author-reviewer discussion period."}, {"review_id": "TD34SOzMeJ", "reviewer": "Reviewer_N6FN", "summary": "The authors propose a new adversarial attack on the discriminative vision models called Level Set Traversal (LST). Contrary to the previous attacks, this new algorithm exploits the orthogonal component of the network's gradient to produce samples that can bypass existing adversarially-trained classification networks.\n\nMoreover, the authors showcase that the samples obtained using this algorithm lie on the star-shaped manifold of high-confidence predictions. This is new compared to the previous adversarial attacks, which typically yield samples not connected via high-confidence linear paths.", "questions": "* Did the authors benchmark multiple state-of-the-art adversarial defense methods against their attack?\n\n* Why does the discovered high-confidence manifold have more high-confidence regions for adversarially-trained models than non-AT variants? Have the authors evaluated multiple AT methods to explore whether or not the shape of this region depends on the network architecture (ResNet vs ViT) or the specific AT method?", "limitations": "The limitations were adequately addressed.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ The presentation of the paper is great.\n\n+ The motivation behind the proposed method is clear. I also like that the paper is theoretically driven and includes experimentally confirmed results on real-world datasets.\n\n+ To me, the results look really convincing and exciting. I was surprised at the existence of the linearly interconnected sets of adversarial examples, even in adversarially trained models.\n\n+ The ablation study of the method is quite extensive and includes robustness tests for multiple hyperparameters.", "weaknesses": "- I would suggest the authors try and make connections to the area of model ensembling, where works such as [1] showcased the existence of piecewise-linear paths with low training error in the space of the model weights.\n\n- I also suggest including section 1 of the appendix in the main paper, as well as extending Figure 4 to include the visualization for multiple attack types. Without reading the appendix, the advantage of the proposed algorithm over previous methods remains unclear. Extended proof of Lemma 1, in my opinion, could be instead moved to the appendix, as it breaks the flow of the text.\n\n- The authors have only validated their approach against one type of adversarial defense for each proposed network. Extending such evaluation would benefit the community and provide a benchmark for future works.\n\n[1] Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, NeurIPS 2018", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We are encouraged that the reviewer found the paper well-motivated, clear, convincing and exciting. We respond to the questions raised below:\n\n- *Connections to model ensembling:* We thank the reviewer for the suggestion. The existence of piece-wise linear paths in model parameter space that achieve low loss (on a fixed training or testing set of images) as found by several recent works suggest that those model weights lie within the low-loss level sets and imply similar connected topologies of these sets. By altering the fixed choice of training/testing images using some well-defined perturbations/modifications, the loss function itself would change and induce different level sets in model-weight space for each such modification. We hypothesize that it might be possible to jointly consider the model parameter space and input image space as orthogonal components in a combined high-dimensional product space, wherein a meta-level linear functional can be defined to define level sets and connectivity. This potentially could help identify subsets of model weights that generalize well across different variations of the input image distribution.\n\n- We thank the reviewer for feedback on the section organization, we will certainly incorporate it in future versions.\n- *Extending Evaluations and Benchmarking:* We thank the reviewer for the suggestion to create a common benchmark to evaluate the existing models, especially amongst different adversarial defenses. Our codebase is compatible with loading adversarially trained models from the Model-Zoo offered by RobustBench [Croce et al. 2021]. We additionally present results obtained on other popular adversarial defenses such as TRADES [Zhang et al. 2019] and the current best-performing WideResNet-34-10 model on CIFAR-10 listed on RobustBench (trained with extra data by Rade et al. 2021) below:\n\nImage Metrics between LST output and Target Image:\n\n| | $\\ell_2$ dist: $\\mu \\pm \\sigma$ | $\\ell_\\infty$ dist: $\\mu \\pm \\sigma$ | SSIM: $\\mu \\pm \\sigma$ | LPIPS dist: $\\mu \\pm \\sigma$ |\n|-------|------------------------ |------------------------ |------------------------ |------------------------ |\nTrades | 3.6 $\\pm$ 2.2 | 0.467 $\\pm$ 0.137 | 0.874 $\\pm$ 0.114 | 0.033 $\\pm$ 0.03|\nRade et al. | 4.0 $\\pm$ 2.6 | 0.51 $\\pm$ 0.145 | 0.846 $\\pm$ 0.132 | 0.037 $\\pm$ 0.034|\n\n\nQuantitative measures ($\\mu \\pm \\sigma$) of model confidence invariance over the triangular convex hull ($\\Delta$) of a given source image and all possible target image pairs and over linear interpolant paths between all possible source-target image pairs:\n\n|| $~~~~~~p_{\\text{src}}$  | Avg $\\Delta$ Conf.  | Avg $\\Delta$ Frac $(\\delta=0.0)~~~$ |  $~~~~(\\delta=0.1)$ |  $~~~(\\delta=0.2)$ | $~~~(\\delta=0.3)$|  Avg Path Conf. |\n|--------------|------------------------ |------------------------|------------------------ |------------------------|------------------------ |------------------------|------------------------ |\nTrades | 0.75 $\\pm$ 0.21 | 0.69 $\\pm$ 0.21 | $~~~~$0.24 $\\pm$ 0.20 | 0.72 $\\pm$ 0.21 | 0.87 $\\pm$ 0.14 | 0.94 $\\pm$ 0.09 | 0.74 $\\pm$ 0.20|\nRade et al. | 0.75 $\\pm$ 0.18 | 0.70 $\\pm$ 0.19 | $~~~~$0.24 $\\pm$ 0.18 | 0.77 $\\pm$ 0.19 | 0.90 $\\pm$ 0.11 | 0.96 $\\pm$ 0.07 | 0.74 $\\pm$ 0.17|\n---\n\n\n>The authors have only validated their approach against one type of adversarial defense for each proposed network. Extending such evaluation would benefit the community and provide a benchmark for future works.\n>  Did the authors benchmark multiple state-of-the-art adversarial defense methods against their attack?\n> Have the authors evaluated multiple AT methods to explore whether or not the shape of this region depends on the network architecture (ResNet vs ViT) or the specific AT method?\n\n- *Shape of Level Sets:* Due to differences in architecture between ResNet and ViT networks, and differences in training methodologies, the level sets do tend to have different shapes in the two cases. For instance, the ResNet-50 AT model is more confident over a more expansive set of inputs as compared to the DeiT-S AT model, indicated by the larger fraction of the triangular convex hull of a given source image and all possible target image pairs for various confidence threshold settings as presented in Table-2. Between different adversarial training methods on the same architecture, we observe that the extent of the level set generally depends on the extent of regularization utilized, as it explicitly promotes smoothness of network outputs over a predefined input region, and that the AT models tend to be under-sensitive over subsets of the input domain that lie well beyond its original threat model.\n\n> Why does the discovered high-confidence manifold have more high-confidence regions for adversarially-trained models than non-AT variants? \n\n- During adversarial training, the model is explicitly regularized to mitigate oversensitivity to perturbations in input space within the subset of images as defined by the original threat model of interest. It is well known that adversarially trained models are smoother than normally trained models within the original threat model, given that large fluctuations in model prediction are discouraged during adversarial training. We suspect that these adversarially trained models also tend to be smoother over subsets of the input domain that lie well beyond its original threat model due to the regularization encountered during adversarial training, and thus the high confidence region is also more expansive as compared to normally trained models.\n\n\n\nWe thank the reviewer for the strong support for acceptance. We greatly appreciate the valuable comments and suggestions, and we will certainly incorporate them in the final version of the paper.\n"}, {"review_id": "0IJK3oT92q", "reviewer": "Reviewer_o1gw", "summary": "The paper studies the under-sensitivity of such deep neural networks, where it is possible to find large perturbations in the input space (such as transforming one image to another) without significant changes to the activations/predictions. Towards this goal, the paper proposes Level Set Traversal, LST, which adds perturbations in an incremental manner to move along regions orthogonal to the gradient (w.r.t input) space computed on the classification loss. Evaluations are done on ImageNet and CIFAR-10, and results show that LST is able to find not only large perturbations that continue to yield high confidence on the source class, but a path of high confidence from the source to the target. Experiments also show that the convex hull formed by a source image and a pair of LST output images for two targets enclose a region of high-confidence for adversarially trained models, unlike non-adversarially trained ones.", "questions": "What are the main failure cases of LST - under what situations does LST work best, and when is LST unable to find such paths or outputs of low loss/high confidence?", "limitations": "Limitations are discussed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The explanation and intuition of the method is very clear, and it also appears to be highly effective in finding an entire path of high confidence/similar loss from two highly visually different images.\n- Theoretical analysis presented in Sec 4 is also helpful and insightful.\n- Additional experiments on CIFAR-10 and ablation studies in the supplementary material are also appreciated.", "weaknesses": "1) The experiment details mention 1000 images from ImageNet are chosen as the source images, but only 5 target images (the ones in the figure) are used. It is difficult to tell whether the results are overfitted to the 5 target images. More extensive evaluation can be done, for example simply randomly choosing source-target pairs from the 1000 chosen images.\n\n2) I do not see any metric measuring the confidence of the output of LST alone. This metric is highly important especially for Table 1, to show that LST can both jointly optimize for minimal distance from target image as well as preserving confidence. Measuring path confidence averages the results across the entire path is not sufficient since the confidence result can be biased by points closer to the source image and hence more likely to exhibit high confidence. \n\n3) Measurement of computational cost (wall-clock time) of the LST algorithm might be useful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We are encouraged that the reviewer found that the proposed method is explained clearly and intuitively, is effective in practice and supported by theoretical insights. We respond to the questions raised below:\n\n\n>The experiment details mention 1000 images from ImageNet are chosen as the source images, but only 5 target images (the ones in the figure) are used. It is difficult to tell whether the results are overfitted to the 5 target images. More extensive evaluation can be done, for example simply randomly choosing source-target pairs from the 1000 chosen images.\n\n- *Choice of Target Classes:* For CIFAR-10, we consider all remaining 9 classes as target classes in our evaluations. For ImageNet, we pick 5 arbitrary classes for our experiments in the paper, but observe that the choice of the target images/classes does not affect our results much. To show this, we rerun our experiments twice more for a normally trained ResNet-50 model wherein we randomly sample another set of 5 classes each time. We also randomly sample two target images for each source image in the last row. We present our results in the tables below:\n\n\n| | $\\ell_2$ dist: $\\mu \\pm \\sigma$ | $\\ell_\\infty$ dist: $\\mu \\pm \\sigma$ | SSIM: $\\mu \\pm \\sigma$ | LPIPS dist: $\\mu \\pm \\sigma$ |\n|-------|------------------------ |------------------------ |------------------------ |------------------------ |\n|Paper| 2.6 $\\pm$ 1.1       | 0.033 $\\pm$ 0.011 | 0.995 $\\pm$ 0.006 | 0.003 $\\pm$ 0.004|\n|Random sampling 1| 2.8 $\\pm$ 0.9 | 0.031 $\\pm$ 0.012 | 0.996 $\\pm$ 0.004 | 0.002 $\\pm$ 0.004|\n|Random sampling 2| 2.9 $\\pm$ 1.3 | 0.055 $\\pm$ 0.031 | 0.996 $\\pm$ 0.005 | 0.002 $\\pm$ 0.003|\n| Random sampling per image| 3.5 $\\pm$ 4.9 | 0.044 $\\pm$ 0.035 | 0.988 $\\pm$ 0.029 | 0.003 $\\pm$ 0.016|\n\n\n\n| | $~~~~~~p_{\\text{src}}$  | Avg $\\Delta$ Conf.  | Avg $\\Delta$ Frac  $(\\delta=0.0)$ | $~~~(\\delta=0.1)$ |  $~~(\\delta=0.2)$ | $~~(\\delta=0.3)$|  Avg Path Conf. |\n|----|------------------------ |------------------------|------------------------ |------------------------|------------------------ |------------------------|------------------------ |\n|Paper| 0.96 $\\pm$ 0.12 | 0.49 $\\pm$ 0.11 | $~~~~~$0.13 $\\pm$ 0.12 | 0.42 $\\pm$ 0.12 | 0.46 $\\pm$ 0.12| 0.48 $\\pm$ 0.11 | 0.87 $\\pm$ 0.09 |\n|Random sampling 1| 0.95 $\\pm$ 0.14 | 0.47 $\\pm$ 0.12 | $~~~~~$0.13 $\\pm$ 0.13 | 0.41 $\\pm$ 0.13 | 0.44 $\\pm$ 0.13 | 0.46 $\\pm$ 0.13 | 0.81 $\\pm$ 0.11|\n|Random sampling 2| 0.95 $\\pm$ 0.13 | 0.42 $\\pm$ 0.11 | $~~~~~$0.11 $\\pm$ 0.12 | 0.36 $\\pm$ 0.11 | 0.39 $\\pm$ 0.12 | 0.41 $\\pm$ 0.12 | 0.81 $\\pm$ 0.11 |\n| Random sampling per image |  0.94 $\\pm$ 0.14 | 0.43 $\\pm$ 0.14 | $~~~~~$0.11 $\\pm$ 0.13 | 0.36 $\\pm$ 0.15 | 0.39 $\\pm$ 0.15 | 0.42 $\\pm$ 0.16 | 0.79 $\\pm$ 0.14|\n\n\n-  The confidence of the LST output itself generally depends on a combination of hyperparameter choices, such as the confidence threshold $\\delta$, number of steps $m$, and steps sizes for the perpendicular and parallel components to the local gradient. For example, if the step size perpendicular to the gradient is made larger, the network confidence can drop below $p_{src} - \\delta$, terminating the LST algorithm. Thus when the LST algorithm stops before the max-iterations is completed, the penultimate point that has network confidence above $p_{src} - \\delta$ is set as the LST output. However, early termination is rarely observed with an appropriate choice of hyperparameters. Thus in all cases, the LST output is guaranteed to have network confidence above $p_{src} - \\delta$, for all LST image metrics shown in Table-1. For reference, we set the confidence threshold  $\\delta=$ 0.2 for ImageNet, which means that the confidence on the LST output never drops below the confidence of the source image by more than 0.2. This can also be seen individually for all LST output images which are the off-diagonal images in Figures-2,3 (original source images are the diagonal images), indicating that the LST outputs have high confidence, generally  well-above the lower bound given by $p_{src} - \\delta$ = $p_{src}- $  0.2.\n\n* *Wall Clock Time:* We record wall clock time on a single RTXA5000 GPU with a ResNet-50 model on ImageNet, using a batch size of 100, and report mean and standard deviation ($\\mu \\pm \\sigma$) statistics over 5 independent mini-batches. \nThe proposed LST algorithm is seen to require 101.8 seconds ($\\pm$ 0.4s) for 400 iterations, and 52.0 seconds ($\\pm$0.3s) for 200 iterations. \nIn comparison, a 100-step PGD attack of the same model takes 25.1 seconds ($\\pm$1.4s). Thus, per iteration, the LST algorithm takes the same amount of time as a PGD attack iteration (both about 0.25 seconds). In total, the LST algorithm of 200/400 iterations requires 2x/4x the computation time of a standard 100-step PGD adversarial attack.\n\n- *Best and Worst cases:* Given that the LST algorithm depends primarily on the parallel and orthogonal components of the model gradient, LST works best on deep networks that do not suffer from gradient obfuscation or gradient masking issues, which encompasses nearly all commonly used vision models. On the other hand, for models with non-differentiable or randomized inference-time components, the LST algorithm would have to be modified to utilize tricks such as Backward Pass Differentiable Approximation (BPDA) and Expectation over Transformation (EoT) tricks as proposed in past adversarial attack literature [Athalye et al. 2018] that helps overcome shattered, stochastic, exploding and vanishing gradients. We also note that these modifications also often require an sizable increase in the maximum iteration count to achieve adequate convergence levels.\n\n\nWe thank the reviewer for the support for acceptance and greatly appreciate the valuable comments and suggestions. We kindly ask if the reviewer would consider increasing their score if their concerns or questions have been addressed. We would be glad to engage further during the discussion period.\n"}, {"review_id": "fqzDDOUr2X", "reviewer": "Reviewer_yXeg", "summary": "The authors explore the output space of vision models. Specifically, they propose an algorithm (Level Set Traversal, LST) which, starting from a \"source\" input image transforms it into an image that looks completely different (e.g. an image from a different class) while still confidently predicting the \"source\" class. This proves that neural networks have \"blind spots\" they further analyze the behavior of networks, and show that the paths between these inputs are connected.", "questions": "* The authors mention that doing only 10ish iterations produces much lower-quality results. I would be curious to see what these would look like (though this is just personal curiosity, so feel free to ignore this).\n\n* I don't understand what the colors within the triangles in Figure 4 represents, could you explain it?\n\n* Do the images you obtain transfer to other architectures? E.g. If we take the off-diagonal images from Fig. 2 and put them into a different network than the one used to create them, would they be classified as target or source class?", "limitations": "The authors have adequately discusses limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Originality: the paper improves upon previous work in the area, which only applied to very specific network architectures (invertible resnets). In contrast, the novel LST algorithm can be applied to virtually any network.\n\nQuality: Both the theoretical and the empirical work appear solid to me.\n\nClarity: I was not able to follow all the math in detail, but was nonetheless able to follow along nicely.\n\nSignificance: I think this paper is a solid contribution to the field. It provides both some theoretical analysis of the output space of vision models, as well as an algorithm for producing \"blind spot\" examples that is easy to understand and implement, and seems to work well.", "weaknesses": "* The authors state that the algorithm takes 200-400 iterations for each input image. To put this into context it would be nice if the authors could give a wall clock time estimate on how long this would take.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their valuable feedback. We are encouraged that the reviewer found that the proposed method is original, explained clearly, and is a significant contribution to the field. We respond to the questions raised below:\n\n> The authors state that the algorithm takes 200-400 iterations for each input image. To put this into context it would be nice if the authors could give a wall clock time estimate on how long this would take.\n- *Wall Clock Time:* We record wall clock time on a single RTXA5000 GPU with a ResNet-50 model on ImageNet, using a batchsize of 100, and report mean and standard deviation ($\\mu \\pm \\sigma$) statistics over 5 independent minibatches. \nThe proposed LST algorithm is seen to require 101.8 seconds ($\\pm$ 0.4s) for 400 iterations, and 52.0 seconds ($\\pm$0.3s) for 200 iterations. \nIn comparison, a 100-step PGD attack of the same model takes 25.1 seconds ($\\pm$1.4s). Thus, per iteration, the LST algorithm takes the same amount of time as a PGD attack iteration (both about 0.25 seconds). In total, the LST algorithm of 200/400 iterations requires 2x/4x the computation time of a standard 100-step PGD adversarial attack.\n\n- In Figure-2 of the Rebuttal pdf, we present the LST outputs as obtained with just 10 iterations. Here, we clearly observe that the LST output images (off-diagonal entries) are not adequately similar perceptually to the target image of interest, but are often rather more similar to the source image. Thus, in practice, we set the number of iterations to be between 200 and 400 to produce LST outputs very similar to the target image.\n\n> I don't understand what the colors within the triangles in Figure 4 represents, could you explain it?\n\n- In order to visualize the extent of level sets over a two-dimensional subset, we evaluate the model confidence over the triangular convex hull obtained by linearly interpolating over three reference points, namely the source image and the two target blindspot images. The prediction confidence (in the range [0,1]) assigned by the model with respect to the source class is mapped to the continuous color-bar (shown in the rightmost part of Figure 4), with high-confidence (close to 1.0) points appearing as bright yellow, and low-confidence (close to 0.0) points appearing as dark violet.   \n\n\n> Do the images you obtain transfer to other architectures? E.g. If we take the off-diagonal images from Fig. 2 and put them into a different network than the one used to create them, would they be classified as target or source class?\n- *Transferability of LST outputs:* In general, we observe little transferability between different networks, that is, the off-diagonal images generated by LST on one network are generally assigned low confidence by a different network with respect to the source class. This is likely because the LST algorithm optimizes the final output to be highly specific to the original network (especially with the choice of relatively small step sizes over several iterations), and transferability is not explicitly incorporated into its design. However, we hypothesize that transferability of LST outputs can be improved significantly by incorporating techniques proposed in a vast body of existing literature that works towards boosting transferability of standard adversarial attacks [1,2,3] and Universal Adversarial Perturbations (UAP) [4,5,6].\n\n\nWe thank the reviewer for the support for acceptance. We greatly appreciate the valuable comments and suggestions, and we will certainly incorporate them in the final version of the paper.\n\n\nReferences:\n\n[1] Xie et al.,  Improving Transferability of Adversarial Examples with Input Diversity, CVPR 2019\n\n[2] Wu et al., Boosting the Transferability of Adversarial Samples via Attention, CVPR 2020\n\n[3] Qin et al., Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbation, NeurIPS 2022\n\n[4] Dezfooli et al., Universal adversarial perturbations, CVPR 2017\n\n[5] Mopuri et al., Fast feature fool: A data independent approach to universal adversarial perturbations, BMVC 2017\n\n[6] Benz et al., Double Targeted Universal Adversarial Perturbations, ACCV 2020\n\n"}]

Author rebuttal:
**A note to all Reviewers**

We sincerely thank the reviewers for their time and valuable feedback on our work. We are glad that the reviewers appreciate the presentation and motivation of the proposed Level Set Traversal (LST) algorithm, its effectiveness in identifying the extent and geometry of equi-confidence level sets of deep networks, whereby subsequently uncovering the linear path connectivity and star-like substructure of these level sets. Furthermore, we are happy to note that the reviewers appreciate that the paper makes noteworthy contributions on the theoretical front towards the analysis of level set submanifolds, juxtaposed with the practical applicability of the LST algorithm to general vision models such as CNNs and ViTs (while prior approaches rely on special network architectures), thereby enabling novel analysis of the under-sensitivity and blind-spots of common models empirically on standard datasets such as ImageNet and CIFAR-10.

We greatly appreciate the valuable comments and suggestions, and we will certainly try to incorporate these in the final version of the paper.


Meta-review: The paper presents an analysis of under-sensitivity in vision models, specifically studying blind spots and the geometry of level sets using a Level Set Traversal (LST) algorithm. The authors investigate the phenomenon of level sets in deep neural networks and propose that despite being perceptually similar, certain inputs can maintain high-confidence predictions similar to a given source image. The authors present their findings on the expansive extent of under-sensitivity in various models, contrasting normal and adversarially trained networks. 

Reviewer QPKJ (giving a 6 rating) highlights the need for clearer motivation and problem statement. In their response, the authors address the reviewer's concerns regarding the motivation of their work, emphasizing the significance of their findings in uncovering the extent of under-sensitivity. They clarify the mathematical foundations of their approach, discussing uniqueness and extremity properties. 

Reviewer LWdq's (giving a 5 rating) concerns regarding the significance of the discovered star-like structure and the paper's practical insights are addressed in the author's rebuttal, where they emphasize the theoretical and empirical significance of their methodology and findings. The authors explain that the level set's existence is notable due to its expansive nature and its connection to under-sensitivity, as well as its potential implications for understanding model behavior beyond adversarial training. The reviewer's remaining questions are adequately addressed in the authors' response, where they provide further explanations and clarification on the methodological and theoretical aspects of the paper. The reviewer appreciates the response and increases their score to a borderline acceptance.

Reviewer N6FN (giving a 8 rating) suggests improvements for better clarity. The authors address the need for benchmarking their approach against multiple state-of-the-art adversarial defense methods and exploring the shape of level sets across different network architectures and adversarial training methods. The authors present additional evaluation results, demonstrating compatibility with existing adversarially trained models and discussing how different architectural and training differences influence level set shapes.

Reviewer o1gw (rating 7) wants more extensive evaluation beyond the chosen 5 target images and suggest considering a metric to measure LST output confidence independently. They also suggest providing computational cost details and inquire about failure cases and best scenarios for LST. In their rebuttal, the authors address concerns about overfitting, showing that the method's effectiveness extends to various ImageNet classes. They explain the impact of hyperparameters on LST output confidence and provide additional results, including computational cost analysis. They discuss limitations related to gradient obfuscation and provide insights into best and worst-case scenarios for LST's performance. The reviewer is happy with the rebuttal and has raised their score to 7.

Review yXeg (also giving a rating 7) is advocating for paper's acceptance, and rebuttal interactions reflect a positive evaluation of the paper's contributions, clarity, and significance. 

The ACs concur with the evaluations given by the reviewers and hold the view that the wide-ranging applicability of the proposed LST algorithm and the revelations it offers into neural network behavior carry significant value for the research community. This, in turn, substantiates its acceptance as a notable and impactful contribution. Therefore, the Area Chairs recommend acceptance.
Decision: Accept (spotlight)