Title: Exploring Geometry of Blind Spots in Vision models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Neural networks

Vision models

blind spots

undersensitivity

invariance

level set geometry

input connectivity
Abstract: Despite the remarkable success of deep neural networks in a myriad of settings, several works have demonstrated their overwhelming sensitivity to near-imperceptible perturbations, known as adversarial attacks. On the other hand, prior works have also observed that deep networks can be under-sensitive, wherein large-magnitude perturbations in input space do not induce appreciable changes to network activations. In this work, we study in detail the phenomenon of under-sensitivity in vision models such as CNNs and Transformers, and present techniques to study the geometry and extent of “equi-confidence” level sets of such networks. We propose a Level Set Traversal algorithm that iteratively explores regions of high confidence with respect to the input space using orthogonal components of the local gradients. Given a source image, we use this algorithm to identify inputs that lie in the same equi-confidence level set as the source image despite being perceptually similar to arbitrary images from other classes. We further observe that the source image is linearly connected by a high-confidence path to these inputs, uncovering a star-like structure for level sets of deep networks. Furthermore, we attempt to identify and estimate the extent of these connected higher-dimensional regions over which the model maintains a high degree of confidence.

Exploring Geometry of Blind Spots in Vision Models

Abstract

Despite the remarkable success of deep neural networks in a myriad of settings, several works have demonstrated their overwhelming sensitivity to near-imperceptible perturbations, known as adversarial attacks. On the other hand, prior works have also observed that deep networks can be under-sensitive, wherein large-magnitude perturbations in input space do not induce appreciable changes to network activations. In this work, we study in detail the phenomenon of under-sensitivity in vision models such as CNNs and Transformers, and present techniques to study the geometry and extent of 'equi-confidence' level sets of such networks. We propose a Level Set Traversal algorithm that iteratively explores regions of high confidence with respect to the input space using orthogonal components of the local gradients. Given a source image, we use this algorithm to identify inputs that lie in the same equi-confidence level set as the source image despite being perceptually similar to arbitrary images from other classes. We further observe that the source image is linearly connected by a high-confidence path to these inputs, uncovering a star-like structure for level sets of deep networks. Furthermore, we attempt to identify and estimate the extent of these connected higher-dimensional regions over which the model maintains a high degree of confidence. The code for this project is publicly available at this URL.

1 Introduction

Deep neural networks have demonstrated remarkable success in various diverse domains including computer vision and natural language processing, surpassing the performance of classical algorithms by a significant margin. However, though they achieve state of the art results on many computer vision tasks like image classification, sometimes even exceeding human-level performance [He et al., 2015, 2016], the overall visual processing conducted by such models can deviate significantly from that effectively observed in the human visual system. Perhaps most iconic and representative of these differences lies in the overwhelming susceptibility of neural networks to near-imperceptible changes to their inputs - commonly called adversarial attacks [Szegedy et al., 2014] - an extraordinary failure mode that highlights the over-sensitivity of such models. Indeed, an active area of research over the past few years has been focused towards analyzing adversarial perturbations under different threat models [Goodfellow et al., 2015, Tramèr et al., 2018, Wong et al., 2019, Laidlaw et al., 2021, Croce and Hein, 2022] and in addressing adversarial vulnerabilities using robust training methodologies [Madry et al., 2018, Zhang et al., 2019b, Singla and Feizi, 2020, Wu et al., 2020, Laidlaw et al., 2021, Levine and Feizi, 2021].

On the other hand, it has been shown that such models may also be under-sensitive , wherein input images that are unmistakably disparate to a human oracle induce near identical network activations or predictions [Jacobsen et al., 2018a]. To tractably analyze this phenomenon, Jacobsen et al. [2018a] utilize a special class of neural networks that are bijective functions, called fully Invertible RevNets [Jacobsen et al., 2018b, Kingma and Dhariwal, 2018], to craft large-magnitude semantic perturbations in input space that are designed to leave its corresponding network representations unchanged. Furthermore, Tramèr et al. [2020] show that robust training with ℓ p bounded adversaries can be a source of excessive model-invariance in itself, due to the poor approximation of the true imperceptible threat model of human oracles by ℓ p norm-bounded balls in RGB-pixel space [Laidlaw et al., 2021]. Indeed, the authors 'break' a provably robust defense on MNIST [Zhang et al., 2019a] with a certified accuracy of 87% , by crafting perturbations within the certified ℓ ∞ radius of 0 . 4 , that however cause model agreement with human oracles to diminish to 60% .

However, these methods either rely upon special invertible network architectures or the selection of the nearest training image of another class as a target followed by a sequence of complex alignment and spectral clustering techniques, so as to semantically alter the input in a conspicuous manner that induces a change in the human assigned oracle label, while leaving the model prediction unchanged. This leads us to our research question: Is it possible to analyze the phenomenon of under-sensitivity of general vision models in a systematic manner on natural image datasets, and characterize the geometry and extent of 'blind spots' of such networks? Indeed, we empirically demonstrate the veracity of this claim - in this work, we present a novel Level Set Traversal algorithm to explore the 'equi-confidence' level sets of popular vision models. Given an arbitrary source and target image pair, our proposed algorithm successfully finds inputs that lie in the same level set as the source image, despite being near-identical perceptually to the target image. Furthermore, the proposed algorithm identifies a connected path between the original source image and the 'blind spot" input so generated, wherein high prediction confidence with respect to the source class is maintained throughout the path. In summary, we make the following contributions in this work:

We present a novel Level Set Traversal algorithm that iteratively uses orthogonal components of the local gradient to identify the 'blind spots' of common vision models such as CNNs and ViTs on CIFAR-10 and ImageNet.

We thereby show that there exist piecewise-linear connected paths in input space between images that a human oracle would deem to be extremely disparate, though vision models retain a near-uniform level of confidence on the same path.

Furthermore, we show that the linear interpolant path between these images also remarkably lies within the same level set; as we observe the consistent presence of this phenomenon across arbitrary source-target image pairs, this unveils a star-like set substructure within these equi-confidence level sets.

We demonstrate that adversarially robust models tend to be under-sensitive over subsets of the input domain that lie well beyond its original threat model, and display level-sets of high-confidence that extend over a significant fraction of the triangular convex hull of a given source image and arbitrary pair of target images.

2 Preliminaries

Notation: In this paper, we primarily consider the setting of classification with access to a labelled dataset. Let x ∈ X denote d -dimensional input images, and let their corresponding labels be denoted as y ∈ { 1 , . . . , N } . Let f : X → [0 , 1] N denote the classification model considered, where f ( x ) = ( f 1 ( x ) , . . . , f N ( x ) ) represents the softmax predictions over the N -classes. Further, let C f ( x ) be the argmax over the N -dimensional softmax output, representing the class predicted by the model for input x . For a given data sample ( x , y ) , let the cross-entropy loss achieved by the model be denoted as CE ( f ( x ) , y ) . Given a prediction confidence value p ∈ [0 , 1] , and a class j ∈ { 1 , . . . , N } we define the Level Set L f ( p, j ) , and Superlevel Set L + f ( p, j ) for the function f as follows:

Given a pair of inputs x 1 and x 2 , we define the linear interpolant path between them as P ( λ ; x 1 , x 2 ) = λ · x 1 + (1 -λ ) · x 2 , for λ ∈ [0 , 1] . A given set S is thus said to be convex if P ( λ ; x 1 , x 2 ) ∈ S, ∀ x 1 , x 2 ∈ S and λ ∈ [0 , 1] . Further, a set S is said to be star-like if there exists some s 0 ∈ S such that P ( λ ; s 0 , x ) ∈ S, ∀ x ∈ S and λ ∈ [0 , 1] .

2.1 Conjugate Nature of Adversarial and Confidence Preserving Perturbations

̸

Given a classification model f and a correctly classified benign input ( x , y ) , an adversarial image is a specially crafted image ˜ x = x + ε such that both x and ˜ x appear near-identical to a human oracle, but induces the network to misclassify, i.e. C f ( x + ε ) = C f ( x ) = y . To enforce imperceptibility in a computationally tractable manner, several adversarial threat models have been proposed, with the ℓ 2 and ℓ ∞ norm constraint models being the most popular. Amongst the earliest adversarial attacks specific to the latter threat model was the Fast Gradient Sign Method (FGSM) attack, proposed by Goodfellow et al. [2015], wherein the adversarial perturbation is found by single-step direct ascent along the local gradient with pixel-wise clipping. A stronger multi-step variant of this attack called Iterated-FGSM (IFGSM) was later introduced by [Kurakin et al., 2017], wherein iterated gradient ascent is performed alternately with projection operations onto the constraint set. A popular variant, called the Projected Gradient Descent (PGD) attack was introduced by Madry et al. [2018] which incorporates a initial random perturbation to the clean image, which was observed to help mitigate gradient masking effects Kurakin et al. [2016]. A large class of adversarial attacks [Croce and Hein, 2020, Gowal et al., 2019, Carlini et al., 2019, Sriramanan et al., 2020] thus utilize perturbations parallel to the gradient, with appropriate projection operations to ensure constraint satisfaction.

̸

In contrast, perturbations that leave the network prediction confidence unchanged, are locally orthogonal to the gradient direction. Indeed, for any differentiable function g : R d → R , denote the level set as L g ( c ) for a given output c ∈ R . Let γ ( t ) : [0 , 1] → L g ( c ) be any differentiable curve within the level set. Then, g ( γ ( t )) = c ∀ t ∈ [0 , 1] . Thus, d dt ( g ( γ ( t ))) = 0 = ⟨∇ g ( γ ( t )) , γ ′ ( t ) ⟩ , implying that γ ′ ( t ) is orthogonal to ∇ g ( γ ( t )) , ∀ t ∈ [0 , 1] . Since this is true for any curve γ contained in the level set, we conclude that the gradient vector is always perpendicular to the level set. Furthermore, we can additionally show that the level set L g ( c ) is often a differentiable submanifold, with mild additional conditions on the gradient. Indeed, a level set L g ( c ) is said to be regular if ∇ g ( x ) = 0 ∀ x ∈ L g ( c ) .

Lemma 1. If g : R d → R is a continuously differentiable function, then each of its regular level sets is a ( d -1) dimensional submanifold of R d .

We present the proof of Lemma 1 in Section A of the Appendix. We thus observe that adversarial perturbations, largely parallel to the gradient, are locally orthogonal to confidence preserving directions which correspond to the ( d -1) dimensional tangent space of the level set. We take inspiration from this observation to develop a general framework that applies to a broad class of differentiable neural networks, as opposed to previous works [Jacobsen et al., 2018a] that require the network to be invertible to identify confidence preserving perturbations.

We also remark that adversarial attacks and level set preserving perturbations are complementary from another perspective as well, as noted by prior works: the former attempts to find inputs that change model predictions without modifying human oracle assignments, while the latter attempts to keep network predictions unchanged though human oracles would likely change their original label assignment. Thus, both classes of adversarial attacks and level set preserving perturbations induce misalignment between oracle and model predictions, and cast light onto independent means of evaluating the coherence of models towards human expectations.

3 Proposed Method: Level Set Traversal

We now describe our algorithm for traversing the level set, which we call the Level Set Traversal (LST) algorithm (Algorithm 1). We try to find a path from a source image to a target image such that all points on that path are classified by the model as the source class with high confidence. Given that these ( d -1) dimensional level set submanifolds can be potentially highly complex in their geometries, we use a discretized approximation using small yet finite step sizes to tractably explore these regions. Let the source image be x s with true label y s , and g = ∇ x CE ( f ( x ) , y s ) be the gradient of the cross entropy loss of the model ( f ) prediction with respect to x . The key idea is to get as close as possible to a target image x t from some image x by computing the projection of ∆ x = x t -x onto the orthogonal complement of g , to obtain a new image x new that leaves the model confidence unchanged. We can compute the projection of ∆ x on the orthogonal complement by subtracting the component of ∆ x parallel to g , that is, x || g = g ( ∆ x ⊤ g ∥ g ∥ 2 ) (L6). Then, using a scale factor η , the vector update to x can be expressed as ∆ x ⊥ = η (∆ x -x || g ) (L7), and therefore x new = x +∆ x ⊥ . Starting from the source image x s , we repeatedly perform this iteration to get reasonably close to the target image

C

（c-C//9)

Cnew

1: Input: Source image x s with label y , target image x t , model f , max iterations m , scale factor η , stepsize ϵ , confidence threshold δ

2: Initialize x = x s , x || = 0

3: for i = 1 to m do

4: ∆ x = x t - x

5: g = ∇ x CE ( f ( x ) , y )

6: c // = ( g · ∆ x ) / || g || 2

7: ∆ x ⊥ = η (∆ x - c // g )

8: x || = Π ∞ ( x || - ϵ g , - ϵ, ϵ )

9: x new = x +∆ x ⊥ + x ||

10: 11: return x

if f ( x s )[ j ] - f ( x new )[ j ] > δ then

12: x = x new

13: return x

Step:0

Step:10

Step:20

Step:40

Step:60

Step:80

Step: 120

Step: 160

Step:200

Step:300

Step:400

Confidence:0.997

Confidence:0.998

Confidence:0.999

Confidence:0.999

Confidence:1.0

Confidence:1.0

Confidence:1.0

Confidence:1.0

Confidence:1.0

Confidence:1.0

Confidence:1.0

while carefully ensuring that the confidence of the model prediction at any given point does not drop below a preset confidence threshold δ compared to the source image (L10).

While the above method works well with a relatively large η if the curvature of f is low, it risks a non-trivial drop in model confidence (or even a change in the model prediction) if the curvature of f at x is high enough. Therefore, after each step, we add a small perturbation vector x || in the direction of -g scaled by a factor of ϵ . This step decreases the cross entropy loss, and thus increases the confidence so as to offset any confidence drops incurred due to the addition of ∆ x ⊥ . Thus, we can maintain a higher model confidence over the path. In order to ensure that the norm of this perturbation is not arbitrarily large, we bound the ℓ ∞ norm by projecting the vector to an ℓ ∞ ball of radius ϵ . Then, the step x || can be computed as clamp ( ϵ g , -ϵ, ϵ ) , where the function clamp ( v , a, b ) is the element-wise application of the function min(max( · , a ) , b ) on all elements of v . Thus, we modify L9 so that now x new = x +∆ x ⊥ -x || . We present a pictorial schematic of Algorithm 1 in Fig 1. We further employ an exponential moving average of x || , in order to smoothen components that potentially undulate heavily during the iterative traversal. Thus, since the frequent changes in x || are smoothened out, we find that the final output images are often linearly connected to the source image with high confidence over the linear interpolations (see Fig 6).

We now apply this algorithm to explore the level sets of standard and robust ResNet-50 models. In Fig 2, we present the path traversed by the LST algorithm, wherein the model predicts the source 'goose' class with very high confidence over the entire path, though the target blindspot found by LST clearly appears as a 'dog' to human oracles. To substantiate the efficacy of the LST algorithm, we randomly select five images from five arbitrary ImageNet classes ('goose', 'Scottish Terrier', 'meerkat', 'academic gown', 'cleaver'), and compute LST blindspots for all possible source-target image pairs. We show the final images output by the LST algorithm in Fig 3 as an image-grid. If we order the five selected images, the i th row and j th column of the image-grid is the LST output obtained using the i th image as target and j th image as the source. Thus, each column represents the source image being transformed iteratively into the other target images. The confidence of the model prediction for the source class (names on top of each column) is displayed just below each image. For both the normally trained and adversarially trained model, these images are almost indistinguishable from the target while retaining high model confidence for the source class. Since adversarially robust models have

Algorithm 1 Level Set Traversal (LST)

Class Label Perceived by Humans

Class Label Perceived by Humans

ScottishTerrier

meerkat

academicgown

cleaver

0.9953

0.9999

0.9999

0.9999

1.0

Scottish Terrier

1.0

1.0

1.0

1.0

1.0

meerkat

1.0

0.9999

0.9971

0.9999

1.0

academicgown

1.0

1.0

66660

0.9991

1.0

1.0

1.0

1.0

1.0

1.0

ScottishTerrier

meerkat

academicgown

cleaver

0.8171

0.9705

0.9867

0.9298

8660

ScottishTerrier

0.9972

0.6825

0.9701

0.9534

0.9972

meerkat

0.9987

0.9479

0.9489

0.9332

0.9993

academicgown

0.9989

0.9875

0.9789

0.7371

0.9984

cleaver

0.9966

0.9742

0.9604

0.9272

0.6394

perceptually aligned gradients, we can sometimes visually notice a few traces of the source image in the final LST image; for example the 'meerkat' image in the 3rd row, 2nd column in the right side of Fig 3 has some traces of the source 'terrier' image, but differences are usually hard to perceive.

We also examine the model confidence over linear interpolations between the source image and LST outputs for all target pairs in Fig 6. Formally, consider a source image x s with label y and let x op represent the LST output when applied toward a target image x t . Denote the difference vector as ∆ v = x op -x s . Then, we observe that x s + α ∆ v is assigned high confidence with respect to class y by the model ∀ α ∈ [0 , 1] , which represents the entire linear interpolant path between x s and x op. Furthermore, we observe that the path discovered by the Level Set Traversal algorithm enjoys two key properties: (1) Uniqueness (once the target image is fixed) and (2) Extremality:

(1) Uniqueness: Since the local tangent space of the level set is ( d -1) dimensional, several independent directions are orthogonal to the local gradient, and apriori do not yield a unique path like a gradient-flow. However, once we fix our target image, we use its difference vector with respect to the current iterate (L4) and compute its projection onto the local tangent space (L7) of the level set, thereby generating a uniquely defined path .

(2) Extremality: Though this flow-based path may be non-linear, we additionally discover that the final output-point of this flow is surprisingly linearly connected with high-confidence to the source image after we apply discretized approximations in practical settings for common vision models etc. Formally, the LST output x op is linearly extremal in the sense that x s +(1+ ϵ )∆ v is rapidly assigned low-confidence by the model even for extremely small values of ϵ > 0 , where ∆ v = x op -x s .

Thus, using the LST algorithm, we find that the level sets of common models extend outwards in an expansive, connected manner to include images of arbitrary classes, which human oracles would never state as being similar. Since the linear path from any given source image to LST outputs for arbitrary target images retains high model confidence throughout, it unveils a remarkable star-like substructure for superlevel sets as shown in Fig 4, where the number of 'limbs' or linear protuberances of the star-like structure is extraordinarily large , plausibly as large as the number of images in all other classes. Furthermore, to study the size of the level sets beyond the one-dimensional interpolant paths, we analyze the two-dimensional triangular convex hull between a given source image and two LST output blindspot images, using quantitative metrics in Section 6.

Scottish Terrier

meerkat

{ E : fgoose(αx) ≥

fgoose(cs) - 8}

goose

cs

academicgown

meerkat

1.0

-->

Prediction Confidence ·

0.8

0.6

0.4

LevelSetTraversal

0.2

TargetedFeatureAttack

TargetedAttack

0.0

0.1

0.2

0.3

0.4

0.5

0.60.7

0.80.91.0

<--LinearInterpolantPathCoefficient-->

4 Disconnected Nature of Standard Adversarial Attacks

At first glance, the images output by the LST algorithm may seem similar to those produced using targeted variants of standard adversarial attacks. In this section, we explore the connectivity of high-confidence paths as induced by standard adversarial attacks, and show that adversarially attacked source images are not linearly connected with high-confidence paths to their target image. In detail, let ( x 1 , y 1 ) and ( x 2 , y 2 ) be any two data samples from different classes ( y 1 = y 2 ) . A targeted adversarial attack [Carlini and Wagner, 2017] on input x 1 with respect to class y 2 is formulated as solving ε 12 = argmin ε 12 : || ε 12 ||≤ ϵ CE ( f ( x 1 + ε 12 ) , y 2 ) . A related variant, called a feature-level targeted adversarial attack [Sabour et al., 2015] instead uses intermediate network activations to craft image perturbations. If f | L ( x ) represents the network activations from hidden layer L for an input x , then this attack attempts to match features by optimizing ε 12 = arg min ε 12 : || ε 12 ||≤ ϵ || f | L ( x 1 + ε 12 ) -f | L ( x 2 ) || . The hidden layer ( L ) that is often selected corresponds to pre-activations of the final fully-connected layer, and thus often induces misclassification.

̸

Using this framework, we can thus analyze the connectivity of high-confidence paths with respect to class y 2 between the target benign sample ( x 2 ) and adversarial examples (such as x 1 + ε 12 targeted towards x 2 ), using the linear interpolant path between the two. By doing so, we can analyze the level set with respect to class y 2 using targeted adversarial examples alone, though the perturbation utilized is inherently norm-limited. In Fig 5, we plot the median confidence of a normally trained ResNet-50 model over the linear interpolant paths for 1000 source-target pairs. We find that though the model confidence with respect to class y 2 is high for both the target benign input x 2 and the targeted adversarially attacked input x 1 + ε 12 (that is, at the end-points of the linear path), the model does not maintain high confidence over the linear interpolant path between the two, but sharply declines to a valley of near zero-confidence over the path. For specific inputs, we often observe sharp, erratic diminutions in target-class prediction confidence over the convex combination, with a non-trivial region of near-zero confidence. This contrasts sharply with the existence of linear-connectivity observed between a source image and LST blindspot images. We thus crucially note that adversarial attacks are standalone insufficient to study the structure of level sets of common vision models. However, we incorporate them into our proposed Level Set Traversal algorithm in a fruitful manner.

5 Theoretical Analysis for Common Models

To better understand the geometry of level set submanifolds for a given prediction confidence threshold, we analyze the nature of confidence preserving perturbations in a few simplified model settings.

1) Linear Functional: First, consider the classification model to be a linear functional f : R d → R , that is, f ( x 1 + x 2 ) = f ( x 1 ) + f ( x 2 ) and f ( λ x ) = λf ( x ) . Then, by the Riesz Representation theorem, there exists a unique vector w f ∈ R d such that f ( x ) = ⟨ w f , x ⟩ ∀ x ∈ R d . We thus observe that for any vector v orthogonal to w f , f ( x + v ) = ⟨ w f , x + v ⟩ = ⟨ w f , x ⟩ + ⟨ w f , v ⟩ = ⟨ w f , x ⟩ . Thus, in this setting, the level sets of f are ( d -1) dimensional linear subspaces spanned by the set { v ∈ R d : ⟨ w , v ⟩ = 0 } . We observe that a similar argument can be extended to affine functions of the form f ( x ) = ⟨ w f , x ⟩ + c , where c ∈ R .

We remark that by applying a first-order Taylor series approximation for general real-valued smooth functions, we observe near-affine behavior locally within a small neighborhood: f ( x + ϵ a ) = f ( x )+ ϵ ⟨∇ f ( x ) , a ⟩ + O ( ϵ 2 ∥ a ∥ 2 ) . Thus locally, we observe that confidence-preserving perturbations can arise from a ( d -1) dimensional plane orthogonal to the gradient. Indeed we incorporate this implicitly in our proposed Level Set Traversal algorithm, wherein the orthogonal projection ∆ x ⊥ with respect to the local gradient is iteratively computed, and its relative success can partly be attributed to the orthogonal hyperplane having a large dimension, namely ( d -1) .

Another related setting worthy of note is that of neural networks that utilize ReLU activations, which induces a piece-wise linear structure over tessellated subsets of the input domain. Thus, a given output neuron of a ReLU network functionally has the form f ( x ) = ⟨ w , x ⟩ + c within a given tessellated region, and thus has a constant gradient within the same region. Further, between two such adjacent regions, the two orthogonal ( d -1) dimensional hyperplanes typically intersect over an affine space over dimension at least ( d -2) . Thus if x 1 , x 2 are two inputs such that their linear interpolant path cuts across n distinct tessellated regions, the common intersection of these orthogonal hyperplanes will typically be ( d -n ) dimensional, indicating that there exist perturbations which lie in the common null-space of the gradients as defined along each tessellated region that is crossed. Indeed in the following section, we demonstrate empirically for Residual Networks that though the exact iterative path followed by the Level Set Traversal algorithm is discretized and non-linear, the final outputs so found are often linearly connected through paths of high confidence to the source image, thereby lying within the level set of the original source class. This indicates that the overlap of different ( d -1) dimensional hyperplanes is non-trivial at a non-local scale in image space, whereby we observe extended connected regions of high confidence.

2) Full-Rank Linear Transformations: Let us now consider a setting apart from classification, such as regression, wherein the complete vector representation of the output is of principal interest. Let the model be of the form f : R d → R d , f ( x ) = A x , where A ∈ R d × d is a full-rank matrix. In this setting, we observe that if f ( x 1 ) = f ( x 2 ) = A x 1 = A x 2 , implying that A ( x 1 -x 2 ) = 0 vector. But since A is full-rank, this implies that x 1 -x 2 = 0 , i.e, x 1 = x 2 . Thus, f is a bijective function and has a trivial null space. Thus in this setting, we necessarily have to relax the problem minimally

̸

, the zeroto be that of identifying perturbations of large magnitude to the input that change the function output: that is, we solve for min v = 0 ∥ A v ∥ 2 ∥ v ∥ 2 . Indeed, let the Singular Value Decomposition (SVD) of the full rank matrix A be given by A = U Σ V T , where U, V are d × d orthogonal matrices, and Σ is a diagonal matrix consisting of the positive singular values σ i for 1 ≤ i ≤ d , in descending order without loss of generality. Then, ∥ A v ∥ = ∥ U Σ V T v ∥ = ∥ U (Σ V T v ) ∥ = ∥ Σ V T v ∥ since U is an orthogonal matrix. Similarly, since V is orthogonal as well, let z = V T v , so that ∥ z ∥ = ∥ v ∥ . Thus, min v = 0 ∥ A v ∥ 2 ∥ v ∥ 2 = min z =0 ∥ Σ z ∥ 2 ∥ z ∥ 2 = min z ∥ Σ z ∥ 2 such that z T z = 1 . But since Σ is diagonal, d

̸

̸

min z ∥ Σ z ∥ 2 = ∑ i =1 σ 2 i z 2 i , under the constraint that || z || 2 = 1 . It is then easy to observe that the minimum value attained is σ 2 d = σ 2 min , and is attained when the input vector to f is the right-singular vector corresponding to the minimum singular value of A .

̸

In this setting, we remark that adversarial perturbations, in contrast, can be formulated in this setting as max v = 0 ∥ A v ∥ 2 ∥ v ∥ 2 , with the maximum value given by σ 2 max and attained by the right-singular vector corresponding to the maximum singular value of A , highlighting the complementary nature of the two problems as indicated previously in Section 2.1. Indeed, the condition number κ of an invertible matrix A is defined as κ = σ max /σ min = ∥ A ∥ · ∥ A -1 ∥ . If condition number is large with κ ≫ 1 , the matrix A is said to be ill-conditioned, and induces an inescapable compromise: if σ max = ∥ A ∥ ≈ 1 (as potentially expected in "robust" networks), then 1 /σ min = ∥ A -1 ∥ ≫ 1 is necessarily large, thereby inducing extreme under-sensitivty along some dimensions; while if 1 /σ min = ∥ A -1 ∥ ≈ 1 , then σ max = ∥ A ∥ ≫ 1 and the model is extremely over-sensitive , similar to the phenomenon of adversarial vulnerability.

6 Quantifying Under-Sensitivity of Vision Models

In this paper, we primarily consider standard vision datasets such as ImageNet [Deng et al., 2009] and CIFAR-10 [Krizhevsky et al., 2009] (latter in Section C of the Appendix). We thereby explore the 'blind-spots' of popular vision models such ResNet [He et al., 2016] and Vision Transformers [Dosovitskiy et al., 2020, Touvron et al., 2021]. Furthermore, we explore the connectivity of such level sets on normally trained variants of such networks, as well as adversarially robust counterparts. For the latter, we analyze robust models trained adversarially against (4 / 255) ℓ ∞ constrained adversaries, available on RobustBench [Croce et al., 2021]. Specifically, we utilize a robust ResNet-50 model from Salman et al. [2020] and a robust DeiT model from Singh et al. [2023]. We also fix the hyperparameters of LST for all models for a fair comparison (detailed in Section E of the Appendix).

Image Quality Metrics: We now proceed to quantitatively verify our observations in Section 3. First, to help quantify the deviation between target images and blindspot outputs generated by our proposed Level Set Traversal algorithm, we utilize a combination of classical image metrics such as RMSE, ℓ ∞ and Structural Similarity Index (SSIM), and perceptual measures such as LPIPS distance using AlexNet [Zhang et al., 2018]. For SSIM, a higher value indicates a closer match, while for all other metrics, a lower value indicates a closer match. To calculate these metrics, we sample around 1000 source images from ImageNet, and select five other random target images of different classes for each source image. We present the image quality metrics for blindspots discovered by LST in Table 1. Here the standard deviation is over the different randomly chosen source and target images.

Metrics for Model Confidence: To evaluate the extent of the model invariance over the regions between the LST outputs and the source image, we evaluate the model confidence with respect to the source class over one-dimensional linear interpolant paths and over two-dimensional subspaces as well. For the latter, we evaluate the model confidence over the triangular convex hull obtained by linear interpolation over three reference points, namely the source image and the two target blindspot images produced using LST. For example, the input at the 'centroid' of the triangle formed by a source image and pair of target blindspots is the arithmetic mean of the three images. We visualize these in Fig 6, wherein the prediction confidence (in the range [0 , 1] ) assigned by the model with respect to the source class is mapped to a continuous colorbar, with high-confidence points (close to 1.0) appearing as bright yellow, and low-confidence points (close to 0.0) appearing as dark violet. Specifically, we use the following metrics: (a) Average Triangle ( ∆ ) Confidence : the mean of the model's source class confidence over the enclosed triangle, (b) Average Triangle ( ∆ ) Fraction for various values of δ : the fraction of inputs in the triangular region for which the model confidence is greater than p src -δ , averaged over all possible target blindspot pairs, where p src is the confidence of the source image, (c) Average Path Confidence : the average model confidence over all linear paths from the source image to all LST blindspot images. The higher these metrics, the more confident, and thus invariant, the model is in this region. For computing these metrics, we use linear interpolations between the source images and the 5 LST outputs found previously for computing the distance metrics. We thus use ( 5 2 ) = 10 triangles for each source image, and sample this triangular area in an equispaced manner to obtain 66 images for computation of the triangle ( ∆ ) metrics. We present these metrics in Table 2, along with the mean (and standard deviation) of model confidence on the source class images ( p src ) for reference. Here, the standard deviation is over the different randomly chosen source images.

We can now quantitatively confirm many of the trends we qualitatively observed in Section 3. In Table 1, we observe that the LST outputs found for the models are closer to the targets as compared to the adversarially trained models. The difference is particularly stark when comparing the ℓ ∞ distances, as ℓ ∞ is a particularly sensitive metric and is also the threat model against which the models were trained. However, for other distance metrics like RMSE or LPIPS, the difference between normally trained and adversarially trained ResNet-50 is not as high. In particular, LPIPS

distances for both models are low, which indirectly implies that the perceived difference between the images for humans oracles is relatively low. This can also be confirmed by visually inspecting the images in Fig 3. For the normally trained DeiT-S, the distance metrics are very similar to that of ResNet-50, with slightly higher RMSE and LPIPS distances. However, the adversarially trained (AT) variant is significantly less under-sensitive compared to both its normal counterpart and the ResNet-50 (AT). Specifically, the LPIPS distance metric is much greater for DeiT-S (AT), which implies there exist significant human-perceptible differences in the image. We can confirm this in Fig 7, where the LST output for the adversarially trained DeiT-S model contains visible traces of the source image. However, the LST outputs are still clearly much closer to the target class as compared to the source, which indicates that there are still some significant blind spots in DeiT-S (AT).

However, when we measure the extent of model invariance over the convex regions enclosed between the LST output and the target images, we find that adversarially trained ResNet-50 are overall more invariant (or under-sensitive) as compared to the normally trained variant. For example, the average triangle confidence for ResNet-50 (AT) is higher than that of normally trained ResNet-50, even though its source confidence is much lower. We also find that a much larger fraction of the triangular convex hull lies within the superlevel set for p src -δ for ResNet-50 (AT) as compared to normal ResNet-50 for all values of δ . The average path confidence is much closer to p src for ResNet-50 (AT) as compared to normally trained ResNet-50. This quantitatively verifies the observation made in Fig 6 that adversarial training demonstrably exacerbates under-sensitivity. Interestingly, these robust models are under-sensitive over subsets of the input domain that lie well beyond the original threat model used in its training. Moreover, between the normally trained DeiT-S and ResNet-50 models, the former appears to be more invariant with greater average confidence over the triangular convex hulls, despite having lower source image confidences p src. For the robust variant of DeiT-S however, the trend is less apparent due to the significantly lower average source image confidences p src. However the average relative ∆ fraction becomes higher for larger values of δ (such as 0 . 3 ), indicating that the superlevel sets are indeed expansive, albeit for lower confidence thresholds.

7 Discussion

While the existence of level sets alone is not very significant in itself, using the proposed LST algorithm, we find that the level set for common vision models is remarkably expansive - large enough to contain inputs that look near-identical to arbitrary target images from other classes. Since the linear path from any given source image to LST blindspot outputs retain high model confidence throughout, the level sets have a star-like connected substructure, where the number of 'limbs' or linear protuberances of the star-like structure is extraordinarily large , plausibly as large as the number of images in all other classes. This is considerably noteworthy since it indicates the hitherto unknown and unappreciated scale and extent of under-sensitivity in common vision models. Moreover this hints at the potential degree of difficulty towards adequately mitigating this phenomenon in practical settings. For instance, if the level set for images of class y 1 contained sizable protuberances towards only one other class y 2 alone, the problem could perhaps be tackled by introducing a contrastive training objective that encourages the network to better discriminate between y 1 -y 2 image pairs by utilizing a denser sampling of related image augmentations, likely resulting in the diminution of these specific 'directed' protuberances (assuming reasonable train-test generalization). But since the star-like connected substructure uncovered by LST implies that such protuberances exist towards any generic image of any other class, such simple approaches will likely be ineffective and possibly computationally infeasible from a combinatorial perspective. Thus, based on the observations uncovered with LST, we hypothesize that addressing the pervasive issue of under-sensitivity in conventional vision models might present a significantly non-trivial challenge.

8 Related Work

The phenomenon of under-sensitivity in classification models was first pointed out by Jacobsen et al. [2018a], wherein they utilize a class of invertible neural networks called fully Invertible RevNets [Jacobsen et al., 2018b, Kingma and Dhariwal, 2018] to specifically craft input images that do not affect network activations at a given layer. In contrast, our proposed algorithm is applicable to general network architectures since we solely utilize input-gradients to perform the traversal over image space. Tramèr et al. [2020] further demonstrated that due to the misalignment of ℓ p norm bounded balls and the ideal set of human-imperceptible perturbations, networks that are adversarially trained against such ℓ p bounded perturbations of relatively large radius are overly-smooth, and become excessively susceptible to invariance-based attacks within the same ℓ p radius. To find such images that lie within a given ℓ p threat model, but induce human oracles to change their label assignment, the authors propose to identify the training image from another class closest in image space and apply a series of semantic-preserving transformations, and additionally use techniques such as realignment and spectral clustering. Given that these operations are fairly complex, the attack algorithm is slow, with alignment alone requiring an amortized time of minutes per input example. In contrast, our proposed method relies upon gradient-backpropagation steps which are efficiently parallelized across a minibatch of input images. Furthermore, our technique is seen to be successful for arbitrary source-target image pairs, since we do not utilize near-neighbour training images as the target. On the theoretical front, Jayaram et al. [2020] analyze the problem of span-recovery of neural networks given only oracle access to the network predictions. They characterize the feasibility of span recovery, and thus approximations of the null space of networks in a provable setting, but remark that its success in practical settings is potentially limited to networks that are extremely thin.

9 Conclusions

In this work, we investigate the phenomenon of under-sensitivity of classification models, wherein large-magnitude semantic perturbations leave network activations unchanged. To identify such 'blind spots' that occur within high-confidence level sets of common vision models, we develop a novel Level Set Traversal algorithm that iteratively perturbs a given source image to appear visually similar to an arbitrary target image by utilizing orthogonal projections with respect to the local input gradient. The proposed method is applicable to general neural networks, and helps uncover a star-like substructure for the level and superlevel sets of CNNs and ViTs on common datasets. We further observe that adversarially trained models retain a high-degree of confidence over regions that lie far beyond its original threat model, with super-level sets that extend over a significant fraction of the triangular convex hull between a given source image and arbitrary pair of blindspot images.

References

Athalye, N. Carlini, and D. Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. In International Conference on Machine Learning (ICML) , 2018.

Carlini and D. Wagner. Towards evaluating the robustness of neural networks. In IEEE Symposium on Security and Privacy (SP) . IEEE, 2017.

Carlini, A. Athalye, N. Papernot, W. Brendel, J. Rauber, D. Tsipras, I. Goodfellow, and A. Madry. On evaluating adversarial robustness. arXiv preprint arXiv:1902.06705 , 2019.

Croce and M. Hein. Minimally distorted adversarial examples with a fast adaptive boundary attack. In International Conference on Machine Learning (ICML) , 2020.

Croce and M. Hein. Adversarial robustness against multiple and single l p -threat models via quick fine-tuning of robust classifiers. In K. Chaudhuri, S. Jegelka, L. Song, C. Szepesvari, G. Niu, and S. Sabato, editors, Proceedings of the 39th International Conference on Machine Learning , volume 162 of Proceedings of Machine Learning Research , pages 4436-4454. PMLR, 17-23 Jul 2022. URL https://proceedings.mlr.press/v162/croce22b.html .

Croce, M. Andriushchenko, V. Sehwag, E. Debenedetti, N. Flammarion, M. Chiang, P. Mittal, and M. Hein. Robustbench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2021. URL https://openreview.net/forum?id=SSKZPJCt7B .

Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009. doi: 10.1109/CVPR.2009.5206848.

Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

J. Goodfellow, J. Shlens, and C. Szegedy. Explaining and harnessing adversarial examples, 2015.

Gowal, J. Uesato, C. Qin, P.-S. Huang, T. Mann, and P. Kohli. An alternative surrogate loss for pgd-based adversarial testing. arXiv preprint arXiv:1910.09338 , 2019.

He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. arXiv preprint arXiv: Arxiv-1502.01852 , 2015.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2016.

Jacobsen, J. Behrmann, R. S. Zemel, and M. Bethge. Excessive invariance causes adversarial vulnerability. CoRR , abs/1811.00401, 2018a. URL http://arxiv.org/abs/1811.00401 .

J.-H. Jacobsen, A. Smeulders, and E. Oyallon. i-revnet: Deep invertible networks. arXiv preprint arXiv:1802.07088 , 2018b.

Jayaram, D. P. Woodruff, and Q. Zhang. Span recovery for deep neural networks with applications to input obfuscation, 2020.

P. Kingma and P. Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems , 31, 2018.

Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.

Kurakin, I. Goodfellow, and S. Bengio. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 , 2016.

Kurakin, I. Goodfellow, and S. Bengio. Adversarial machine learning at scale. In International Conference on Learning Representations (ICLR) , 2017.

Laidlaw, S. Singla, and S. Feizi. Perceptual adversarial robustness: Defense against unseen threat models. In ICLR , 2021.

J. Levine and S. Feizi. Improved, deterministic smoothing for l_1 certified robustness. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 6254-6264. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/levine21a.html .

Madry, A. Makelov, L. Schmidt, T. Dimitris, and A. Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations (ICLR) , 2018.

Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32 , pages 8024-8035. Curran Associates, Inc., 2019. URL http://papers.neurips.cc/paper/ 9015-pytorch-an-imperative-style-high-performance-deep-learning-library. pdf .

Rade and S.-M. Moosavi-Dezfooli. Reducing excessive margin to achieve a better accuracy vs. robustness trade-off. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=Azh9QBQ4tR7 .

Sabour, Y. Cao, F. Faghri, and D. J. Fleet. Adversarial manipulation of deep representations. arXiv preprint arXiv:1511.05122 , 2015.

Salman, A. Ilyas, L. Engstrom, A. Kapoor, and A. Madry. Do adversarially robust imagenet models transfer better? CoRR , abs/2007.08489, 2020. URL https://arxiv.org/abs/2007.08489 .

D. Singh, F. Croce, and M. Hein. Revisiting adversarial training for imagenet: Architectures, training and generalization across threat models. arXiv preprint arXiv:2303.01870 , 2023.

Singla and S. Feizi. Second-order provable defenses against adversarial attacks. In H. D. III and A. Singh, editors, Proceedings of the 37th International Conference on Machine Learning , volume 119 of Proceedings of Machine Learning Research , pages 8981-8991. PMLR, 13-18 Jul 2020. URL https://proceedings.mlr.press/v119/singla20a.html .

Sriramanan, S. Addepalli, A. Baburaj, and R. Venkatesh Babu. Guided Adversarial Attack for Evaluating and Enhancing Adversarial Defenses. In Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus. Intriguing properties of neural networks, 2014.

Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In International conference on machine learning , pages 10347-10357. PMLR, 2021.

Tramèr, A. Kurakin, N. Papernot, I. Goodfellow, D. Boneh, and P. McDaniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations (ICLR) , 2018.

Tramèr, J. Behrmann, N. Carlini, N. Papernot, and J.-H. Jacobsen. Fundamental tradeoffs between invariance and sensitivity to adversarial perturbations, 2020.

Wong, F. Schmidt, and Z. Kolter. Wasserstein adversarial examples via projected sinkhorn iterations. In International Conference on Machine Learning , pages 6808-6817. PMLR, 2019.

Wu, S.-T. Xia, and Y. Wang. Adversarial weight perturbation helps robust generalization. Advances in Neural Information Processing Systems , 33, 2020.

Zhang, H. Chen, C. Xiao, S. Gowal, R. Stanforth, B. Li, D. Boning, and C.-J. Hsieh. Towards stable and efficient training of verifiably robust neural networks. arXiv preprint arXiv:1906.06316 , 2019a.

Zhang, Y. Yu, J. Jiao, E. Xing, L. El Ghaoui, and M. I. Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning (ICML) , 2019b.

Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric, 2018.

̸

Appendix

A Theoretical Characterization of Level Sets

Let g : R d → R be a differentiable function. For c ∈ R , the level set L g ( c ) is said to be regular if the gradient of g is non-vanishing for all points in the level set, that is, ∇ g ( x ) = 0 ∀ x ∈ L g ( c ) . We can then show that the regular level sets of continuously differentiable functions are ( d -1) dimensional submanifolds of R d (Lemma 1), using the Implicit Function Theorem. We restate the lemma here for the readers' convenience, and present its proof below.

̸

Lemma 1. If g : R d → R is a continuously differentiable function, then each of its regular level sets is a ( d -1) dimensional submanifold of R d .

̸

Proof. Without loss of generality, let L g (0) be a regular level set of g , and let a ∈ L g (0) (if c = 0 , we can define a new function shifted by the constant -c , so it suffices to consider L g (0) ). By definition of regularity, we know that ∇ g ( a ) = 0 , and at least one of its components must be non-zero. By renumbering the coordinates, without loss of generality, we can assume the first vector component is non-zero, i.e. ∇ 1 g ( a ) = ∂ ∂x 1 g ( a ) = 0 . Thus we have a function g such that g : R × R d -1 → R with

̸

Since the 1 × 1 Jacobian submatrix [ ∂ ∂x 1 g ( a ) ] is non-zero and thus invertible, we can thus apply the Implicit Function Theorem to g at a , which states that there exist open sets U, V such that U ⊂ R with a 1 ∈ U and V ⊂ R d -1 with ( a 2 , . . . , a d ) ∈ V , and an "implicit" continuously differentiable function t : V → U such that t ( a 2 , . . . , a d ) = a 1 with the property that for all ( x 2 , . . . , x d ) ∈ V , we have:

Further, we observe that the set W = U × V ⊂ R d is open since U, V are open. Now, if x ∈ L g (0) ∩ W , x 1 = t ( x 2 , . . . , x d ) . Thus, consider ϕ a : L g (0) ∩ W → R d with

which is a map with its first coordinate identically zero. Thus, if Π -1 ( x 1 , x 2 , . . . , x d ) = ( x 2 , . . . , x d ) is the projection map from R d to R d -1 that omits the first coordinate, Π -1 ◦ ϕ a is a continuously differentiable chart from L g (0) ∩ W to R d -1 containing a . Since this is true pointwise for each a ∈ L g (0) , L g (0) is a ( d -1) dimensional submanifold of R d .

B Implementation Details and Methodology

B.1 Details on Datasets and Pretrained Models

In this paper, we present results on vision datasets such as ImageNet [Deng et al., 2009] and CIFAR10 [Krizhevsky et al., 2009], given that they have come to serve as benchmark datasets in the field. ImageNet Large Scale Visual Recognition Challenge (ILSVRC) consists of over a million labelled RGB color images arising from a wide variety of objects, totalling to 1000 distinct classes. The images themselves are of relatively high resolution, with common vision models utilizing an input shape of 224 × 224 pixels. On the other hand, CIFAR-10 is a smaller dataset, consisting of 32 × 32 pixel RGB images from ten classes: "airplane", "automobile", "bird", "cat", "deer", "dog", "frog", "horse", "ship" and "truck". In this paper, all training and experimental evaluations were performed using Pytorch [Paszke et al., 2019].

The ℓ ∞ norm constrained threat model is the most commonly used setting for adversarial robustness in both these datasets. On ImageNet, we analyse robust models trained with adversaries constrained within a radius of 4 / 255 , while on CIFAR-10, we consider robust models trained with adversaries over a larger radius of 8 / 255 . We use adversarially trained models as made available on RobustBench [Croce et al., 2021] to analyze popular vision models. For the class of Residual Networks [He et al., 2016], we analyze a normally trained ResNet-50 model from the Pytorch Image Models (timm) repository, and the adversarially robust ResNet-50 model from Salman et al. [2020]. For the class of Vision transformers [Dosovitskiy et al., 2020, Touvron et al., 2021], we utilize a robust DeiT model from Singh et al. [2023]. For CIFAR-10, we utilize adversarially trained WideResNet-34-10 models

̸

Scottish Terrier

meerkat

academic gown

cleaver

goose

ScottishTerrier

meerkat

academic gown

cleaver

0.7856

0.9958

0.9964

0.9938

8660

0.7556

0.7214

0.7061

0.748

0.8586

Scottish Terrier

Scottish Terrier

0.9913

0.837

0.9932

0.9947

0.9965

0.8336

0.8213

0.797

0.7199

0.9145

meerkat

0.9947

0.9953

0.7848

0.9943

0.9963

0.8232

0.778

0.6734

0.7277

0.8627

academic gown

UMOb

0.9761

0.9834

0.986

0.8003

0.9951

0.8127

0.8751

0.7133

0.687

0.8446

cleaver

99660

0.7088

0.727

0.9949

0.9914

0.9868

0.8017

0.7593

0.8085

0.7415

from Wu et al. [2020], Zhang et al. [2019b] and Rade and Moosavi-Dezfooli [2022], alongside a normally trained counterpart.

B.2 Algorithm and Hyperparameter Details

ImageNet : In the main paper, we fix the parameters of the LST algorithm for all the visualizations (Fig 3,6,7 and Tables 1,2 in the Main paper). The scale factor for the step perpendicular to the gradient, or η , is 10 -2 . The stepsize for the perturbation parallel to the gradient -∇ CE ( f ( x ) , y ) , or ϵ , is 2 × 10 -3 . The confidence threshold ( δ ) is 0 . 2 , which means that the confidence never drops below the confidence of the source image by more than 0 . 2 . In practice, we rarely observe such significant drops in the confidence during the level set traversal. The algorithm is run for m = 400 iterations. The number of iterations required is relatively high compared to standard adversarial attacks because of the much larger distance between the source and the target images. We also present results with different parameters in the subsequent sections.

CIFAR-10 : We fix the parameters of LST when applied on CIFAR-10 in a similar manner. The scale factor for the step perpendicular to the gradient, or η , is 1 × 10 -2 . The stepsize for the perturbation parallel to the gradient -∇ CE ( f ( x ) , y ) , or ϵ , is 2 × 10 -3 . The confidence threshold ( δ ) is 0 . 25 , and we run the algorithm for m = 300 iterations to obtain high fidelity output images. The adversarially trained model is observed to have low prediction confidence even on the source images, and thus again requires a large number of iterations for good convergence.

C Results on CIFAR-10

In this section, we present qualitative and quantitative results obtained on the CIFAR-10 dataset. Similar to Section 6 of the Main paper, we consider RMSE, ℓ ∞ , the Structural Similarity Index (SSIM), and LPIPS with AlexNet [Zhang et al., 2018]. For SSIM, a higher value indicates a closer match, while for all other metrics, a lower value indicates a closer match. To calculate these metrics, we sample at 1000 images from CIFAR-10 to serve as the source images, and consider 9 fixed images from all the remaining 9 classes as targets as shown in Figure-8. By visual inspection, we see that any source image can be iteratively modified by LST to resemble any given target image, while retaining high confidence on the original class as indicated in each column of the figure. Indeed, we quantify this in Table-3 where we observe that the final output images of the LST traversal algorithm are close

airplane

automobile

bird

Cat

deer

bop

horse

ship

1.0

1.0

86660

1.0

LU

1.0

1.0

0T

1.0

L

1.0

to the target over these different metrics; in particular the high SSIM similarity and small LPIPS distance implies high visual similarity for human oracles.

Similar to the Main paper, we again evaluate the extent of the model invariance over the regions between the LST outputs and the source image, and compute various metrics which evaluate the model's confidence in the source class on all triangular convex hulls enclosed by the source image and any two target images. We observe that the extent of convexity is greatly pronounced on CIFAR-10, with high confidence maintained over a significant fraction of the convex hull. Indeed, to quantify these observations we use the following metrics: (a) Average Triangle ( ∆ ) Confidence : the mean of the model's source class confidence over the enclosed triangle, (b) Average Triangle ( ∆ ) Fraction for various values of δ : the fraction of images in the triangular region for which the model confidence is greater than p src -δ , averaged over all possible target pairs. Here p src is the confidence of the source image, (c) Average Path Confidence : the average model confidence overall linear paths from the source image to all target images. The higher these metrics, the more confident, and thus invariant, the model is in this region. For all of these metrics, we use ten linear interpolations between the source and each of the LST output images and around 65 images that cover the triangular region. We present these metrics in Table 4, along with the mean (and standard deviation of) confidence of the model on the source class ( p src) for reference. We thus observe that the average confidence over the triangular convex hull is extremely high on average, and that the super-level sets corresponding to δ = 0 , 0 . 1 , 0 . 2 , 0 . 3 cover a significant fraction of the triangular hull. Further, we observe that the linear interpolant paths maintain a very high degree of confidence as well.

airplane

automobile

ship

1.0

L0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

1.0

L.C

1.0

1.0

1.0

66660

D Limitations and Computational Complexity

We note that the proposed Level Set Traversal algorithm typically requires around 200 - 400 iterations to find images which are near-identical to the target image while still maintaining high confidence for the source class since both the semantic and pixel level changes required are of relatively large magnitude. If restricted to a much smaller budget such as 10 steps, we observe that the final image is visually distinct from the target image. Thus, it is non-trivial to design experiments which explicitly train new networks against outputs found by our proposed LST algorithm, akin to that performed

within standard adversarial training [Madry et al., 2018]. Furthermore, our quantitative experiments could be potentially extended over higher-dimensional simplices as well, if the curse of dimensionality could be addressed effectively.

D.1 Best and Worst case scenarios for LST based exploration

Given that the LST algorithm depends primarily on the parallel and orthogonal components of the model gradient, LST is efficacious on deep networks that do not suffer from gradient obfuscation or gradient masking issues, which encompasses nearly all commonly used vision models. On the other hand, for models with non-differentiable or randomized inference-time components, the LST algorithm would have to be modified to utilize techniques such as Backward Pass Differentiable Approximation (BPDA) and Expectation over Transformation (EoT), as proposed in past adversarial attack literature [Athalye et al., 2018], that helps overcome shattered, stochastic, exploding and vanishing gradients. We also note that these modifications also often require a sizable increase in the maximum iteration count to achieve adequate convergence levels.

D.2 Wall Clock Time

We record wall clock time on a single RTXA5000 GPU with a ResNet-50 model on ImageNet, using a batchsize of 100, and report mean and standard deviation ( µ ± σ ) statistics over 5 independent minibatches. The proposed LST algorithm is seen to require 101.8 seconds ( ± 0.4s) for 400 iterations, and 52.0 seconds ( ± 0.3s) for 200 iterations. In comparison, a 100-step PGD attack of the same model takes 25.1 seconds ( ± 1.4s). Thus, per iteration, the LST algorithm takes the same amount of time as a PGD attack iteration (both about 0.25 seconds). In total, the LST algorithm of 200/400 iterations requires 2x/4x the computation time of a standard 100-step PGD adversarial attack.

We generally observe that for adversarially robust models, near-optimal convergence to the target image (wherein LST output looks essentially identical to the target image) requires a larger number of steps as compared to normally trained models. While we fix the max iterations to 400 across all ImageNet models for instance, it is possible to reduce this parameter for normally trained models while maintaining good visual quality.

E Ablation and Sensitivity Analysis

Here we perform ablation studies on LST hyperparameters to study the impact of each one independently. We use default parameters - number of LST iterations m = 400 , step size perpendicular to the gradient η = 10 -2 , step size parallel to the gradient ϵ = 0 . 002 , confidence threshold δ = 0 . 2 -unless mentioned otherwise. We evaluate our ablative analysis on a set of 500 images from ImageNet using a normally trained ResNet-50 model. In general, we observe a trade-off between optimizing for

visual similarity towards the target image of another class, and high-confidence metrics associated with the LST blindspot output so produced.

A1. Varying number of iterations m for which LST is run (see Tables 5 and 6): As the number of iterations increases, the distance metrics between the output of LST and the final images decrease. We ideally desire these deviations between LST outputs and target images to be low, to ensure good visual quality. Hence, a higher m value is preferred for performing LST.

A2. Varying adversarial step size ϵ parallel to the negative gradient for LST iterations m = 100 (see Tables 7 and 8): With fewer iterations for LST, a larger adversarial step size ϵ is greatly beneficial. As shown in the Tables, a larger ϵ results in better confidence metrics over the triangular convex hulls.

A3. Varying step size η perpendicular to the gradient for LST iterations m = 100 (see Tables 9 and 10): From the Tables, we can see that as the step size η increases, LST output images become closer to the target images for a smaller value of m = 100 . Also, with larger η , we can achieve larger triangular confidence scores. However, the best confidence scores are achieved using larger m values.

A4. Varying step size η perpendicular to the gradient (see Tables 11 and 12): As expected, the distance metrics between the LST output images and the target images are larger with a lower η . η = 0 . 01 , 0 . 05 achieve desirable small distance scores. Among these, our default parameter η = 0 . 05 achieves the best confidence scores, alongside adequate convergence with minimal distance metrics towards the target image.

A5. Varying step size ϵ parallel to the gradient (see Tables 13 and 14): A larger ϵ results in LST output images that are closer to the final images in terms of our distance metrics. We also note that a larger adversarial perturbation helps in achieving higher triangular confidence scores. We observe that setting the adversarial step size to zero results in output images that are perceptually different compared to the target image, as observed with the larger image distance metrics such as LPIPS distance. However, we also observe that even extremely small non-zero adversarial step-sizes are very effective in improving convergence towards the target images.

F Additional Level Set Traversal Examples

Figures 11, 12, 13, 14, and 15 show examples images output by our LST algorithm using ImageNet images for a normally trained ResNet-50 from ablation studies A1, A2, A3, A4, and A5, respectively. Figures 16, 17, 18, 19, and 20 show their respective triangular convex hull.

black swan

TibetanTerrier

tigerbeetle

accordion

cliff dwelling

black swan

TibetanTerrier

tigerbeetle

accordion

cliff dwelling

black swan

black swan

0.8279

0.9877

0.9109

0.9684

0.9565

1.0

66660

66660

1.0

0.9997

Tibetan Terrier

Tibetan Terrier

0.9421

0.8453

0.9124

0.8914

0.9489

1.0

1.0

0.9999

0.9997

0.9999

tiger beetle

0.9492

0.9637

0.8809

0.922

0.9448

0.9998

0.9998

0.9999

0.9998

0.9999

accordion

accordion

0.844

0.9097

0.78

0.8597

0.8074

0.9988

0.998

0.9992

1.0

0.9962

cliff dwelling

cliff dwelling

0.9258

0.9824

0.966

0.9367

0.8641

1.0

1.0

1.0

0.9997

1.0

black swan

Tibetan Terrier

tigerbeetle

accordion

cliff dwelling

black swan

TibetanTerrier

tigerbeetle

accordion

cliff dwelling

black swan

black swan

0.8279

0.9877

0.9109

0.9684

0.9565

1.0

8666°0

0.9998

1.0

0.9997

Tibetan Terrier

Tibetan Terrier

0.9421

0.8453

0.9124

0.8914

0.9489

1.0

1.0

0.9998

0.9999

0.9989

tiger beetle

tiger beetle

0.9492

0.9637

0.8809

0.922

0.9448

1.0

0.9999

0.9999

1.0

1.0

accordion

0.844

0.9097

0.78

0.8597

0.8074

0.9991

0.9984

0.9967

1.0

0.9969

cliff dwelling

cliff dwelling

0.9258

0.9824

0.966

0.9367

0.8641

0.9999

0.9999

0.9999

1.0

1.0

black swan

TibetanTerrier

tigerbeetle

accordion

cliff dwelling

black swan

TibetanTerrier

tigerbeetle

accordion

cliff dwelling

black swan

black swan

0.8279

0.9877

0.9109

0.9684

0.9565

1.0

6660

0.9987

0.998

0.9984

Tibetan Terrier

Tibetan Terrier

0.9421

0.8453

0.9124

0.8914

0.9489

0.9995

1.0

0.9988

0.9968

0.9979

tiger beetle

tiger beetle

0.9492

0.9637

0.8809

0.922

0.9448

0.9989

0.9978

66660

0.9968

0.998

accordion

accordion

0.844

0.9097

0.78

0.8597

0.8074

0.9773

0.8858

0.8012

1.0

0.8779

cliff dwelling

cliff dwelling

0.9258

0.9824

0.966

0.9367

0.8641

0.9999

0.9993

0.9996

0.9946

1.0

black swan

TibetanTerrier

tigerbeetle

accordion

cliff dwelling

black swan

TibetanTerrier

tigerbeetle

accordion

cliff dwelling

black swan

black swan

1.0

1.0

1.0

1.0

1.0

1.0

86660

0.9997

66660

0.9998

Tibetan Terrier

Tibetan Terrier

1.0

1.0

1.0

1.0

1.0

0.9998

1.0

0.9996

0.9995

0.9995

tiger beetle

tiger beetle

1.0

1.0

0.9999

1.0

1.0

8666°0

86660

0.9999

0.9997

0.9998

accordion

accordion

1.0

1.0

0.9999

1.0

1.0

0.9982

0.7988

0.9974

1.0

0.9988

cliff dwelling

cliff dwelling

1.0

1.0

1.0

1.0

1.0

1.0

0.9998

0.9999

0.9997

1.0

black swan

Tibetan Terrier

tigerbeetle

accordion

cliff dwelling

black swan

TibetanTerrier

tigerbeetle

accordion

cliff dwelling

black swan

black swan

1.0

0.8098

0.8305

0.8221

0.789

1.0

1.0

1.0

1.0

1.0

Tibetan Terrier

Tibetan Terrier

0.7945

1.0

0.8275

0.7995

0.7903

1.0

1.0

1.0

1.0

1.0

tiger beetle

tiger beetle

0.787

0.769

0.9999

0.7535

0.6897

1.0

1.0

0.9999

0.9999

1.0

accordion

0.7907

0.813

0.8131

1.0

0.7894

0.9998

0.9997

0.9999

1.0

0.9998

cliff dwelling

cliff dwelling

0.8144

0.8139

0.8297

0.7958

1.0

1.0

1.0

1.0

1.0

1.0

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors explore the output space of vision models. Specifically, they propose an algorithm (Level Set Traversal, LST) which, starting from a \"source\" input image transforms it into an image that looks completely different (e.g. an image from a different class) while still confidently predicting the \"source\" class. This proves that neural networks have \"blind spots\" they further analyze the behavior of networks, and show that the paths between these inputs are connected.\n\nSTRENGTHS:\nOriginality: the paper improves upon previous work in the area, which only applied to very specific network architectures (invertible resnets). In contrast, the novel LST algorithm can be applied to virtually any network.\n\nQuality: Both the theoretical and the empirical work appear solid to me.\n\nClarity: I was not able to follow all the math in detail, but was nonetheless able to follow along nicely.\n\nSignificance: I think this paper is a solid contribution to the field. It provides both some theoretical analysis of the output space of vision models, as well as an algorithm for producing \"blind spot\" examples that is easy to understand and implement, and seems to work well.\n\nWEAKNESSES:\n* The authors state that the algorithm takes 200-400 iterations for each input image. To put this into context it would be nice if the authors could give a wall clock time estimate on how long this would take.\n\nQUESTIONS:\n* The authors mention that doing only 10ish iterations produces much lower-quality results. I would be curious to see what these would look like (though this is just personal curiosity, so feel free to ignore this).\n\n* I don't understand what the colors within the triangles in Figure 4 represents, could you explain it?\n\n* Do the images you obtain transfer to other architectures? E.g. If we take the off-diagonal images from Fig. 2 and put them into a different network than the one used to create them, would they be classified as target or source class?\n\nLIMITATIONS:\nThe authors have adequately discusses limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the under-sensitivity of such deep neural networks, where it is possible to find large perturbations in the input space (such as transforming one image to another) without significant changes to the activations/predictions. Towards this goal, the paper proposes Level Set Traversal, LST, which adds perturbations in an incremental manner to move along regions orthogonal to the gradient (w.r.t input) space computed on the classification loss. Evaluations are done on ImageNet and CIFAR-10, and results show that LST is able to find not only large perturbations that continue to yield high confidence on the source class, but a path of high confidence from the source to the target. Experiments also show that the convex hull formed by a source image and a pair of LST output images for two targets enclose a region of high-confidence for adversarially trained models, unlike non-adversarially trained ones.\n\nSTRENGTHS:\n- The explanation and intuition of the method is very clear, and it also appears to be highly effective in finding an entire path of high confidence/similar loss from two highly visually different images.\n- Theoretical analysis presented in Sec 4 is also helpful and insightful.\n- Additional experiments on CIFAR-10 and ablation studies in the supplementary material are also appreciated.\n\nWEAKNESSES:\n1) The experiment details mention 1000 images from ImageNet are chosen as the source images, but only 5 target images (the ones in the figure) are used. It is difficult to tell whether the results are overfitted to the 5 target images. More extensive evaluation can be done, for example simply randomly choosing source-target pairs from the 1000 chosen images.\n\n2) I do not see any metric measuring the confidence of the output of LST alone. This metric is highly important especially for Table 1, to show that LST can both jointly optimize for minimal distance from target image as well as preserving confidence. Measuring path confidence averages the results across the entire path is not sufficient since the confidence result can be biased by points closer to the source image and hence more likely to exhibit high confidence. \n\n3) Measurement of computational cost (wall-clock time) of the LST algorithm might be useful.\n\nQUESTIONS:\nWhat are the main failure cases of LST - under what situations does LST work best, and when is LST unable to find such paths or outputs of low loss/high confidence?\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new adversarial attack on the discriminative vision models called Level Set Traversal (LST). Contrary to the previous attacks, this new algorithm exploits the orthogonal component of the network's gradient to produce samples that can bypass existing adversarially-trained classification networks.\n\nMoreover, the authors showcase that the samples obtained using this algorithm lie on the star-shaped manifold of high-confidence predictions. This is new compared to the previous adversarial attacks, which typically yield samples not connected via high-confidence linear paths.\n\nSTRENGTHS:\n+ The presentation of the paper is great.\n\n+ The motivation behind the proposed method is clear. I also like that the paper is theoretically driven and includes experimentally confirmed results on real-world datasets.\n\n+ To me, the results look really convincing and exciting. I was surprised at the existence of the linearly interconnected sets of adversarial examples, even in adversarially trained models.\n\n+ The ablation study of the method is quite extensive and includes robustness tests for multiple hyperparameters.\n\nWEAKNESSES:\n- I would suggest the authors try and make connections to the area of model ensembling, where works such as [1] showcased the existence of piecewise-linear paths with low training error in the space of the model weights.\n\n- I also suggest including section 1 of the appendix in the main paper, as well as extending Figure 4 to include the visualization for multiple attack types. Without reading the appendix, the advantage of the proposed algorithm over previous methods remains unclear. Extended proof of Lemma 1, in my opinion, could be instead moved to the appendix, as it breaks the flow of the text.\n\n- The authors have only validated their approach against one type of adversarial defense for each proposed network. Extending such evaluation would benefit the community and provide a benchmark for future works.\n\n[1] Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs, NeurIPS 2018\n\nQUESTIONS:\n* Did the authors benchmark multiple state-of-the-art adversarial defense methods against their attack?\n\n* Why does the discovered high-confidence manifold have more high-confidence regions for adversarially-trained models than non-AT variants? Have the authors evaluated multiple AT methods to explore whether or not the shape of this region depends on the network architecture (ResNet vs ViT) or the specific AT method?\n\nLIMITATIONS:\nThe limitations were adequately addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces the idea of level sets for image classification models. Image classification models are said to be ‘under-sensitive’ when two visually distinct images produce the same output (class). The authors propose a method to compute ‘equi-cconfidence’ level sets such that two images belonging to this set produce the same output when passed through a classifier. \n\nThe Level Set Traversal (LST) algorithm takes a source image and target image (visually distinct from the source, different class) and produces an image visually similar to the target while maintaining the output prediction of the source image. This is done by iteratively updating the source image in the direction perpendicular to the gradient. The paper then goes on to show (theoretically) the behavior or level sets in some basic ML settings (linear classifier, rely neural networks etc).\n\nAdditionally, the authors also highlight the complimentary nature of adversarial examples and level sets: Adversarial examples try to change model output while keeping human predictions the same while levels sets try to keep model output the same when human predictions differ.\nExperiments are done on ResNet-50 and DeiT on ImageNet. Empirical results show adversarially trained models exacerbate the problem of under sensitivity compared to vanilla models.\n\nSTRENGTHS:\n1. The LST algorithm is the main contribution of the paper and is a novel way to calculate equity-confidence sets given a source-target pair.\n2. The paper is well written and concepts are introduced in an easy to understand fashion.\n3. Significance: Understanding what causes under-sensitivity in a model is a major open problem.\n4. I really like the explanation of the under-sensitivity of adversarial models using the LST algorithm. I think that is the major insight from this paper (apart from the algorithm itself of course).\n\nOverall, I really like the idea and the execution of the paper so I'm conflicted whether to recommend acceptance  (see my comments below). I'm looking forward to hearing from the authors and seeing other reviewers' comments.\n\nWEAKNESSES:\n1. I think the main weakness of the paper is the lack of analysis around the level set found for a particular source-target pair. For example, in the adversarial example setting, adversarial examples don’t need to be w.r.t. some target, they just need to change the label with a perturbation. In this paper’s case, the level set itself depends on a target label. This isn’t necessarily an issue on it’s own, however, it is difficult to understand what showing the mere existence of a level set is supposed to show about the under-sensitivity of the model. Is there something about the level set we've found that tells us something about the under-sensitivity of the model? Other questions: How does choice of target affect the level set? Is there a ’size’ for a level set? Are level sets from source -> target and target -> source symmetric?\n2. Related to presentation: I think including a figure showing images along a path from source to the final image generated by the level set algorithm would be useful to understand what exactly is within a level set. This figure could potentially replace Figure 3 which, in my opinion, does not add anything new (it shows the same thing as Figure 2 except a change in architecture)\n\nQUESTIONS:\n1. How are the images inside the convex hull in Fig 4 obtained? The images on the edges are calculated via linear interpolation but I am not sure about those inside the convex hull. \n2. How do you distinguish ‘under-sensitivity’ from just poor training? Suppose you chose an architecture/set of hyperparameters which is bad fit for the dataset. Wouldn’t this setting also be ‘under-sensitive’? Is there any way to tell them apart?\n\nLIMITATIONS:\n1. (Relatively) Large computational budget required to calculate images close to the target via LST. (Mentioned by the authors)\n2. Level Sets are a property of target class in addition to source class (unlike adversarial examples)", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors aim to systematically study specific shortcomings (called blind spots) of vision models caused by their under-sensitivity.\nFor this purpose, they devise an algorithm that given an arbitrary pair of images (called source and target) can produce inputs that result in the same prediction output as the source image despite being \"perceptually\" similar to the target image (as quantified using LPIPS).\nThe authors study the geometry of the generated inputs and compare it under different training criteria (e.g. standard vs. adversarially robust models).\n\nSTRENGTHS:\n- Interesting analysis that can potentially shed light into inherent shortcomings of vision models.\n- The authors provide their source code\n\nUpdate after reubttal\n------------------------\nI would like to thank the authors for their elaborate response to the review remarks. My remarks are largely addressed in these responses, and I am hence leaning for accepting this contribution. In that case, I urge the authors to highlight the significance of their work and its utility to diagnose the sensitivity issues in adversarial training upfront. The manuscript is heavy on technical jargon that, while important, could be simplified in the main text and presented in the appendix.\nAs an additional suggestion, the following workshop at NeurIPS '23 would be a great fit to present some of the theoretical parts / foundations of this work https://www.neurreps.org/\n\nWEAKNESSES:\n- The contribution is rather limited. It is unclear what actionable insights we can gain using the proposed level-set analysis. There were no sufficient findings that can inform their design and training of vision models.\n- The observations with respect to the \"geometry of blind spots\" are highly anecdotal \n- The novelty is rather limited. A wide variety of methods have been devised to minimally perturb an image in order to fool the model to make a specific prediction. In the interpretability domain, various methods have meaningfully utilized minimal perturbation for the purpose of visualizing which image areas are most relevant for the input. In fact, similar perturbations are used in Integrated Gradients (Sundararajan et al. 2017), however, for a tangible application (feature attribution). See also the work by Wagner et al:\n\"Interpretable and Fine-Grained Visual Explanations for Convolutional Neural Networks\" (CVPR '19)\n\nMinor: There were frequent language issues. Below are the ones I noted:\n- sensitivty\n- dimnesional\n- upto human expectations => unclear (did you mean, compared with?)\n- phenonmenon\n- atttacks\n- near-neighbour training images => nearest-neighbour images?\n- simalar\n\nQUESTIONS:\nHow sensitive is your analysis to the parameters of the level set algorithm, besides delta (e.g. max iterations, scale factor and stepsize)?\n\nLIMITATIONS:\nPartially discussed", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-26"}