Title: Batchnorm Allows Unsupervised Radial Attacks
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adversarial

Batch normalization

Robustness

Geometric

radial
Abstract: The construction of adversarial examples usually requires the existence of soft or hard labels for each instance, with respect to which a loss gradient provides the signal for construction of the example. We show that for batch normalized deep image recognition architectures, intermediate latents that are produced after a batch normalization step by themselves suffice to produce adversarial examples using an intermediate loss solely utilizing angular deviations, without relying on any label. We motivate our loss through the geometry of batch normed representations and their concentration of norm on a hypersphere and distributional proximity to Gaussians. Our losses expand intermediate latent based attacks that usually require labels. The success of our method implies that leakage of intermediate representations may create a security breach for deployed models, which persists even when the model is transferred to downstream usage. Removal of batch norm weakens our attack, indicating it contributes to this vulnerability. Our attacks also succeed against LayerNorm empirically, thus being relevant for transformer architectures, most notably vision transformers which we analyze.

Batchnorm Allows Unsupervised Radial Attacks

Abstract

The construction of adversarial examples usually requires the existence of soft or hard labels for each instance, with respect to which a loss gradient provides the signal for construction of the example. We show that for batch normalized deep image recognition architectures, intermediate latents that are produced after a batch normalization step by themselves suffice to produce adversarial examples using an intermediate loss solely utilizing angular deviations, without relying on any label. We motivate our loss through the geometry of batch normed representations and their concentration of norm on a hypersphere and distributional proximity to Gaussians. Our losses expand intermediate latent based attacks that usually require labels. The success of our method implies that leakage of intermediate representations may create a security breach for deployed models, which persists even when the model is transferred to downstream usage. Removal of batch norm weakens our attack, indicating it contributes to this vulnerability. Our attacks also succeed against LayerNorm empirically, thus being relevant for transformer architectures, most notably vision transformers which we analyze.

1 Introduction

Adversarial examples x adv are commonly defined as data instances which lie ϵ away in some norm (usually L ∞ ) from an actual data instance x real . To humans, ϵ is small and x real , x adv share labels, yet to a classifier neural network, they do not. Since their discovery [Szegedy et al., 2013, Goodfellow et al., 2014], adversarial examples have spurred research in both attacking (i.e., generating adversarial data instances) [Akhtar and Mian, 2018, Tramer et al., 2020] and defending [Yuan et al., 2019] neural networks against them. Common methods for generating adversarial images use loss gradients. A pioneering attack is Fast Gradient Sign Method (FGSM) [Goodfellow et al., 2014], possibly strengthened via Projected Gradient Descent (PGD) [Madry et al., 2017] - an iterative form of FGSM. Defenses against these attacks have been derived, including adversarial training [Shrivastava et al., 2017], where adversarial images are fed to the network for training. Adversarial training is expensive but powerful [Salman et al., 2019] and scalable [Wong et al., 2020, Shafahi et al., 2019]. Defenses can give illusory security by obfuscating gradients [Athalye et al., 2018]. An unsaid aspect of defense is hiding model parameters - changing attacks from the white-box (model parameters known) scenario to black-box (models only accessible via outputs) scenario, which makes them harder to attack.

We consider the family of intermediate-level attacks, exemplified by members such as Intermediate Level Attack Projection (ILAP) [Huang et al., 2019] and its variants [Li et al., 2020b]. In this, an initial direction in the latent space of the hidden intermediate layers is determined via FGSM or other attacks. Gradient steps then maximally perturb the intermediate hidden representation to find a suitable x adv , unlike directly using the label. Such methods can outperform the method used for

the initial direction itself and be more transferable across different models. ILAP uses FGSM to set initial directions, but once the direction arises, the layers after the intermediate layer play no role.

̸

We now ask : Given an unlabeled data point x along with full access to a deep network upto K layers, where K is less than the network depth, could we craft an adversarial example out of x without any other information - e.g., the label of x , any x ′ = x - so long as the deep network utilized batch normalization ? We answer this question positively when x is an image for several popular families of image recognition architectures, exploiting the hyperspherical geometry of latent spaces to propose an angular adversarial attack. The label access of the black box case is absent - we do not allow access to hard labels (true label of x ) or soft labels (logits obtained at penultimate layers). We adapt the method advanced in ILAP to remove the need for label-based FGSM, and proceed instead with an angular loss based on the geometry of batch normalization. Our concrete contributions are :

To provide a label-free attack that relies on only gradient access upto intermediate layers

Ablation of our method to show it is not brittle to choices of intermediate layers, (small variations between using layers i, i +1 if i is sufficiently deep)

To show prominent vision models - ResNets and EfficientNets - fall to our attack. Removing batchnorm in ResNets via Fixup neuters the attack - suggesting it is playing a role in this.

To show that the attack empirically works even for transformer architectures (ViT) that employ Layernorm over batchnorm.

To show that these attacks persist in the transfer learning setting, where the models are frozen upto a certain layer and then fine-tuned for downstream usage.

To improve the supervised attack case when labels are available by using our loss in conjunction with the loss from the true label.

̸

This implies that releasing the first K layers of a model publicly as a feature extractor, without any label access, can create adversarial attacks, which has security implications. Weaknesses persist even when the unseen, later layers of the network are tuned on different datasets (e.g. CIFAR10/100) adversarial examples can be crafted for CIFAR without using the changed, later layers only using this first K . We exploit the radial geometry of batch normalization and assume the norm of the neural representations concentrates around a constant, allowing us to exploit the geometry formed. This indicates batch normalization may make networks not just less robust to conventional attacks [Galloway et al., 2019], but open up new attacks entirely. We ablate our method and compare to scenarios of naive baselines and having access to x ′ = x to showcase its competitiveness.

1.1 Relation to Related Work

We tie together two areas : analyzing batch normalization from a geometric standpoint under convergence assumptions, and adversarial attacks with partial knowledge of architectures. Batch normalization has been analyzed from the standpoint of deep learning theory and model optimization [Santurkar et al., 2018]. While its detrimental effects on model robustness [Galloway et al., 2019, Benz et al., 2020] are known, most analysis focuses on how batch normalization makes the model weaker to already-existing attacks. We propose that batch normalization might impact the network by making it more vulnerable to novel attacks. We are to our knowledge the first to connect this with the concentration of norm phenomenon implied by results such as the dynamics of batch normalization at high depths [Daneshmand et al., 2021]. These tie into results such as neural collapse [Papyan et al., 2020], which find that later layers in deep networks have predictable geometric patterns.

Research in adversarial examples has focused in specific settings such as white/grey/black box settings denoting levels of model access/knowledge. Even in the black box case, it is often assumed we can feed an input x into the model M under attack and retrieve M ( x ) , the model's prediction of the label of x . We consider the case of partial model knowledge and attacking a model with no label outputs. This is relevant with the presence of large-scale pretraining, where models are fed large training sets of data unrelated to any one domain to capture the overall distribution of a particular type of modality, such as GPT-3 [Brown et al., 2020] with natural language. Deployed downstream, these public models may fine tune on a smaller dataset, and most of their weights are not fine-tuned. Public models may act as feature extractors that distill large amounts of data beyond dataset scopes. They output not labels, but representation vectors, and our attack targets this case. Previous works do use latent representations to construct adversarial examples [Park and Lee, 2021, Creswell et al.,

2017, Yu et al., 2021] through moving in latent space towards points of differing labels or maximizing deviations in latent space [Kos et al., 2018]. Along with the related no-box [Li et al., 2020a, Zhang et al., 2022b] setting (assuming no model knowledge), these methods require extra parts, such as representations of instances of different labels as targets to move towards, surrogate models, or some soft labels, often with gradient access - all of which we do not require. They are also ad hoc in their choice of layers to attack, but we give a definite answer of which layers and architectures are the most vulnerable - batch norm layers situated deeper in the network. The closest case to our attack are label-free, single-stage latent-based attacks [Zhou et al., 2018, Inkawhich et al., 2019, Ganeshan et al., 2019, Inkawhich et al., 2020b, Zhang et al., 2022a, Wang et al., 2021, Inkawhich et al., 2020a] which we outperform. These methods use the latent to boost the supervised scenario and not exploit batch normalization or two-step optimization in an unsupervised setting as we do.

1.2 Setup - FGSM and PGD Attacks

We describe FGSM aka Fast Gradient Sign Method [Goodfellow et al., 2014] for generating an adversarial example. The task is to find x ′ which differs from x in the label assigned by a classifier C , which outputs a distribution over K classes ( K positive numbers that sum to 1 ).

̸

Generally, we only restrict attacks to cases where arg max i [ C ( x )] i is the true label, i.e., C is correct. The norm ∥ . ∥ used to define the attack is almost always the L ∞ norm, though other choices such as L 2 [Carlini and Wagner, 2017] have been tested as well. We will assume the norm to be L ∞ for the description. Computing x ′ (possibly with projection to a valid domain) proceeds as follows:

Apply C to x to obtain C ( x ) .

Compute gradient ∇ L wrt x where L is the loss between C ( x ) and the true label vector v l .

Compute x ′ i = x i + ϵ × sign( ∇ L ) i , i.e., move by ± ϵ based on sign of gradient.

We understand the sign function to map to -1 , 0 , 1 respectively as the argument is < 0 , 0 , or > 0 . FGSM uses gradient information, which may vary rapidly around x . A stronger variant, the Projected Gradient Descent (PGD) attack [Madry et al., 2017] uses multiple steps near x taking gradients at each step. Under PGD, there is a step size α , typically << ϵ , which iteratively updates x t i as :

With a clamping step on x t i that enforces the constraint of ∥ x ′ -x ∥≤ ϵ , this step after initializing x 0 i at x i yields much stronger adversarial examples than FGSM at higher t .

1.3 Intermediate Level Attack Projection (ILAP)

These methods - FGSM and PGD - rely on the input and the final layer's output loss with respect to the true label. We can consider a neural network N of depth D as :

With 1 based indexing, N j,k is the neural sub-network via composing layers from depth j to k . Let z i = N 1 ,i ( x ) be the latent representation at depth i . Consider x adv from any baseline method, and let

We then seek an alternate adversarial example, x ILAP i which seeks to work with the loss function :

The L can then be plugged into either FGSM or PGD. Put simply, this loss encourages latent space movement directionally similar to FGSM/PGD. L increases the inner product with ∆ adv i and later iterations use it over the true loss. This alternate adversarial example is highly transferable - e.g., adversarial examples from ResNet18 have higher success rates vs different models such as DenseNet, when created by an ILAP process than directly using FGSM. It can even beat FGSM on the source model itself. [Huang et al., 2019] ILAP demonstrates that latent representations provide ammunition for adversarial attacks, but requires an initial adversarial example x adv created with full model access and label. To remedy this, we craft an attack needing neither. We show that if the correct layers are attacked with an angular version of the same loss, the initial adversarial example is unnecessary.

2 Geometry of Batch Normalization

Given a minibatch of inputs x i of dimensions n with mean µ ij and standard deviation σ ij at index j , consider batch normalization [Ioffe and Szegedy, 2015] or just BN as:

Such normalization (with affine shift) has become commonplace in neural networks. During training, the empirical average of the minibatch is used and during testing, computed averages (mean/standard deviation) are used. By batchnorm, we mean only the batchnorm layer without an affine shift. Layer normalization [Ba et al., 2016] is an alternate form of normalization that normalizes over all neurons in a hidden layer. For space reasons, we discuss batch normalization in the main text and layer normalization in Appendix A. Empirically, our attack succeeds on layer norm and architectures using it, such as transformers [Vaswani et al., 2017] and vision transformers [Dosovitskiy et al., 2020].

Converged regime: Suppose in training, the sample means/standard deviations in a pre-BN layer converge to sample statistics. Post-batchnorm, representation vector Z satisfies, at every index j :

i.e., for a layer of dimensionality d and denoting the entire latent vector as Z , E ( Z ) = 0 , E ( ∥ Z ∥ 2 ) = d by linearity of expectation. The above convergence is with respect to the training set, but carries over to the test set (approximately) under the I.I.D assumption when learning takes place. Suppose ∥ Z ∥ 2 was concentrated about its expected value - i.e., nearly all ∥ Z ∥ values were ≈ √ d . We first discuss the implications of such a scenario and what attacks they allow. We assume that in this latent space, the inner product determines 'similarity'. Two latents Z a , Z b from different instances X a , X b will be closer as ⟨ Z a , Z b ⟩ rise. Under this metric , the most dissimilar point to Z a is parallel to -Z a . We will formulate our attack assuming that ∥ Z ∥ = √ d i.e. a hyperspherical latent space. After we have formulated our attack, we consider scenarios of such spaces and concentrations of norm.

2.1 Angular Attack Based on Converged Batchnorm

Let N 1 ,i be the network upto layer i with dimensionality d . If layer i is a batch norm layer (without affine shift), z = N 1 ,i ( x ) lies (with converged batchnorm and ∥ Z ∥ 's norm concentrated) approximately on a shell of radius √ d . The natural distance metric between two z, z ′ is angular distance:

where ⟨ z, z ′ ⟩ is at a local maximum on the hypersphere ∥ z ∥ = constant when z = z ′ and no gradient exists. We need another gradient initially, and propose the following attack algorithm. Given a real example x 0 with latent z 0 = N 1 ,i ( x 0 ) , z t = N 1 ,i ( x t ) we consider the loss L t init = -∥ z t ∥ = -∥ N 1 ,i ( x t ) ∥ for the first t init iterations. We then iteratively generate x t :

After t init iterations, we modify the angular loss to L radial, using the deviations obtained so far. L radial is angularly defined, denoting z t i the latent at depth i and iteration t :

We then iteratively update as follows (graphically depicted in Figure 1):

The figure geometrically depicts how intermediately moving from a point to another in latent space lowers the radial norm ( A i , left). Directly lowering the radial norm via a path from A 0 to A 1 (right) can create an initial deviation. But extending this to C as in the figure ends up with a lower angular deviation than maximizing angular deviation directly and ending up at A adv . Methods which directly increase the L 2 deviation starting from an initial method would result in points such as C .

x

y

z

A

C

A

i

C'

Aadv

A

Both losses depend on the layer i . With access till layer D , multiple candidate i values exist. For simplicity, we only use D,D -1 and replace sign( ∇ L init ) with 1 2 [sign( ∇ L init ,D -1 )+sign( ∇ L init ,D )] where the subscript L init ,D -1 denotes setting i as D -1 , and similarly we set i = D and average the sign for sign( ∇ L radial ) . This averaged term becomes Ψ in our pseudocode (Algorithm 1). We clamp x k iterates to satisfy ∥ x k -x 0 ∥≤ ϵ . Our attack resembles ILAP, with the initial direction ∆ adv i from an unsupervised radial loss without labels. We then maximize the angular deviation from the original latent, and do not 'follow the leader' on the original angular deviation after t init . The intuition is as follows. The surrogate loss -|| z t || moves along a chord in the hypersphere's interior from z 0 towards -z 0 , the most dissimilar point, while latent representations are on the surface . We use the last 2 layers for simplicity - stronger attacks might exist using all layers. Concentration of norm forms the hypersphere, and the 2-step process is key. 1 -step methods, e.g. single-stage works discussed under related works, use methods such as random initialization and angular minimization, but perform worse (see Appendix P). The concentration is in L 2 , but the adversarial example is in the L ∞ metric.

Algorithm 1 Angular attack algorithm

Input: Neural Network N ; access of model parameters of N till layer D , N 1 ,i : sub-network of N upto layer i , a starting real sample x 0 , perturbation α

Moving from z 0 along -∇∥ z t ∥ does not move linearly towards the origin when the optimization is imperfect - e.g. using the gradient sign. We manipulate x i , not z i - it may not be ∃ x ′ such that N 1 ,i ( x ′ ) = γz 0 , ∀ γ, 0 ≤ γ < 1 (the ray joining originz 0 may lack points with pre-images in x ). Another issue is α movement direction-wise (ignoring sign averaging) - e.g. let Z i ∈ R 2 , N 1 ,i ( x ) = x - the representation as identity map. Let x = (1 , ϵ ) lie on the hypersphere of radius √ 1 + ϵ 2 . The sign of ∇ L init is ( -, -) and x 0 = (1 -α, ϵ -α ) - not necessarily collinear with (0 , 0) -(1 , ϵ ) . Batch

normalization leaks mean/variance to adversaries. Suppose we had a perfect network N (could find x mapping to any z , i.e., was onto), and were assured the latent space was a zero mean hypersphere and we get adversarial instances if we move the latent from z to -z . This would be insufficient if all latents were translated by some vector c , i.e., the mean of the hypersphere was nonzero. To find the point opposite z on the new hypersphere ( c -( z -c ) ), we need c . Batch norm gives us this c . This weakness outweighs knowing z ′ arising from x ′ of a different label, as we show in the results.

2.2 When Does the Norm Concentrate ?

We assumed that after a batch normalization step, ∥ Z ∥ concentrates. Consider sufficiency conditions. Let Z ij be the j -th entry of the i -th latent Z i formed from X i with expectation 0 , variance 1 (batch norm). Consider independent instances X i and assume no instances indexed by i were used for training to impact model parameters, which affects the other latents and makes them dependent - i.e. consider test/validation sets. Then, Z ij , Z i ′ j ′ are independent if i = i ′ , ∀ j ′ = j . Now :

This sums d random variables (assuming Z i has dimension d ) and depends on independence structures and the marginal distributions of each variable. Under certain cases, independence of Z ij , Z ij ′ can occur. Suppose Z ij was constrained in {-1 , 1 } , and had to encode 2 D X i 's. Then, Z ij , Z ij ′ are independent under the optimal encoding, assigning for each X i a unique D -length code with entries ∈ {-1 , 1 } . When Z ij , Z ij ′ are independent, ∥ Z i ∥ 2 = ∑ j Z 2 ij will concentrate around its expected value of d under mild conditions e.g. bounded 4 -th moments of Z ij (Chebyshev's inequality), but concentration tails may not be sub-Gaussian. By tails of concentrations, we mean bounds of form:

For example, sub-Gaussian concentration implies f ( δ ) = O (exp( -δ 2 )) . Suppose each Z ij is marginally distributed as a (sub)-Gaussian. With independence, Chernoff's inequality leads to a norm concentration for ∥ Z ∥ with stronger sub-Gaussian tails over Chebyshev's (tails of 1 δ 2 ). Inequalities and implied tails can be consulted from e.g. [Vershynin, 2018]. We consider some previous results. For deep linear models of width(dimensionality) d , we have [Daneshmand et al., 2020, 2021]:

Z is the distribution after N 1 ,i i.e., i layers, α is a constant, b is the training batch size. The LHS (termed the orthogonality gap) indicates that the distribution of the latents convergences to the isotropic Gaussian using KL divergence, which is a stronger condition leading to the latent norms being clustered around √ d . Though derived for linear networks, the original paper offers evidence that the conjecture holds for general multi-layer networks. This indicates deeper layers i.e., increasing i will have the largest effect on our assumptions holding, as this exponentially drops the KL divergence to a Gaussian (and concentrates the norm), with a secondary benefit from increasing width ( d ). Beyond linear networks, general networks with depth and width reduce to Gaussian Processes [Yang et al., 2019, Yang and Schoenholz, 2018, Yang and Hu, 2020, Yang, 2019, Neal, 2012]. These results usually apply to marginal distributions and not independence structures. Empirically, we find dependence between different variables (measured as correlation) falls as the network latent grows deeper, which aligns with theory. Dimensionality falls as the network deepens recall the toy example of encoding 2 D instances with D binary variables which forces independence. ILAP's empirical ablation studies found the optimal i to create the latents for the attack occurred in the range 0 . 6 to 0 . 8 (network depth normalized to 1 ). In sum, we expect our optimal layers for the latent to lie near the end of the network . Our methods work without assumptions in hyperspherical representation spaces [Wang and Isola, 2020, Schroff et al., 2015, Mettes et al., 2019]. Analysis and empirical findings on the concentration and tails is included in Appendix R.

3 Results

We carry out an extensive set of experiments on Imagenet [Russakovsky et al., 2015], utilizing several ResNet [He et al., 2016] models - ResNet-{ 18 , 34 , 50 , 101 , 152 } and EfficientNet models [Tan and

̸

0.00

0.03

0.06

0.10

Epsilon values

0

10

20

30

40

50

60

70

80

90

Resnet34 FGSM

Resnet50 FGSM

Resnet101 FGSM

Resnet152 FGSM

Resnet18 Angular

Resnet34 Angular

Resnet50 Angular

Resnet101 Angular

Resnet152 Angular

Original Resnet

0.00

0.03

0.06

0.10

Epsilon values

10

20

30

40

50

60

70

80

90

Fixup Resnet18 FGSM

Fixup Resnet34 FGSM

Fixup Resnet50 FGSM

Fixup Resnet101 FGSM

Fixup Resnet152 FGSM

Fixup Resnet18 Angular

Fixup Resnet34 Angular

Fixup Resnet50 Angular

Fixup Resnet101 Angular

Fixup Resnet152 Angular

0.00

0.03

0.06

0.10

Epsilon values

0

10

20

30

40

50

60

70

80

90

B1 EffNet FGSM

B2 EffNet FGSM

B3 EffNet FGSM

B4 EffNet FGSM

B5 EffNet FGSM

B1 EffNet Angular

B2 EffNet Angular

B3 EffNet Angular

B4 EffNet Angular

B5 EffNet Angular

EfficientNet

Le, 2019] B1 to B5 . We chose these models because they have a common structure at construction. Except a few initial layers, they are constructed by repeatedly stacking the same blocks ('Basic' or 'Bottleneck' blocks for ResNets, and Mobile Convolutional blocks [Howard et al., 2017] for EfficientNets). Every block possesses at least one batch norm layer, allowing the extraction of the appropriate latent. All models were ran on Imagenet with a standard normalization pre-processing step. Results of all attacks on all models, a sanity check with a fixed norm model (explicitly hyperspherical, i.e. always concentrated in norm), improving supervised (label present) cases on top of FGSM/PGD, comparisons to single-stage attacks previously proposed, interactions with defence, ablations, visualizations, statistical testing, confidence intervals and other transfer learning results and comparisons appear in the Appendices. As the alternative to batchnormed models, we choose Fixup [Zhang et al., 2019] - a different way to initialize the models. Core results are summarized in Figure 2. On the batchnorm-free Fixup Resnet, the FGSM attack greatly outperforms our angular attack. On the other two architectures, this is not the case. FGSM's performance and clean accuracy vary far less than the success of the angular attack. That implies overall model quality and robustness to common attacks remains the same, and vulnerability to the angular attack specifically is what varies. We train Fixup resnets on imagenet as alternatives to the batchnormed models for Resnet only, as methods in Fixup do not generalize to efficientnets. Details of these training steps are in the Appendix B, and we also attach our training and inference codebases. We obtained the models from publicly available repositories for PyTorch [Paszke et al., 2019], namely torchvision and EfficientNetsPyTorch, and did not change the original weights. For the choice of ϵ for the adversarial attack, we chose 0 . 03 , 0 . 06 , 0 . 1 and carried out all attacks using an α = 0 . 01 over 40 iterations.

Vision Transformers and LayerNorm. Recently, Layer normalization [Ba et al., 2016] has emerged as an alternate mode of normalization to batch norm. It finds prominent usage in the transformer architecture [Vaswani et al., 2017] and for images, in the ViT (Vision Transformer) models. We used our attacks on the case of ViT models [Dosovitskiy et al., 2020] (B/L)-(16/32). In all cases, the attacks were successful as with Resnets/batchnorm. This indicates our attack can succeed even when instead of batch normalization, layer normalization is utilized instead. We discuss this in Appendix A, but make batch norm the focus as it has more pre-existing theoretical analysis [Daneshmand et al., 2020].

We examine the results on all Resnets in Tables 1,2,3 with Table 8 denoting clean accuracies. Angular attacks - especially in terms of their top 5 accuracy, at higher ϵ and on deeper networks - dominate FGSM attacks, while losing to PGD attacks. No random perturbation performs well. Confidence intervals rule out a fluke (statistical tests Appendix C). Raw numbers for Fixup Resnets (in Appendix D) exhibit a different pattern - the angular attack performs as well as a random method, while FGSM, and PGD perform as normal. EfficientNets (in Appendix E) follow ResNets in their result. In Table 4, we exhibit the max over single-stage benchmarks among [Zhou et al., 2018, Inkawhich et al., 2019,

Ganeshan et al., 2019, Inkawhich et al., 2020b, Zhang et al., 2022a, Wang et al., 2021, Inkawhich et al., 2020a]. The performance is worse than our method. We also showcase results of ViT models on ϵ = 0 . 1 for imagenet in Table 5. (Results for other values of ϵ are in Appendix F). Finally in Table 6, we attack Resnets with the angular attack but move the layer being attacked to the layer before the BN layer. This noticeably weakens the attack, suggesting a relationship between the two.

We examine the drop in accuracy as a function of the access to various blocks. We run our attacks consistently accessing the 3 / 4 -th layer of a net, i.e., for a Resnet50 of blocks [3 , 4 , 6 , 3] - total 16 -we use only the first 12 blocks and sum our angular losses over block 11 , 12 . On Resnet-34 access to deeper blocks strongly strengthens the attack in Table 7. This agrees with the theory of batch normalization discussed previously [Daneshmand et al., 2021] and with the empirical findings of ILAP [Huang et al., 2019], which found that the most optimal layers to perturb lay between 50% and 80% of the network's length. Further depth ablations are in Appendix G. Concentration of norm relies on convergence of batchnorm and independence. Assuming the former, we can check independence by examining the absolute, off-diagonal correlations in the correlation matrices of the latent representations. We note a decorrelation effect with depth, which matches the success of our attacks. We compare the fall in absolute correlation among the latent dimensions in Table 9, across three 'block groups' of Resnet-18 which has 8 blocks organized as [2 , 2 , 2 , 2] . Correlation - a proxy for independence - decreases over the last three block groups.

Transfer learning. For transfer learning, we add a linear classifier that can access the last extracted latent layer of the resnet, and we unfreeze this classifier and the last group of blocks. Every other layer is frozen. The setup is fine-tuned, downstream, on CIFAR-10 and CIFAR-100 [Krizhevsky et al., 2009]. We show results on CIFAR-100 with Resnet-50 in Table 10. Although Table 10 indicates that targeted attacks perform worse than the original angular attacks, it does not mean that we cannot use the label information. It only means that when we randomly pick an instance of a different label, the angular attack often does better. But, the combination of label and angular attack can do better than either attack, so having an instance of a different label can still be very helpful. We provide some cases where our loss is provided in ensemble with the targeted loss and these perform better than either (see Appendix H). Results for CIFAR-10 / other resnets/datasets are in Appendix J. Our method outperforms all but PGD.

Ablation against other cases. The radial loss is an implicit movement towards the opposite point on a hypersphere. This assumes that the hypersphere exists at all. If we have a different latent Z j from a point X j of a different label from X i , we check if our attack might be more effective using an initial loss of the form -∥ Z i -Z j ∥ . Increasing this loss would be lowering the distance between Z i , Z j i.e. moving towards the other point in latent space. Yet, this targeted attack - shown in Table 10 - is actually inferior to our attack, which lacks the access.

4 Conclusion

We have shown a powerful, label-free attack which only utilizes a portion of all the layers available for a network to construct adversarial examples that fool the entire network. It succeeds at this without knowing the label or having gradient access to the full model, and these adversarial methods

generalize to the case where the model was fine-tuned afterwards. These results have relevance at the intersection of the theory and practice of adversarial robustness and the growing study of batch normalization and its weaknesses and drawbacks. We provide support to the notion that batch normalization may open up unseen avenues of perturbations that unexpectedly impact model robustness, a viewpoint supported by previous literature [Galloway et al., 2019, Benz et al., 2020, Wang et al., 2022]. We also extend our results to LayerNorm, which is increasingly relevant with the advent of transformer architectures [Dosovitskiy et al., 2020].

Acknowledgments and Disclosure of Funding

Resources used in this work were provided by the Province of Ontario, the Government of Canada through CIFAR, companies sponsoring the Vector Institute https://vectorinstitute. ai/partners/ and the Natural Sciences and Engineering Council of Canada.

References

Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access , 6:14410-14430, 2018.

Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420 , 2018.

Kazuoki Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical Journal, Second Series , 19(3):357-367, 1967.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Philipp Benz, Chaoning Zhang, and In So Kweon. Batch normalization increases adversarial vulnerability: Disentangling usefulness and robustness of model features. 2020.

Sergei Bernstein. On a modification of chebyshev's inequality and of the error formula of laplace. Ann. Sci. Inst. Sav. Ukraine, Sect. Math , 1(4):38-49, 1924.

Dean A Bodenham and Niall M Adams. A comparison of efficient approximations for a weighted sum of chi-squared random variables. Statistics and Computing , 26(4):917-928, 2016.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy (SP) , pages 39-57. IEEE, 2017.

Antonia Casta˜ no-Mart´ ınez and Fernando L´ opez-Bl´ azquez. Distribution of a sum of weighted noncentral chi-square variables. Test , 14(2):397-415, 2005.

Antonia Creswell, Anil A Bharath, and Biswa Sengupta. Latentpoison-adversarial attacks on the latent space. arXiv preprint arXiv:1711.02879 , 2017.

Hadi Daneshmand, Jonas Kohler, Francis Bach, Thomas Hofmann, and Aurelien Lucchi. Batch normalization provably avoids ranks collapse for randomly initialised deep networks. Advances in Neural Information Processing Systems , 33:18387-18398, 2020.

Hadi Daneshmand, Amir Joudaki, and Francis Bach. Batch normalization orthogonalizes representations in deep random networks. Advances in Neural Information Processing Systems , 34: 4896-4906, 2021.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W Taylor. Batch normalization is a cause of adversarial vulnerability. arXiv preprint arXiv:1905.02161 , 2019.

Aditya Ganeshan, Vivek BS, and R Venkatesh Babu. Fda: Feature disruptive attack. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 8069-8079, 2019.

Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572 , 2014.

Torben Hagerup and Christine R¨ ub. A guided tour of chernoff bounds. Information processing letters , 33(6):305-308, 1990.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

Dan Hendrycks and Kevin Gimpel. Early methods for detecting adversarial images. arXiv preprint arXiv:1608.00530 , 2016.

Wassily Hoeffding. Probability inequalities for sums of bounded random variables. In The collected works of Wassily Hoeffding , pages 409-426. Springer, 1994.

Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:1704.04861 , 2017.

Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. In Proceedings of the IEEE/CVF international conference on computer vision , pages 4733-4742, 2019.

Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7066-7074, 2019.

Nathan Inkawhich, Kevin Liang, Binghui Wang, Matthew Inkawhich, Lawrence Carin, and Yiran Chen. Perturbing across the feature hierarchy to improve standard and strict blackbox attack transferability. Advances in Neural Information Processing Systems , 33:20791-20801, 2020a.

Nathan Inkawhich, Kevin J Liang, Lawrence Carin, and Yiran Chen. Transferable perturbations of deep feature distributions. arXiv preprint arXiv:2004.12519 , 2020b.

Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167 , 2015.

Jernej Kos, Ian Fischer, and Dawn Song. Adversarial examples for generative models. In 2018 IEEE security and privacy workshops (SPW) , pages 36-42. IEEE, 2018.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Beatrice Laurent and Pascal Massart. Adaptive estimation of a quadratic functional by model selection. Annals of Statistics , pages 1302-1338, 2000.

Qizhang Li, Yiwen Guo, and Hao Chen. Practical no-box adversarial attacks against dnns. Advances in Neural Information Processing Systems , 33:12849-12860, 2020a.

Qizhang Li, Yiwen Guo, and Hao Chen. Yet another intermediate-level attack. In European Conference on Computer Vision , pages 241-257. Springer, 2020b.

Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 , 2017.

A Markov. On certain applications of algebraic continued fractions. Unpublished Ph. D. thesis, St Petersburg , 1884.

Colin McDiarmid et al. On the method of bounded differences. Surveys in combinatorics , 141(1): 148-188, 1989.

Pascal Mettes, Elise van der Pol, and Cees Snoek. Hyperspherical prototype networks. Advances in neural information processing systems , 32, 2019.

Radford M Neal. Bayesian learning for neural networks , volume 118. Springer Science & Business Media, 2012.

Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117(40): 24652-24663, 2020.

Geon Yeong Park and Sang Wan Lee. Reliably fast adversarial training via latent adversarial perturbation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7758-7767, 2021.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32, 2019.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115(3):211-252, 2015.

Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in Neural Information Processing Systems , 32, 2019.

Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normalization help optimization? Advances in neural information processing systems , 31, 2018.

Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 815-823, 2015.

Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems , 32, 2019.

Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2107-2116, 2017.

Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 , 2014.

Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 , 2013.

Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning , pages 6105-6114. PMLR, 2019.

Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems , 33:1633-1645, 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Roman Vershynin. High-dimensional probability, volume 47 of cambridge series in statistical and probabilistic mathematics. Cambridge University Press, Cambridge , 7:14-22, 2018.

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.

Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, and Zhangyang Wang. Removing batch normalization boosts adversarial training. In International Conference on Machine Learning , pages 23433-23445. PMLR, 2022.

Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. arXiv preprint arXiv:2005.10242 , 2020.

Zhibo Wang, Hengchang Guo, Zhifei Zhang, Wenxin Liu, Zhan Qin, and Kui Ren. Feature importanceaware transferable adversarial attacks. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7639-7648, 2021.

Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994 , 2020.

Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902.04760 , 2019.

Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522 , 2020.

Greg Yang and Sam S Schoenholz. Deep mean field theory: Layerwise variance and width variation as methods to control gradient explosion. 2018.

Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz. A mean field theory of batch normalization. arXiv preprint arXiv:1902.08129 , 2019.

Yunrui Yu, Xitong Gao, and Cheng-Zhong Xu. Lafeat: piercing through adversarial defenses with latent features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5735-5745, 2021.

Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems , 30(9):2805-2824, 2019.

Anru R Zhang and Yuchen Zhou. On the non-asymptotic and sharp lower tail bounds of random variables. Stat , 9(1):e314, 2020.

Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321 , 2019.

Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R Lyu. Improving adversarial transferability via neuron attribution-based attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14993-15002, 2022a.

Qilong Zhang, Chaoning Zhang, Chaoqun Li, Jingkuan Song, Lianli Gao, and Heng Tao Shen. Practical no-box adversarial attacks with training-free hybrid image transformation. arXiv preprint arXiv:2203.04607 , 2022b.

Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable adversarial perturbations. In Proceedings of the European Conference on Computer Vision (ECCV) , pages 452-467, 2018.

A Layer normalization

Layer normalization [Ba et al., 2016] is similar to batch normalization. However, unlike batch norm, Layer norm computes the mean and standard deviation over all activations in a layer. This further means it can be used in the online setting or in the test setting without caching statistics, as the statistics are calculated in real time.

Now, it is clear that if the input is of the form ( N,H ) where N denotes the minibatch size and H the hidden dimension, layer normalization ensures that each instance i satisfies, for a layer of dimension d over all activations H i,j :

i.e. a hyperspherical space. This means our assumptions hold exactly. However, this assumes the layer normalization occurs over the entire hidden dimension. This may not necessarily be the case. For example, if the dimensions are ( N,C,H,W ) it is possible that normalization only occurs over one of the dimensions of C, H, W . However, in this case, we can still recover an appropriate hyperspherical latent by summing over the remaining dimensions.

B Training Details

Recall that we use the losses of the following form (section 2 . 1 of the main text) :

We take the last two layers upto which we have access as i, i ′ to sum the signs of the angular loss. We cut the α by 2 to adjust for the sum of the signs, resulting in the overall averaging of signs referenced in the text. Also, before switching over to the angular loss, we can keep using L init for more than the initial iterations. We utilize 20 such iterations before switching, at α ′ = α/ 40 . Hence, in total, our angular PGD attack consists of 20 iterations of finding an unsupervised radial direction of moving to the antipodal point in latent space, and 20 iterations of maximizing the angular deviation given the initial movement.

Projection back to the valid PyTorch Tensor space for the adversarial instance (adjusting for normalization layers) proceeds as per normal PGD methods. The ϵ values here are calculated in the normalized Tensor space i.e. after normalizing the [0 , 1] tensor Imagenet image with parameters of mean as [0 . 485 , 0 . 456 , 0 . 406] and standard deviation as [0 . 229 , 0 . 224 , 0 . 225] channelwise. The projection is such that it respects both the original tensor's range and the ϵ in the normalized space.

Normalization settings for Imagenet for Resnet were kept as-is from the pytorch examples 2 and also as per the Fixup repository 3 as well as for Efficientnet 4 . For training, we utilize SGD. Please consult the attached codebase for all code-based details. Everything was run on a single Tesla V100 GPU, on Torch 1.6, torchvision 0.7.0. All confidence intervals appearing in this paper were generated by bootstrapping and asymmetric intervals were written in ± of the max one sided deviation. We use the Wilcoxon sign-rank test for determination of statistical significance, with p < 10 -3 as the cutoff, we refer to anything more significant than this p -value as passing the test here on. Cosine learning rate is used to train the Fixup resnet as per [Zhang et al., 2019].

C Original ResNets

We recall that in this class fall 5 models, namely Resnet18 , 34 , 50 , 101 , and 152 . In terms of block count, they respectively possess 8 , 16 , 16 , 33 , 50 layers. All radial attacks are performed with the last 2 layers upto which we have access. Access is granted upto blocks 6 , 12 , 12 , 25 , 38 respectively. So for example, on Resnet-18 the radial attack uses the loss signal from blocks 5 , 6 alone to craft the example. Every value provided in every table is a percentage accuracy metric.

At higher values of ϵ , angular attack outperforms the FGSM one. The confidence intervals (constructed by bootstrapping, denoted by ± ) mark the 5 and 95 percentile confidence intervals and can be used to gauge statistical significance. The wilcoxon signed rank test passed for every case where the point estimate of the angular attack was better than the FGSM one, with the exception of top1 on Resnet-101.

D Fixup ResNets

Just as with original Resnets, here we have 5 models, namely Resnet18 , 34 , 50 , 101 , and 152 . In terms of block count, they respectively possess 8 , 16 , 16 , 33 , 50 layers. All radial attacks are performed with the last 2 layers upto which we have access. Access is granted upto blocks 6 , 12 , 12 , 25 , 38 respectively. So for example, on FixUpResnet-18 the radial attack uses the loss signal from blocks 5 , 6 alone to craft the example. Every value provided in every table is a percentage accuracy metric. In short, everything is performed just as with the original Resnets.

Unlike the original resnets, the angular values do not even come close to the FGSM counterparts. This strongly suggests that removal of batch norm completely fixes this mode of vulnerability.

E EfficientNets

Here we have 5 models, namely EfficientNets B1 to B5. In terms of block count, they respectively possess 23 , 23 , 26 , 32 , 39 layers. All radial attacks are performed with the last 2 layers upto which we have access. Access is granted upto blocks 17 , 17 , 19 , 24 , 31 respectively. So for example, on B1 the radial attack uses the loss signal from blocks 16 , 17 alone to craft the example. Every value provided in every table is a percentage accuracy metric. In short, everything is performed just as with the original Resnets.

With the re-introduction of Batchnorm, the radial attack is again competitive and beats FGSM. We re-perform the same procedure as for Resnets to determine statistical significance against FGSM. The wilcoxon signed rank test passed for every case where the point estimate of the angular attack was better than the FGSM one.

F Results on ViT models

These results should be taken in context of our discussions on Layer normalization. Note that all models are from the PyTorch implementation of ViT [Dosovitskiy et al., 2020].

For context, we show the clean accuracies as follows :

G Ablations on Resnets-18,34,50

We first check if the independence structure of latents grows more independent with depth by examining the cross-diagonal average absolute correlation across block groups in Resnet-18. This is seen to decay with depth.

Now, we present results that exhibit the variation in accuracies as the layers being attacked grow further in the network. As a reminder, resnet-18 possesses 8 blocks divided into 4 groups as [2 , 2 , 2 , 2] while resnets 34 and 50 both possess 16 blocks divided as [3 , 4 , 6 , 3] . We will use 1 -based indexing to denote the layer upto which we have access, and the last 2 layers upto which we have access will provide the entire signal. Under the Wilcoxon sign rank test, the fall was monotonic, as suggested by the point estimates.

The point we wish to make clear is this : the fall is fairly unidirectional across all models with depth, as suggested by our previous analysis. Fixing the ϵ = 0 . 03 , which shows the biggest gap between layers as larger values of ϵ decrease every accuracy to zero, we can examine more layers to see this for the angular attack :

H Ablations with respect to number of iterations, number of layers averaged over and ensembling

Table 37 reports the difference in performance with respect to the default case ( t init = 20 ) as we vary the number of initial iterations. All results were performed on the vanilla Resnets. Positive values indicate stronger attack. There is no clear trend except that 20 is actually not the optimal value (15 is a bit stronger).

Table 38 compares performance of our attack as we vary the number of iterations. Negative values indicate worse performance relative to baseline (40 iterations). We can see there is some gain in moving to 50 iterations. However the standard evaluation practice is with 40, so we did not optimize over this step.

Table 39 compares the performance of our attack as we vary the number of layers that we average over. Negative values are worse relative to baseline (2 layers to be averaged over). We can clearly see the 2-layer averaging is actually quite helpful. Bigger gains are possible with 3 or 4 layers averaged over, but at the cost of more evaluations.

Table 40 reports the performance of ensembling targeted and angular losses. Positive values indicate gains with respect to the angular attack on its own.

I Baselines of Targeted Attacks and Stronger Random Attacks

Here, we check if providing a latent, generated from an instance of a different label, as the initial direction of ILAP is useful as opposed to using -|| z i || . To be clear, in this case, || z i -z j || is minimized, where z j arises from an instance of a different label. Then, the initial deviation is increased via ILAP.

We also construct an alternative random baseline, termed 'naive ILAP'. In this, an initial random deviation of a random, not necessarily batchnormed layer (we select a random ReLU layer per instance), is increased via ILAP, in the L 2 norm.

We run these baselines for ϵ = 0 . 03 , 0 . 06 as at 0 . 1 the values rapidly approach zero for any method. It is apparent that the targeted method is roughly on par with ours and the naive method performs worse. Hence, our method is basically as good as having these points to perturb towards, 'for free' just from the manifold structure.

J Transfer Results on CIFAR-10 and CIFAR-100

We show accuracies under the radial attack when Resnet-18,34,and 50 are subjected to it after transfer learning on CIFAR-10 and CIFAR-100. We also show the corresponding baseline drops under FGSM, PGD, random noise and no noise. For Resnets, the final block group (i.e. for example, Resnet-50 has [3 , 4 , 6 , 3] as its block groups, so the last 3 ) is tuned along with a linear classifier. The wilcoxon signed rank test passed for every case where the point estimate of the angular attack was better than the FGSM one.

Note that our attack is markedly more successful on CIFAR-100 and in general seems to reduce the classifier to randomly guessing and thus has difficulty lowering it below 10 and 50 percent on CIFAR-10 for top1 and top5 respectively.

J.1 Results on the Caltech Birds Dataset (CUB)

These experiments were performed on Resnet-50 on the Caltech Birds dataset [Wah et al., 2011]. Everything in the setup is exactly as per the CIFAR-10 and CIFAR-100 case.

K Transfer results between architectures

First, we consider results of adversarial attacks as we transfer between different types of Resnets. We consider the difference in transferability between our method and that of PGD for all cases (positive values denote our attack is stronger).

We also will show a case where the model being attacked can be a non-batchnormed model, in this case, VGG16 [Simonyan and Zisserman, 2014]. The results appear below. In all cases, the source model is a Resnet, the attacked model is VGG16, and the result is the difference between PGD and our attack (positive values denote our attack is stronger).

L Interactions with defenses and robust models

We consider the most key defence - adversarial training [Shrivastava et al., 2017]. For this process, we carried out adversarial training with an even mixture of FGSM, PGD, and radial attack (1 each for each training instance) for Resnet-18,34,50 for 20 epochs. Even after the training process, the radial attack remained effective. We show the results so obtained.

We also considered various robust models and found that our method had, on average (averaging over all ϵ and all robust checkpoints) a 6 . 8% advantage over FGSM in top-1 accuracy and 18 . 9% in top-5. We also note that our achievement was consistent over all ( ϵ , model) pairs : the lowest top-1 advantage was 2 . 6% (positive) and lowest top-5 advantage was 3 . 4% . Therefore, it seems the canonical adversarially robust models also do fall prey to our attack (at least to a greater extent than to FGSM).

M Ablations for affine-shifted batchnorm and for non-batch-normalized layers

Suppose that a batch normed representation ( Z ) is followed by a linear layer ( AZ + B ) . We note that if Z is perfectly isotropic Gaussian, we can estimate that E ( Z ) = 0 , and Cov ( Z ) = I . Correspondingly, if we are given the linear layer, we can have that E ( AZ + B ) = E ( B ) = B . We also have that Cov ( AZ + B ) = Cov ( AZ ) = A Cov ( Z ) A T = AA T . Therefore, if we can estimate the empirical covariance matrix from observations of AZ + B , we can find AA T (approximately). We can also factorize this to get BB T = AA T . It is known in this case that A and B are related by an orthogonal transformation i.e. A = BQ for some orthogonal matrix Q . So, we can recover Z (up to an orthogonal rotation) by inverting B as if we have AZ + B , we can subtract B (by estimating B via E [ AZ + B ] ) and then having AZ , multiply with B -1 to get B -1 AZ = QZ . Now an orthogonal matrix is a rotation, and the Gaussian ( Z ) upon a rotation ( Q ) is also a Gaussian ( QZ ) (the spherical geometry we have shown is also invariant under rotation). So our results carry over if there are sufficient samples to estimate the covariance. Note that this assumes A is not singular.

However, in practice, this covariance estimation can be very expensive and there are not enough samples. The exact inversion must be replaced by a pseudo-inversion which works but has less justification. Since we require more samples there can also be the question of more powerful methods becoming available which can utilize more samples more efficiently. We can also try our method on AZ instead of trying to invert A (this is simpler as it just requires calculating B , and AZ is also Gaussian). We were able to recreate attack numbers close to our results in the main text for all resnets with 5,000 samples. It can be seen that the results are close to the actual angular attack.

Our hypothesis is that BN makes the feature after BN Gaussian. Let us write the flow of representations as :

where X is the original feature, NL is the last nonlinear layer before BN, and Y is the pre-BN layer. Our prediction is that Z is Gaussian.

If, instead of the post-BN feature ( Z ) , the pre-BN feature ( Y ) is attacked, the result will still succeed, as the BN mapping is an invertible linear mapping in inference phase, and thus if Z is Gaussian, so is Y . (Our hypothesis works for any Gaussianized layers). But this is not a refutation of our method, because Y is Gaussian due to the existence of the BN layer as well and the cause is the same. Even Z ′ is a valid target.

This does not mean all layers are valid targets and there is nothing special about BN - our method does not over-predict and is very precise. Only Y and any linear maps on Z enjoy the property that if Z is Gaussian, so are they. X is not under such a guarantee. We now attach results by moving all corresponding attacks from Z to X , and the results are uniformly worse across the board, showing that the attack benefits from attacking Gaussian layers. Thus, the overall hierarchy of success is attacking a batch normalized layer > attacking a linearly-transformed batch norm layer > attacking a layer not related to batch norm by any linear fashion.

N Visualizations

We show some visualizations of adversarial images created by the radial attack on Imagenet to contrast to FGSM and PGD in table 60. As might be expected, the images sometimes look very similar to the baseline images even at ϵ = 0 . 1 , and sometimes have clear noise over them.

Now, has our method just rediscovered what FGSM and PGD do, or is it independent ? To answer this, we checked the absolute L 1 distance between various images at ϵ = 0 . 1 . The results are as follows :

Average L 1 distance of 15887 . 2 between FGSM and angular attacks

Average L 1 distance of 10282 . 5 between PGD and angular attacks

Average L 1 distance of 12866 . 4 between FGSM and PGD attacks

Average L 1 distance of 8968 . 8 between the clean image and the angular attack.

As can be seen, our method is not too similar to either FGSM or PGD and instead is closer to the base image, indicating that PGD and angular attacks diverge in opposite directions from the original image and do not merely copy each other. This is good as we are not just re-creating some other method. Indeed the closeness of our method is more to PGD than to FGSM, which is a good sign, because PGD is the superior attack.

Baseline

FGSM

Angular

PGD

O Extensions to the With-label Case

Here, we 'ensemble' our loss with the label-based loss via ILAP. We set an initial direction by adding the signs of the radial losses with the classification loss, then maximize the angular deviation. (This is done and scaled in the same manner as Ψ in the main text's algorithm). Since even at ϵ = 0 . 03 the attack brings the top1 accuracies near zero (less than 0 . 1% ) under PGD, we compare the top5 accuracies which are noticeably improved. In the other cases, both the methods result in near-zero accuracy.

P Single-Stage Benchmarks

Previous latent-representation based attacks often plan for the labeled scenario and add a latent loss to a label-dependent loss to formulate an overall supervised attack. Since our method focuses on the unsupervised case, we remove the label-based component from several state of the art methods, namely from [Zhou et al., 2018, Inkawhich et al., 2019, Ganeshan et al., 2019, Inkawhich et al., 2020b, Zhang et al., 2022a, Wang et al., 2021, Inkawhich et al., 2020a] for a comparison against ours. Most of these methods may be characterized as a single-stage latent disturbance method which initializes a random perturbation and moves in the L 2 norm in latent space. That is, they resemble our method but lack its angular viewpoint and justification. We take the best of each method. The performance, though noticeably above the random benchmarks, is below ours. For convenience, Table 3 in the main paper is repeated as Table 62 below.

Table 63 reports the difference in performance for each single stage attack with respect to the best attack (denoted by '-') whose performance is reported in Table 62

Q Results on the Hypersphere

On hyperspherical representation spaces, the concentration of norm is perfect. We can ensure this by concatenating the batch norm layers with a hyperspherical ( L 2 ) projection layer and re-training the model. The corresponding angular accuracies are presented below on Resnets18 , 34 , 50 for Imagenet. It can be seen that these cases perform roughly as well as the batch normed scenario.

R Discussion of the Concentration of Norm

From here on, we discuss the case of the concentration around expectation of a random variable V of the following form, with M i 's being other random variables of mean 0 , variance 1 :

We can see that V is the squared norm of the batch norm latent of interest and each M i is an individual hidden neuron, i.e. going dimension-wise along the latent. To avoid writing the square, let us always write :

We understand concentration to occur with associated functions f ( N,δ ) , g ( N ) - where g ( N ) is usually a monotonically increasing function in N obeying g ( N ) →∞ when N →∞ - to mean any result that shows :

Note that by linearity of expectation, E ( V ) = N . (This holds even if the M i 's are not independent.) Ideally, this function g ( N ) will obey:

δ →∞ implies f ( N,δ ) → 0 - large deviations are unlikely

g ( N ) N → 0 as N →∞ -g ( N ) grows slower than N .

If we combine the above points, we get that ∀ δ > 0 , ∀ ϵ > 0 , ∃ N such that

When q ( N ) is any function which obeys q ( N ) g ( N ) →∞ as N →∞ . That is, we can always increase the sample size so that deviations from the expectation are small relative to a monotonically increasing function of N which increases faster than g ( N ) that is, q ( N ) . In particular, consider q ( N ) = N and g ( N ) = √ N . This implies that if our conditions are met, f ( N,δ ) → 0 as N →∞ , and V E ( V ) → 1 . In particular, generally we aim for Gaussian concentration . Let the variance of V be S ( N ) , a function of N . Then we hope to have f ( N,δ ) = O (exp( -δ 2 )) , with g ( N ) = √ S ( N ) . This is alright so long as S ( N ) is o ( N 2 ) making g ( N ) as o ( N ) . Thus g ( N ) gives the size of the tails, and f bounds the tail concentration .

Clearly, the above f, g depend on two things: how the M i 's are distributed, and how they depend on each other.

We next note some realistic conditions and situations under which concentration and similar bounding will occur in V and its feasibility in our context (a neural network hidden layer, post batch normalization). We proceed from the most general to the most assumption-heavy cases.

M i 's are all distributed in an unknown fashion, possibly not independent of each other, but we can check the covariance between any two P i 's, and the variance of each P i is finite. Then, we have that, with S ( N ) the variance of V , S ( P i ) variance of a P i , CS ( P i , P j ) the covariance of any two P i , P j :

We have shown that in practice the correlation terms (off-diagonal, i.e. covariances) do fall with depth on Resnet-18. This makes this approach the most general. Once we have the variance, we can plug in Chebyshev's inequality [Markov, 1884] which yields that deviations of k √ S ( N ) occur with a probability ≤ 1 k 2 . That is,

This bound is extremely loose in practice, yet it is the most applicable. This is because we can usually estimate the covariance terms directly with reasonable precision and need not make further assumptions.

We know the M i 's are dependent, and we know that they are Gaussian, and that they are jointly Gaussian. Let the covariance matrix describing the M i 's be Σ . Let LL T = Σ . Then L -1 M where M is the vector of all M i 's is an isotropic Gaussian random variable i.e ∼ N (0 , I N ) = Z N . And, we can see that V = M T M is then ( Z N ) T Σ( Z N ) . This is a quadratic form of a Gaussian random variable. The result is known - it is a non central chi squared distribution. Specifically, let µ 1 , µ 2 , . . . , µ N be the eigenvalues of the matrix Σ (which is real symmetric and must have all eigenvalues as real), then the distribution is :

Where, χ 2 1 denotes the chi-square distribution of degree of freedom 1 . In this case, concentration bounds should be derived upon careful examination of the actual covariance Σ . However, in most cases, bounds obey [Bodenham and Adams, 2016, Laurent and Massart, 2000, Zhang and Zhou, 2020, Casta˜ no-Mart´ ınez and L´ opez-Bl´ azquez, 2005] at worst an exponential concentration, i.e. :

In certain cases, numerical methods are feasible, and one can also use Chernoff's inequality on the moment generating function. In this case, the assumptions are purely on how M i 's are distributed, we do not ask for independence.

Now, we move on to noting how independence may offer alternate ways of bounding V . For the following cases, we assume the M i 's and hence the P i 's are independent.

M i 's are independent, and all P i 's occur in a bounded range. Note that this is nearly impossible to guarantee and indeed this is not true if M i is a Gaussian. However, we can modify this to the condition that, with probability ≥ 1 -p bound , P i occurs in a bounded range. Then, we can apply the union bound to get that, with probability ≥ 1 -Np bound , all P i 's occur in a bounded range. This is reasonable if we expect at most some p bound fraction of outliers. In this scenario, we may employ either Azuma's inequality [Azuma, 1967], Hoeffding's inequality [Hoeffding, 1994], or McDiarmid's inequality [McDiarmid et al., 1989] to get the sought result : V concentrates with Gaussian-like concentration,i.e. with probability ≥ 1 -Np bound :

where g ( N ) (replacing the √ S ( N ) ) is an upper bound derived from the range of occurrence, e.g. g ( N ) = √ ( b -a ) 2 × N (this is correct upto constant factors for all of the three inequalities i.e. Azuma/Hoeffding/McDiarmid). Note that any random variable which must occur in the range [ a, b ] has variance upper bounded by ( b -a ) 2 . S ( N ) thus scales as ( b -a ) 2 N - we replace it with g ( N ) here and this is the link between the two methods. This gives us that, with high probability, the deviation around E ( V ) is of O ( √ N ) . When we have a notion of the variance of P i as well, we can utilize Bernstein's family of inequalities [Bernstein, 1924] for a more accurate bound.

In the case where the M i 's are sub-Gaussians - i.e. random variables which for some constants α, β obey, taking M i as the example :

Trivially, a Gaussian random variable is sub gaussian. Then, the P i 's are sub-exponential random variables. They are known to exhibit [Vershynin, 2018] mixtures of Gaussian ( exp( -δ 2 ) ) and exponential ( exp( -δ ) ) tail concentrations (RHS), with g ( N ) ≈ O ( √ N ) . The exponential tail does not occur for small deviations. Note that this relies on the moment generating function of M i . In this case, no bound on the support is required.

In general, all the inequalities above rely on some variants of Chernoff's inequality [Hagerup and R¨ ub, 1990], which is merely Markov's inequality on the moment generating function.

R.1 Empirical Findings

To corroborate our findings, we define the dispersion ratio . Given an univariate random variable Z , which always has Z > 0 of mean µ , the dispersion ratio D [ a,b ] with b ≥ a, 0 ≤ a, b ≤ 1 is defined as :

where F, G are respectively inverse CDF (quantile) functions of Z and a chi squared distribution with degree of freedom equal to the empirical mean of Z (rounded). We do not use the true mean µ , as it is unknown. In simple terms, this ratio measures how close the tails of the random variable match Chi-square tails. It can be 0 - indicating a delta distribution and perfect concentration - as well. When Z has chi square tails, D [ a,b ] ≈ 1 for most values of a, b .

We performed the following test. For Resnets-18,34,50, we picked out layers at depth 1 / 4 and 3 / 4 (upto the closest integer, discounting initial non-repeating blocks) into the network and randomly sampled a, b 1000 times uniformly from [0 , 1] to calculate the empirical dispersion (on average), with the random variable being the square of the latent norm at depth i i.e. ∥ N 1 ,i ( x ) ∥ 2 . By hypothesis, when the individual neurons are gaussian, this r.v. will be a sum of squares of Gaussians, each of which is a chi-square, and will be another Chi-square. We also did the same for FixUp Resnets. The results indicate that deeper layers are much closer in dispersion, and that FixUp Resnets are far more dispersed (values are far from 1 ). This strongly supports our conclusions which line up with conjectures in [Daneshmand et al., 2020, 2021] and the general 'Gaussianization hypothesis' [Neal, 2012, Yang and Schoenholz, 2018, Yang et al., 2019].

S Limitations, Ethical and Societal Impact

S.1 Limitations

Our work is necessarily limited in that it tackles deep networks utilizing batch normalization. As such, it cannot be easily extended to networks utilizing alternative forms of normalization such as LayerNorm [Ba et al., 2016] or networks that employ no normalization whatsoever. We fully acknowledge these limitations and do not claim to have crafted an adversarial attack that, in this present form, succeeds against batch norm free networks. Instead, we are incentivizing movement away from batch normalization altogether, and view our work as located closer to research which analyzes batch normalization [Santurkar et al., 2018, Galloway et al., 2019, Benz et al., 2020] than to proposing new methods of adversarial attacks, although it exists at the intersection of both these aspects.

S.2 Broader impacts and safeguards

Research which proposes avenues of adversarial attacks may be misused by malicious parties to attack objects in the real world [Hendrycks and Gimpel, 2016, Athalye et al., 2018]. Nevertheless, such research is important to analyze the actual robustness of the model itself and ultimately helps create better systems in the long run. Thus, we view our work as a net societal positive in the long run, even if its short term benefits may not necessarily be so.

We do not presently believe our attack requires specific safeguarding against real world malicious actors. If this situation changes, we would recommend that the real world systems in question have their normalization processes changed, e.g. via fixup.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows that for batch normalized deep image recognition architectures, intermediate latents that are produced after a batch normalization step by themselves suffice to produce adversarial examples using an intermediate loss solely utilizing angular deviations, without relying on any label. The success of the proposed method implies that leakage of intermediate representations may create a security breach for deployed models, which persists even when the model is transferred to downstream usage. The proposed attack also empirically works even for transformer architectures (ViT) that employ Layernorm over batchnorm.\n\nSTRENGTHS:\n1.\tThe proposed method is well motivated and technically sound.\n\nThe proposed loss is motivated by the geometry of batch normed representations and their concentration of norm on a hypersphere and distributional proximity to Gaussians. Through theoretical analysis, the authors show the contribution of batch normalization to an unsupervised radial attack and the optimal layers for the latent to lie near the end of the network. These theoretical findings are also supported by their experimental results. Therefore, the proposed method is well motivated and technically sound.\n\n2.\tThe experimental results are promising.\n\nComparing to single-stage attacks, the proposed method can obtain higher attack success rates, which shows the effectiveness of the proposed attack method and loss function.\n\nWEAKNESSES:\n1.\tThe significance of the proposed method is not clear.\n\nWhile the proposed method can get rid of the reliance on label information, as shown in their experiments, the obtained attack success rates are inferior to well-recognized attacks, i.e., PGD. Therefore, it is unclear what is the significance of the proposed method. Since the authors consider white-box settings, attackers can use white-box attacks, like PGD, with predicted labels of the target images, which can also avoid the usage of ground-truth label information. A potential application of the proposed method seems to be the transfer learning setting. However, from the experimental results, i.e., Table 10, the proposed method is also inferior to PGD. Therefore, it is unclear what the proposed method can be used for.\n\n\n\n2.\tSome claims are not validated or clearly explained.\n\n1)\tThe authors failed to clearly explain the meaning of Figure 1, which is closely related to their motivation. The notation of Figure 1 is different from that in the text, which can be confusing.\n2)\tThe authors claimed that “the 2-step process is key.” However, they failed to explain why the proposed 2-step process is better than 1-step methods.\n3)\tThe meaning of lines 171-173 is confusing, can the authors further clarify their claims?\n4)\tWhen we have a different latent Zj from a point Xj of a different label from Xi, the authors proposed to use this latent in their loss function to generate adversarial samples. However, this scheme is inferior to the proposed method. Can the authors explain why?\n\n3.\tPresentations can be improved.\n\n1)\tThe authors split the attack results of their method and baseline methods into different tables, i.e., Tables 1-4 and Table 8, which makes it inconvenient to compare different attacks. The authors can merge these tables for better readability. \n2)\tThe authors claimed one of their contributions as “To improve the supervised attack case when labels are available by using our loss in conjunction with the loss from the true label.” However, the corresponding experimental results are put into the appendix. This reviewer would suggest the authors put these experimental results into the main paper to support their claim.\n\nQUESTIONS:\n1.\tWhat can the proposed method be used for?\n\n2.\tWhy is the proposed 2-step process better than 1-step methods?\n\n3.\tThe meaning of lines 171-173 is confusing, can the authors further clarify their claims?\n\n4.\tWhen we have a different latent Zj from a point Xj of a different label from Xi, the authors proposed to use this latent in their loss function to generate adversarial samples. However, this scheme is inferior to the proposed method. Can the authors explain why?\n\nLIMITATIONS:\nThe authors have acknowledged the limitations of their work that it cannot be easily extended to networks utilizing alternative forms of normalization such as LayerNorm or networks that employ no normalization whatsoever. It is acceptable that this study focuses on a specific but widely used module in modern neural networks. Please see the weaknesses section for the other limitations of this work.\n\nThe authors have discussed the potential negative societal impact of their work. The authors also proposed a mitigation strategy to alleviate potential harm of this study.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors proposed a label-free attack utilizing a portion of all layers, which does not require to have gradient access to the full model, and the generated adversarial methods generalize to the case where the model was fine-tuned afterwards. These results have relevance at the intersection of the theory and practice of adversarial robustness and the growing study of batch normalization and its weaknesses and drawbacks.\n\nSTRENGTHS:\nThe proposed attach method is a label-free method, and it is a quite strong one. Moreover, the authors use the diagram to demonstrate the idea on their proposed method, which is very clear. The paper is well-written, and easy to follow. Moreover, the authors extended their work to the layer normalization case, showing the proposed method is a quite generic one.\n\nWEAKNESSES:\nThe proposed method only tested in the white-box setting, which means the adversary has the access to all parameters of the model, while in practice, this might be not practical, as usually the service provider only provides the api for the service. I was wondering how the model performs under the black box setting.\n\nThe paper mainly presents the experimental results, but lacks of some theoretical insights why intuitively it would work.\n\nQUESTIONS:\nSee the weakness. Besides, I was wondering how the initial stage affects the attack results, i.e., say we have total budget on calling gradient of $L$, then to best use this method, how should we trade off the $t_{\\textrm{init}}$ and $t_{\\textrm{radial}}$.\n\nIn figure 2, it shows that the method does not work well on fixup resnet. I was wondering if the authors could conduct more experiments on different architectures to have more evidence.\n\nSome typos: for example\n\nLine: above Line 145, should be $\\frac{<z, z'>}{||z|| ||z'||},$ and Line 145 starts with `where`.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
The authors present and evaluate an algorithm to construct adversarial examples without labels by minimizing the cosine similarity between intermediate layers. The authors show that the attack only works with BatchNorm.

STRENGTHS:
- The paper presents strong evidence that the attack is successful on multiple datasets and with multiple architectures

- Despite not achieving state-of-the-art attack success rate when compared to attacks with labels, the paper demonstrates that attacks on BatchNorm architectures are possible without knowledge of the labels, which is required by previous methods

- Paper is generally well written and thorough

WEAKNESSES:
- There are some clarity issues regarding the explanation for the attack, see Questions

- The authors make the assumption that norm is concentrated, and make an argument that it should be concentrated based on statistical assumptions about the data, but it seems that this should be verified empirically by plotting the distribution of the norms, rather than relying on an imprecise argument.

- The assumptions aren't verified to be necessary. It would be helpful to include an ablation where the norm concentration is broken.

QUESTIONS:
- I don't fully understand why the attack relies on concentration of norm. It's true that without concentration of norm then for a latent representation Z that there is no unique "most dissimilar point", but presumably points opposite Z would still be dissimilar without this assumption.

LIMITATIONS:
Authors address limitations adequately.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

comments: SUMMARY:
This paper proposes an adversarial attack (angular attack) method that does not require any label information and works by only accessing the network's first part (up to a specific layer). The attack is based on the assumption that the BN layer converges and forms a hyperspherical latent space, where an angular loss is applied to guide the direction of the adversarial updates.

STRENGTHS:
1. The proposed angular attack requires no labels, and only partial access to the network.
2. The hyperspherical assumption about the geometry of BN makes sense. Both positive and negative results (e.g., angular attack on Fixup ResNet) support this assumption.

WEAKNESSES:
1. Although Figure 2 is straightforward and informative, the other table results, including those in the appendix, are unclear. For example, it is unclear whether the numbers in Table 9 are absolute mean correlation or the fall in absolute correlation. Table 4 and Table 47 in the appendix are identical tables that only report the “max” over seven baseline methods. I think it makes more sense to include the full results of each baseline.
2. There are some missing ablation studies that I think should be included in the experiment section. a. How does the number of iterations affect the method (currently, the experiments only use a default iteration of 40). b. The current angular loss is computed by averaging the loss over the last two layers; I'd like to see more results of computing losses over more layers or just the last layer.

QUESTIONS:
1. Table 10 (b) shows that the targeted attack performs worse than the original angular attack. Does that mean the proposed attack method cannot be used for targeted attacks (when some label information is available)?

LIMITATIONS:
See weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07