Batchnorm Allows Unsupervised Radial Attacks
Authors: Amur Ghose
Apurv Gupta
Yaoliang Yu
Pascal Poupart
Venue: NeurIPS 2023 poster (2023)
Keywords: Adversarial
Batch normalization
Robustness
Geometric
radial
TL;DR: Batch normalization can open up unforeseen attacks on deep neural vision architectures.

Abstract: The construction of adversarial examples usually requires the existence of soft or hard labels for each instance, with respect to which a loss gradient provides the signal for construction of the example. We show that for batch normalized deep image recognition architectures, intermediate latents that are produced after a batch normalization step by themselves suffice to produce adversarial examples using an intermediate loss solely utilizing angular deviations, without relying on any label. We motivate our loss through the geometry of batch normed representations and their concentration of norm on a hypersphere and distributional proximity to Gaussians. Our losses expand intermediate latent based attacks that usually require labels. The success of our method implies that leakage of intermediate representations may create a security breach for deployed models, which persists even when the model is transferred to downstream usage. Removal of batch norm weakens our attack, indicating it contributes to this vulnerability. Our attacks also succeed against LayerNorm empirically, thus being relevant for transformer architectures, most notably vision transformers which we analyze.

Batchnorm Allows Unsupervised Radial Attacks
Amur Ghose" 1.2, Apury Gupta', Yaoliang Yu!.2, Pascal Poupart .22
David R. Cheriton School of Computer Science, University of Waterloo,
Vector Institute. 'Columbia University
aSghose@uwaterroo. ca, apuvgupaa196@@mail.cmmm yaoliang yww.ungeuwwwwaarroooooooo ca, wtt.:oo.....ppppprrrrttttro.....
Abstract
The construction of adversarial examples usually requires the existence soft or hard labels for each instance, with respect which loss gradient provides the signal for construction of the example. We show that for batch normalized deep image recognition architectures, intermediate latents that are produced after batch normalization step by themselves suffice to produce adversarial examples using intermediate loss solely utilizing angular deviations, without relying any label. We motivate our loss through the geometry of batch normed representations and their concentration of norm to Gaussians. Our losses expand intermediate latent based attacks that usually require labels. The success of our method implies that leakage of intermediate representations may create a security breach for deployed models, which persists when the model is transferred to downstream usage. Removal of batch weakens our attack, indicating it contributes to this vulnerability. Our attacks also succeed against LayerNorm empirically, thus being relevant for transformer architectures, most notably vision transformers which we analyze.
hypersphere and distributional proximity
Introduction
Adversarial examples Fody are commonly defined as data instances which away in some norm (usually L.o) from an actual data instance Treal To humans, is small and Treat, Tady share labels, yet to a classifier neural network, they do not. Since their discovery (Szegedy et al. 2013 Goodfellow al. 2014). adversarial examples have spurred research in both attacking (i.e., generating adversarial data instances) [Akhtar and Mian 2018 Tramer et al. 2020) and defending (Yuan et al. 2019
neural networks against them. Common methods for generating adversarial images use loss gradients. pioneering attack is Fast Gradient Sign Method (FGSM) [Goodfellow et al. 2014). possibly strengthened via Projected Gradient Descent (PGD) [Madry et al. 2017] iterative form of FGSM. Defenses against these attacks have been derived, including adversarial training [Shrivastava et al.,
2017]. where adversarial images are fed to the network for training. Adversarial training is expensive but powerful [Salman et al. 2019] and scalable [Wong et al. [020 Shafahi et al. 2019 Defenses can give illusory security by obfuscating gradients [Athalye et al. 2018). An unsaid aspect of defense is hiding model parameters changing attacks from the white-box (model parameters known) scenario black-box (models only accessible via outputs) scenario, which makes them harder attack.
We consider the family of intermediate-level attacks, exemplified by members such as Intermediate Level Attack Projection (ILAP) (Huang et al., 2019) and its variants (Li et al., 2020b). In this, initial direction in the latent space of the hidden intermediate layers is determined via FGSM or other attacks. Gradient steps then maximally perturb the intermediate hidden representation to find a suitable Fado. unlike directly using the label. Such methods can outperform the method used for
"Corresponding author: Amur Ghose, email: 43ghose@uwaterloo..
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

the initial direction itself and be more transferable across different models. ILAP uses FGSM to set initial directions, but once the direction arises, the layers after the intermediate layer play no role.
We now ask Given an unlabeled data point x along with full access to a deep network upto K layers, where K is less than the network depth, could we craft adversarial example out of without any other information - c.g., the label of I any x' * x so long as the deep network utilized batch normalization ? We answer this question positively when x an image for several popular families of image recognition architectures, exploiting the hyperspherical geometry of latent spaces to propose an angular adversarial attack. The label access of the black box case is absent- we do not allow access to hard labels (true label of 2) or soft labels (logits obtained at penultimate layers). We adapt the method advanced in ILAP to remove the need for tabel-based FGSM, and proceed instead with an angular loss based on the geometry of batch normalization. Our concrete contributions are:
To provide label-free attack that relies on only gradient access upto internediate layers Ablation of our method to show it is not brittle to choices of intermediate layers, (small variations between using layers i + if is sufficiently deep)
To show prominent vision models- ResNets and EfficientNets fall to our attack. Removing batchnorm in ResNets via Fixup neuters the attack suggesting it is playing a role in this.
To show that the attack empirically works even for transformer architectures (ViT) that employ Layernorm over batchnorm.
To show that these attacks persist in the transfer learning setting, where the models are frozen upto a certain layer and then fine-tuned for downstream usage.
To improve the supervised attack case when labels are available by using our loss in conjunction with the loss from the true label.
This implies that releasing the first K layers of model publicly as a feature extractor, without any label access, can create adversarial attacks, which has security implications. Weaknesses persist even when the unseen, later layers of the network are tuned on different datasets (e.g. CIFAR 10/100) adversarial examples can be crafted for CIFAR without using the changed, later layers only using this first K. We exploit the radial geometry of batch normalization and assume the norm of the neural representations concentrates around a constant, allowing us to exploit the geometry formed. This indicates batch normalization may make networks not just less robust to conventional attacks [Galloway et al., 2019) but open up new attacks entirely. We ablate our method and compare to scenarios of naive baselines and having access to x' * x: to showease its competitiveness.
1.1 Relation Related Work
We tie together two areas : analyzing batch normalization from a geometric standpoint under convergence assumptions, and adversarial attacks with partial knowledge of architectures. Batch normalization has been analyzed from the standpoint of deep learning theory and model optimiza- tion [Santurkar et al. 2018). While its detrimental effects on model robustness [Galloway et al., 2019 Benz et al. [020] are known, most analysis focuses on how batch normalization makes the model weaker to already-exisst existing attacks. We propose that batch normalization might impact the network by making it more vulnerable to novel attacks. We are to our knowledge the first to connect this with the concentration of norm phenomenon implied by results such as the dynamics of batch normalization at high depths [Daneshmand et al.) 2021]. These tie into results such as neural collapse (Papyan et al., 2020), which find that later layers in deep networks have predictable geometric patterns.
Research in adversarial examples has focused in specific settings such as white/grey/black box settings denoting levels of model access/knowledge. Even in the black box case, is often assumed we can feed an input r into the model AM under attack and retrieve M (z). the model's prediction of the label of æ. We consider the case of partial model knowledge and attacking model with no label outputs. This is relevant with the presence of large-scale pretraining, where models are fed large training sets of data unrelated any one domain to capture the overall distribution of particular type of modality, such as GPT-3 [Brown et al.) 2020] with natural language. Deployed downstream, these public models may fine tune on a smaller dataset, and most of their weights are not fine-tuned. Public models may act as feature extractors that distill large amounts of data beyond dataset scopes. They output not labels, but representation vectors, and our attack targets this case. Previous works do use latent representations to construct adversarial examples (Park and Lee 202 I Creswell et al.)

---

2017 Yu et al. 2021] through moving in latent space towards points of differing labels or maximizing deviations latent space [Kos et al.) 2018]. Along with the related no-box [Li et al.) 2020al Zhang model knowledge), these methods require extra parts, such as representations of instances of different labels as targets to move towards, surrogate models, or some soft labels, often with gradient access all of which we do not require. They are also ad hoc in their choice of layers to attack, but we give a definite answer of which layers and architectures are the most vulnerable - batch norm layers situated deeper in the network. The closest case to our attack are label-free, single-stage latent-based attacks [Zhou et al. 2018 [nkawhhch et al. 2019 Ganeshan eet al., 2019 Inkawhich al., 2020b Zhang et | 2002a Wang et al., [202 Inkawhich et al.) [2020a] which we outperform. These methods use the latent to boost the supervised scenario and not exploit batch normalization or two-step optimization in an unsupervised setting as we do.
et al. 2022b] setting (assuming
|
1.2 Setup- FGSM and PGD Attacks
We describe FGSM aka Fast Gradient Sign Method [Goodfellow et 2014] for generating an adversarial example. The task is to find x' which differs from x in the label assigned by classifier C, which outputs a distribution over K classes (K positive numbers that sum to 1).
| - x E
and
arg (aaa(((())) + arg Nggmax(C(22)) :
(I)
Generally, we only restrict attacks to cases where arg max,(C(x)] is the true label, i.e., C' is correct. The norm ||||| used define the attack almost always the L norm, though other choices such as L2 [Carlini and Wagner 2017] have been tested as well. We will assume the norm to be Lxe for the description. Computing x' (possibly with projection to valid domain) proceeds as follows:
* Apply C' to x to obtain C(x).
Compute gradient V L. wrt where T. is the loss between C(x) and the true label vector un- Compute x = 24 + x sign( L). i.e., move by +e based on sign of gradient.
We understand the sign function to map to -1, 0, respectively as the argument is < O, 0, or >> O. FGSM uses gradient information, which may vary rapidly around x. A stronger variant, the Projected Gradient Descent (PGD) attack (Madry et al. 2017] uses multiple steps near taking gradients at each step. Under PGD, there is a step size a, typically << c, which iteratively updates of as:
z = of With a clamping step on ai that enforces the constraint of ||a' x c, this step after initializing 2! at i; yields much stronger adversarial examples than FGSM at higher t.
x sign(VL);
1.3 Intermediate Level Attack Projection (ILAP)
These methods FGSM and PGD rely on the input and the final layer's output loss with respect to the true label. We can consider a neural network N of depth D as
(z)) (N1,)
With 1 based indexing, Nz.1 is the neural sub-network via composing layers from depth to k. Let At = (1)) be the latent representation at depth Consider Fadr from any baseline method, and let
ade N (T,d())
2 orig = N1,((r): Addu adv
orig
2,
We then seek an alternate adversarial example, xz! LAP which seeks to work with the loss function
L (A A! !LLAP) where S!LAP !LAA 2, orig and 2 2!LAP (=!
The L can then be plugged into either FGSM or PGD. Put simply, this loss encourages latent space movement directionally similar FGSM/PGD. L increases the inner product with A Adde and later iterations use it over the true loss. This alternate adversarial example is highly transferable e.g.. adversarial examples from ResNet18 have higher success rates vs different models such as DenseNet, when created by an ILAP process than directly using FGSM. It can even beat FGSM on the source model itself. (Huang et al. 2019) ILAP demonstrates that latent representations provide ammunition for adversarial attacks, but requires an initial adversarial example Fade created with full model access and label. To remedy this, we craft an attack needing neither. We show that if the correct layers are attacked with an angular version of the same loss, the initial ad adversarial example is unnecessary.
3

---

2 Geometry of Batch Normalization
Given minibatch of inputs z; of dimensions n with mean Paj and standard deviation di at index j. consider batch normalization [lotfe and Szegedy 2013] or just BN as:
[2N()));
Fij Pij
(2)
Such normalization (with affine shift) has become commonplace in neural networks. During training, the empirical average of the minibatch is used and during testing, computed averages (mean/standard deviation) are used. By batchnorm, we mean only the batchnorm layer without an affine shift. Layer normalization [Ba al.) 2016] is alternate form of normalization that normalizes over all neurons in a hidden layer. For space reasons, we discuss batch normalization in the main text and layer normalization in Appendix[A] Empirically,
attack succeeds layer norm and architectures using it, such as transformers [Vaswani et al., [017] and vision transformers [Dosovitskiy et al. 2020
Converged regime: Suppose in training, the sample means/standard deviations in pre-BN layer converge to sample statistics. Post-batchnorm, representation vector Z satisfies, at every index j:
(Z,z 0, E (Z))
i.e., for a layer of dimensionality d and denoting the entire latent vector as Z, F(Z) 0, [[ d by linearity of expectation. The above convergence is with respect to the training set, but carries over to the test (approximately) under the I.I.D assumption when learning takes place. Suppose || was concentrated about its expected value i.e., nearly all Z|| values were vd. We first discuss the implications of such a scenario and what attacks they allow. We assume that in this latent space, the inner product determines 'similarity". Two latents Ze. Zi from different instances X. X₁ will be closer as (Za, Zo) rise. Under this metric, the most dissimilar point to Za is parallel to - -Ze- We will formulate our attack assuming that |= vd i.e. a hyperspherical latent space. After we have formulated our attack, we consider scenarios of such spaces and concentrations norm.
2.1 Angular Attack Based on Converged Batchnorm
Let N1., be the network upto layer with dimensionality d. If layer is batch norm layer (without affine shift), z (x) lies (with converged batchnorm and [|]] norm concentrated) approxi- mately on a shell of radius dd. The natural distance metric between two z, is angular distance:
(z,
| 2
where (2,2") is at local maximum on the hypersphere || constant when 2 = 2' and no gradient exists. We need another gradient initially, and propose the following attack algorithm. Given real example oº with latent 20 N1,(2"),,* = (='( we consider the loss T = = (=")], for the first aam iterations. We then iteratively generate xf:
+ x sign (VL
1151 S Linit
After finit iterations, we modify the angular loss to Leadial- using the deviations obtained so far. Lendial angularly defined, denoting 2 the latent at depth and iteration t:
(2) =t) ||
Tradial
No, Ni,a(x")
2
We then iteratively update as follows (graphically depicted in Figure
x 1 + a x sign (VLaal)
Aiiit
< t S fyaiial
The figure geometrically depicts how intermediately moving from a point to another in latent space lowers the radial norm (A,, left). Directly lowering the radial norm via a path from Ao to A₁ (right) can create an initial deviation. But extending this to C' as in the figure ends up with a lower angular deviation than maximizing angular deviation directly and ending up at Aadv- Methods which directly increase the Le deviation starting from initial method would result in points such C.

---

A
0
y
Ag
Ag
Figure L: A,C' are latent image representations. A, lies on chord AC. Targeted latent space movements lower norm. R: Initial radial loss towards O forms A with implied movement towards C. , follow. Implied movement towards C' further than C from A in 0 (angles).
Both losses depend the layer i. With access till layer D, multiple candidate i values exist. For simplicity, we only use D, D and replace sign (V Lima) with 2 [signn( V Lini, DD + ( Timit, where the subscript D-1 D-1 denotes setting as D 1, and similarly we set = and average the sign for sign( V Laddai). This averaged term becomes V/ in our pscudocode (Algorithm I We clamp x² iterates to satisfy || - = c. Our attack resembles ILAP, with the initial direction Adde from an unsupervised radial loss without labels. We then maximize the angular deviation from the original latent, and do not "follow the leader" on the original angular deviation after Linit. The intuition is as follows. The surrogate loss -| moves along chord in the hypersphere' interior from 20 towards -20, the most dissimilar point, while latent representations are on the surface We use the last 2 layers for simplicity stronger attacks might exist using all layers. Concentration of norm forms the hypersphere, and the 2-step process is key. 1-step methods, e.g. single-stage works discussed under related works, use methods such as random initialization and angular minimization, but perform worse (see Appendix P). The concentration is in L2, but the adversarial example is in the L metric.
Algorithm Angular attack algorithm
Input: Neural Network N; access model parameters of N till layer D, N sub-network of N upto layer é a starting real sample x perturbation a Result:Adversaria example foout
linic f Num iterations with loss Linit; (radial + Num iterations with loss Leadial
< 1, n +- iii + 1
while &s < fiit do
I - - (1,,
T sign (VLmD.-1)) sign (VI (VI
o x- --= + o v; xk c ,e) k ++1
end while while S tradial do
(2), (1, v sign V V..mm. D-1) sign (VIm-1 [1,
radial,
|| W xh +- clamp( (r" ,e); t- n
end while Return: zfona
Moving from 20 along V||2*|| does not move linearly towards the origin when the optimization is imperfect e.g. using the gradient sign. We manipulate xt, not z it may not be 3 x' such that
(x') = 72", vy, < y (the ray joining origin-2 may lack points with pre-images in x:). Another issue a movement direction-wise (ignoring sign averaging) e.g. let E ( the representation as identity map. Let x = (1,+) lie on the hypersphere of radius / + e², The sign of V Limit is (-,-) and d (1 a, a) not necessarily collinear with (0, 0) (1, Batch
5

---

normalization leaks mean/variance to adversaries. Suppose we had perfect network N (could find mapping to any 2, i.e. was onto), and were assured the latent space was zero mean hypersphere and we get adversarial instances if we move the latent from to - This would be insufficient if all latents were translated by some vector c, i.e., the mean of the hypersphere was nonzero. To find the point opposite on the new hypersphere (c- - (z c)), we need c. Batch norm gives us this This weakness outweighs knowing z' arising from x' of a different label, as we show in the results.
2.2 When Does the Norm Concentrate
We assumed that after a batch normalization step. ||] concentrates. Consider sufficiency conditions. Let Ziy be the j-th entry of the i-th latent Z, formed from X; with expectation 0, variance (batch norm). Consider independent instances X; and assume no instances indexed by were used for training impact model parameters, which affects the other latents and makes them dependent i.e. consider test/validation sets. Then, Zij: z are independent if Vj' = Now
d
12 22
This sums d random variables (assuming Z, has dimension d) and depends on independence structures and the marginal distributions of each variable. Under certain cases, independence of Zaj, Zag can occur. Suppose Zi was constrained in {-1, and had to encode 2n X;'s. Then, Zıj= Ziy independent under the optimal encoding, assigning for each X; a unique D-length code with entries {--, When Zıj. Ziz' independent, || 1= E, Zł will concentrate around its expected value of d under mild conditions e.g. bounded 4-th moments of Z,y (Chebyshev's inequality), but concentration tails may not be sub-Gaussian By tails of concentrations, we mean bounds of form:
P p ( 2 8) s f())
For example, sub-Gaussian concentration implies f()) = O(exp(-5²)). Suppose each Zij is marginally distributed a (sub)-Gaussian. With independence, Chernoff's inequality leads norm concentration for | with stronger sub-Gaussian tails over Chebyshev's (tails of (t). Inequalities and implied tails can be consulted from e.g. [Vershynin| 2018 We consider some previous results. For deep linear models of (ndd(diimnssionaaii d, we have [Daneshmand et al. 2020 2021
E|DRr (2DK, - (1- avd
Z is the distribution after N1.1 i.c.. i layers, a is a constant, b is the training batch size. The LHS (termed the orthogonality gap) indicates that the distribution of the latents convergences to the isotropic Gaussian using KI. divergence, which a stronger condition leading to the latent norms being clustered around vd. Though derived for linear networks, the original paper offers evidence that the conjecture holds for general multi-layer networks. This indicates deeper layers i.e., increasing i will have the largest effect on our assumptions holding, as this exponentially drops the KI. divergence increasing width (d). Beyond linear networks, general networks with depth and width reduce to Gaussian Processes (Yang al. 2019 Yang and Schoenholz 2018 Yang and Hul 2020 Yang 2019 Neal 2012]. These results usually apply to marginal distributions and not independence structures. Empirically, we find dependence between different variables (measured as correlation) falls as the network latent grows deeper, which aligns with theory. Dimensionality falls as the network deepens. recall the toy example of encoding 2n instances with D binary variables which forces independence. ILAP's empirical ablation studies found the optimal i to create the latents for the attack occurred in the range 0.6 to 0.8 (network depth normalized to 1). In sum, we expect our optimal layers for the latent to lie near the end of the network. Our methods work without assumptions in hyperspherical representation spaces [Wang and Isola 2020 [Schroff et al, 2015 Mettes et al. 2019). Analysis and empirical findings on the concentration and tails is included Appendix R
Gaussian (and concentrates the norm), with a secondary benefit from
3 Results
We carry out an extensive set of experiments on Imagenet [Russakovsky et al. 2015). utilizing several ResNet He et al. 2016) models ResNett-11s 34, 50, 101, 152) and EfficientNet models Tan and

---

30
63480 SAS
Escko s000
Original Resnet
Fixup Resnet
EfficientNet
Figure 2: Accuracies (%) of networks under FGSM (blue) and our Angular attack (red). (Color intensity proportional to network size). Accuracies under angular attack are lower without Fixup.
Le 2019 B-1 to B-5. We chose these models because they have a common structure construction. Except a few initial layers, they are constructed by repeatedly stacking the same blocks ("Basic" or "Bottleneck" blocks for ResNets, and Mobile Convolutional blocks [Howard al., 2017] for EfficientNets). Every block possesses at least one batch norm layer, allowing the extraction of the appropriate latent. All models were ran on Imagenet with a standard normalization pre-processing step. Results of all attacks on all models, a sanity check with a fixed norm model (explicitly hyperspherical, i.e. always concentrated in norm), improving supervised (label present) cases on top of FGSM/PGD, comparisons to single- stale attacks previously proposed, interactions with defence, ablations, visualizations. statistical testing, confidence intervals and other transfer learning results and comparisons appear in the Appendices. As the alternative to batchnormed models, we choose Fixup Zhang et al. 2019] different way initialize the models. Core results are summarized in Figure 2 On the batchnorm-fres free Fixup Resnet, the FGSM attack greatly outperforms our angular attack. On the other two architectures, this not the case. FGSM's performance and clean accuracy vary far less than the success of the angular attack. That implies overall model quality and robustness to common attacks remains the same, and vulnerability to angular attack specifically is what varies. We train Fixup resnets imagenet alternatives to the batchnormed models for Resnet only, as methods in Fixup do not generalize to efficientnets. Details of these training steps are the Appendix B and also attach our training and inference codebases. We obtained the models from publicly available repositories for PyTorch [Pazkk et al. 2019), namely torchvision and EfficientNets: PyTorch, and did not change the original weights. For the choice of for the ad adversarial attack, chose 0.03, 0.06, 0.1 and carried out all attacks using an a 0.01 over 40 iterations.
Vision Transformers and LayerNorm. Recently, Layer normalization [Ba et al. 2016) has emerged as an alternate mode of normalization batch norm. It finds prominent usage in the transformer architecture |Vaswani et al.) 2017] and for images, in the ViT' (Vision Transformer) models. We used our attacks on the case of ViT models (Dosovitskiy al. 2020) (B/L)-(16/32). In all cases, attacks were successful as with Resnets/batchnorm. This indicates our attack can succeed even when instead of batch normalization, layer normalization utilized instead. We discuss this in Appendix A but make batch norm the focus it has pre-existing theoretical analysis [Daneshmand et al. 2020).
d clean accuracies. Angular attacks especially in terms of their top accuracy. at higher and on deeper networks dominate FGSM attacks, while losing to PGD attacks. No random perturbation performs well. Confidence intervals rule out fluke (statistical tests Appendix C. Raw numbers for Fixup Resnets (in Appendix D exhibit a different pattern - the angular attack performs as well as random method, while FGSM, and PGD perform as normal. EfficientNets (in Appendix()) follow ResNets in their result. Table exhibit the max over single-stage benchmarks among Zhou al.) 2018 Inkawhich et al. 2019
We examine the results on all Resnets in Tables 223 with Table denoting
7

---

Table 1: Comparison of FGSM and Random noise, Top-1 accuracy, Top-5 in brackets.
Net Type
0.03
0.06
0.1
FGSM
Random
FGSM
Random
FGSM
Randorn
Resnet- 18
1.95 (21 933
69.40 68840 (88.94) 94)
1,16 (15.39) 39)
68.58 (88.51)
1.22 (13.74)
67.11 (87 (6)
Resnet-34
4.36 (30.54)
73.06 (9)
2.95 (23.65)
72.43 (90.96)
3.13 (22.08)
71.23 (90.26)
Resnet-50
8.18 (39.04)
75.79
6.62 (33.33)
74.89 (92.32)
7.10 (32.58)
73.57 (91 (91.5)
Resnet-10)
9.93 (43.27)
77.24 (93.52) 52)
8.32 (37 88)
76.79 (93.25)
9.10 (37.61)
75.57 (92.75)
Resnet-152
77.21 (93.78)
8.76 (39.23)
9.82 (38.25)
76.28 (92.98)
Table 2: Comparing Angular attacks, top-1/5 accuracy, with confidence intervals on different Resnets.
Type
0.03
0.06
0.1
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
23,416
1.222
40.034
12.12
2.422
-.0.22
5,980 12
0.454 1:011
1,602 00.39
7.164
1.28
14.320
22.48
0.592
+0.64
1.16 00.32
0,216 #0.04
0.814 +0.27
13.968
+2.82
24.796
25.22
22.22
3,680
+0.88
+0.12
1.478 20.26
101
7.012
10.62
13,886
11.23
0,874 + 1.6
2,606 10.78
0.390 10.122
1.346 0.
1.0.05
Table 3: Comparing PGD attacks, top-1/5 accuracy, with confidence intervals on Resnets.
Type
0.03
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
18
0.006 +:0 000
3.374
+0.04
0.0 +0.0
0.484 +0.04
0.0 =0.0
0.17 #0.02
4,38
0.006 20.001
40.01
0.004 #0.001
0.288
50
0.028
00.
8.076 1.22
0.006 +0.002
2.914 #0.4
0.0 =0.0
1.956 #0.3
0.032 +0.07
9.648 +1.76
0.008 00.002
3,902 60
0.006 00.002
2,702 =0.51
152
0.042 10.08
9.864 11883
0.014 10.003
4.16 20.73
0.015 10.004
3,253 =0.64
Table 4: Comparing single-stage attacks, top- 1/5 accuracy, with confidence intervals on Resnets.
Type
0.03
0.06
0.1
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
18
30.45
48.34
6.98
16.87
3.89
10.22
34
16.78
35.89
1.45
8.32
1.68
4.54
50
22.68
39.10
4.62
10.45
4.71
6.58
101
15.65
34.67
5.82
11.28
5.28
8.95
152
8.29
18.53
1.78
5.67
0.95
3.22
Table 6: Top- 1/5 accuracy for ResNets,
Table 5: Top-1 (& Top-5) accuracy for
Vision Transformers: 0.1
angular attack not on BN layer, e
0.03
ResNet
Top-1
Top-5
Model
FGSM
PGD
Angular
18
27.3
46.2
ViT-B-16
35.1 (60.6)
5.7 (18.5)
16.7 (28.5)
34
11.5
19.2
VIT-B-32
34.8 (59.4)
5.4 (21.1)
17.8 (31.8)
50
17.3
28.5
ViT-L-16
36.0 (65.6)
8.2 (17.7)
14.5 (33.8)
101
11.32
19.5
ViT-L-32
34.1 (61.7)
5.8 (22.9)
13.9 (38.5)
152
8.2
11.9
Table 7: Resnet-34 ablations. Last 2 layers of the net granted access are used for angular attack.
Access till
0.03
0.06
0.1
Top 1
Top
Top
Top 5
Top
Top 5
8
54.46
11.78
77.35
0066
18.44 10.45
32.68
10.68
4.95 10.86
11.68 11.26
9
35.84
+0.78
59.32
10.82
6.32 10.56
13.84 10.56
1.39 10.37
13.84 11.78
10
26.22
+1.56
41.5
1.58 0038
3.88
+1.02
0.60 +0.12
1.52 +0.37
13.20 +0.26
-0.86 22.68
0.94 =0.31
2.34
#0.22
0.34 #0.06
0.92 #0.17
12
6.40
#0.72
12.05 +1.89
0.45 +0.12
2.27
10.56
0.20 +0.08
0.65 #0.22

---

Table 8: Clean accuracies for Resnets
Table 9: Resnet-18 absolute correlations
ResNet type
18
34
50
101
152
Acce1
69.75
73.31
76.13
77.37
78.31
Black group
2
3
Accie5
89.07
91.42
92.86
93.54
94.04
Absolute mean correlation
0.41
0.32
0.19
Table 10: Angular transfer attack results and targeted benchmarks Angular results CIFAR-100 100, Resnet-50
b) Targeted benchmark Resnets
Net Type
0.03
0.06
Method
0.06
Top
Top
Top
Top
Top I
Top 5
Top
Top
Top
Top
Resnet-18
25.16
42.58
PGD
0.16
0.20
0.14
0.14
0.14
0.14
Random
53.1
38.63
Resnet:34 34
Resnet-50
13.55
12.2
30.97
21.8
0.65
1.9
2.86
5.16
FGSM
29.28
59.02
25.86
51.95
15.01
35.14
Resect-101
5.85
13.66
2.86
6.67
Angular
2.32
10.79
2.16
9.72
1.88
9.49
Resnet-152
9.68
16.77
1.29
3.87
Ganeshan et al. 2019 Inkawhich et al. 2020b Zhang et al., 2022a Wang et al. 2021 Inkawhich et al.] 2020a]. The performance is worse than our method. We also showease results of ViT models on 0.1 for imagenet in Table [ (Results for other values of are in Appendix Finally in Table 6, we attack Resnets with the angular attack but move the layer being attacked to the layer before the BN layer. This noticeably weakens the attack, suggesting relationship between the two.
We examine the drop in accuracy as a function of the access to various blocks. We run our attacks consistently accessing the 3/4-th layer of net, i.e., for a Resnet-50 of blocks [3, 4, 6,3] total 16- we use only the first 12 blocks and sum our angular losses over block 11, 12. On Resnet-34 access to deeper blocks strongly strengthens the attack in Table 7 This agrees with the theory of batch normalization discussed previously (Daneshmand et al., 2021] and with the empirical findings of ILAP Huang et al. [211]] which found that the most optimal layers to perturb lay between 50% and 80% of the network's length. Further depth ablations are in Appendix G Concentration of norm relies convergence of batchnorm and independence. Assuming the former, we can check independence by examining the absolute, off-diagonal correlations in the correlation matrices of the latent representations. We note a decorrelation effect with depth, which matches the success of our attacks. We compare the fall in absolute correlation among the latent dimensions in Table 9 across three "block groups" of Resnet- 18 which has 8 blocks organized as 2, 2, 2). Correlation proxy for independence decreases over the last three block groups
Transfer learning. For transfer learning, we add a linear classifier that can access the last extracted latent layer of the resnet, and we unfreeze this classifier and the last group of blocks. Every other layer is frozen. The setup is fine-tuned, downstream, on CIFAR-I 10 and CIFAR-100 Krizhevsky et al.) 2009]. We show results on CIFAR- with Resnet-50 in Table 10 Although Table [Ojindicates that targeted attacks perform worse than the original angular attacks, it does not mean that we cannot use the label information. only means that when we randomly pick an instance of a different label, the angular attack often does better. But, the combination of label and angular attack can do better than either attack, so having an instance of a different label can still be very helpful. We provide some cases where our loss is provided in ensemble with the targeted loss and these perform better than either (see Appendix HH. Results for CIFAR-10 other resnets/datasets are in Appendix Our method outperforms all but PGD.
Ablation against other cases. The radial loss an implicit movement towards the opposite point on hypersphere. This assumes that the hypersphere exists at all. If we have a different latent 2, from point X, of a different label from X,, we check if our attack might be more effective using an initial loss of the form Z Zj ||. Increasing this loss would be lowering the distance between Z,, Z, i.e. moving towards the other point in latent space. Yet, this targeted attack shown in Table TO is actually inferior to attack, which lacks the access.
4 Conclusion
We have shown a powerful, label-free attack which only utilizes portion of all the layers available for a network to construct adversarial examples that fool the entire network. succeeds at this without knowing the label or having gradient access to the full model, and these adversarial methods

---

generalize to the case where the model was tine-tuned afterwards. These results have relevance the intersection of the theory and practice of adversarial robustness and the growing study of batch normalization and its weaknesses and drawbacks. We provide support to the notion that batch normalization may open up unseen avenues of perturbations that unexpectedly impact model robustness, viewpoint supported by previous literature (Galloway et al. 2019 [Bnzz et al.) 20200] Wang et al., 2022]. We also extend our results to LayerNorm, which is increasingly relevant with the advent of transformer architectures Dossvviskkiy et al. 2020
Acknowledgments and Disclosure of Funding
Resources used in this work were provided by the Province of Ontario, the Govemment of Canada through CIFAR, companies sponsoring the Vector Institute hatp:://veetorinsstiii ai/partners/ and the Natural Sciences and Engineering Council of Canada.
References
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access, 6:14410- 114410-14430, 2018
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.000 004200000 2018.
Kazuoki Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical
Journal, Second Series, 19(3)337-367,, 1967.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. arXiv preprint arXfv:1607.06450, 2016.
Philipp Benz, Chaoning Zhang, and In So Kweon. Batch normalization increases adversarial vulnerability: Disentangling usefulness and robustness of model features. 2020.
Sergei Bernstein. On a modification of chebyshev's inequality and of the error formula of laplace. Ann. Sci. Inst. Sav. Ukraine, Sect. Math, 1(4):38-49, 1924.
Dean Bodenham and Niall M Adams. comparison of efficient approximations for a weighted sum of chi-squared random variables. Statistics and Computing. 26(4):917-928 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy (SP), pages 39-57. IEEE, 2017.
Antonia Gastaño-Marting and Fernando LláezeBlázquez. Distribution of sum of weighted noncentral chi-square variables. Test, 14(2):397-415, 2005.
Antonia Creswell, Anil A Bharath, and Biswa Sengupta. Latentpoison-adveesaiiii attacks on the latent space. arXiv preprint arXiv:1711.028799 2017.
Hadi Daneshmand, Jonas Kohler, Francis Bach, Thomas Hofmann, and Aurelien Lucchi. Batch normalization provably avoids ranks collapse for randomly initialised deep networks. Advances in Neural Information Processing Systems, 13:18387-18899 2020.
Hadi Daneshmand, Amir Joudaki, and Francis Bach. Batch normalization orthogonalizes repre- sentations in deep random networks. Advances in Neural Information Processing Systems, 34: 4896-4906. 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint 2rX1v:2010.11929, 2020.
10

---

Angus Galloway, Anna Golubeva, Thomas Tanay, Medhat Moussa, and Graham W Taylor. Batch normalization a cause of adversarial vulnerability, arXiv preprint arXiv:1905.02161, 2019.
Aditya Ganeshan, Vivek BS, and Venkatesh Babu. Fda: Feature disruptive attack. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 0069-8079, 2019.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv: 2014.
Torben Hagerup and Christine Rüb. A guided tour of chernoff bounds. Information processing letters, 33(6):305 308, 1990.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Early methods for detecting adversarial images. arXiv preprint 0rXiv:1608.00530, 2016.
Wassily Hoeffding. Probability inequalities for sums of bounded random variables. In The collected works of Wassily Hoeffding, pages 409-426. Springer, 1994.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: (00.00/674.44 2017.
Qian Huang, Isay Katsman, Horace He, Zeqi Gu, Serge Belongie, and Ser-Nam Lim. Enhancing adversarial example transferability with an intermediate level attack. Proceedings of the IEEE/CVF international conference on computer vision, pages 4733-4744 2019.
Nathan Inkawhich, Wei Wen, Hai Helen Li, and Yiran Chen. Feature space perturbations yield more transferable adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7066-70744 2019.
Nathan Inkawhich, Kevin Liang, Binghui Wang, Matthew Inkawhich, Lawrence Carin, and Yiran Chen. Perturbing across the feature hierarchy to improve standard and strict blackbox attack transferability. Advances in Neural Information Processing Systems, 33:20791-20801, 2020a.
Nathan Inkawhich, Kevin J Liang, Lawrence Carin, and Yiran Chen. Transferable perturbations of deep feature distributions. arXiv preprint arXiv:2004. 12519, 2020b.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv: 2015.
Jernej Kos, Ian Fischer, and Dawn Song. Adversarial examples for generative models. In 2018 IEEE security and privacy workshops (SPW), pages 36-42. IEEE, 2018
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Beatrice Laurent and Pascal Massart. Adaptive estimation of quadratic functional by model selection. Annals of Statistics, pages 1302-1338, 2000.
Qizhang Li, Yiwen Guo, and Hao Chen. Practical no-box adversarial attacks against dnns. Advances in Neural Information Processing Systems, 33:12849-12860, 2020a.
Qizhang Li, Yiwen Guo, and Hao Chen. Yet another intermediate-level attack. In European Conference Computer Vision, pages 241-257. Springer, 2020b.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint 2017.
Markov. On certain applications of algebraic continued fractions. Unpublished Ph. thesis, St Petersburg, 1884.
11

---

Colin McDiarmid et al. On the method of bounded differences. Surveys in combinatorics, 141(1): 148-188, 1989.
Pascal Mettes, Elise van der Pol, and Cees Snoek. Hyperspherical prototype networks. Advances in neural information processing systems, 32, 2019.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663, 2020.
Geon Yeong Park and Sang Wan Lee. Reliably fast adversarial training via latent adversarial perturbation. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 7758-7767, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):211-252 2015.
Hadi Salman, Jerry Li, Ilya Razenshteyn, Pengchuan Zhang, Huan Zhang, Sebastien Bubeck, and Greg Yang. Provably robust deep learning via adversarially trained smoothed classifiers. Advances in Neural Information Processing Systems, 32, 2019.
Shibani Santurkar, Dimitris Tsipras, Andrew Ilyas, and Aleksander Madry. How does batch normal- ization help optimization? Advances in neural information processing systems, 31, 2018.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 815-823, 2015.
Ali Shafahi, Mahyar Najibi, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer, Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Advances in Neural Information Processing Systems, 32, 2019.
Ashish Shrivastava, Tomas Pfister, Oncel Tuzel, Joshua Susskind, Wenda Wang, and Russell Webb. Learning from simulated and unsupervised images through adversarial training. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2107-2116, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.155 2014.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6799 2013.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-6114 14. PMLR, 2019.
Florian Tramer, Nicholas Carlini, Wieland Brendel, and Aleksander Madry. On adaptive attacks to adversarial example defenses. Advances in Neural Information Processing Systems, 33:1633-1645, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
Roman Vershynin. High-dimensional probability, volume 47 of cambridge series in statistical and probabilistic mathematics. Cambridge University Press, Cambridge, 7:14-22, 2018.
12

---

Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, and Zhangyang Wang. Removing batch normalization boosts adversarial training. In International Conference on Machine Learning, pages 23433 23445. PMLR, 2022.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through align- ment and uniformity the hypersphere. arXiv preprint vrXiv:2005.10242 2020.
Zhibo Wang, Hengchang Guo, Zhifei Zhang, Wenxin Liu, Zhan Qin, and Kui Ren. Feature importance- aware transferable adversarial attacks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7639-7648, 2021.
Eric Wong, Leslie Rice, and Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994, 2020.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint 2019.
Greg Yang and Edward J Hu. Feature learning in inftnite-width neural networks. arXiv preprini arXiv:201 1, 14522, 2020.
Greg Yang and Sam S Schoenholz. Deep mean field theory: Layerwise variance and width variation as methods to control gradient explosion. 2018.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel Schoenholz. mean field theory of batch normalization. arXiv preprint arXiv:1902.08/29, 2019.
Yunrui Yu, Xitong Gao, and Cheng Zhong- Xu. Lafeat: piercing through adversarial defenses with latent features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5735-5745, 2021.
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. JEEE transactions on neural networks and learning systems, 3009)2885522244, 2019.
Anru R Zhang and Yuchen Zhou. On the non-asymptotic and sharp lower tail bounds of random variables. Star, (1))e311, 2020.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint 190.iv::90..00. 00321, 2019.
Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael Lyu. Improving adversarial transferability via neuron attribution-based attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14993-15002, 2022a.
Qilong Zhang, Chaoning Zhang, Chaoqun Li, Jingkuan Song, Lianli Gao, and Heng Tao Shen. Practical no-box adversarial attacks with training-free free hybrid image transformation. arXiv preprint arXiv:2203.046 04607, 2022b.
Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable adversarial perturbations. In Proceedings of the European Conference on Computer Vision (ECCV), pages 452-463 2018.
13

---

A Layer normalization
Layer normalization [Ba et al. 2016 is similar batch normalization. However, unlike batch norm, Layer norm computes the mean and standard deviation over all activations in a layer. This further means it can be used in the online setting or in the test setting without caching statistics, as the statistics are calculated in real time.
Now, it clear that if the input of the form (N, H) where N denotes the minibatch size and H the hidden dimension, layer normalization ensures that each instance satisfies, for a layer of dimension d over all activations H.g
Σ 15iSd
H4,
Σ Mh d 13J5d
i.e. a chyerspperical space. This means our assumptions hold exactly. However, this assumes the layer normalization occurs over the entire hidden dimension. This may not necessarily be the case. For example, if the dimensions are (N, C, H. W) it is possible that normalization only occurs over one of the dimensions of C, H, W. However, in this case, we can still recover an appropriate hyperspherical latent by summing over the remaining dimensions.
B
Training Details
Recall that we use the losses of the following form (section 2.1 of the main text):
(20 NN,,
Tradial
|| ||
= =
We take the last two layers upto which have access as i, i' to sum the signs of the angular loss. We cut the a by to adjust for the sum of the signs, resulting in the overall averaging of signs referenced in the text. Also, before switching over the angular loss, we keep using Linit for more than the initial iterations. We utilize 20 such iterations before switching, at a' = a/40. Hence, total, our angular PGD attack consists of 20 iterations of finding an unsupervised radial direction of moving to the antipodal point in latent space, and 20 iterations of maximizing the angular deviation given the initial movement.
Projection back to the valid PyTorch Tensor space for the adversarial instance (adjusting for nor- malization layers) proceeds as per normal PGD methods. The values here calculated in the normalized Tensor space i.c. after normalizing the [0, tensor Imagenet image with parameters of mean [0.485, 0.456, 0.406] and standard deviation [0.229, 0.224, 0.225] channelwise. The projection is such that respects both the original tensor's range and the e in the normalized space.
Normalization settings for Imagenet for Resnet were kept as-is from the pytorch examples and also as per the Fixup repository[ as well as for Efficientne("] For training, we utilize SGD. Please consult the attached codebase for all code-based details. Everything on Torch 1.6, torchvision 0.7.0. All confidence intervals appearing in this paper were generated by bootstrapping and asymmetric intervals were written in + of the max sided deviation. We use the Wilcoxon sign-rank test for determination of statistical significance, with p < 10 as the cutoff, we refer to anything more significant than this value passing the test here on. Cosine learning rate is used to train the Fixup resnet as per (Zhang et al. 2019
run single Tesla V100 GPU,
hhups://githhbbb hhtpps//gihuuh.co///ngggg hww.s/////ooooooooo////ii//innon
14

---

C Original ResNets
We recall that in this class fall models, namely Resnct-18,34,50,100. and 152. In terms of block count, they respectively possess 8, 16, 16, 33, 50 layers. All radial attacks are performed with the last 2 layers upto which we have access. Access is granted upto blocks 6, 12, 12, 25, respectively. So for example, on Resnet- 18 the radial attack uses the loss signal from blocks 5, alone to craft the example. Every value provided every table percentage accuracy metric.
Table 11: Clean accuracies for ResNets
ResNet type
18
34
50
101
152
Acc@1
69.75
73.31
76.13
77.37
78.31
Acc@5
89.07
91.42
92.86
93.54
94.04
Table 12: Comparison of FGSM and Random noise, Top-1 1 accuracy.
Net Type
0.03
0.06
0.1
FGSM
Random
FGSM
Random
FGSM
Random
Resnet-18 18
1.95
69.40
1.16
68.58
1.22
67.11
Resnet-34
4.36
73.06
2.95
72.43
3.13
71.23
Resnet-50
8.18
75.79
6.62
74.89
7.10
73.57
Resnet-100 101
9.93
77.24
8.32
76.79
9.10
75.57
Resnet- 152
10.23
77.21
8.76
76.76
9.82
76.28
Table 13: Comparison of FGSM and Random noise, Top-5 accuracy.
Net Type
0.03
0.06
0.1
FGSM
Random
FGSM
Random
FGSM
Random
Resnet- 18
21.93
88.94
15.39
88.51
13.74
87.46
Resnet-34
30.54
91.31
23.65
90.96
22.08
90.26
Resnet-50
39.04
92.07
33.33
92.32
32.58
91.58
Resnet- 101
43.27
93.52
37.88
93.25
37.61
92.75
Resnet-152
44.65
93.78
39.23
93.48
38.25
92.98
Table 14: Comparison of PGD attacks, top-1 and top-5 accuracy, with confidence intervals on different Resnet types.
Type
Top
Top-5
Top-1
Top
Top-1
Top-5
18
0.006
10.001
1.374 00.04
0.0 +:000
0.484 10.04
0.0 =0.0
0.17 10.02
34
0.006 #0.001
4,386 10.03
0.006 #0.001
0.666
0.004 #0.001
0.288 20.03
50
0.028
1.0.00
8.076 1.23
0.006 10.002
2.914
0.0 :0.0
1.956 10033
00.07
9.648
40.002
3,902
0.006 +:0 002
2.702 10.54
152
0.042 #0.08
9.864 11.83
0.014 #0.003
4.16 =0.73
0.015 #0.001
3,253 #0.61
15

---

Table 15: Comparison of Angular attacks, top-1 and top-5 accuracy, with confidence intervals on different Resnet types.
Type
0.03
0.06
0.1
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
23,416
11.22
5,980
=0.11
1:0.30
34
7.164
1.28
14.320
1:248
0.592
=0.64
1.916 1:0.22
0.216 10.04
0.814 +:0.27
50
13.968
12.82
24,796
15.22
1.418
10.22
3,680 1.0.83
0.432 10.12
1.478
1.026
7.012
00.62
13.886
1.23
0.874
126
2,606
40.78
0.390 +0.122
1.346
152
5,030
#0.1
10.438
1.02
0.534
#0.12
1,572
+0.265
0.248 #0.05
0,808 =0.21
At bigher values ef angular attack outperforms FGSM one. The confidence intervals (constructed by bootstrapping, denoted by =) mark the and 95 percentile confidence intervals and can be used to gauge statistical significance. The wilcoxon signed rank test passed for every case where the point estimate of the angular attack was better than the FGSM one, with the exception of top-1 on 101
16

---

D Fixup ResNets
Just with original Resnets, here we have 5 models, namely Resmct-18.34,5001104 and 152. In terms of block count, they respectively possess 8, 16, 16, 33, 50 layers. All radial attacks are performed with the last layers upto which we have access. Access is granted upto blocks 6, 12, 12, 25, 38 respectively. So for example, on FixUpResnet- 18 the radial attack uses the loss signal from blocks 5, alone to craft the example. Every value provided in every table is a percentage accuracy metric. In short, everything is performed just as with the original Resnets.
Table 16:
Clean accuracies for ResNets
ResNet type
18
34
50
101
152
Acc@)
68.212
70.466
72.938
73.596
73.866
Acc@5
87.910
89,470
90.932
91.273
91.528
Table 17: Comparison of FGSM and Random noise, Top- accuracy.
Net Type
0.03
0.06
0.1
FGSM
Random
FGSM
Random
FGSM
Random
Resnet- 18
2.09
68.18
1.25
67.61
1.19
66.28
Resnet-34
3.73
70.79
2.18
70.39
2.08
69.36
Resnet-50
5.34
73.29
3.63
72.80
3.51
71.61
Resnet- 101
6.37
74.41
4.17
73.93
4.17
72.76
Resnet-152
6.59
74.68
4.71
74.21
4.60
73.06
Table 18: Comparison of FGSM and Random noise, Top-5 accuracy.
Net Type
0.03
0.06
0.1
FGSM
Random
FGSM
Random
FGSM
Random
Resnet- 18
23.17
87.91
14.23
87.56
11.33
86.75
Resnet-34
29.86
89.52
19.71
89.28
16.05
88.58
Resnet-50
33.67
91.12
23.32
90.83
19.95
90.10
Resnet-100 101
37.22
91.83
26.18
91.63
22.50
91.00
Resnet- 152
37.64
92.09
26.29
91.80
22.70
91.24
Table 19: Comparison of PGD attacks, top-1 and top-5 accuracy, with confidence intervals different Resnet types.
Type
0.03
0.06
0.1
Top-1
Top-5
Top-1
Top-1
Top-5
0.034 =0.003
4.490
+0.32
0.014 +0.003
0.322 =0.05
0.008 #0.002
0.058 =00.013
34
0.106 1:0.22
6.524 1:0.8
0.046 1:0.08
0.608 1:0.108
0.030
1:0.007
0.142 1:0.03
0.266
101
040
+0.03
6.988
1.13
0.018 +0.005
0.317 20.04
0.006 +0.001
0.048 40.004
0.124 00.08
7.474 1.18
0.020 0..
0.340 =0.06
0.004 +0.001
0.030 00.005
17

---

Table 20: Comparison of Angular attacks, top-1 and top-5 accuracy, with confidence intervals on different Resnet types.
0.03
0.06
0.1
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
11.78
10.68
83,970
54.666
77.336
69.9668
10.84
88.972
=0.67
67.012
1:127
86,780
10.84
61.868
=0.94
83.034
=0.53
72.914
10.42
90.434
10.62
68.368
11.2
87.818
10.67
61.664
1.1.2
82.570
0..87
73.306
1.07
91.128
#0.48
+0.47
88.686
40.34
62.550
#0.81
83.318
=0.56
152
73.626
+0.72
91.112 #0.37
70.224
40.84
89.078 +0.64
63.700
=0.81
84.116
=0.71
Unlike the original resnets, the angular values not even come close to the FGSM counterparts. This strongly suggests that removal of batch norm completely fixes this mode of vulnerability.
18

---

E EfficientNets
block count, they respectively possess 23, 23, 26, 32, 39 layers. All radial attacks are performed with the last layers upto which we have access. Access is granted upto blocks 17, 17. 19, 24, 31 respectively. So for example, on BI the radial attack uses the loss signal from blocks 16, 17 alone to craft the example. Every value provided in every table percentage accuracy metric. In short, everything performed just as with the original Resnets.
Here have 5 models, namely EfficientNets B1 B5, In
Table 21: Clean accuracies for EfficientNets
Efficientnet type
B1
B2
B3
B4
B5
Clean accuracy (top-1)
78.382
79.808
81,532
83.026
83.778
Clean accuracy (top-5)
94.036
94.732
95.646
96.342
96.710
Table 22: Comparison of FGSM and Random noise, Top-1 accuracy.
Net Type
0.03
0.06
0.1
FGSM
Random
FGSM
Random
FGSM
Random
B1
21.314
77.863
16.860
77.950
14.220
77.296
B2
26.522
79.604
21,926
79.302
20.218
78.564
B3
30.700
81.430
26.526
81.064
25.164
80.678
B4
39.380
82.842
34.820
82.712
33.274
82.344
B5
42.012
83.744
38.002
83.628
36.764
83.424
Table 23: Comparison of FGSM and Random noise, Top-5 accuracy.
Net Type
0.03
0.06
0.1
FGSM
Random
FGSM
Random
FGSM
Random
B1
47.767
93.663
39,758
93.681
38.54
93.582
B2
52.634
94.696
45.114
94.532
42.356
94.222
B3
56.612
95.650
50.478
95,484
48.448
95.282
B4
63.678
96.314
58.024
96.226
56.314
96.116
B5
67.620
96.740
62.750
96.708
61.226
96.626
Table 24: Comparison of PGD attacks, top-1 1 and top-5 accuracy, with confidence intervals.
Type
Top-1
Top-5
Top
Top
Top-1
Top-5
BI
0.430 +:0
1.318
10.43
0.078
20.012
0.386 4:005
0.017
40.0044
0.197
10.022
0.618 =0 102
3.092
20.52
0.064 #0.006
0.340 #0.07
0.012 00.0000000 002
0.218
B3
0.698 1.0. 10..00 108
2.482
10.56
0.136 1:0.02
0.484 10007
0.044 10.006
0.318 10.07
-0.08
10.4
40.08
40.004
+0.05
B5
0.812 10.12
1.728
+0.35
0.135 +0.02
0.260 #0.01
0.073 #0.008
0.125 #0.03
19

---

Table 25: Comparison of Angular attacks, top-1 and top-5 accuracy, with confidence intervals.
Net Type
0.03
0.06
0.1
Top-1
Top-5
Top-1
Top.5
Top
Top
B1
11.49
+: 1.
19.501
11.8
0.592 +0.063
1.666 20
0.156 40.024
0.513 +0.083
B2
13.902
11.74
22.421
+1.54
+0.12
2.208 +0.37
0.153 #0.024
0,589
B3
9.596
#0.61
16:476
#1.12
0.442 00.07
1.267 10.112
0.130 0.130 #0.002
0.474 +0.028
B4
12,045
+1.54
19.192
22.06
0.377
#0.042
1,104 #0.17
0.091 #0.008
0.377 #0.042
#2.41
+1.54
1.793
#0.014
0.468 #0.017
With the re-introduction of Batchnorm, the radial attack is again competitive and beats FGSM. We re-perform the same procedure as for Resnets to determine statistical significance against FGSM. The wilcoxon signed rank test passed for every case where the point estimate of the angular attack was better than the FGSM one.
20

---

Results on VIT models
These results should be taken in context of our discussions Layer normalization. Note that all models are from the Py Torcc implementation of VIT [Dosovitskiy et al. 2020
Table 26:
Attack results for 0.03. Main values represent top-1 accuracy, and values in brackets represent top-5 accuracy.
Model
FGSM
PGD
Angular
ViT-B-16
45.5(73.5)
21.8(67.5)
38.2(69.4)
ViT-B-32
42.9(70.2)
24.7(68.6)
36.5(69.3)
ViT-L-16
55.4(84.1)
15.9(63.5)
38,4(70.8)
ViT-L-32
43.6(73.4)
29.7(73.8)
34.8(73.2)
Table 27: Attack results for 0.06. represent top-5 accuracy.
Main values represent top-1 accuracy, and values in brackets
Model
FGSM
PGD
Angular
ViT-B-16
40.2(67.2)
13.5(44.3)
26.8(52.3)
ViT-B-32
37.5(64.3)
17.4(40.2)
28.5(50.8)
ViT-L-16
45.3(73.8)
12.5(32.8)
31.7(45.2)
ViT-L-32
38.7(68.4)
15.2(44.2)
22.8(52.8)
Table 28: Attack results for c = 0.1. Main values represent top-1 accuracy, and values in brackets represent top-5 accuracy.
Model
FGSM
PGD
Angular
ViT-B-16
(5.16)0.6)
5.7(18.5)
16.7(28.5)
VIT-B-32
34.8(59.4
5.4(21.1)
17.8(3188
ViT-L-16
36.0(65.6)
8.2(17.7)
14.5(33.8)
ViT-L-32
34.1(61.7)
5.8(22.9)
13.9(38.5)
For context, we show the clean accuracies as follows:
Table 29: Top-1 and Top-5 accuracies on ImageNet-IK: for various models.
Model
Top-1 Acc.
Top-5 Acc.
ViT-B-16
81.072
95.318
ViT-B-32
75.912
92.466
ViT-L-16
79.662
94.638
ViT-L-32
76.972
93.07
21

---

G Ablations on Resnets-18,34,50
We first check if the independence structure of latents grows more independent with depth by examining the cross-diagona average absolute correlation across block groups in Resnet-18. 18. This is seen to decay with depth.
Table Comparing independence structures on Resnet-18 by comparing absolute value of cross- diagonal correlations, across block groups.
Block count
2
Average off-diagonal coefficient (absolute)
0.41
0.32
0.19
Now, we present results that exhibit the variation accuracies as the layers being attacked grow further in the network. As reminder, resnet- 18 possesses 8 blocks divided into 4 groups as [2, 2, while resnets 34 and 50 both possess 16 blocks divided as [3, 4, 6, 3]. We will use 1-based indexing to denote the layer upto which we have access, and the last 2 layers upto which we have access will provide the entire signal. Under the Wilcoxon sign rank test, the fall was monotonic, as suggested by the point estimates.
Table 31: Ablation on Resnet-18 18. Last 2 layers of the net granted access are used to craft the attack.
Access till
0.03
0.06
0.1
Top
Top 5
Top 1
Top
Top
Top 5
5
47.55 +2.2
71.09 +33.2
12.6 +1.2
27.8 +211
3.38 =0.4
8.28 +:066
6
22.57 +1.4
39.82 +1.2
2.12 +0.3
6.55 +0.4
0.60 +0.1
1.81 +0.3
7.84 10.83
16.58 0076
0.18 10.03
1.44 10.16
0.13 10.02
0.54 #0.12
Table 32: Ablation on Resnet-34. Last 2 layers of the net granted access to are used to craft the attack.
Access till
0.03
0.06
0.1
Top
Top 5
Top
Top
Top 1
Top 5
8
54,46
11.78
77.35
4-0.68
18.44 +0.45
32.68
+-0.68
4.95
+0.86
11.68
+..26
9
35.84
+00.78
59.32 +0.82
6.32 +0.56
13.84
+0.56
1.39 +0.37
13.84 +1.78
10
26.22
+1.56
41.5 +1.45
1.58 +0.38
3.88 +1.02
0.60 +0.12
1.52 -0.37
11
13.20
+0.26
+00.6 22.68
0.94 +0.31
2.34 +0.22
0.34 +0.06
0.92 +-017
12
6.40
+0.72
12.05
+1.89
0.45 +0.12
2.27 10.56
0.20 10.08
0.65 #0.22
Table 33: Ablation on Resnet-50. Last 2 layers of the net granted access are used to craft the attack.
Access till
0.03
0.06
0.1
Top 1
Top 5
Top
Top 5
Top 1
Top 5
9
53.06 11.89
72.74 1:2.76
14.48 +1.74
25.52 +1.29
2.16 +0.48
5.84 +0.62
10
38.10 +1.44
54.3 +00.7
6.44 +0.42
12.24 +0.82
1.08 +0.24
2.24 1:0.5
11
24.55
+1.22
+1.04 42.78
2.84 +0.32
8.44 +0.18
0.14 +0006
2.04 +-0.8
12
8.56
+1.08
14.08 +1.78
1.76 +0.035
3.22 +0.83
0.55 10.12
1.55 +0.37
22

---

The point we wish to make clear is this the fall is fairly unidirectional across all models with depth, as suggested by our previous analysis. Fixing the 0.03, which shows the biggest gap between layers as larger values of decrease every accuracy to zero, we can examine more layers to see this for the angular attack
Table 34; Resnet-18 18: Top-1 and Top-5 under angular attack
Depth
8
Top-1
47.55
22.57
7.84
2.78
Top-5
71.09
39.82
16.58
7.89
Table 35: Resnet-34: Top-1 and Top-5 under angular attack
Depth
8
10
11
12
13
14
15
16
Top-1
54.46
35.84
26.22
13.20
6.40
6.32
5.92
5.7
5.2
Top-5
77.35
59.32
41.50
22.68
12.05
11.86
10.53
10.46
10.29
Table 36: Resnet-50: Top-1 and Top-5
under angular attack
Depth
8
9
10
11
12
13
14
15
Top-1
65.54
53.06
38.10
24.55
8.56
7.2
5.42
2.95
2.42
Top-5
87.55
72.74
54.3
42.78
14.08
12.56
9.87
6.28
5.76
23

---

H Ablations with respect to number of iterations, number of layers averaged over and ensembling
Table 37 reports the difference in performance with respect to the default case (f = 20) as we vary the number of initial iterations. All results were performed on the vanilla Resnets. Positive values indicate stronger attack. There is no clear trend except that 20 is actually not the optimal value (15 is a bit stronger).
Table 37: Ablation with respect to number initial iterations Limit with Resnets trained on Imagenet
(e=0.03 and average over layers). Top-1 results (top-5 results in brackets).
Resnet
Linit
linit 10
linit 15
limit
20
Linit = 25
linit 30
18
-0.5(-0.6)
0.1(1.3)
-0.2(0.5)
0.0
0.6(-1.4)
1.7(0.1)
34
1.5(0.4)
-1.5(-0.5)
0.8(0.3)
0.0
-1.3(1.1)
-1.9(1.7)
50
1.9(-1.6)
-1.3(-0.4)
1.5(1.7)
0.0
-0.1(-1.4)
0.5(-0.5)
101
1.3(-0.4)
0.1(1.3)
1.8(-100
0.0
-1.1(1.3)
-0.8(1.6)
152
0.6(-0.3)
1.6(1.1)
1.2(0.2)
0.0
-0.1(-1.3)
-0.3(-0.5)
Table 38 compares performance of our attack as we vary the number of iterations. Negative values indicate worse performance relative to baseline (40 iterations). We can see there is some gain in moving to 50 iterations. However the standard evaluation practice with 40, so we not optimize over this step.
Table 38: Ablation with respect to the total number of iterations (finit + fradial) with Resnets trained
on Imagenet (c=0.03 and average over 2 layers). Top-1 results (top-5 results in brackets).
Resnet
30 iterations
40 iterations
50 iterations
18
-0.52(-0.43)
0.0(0.0)
0.38(1.97)
34
-1.36(-0.7)
0.0(0.0)
0.3(0.87)
50
-1.16(-0.34)
0.0(0.0)
0.28(0.02)
101
-0.21(-0.02)
0.0(0.0)
0.63(0.74)
152
-1.56(-0.19)
0.0(0.0)
0.11(1.86)
Table 39
compares the performance of attack vary the number of layers that average over. Negative values are worse relative to baseline (2 layers to be averaged over). We can clearly see the 2-layer averaging actually quite helpful. Bigger gains possible with 3 or 4 layers averaged over, but at the cost of more evaluations.
Table 39: Ablation with respect to number of layers averaged over with Resnets trained on Imagenet for 40 iterations (e=0.03). Top-1 results (top-5 results in brackets).
Resnet
layer
2 layers
3 layers
4 layers
18
-1.3((-1.21)
0.0(0.0)
1.09(1.79)
3.49(2.93)
34
-2.64(-3.08)
0.0(0.0)
2.95(2.67)
3.47(0.93)
50
-2.79(-2.95)
0.0(0)
2.8(2.15)
3.8(3.18)
101
-3.27(-3.55)
0.0(0.0)
0.47(0.31)
0.39(0.5)
152
-2.74(-2.98)
0.0(0.0)
0.66(0.45)
0.83(0.33)
Table 40 reports the performance of ensembling targeted and angular losses, Positive values indicate gains with respect to the angular attack on its own.
24

---

Table 40: Ensemble (targeted loss with angular loss) for Resnets trained on Imagenet for 40 iterations (e=0.03 and average over 2 layers). Top-1 results (top-5 results in brackets).
Resnet
ensemble
3.38(1.62)
34
0.88(2.05)
50
0.64(3.81)
101
0.89(1.79)
152
0.99(2.1)
25

---

I Baselines of Targeted Attacks and Stronger Random Attacks
Here, check if providing a latent, generated from an instance different label, the initial direction of ILAP is useful as opposed to using -||2|||- To be clear, in this case, || - 2yll is minimized, where 2j arises from an instance of different label. Then, the initial deviation increased via ILAP.
We also construct alternative random baseline, termed "naive ILAP". In this, initial random deviation of a random, not necessarily batchnormed layer (we select random ReLU layer per instance), is increased via II.,,, in the L2 norm.
We run these baselines for с 0.03, 0.06 as at 0.1 the values rapidly approach zero for any method. It is apparent that the targeted method is roughly par with ours and the naive method performs worse. Hence, our method basically as good as having these points to perturb towards, "for free" just from the manifold structure.
Table 41: Targeted benchmark on original Resnets
Net Type
0.03
0.06
Top
Top 5
Top
Top 5
Resnet- 18
25.16
42.58
1.46
5.85
Resnet-34
12.2
21.8
1.9
2.86
Resnet-50
13.55
30.97
0.65
5.16
Resnet- 101
5.85
13.66
2.86
6.67
Resnet- 9.68
16.77
1.29
3.87
Table 42: Naive ILAP on all original Resnets
Net Type
0.03
c
0.06
Top
Top 5
Top
Top 5
Resnet- 18
31.11
52.11
9.92
20.66
Resnet-34
19.57
37.59
6.34
15.15
Resnet-50
25.37
36.59
5.81
14.84
Resnet- 101
25.37
37.56
5.71
14.29
Resnet- 152
22.58
36.13
3.23
8.39
26

---

Transfer Results on CIFAR-10 10 and CIFAR-100
We show accuracies under the radial attack when Resnet- 1aiect-88,34,, 18,34,and 50 are subjected to it after transfer learning on CIFAR-10 10 and CIFAR-100. We also show the corresponding baseline drops under FGSM, PGD, random noise and no noise. For Resnets, the final block group (i.e. for example, Resnet-50 has [3, 4, 6, 3] as its block groups, so the last 3) is tuned along with a linear classifier. The wilcoxon signed rank test passed for every case where the point estimate of the angular attack was better than the FGSM one.
Note that attack is markedly more successful CIFAR-100 and in general seems reduce the classifier to randomly guessing and thus has difficulty lowering it below 10 and 50 percent on CIFAR-I0 10 for top-1 and top-5 respectively.
Table 43: Clean accuracies on CIFAR-10
Network type
Resnet-18
Resnet-34
Resnet-50
Clean accuracy (top-1) 89.380
89.500
91.210
Clean accuracy (top-5)
99.640
99.790
99.740
Table 44: Results on CIFAR-10, Resnet-50
Method
0.03
0.06
0.1
Top
Top 5
Top
Top 5
Top 1
Top
PGD
4.230
16.680
4.250
15.360
4.250
15.320
Random
83,930
99,480
71.730
98.520
59.180
83.220
FGSM
52.950
94.770
48.250
92.00
38.300
83.220
Angular
15.020
58.700
14.160
55.310
14.080
54.200
Table 45: Results on CIFAR-10, Resnet-34
Method
0.03
0.06
0.1
Top
Top 5
Top
Top
Top 1
Top 5
PGD
2.88
13.74
2.81
8.67
2.84
7.97
Random
81.59
99.19
66.7
96.95
52.25
93.30
FGSM
47.00
92.08
42.77
89.58
30.79
83.18
Angular
10.29
54.79
9.4
52.17
8.940
52.100
Table 46: Results on CIFAR-10, Resnet-18
Method
0.03
0.06
0.1
Top
Top
Top
Top 5
Top
Top 5
PGD
4.3
6.3
4.3
5.6
4.28
5.58
Random
84.10
99.25
73.69
98.05
68.22
97.08
FGSM
36.82
88.74
36.52
87.33
30.65
82.2
Angular
10.83
52.38
10.82
49.22
10.51
49.14
27

---

Table 47: Clean accuracies CIFAR-100
Network type
Resnet-18 18
Resnet-34
Resnet-50
Clean accuracy (top-1)
67.410
69.500
71.170
Clean accuracy (top-5)
90.270
91.140
92.510
Table 48: Results LIFAR-100, Resnet-50
Method
0.03
0.06
= 0.1
Top
Top 5
Top
Top 5
Top
Top 5
PGD
0.16
0.20
0.14
0.14
0.14
0.14
Random
64.42
88.18
53.1
79.76
38.63
66.23
FGSM
29.28
59.02
25.86
51.95
15.01
35.14
Angular
2.32
10.79
2.16
9.72
1.88
9.49
Table 49: Results on CIFAR-100, Resnet-34
Method
0.03
0.06
0.1
Top
Top
Top
Top
Top
Top 5
PGD
1.16
1.38
1.13
1.2
1.14
1.22
Random
60.37
85.01
46.68
73.09
34.37
60.34
FGSM
25.97
52.32
25,27
48.92
20.08
40.94
Angular
1.63
7.55
1.440
6.22
1.39
6.16
Table 50: Results on CIFAR-I00 100, Resnet-19 18
Method
0.03
0.06
0.1
Top
Top
Top
Top
Top
Top
PGD
0.0
0.0
0.0
0.0
0.0
0.0
Random
60.88
86.19
49.94
78.68
42.41
71.08
FGSM
13.28
36.48
15.88
37.77
13.84
32.8
Angular
2.17
8.57
1.57
7.04
1.34
6.48
28

---

J.1 Results on the Caltech Birds Dataset (CUB)
These experiments were performed on Resnet-50 on the Caltech Birds dataset Wah al. 2011 Everything in the setup is exactly as per the CIFAR-10 10 and CIFAR- 100 case.
Table 51: Results Caltech-Birds, Resnet-50
Method
0.03
c
0.06
0.1
Top
Top 5
Top 1
Top 5
Top
Top 5
PGD
0.0
0.03
0.0
0.0
0.0
0.0
Baseline
61.02
88.9
61.02
88.9
61.02
88.9
FGSM
1.2
11.7
1.0
10.5
0.7
9.5
Angular
10.31
25.15
0.8
6.2
0.5
3.8
29

---

K
Transfer results between architectures
First, we consider results of adversarial attacks as we transfer between different types of Resnets. We consider the difference in transferability between our method and that of PGD for all cases (positive values denote our attack stronger).
Table 52: Transfer results at e = 0.03. Row denotes the source model, column denotes the model attacked. All values denote differences relative the corresponding PGD values (positive values denote our attack stronger).
Resnet types
18
34
50
101
18
1.4
2.2
-1.2
34
1.5
2.1
1.5
50
1.7
2.2
2.4
101
1.3
3.7
2.5
Table 53: Transfer results at e = 0.06. Row denotes the source model, column denotes the model attacked. All values denote differences relative the corresponding PGD values (positive values denote our attack is stronger).
Resnet types
34
101
18
-0.8
0.7
-1.2
34
2.6
3,2
4.5
50
4.8
3.4
1.5
101
2.1
5.2
3.2
Table 54: Transfer results 0.1. Row denotes the source model, column denotes the model attacked. All values denote differences relative the corresponding PGD values (positive values denote our attack is stronger).
Resnet types
18
34
50
101
18
3.5
4.6
5,2
34
6.5
5.2
6,4
50
5.9
9.1
7.2
101
6.4
5.9
7.0
We also will show case where the model being attacked can be non-batchnormed model, in this case, VGG16 [Simonyan and Zisserman, [0014]. The results appear below. In all cases, the source model Resnet, the attacked model is VGGI6, and the result is the difference between PGD and our attack (positive values denote our attack is stronger).
Table 55: Difference between PGD and our attack (positive values indicate that our attack is stronger). The source model is a Resnet, the attacked model is VGGI6.
Source Resnet
0.03
0.06
0.1
18
+3.8
+5.4
+7.5
34
+4.3
+5,8
+8.6
50
+3.2
+4.7
+ +10.5
101
+4.8
+6,2
+9.2
30

---

L Interactions with defenses and robust models
We consider the most key defence adversarial training [Shrivastava al. 2017]. For this process, we carried out adversarial training with an even mixture of FGSM, PGD, and radial attack (1 each for each training instance) for Resnet-18,34,5 for 20 epochs. Even after the training process, the radial attack remained effective. We show the results so obtained.
Table 56: Top-1 I Values, Angular outside bracket, and FGSM in-bracket
Resnet type
0.03
0.06
0.1
18
38.9(22.6)
10.9(14.8)
8.6(12.2)
34
15.9(26.7)
13.5(24.1)
9.8(22.5)
50
22.6(30.2)
16.7(23.5)
11.6(23.2)
Table 57: Top-5 Values, Angular outside bracket, and FGSM in-bracket
Resnet type
0.03
0.06
0.1
18
44.9(43.5)
18.5(32.7)
14.4(26.8)
34
41.0(48.2)
30.8(43.5)
21.5(38.5)
50
37.0(51.2)
26.9(44.0)
18.4(41.8)
We also considered various rrobust models and found that our method had, average (averaging over all с and all robust checkpoints) a 6.8% advantage over FGSM in top-1 accuracy and 18.9% in top-5. We also note that our achievement was consistent over all model) pairs the lowest top-1 advantage was 2.6% (positive) and lowest top-5 advantage was 3.4%. Therefore, it seems the canonical adversarially robust models also do fall prey attack (at least a greater extent than to FGSM).
31

---

M Ablations for affine-shitted batchnorm and for non-batch-normaaized
layers
Suppose that batch normed representation (2) is followed by linear layer (AZ + B). We note that if Z perfectly isotropic Gaussian, can estimate that E 0, and Cov(Z) = Correspondingly. if we are given the linear layer, we can have that E + B) E(B) also have that Cov(AZ B) Cov(AZ) = ACov(Z) (Z AAT. Therefore, if we can estimate the empirical covariance matrix from observations of AZ + B, we can find AAA (approximately). known in this case that A and B are related by
B. We
We can also factorize this to get B AA" an orthogonal transformation i.e. A BQ for some orthogonal matrix Q. So, we can recover Z (up to an orthogonal rotation) by inverting B as we have AZ + B, we can subtract B (by estimating B via E[AZ + B)) and then having AZ, multiply with B get B AZ QZ. Now orthogonal matrix a rotation, and the Gaussian (Z) upon a rotation (Q) is also a Gaussian (QZ) (the spherical geometry have shown is also invariant under rotation). So results carry over if there sufficient samples to estimate the covariance. Note that this assumes not singular.
However, in practice, this covariance estimation can be very expensive and there are not enough samples. The exact inversion must be replaced by pseudo-inversion which works but has less justification. Since require more samples there can also be the question of more powerful methods becoming available which can utilize more samples more efficiently. We can also try our method on AZ instead of trying to invert A (this is simpler as it just requires calculating B, and AZ is also Gaussian). We were able to recreate attack numbers close to our results in the main text for all resnets with 5,000 samples. It can be seen that the results are close the actual angular attack.
Table 58: Comparison of angular attacks, top-1 I and top-5 accuracy, on different Resnet types for different values of e, when a batch normalized layer is attacked but a linear layer is present.
Type
0.03
0.06
0.1
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
18
24.1
40.5
3.5
7.2
0.9
1.9
34
7.9
14.8
1.1
2.8
0.3
0.8
50
14.8
25.9
2.6
4.9
0.5
1.4
101
8.5
14.3
1.8
3.6
0.5
1.8
152
6.3
11.5
1.0
3.3
0.4
1.5
Our hypothesis is that BN makes the feature after BN Gaussian. Let us write the flow of representa- tions as :
X NL - Y BN + Z Possible linear layer + Z'
where X the original feature, N L is the last nonlinear layer before BN, and Y is the pre-BN layer. Our prediction is that Z is Gaussian.
If, instead of the Post-BN feature (Z), the pre-BN feature (Y) is attacked, the result will still succeed, as the BN mapping an invertible linear mapping in inference phase, and thus if Z is Gaussian, so is Y. (Our hypothesis works for any Gaussianized layers). But this is not refutation of our method, because Y is Gaussian due to the existence of the BN layer as well and the cause is the same. Even Z' is valid target.
This does not mean all layers are valid targets and there is nothing special about BN our method does not over-predict and is very precise. Only Y and any linear maps Z enjoy the property that if Z is Gaussian, so are they. X is not under such a guarantee. We now attach results by moving all corresponding attacks from Z to X, and the results are uniformly worse across the board, showing that the attack benefits from attacking Gaussian layers. Thus, the overall hierarchy of success is attacking batch normalized layer >> attacking a linearly- transfomeed batch norm layer attacking layer not related to batch norm by any linear fashion.
32

---

Table 59: Comparison of Angular attacks, top-1 and top-5 accuracy, on different Resnet types for different values of when the layer under attack related to batch normalized layer via linear transformations.
Type
0.03
0.06
0.1
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
18
27.3
46.2
5.6
8.5
0.8
1.7
34
11.5
19.2
1.3
3.8
0.5
1.3
50
17.3
28.5
2,2
5.6
0.8
2.7
101
11.32
19.5
1.1
3.2
1.0
2.4
152
8.2
11.9
0.9
1.6
0.6
1.2
33

---

Visualizations
We show some visualizations of adversarial images created by the radial attack on Imagenet to contrast to FGSM and PGD in table [0] As might be expected, the images sometimes look very similar the baseline images even c 0.1, and sometimes have clear noise over them.
Now, has our method just rediscovered what FGSM and PGD do, or is it independent To answer this, we checked the absolute Lı distance between various images follows:
0.1. The results as
Average L1 distance 15887.2 between FGSM and angular attacks
Average I, distance of 10282.5 between PGD and angular attacks
Average Lı distance of 12866.4 between FGSM and PGD attacks
Average L1 distance of 8968.8 between the clean image and the angular attack.
As can be seen, our method is not too similar to either FGSM or PGD and instead is closer to the base image, indicating that PGD and angular attacks diverge opposite directions from the original image and do not merely copy each other. This is good as we are not just re-creating some other method. Indeed the closeness of our method more to PGD than to FGSM, which is good sign, because PGD is the superior attack.
34

---

Table 60: Table of figures of adversarial images produced under FGSM, PGD, and our Angular method Imagenet with 0.1, with Resnet18 being attacked.
Baseline
FGSM
Angular
PGD
35

---

Extensions to the With-label Case
Here, we "ensemble" our loss with the label-based loss via ILAP. We set initial direction by adding the signs of the radial losses with the classification loss, then maximize the angular deviation. (This done and scaled in the same manner as V in the main text's algorithm). Since even at 0.03 the attack brings the top-1 accuracies near zero (less than 0.1%) under PGD, we compare the top-5 accuracies which are noticeably improved. the other cases, both the methods result in near-zero accuracy.
Table 61: Accuracies with "ensembled PGD",
0.03,
top-5
ResNet type
18
34
50
101
152
PGD alone
3.37
4.39
8.08
9.65
9.86
PGD with angular
2.26
2.71
6.20
7.65
8.32
36

---

P Single-Stage Benchmarks
Previous latent-representation based attacks often plan for the labeled scenario and add latent loss to label-dependent loss to formulate an overall supervised attack. Since our method focuses on the unsupervised case, we remove the basel component from several state of the methods, namely from [Zhou et al. 2018 Inkawhich et al. 2019 Ganeshan et al., 2019 Inkawwhchh et al. 2020b Zhang et al., 2022a Wang al. 2021 Inkawhich al. 2020a] for a comparison against ours. Most of these methods may be characterized as single-stage latent disturbance method which initializes a random perturbation and moves in the L2 norm in latent space. That is, they resemble method but lack its angular viewpoint and justification. We take the best of cach method. The performance, though noticeably above the random benchmarks, is below ours. For convenience, Table in the main paper is repeated as Table 62 below.
Table 62: Comparison of songll-stage top-1 and top-5 attacks on Resnets.
Type
0.03
0.06
0.1
Top-1 1
Top-5
Top-1
Top-5
Top-1 1
Top-5
18
30.45
48.34
6.98
16.87
3,89
10.22
34
16.78
35.89
1.45
8.32
1.68
4.54
50
22.68
39.10
4.62
10.45
4.71
6.58
101
15.65
34.67
5.82
11.28
5,28
8.95
152
8.29
18.53
1.78
5.67
0.95
3.22
Table 63 reports the difference in performance for each single stage attack with respect to the best attack (denoted by "-") whose performance is reported in Table
Table 63: Detailed comparison of single stage attacks for 0.03. indicates the best attack whose performance is reported in Table 62bble622an and the other entries indicate the difference in performance with respect to the best attack. Top-1 results (top-5 results in brackets)
Type
Zhou at
2018
lakawbich
2019
Ganeshas et
2019
Inkowbcch et
20206
Zhang et al
2022
Wong al. 2021
Inkowhich et
20200
18
6.8(2.23)
6.97(6.54)
3.96(2.33)
3.6(2.49)
1.08(1.))
3.78(2.53)
34
3.9(6.31)
3.07(3.36)
3.03(1.73)
2.26(3.2)
1.95(1.96)
2.77(4.63)
50
0.32(0.04)
5.55(7.57)
1.97(0.33)
3.58(4.16)
0.63(7.49)
2.37(4.65)
101
3.18(6.04)
3.27(3.6)
5.55(6.22)
7.55(8.65)
1.05(1.68)
3.14(5.01)
152
3.34(6.15)
2.45(3.64)
3.24(5.28)
1.18(1.95)
6.91(9.91)
1.06(2.62)
37

---

Results on the Hypersphere
On hyperspherical representation spaces, the concentration of norm is perfect. We can ensure this by concatenating the batch norm layers with hyperspherical (I2) projection layer and re-training the model. The corresponding angular accuracies are presented below on Resnets- 18, 34, 50 for Imagenet. It can be seen that these cases perform roughly well as the batch normed scenario.
Table 64: Comparison of Angular attacks, top-1 and top-5 accuracy, with different Resnet types under hyperspherical ablation.
Type
0.03
0.06
Top-1
Top-5
Top-1
Top-5
Top-1
Top-5
18
19.45
35.22
1.98
4.32
0.22
1.12
34
5.22
9.45
0.51
2.12
0.18
0.95
50
11.22
20.85
1.32
3.09
0.45
1.56
38

---

R Discussion of the Concentration of Norm
From here on, we discuss the case of the concentration around expectation of random variable of the following form, with M2 's being other random variables of mean 0, variance 1:
V
M2
We can see that V is the squared norm of the batch norm latent of interest and each M, is an individual hidden neuron, i.c. going dimension wise along the latent. To avoid writing the square, let always write:
N Σ P.
Pj M2. V
We understand concentration occur with associated functions f 6), g(V) where g g(N) is usually a monotonically increasing function N obeying g(N) oo when - oo to mean any result that shows:
P(IV - E(V) > 5g(N)) S f(N,6)
Note that by linearity of expectation, E(V) .N. (This holds even if the M, 's are not independent.) Ideally, this function g(N) will obey:
5 + 0o implies f(N,6) - 0 large deviations are unlikely
a(N) O N + g( (N) grows slower than N.
If we combine the above points, we get that A5 > O, Ve > O, EN such that
P(IV - B(V) bn(N)) S J(N, ((N) g N)
<e
When ((N) is any function which obeys g(N)
as N 0o. That is, we can always increase the sample size so that deviations from the expectation are small relative to a monotonically increasing function of N which increases faster than g(NV that is, g(NV). In particular, consider q(N) N and g((N) VN. This implies that if our conditions are met, S(N, - N - cc, and V B[V] - 1. In particular, generally we aim for Gaussian concentration. Let the variance of V be S(N), a function of N. Then we hope to have S(N, 6) = O(exp(-o²)), with g(N) = (SN). This is alright so long as S(N) is of (2) making gNN as o(N). Thus g(N) gives the size the tails, and f bounds the tail concentration.
g(N)
Clearly, the above f. g depend on two things: how the Mj's k are distributed, and how they depend on each other.
We next note some realistic conditions and situations under which concentration and similar bound- ing will occur in V and its feasibility in our context (a neural network hidden layer, post batch normalization). We proceed from the most general the most assumption-heat heavy cases.
M,'s are all distributed in unknown fashion, possibly not independent of each other, but we can check the covariance between any two Pt's, and the variance of each P. is finite. Then, have that, with S(N) the variance of V. ((P:) variance of P., CS((P, P5) the covariance of any two P, Pz:
N S(N) Σ S(P.) + Σ 2CS(P..P.)
1=1
39

---

We have shown that in practice the correlation terms (off- diagonal, i.e. covariances) do fall with depth on Resnet-18. 18. This makes this approach the most general. Once we have the variance, we can plug in Chebyshev's is inequality [Markov 1884] which yields that deviations of k (NN) occur with a probability I t.. That is.
P (IV E(V) 2 BVS(()) S
1
This bound extremely loose in practice, yet is the most applicable. This is because we usually estimate the covariance terms directly with reasonable precision and need not make further assumptions.
We know the M, 's are dependent, and we know that they are Gaussian, and that they are jointly Gaussian. Let the covariance matrix describing the M,'s 's be Σ. Let LL" E. Then L-'M where M is the vector of all M,'s is an isotropic Gaussian random variable i.e NO, IN) = Zy. And, we can see that V M M then (Zx)")(Zxx) This is quadratic form of Gaussian random variable. The result is known it is a non central chi squared distribution. Specifically, let ft: M2...... AN be the eigenvalues of the matrix E (which is real symmetric and must have all eigenvalues as real), then the distribution is:
(IN)
P4XI
==1
Where, xi denotes the chi-square distribution of degree of freedom 1. In this case, con- centration bounds should be derived upon careful examination of the actual covariance E. However, in most cases, bounds obey [Bodenham and Adams 2016 Laurent and Massart
2000, /Zhang and Zhou 2020 Castano-Martinez and López-Blázquee 2005 at worst exponential concentration, i.c.
P
(IV E(V) 27012617 OVN) & O(exp(-8))
In certain cases, numerical methods are feasible, and one can also use Chernoff's 's inequality on the moment generating function. In this case, the assumptions are purely on how M, are distributed, we do not ask for independence.
Now, move to noting how independence may offer alternate ways of bounding V. For the following cases, we assume the M,'s 's and hence the P.'s ss are independent.
Ml' are independent, and all P.'s occur in a bounded range. Note that this is nearly impossible guarantee and indeed this is not true if M, is Gaussian. However, we can modify this to the condition that, with probability 2 Phound- P. occurs bounded range. Then, we can apply the union bound get that, with probability all P.'s occur in a bounded range. This is reasonable if we expect at most some Phoand fraction of outliers. In this scenario, we may employ either Azuma's inequality [Azuma 1967). Hoeffding's inequality (Hoefiding 1994), or McDiarmid's inequality (McDiarmid et 1989 get the sought result: V concentrates with Gaussian- concentration,iee with probability 2 1 N
Npoound-
P(VV-(()) 2 8g( < O(exp(-6"))
where g(N) (replacing the VS(N)) is upper bound derived from the range of occur- (this is correct upto constant factors for all of the three incqualities i.c. Huum//Hoorrdin//Maiiami Note that any random variable which must occur the range [a, bj has variance upper bounded by (b (b-a)² a)² S(N) thus scales as (b 2 we replace it with g(N) here and this is the link between the two meth- ods. This gives us that, with high probability, the deviation around B(V) is of O(vN). When we have a notion of the variance of P; as well, we can utilize Bernstein's family of inequalities [Bernstein 1924 for more accurate bound.
rence, c.g. (NN (bb a)²
40

---

In the case where the Mj's are sub-Gaussians i.e. random variables which for some constants a, 3 obey, taking M, as the example:
P(I 8) a(cxp(-BB²))
Trivially, a Gaussian random variable is sub gaussian. Then, the P.'s are sub-exponential random variables. They are known to exhibit [Vershynim, 2018 mixtures of Gaussian (exp(-62)) and exponential (exp(-6)) tail concentrations (RHS), with g(N) x O( (V The exponential tail does not occur for small deviations. Note that this relies on the moment generating function of M,. In this case, no bound on the support is required.
In general, all the inequalities above rely some variants of Chernoff's s inequality (Hagerup and Rub 1990]. which is merely Markov's 's inequality on the moment generating function.
R.1 Empirical Findings
To corroborate our findings, we define the dispersion ratio. Given an univariate random variable Z, which always has 0 of mean μ. the dispersion ratio D(a.0) with a, a, S is defined as
F-) (- G (-)
where F, G are respectively inverse CDF (quantile) functions of Z and a chi squared distribution with degree of freedom equal to the empirical mean Z (rounded). We do not use the true mean 14, as unknown. In simple terms, this ratio measures how close the tails of the random variable match Chi-square tails. It can be indicating a delta distribution and perfect concentration as well. When Z has chi square tails, (a R for most values of a, 6.
We performed the following test. For Resnets- 18,3,,50, we picked out layers at depth 1/4 and 3/4 (upto the closest integer, discounting initial non-repeating blocks) into the network and randomly sampled a, 1000 times uniformly from 00, 1] to calculate the empirical dispersion (on average), with the random variable being the square of the latent norm depth i i.e. (=))1, By hypothesis. when the individual neurons are gaussian, this r.v. will be a sum of squares of Gaussians, each of which chi-square, and will be another Chi-square. We also did the same for FixUp Resnets. The results indicate that deeper layers are much closer in dispersion, and that FixUp Resnets are far more dispersed (values are far from 1). This strongly supports our conclusions which line up with conjectures in [Daneshmand et al. 2020 2021 and the general "Gaussianization hypothesis" [Neal 2012 Yang and Schoenholz 2018
Yang al. 2019].
Table 65: Concentration for Normal Resnets
ResNet type
18
50
Dispersion 1/4
2.32
2.22
2.42
Dispersion 3/4
1.68
1.86
1.74
Table 66: Concentration for FixUp Resnets
ResNet type
18
34
50
Dispersion I/4
4.28
4.67
5,22
Dispersion 3/4
2.84
4.05
3.68
41

---

Limitations, Ethical and Societal Impact
S.1 Limitations
Our work is necessarily limited in that it tackles deep networks utilizing batch normalization. As such, cannot be easily extended networks utilizing alternative forms of normalization such as LayerNorm [Ba et al.) [0166 or networks that employ no normalization whatsoever. We fully acknowledge these limitations and do not claim to have crafted an adversarial attack that, in this present form, succeeds against batch norm free networks. Instead, we are incentivizing movement away from batch normalization altogether, and view our work as located closer research which analyzes batch normalization [Santurkar et al. 2018 Galloway et al.) 2019 Benz et al. 2020) than to proposing new methods of adversarial attacks, although exists at the intersection of both these aspects.
S.2 Broader impacts and safeguards
Research which proposes avenues of adversarial attacks may be misused by malicious parties to attack objects in the real world [Hendryeks and Gimpel 2016 Athhaly et al. 2018).. Nevertheless, such research is important analyze the actual robustness of the model itself and ultimately helps create better systems in the long run. Thus, we view our work as net societal positive in the long run, even if its short term benefits may not necessarily be so.
We do not presently believe our attack requires specific safeguarding against real world malicious actors. If this situation changes, would recommend that the real world systems question have their normalization processes changed, e.g. via fixup.
42


Reviews:
[{"review_id": "DzcYo2bh1G", "reviewer": "Reviewer_fVuA", "summary": "This paper proposes an adversarial attack (angular attack) method that does not require any label information and works by only accessing the network's first part (up to a specific layer). The attack is based on the assumption that the BN layer converges and forms a hyperspherical latent space, where an angular loss is applied to guide the direction of the adversarial updates. ", "questions": "1. Table 10 (b) shows that the targeted attack performs worse than the original angular attack. Does that mean the proposed attack method cannot be used for targeted attacks (when some label information is available)?", "limitations": "See weaknesses. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The proposed angular attack requires no labels, and only partial access to the network.\n2. The hyperspherical assumption about the geometry of BN makes sense. Both positive and negative results (e.g., angular attack on Fixup ResNet) support this assumption. \n", "weaknesses": "1. Although Figure 2 is straightforward and informative, the other table results, including those in the appendix, are unclear. For example, it is unclear whether the numbers in Table 9 are absolute mean correlation or the fall in absolute correlation. Table 4 and Table 47 in the appendix are identical tables that only report the “max” over seven baseline methods. I think it makes more sense to include the full results of each baseline.\n2. There are some missing ablation studies that I think should be included in the experiment section. a. How does the number of iterations affect the method (currently, the experiments only use a default iteration of 40). b. The current angular loss is computed by averaging the loss over the last two layers; I'd like to see more results of computing losses over more layers or just the last layer. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Hi, thank you for the review. We address the points raised as follows :\n\nWeakness 1 : Table 9 denotes the absolute correlation. It is meant to demonstrate that this value falls monotonically.\n\nYes, Table 4 and 47 are the same, we apologize for this confusion. We have broken out the results of each method in the common rebuttal. We will include this in the appendix. Due to lack of space, we utilized the max approach for the main paper's table.\n\nWeakness 2 : Yes, we chose 40 as this is a commonly used default iteration # across various papers e.g. the original work on PGD. We attach some ablations with different iteration #.\n\nThe averaging only occurs over the last 2 layers due to our preference of keeping things simple - we could certainly optimize over the parameter. Our point was to show that even without this optimization, the attack performs well. We attach ablations over this.\n\nQuestion : \n\nNo, it does not mean that we cannot use the label information. It only means that when we randomly pick an instance of a different label, the angular attack often does better. But, the *combination* of label and angular attack can do better than either attack, so having an instance of a different label can still be very helpful. We provide some cases where our loss is provided in ensemble with the targeted loss and these perform better than either.\n\nAll ablations below performed on vanilla Resnets $\\epsilon=0.03$ as this is where the accuracy fluctuates the most and at a higher epsilon many accuracies are directly going to zero.\n\n**Table of attacks**\n\nIn order in which they are cited (appendix O, Line 219-220). Each value in the column represents added delta from lowest (i.e. gap by which a method is worse). All tests on $\\epsilon = 0.03$. Top-1 in main, Top-5 in brackets. \"-\" denotes optimal.\n\n| ResNet |          |          |          |          |          |          |          |\n|--------|----------|----------|----------|----------|----------|----------|----------|\n| 18     | - | 6.8(2.23) | 6.97(6.54)| 3.96(2.33)| 3.6(2.49) | 1.08(1.1) | 3.78(2.55)|\n| 34     | 3.9(6.31)| 3.07(3.36)| 3.03(1.73)| 2.26(3.2) | 1.95(1.96)| 2.77(4.63)| -  |\n| 50     | 0.32(0.04)| 5.55(7.57)| -  | 1.97(0.33)| 3.58(4.16)| 0.63(7.49)| 2.37(4.65)|\n| 101    | 3.18(6.04)| 3.27(3.6) | 5.55(6.22)| 7.55(8.65)| 1.05(1.68)| -  | 3.14(5.01)|\n| 152    | 3.34(6.15)| 2.45(3.64)| 3.24(5.28)| 1.18(1.95)| 6.91(9.91)| 1.06(2.62)| -  |\n\n**Table of iterations ablated**\n\nNegative values indicate worse performance relative to baseline (40 iterations). We can see there is some gain in moving to 50 iterations. However the standard evaluation practice is with 40, so we did not optimize over this step. As above, we use vanilla Resnets, Imagenet, $\\epsilon = 0.03$, top-1(top-5 in bracket) and so on.\n\n| ResNet |        30      |    40          |    50          |\n|--------|--------------|--------------|--------------|\n| 18     | -0.52(-0.43) | 0.0(0.0)     | 0.38(1.97)   |\n| 34     | -1.36(-0.7)  | 0.0(0.0)     | 0.3(0.87)    |\n| 50     | -1.16(-0.34) | 0.0(0.0)     | 0.28(0.02)   |\n| 101    | -0.21(-0.02) | 0.0(0.0)     | 0.63(0.74)   |\n| 152    | -1.56(-0.19) | 0.0(0.0)     | 0.11(1.86)   |\n\n**Number of averaging over layers**\n\nAs above, negative values are worse relative to baseline (2 layers to be averaged over). Vanilla Resnets on Imagenet, $\\epsilon = 0.03$ , $40$ iterations, top-1 with top-5 in brackets.\n\n| ResNet |      1         |    2      |     3           |        4      |\n|--------|---------------|----------|--------------|--------------|\n| 18     | -1.37(-1.21)  | 0.0(0.0) | 1.09(1.79)   | 3.49(2.93)   |\n| 34     | -2.64(-3.08)  | 0.0(0.0) | 2.95(2.67)   | 3.47(0.93)   |\n| 50     | -2.79(-2.95)  | 0.0(0)   | 2.8(2.15)    | 3.8(3.18)    |\n| 101    | -3.27(-3.55)  | 0.0(0.0) | 0.47(0.31)   | 0.39(0.5)    |\n| 152    | -2.74(-2.98)  | 0.0(0.0) | 0.66(0.45)   | 0.83(0.33)   |\n\nWe can clearly see the 2-layer averaging is actually quite helpful. Bigger gains are possible with $3,4$ - but at the cost of more evaluations.\n\n**Ensembling**\n\nThese gains are as above - on vanilla Resnets, $\\epsilon=0.03$, Imagenet, 40 iters, top-1 with top-5 in brackets. Positive indicates gains. We simply combine the targeted and the angular loss to produce these results.\n\n| ResNet |               |\n|--------|---------------|\n| 18     | 3.38(1.62)  |\n| 34     | 0.88(2.05)  | \n| 50     | 0.64(3.81)  | \n| 101    | 0.89(1.79)  | \n| 152    | 0.99(2.1)  | \n\nPlease note that a similar ablation with PGD also appears in appendix (Appendix N).\n\nWe hope that our rebuttal answers all your queries to your satisfaction and resolves your issues with the paper."}, {"review_id": "kIlnEjrb0c", "reviewer": "Reviewer_V7by", "summary": "The authors present and evaluate an algorithm to construct adversarial examples without labels by minimizing the cosine similarity between intermediate layers. The authors show that the attack only works with BatchNorm.", "questions": "- I don't fully understand why the attack relies on concentration of norm. It's true that without concentration of norm then for a latent representation Z that there is no unique \"most dissimilar point\", but presumably points opposite Z would still be dissimilar without this assumption.", "limitations": "Authors address limitations adequately.", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper presents strong evidence that the attack is successful on multiple datasets and with multiple architectures\n\n- Despite not achieving state-of-the-art attack success rate when compared to attacks with labels, the paper demonstrates that attacks on BatchNorm architectures are possible without knowledge of the labels, which is required by previous methods\n\n- Paper is generally well written and thorough", "weaknesses": "- There are some clarity issues regarding the explanation for the attack, see Questions\n\n- The authors make the assumption that norm is concentrated, and make an argument that it should be concentrated based on statistical assumptions about the data, but it seems that this should be verified empirically by plotting the distribution of the norms, rather than relying on an imprecise argument.\n\n- The assumptions aren't verified to be necessary. It would be helpful to include an ablation where the norm concentration is broken.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review. We would first like to note that you state \"The authors show that the attack only works with BatchNorm.\" Empirically, the attack is successful on other architectures - we carry out experiments on Layernorm architectures (ViT).\n\nRegarding weaknesses : \n\nWeakness 1 : Moved to question, and addressed under questions.\n\nWeakness 2 : Our assumption on the concentration of norm is actually tested in the appendix (page 27) in Tables 49 and 50 empirically. While plots can provide visual intuition, statistical measures are more definitive and we show that FixUp resnets do not concentrate to the same degree normal ones do.\n\nWeakness 3 : Please note that Appendix L contains ablations of non-concentrated layers unrelated to batch norm linearly inside the batch normalized architectures already in table 44, and might fit your needs. Another relevant ablation appears in page 12 of the appendix (table 28)\n\nIn general, regarding necessity of concentration - it is very difficult to break the norm concentration in a way that does not also change other things about the setup, making it difficult to test this assumption beyond what we have done in appendix L and FixUp. We have provided evidence that FixUp, for instance, breaks the attack and breaks concentration (see also, the above point). While this **could** result from other attributes of FixUp, this criticism can be raised about any way to not make the norm concentrate as well. If we remove batch norm and have no normalization at all, we will break concentration, but also the other things about the model that batch norm provides. As such, we believe appendix L should meet your requirements.\n\n**If you are not satisfied with the ablations provided, you may provide details about what you would like to see in this ablation, we can try to test it and report here in time.**\n\nWe would also note that in general, if an attack proves to be successful outside the conditions we think are necessary, i.e. a concentrated norm, it makes the attack *stronger*, not weaker, from a purely empirical point of view as it allows attacking non-normalized models at least some of the time - even though it does mean the causation might be more complicated.\n\nRe - Question : To be clear, we are **not** saying that the attack necessarily fails in the absence of norm concentration. We are saying that a simple, easily available piece of information like \"This network uses batch norm\" allows us to conclude concentration of norm is likely, and thus guess our attack might work. Whereas, in the absence of such a guarantee, we are simply less certain about its success. We are not saying it will fail in such a case. In other words, if our attack fails, the norm likely is not concentrated, but if the norm is not concentrated, it does not mean the attack is sure to fail. For example, our attack transfers to VGG16 (Appendix, page 16), which is batch-norm-free.\n\nWe hope this addresses your concerns with the paper, while clarifying and resolving any outstanding issues."}, {"review_id": "C7I5HJ1Zge", "reviewer": "Reviewer_7ce2", "summary": "In this paper, the authors proposed a label-free attack utilizing a portion of all layers, which does not require to have gradient access to the full model, and the generated adversarial methods generalize to the case where the model was fine-tuned afterwards. These results have relevance at the intersection of the theory and practice of adversarial robustness and the growing study of batch normalization and its weaknesses and drawbacks.", "questions": "See the weakness. Besides, I was wondering how the initial stage affects the attack results, i.e., say we have total budget on calling gradient of $L$, then to best use this method, how should we trade off the $t_{\\textrm{init}}$ and $t_{\\textrm{radial}}$.\n\nIn figure 2, it shows that the method does not work well on fixup resnet. I was wondering if the authors could conduct more experiments on different architectures to have more evidence.\n\nSome typos: for example\n\nLine: above Line 145, should be $\\frac{<z, z'>}{||z|| ||z'||},$ and Line 145 starts with `where`.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The proposed attach method is a label-free method, and it is a quite strong one. Moreover, the authors use the diagram to demonstrate the idea on their proposed method, which is very clear. The paper is well-written, and easy to follow. Moreover, the authors extended their work to the layer normalization case, showing the proposed method is a quite generic one.\n\n", "weaknesses": "The proposed method only tested in the white-box setting, which means the adversary has the access to all parameters of the model, while in practice, this might be not practical, as usually the service provider only provides the api for the service. I was wondering how the model performs under the black box setting.\n\nThe paper mainly presents the experimental results, but lacks of some theoretical insights why intuitively it would work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your in-depth review. We address your concerns below :\n\nRegarding weakness 1 : This is right, sometimes the service provider does not provide details. However, we often do know enough about the model’s initial layers as they are often based on some available public model. For example, Resnet, used here, has been used to initialize many different CV models. If the CV model is only provided via a service but we know it is Resnet-based, we can attack it with this method. Note that the adversarial examples exhibit good transfer properties (appendix section J) and simply knowing it is Resnet leaks enough information for us to attack it.\n\nRegarding weakness 2 : In the field of theoretical ML, the presence of BN inducing norm concentration is currently a conjecture with some strong partial evidence behind it - as we show in our key citation of Daneshmand et. al. In fact, we believe this work (https://arxiv.org/abs/2106.03970 - Neurips 2021) to be sufficiently motivating from a theoretical POV in our attack as we argue in section 2.2. Due to space constraints in the main paper, we could not explicate ourselves fully - we promise to expand and build on this connection. We would urge the reviewer to look at the orthogonality gap concept in the linked paper which motivates our attack, especially section 4.1 and appendix F which conjectures that the Gaussianization hypothesis extends to networks which are not linear, i.e. the class we consider. **Despite being aware of these theoretical motivations, we did not wish to simply repeat a different paper's contents - and just cited Daneshmand**. For lack of space, these connections were cut from the main text. If you feel it is more appropriate to re-tell this story in the main paper and also cite the paper, we will do this and write out the motivation following the linked paper to explain why the attack actually has a strong partial theoretical motivation already existing in the literature.\n\nTo be clear, fully (not partially) proving the Daneshmand et. al conjecture will be a separate work in its own right, and in theoretical ML. We are focusing on the empirical side, and showing that the partial conjecture is enough to create a good attack. But, we want to make it clear that the theoretical motivation does exist, only in a different paper that we have cited. We will summarize its main content in the paper going forward to make this connection very clear.\n\nRegarding questions :\n\n1) We have performed some $t_{init}$ and $t_{radial}$ ablations, and attach them appropriately.\n\n2) We have already performed experiments on several other architectures beyond Resnet as well, like ViT, Efficientnet, VGG16, and have put the results in the paper and the appendix. If you could name the *specific* architecture you wish to see that is not already present, that would be helpful - we will provide it.\n\n3) Thank you for catching the typos, we will certainly correct them !\n\n**Results upon varying the initial and angular loss tradeoff** :\n\nAll results were performed on the vanilla Resnets. The difference is expressed as a difference from the original case (20 initial iterations). The # iterations varied below is the # of such initial iterations. Positive values indicate stronger attack. There is no clear trend except that 20 is actually not the optimal value (15 is a bit stronger). The changes in top-1 accuracy appear in the main value and top-5 in parentheses for all these Resnets with $\\epsilon = 0.03$ as that $\\epsilon$ avoids accuracy going to $0$, making changes clear and visible.\n\n| Resnet | 5          | 10  | 15  | 20 | 25    | 30 |\n|--------|------------|-----|-----|----|-------|----|\n| 18 | -0.5(-0.6) | 0.1(1.3) | -0.2(0.5) | 0.0 | 0.6(-1.4) | 1.7(0.1) | \n| 34 | 1.5(0.4) | -1.5(-0.5) | 0.8(0.3) | 0.0 | -1.3(1.1) | -1.9(1.7) | \n| 50 | 1.9(-1.6) | -1.3(-0.4) | 1.5(1.7) | 0.0 | -0.1(-1.4) | 0.5(-0.5) | \n| 101 | 1.3(-0.4) | 0.1(1.3) | 1.8(-1.0) | 0.0 | -1.1(1.3) | -0.8(1.6) | \n| 152 | 0.6(-0.3) | 1.6(1.1) | 1.2(0.2) | 0.0 | -0.1(-1.3) | -0.3(-0.5) | \n\nLet us know if you have any other concerns. We hope we have addressed your issues regarding this paper appropriately.\n"}, {"review_id": "YYVPDgiGs5", "reviewer": "Reviewer_ykjk", "summary": "This paper shows that for batch normalized deep image recognition architectures, intermediate latents that are produced after a batch normalization step by themselves suffice to produce adversarial examples using an intermediate loss solely utilizing angular deviations, without relying on any label. The success of the proposed method implies that leakage of intermediate representations may create a security breach for deployed models, which persists even when the model is transferred to downstream usage. The proposed attack also empirically works even for transformer architectures (ViT) that employ Layernorm over batchnorm.", "questions": "\n1.\tWhat can the proposed method be used for?\n\n2.\tWhy is the proposed 2-step process better than 1-step methods?\n\n3.\tThe meaning of lines 171-173 is confusing, can the authors further clarify their claims?\n\n4.\tWhen we have a different latent Zj from a point Xj of a different label from Xi, the authors proposed to use this latent in their loss function to generate adversarial samples. However, this scheme is inferior to the proposed method. Can the authors explain why?\n", "limitations": "\nThe authors have acknowledged the limitations of their work that it cannot be easily extended to networks utilizing alternative forms of normalization such as LayerNorm or networks that employ no normalization whatsoever. It is acceptable that this study focuses on a specific but widely used module in modern neural networks. Please see the weaknesses section for the other limitations of this work.\n\nThe authors have discussed the potential negative societal impact of their work. The authors also proposed a mitigation strategy to alleviate potential harm of this study.\n\n", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1.\tThe proposed method is well motivated and technically sound.\n\nThe proposed loss is motivated by the geometry of batch normed representations and their concentration of norm on a hypersphere and distributional proximity to Gaussians. Through theoretical analysis, the authors show the contribution of batch normalization to an unsupervised radial attack and the optimal layers for the latent to lie near the end of the network. These theoretical findings are also supported by their experimental results. Therefore, the proposed method is well motivated and technically sound.\n\n2.\tThe experimental results are promising.\n\nComparing to single-stage attacks, the proposed method can obtain higher attack success rates, which shows the effectiveness of the proposed attack method and loss function. \n", "weaknesses": "1.\tThe significance of the proposed method is not clear.\n\nWhile the proposed method can get rid of the reliance on label information, as shown in their experiments, the obtained attack success rates are inferior to well-recognized attacks, i.e., PGD. Therefore, it is unclear what is the significance of the proposed method. Since the authors consider white-box settings, attackers can use white-box attacks, like PGD, with predicted labels of the target images, which can also avoid the usage of ground-truth label information. A potential application of the proposed method seems to be the transfer learning setting. However, from the experimental results, i.e., Table 10, the proposed method is also inferior to PGD. Therefore, it is unclear what the proposed method can be used for.\n\n\n\n2.\tSome claims are not validated or clearly explained.\n\n1)\tThe authors failed to clearly explain the meaning of Figure 1, which is closely related to their motivation. The notation of Figure 1 is different from that in the text, which can be confusing.\n2)\tThe authors claimed that “the 2-step process is key.” However, they failed to explain why the proposed 2-step process is better than 1-step methods.\n3)\tThe meaning of lines 171-173 is confusing, can the authors further clarify their claims?\n4)\tWhen we have a different latent Zj from a point Xj of a different label from Xi, the authors proposed to use this latent in their loss function to generate adversarial samples. However, this scheme is inferior to the proposed method. Can the authors explain why?\n\n3.\tPresentations can be improved.\n\n1)\tThe authors split the attack results of their method and baseline methods into different tables, i.e., Tables 1-4 and Table 8, which makes it inconvenient to compare different attacks. The authors can merge these tables for better readability. \n2)\tThe authors claimed one of their contributions as “To improve the supervised attack case when labels are available by using our loss in conjunction with the loss from the true label.” However, the corresponding experimental results are put into the appendix. This reviewer would suggest the authors put these experimental results into the main paper to support their claim.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the review, our comments in response are as follows to the weaknesses : \n\nWeakness 1: The proposed method is most suitable for the cases where a feature extractor (known to us) has been used to train a model (unknown to us). Generally in these cases the first K layers of the new fine-tuned model are left unchanged. Hence, we must attack this model without knowing anything except what the first K layers are. In fact, in these cases, not even a predicted label is usable, as the feature extractor does not emit labels but representation vectors. This case arises naturally in scenarios like :\n\n- Knowing a model is fine-tuned from e.g. Resnet => the first few layers are likely to be unchanged, thus this applies\n- Knowing a model uses a ready-made representation model and then trains on top of it\n\nOur attack is successful in that scenario, whenever these first K layers utilize batchnorm or Layer norm, which is a significant fraction of all models. We believe this scenario to be more and more common as pre-trained and fine-tuned models become the norm instead of training from scratch. Note that in this case, PGD is not applicable as we do not know the full new model and cannot even get soft labels. In fact, we may not even know the new set of labels (when we pre-train on Imagenet, and we are given a new model that is for CIFAR-10, we can use our attack without knowing that it is CIFAR-10, or it has 10 categories, but the soft label approach requires this knowledge of what the label structure of the downstream dataset is)\n\n2.1) We apologize for the oversight. The notation is different as the complete notation had not been introduced at this point in the paper, however, we believe that the figure should be as early as possible for introducing visual intuition to the reader. We will move the notation earlier and align the caption going forward.\n\n2.2) For our attack, we explicitly state why the simple 1-stage solution does not work under angular similarity - If the angular loss is used directly, the exact hyper-spherical condition has a differentiability problem (Section 2.1). Of course, *other* attacks might be 1-stage, but they are empirically worse, as we show in our comparison to single-stage attacks (appendix O)\n\n2.3) Regarding lines 171-173 : Suppose we are in a simple regime where we have an input $X$ with output either 0 or 1. For simplicity, say $X_1$ is mapping to 0. We are told that all valid inputs $X$ lie on a hypersphere, and that opposing points on the hypersphere are the most dissimilar, and that given an $X_1$, we must find $X_2$ with a different label i.e. 1. In the batch normed regime, the center of the hypersphere is the origin. In the non batch normed regime, there is not enough information to solve the problem, but in batch normed regime, we can reflect $X_1$ to get $X_2 = -X_1$ as our candidate for label 1. The point of the lines is to show batch norm gives us one more point of information which can be enough information leakage to make the robustness weaker for toy cases.\n\n2.4) Consider the toy case of 2.3, and suppose the labels are one of : 0, 0.01, and 1 (Assume that the similarity of the labels reflects the numerical gap). In this case, given a point of label 0, the random point of a different label might be one with label 0.01. Due to high similarity of the two labels, there is no effective movement. When points of different label are being drawn at random, there is no way to check against this. Thus, under the case where $-X_1 = X_2$ is the most dissimilar point to $X_1$ and all similarity metrics are angular, a differently labeled $X_3$ to $X_1$ might result in only a small angular movement vs. moving towards $X_2 = - X_1$. Especially when $X_3$ and $X_1$ are of closely related classes (e.g. cat and dog vs airplane), the projection $- X_1 = X_2$ is a more effective target to move to vs randomly selecting an element of another class. \n\n3.1) We will certainly merge these tables. Thank you for the catch. \n\n3.2) These results will be put in the main paper instead.\n\nWe hope this addresses the questions / weaknesses. Please let us know if these address your concerns. Note that the questions have been answered under weaknesses (Question 1 as weakness 1, Q2 as 2.2, Q3 as 2.3, Q4 as 2.4).\n\nWith respect to your \"Limitations\" section, we would also like to note that Layernorm is empirically vulnerable to our attack and analyzed in the appendix for our method.\n\nWe hope we have addressed your concerns in full and resolved your issues with the paper - if not, please let us know.\n"}]

Author rebuttal:
Dear Reviewers,

We thank you for your reviews, and have directly responded to all of you without any common rebuttal. Please let us know if your concerns are appropriately addressed fully.



Meta-review: The paper presents a method for creating adversarial examples using batch normalization without relying on labels. After carefully considering the reviewers' comments and the authors' rebuttals, I recommend accepting this paper for the following reasons:

Theoretical Soundness and Experimental Promises: As highlighted by several reviewers, the proposed method is technically sound and well-motivated (Reviewer ykjk: "The proposed method is well motivated and technically sound"). The experimental results are promising, with successful attack rates on different datasets and architectures (Reviewer V7by: "The paper presents strong evidence that the attack is successful on multiple datasets").

Response to Concerns: The authors effectively addressed most of the reviewers' concerns in the rebuttal. They provided additional results, clarified misunderstandings, and committed to improving the presentation. For example, Reviewer fVuA's concerns about ablation studies and table clarity were addressed, leading to the statement: "The additional results answered my questions, and I am happy to raise my score."

Significance of Contribution: Although there were concerns about the practical applicability and significance of the method, the authors clarified its relevance in scenarios involving pre-trained and fine-tuned models (Authors: "The proposed method is most suitable for the cases where a feature extractor (known to us) has been used to train a model (unknown to us)").

Limitations and Presentation: Concerns about clarity, presentation, and comparison with existing methods like PGD were acknowledged by the authors. They have promised to address these in the final version, and their detailed explanations seem to support this commitment (Authors: "We will put all this in the paper in terms of content").

While one reviewer maintained a Borderline reject rating (Reviewer ykjk), the overall consensus leans toward acceptance, and the authors have provided satisfactory responses to many of the raised issues.

In conclusion, the paper's theoretical grounding, experimental support, and the authors' diligent response to the reviewers' concerns lead to a recommendation for acceptance.
Decision: Accept (poster)