SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 56: Top-1 Values, Angular outside bracket, and FGSM in-bracket
Resnet type	0.03	0.06	0.1
18	38.9(22.6)	10.9(14.8)	8.6(12.2)
34	15.9(26.7)	13.5(24.1)	9.8(22.5)
50	22.6(30.2)	16.7(23.5)	11.6(23.2)
Table 57: Top-5 Values, Angular outside bracket, and FGSM in-bracket
Resnet type	0.03	0.06	0.1
18	44.9(43.5)	18.5(32.7)	14.4(26.8)
34	41.0(48.2)	30.8(43.5)	21.5(38.5)
50	37.0(51.2)	26.9(44.0)	18.4(41.8)
We also considered various robust models and found that our method had, on average (averaging over all  $\epsilon$  and all robust checkpoints) a  $6.8\%$  advantage over FGSM in top-1 accuracy and  $18.9\%$  in top-5. We also note that our achievement was consistent over all ( $\epsilon$ , model) pairs: the lowest top-1 advantage was  $2.6\%$  (positive) and lowest top-5 advantage was  $3.4\%$ . Therefore, it seems the canonical adversarially robust models also do fall prey to our attack (at least to a greater extent than to FGSM).
M Ablations for affine-shifted batchnorm and for non-batch-normalized layers
Suppose that a batch normed representation  $(Z)$  is followed by a linear layer  $(AZ + B)$ . We note that if  $Z$  is perfectly isotropic Gaussian, we can estimate that  $E(Z) = 0$ , and  $\operatorname{Cov}(Z) = I$ . Correspondingly, if we are given the linear layer, we can have that  $E(AZ + B) = E(B) = B$ . We also have that  $\operatorname{Cov}(AZ + B) = \operatorname{Cov}(AZ) = A\operatorname{Cov}(Z)A^T = AA^T$ . Therefore, if we can estimate the empirical covariance matrix from observations of  $AZ + B$ , we can find  $AA^T$  (approximately). We can also factorize this to get  $BB^T = AA^T$ . It is known in this case that  $A$  and  $B$  are related by an orthogonal transformation i.e.  $A = BQ$  for some orthogonal matrix  $Q$ . So, we can recover  $Z$  (up to an orthogonal rotation) by inverting  $B$  as if we have  $AZ + B$ , we can subtract  $B$  (by estimating  $B$  via  $E[AZ + B]$ ) and then having  $AZ$ , multiply with  $B^{-1}$  to get  $B^{-1}AZ = QZ$ . Now an orthogonal matrix is a rotation, and the Gaussian  $(Z)$  upon a rotation  $(Q)$  is also a Gaussian  $(QZ)$  (the spherical geometry we have shown is also invariant under rotation). So our results carry over if there are sufficient samples to estimate the covariance. Note that this assumes  $A$  is not singular.
However, in practice, this covariance estimation can be very expensive and there are not enough samples. The exact inversion must be replaced by a pseudo-inversion which works but has less justification. Since we require more samples there can also be the question of more powerful methods becoming available which can utilize more samples more efficiently. We can also try our method on  $AZ$  instead of trying to invert  $A$  (this is simpler as it just requires calculating  $B$ , and  $AZ$  is also Gaussian). We were able to recreate attack numbers close to our results in the main text for all resnets with 5,000 samples. It can be seen that the results are close to the actual angular attack.
Table 58: Comparison of angular attacks, top-1 and top-5 accuracy, on different Resnet types for different values of  $\epsilon$ , when a batch normalized layer is attacked but a linear layer is present.
Type	ε = 0.03	ε = 0.06	ε = 0.1
Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
18	24.1	40.5	3.5	7.2	0.9	1.9
34	7.9	14.8	1.1	2.8	0.3	0.8
50	14.8	25.9	2.6	4.9	0.5	1.4
101	8.5	14.3	1.8	3.6	0.5	1.8
152	6.3	11.5	1.0	3.3	0.4	1.5
Our hypothesis is that BN makes the feature after BN Gaussian. Let us write the flow of representations as :
$$
X \rightarrow N L \rightarrow Y \rightarrow B N \rightarrow Z \rightarrow \text {P o s s i b l e}
$$
where  $X$  is the original feature,  $NL$  is the last nonlinear layer before BN, and  $Y$  is the pre-BN layer. Our prediction is that  $Z$  is Gaussian.
If, instead of the post-BN feature  $(Z)$ , the pre-BN feature  $(Y)$  is attacked, the result will still succeed, as the BN mapping is an invertible linear mapping in inference phase, and thus if  $Z$  is Gaussian, so is  $Y$ . (Our hypothesis works for any Gaussianized layers). But this is not a refutation of our method, because  $Y$  is Gaussian due to the existence of the BN layer as well and the cause is the same. Even  $Z'$  is a valid target.
This does not mean all layers are valid targets and there is nothing special about BN - our method does not over-predict and is very precise. Only  $Y$  and any linear maps on  $Z$  enjoy the property that if  $Z$  is Gaussian, so are they.  $X$  is not under such a guarantee. We now attach results by moving all corresponding attacks from  $Z$  to  $X$ , and the results are uniformly worse across the board, showing that the attack benefits from attacking Gaussian layers. Thus, the overall hierarchy of success is attacking a batch normalized layer  $>$  attacking a linearly-transformed batch norm layer  $>$  attacking a layer not related to batch norm by any linear fashion.
Table 59: Comparison of Angular attacks, top-1 and top-5 accuracy, on different Resnet types for different values of  $\epsilon$ , when the layer under attack is not related to a batch normalized layer via linear transformations.
Type	ε = 0.03	ε = 0.06	ε = 0.1
Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
18	27.3	46.2	5.6	8.5	0.8	1.7
34	11.5	19.2	1.3	3.8	0.5	1.3
50	17.3	28.5	2.2	5.6	0.8	2.7
101	11.32	19.5	1.1	3.2	1.0	2.4
152	8.2	11.9	0.9	1.6	0.6	1.2
N Visualizations
We show some visualizations of adversarial images created by the radial attack on Imagenet to contrast to FGSM and PGD in table 60. As might be expected, the images sometimes look very similar to the baseline images even at  $\epsilon = 0.1$ , and sometimes have clear noise over them.
Now, has our method just rediscovered what FGSM and PGD do, or is it independent? To answer this, we checked the absolute  $L_{1}$  distance between various images at  $\epsilon = 0.1$ . The results are as follows:
- Average  $L_{1}$  distance of 15887.2 between FGSM and angular attacks
- Average  $L_{1}$  distance of 10282.5 between PGD and angular attacks
- Average  $L_{1}$  distance of 12866.4 between FGSM and PGD attacks
- Average  $L_{1}$  distance of 8968.8 between the clean image and the angular attack.
As can be seen, our method is not too similar to either FGSM or PGD and instead is closer to the base image, indicating that PGD and angular attacks diverge in opposite directions from the original image and do not merely copy each other. This is good as we are not just re-creating some other method. Indeed the closeness of our method is more to PGD than to FGSM, which is a good sign, because PGD is the superior attack.
Table 60: Table of figures of adversarial images produced under FGSM, PGD, and our Angular method onImagenet with  $\epsilon = 0.1$ , with Resnet18 being attacked.
Baseline	FGSM	Angular	PGD
O Extensions to the With-label Case
Here, we "ensemble" our loss with the label-based loss via ILAP. We set an initial direction by adding the signs of the radial losses with the classification loss, then maximize the angular deviation. (This is done and scaled in the same manner as  $\Psi$  in the main text's algorithm). Since even at  $\epsilon = 0.03$  the attack brings the top-1 accuracies near zero (less than  $0.1\%$ ) under PGD, we compare the top-5 accuracies which are noticeably improved. In the other cases, both the methods result in near-zero accuracy.
Table 61: Accuracies with "ensembled PGD",  $\epsilon = {0.03}$  ,top-5
ResNet type	18	34	50	101	152
PGD alone	3.37	4.39	8.08	9.65	9.86
PGD with angular	2.26	2.71	6.20	7.65	8.32
P Single-Stage Benchmarks
Previous latent-representation based attacks often plan for the labeled scenario and add a latent loss to a label-dependent loss to formulate an overall supervised attack. Since our method focuses on the unsupervised case, we remove the label-based component from several state of the art methods, namely from [Zhou et al., 2018, Inkawich et al., 2019, Ganeshan et al., 2019, Inkawich et al., 2020b, Zhang et al., 2022a, Wang et al., 2021, Inkawich et al., 2020a] for a comparison against ours. Most of these methods may be characterized as a single-stage latent disturbance method which initializes a random perturbation and moves in the  $L_{2}$  norm in latent space. That is, they resemble our method but lack its angular viewpoint and justification. We take the best of each method. The performance, though noticeably above the random benchmarks, is below ours. For convenience, Table 3 in the main paper is repeated as Table 62 below.
Table 62: Comparison of single-stage top-1 and top-5 attacks on Resnets.
Type	ε = 0.03	ε = 0.06	ε = 0.1
Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
18	30.45	48.34	6.98	16.87	3.89	10.22
34	16.78	35.89	1.45	8.32	1.68	4.54
50	22.68	39.10	4.62	10.45	4.71	6.58
101	15.65	34.67	5.82	11.28	5.28	8.95
152	8.29	18.53	1.78	5.67	0.95	3.22
Table 63 reports the difference in performance for each single stage attack with respect to the best attack (denoted by " -") whose performance is reported in Table 62
Table 63: Detailed comparison of single stage attacks for  $\epsilon = 0.03$ . "-" indicates the best attack whose performance is reported in Table 62 and the other entries indicate the difference in performance with respect to the best attack. Top-1 results (top-5 results in brackets)
Type	[Zhou et al., 2018]	[Inkawich et al., 2019]	[Ganeshan et al., 2019]	[Inkawich et al., 2020b]	[Zhang et al., 2022a]	[Wang et al., 2021]	[Inkawich et al., 2020a]
18	-	6.8(2.23)	6.97(6.54)	3.96(2.33)	3.6(2.49)	1.08(1.1)	3.78(2.55)
34	3.9(6.31)	3.07(3.36)	3.03(1.73)	2.26(3.2)	1.95(1.96)	2.77(4.63)	-
50	0.32(0.04)	5.55(7.57)	-	1.97(0.33)	3.58(4.16)	0.63(7.49)	2.37(4.65)
101	3.18(6.04)	3.27(3.6)	5.55(6.22)	7.55(8.65)	1.05(1.68)	-	3.14(5.01)
152	3.34(6.15)	2.45(3.64)	3.24(5.28)	1.18(1.95)	6.91(9.91)	1.06(2.62)	-
Q Results on the Hypersphere
On hyperspherical representation spaces, the concentration of norm is perfect. We can ensure this by concatenating the batch norm layers with a hyperspherical  $(L_2)$  projection layer and re-training the model. The corresponding angular accuracies are presented below on Resnets-18, 34, 50 for Imagenet. It can be seen that these cases perform roughly as well as the batch normed scenario.
Table 64: Comparison of Angular attacks, top-1 and top-5 accuracy, with on different Resnet types under hyperspherical ablation.
Type	ε = 0.03	ε = 0.06	ε = 0.1
Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
18	19.45	35.22	1.98	4.32	0.22	1.12
34	5.22	9.45	0.51	2.12	0.18	0.95
50	11.22	20.85	1.32	3.09	0.45	1.56
R Discussion of the Concentration of Norm
From here on, we discuss the case of the concentration around expectation of a random variable  $V$  of the following form, with  $M_{i}$ 's being other random variables of mean 0, variance 1:
$$
V = \sum_ {i = 1} ^ {N} M _ {i} ^ {2}
$$
We can see that  $V$  is the squared norm of the batch norm latent of interest and each  $M_{i}$  is an individual hidden neuron, i.e. going dimension-wise along the latent. To avoid writing the square, let us always write :
$$
P _ {i} = M _ {i} ^ {2}, V = \sum_ {i = 1} ^ {N} P _ {i}
$$
We understand concentration to occur with associated functions  $f(N, \delta), g(N)$  - where  $g(N)$  is usually a monotonically increasing function in  $N$  obeying  $g(N) \to \infty$  when  $N \to \infty$  - to mean any result that shows :
$$
P \left(\left| V - E (V) \right| \geq \delta g (N)\right) \leq f (N, \delta)
$$
Note that by linearity of expectation,  $E(V) = N$ . (This holds even if the  $M_{i}$ 's are not independent.)
Ideally, this function  $g(N)$  will obey:
-  $\delta \to \infty$  implies  $f(N, \delta) \to 0$ -large deviations are unlikely
-  $\frac{g(N)}{N} \to 0$  as  $N \to \infty - g(N)$  grows slower than  $N$ .
If we combine the above points, we get that  $\forall \delta > 0, \forall \epsilon > 0, \exists N$  such that
$$
P \left(| V - E (V) | \geq \delta q (N)\right) \leq f (N, \frac {q (N)}{g (N)}) <   \epsilon
$$
When  $q(N)$  is any function which obeys  $\frac{q(N)}{g(N)} \to \infty$  as  $N \to \infty$ . That is, we can always increase the sample size so that deviations from the expectation are small relative to a monotonically increasing function of  $N$  which increases faster than  $g(N)$  that is,  $q(N)$ . In particular, consider  $q(N) = N$  and  $g(N) = \sqrt{N}$ . This implies that if our conditions are met,  $f(N, \delta) \to 0$  as  $N \to \infty$ , and  $\frac{V}{E(V)} \to 1$ . In particular, generally we aim for Gaussian concentration. Let the variance of  $V$  be  $S(N)$ , a function of  $N$ . Then we hope to have  $f(N, \delta) = O(\exp(-\delta^2))$ , with  $g(N) = \sqrt{S(N)}$ . This is alright so long as  $S(N)$  is  $o(N^2)$  making  $g(N)$  as  $o(N)$ . Thus  $g(N)$  gives the size of the tails, and  $f$  bounds the tail concentration.
Clearly, the above  $f, g$  depend on two things: how the  $M_{i}$ 's are distributed, and how they depend on each other.
We next note some realistic conditions and situations under which concentration and similar bounding will occur in  $V$  and its feasibility in our context (a neural network hidden layer, post batch normalization). We proceed from the most general to the most assumption-heavy cases.
-  $M_i$ 's are all distributed in an unknown fashion, possibly not independent of each other, but we can check the covariance between any two  $P_i$ 's, and the variance of each  $P_i$  is finite. Then, we have that, with  $S(N)$  the variance of  $V$ ,  $S(P_i)$  variance of a  $P_i$ ,  $CS(P_i, P_j)$  the covariance of any two  $P_i, P_j$ :
$$
S (N) = \sum_ {i = 1} ^ {N} S \left(P _ {i}\right) + \sum_ {i <   j} 2 C S \left(P _ {i}, P _ {j}\right)
$$
We have shown that in practice the correlation terms (off-diagonal, i.e. covariances) do fall with depth on Resnet-18. This makes this approach the most general. Once we have the variance, we can plug in Chebyshev's inequality [Markov, 1884] which yields that deviations of  $k\sqrt{S(N)}$  occur with a probability  $\leq \frac{1}{k^2}$ . That is,
$$
P \left(| V - E (V) | \geq \delta \sqrt {S (N)}\right) \leq \frac {1}{\delta^ {2}}
$$
This bound is extremely loose in practice, yet it is the most applicable. This is because we can usually estimate the covariance terms directly with reasonable precision and need not make further assumptions.
- We know the  $M_i$ 's are dependent, and we know that they are Gaussian, and that they are jointly Gaussian. Let the covariance matrix describing the  $M_i$ 's be  $\Sigma$ . Let  $LL^T = \Sigma$ . Then  $L^{-1}\mathbf{M}$  where  $\mathbf{M}$  is the vector of all  $M_i$ 's is an isotropic Gaussian random variable i.e.  $\sim \mathcal{N}(0, I_N) = Z_N$ . And, we can see that  $V = \mathbf{M}^T M$  is then  $(Z_N)^T\Sigma(Z_N)$ . This is a quadratic form of a Gaussian random variable. The result is known - it is a non central chi squared distribution. Specifically, let  $\mu_1, \mu_2, \ldots, \mu_N$  be the eigenvalues of the matrix  $\Sigma$  (which is real symmetric and must have all eigenvalues as real), then the distribution is:
$$
\left(Z _ {N}\right) ^ {T} \Sigma \left(Z _ {N}\right) \sim \sum_ {i = 1} ^ {N} \mu_ {i} \chi_ {1} ^ {2}
$$
Where,  $\chi_1^2$  denotes the chi-square distribution of degree of freedom 1. In this case, concentration bounds should be derived upon careful examination of the actual covariance  $\Sigma$ . However, in most cases, bounds obey [Bodenham and Adams, 2016, Laurent and Massart, 2000, Zhang and Zhou, 2020, Castanio-Martinez and Lopez-Blazquez, 2005] at worst an exponential concentration, i.e.:
$$
P \left(| V - E (V) | \geq \delta \sqrt {N}\right) \leq O (\exp (- \delta))
$$
In certain cases, numerical methods are feasible, and one can also use Chernoff's inequality on the moment generating function. In this case, the assumptions are purely on how  $M_{i}$ 's are distributed, we do not ask for independence.
Now, we move on to noting how independence may offer alternate ways of bounding  $V$ . For the following cases, we assume the  $M_{i}$ 's and hence the  $P_{i}$ 's are independent.
-  $M_i$ 's are independent, and all  $P_i$ 's occur in a bounded range. Note that this is nearly impossible to guarantee and indeed this is not true if  $M_i$  is a Gaussian. However, we can modify this to the condition that, with probability  $\geq 1 - p_{\text{bound}}$ ,  $P_i$  occurs in a bounded range. Then, we can apply the union bound to get that, with probability  $\geq 1 - Np_{\text{bound}}$ , all  $P_i$ 's occur in a bounded range. This is reasonable if we expect at most some  $p_{\text{bound}}$  fraction of outliers. In this scenario, we may employ either Azuma's inequality [Azuma, 1967], Hoeffding's inequality [Hoeffding, 1994], or McDiarmid's inequality [McDiarmid et al., 1989] to get the sought result:  $V$  concentrates with Gaussian-like concentration, i.e. with probability  $\geq 1 - Np_{\text{bound}}$ :
$$
P \left(\left| V - E (V) \right| \geq \delta g (N)\right) \leq O \left(\exp \left(- \delta^ {2}\right)\right)
$$
where  $g(N)$  (replacing the  $\sqrt{S(N)}$ ) is an upper bound derived from the range of occurrence, e.g.  $g(N) = \sqrt{(b - a)^2 \times N}$  (this is correct up to constant factors for all of the three inequalities i.e. Azuma/Hoeffding/McDiarmid). Note that any random variable which must occur in the range  $[a, b]$  has variance upper bounded by  $(b - a)^2$ .  $S(N)$  thus scales as  $(b - a)^2 N$  - we replace it with  $g(N)$  here and this is the link between the two methods. This gives us that, with high probability, the deviation around  $E(V)$  is of  $O(\sqrt{N})$ . When we have a notion of the variance of  $P_i$  as well, we can utilize Bernstein's family of inequalities [Bernstein, 1924] for a more accurate bound.
- In the case where the  $M_{i}$ 's are sub-Gaussians - i.e. random variables which for some constants  $\alpha, \beta$  obey, taking  $M_{i}$  as the example :
$$
P (| M _ {i} | \geq t) \leq \alpha (\exp (- \beta t ^ {2}))
$$
Trivially, a Gaussian random variable is sub gaussian. Then, the  $P_{i}$ 's are sub-exponential random variables. They are known to exhibit [Vershynin, 2018] mixtures of Gaussian  $(\exp(-\delta^{2}))$  and exponential  $(\exp(-\delta))$  tail concentrations (RHS), with  $g(N) \approx O(\sqrt{N})$ . The exponential tail does not occur for small deviations. Note that this relies on the moment generating function of  $M_{i}$ . In this case, no bound on the support is required.
In general, all the inequalities above rely on some variants of Chernoff's inequality [Hagerup and Rüb, 1990], which is merely Markov's inequality on the moment generating function.
R.1 Empirical Findings
To corroborate our findings, we define the dispersion ratio. Given an univariate random variable  $Z$  which always has  $Z > 0$  of mean  $\mu$ , the dispersion ratio  $D_{[a,b]}$  with  $b \geq a, 0 \leq a, b \leq 1$  is defined as:
$$
\frac {F ^ {- 1} (b) - F ^ {- 1} (a)}{G ^ {- 1} (b) - G ^ {- 1} (a)}
$$
where  $F, G$  are respectively inverse CDF (quantile) functions of  $Z$  and a chi squared distribution with degree of freedom equal to the empirical mean of  $Z$  (rounded). We do not use the true mean  $\mu$ , as it is unknown. In simple terms, this ratio measures how close the tails of the random variable match Chi-square tails. It can be 0 - indicating a delta distribution and perfect concentration - as well. When  $Z$  has chi square tails,  $D_{[a,b]} \approx 1$  for most values of  $a, b$ .
We performed the following test. For Resnets-18,34,50, we picked out layers at depth 1/4 and 3/4 (upto the closest integer, discounting initial non-repeating blocks) into the network and randomly sampled  $a$ ,  $b$  1000 times uniformly from [0, 1] to calculate the empirical dispersion (on average), with the random variable being the square of the latent norm at depth  $i$  i.e.  $\| N_{1,i}(x)\| ^2$ . By hypothesis, when the individual neurons are gaussian, this r.v. will be a sum of squares of Gaussians, each of which is a chi-square, and will be another Chi-square. We also did the same for FixUp Resnets. The results indicate that deeper layers are much closer in dispersion, and that FixUp Resnets are far more dispersed (values are far from 1). This strongly supports our conclusions which line up with conjectures in [Daneshmand et al., 2020, 2021] and the general "Gaussianization hypothesis" [Neal, 2012, Yang and Schoenholz, 2018, Yang et al., 2019].
Table 65: Concentration for Normal Resnets
ResNet type	18	34	50
Dispersion - 1/4	2.32	2.22	2.42
Dispersion - 3/4	1.68	1.86	1.74
Table 66: Concentration for FixUp Resnets
ResNet type	18	34	50
Dispersion - 1/4	4.28	4.67	5.22
Dispersion - 3/4	2.84	4.05	3.68
S Limitations, Ethical and Societal Impact
S.1 Limitations
Our work is necessarily limited in that it tackles deep networks utilizing batch normalization. As such, it cannot be easily extended to networks utilizing alternative forms of normalization such as LayerNorm [Ba et al., 2016] or networks that employ no normalization whatsoever. We fully acknowledge these limitations and do not claim to have crafted an adversarial attack that, in this present form, succeeds against batch norm free networks. Instead, we are incentivizing movement away from batch normalization altogether, and view our work as located closer to research which analyzes batch normalization [Santurkar et al., 2018, Galloway et al., 2019, Benz et al., 2020] than to proposing new methods of adversarial attacks, although it exists at the intersection of both these aspects.
S.2 Broader impacts and safeguards
Research which proposes avenues of adversarial attacks may be misused by malicious parties to attack objects in the real world [Hendrycks and Gimpel, 2016, Athalye et al., 2018]. Nevertheless, such research is important to analyze the actual robustness of the model itself and ultimately helps create better systems in the long run. Thus, we view our work as a net societal positive in the long run, even if its short term benefits may not necessarily be so.
We do not presently believe our attack requires specific safeguarding against real world malicious actors. If this situation changes, we would recommend that the real world systems in question have their normalization processes changed, e.g. via fixup.