SlowGuess/NeurIPS_2023_OCR huggingface dataset:
This sums  $d$  random variables (assuming  $Z_{i}$  has dimension  $d$ ) and depends on independence structures and the marginal distributions of each variable. Under certain cases, independence of  $Z_{ij}, Z_{ij'}$  can occur. Suppose  $Z_{ij}$  was constrained in  $\{-1, 1\}$ , and had to encode  $2^{D}X_{i}$ 's. Then,  $Z_{ij}, Z_{ij'}$  are independent under the optimal encoding, assigning for each  $X_{i}$  a unique  $D$ -length code with entries  $\in \{-1, 1\}$ . When  $Z_{ij}, Z_{ij'}$  are independent,  $\|Z_{i}\|^{2} = \sum_{j} Z_{ij}^{2}$  will concentrate around its expected value of  $d$  under mild conditions e.g. bounded 4-th moments of  $Z_{ij}$  (Chebyshev's inequality), but concentration tails may not be sub-Gaussian. By tails of concentrations, we mean bounds of form:
$$
P \left(\left| \left\| Z _ {i} ^ {2} \right\| - E \left(\left\| Z _ {i} ^ {2} \right\|\right) \right| \geq \delta\right) \leq f (\delta)
$$
For example, sub-Gaussian concentration implies  $f(\delta) = O(\exp(-\delta^2))$ . Suppose each  $Z_{ij}$  is marginally distributed as a (sub)-Gaussian. With independence, Chernoff's inequality leads to a norm concentration for  $\| Z \|$  with stronger sub-Gaussian tails over Chebyshev's (tails of  $\frac{1}{\delta^2}$ ). Inequalities and implied tails can be consulted from e.g. [Vershynin, 2018]. We consider some previous results. For deep linear models of width (dimensionality)  $d$ , we have [Daneshmand et al., 2020, 2021]:
$$
E [ D _ {K L} (Z | | \mathcal {N} (0, I _ {d})) ] = O \left((1 - \alpha) ^ {i} + \frac {b}{\alpha \sqrt {d}}\right)
$$
$Z$  is the distribution after  $N_{1,i}$  i.e.,  $i$  layers,  $\alpha$  is a constant,  $b$  is the training batch size. The LHS (termed the orthogonality gap) indicates that the distribution of the latents converges to the isotropic Gaussian using KL divergence, which is a stronger condition leading to the latent norms being clustered around  $\sqrt{d}$ . Though derived for linear networks, the original paper offers evidence that the conjecture holds for general multi-layer networks. This indicates deeper layers i.e., increasing  $i$  will have the largest effect on our assumptions holding, as this exponentially drops the KL divergence to a Gaussian (and concentrates the norm), with a secondary benefit from increasing width  $(d)$ . Beyond linear networks, general networks with depth and width reduce to Gaussian Processes [Yang et al., 2019, Yang and Schoenholz, 2018, Yang and Hu, 2020, Yang, 2019, Neal, 2012]. These results usually apply to marginal distributions and not independence structures. Empirically, we find dependence between different variables (measured as correlation) falls as the network latent grows deeper, which aligns with theory. Dimensionality falls as the network deepens - recall the toy example of encoding  $2^{D}$  instances with  $D$  binary variables which forces independence. ILAP's empirical ablation studies found the optimal  $i$  to create the latents for the attack occurred in the range 0.6 to 0.8 (network depth normalized to 1). In sum, we expect our optimal layers for the latent to lie near the end of the network. Our methods work without assumptions in hyperspherical representation spaces [Wang and Isola, 2020, Schroff et al., 2015, Mettes et al., 2019]. Analysis and empirical findings on the concentration and tails is included in Appendix R.
We carry out an extensive set of experiments on Imagenet [Russakovsky et al., 2015], utilizing several ResNet [He et al., 2016] models - ResNet-\{18, 34, 50, 101, 152\} and EfficientNet models [Tan and
Original Resnet
Fixup Resnet
EfficientNet
Figure 2: Accuracies  $(\%)$  of networks under FGSM (blue) and our Angular attack (red). (Color intensity proportional to network size). Accuracies under angular attack are lower without Fixup.
[Le, 2019] B-1 to B-5. We chose these models because they have a common structure at construction. Except a few initial layers, they are constructed by repeatedly stacking the same blocks ("Basic" or "Bottleneck" blocks for ResNets, and Mobile Convolutional blocks [Howard et al., 2017] for EfficientNets). Every block possesses at least one batch norm layer, allowing the extraction of the appropriate latent. All models were ran onImagenet with a standard normalization pre-processing step. Results of all attacks on all models, a sanity check with a fixed norm model (explicitly hyperspherical, i.e. always concentrated in norm), improving supervised (label present) cases on top of FGSM/PGD, comparisons to single-stage attacks previously proposed, interactions with defence, ablations, visualizations, statistical testing, confidence intervals and other transfer learning results and comparisons appear in the Appendices. As the alternative to batchnormed models, we choose Fixup [Zhang et al., 2019] - a different way to initialize the models. Core results are summarized in Figure 2. On the batchnorm-free Fixup Resnet, the FGSM attack greatly outperforms our angular attack. On the other two architectures, this is not the case. FGSM's performance and clean accuracy vary far less than the success of the angular attack. That implies overall model quality and robustness to common attacks remains the same, and vulnerability to the angular attack specifically is what varies. We train Fixup resnets onImagenet as alternatives to the batchnormed models for Resnet only, as methods in Fixup do not generalize to efficientnets. Details of these training steps are in the Appendix B, and we also attach our training and inference codebases. We obtained the models from publicly available repositories for PyTorch [Paszke et al., 2019], namely torchvision and EfficientNets-PyTorch, and did not change the original weights. For the choice of  $\epsilon$  for the adversarial attack, we chose 0.03, 0.06, 0.1 and carried out all attacks using an  $\alpha = 0.01$  over 40 iterations.
Vision Transformers and LayerNorm. Recently, Layer normalization [Ba et al., 2016] has emerged as an alternate mode of normalization to batch norm. It finds prominent usage in the transformer architecture [Vaswani et al., 2017] and for images, in the ViT (Vision Transformer) models. We used our attacks on the case of ViT models [Dosovitskiy et al., 2020] (B/L)-(16/32). In all cases, the attacks were successful as with Resnets/batchnorm. This indicates our attack can succeed even when instead of batch normalization, layer normalization is utilized instead. We discuss this in Appendix A, but make batch norm the focus as it has more pre-existing theoretical analysis [Daneshmand et al., 2020].
We examine the results on all Resnets in Tables 1,2,3 with Table 8 denoting clean accuracies. Angular attacks - especially in terms of their top 5 accuracy, at higher  $\epsilon$  and on deeper networks - dominate FGSM attacks, while losing to PGD attacks. No random perturbation performs well. Confidence intervals rule out a fluke (statistical tests Appendix C). Raw numbers for Fixup Resnets (in Appendix D) exhibit a different pattern - the angular attack performs as well as a random method, while FGSM, and PGD perform as normal. EfficientNets (in Appendix E) follow ResNets in their result. In Table 4, we exhibit the max over single-stage benchmarks among [Zhou et al., 2018, Inkawich et al., 2019,
Table 1: Comparison of FGSM and Random noise, Top-1 accuracy, Top-5 in brackets.
Net Type	ε = 0.03	ε = 0.06	ε = 0.1
FGSM	Random	FGSM	Random	FGSM	Random
Resnet-18	1.95 (21.93)	69.40 (88.94)	1.16 (15.39)	68.58 (88.51)	1.22 (13.74)	67.11 (87.46)
Resnet-34	4.36 (30.54)	73.06 (91.31)	2.95 (23.65)	72.43 (90.96)	3.13 (22.08)	71.23 (90.26)
Resnet-50	8.18 (39.04)	75.79 (92.07)	6.62 (33.33)	74.89 (92.32)	7.10 (32.58)	73.57 (91.58)
Resnet-101	9.93 (43.27)	77.24 (93.52)	8.32 (37.88)	76.79 (93.25)	9.10 (37.61)	75.57 (92.75)
Resnet-152	10.23 (44.65)	77.21 (93.78)	8.76 (39.23)	76.76 (93.48)	9.82 (38.25)	76.28 (92.98)
Table 2: Comparing Angular attacks, top-1/5 accuracy, with confidence intervals on different Resnets.
Type	ε = 0.03	ε = 0.06	ε = 0.1
Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
18	23.416 ±1.22	40.034 ±2.12	2.422 ±0.52	5.980 ±1.12	0.454 ±0.11	1.602 ±0.39
34	7.164 ±1.28	14.320 ±2.48	0.592 ±0.64	1.916 ±0.32	0.216 ±0.04	0.814 ±0.27
50	13.968 ±2.82	24.796 ±5.22	1.418 ±0.22	3.680 ±0.83	0.432 ±0.12	1.478 ±0.26
101	7.012 ±0.62	13.886 ±1.23	0.874 ±0.126	2.606 ±0.78	0.390 ±0.122	1.346 ±0.28
152	5.030 ±0.4	10.438 ±1.02	0.534 ±0.12	1.572 ±0.265	0.248 ±0.05	0.808 ±0.21
Table 3: Comparing PGD attacks, top-1/5 accuracy, with confidence intervals on Resnets.
Type	ε = 0.03	ε = 0.06	ε = 0.1
Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
18	0.006 ±0.001	3.374 ±0.04	0.0 ±0.0	0.484 ±0.04	0.0 ±0.0	0.17 ±0.02
34	0.006 ±0.001	4.386 ±0.03	0.006 ±0.001	0.666 ±0.04	0.004 ±0.001	0.288 ±0.03
50	0.028 ±0.007	8.076 ±1.23	0.006 ±0.002	2.914 ±0.4	0.0 ±0.0	1.956 ±0.3
101	0.032 ±0.07	9.648 ±1.76	0.008 ±0.002	3.902 ±0.62	0.006 ±0.002	2.702 ±0.54
152	0.042 ±0.08	9.864 ±1.83	0.014 ±0.003	4.16 ±0.73	0.015 ±0.004	3.253 ±0.64
Table 4: Comparing single-stage attacks, top-1/5 accuracy, with confidence intervals on Resnets.
Type	ε = 0.03	ε = 0.06	ε = 0.1
Top-1	Top-5	Top-1	Top-5	Top-1	Top-5
18	30.45	48.34	6.98	16.87	3.89	10.22
34	16.78	35.89	1.45	8.32	1.68	4.54
50	22.68	39.10	4.62	10.45	4.71	6.58
101	15.65	34.67	5.82	11.28	5.28	8.95
152	8.29	18.53	1.78	5.67	0.95	3.22
Table 5: Top-1 (& Top-5) accuracy for Vision Transformers:  $\epsilon = 0.1$
Model	FGSM	PGD	Angular
ViT-B-16	35.1 (60.6)	5.7 (18.5)	16.7 (28.5)
ViT-B-32	34.8 (59.4)	5.4 (21.1)	17.8 (31.8)
ViT-L-16	36.0 (65.6)	8.2 (17.7)	14.5 (33.8)
ViT-L-32	34.1 (61.7)	5.8 (22.9)	13.9 (38.5)
Table 6: Top-1/5 accuracy for ResNets, angular attack not on BN layer,  $\epsilon = 0.03$
ResNet	Top-1	Top-5
18	27.3	46.2
34	11.5	19.2
50	17.3	28.5
101	11.32	19.5
152	8.2	11.9
Table 7: Resnet-34 ablations. Last 2 layers of the net granted access to are used for angular attack.
Access till	ε = 0.03	ε = 0.06	ε = 0.1
Top 1	Top 5	Top 1	Top 5	Top 1	Top 5
8	54.46 ±1.78	77.35 ±0.68	18.44 ±0.45	32.68 ±0.68	4.95 ±0.86	11.68 ±1.26
9	35.84 ±0.78	59.32 ±0.82	6.32 ±0.56	13.84 ±0.56	1.39 ±0.37	13.84 ±1.78
10	26.22 ±1.56	41.5 ±1.45	1.58 ±0.38	3.88 ±1.02	0.60 ±0.12	1.52 ±0.37
11	13.20 ±0.26	22.68 ±0.86	0.94 ±0.31	2.34 ±0.22	0.34 ±0.06	0.92 ±0.17
12	6.40 ±0.72	12.05 ±1.89	0.45 ±0.12	2.27 ±0.56	0.20 ±0.08	0.65 ±0.22
Table 8: Clean accuracies for Resnets
ResNet type	18	34	50	101	152
Acc@1	69.75	73.31	76.13	77.37	78.31
Acc@5	89.07	91.42	92.86	93.54	94.04
Table 9: Resnet-18 absolute correlations
Block group	1	2	3
Absolute mean correlation	0.41	0.32	0.19
Table 10: Angular transfer attack results and targeted benchmarks
a) Angular results on CIFAR-100, Resnet-50
Method	ε = 0.03	ε = 0.06	ε = 0.1
Top 1	Top 5	Top 1	Top 5	Top 1	Top 5
PGD	0.16	0.20	0.14	0.14	0.14	0.14
Random	64.42	88.18	53.1	79.76	38.63	66.23
FGSM	29.28	59.02	25.86	51.95	15.01	35.14
Angular	2.32	10.79	2.16	9.72	1.88	9.49
b) Targeted benchmark on Resnets
Net Type	ε = 0.03	ε = 0.06
Top 1	Top 5	Top 1	Top 5
Resnet-18	25.16	42.58	1.46	5.85
Resnet-34	12.2	21.8	1.9	2.86
Resnet-50	13.55	30.97	0.65	5.16
Resnet-101	5.85	13.66	2.86	6.67
Resnet-152	9.68	16.77	1.29	3.87
Ganeshan et al., 2019, Inkawich et al., 2020b, Zhang et al., 2022a, Wang et al., 2021, Inkawich et al., 2020a]. The performance is worse than our method. We also showcase results of ViT models on  $\epsilon = 0.1$  for imagenet in Table 5. (Results for other values of  $\epsilon$  are in Appendix F). Finally in Table 6, we attack Resnets with the angular attack but move the layer being attacked to the layer before the BN layer. This noticeably weakens the attack, suggesting a relationship between the two.
We examine the drop in accuracy as a function of the access to various blocks. We run our attacks consistently accessing the 3/4-th layer of a net, i.e., for a Resnet-50 of blocks [3, 4, 6, 3] - total 16 - we use only the first 12 blocks and sum our angular losses over block 11, 12. On Resnet-34 access to deeper blocks strongly strengthens the attack in Table 7. This agrees with the theory of batch normalization discussed previously [Daneshmand et al., 2021] and with the empirical findings of ILAP [Huang et al., 2019], which found that the most optimal layers to perturb lay between  $50\%$  and  $80\%$  of the network's length. Further depth ablations are in Appendix G. Concentration of norm relies on convergence of batchnorm and independence. Assuming the former, we can check independence by examining the absolute, off-diagonal correlations in the correlation matrices of the latent representations. We note a decorrelation effect with depth, which matches the success of our attacks. We compare the fall in absolute correlation among the latent dimensions in Table 9, across three "block groups" of Resnet-18 which has 8 blocks organized as  $[2, 2, 2, 2]$ . Correlation - a proxy for independence - decreases over the last three block groups.
Transfer learning. For transfer learning, we add a linear classifier that can access the last extracted latent layer of the resnet, and we unfreeze this classifier and the last group of blocks. Every other layer is frozen. The setup is fine-tuned, downstream, on CIFAR-10 and CIFAR-100 [Krizhevsky et al., 2009]. We show results on CIFAR-100 with Resnet-50 in Table 10. Although Table 10 indicates that targeted attacks perform worse than the original angular attacks, it does not mean that we cannot use the label information. It only means that when we randomly pick an instance of a different label, the angular attack often does better. But, the combination of label and angular attack can do better than either attack, so having an instance of a different label can still be very helpful. We provide some cases where our loss is provided in ensemble with the targeted loss and these perform better than either (see Appendix H). Results for CIFAR-10 / other resnets/datasets are in Appendix J. Our method outperforms all but PGD.
Ablation against other cases. The radial loss is an implicit movement towards the opposite point on a hypersphere. This assumes that the hypersphere exists at all. If we have a different latent  $Z_{j}$  from a point  $X_{j}$  of a different label from  $X_{i}$ , we check if our attack might be more effective using an initial loss of the form  $-\| Z_{i} - Z_{j}\|$ . Increasing this loss would be lowering the distance between  $Z_{i}, Z_{j}$  i.e. moving towards the other point in latent space. Yet, this targeted attack - shown in Table 10 - is actually inferior to our attack, which lacks the access.
We have shown a powerful, label-free attack which only utilizes a portion of all the layers available for a network to construct adversarial examples that fool the entire network. It succeeds at this without knowing the label or having gradient access to the full model, and these adversarial methods
generalize to the case where the model was fine-tuned afterwards. These results have relevance at the intersection of the theory and practice of adversarial robustness and the growing study of batch normalization and its weaknesses and drawbacks. We provide support to the notion that batch normalization may open up unseen avenues of perturbations that unexpectedly impact model robustness, a viewpoint supported by previous literature [Galloway et al., 2019, Benz et al., 2020, Wang et al., 2022]. We also extend our results to LayerNorm, which is increasingly relevant with the advent of transformer architectures [Dosovitskiy et al., 2020].
Resources used in this work were provided by the Province of Ontario, the Government of Canada through CIFAR, companies sponsoring the Vector Institute https://vectorinstitute.ai/partners/ and the Natural Sciences and Engineering Council of Canada.
Naveed Akhtar and Ajmal Mian. Threat of adversarial attacks on deep learning in computer vision: A survey. IEEE Access, 6:14410-14430, 2018.
Anish Athalye, Nicholas Carlini, and David Wagner. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420, 2018.
Kazuoki Azuma. Weighted sums of certain dependent random variables. Tohoku Mathematical Journal, Second Series, 19(3):357-367, 1967.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450, 2016.
Philipp Benz, Chaoning Zhang, and In So Kweon. Batch normalization increases adversarial vulnerability: Disentangling usefulness and robustness of model features. 2020.
Sergei Bernstein. On a modification of chebyshev's inequality and of the error formula of laplace. Ann. Sci. Inst. Sav. Ukraine, Sect. Math, 1(4):38-49, 1924.
Dean A Bodenham and Niall M Adams. A comparison of efficient approximations for a weighted sum of chi-squared random variables. Statistics and Computing, 26(4):917-928, 2016.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE symposium on security and privacy (SP), pages 39-57. IEEE, 2017.
Antonia Castano-Martínez and Fernando López-Blázquez. Distribution of a sum of weighted noncentral chi-square variables. Test, 14(2):397-415, 2005.
Antonia Creswell, Anil A Bharath, and Biswa Sengupta. Latentpoison-adversarial attacks on the latent space. arXiv preprint arXiv:1711.02879, 2017.
Hadi Daneshmand, Jonas Kohler, Francis Bach, Thomas Hofmann, and Aurelien Lucchi. Batch normalization provably avoids ranks collapse for randomly initialised deep networks. Advances in Neural Information Processing Systems, 33:18387-18398, 2020.
Hadi Daneshmand, Amir Joudaki, and Francis Bach. Batch normalization orthogonalizes representations in deep random networks. Advances in Neural Information Processing Systems, 34: 4896-4906, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929, 2020.
Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-ucsd birds-200-2011 dataset. 2011.
Haotao Wang, Aston Zhang, Shuai Zheng, Xingjian Shi, Mu Li, and Zhangyang Wang. Removing batch normalization boosts adversarial training. In International Conference on Machine Learning, pages 23433-23445. PMLR, 2022.
Tongzhou Wang and Phillip Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. arXiv preprint arXiv:2005.10242, 2020.
Zhibo Wang, Hengchang Guo, Zhifei Zhang, Wenxin Liu, Zhan Qin, and Kui Ren. Feature importance-aware transferable adversarial attacks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 7639-7648, 2021.
Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. arXiv preprint arXiv:2001.03994, 2020.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv:1902.04760, 2019.
Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522, 2020.
Greg Yang and Sam S Schoenholz. Deep mean field theory: Layerwise variance and width variation as methods to control gradient explosion. 2018.
Greg Yang, Jeffrey Pennington, Vinay Rao, Jascha Sohl-Dickstein, and Samuel S Schoenholz. A mean field theory of batch normalization. arXiv preprint arXiv:1902.08129, 2019.
Yunrui Yu, Xitong Gao, and Cheng-Zhong Xu. Lafeat: piercing through adversarial defenses with latent features. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5735-5745, 2021.
Xiaoyong Yuan, Pan He, Qile Zhu, and Xiaolin Li. Adversarial examples: Attacks and defenses for deep learning. IEEE transactions on neural networks and learning systems, 30(9):2805-2824, 2019.
Anru R Zhang and Yuchen Zhou. On the non-asymptotic and sharp lower tail bounds of random variables. Stat, 9(1):e314, 2020.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint arXiv:1901.09321, 2019.
Jianping Zhang, Weibin Wu, Jen-tse Huang, Yizhan Huang, Wenxuan Wang, Yuxin Su, and Michael R Lyu. Improving adversarial transferability via neuron attribution-based attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14993-15002, 2022a.
Qilong Zhang, Chaoning Zhang, Chaoqun Li, Jingkuan Song, Lianli Gao, and Heng Tao Shen. Practical no-box adversarial attacks with training-free hybrid image transformation. arXiv preprint arXiv:2203.04607, 2022b.
Wen Zhou, Xin Hou, Yongjun Chen, Mengyun Tang, Xiangqi Huang, Xiang Gan, and Yong Yang. Transferable adversarial perturbations. In Proceedings of the European Conference on Computer Vision (ECCV), pages 452-467, 2018.
A Layer normalization
Layer normalization [Ba et al., 2016] is similar to batch normalization. However, unlike batch norm, Layer norm computes the mean and standard deviation over all activations in a layer. This further means it can be used in the online setting or in the test setting without caching statistics, as the statistics are calculated in real time.
Now, it is clear that if the input is of the form  $(N,H)$  where  $N$  denotes the minibatch size and  $H$  the hidden dimension, layer normalization ensures that each instance  $i$  satisfies, for a layer of dimension  $d$  over all activations  $H_{i,j}$ :
$$
\sum_ {1 \leq j \leq d} H _ {i, j} = 0, \sum_ {1 \leq j \leq d} H _ {i, j} ^ {2} = d
$$
i.e. a hyperspherical space. This means our assumptions hold exactly. However, this assumes the layer normalization occurs over the entire hidden dimension. This may not necessarily be the case. For example, if the dimensions are  $(N,C,H,W)$  it is possible that normalization only occurs over one of the dimensions of  $C,H,W$ . However, in this case, we can still recover an appropriate hyperspherical latent by summing over the remaining dimensions.
Recall that we use the losses of the following form (section 2.1 of the main text):
$$
L _ {\text {i n i t}} ^ {t} = - \left\| z ^ {t} \right\| = - \left\| N _ {1, i} \left(x ^ {t}\right) \right\|
$$
We take the last two layers up to which we have access as  $i, i'$  to sum the signs of the angular loss. We cut the  $\alpha$  by 2 to adjust for the sum of the signs, resulting in the overall averaging of signs referenced in the text. Also, before switching over to the angular loss, we can keep using  $L_{init}$  for more than the initial iterations. We utilize 20 such iterations before switching, at  $\alpha' = \alpha / 40$ . Hence, in total, our angular PGD attack consists of 20 iterations of finding an unsupervised radial direction of moving to the antipodal point in latent space, and 20 iterations of maximizing the angular deviation given the initial movement.
Projection back to the valid PyTorch Tensor space for the adversarial instance (adjusting for normalization layers) proceeds as per normal PGD methods. The  $\epsilon$  values here are calculated in the normalized Tensor space i.e. after normalizing the [0,1] tensor Imagenet image with parameters of mean as [0.485, 0.456, 0.406] and standard deviation as [0.229, 0.224, 0.225] channelwise. The projection is such that it respects both the original tensor's range and the  $\epsilon$  in the normalized space.