SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Additionally, we assessed the influence of network width on the robust test accuracy of fully-trained networks. The robust test accuracy was determined using  $\ell_{\infty}$ -AutoAttack with  $\epsilon = 0.1$ . The adversarial training was conducted under  $p = \infty$ ,  $q = \infty$ ,  $\epsilon = 0.1$ , and 10 iterations. We employed Adam with a learning rate of 0.001 and epochs set to 200.
From Tabs. A5 and A6, it is evident that the robust test accuracy depends on network width rather than depth. The same superscripts in the tables indicate results from networks with an identical number of parameters. For example, the accuracy for  $(N,L) = (250,5)$  significantly surpasses that
Figure A12: Adversarial loss (Eq. (3)) in vanilla networks with  $d = 100$ ,  $N = 40,000$ ,  $L = 3$ ,  $\sigma_w^2 = 2$ ,  $\sigma_b^2 = 0.01$ , and  $\epsilon = 0.1$ . The description is the same as Fig. 2.
Figure A13: Adversarial loss (Eq. (3)) in residual networks with  $d = 100$ ,  $N = 35,000$ ,  $L = 3$ ,  $\sigma_w^2 = 0.01$ ,  $\sigma_b^2 = 0.01$ , and  $\epsilon = 0.1$ . The description is the same as Fig. 2.
for  $(N,L) = (125,20)$ . Although Thms 5.9 and G.14 is primarily applicable to the early stages of training, these theorems, emphasizing the importance of network width, hold true for fully-trained networks.
Figure A14: Adversarial loss (Eq. (3)) in vanilla networks with  $d = 500$ ,  $K = 100$ ,  $L = 3$ ,  $\sigma_w^2 = 2$ ,  $\sigma_b^2 = 0.01$ ,  $p = 2$ ,  $q = 2$ , and  $\epsilon = 0.1$ . The description is similar to Fig. 2.
Figure A15: Adversarial loss (Eq. (3)) in vanilla networks with  $d = 500$ ,  $N = 40$ , 000,  $K = 100$ ,  $L = 3$ ,  $\sigma_w^2 = 2$ ,  $\sigma_b^2 = 0.01$ ,  $p = 2$ , and  $q = 2$ . The description is similar to Fig. 2.
Figure A16: Adversarial loss (Eq. (3)) in vanilla networks with  $d = 500$ ,  $N = 40$ , 000,  $K = 100$ ,  $\sigma_w^2 = 2$ ,  $\sigma_b^2 = 0.01$ ,  $p = 2$ ,  $q = 2$ , and  $\epsilon = 0.1$ . The description is similar to Fig. 2.
Table A5: Test accuracy  $(\%)$  on MNIST [26] under  $\ell_{\infty}$ -AutoAttack with the perturbation constraint 0.1. We used residual networks with  $\sigma_w^2 = 0.01$  and  $\sigma_b^2 = 0.01$ . For adversarial attack in training, we used  $p = \infty$ ,  $q = \infty$ ,  $\epsilon = 0.1$ , and 10 iterations. We set an optimizer to Adam, a learning rate to 0.001, and epochs to 200. Values marked with the same superscript denote results from networks with the same number of parameters  $LN^2$ .
L=5	L=10	L=15	L=20
N=125	13.8	13.1	9.79	8.69♣
N=250	45.8♣	49.8	47.0	48.1†
N=500	79.2†	76.5	75.3	73.4◇
N=1,000	89.8◇	88.4	87.3	87.9
Figure A17: Adversarial loss (Eq. (3)) calculated with  $\epsilon = 0.1$ . We used vanilla networks with  $d = 28 \times 28$ ,  $N = 10,000$ ,  $K = 10$ ,  $L = 3$ ,  $\sigma_w^2(0) = 2$ , and  $\sigma_b^2(0) = 0.01$ . We trained the vanilla networks normally (not adversarially) with the learning rate 0.001. The description is similar to Fig. 2.
Training step
Figure A18: Time evolution of the weight variance in the vanilla and residual network with  $L = 10$  during adversarial training with  $p = \infty$ ,  $q = \infty$ , and  $\epsilon = 0.3$ . In standard and adversarial training, we set the learning rate to 0.0001. In standard training with or without  $\ell_2$  weight regularization, we set  $N = 1,000$ . In adversarial training, we set  $p = \infty$ ,  $q = \infty$ , and  $\epsilon = 0.3$ . The vanilla network was initialized with  $\sigma_w^2(0) = 2$  and  $\sigma_b^2(0) = 0.01$ . The residual network was initialized with  $\sigma_w^2(0) = 0.1$  and  $\sigma_b^2(0) = 0.01$ . The orange dashed lines are predicted by Thms 5.4 and G.9. To derive the theoretical predictions, we calculated  $t := \text{training steps} \times \text{learning rate}$ . For visibility, the experimental results were shifted parallel to satisfy  $\sigma_w^2(0) = 2$  on the vanilla networks and  $\sigma_w^2(0) = 0.1$  on the residual networks.
Figure A19: Heat map of the training accuracy on the vanilla and residual networks trained on MNIST. The description is the same as Fig. 4. The vanilla network was initialized with  $\sigma_w^2(0) = 2$  and  $\sigma_b^2(0) = 0.01$ . The residual network was initialized with  $\sigma_w^2(0) = 0.1$  and  $\sigma_b^2(0) = 0.01$ . The horizontal axis represents the width of the network and the vertical axis represents the depth.
Figure A20: Fisher-Rao norm of vanilla and residual networks adversarially trained on MNIST. We set  $p = \infty$ ,  $q = \infty$ ,  $\epsilon = 0.1$ , and the learning rate to 0.0001. The vanilla network was initialized with  $\sigma_w^2(0) = 2$  and  $\sigma_b^2(0) = 0.01$ . The residual network was initialized with  $\sigma_w^2(0) = 0.1$  and  $\sigma_b^2(0) = 0.01$ .
Figure A21: Fig. A20 with the origin shifted parallel to zero.
Table A6: Test accuracy (%) on Fashion-MNIST [99] under  $\ell_{\infty}$ -AutoAttack. The description is the same as Tab. A5.
L=5	L=10	L=15	L=20
N=125	7.78	9.22	10.4	11.0♣
N=250	33.1♣	33.6	32.6	34.0†
N=500	51.7†	51.7	51.6	50.8◇
N=1,000	67.9◇	67.3	66.7	66.8