SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: (Norm-based measures) Numerical results of  $\pi_{k}$  and its corresponding standard deviation.
Model Architecture	Training Methods	Steps	Optimizer	Batch-size	Aug	Extra-data	Early Stopping
num.params (7)	-	-0.02±0.04	-0.01±0.08	-0.03±0.01	-0.02±0.03	-0.02±0.02	-0.03±0.00	-0.02±0.01
path_norm (8)	0.35±0.05	0.27±0.13	0.24±0.26	0.34±0.03	0.37±0.03	0.35±0.05	0.35±0.03	0.46±0.14
log.prod_of_spec (9)	0.03±0.13	-0.07±0.07	-0.00±0.01	-0.12±0.03	-0.11±0.05	-0.13±0.01	-0.13±0.01	-0.13±0.06
log_PROD_of_fro (10)	0.37±0.03	0.15±0.11	0.09±0.10	0.18±0.01	0.18±0.04	0.19±0.00	0.19±0.02	0.19±0.04
euclid_init_norm (11)	0.32±0.03	0.06±0.03	0.09±0.10	0.17±0.00	0.20±0.04	0.17±0.01	0.17±0.03	0.20±0.11
Norm-based measures requires fixed model architecture. In many prior works, researchers have demonstrated the effectiveness of norm-based measures in estimating the generalization gap [27, 17]. Among them, the weight norm-based measures, e.g., the product of Frobenius norm (logprod_of_fro) [46], the product of spectral norm (logprod_of_spec) [7], and the distance to the initial weight (euclid_init_norm) [27, 40], are built on theoretical frameworks such as PAC-Bayes [43, 47, 38]. path_norm is also often used to estimate the complexity of a neural network, which calculates the sum of outputs for all-ones input after squaring all parameters [27].
In Table 1, the most of norm-based measures exhibit a low correlation with the robust generalization gap for the metric  $\psi_{k}$ . However, by using the proposed metric  $\pi_{k}$  in Table 2, we discover that log_PROD_of_fro and euclid_init_norm exhibit strong correlations with low standard deviation when the model architecture is fixed. Intuitively, when the model architecture varies, the number of parameters and their corresponding values exhibit different ranges. Indeed, the range of log_PROD_of_fro roughly shows [50, 100] for ResNet18, but shows [140, 200] for WRN28-10. Thus, comparing models with different architectures degrades the precision of weight norm-based measures. Under the fixation of the model architecture, log_PROD_of_fro is positively correlated with the robust generalization gap, which consists to the prior theoretical observations in PAC-Bayesian framework [50] or Lipschitz analysis [58]. For log_PROD_of_spec, we do not observe a strong correlation under any conditions.
Furthermore, we observe that path_norm shows some extent of correlation for all  $\psi_{k}$  and  $\pi_{k}$ . Upon conducting a more in-depth analysis, we find that the log of path_norm yields an almost linear relationship with the robust generalization gap when conditioned with average_ce(PGD), resulting the total  $\tau = 0.68$  (detailed in Appendix A.2). This finding suggests that path_norm can be effectively utilized for estimating the robust generalization gap, with consistent to the prior works under the standard training framework [27, 17].
Table 3: (Margin-based measures) Numerical results of  $\pi_{k}$  and its corresponding standard deviation.
Model Architecture	Training Methods	Steps	Optimizer	Batch-size	Aug	Extra-data	Early Stopping
average_ce (12)	-0.24±0.03	-0.28±0.17	-0.30±0.27	-0.23±0.00	-0.23±0.04	-0.24±0.06	-0.23±0.06	-0.34±0.31
inverse_margin (13)	0.07±0.05	-0.05±0.23	-0.18±0.32	0.09±0.06	0.08±0.09	0.08±0.13	0.09±0.12	0.09±0.12
prob_margin (14)	0.24±0.02	0.28±0.16	0.31±0.27	0.23±0.01	0.23±0.04	0.24±0.06	0.23±0.06	0.34±0.31
boundary Thickness (15)	-0.02±0.01	-0.17±0.15	-0.19±0.23	-0.02±0.01	-0.02±0.02	-0.04±0.06	-0.02±0.05	-0.04±0.27
average_ce(PGD) (12)	-0.78±0.01	-0.58±0.39	-0.47±0.40	-0.79±0.04	-0.79±0.01	-0.80±0.03	-0.79±0.00	-0.78±0.01
inverse_margin(PGD) (13)	0.34±0.02	0.26±0.22	0.05±0.19	0.37±0.03	0.35±0.03	0.34±0.05	0.36±0.13	0.38±0.08
prob_margin(PGD) (14)	0.79±0.01	0.59±0.38	0.48±0.39	0.79±0.04	0.79±0.01	0.80±0.03	0.79±0.00	0.78±0.02
Maximizing margin beyond a certain point harms robust generalization. Traditionally, the maximizing margin is considered as an ultimate goal in the adversarial training framework [42, 61]. Indeed, average cross-entropy loss [42] and margin-related losses [61] are frequently used in adversarial training methods. However, in both Table 1 and Table 3, average_ce(PGD) exhibit a high negative correlation with the robust generalization gap across all variations of training parameters. Similarly,  $\pi_{k}$  and the total  $\tau$  of prob_margin(PGD) are extremely high (0.79). This suggests that lower cross-entropy on PGD examples (and higher margin in the probability space) leads to worse robust generalization. Indeed, Fig. 2 shows a clear negative correlation between average_ce(PGD) and the robust generalization gap. Notably, high test robustness is observed within the range of average_ce(PGD)  $\in [0.5, 1.0]$ , indicating that minimizing average_ce(PGD) beyond a certain point may harm the generalization performance as Ishida et al. [25] observed in the standard training frameworks.
Figure 2: Scatter plot for average_ce(PGD) and the gap. Bright color indicates a higher test robustness. For better visualization, we cutoff average_ce(PGD) > 2.
Given this observation, we argue that the margin maximization in adversarial training methods should be carefully revisited. Recent studies have highlighted that maximizing the margin might not necessarily be the optimal objective in adversarial training due to intricate gradient flow dynamics [59] and the non-cognitive concept of using predicted probabilities [1]. Additionally, in recent work [32], despite the similar robustness of TRADES and AT, their margin distributions on benign and adversarial examples are extremely different. This implies that the margin cannot be the sole determinant of adversarial robustness. Considering these findings and other recent studies [41, 60], the margin maximization should be accompanied by a consideration of other factors such as weight regularization or gradient information.
The cross entropy and margin on benign examples, denoted as average_ce and prob_margin, also show some degree of correlation with the generalization gap. This correlation becomes particularly significant when using early stopping, where their correlations reach up to 0.65. Note that inverse_margin behaves differently because it uses the 10th-percentile of margins over the training dataset rather than the expectation.
Boundary thickness works well for models trained by AT. Yang et al. [67] introduced the concept of boundary thickness, which is an extended version of margin based on adversarial examples. They argue that a thin decision boundary leads to both poor adversarial robustness and the gap. Therefore, boundary Thickness should be negatively correlated with the robust generalization gap. However, as shown in Table 3, it does not correlate well with the robust generalization gap. The main difference between our experiments and those in [67] is that we also considered TRADES and MART, whereas Yang et al. [67] sorely compared models trained with AT. Thus, in Fig. 3, we plot the inner  $\tau$  in  $\pi_{k}$  for each training method. It is evident that the boundary thickness demonstrates a strong correlation with the robust generalization gap when the training method is fixed to AT. This suggests that boundary Thickness is more effective for comparing models trained with AT. Furthermore, in Appendix A.2, we also discover that boundary Thickness becomes more highly correlated with the robust generalization gap when the models are conditioned on average_ce(PGD). Thus, when using boundary Thickness as the sole determinant of robust generalization, we should carefully revisit the choice of training methods and the robust accuracy of models on train datasets.
Figure 3: Box plot of the inner  $\tau$  in  $\pi_k$  Eq. (6), where  $\Theta_k$  is training method. The text corresponds to the method for each outlier, e.g., boundary Thickness performs well when AT is used.
Table 4: (Smoothness-based and Flatness-based measures) Numerical results of  $\pi_{k}$  and its corresponding standard deviation.
Model Architecture	Training Methods	Steps	Optimizer	Batch-size	Aug	Extra-data	Early Stopping
kl_divergence (16)	-0.45±0.01	-0.35±0.29	-0.15±0.19	-0.46±0.10	-0.46±0.08	-0.43±0.12	-0.45±0.02	-0.37±0.29
local_lip (17)	-0.24±0.06	-0.15±0.19	-0.01±0.17	-0.24±0.10	-0.25±0.07	-0.20±0.05	-0.23±0.02	-0.21±0.30
pacbayesflat (18)	0.04±0.09	0.04±0.14	-0.06±0.14	0.06±0.07	0.04±0.08	0.10±0.12	0.08±0.10	0.07±0.16
estimated_sharness (19)	-0.07±0.02	-0.11±0.24	-0.22±0.18	-0.06±0.17	-0.08±0.10	-0.02±0.13	-0.04±0.07	-0.04±0.16
estimated_inv_sharness (20)	0.04±0.02	-0.04±0.32	-0.19±0.14	0.05±0.12	0.04±0.06	0.10±0.13	0.07±0.10	0.03±0.12
averageflat (21)	-0.36±0.01	-0.24±0.21	-0.10±0.15	-0.38±0.07	-0.37±0.06	-0.32±0.13	-0.35±0.00	-0.33±0.25
pacbayesflat (PGD) (18)	0.59±0.04	0.45±0.16	0.30±0.21	0.60±0.00	0.59±0.02	0.62±0.02	0.63±0.03	0.50±0.07
estimated_sharness (PGD) (19)	-0.22±0.01	-0.07±0.12	-0.06±0.14	-0.22±0.10	-0.23±0.07	-0.17±0.07	-0.19±0.06	-0.21±0.32
estimated_inv_sharness (PGD) (20)	-0.24±0.02	-0.12±0.14	-0.10±0.17	-0.26±0.10	-0.26±0.06	-0.20±0.08	-0.23±0.06	-0.23±0.35
Smoothness does not guarantee low robust generalization gap. In the pursuit of achieving adversarial robustness, the smoothness between benign and adversarial examples is often considered as an indicative measure. For instance, TRADES [71] minimizes the  $\mathrm{kl\_divergence}$  between benign and adversarial logits. However,  $\mathrm{kl\_divergence}$  shows a negative correlation for both Table 1 and Table 4. This implies that, similar to average_ce(PGD),  $\mathrm{kl\_divergence}$  cannot serve as an indicator for robust generalization.
While Xu et al. [65] demonstrated that imposing local Lipschitzness (local_lip) leads to better generalization in linear classification, recent research [66] argued an opposing perspective, suggesting that within neural networks, local Lipschitzness might hurt robust generalization. However, this conclusion was built on fewer than 20 models and evaluated solely on test examples. In our large experiment, we cannot observe that local Lipschitzness itself negatively affects robust generalization. Rather, it is more efficient in predicting robust accuracy (detailed in Appendix A.1). These findings are consistent with [45, 39], which highlighted the importance of model architecture or weight norms when evaluating models with local Lipschitzness.
Flatness-based measures are not correlated well with the robust generalization gap. Flatness-based measures have recently regarded as powerful indicators of generalization performance in both standard and adversarial training frameworks [18, 64]. This includes the maximum perturbation size in the weight space that do not dramatically changes the accuracy (pacbayesflat) [43, 27], the loss increment by adversarial weight perturbation (estimated_sharpness) [18], and its scale-invariant version (estimated_inv_sharpness) [36]. However, our analysis reveals that flatness-based measures tend to exhibit poor correlations with the robust generalization gap. In both Table 1 and Table 4, the majority of flatness-based measures exhibit near-zero correlations or even negative values. Only pacbayesflat(PGD) demonstrates a strong correlation with robust generalization because it effectively distinguishes between robust and non-robust models (refer to Appendix A.2).
Recently, Stutz et al. [56] demonstrated the importance of early stopping in the analysis of flatness. Similarly, we observe that, when early stopping is employed, the correlation of estimated_sharpness approaches zero. However, without early-stopping, we discover
Table 5: (Gradient-based measures) Numerical results of  $\pi_{k}$  and its corresponding standard deviation.
Model Architecture	Training Methods	Steps	Optimizer	Batch-size	Aug	Extra-data	Early Stopping
x_grad_norm (22)	-0.42±0.03	-0.45±0.28	-0.31±0.23	-0.43±0.09	-0.42±0.02	-0.40±0.02	-0.41±0.05	-0.38±0.07
w_grad_norm (23)	0.07±0.04	-0.02±0.33	-0.17±0.18	0.08±0.13	0.07±0.08	0.11±0.13	0.09±0.09	0.05±0.08
x_grad_norm(PGD) (22)	-0.34±0.02	-0.24±0.21	-0.09±0.15	-0.36±0.05	-0.34±0.09	-0.32±0.12	-0.34±0.02	-0.25±0.25
w_grad_norm(PGD) (23)	-0.32±0.03	-0.16±0.17	-0.15±0.18	-0.32±0.11	-0.33±0.09	-0.28±0.08	-0.31±0.04	-0.28±0.31
Table 6: Numerical results of  $\pi_{k}$  for each measure when  $\Theta_{k}$  is given by average_ce(PGD) values with a bin size of 0.1, along with its corresponding standard deviation.
Measures	πk	Measures	πk
num.params (7)	-0.23±0.17	estimated_sharpness (19)	-0.12±0.19
path_norm (8)	0.25±0.14	estimated_inv_sharpness (20)	-0.13±0.17
log_PROD_of_spec (9)	0.10±0.13	averageflat (21)	-0.12±0.17
log_PROD_of_fro (10)	0.11±0.10	x_grad_norm (22)	-0.35±0.16
euclid_init_norm (11)	-0.11±0.14	w_grad_norm (23)	-0.06±0.17
average_ce (12)	0.03±0.19	inverse_margin(PGD) (13)	0.05±0.19
inverse_margin (13)	-0.09±0.19	prob_margin(PGD) (14)	0.13±0.16
prob_margin (14)	-0.02±0.16	pacbayesflat(PGD) (18)	-0.19±0.20
boundary Thickness (15)	0.06±0.19	estimated_sharpness (PGD) (19)	0.05±0.19
kl_divergence (16)	-0.09±0.14	estimated_inv_sharpness (PGD) (20)	0.01±0.19
local_lip (17)	-0.11±0.16	x_grad_norm (PGD) (22)	-0.12±0.14
pacbayesflat (18)	-0.25±0.21	w_grad_norm (PGD) (23)	0.13±0.16
that estimated_sharpaness exhibits a significant negative correlation for models have low average_ce(PGD)  $\leq 1.5$ . As shown in Fig. 4, models with low estimated_sharpaness show high robust generalization gaps. This finding aligns with the concurrent work of [5], which demonstrates that flatter solutions generalize worse on out-of-distribution data. The additional results can be found in Appendix A.6.
In the case of averageflat [56], which is calculated with random weight perturbations and their worst-case losses, it demonstrates some degree of correlation. However, it is more efficient in predicting robust accuracy rather than the gap (refer to Appendix A.1). This result suggests that, as the concurrent work [5] observed in the standard training framework, flatness measures may not serve as reliable indicators of correlation in the adversarial training framework even they can be effectively used to achieving better performance.
Figure 4: Scatter plot for the robust generalization gap and estimated_sharpness. Conditioned on average_ce(PGD)  $\leq$  1.5 without early-stopping.
The norm of gradient of inputs robustly captures the gap even for models with similar cross-entropy losses. Although some prior works [53, 24] have argued that regularizing the input gradient norm might improve adversarial robustness, we observe that this cannot be argued as lower input gradient norm is better. Table 5 summarizes  $\pi_{k}$  of the gradient norm of input (x_grad_norm) and the gradient norm of weight (w_grad_norm). Among these, x_grad_norm show a strong correlation with the robust generalization gap. The negative correlation of x_grad_norm indicates that models with a larger input gradient norm are more likely to show lower robust generalization gap.
Furthermore, even when comparing models having similar average_ce(PGD), x_grad_norm is the most robust indicator of the robust generalization gap. Previous works in the standard training framework [27, 17] have argued that the cross-entropy loss is strongly correlated with the robust generalization gap, and thus, they used early stopping based on certain cross-entropy thresholds during training to remove the influence of varying cross-entropy loss. However, within the adversarial training framework, employing the same early stopping based on loss becomes challenging as TRADES and MART minimize different loss functions from AT. Therefore, we categorize the trained models into groups based on average_ce(PGD) values using a bin size of 0.1. This grouping reduces  $\pi_{k}$  of average_ce(PGD) to  $-0.12$ . The results are summarized in Table 6. Compared to all other measures, x_grad_norm exhibits the highest rank correlation with the robust generalization gap even when conditioned on average_ce(PGD). We believe this finding highlights the importance of the norm of input gradients as a valuable regularizer for improving model robustness and its generalization in practical settings.
Figure 5: Experiment on RobustBench [12]. For each dataset, we plot the total  $\tau$  of each measure and highlight the robust measures with  $|\tau| \geq 0.25$  for all datasets with yellow background.
Table 7: Numerical results of  $\pi_{k}$  for norm-based measures when  $\Theta_{k}$  is the model structure along with its corresponding standard deviation. The total  $\tau$  is the same as in Fig. 5. For ImageNet,  $\pi_{k}$  is not applicable due to the limited number of pre-trained models in RobustBench [12].
CIFAR-10	πk	Total τ	CIFAR-100	πk	Total τ
path_norm (8)	0.24±0.26	0.02	path_norm (8)	0.55±0.05	0.09
log_PROD_of_fro (10)	0.12±0.40	0.10	log_PROD_of_fro (10)	0.30±0.30	0.17
5 Broader Impact with Benchmarks
RobustBench [12] provides a set of pre-trained models that achieve high robust accuracy across various datasets, including CIFAR-10, CIFAR-100, and ImageNet. Leveraging this benchmark, we extend our observations to diverse models including transformer-based architectures [3, 13] or trained on diffusion-generated datasets [51, 62]. As shown in Figure 5, we identify that some of our findings in Section 4 also can be applied to these models. Margin-based measures such as average_ce, prob_margin, average_ce(PGD), and prob_margin(PGD) consistently exhibit strong correlations with the robust generalization gap. Additionally, we observe that x_grad_norm consistently shows reliable performance in predicting the robust generalization gap, even when applied to models in the RobustBench across various datasets.
Though a deeper analysis is limited by the absence of training setting details, such as the use of early stopping, we additionally conduct an analysis with the model architecture by analyzing the pre-trained models. Table 7 summarizes  $\pi_{k}$  for models with the same architecture. As we demonstrated in Section 4, norm-based measures exhibit a higher correlation when comparing models with identical architectures. Notably, for CIFAR-100, we find that path_norm shows a strong correlation with the robust generalization gap. Regarding the low correlation and high standard deviation of norm-based measures, we hypothesize that other training settings, such as the choice of activation functions (e.g., Swish and SiLU instead of ReLU) and training datasets, may affect the values of norm-based measures. Further exploration of these aspects is left to future work, as additional research and experiments can provide a more comprehensive understanding of these relationships.
While our study unveils the correlation between various measures and the robust generalization gap over 1,300 models, due to our computational constraints, we focused on ResNet models, CIFAR-10, and PGD with the  $L_{\infty}$  norm. Thus, further investigations on a broader range of hyper-parameters and the use of stronger attacks may uncover new relationships beyond our analysis. We hope that future work would address these limitations.
Through large-scale experiments, we verified the underlying relationships between various measures and the robust generalization gap on CIFAR-10 under the  $L_{\infty}$  norm. Our findings offer valuable insights into robust generalization and emphasize the need for caution when making statements such as, 'model A is superior to model B because model A exhibits a better measure value than model B,' a frequently employed phrase in recent literature. We hope that our discoveries regarding diverse measures can contribute to further advancement in the field of adversarial robustness.
This work was supported by the National Research Founda- tion of Korea (NRF) grant funded by the Korean government (MSIT) (No. 2019R1A2C2002358) and the Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea government (MSIT) (No. 2022-0-00984).
[1] Hiroki Adachi, Tsubasa Hirakawa, Takayoshi Yamashita, and Hironobu Fujiyoshi. Revisiting instance-reweighted adversarial training. 2022.
[2] Jean-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? Advances in Neural Information Processing Systems, 32:12214-12223, 2019.
[3] Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. Advances in neural information processing systems, 34:20014-20027, 2021.
[4] Maksym Andriushchenko and Nicolas Flammarion. Understanding and improving fast adversarial training. Advances in Neural Information Processing Systems, 33, 2020.
[5] Maksym Andriushchenko, Francesco Croce, Maximilian Müller, Matthias Hein, and Nicolas Flammarion. A modern look at the relationship between sharpness and generalization. arXiv preprint arXiv:2302.07011, 2023.
[6] Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-156, 2006.
[7] Peter L Bartlett, Dylan J Foster, and Matus J Telgarsky. Spectrally-normalized margin bounds for neural networks. Advances in neural information processing systems, 30, 2017.
[8] Samuel Rota Bulo, Lorenzo Porzi, and Peter Kontschieder. In-place activated batchnorm for memory-optimized training of dnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5639–5647, 2018.
[9] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C Duchi, and Percy S Liang. Unlabeled data improves adversarial robustness. In Advances in Neural Information Processing Systems, pages 11192-11203, 2019.
[10] Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. In International Conference on Machine Learning, pages 854-863. PMLR, 2017.
[11] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pages 2206-2216. PMLR, 2020.
[12] Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flammarion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robustbench: a standardized adversarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.
[13] Edoardo Debenedetti, Vikash Sehwag, and Prateek Mittal. A light recipe to train robust vision transformers. In First IEEE Conference on Secure and Trustworthy Machine Learning, 2023. URL https://openreview.net/forum?id=IztT98kyOcKs.
[14] Gavin Weiguang Ding, Yash Sharma, Kry Yik Chau Lui, and Ruitong Huang. Mma training: Direct input space margin maximization through adversarial training. In International Conference on Learning Representations, 2019.
[69] Stone Yun and Alexander Wong. Do all mobilenets quantize poorly? gaining insights into the effect of quantization on depthwise separable convolutional networks through the eyes of multi-scale distributional dynamics. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2447-2456, 2021.
[70] Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard C. Wilson and William A. P. Smith, editors, Proceedings of the British Machine Vision Conference (BMVC), pages 87.1-87.12. BMVA Press, September 2016. ISBN 1-901725-59-6. doi: 10.5244/C.30.87. URL https://dx.doi.org/10.5244/C.30.87.
[71] Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pages 7472-7482. PMLR, 2019.
[72] Yang Zhao, Hao Zhang, and Xiuyuan Hu. Penalizing gradient norm for efficiently improving generalization in deep learning. In International Conference on Machine Learning, pages 26982-26992. PMLR, 2022.
[73] Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha C Dvornek, James s Duncan, Ting Liu, et al. Surrogate gap minimization improves sharpness-aware training. In International Conference on Learning Representations, 2021.
A.1 Estimating Test Robust Accuracy