Complexity Matters: Effective Dimensionality as a Measure for Adversarial Robustness

## Abstract

Abstract Quantifying robustness in a single measure for the purposes of model selection, development of adversarial training methods, and anticipating trends has so far been elusive. The simplest metric to consider is the number of trainable parameters in a model but this has previously been shown to be insufficient at explaining robustness properties. A variety of other metrics, such as ones based on boundary thickness and gradient flatness have been proposed but have been shown to be inadequate proxies for robustness. In this work, we investigate the relationship between a model’s effective dimensionality , which can be thought of as model complexity, and its robustness properties. We run experiments on commercial-scale models that are often used in real-world environments such as YOLO and ResNet. We reveal a near-linear inverse relationship between effective dimensionality and adversarial robustness, that is models with a lower dimensionality exhibit better robustness. We investigate the effect of a variety of adversarial training methods on effective dimensionality and find the same inverse linear relationship present, suggesting that effective dimensionality can serve as a useful criterion for model selection and robustness evaluation, providing a more nuanced and effective metric than parameter count or previously-tested measures.

## 1 Introduction

Robustness to adversarial perturbations has been a desired but often ill-achieved property of modern neural networks, that can otherwise be increasingly found in commercial real-world applications (Tocchetti et al., ). There has been a myriad of techniques proposed to improve robustness, ranging from preproccessor defenses (Diallo & Patras, ) (that have since been shown to be flawed by Carlini ()) to various forms of adversarial training methods. Adversarial training, since its introduction by Madry et al. (), has become one of the most common methods of defending against adversarial examples owing to its ease of implementation and deployment. It was however shown recently that adversarial training is far from a catch-all approach and suffers from overfitting (Rice et al., ).

A variety of different measures have been proposed to quantify the inherent robustness of trained models, but they were mainly found to be limited in their applicability Kim et al. (). We investigate the use of effective dimensionality as a measure for adversarial robustness. This measure was originally proposed by MacKay () and then expanded to deep neural networks by Maddox et al. (). We use effective dimensionality as a metric of model complexity – it is a joint measure of both the expressivity of the model’s architecture and the inherent properties of the dataset the model was trained on. This was used as it was found that parameter count alone does not provide a sufficient metric to represent the joint ‘complexity’ of a model and it’s underlying trained data.

We conduct experiments across a wide range of model architectures, such as ResNet (He et al., ), ShuffleNet (Zhang et al., ), and YOLO (Varghese & M., ), as well as over CIFAR-10, CIFAR-100, and ImageNet. Our key research contributions are as follows:

- 1.
We present a large-scale investigation into the effective dimensionality of classification models, most of which are often used in production.
- 2.
We find a near-linear negatively-correlated relationship between a model’s robustness to adversarial examples and its effective dimensionality. Namely, we find that models with a lower effective dimensionality exhibit improved robustness properties.
- 3.
We conduct an empirical investigation into the effect of adversarial training techniques on a model’s effective dimensionality. We show how these techniques reduce the models’ effective dimensionality and derive a linear trend in-line with the above findings.

## 2 Related Work

### 2.1 Adversarial Robustness

Threat Model For a given model $f_{\theta}:\mathcal{X}\rightarrow\mathbb{R}^{C}$ parameterised by $\theta$, an input $\mathbf{x}\in\mathcal{X}$ and label $y=_{\text{argmax}}f_{\theta}(x)\in\{0,\cdots,C\}$, we consider a perturbation $\mathbf{\delta}$ to be adversarial if

$$ $f_{\theta}(\mathbf{x}+\mathbf{\delta})\neq_{\text{argmax}}y$ (1) $$

Working from established literature such as Croce et al. (), we assume a white-box threat model for the attacker (i.e. full knowledge of both $f$ and $\theta$), and measure adversarial robustness via robust accuracy under attack for the $l_{\infty}$ norm. Details regarding the exact metrics used are discussed in Section 3.

Adversarial Example Generation Over the years, various strategies have been developed to assess and challenge the robustness of models. Among the most notable are Projected Gradient Descent (PGD) (Madry et al., ), AutoAttack (Croce & Hein, ), and Gaussian noise (Ford et al., ).

PGD, introduced by Madry et al. (), is one of the most widely studied adversarial example generation methods due to its effectiveness and simplicity. PGD is an iterative method that builds on the idea of applying small, constrained gradient-based perturbations to an input in order to maximize the model’s loss function. It is an iterative method, with the perturbation being projected back into a predefined bound around the original input to ensure the perturbation stays within acceptable limits. PGD is considered one of the strongest first-order methods and serves as the basis for adversarial training, as discussed later.

AutoAttack, proposed by Croce & Hein (), introduces a suite of strong adversarial attacks. AutoAttack automates the process of adversarial example generation generation and evaluation by combining four different attacks, including PGD and Fast Adaptive Boundary (Croce & Hein, ). This ensemble approach ensures comprehensive robustness evaluation by removing potential biases inherent in single-attack methods. It is designed to be parameter-free and eliminates the need for tuning, providing more consistent and robust adversarial performance testing across a variety of neural network architectures.

Ford et al. () explored the use of simple Gaussian noise as a baseline adversarial perturbation. While Gaussian noise is not a crafted example in the same sense as PGD or AutoAttack, it serves as a valuable point of comparison for more sophisticated methods. Though less effective than gradient-based attacks in fooling models, Gaussian noise highlights a model’s inherent robustness to random variations and provides insights into the stability of learned representations. Ford et al. () demonstrated that while Gaussian noise is often ineffective in generating adversarial examples, it remains a useful tool for probing the overall robustness of models, particularly in scenarios where adversarial examples are not deliberately targeted.

Adversarial Training We explore the effects of adversarial training and other derivative methods. This robust training technique was first formulated by Madry et al. (), utilizing the following min-max optimization with notation similar to the one used above:

$$ $\text{min}_{f}\mathbb{E}_{(\mathbf{x},y)\sim D}[\text{max}_{\delta}l(f(\mathbf {x}+\mathbf{\delta}),y)]$ (2) $$

The inner maximization is often approximated by repeated iterations of PGD, which greatly increases computation costs. Adversarial training is considered to be the state-of-the-art method for ensuring robustness, with recent papers greatly reducing involved costs by re-using gradient information during training (Shafahi et al., ).

### 2.2 Robustness Measures

Several measures have been proposed to quantify model robustness. Boundary thickness, introduced by Yang et al. (), focuses on exploring the relationship between the distance between decision boundaries and data points. It was found that models with thicker boundaries tend to exhibit greater robustness, as adversarial perturbations are less likely to cross decision boundaries with small perturbations. Flatness-based measures, studied by Stutz et al. (), focus on the curvature of the loss surface around the input data. Models with flatter loss landscapes are thought to generalize better and resist adversarial perturbations. Yang et al. () proposed the use of local Lipschitzness to assess robustness, relating to the smoothness of the learned decision function. It was found that a lower local Lipschitz constant implies greater robustness, as small perturbations induce minimal output changes. However, calculating the local Lipschitz constants directly is often computationally-intractable, and estimators need to be used which are in themselves computationally expensive on large models (Fazlyab et al., ).

Despite these advances, Kim et al. () demonstrated that none of these metrics provide a comprehensive measure of robustness. Each captures only specific aspects, leaving room for the development of more holistic and effective metrics.

### 2.3 Effective Dimensionality as a Complexity Metric

We work under the intuition that the robustness properties of a model are related to it’s complexity. The issue arrises with trying to concretely define ‘complexity’ when it comes to neural networks. A simple measure would be model size in terms of number of trainable parameters or amount of compute required, but this has recently been shown to only have a small correlation to robustness (Debenedetti et al., ). Grant & Wu () investigated the relationship between generalization and generalized degrees-of-freedom. We investigated this metric, but found that the method described in the paper to be generally computationally intractable and not converge for larger models.

Hence, we decided to explore the relationship between effective dimensionality, as defined by Maddox et al. (), and adversarial robustness. We found this metric to be appropriate due to it being a joint measure of both the expressivity of the model’s architecture and the inherent properties of the dataset the model was trained on. We felt that this captured the notion of what a neural network’s complexity should be in accordance to the universal approximation theorem (Hornik et al., ) :- the architecture of the model itself does not affect complexity until after the parameters are trained on, and the complexity would naturally depend on the kind of data the model was being fitted to.

The effective dimensionality of a symmetric matrix $A\in\mathbb{R}^{k\times k}$ is defined to be

$$ $N_{\text{eff}}(A,z)=\sum_{i=1}^{k}\frac{\lambda_{i}}{\lambda_{i}+z}$ (3) $$

where $\lambda_{i}$ are the eigenvalues of $A$ and $z>0$ is a regularization constant (MacKay, ). We compute the effective dimensionality of a neural network based on the eigenspectrum of the Hessian of the loss on the test data, according to the method described by Maddox et al. (). This metric has previously been shown to accurately track generalization properties of networks.

Intuitively, the effective dimensionality of a model explains the number of parameters that have been determined by the data, which corresponds to the number of parameters the model is using to make predictions. A model with a low effective dimensionality have a simpler function space, embodying Occam’s razor and generally avoid overfitting. The metric is directly related to the number of parameter directions in which the functional form of the model is sensitive to perturbation (Maddox et al., ).

## 3 Experiments

We conduct experiments over the CIFAR10, CIFAR100 (Krizhevsky, ), and ImageNet (Deng et al., ) datasets.

For ImageNet, we test the following model families: ResNet (resnet18, resnet34, resnet50, resnet101), ShuffleNetV2 (shufflenetv2_x0_5, shufflenetv2_x1_0, shufflenetv2_x1_5, shufflenetv2_x2_0), and YOLOv8 (yolov8n-cls, yolov8s-cls, yolov8m-cls, yolov8l-cls, yolov8x-cls).

For CIFAR10 and CIFAR100, we test the following models families: ResNet (resnet20, resnet32, resnet44, resnet56), MobileNetV2 (mobilenetv2_x0_5, mobilenetv2_x1_0, mobilenetv2_x1_5, mobilenetv2_x2_0), RepVGG (repvgg_a0, repvgg_a1, repvgg_a2), ShuffleNetV2 (shufflenetv2_x0_5, shufflenetv2_x1_0, shufflenetv2_x1_5, shufflenetv2_x2_0), VGG (vgg11_bn, vgg13_bn, vgg16_bn, vgg19_bn)

We conduct the following three large-scale experiments:

Measuring effective dimensionality We calculate the effective dimensionality of a large number of commercial-grade models on the aforementioned vision datasets. As described in Section 2.3, we use a slightly modified version of the code provided by Maddox et al. () to calculate effective dimensionality.

Robustness at different effective dimensionalities We calculate the robustness of the various models listed above. We use the implementation provided by Kim () to enable this. We run PGD, Auto, GN adversarial examples at varying increasing capacities ($\epsilon\in\{1/255,2/255,\cdots,8/255\}$ and $\sigma\in\{0.05,0.1,\cdots,0.4\}$ respectively). The robustness is reported as classification accuracy under attack $p^{*}$. To account for the fact that different models vary in their baseline (i.e. without any perturbations being applied) accuracy $p$, we report the relative performance under attack $p_{r}=\frac{p^{*}}{p}$. This gives a more practical metric for robustness that measures the relative degradation in performance under attack.

Effect of adversarial training on effective dimensionality We measure the effect of adversarial training and related techniques on effective dimensionality and relate this to the robustness properties of the respective models. We test ResNet18, WRN28, WRN34 using the MAIR framework (Kim et al., ) and the techniques described there. Namely, we consider the effect of extra data (Carmon et al., ), and using Adversarial Weight Perturbation (AWP) (Wu et al., ) as an optimizer. We look at and compare various training methods: Standard, AT (Madry et al., ), TRADES Zhang et al. (), MART (Wang et al., ).

## 4 Results

Figure: Figure 1: Measuring the effect of scale on a model’s effective dimensionality, on various datasets. The exact models tested from each model family are listed in Section 3.
Refer to caption: x1.png

### 4.1 Effective Dimensionality vs Model Size

In Figure 1 we demonstrated how model size, measured as the number of trainable parameters, affected the model’s effective dimensionality. These results are in-line with what would be expected based on the smaller models tested by Maddox et al. (). With the exception of outliers resnet on ImageNet, and vgg on the CIFAR datasets, we can see a clear polynomial
trend :- the effective dimensionality of the models decays as they get larger. In the outliers’ cases, the effective dimensionality remains more-or-less consistent or without any noticeable trend. We note in Section 4.3 that the robustness of these outliers tends to follow the trend of their effective dimensionality regardless.

### 4.2 Adversarial Performance vs Model Size

We corroborate past findings presented by Debenedetti et al. (); Bartoldson et al. () that larger, in terms of parameter count, models tend to be more less effected by adversarial examples with a given parameter budget. This trend however does not hold in all tested model classes and is marginal at times. This marginal relationship corresponds to recent similar findings in exploring robustness in LLMs given size that found little-to-no correlation between robustness and size as in Howe et al. ().

We find that higher-parameterised models tend to be more robust within the same model class. However, it was found that there is little correlation between model size and robustness between different model classes. A more detailed discussion of these results can be found in Section A.1.

### 4.3 Adversarial Performance vs Effective Dimensionality

Figure: Figure 2: Relative adversarial performance, plotted against the respective model’s effective dimensionality. A description of the performance metric is given in Section 3. We report the top-5 accuracy for AutoAttack and the top-1 accuracy for PGD and GN.
Refer to caption: x4.png

The graphs shown in Figure 2 illustrate the relationship between effective dimensionality and the relative performance of the different models (listed in Section 3) under different adversarial examples (described in Section 2.1), across multiple datasets (ImageNet, CIFAR100, CIFAR10). We observe the following trends:

AutoAttack Across all datasets (ImageNet, CIFAR100, CIFAR10), there is a general negative correlation between effective dimensionality and relative performance. Models with higher effective dimensionality tend to suffer more under AutoAttack, especially on ImageNet and CIFAR100. The main outliers to the general trends are ResNet (ImageNet) and VGG (CIFAR), much like in Section 4.1.

PGD The PGD results show a similar trend across ImageNet and CIFAR datasets, where models with lower effective dimensionality generally exhibit higher robustness. However, there are some inconsistencies, particularly in CIFAR100 and CIFAR10, where certain models (e.g., VGG on CIFAR datasets) exhibit an increase in performance at higher dimensionalities, likely due to model-specific behavior.

Gaussian noise When subject to Gaussian noise, models across all datasets show a clear inverse relationship between effective dimensionality and performance. Higher dimensionality again corresponds to lower robustness.

Overall, the results suggest that lower effective dimensionality generally correlates with better adversarial robustness across various adversarial example generation methods and datasets, though model-specific variations do exist.

### 4.4 Effect of Adversarial Training on Effective Dimensionality

In Figures 3 and 4 we demonstrate the effect of various adversarial training techniques on the effective dimensionality of models and how this relates to their adversarial robustness.

The results are incredibly clear-cut. We see that in nearly-all cases the Standard-None setup has the highest dimensionality, with any additional adversarial training measures significantly lowering this metric. For ResNet18, AWP reduced dimensionality by $24.0\%$, and AWP+ED by $29.3\%$ relative to the baseline, on average. For WRN28, AWP reduced dimensionality by $23.2\%$ and AWP+ED by $30.4\%$, on average. For WRN34, AWP reduced dimensionality by $19.1\%$ and AWP+ED by $31.3\%$, on average.

The effect of these dimensionality reductions on adversarial robustness can be seen in Figure 4. We remove outliers (mainly poorly pre-trained models) when plotting these results. Here we see a highly-correlated (lowest $R^{2}$ is $0.73$ in the WRN34 case) relationship between effective dimensionality and the relative adversarial performance. Based on the linear regression, we can see that drop in effective dimesionality of $10$ points corresponds to roughly an absolute increase of $5.5\%$ in relative adversarial performance. These results are in line with the ones described in Section 4.3 above, namely that lower effective dimensionality generally correlates with higher robustness.

Figure: Figure 3: Effect of various adversarial training methods, described in Section 3, on the effective dimensionality of the respective models. AWP corresponds to Adversarial Weight Perturbation, and AWP+ED involves AWP and extra training data.
Refer to caption: x13.png

Figure: Figure 4: Relative adversarial performance under a various adversarial training methods, plotted against the respective model’s effective dimensionality. A description of the performance metric is given in Section 3. AWP corresponds to Adversarial Weight Perturbation, and AWP+ED involves AWP and extra training data.
Refer to caption: x16.png

## 5 Discussion

The results presented in Section 4 build upon the work of Maddox et al. () who claim that the effective dimensionality of a model is a good mechanism for model selection. We show that this selection criteria does not only correspond to an improvement in generalization but also to adversarial robustness.

Across all experiments, a clear inverse relationship between effective dimensionality and robustness under adversarial examples emerges, consistent with previous findings that models with lower effective dimensionality tend to be more robust to adversarial perturbations. The adversarial training methods tested consistently reduced the effective dimensionality of models compared to standard training, confirming that adversarial training not only enhances robustness but also simplifies the model’s internal structure in terms of its dimensionality.

Although effective dimensionality has shown promise as a proxy for measuring adversarial robustness, it is not a perfect predictor as shown by the slightly mixed results presented in Section 4.3. Other factors, such as the loss landscape flatness, boundary thickness, or specific adversarial training mechanisms, may also contribute significantly to the model’s ability to withstand adversarial examples. This aligns with the conclusion of Kim et al. () that no single robustness measure is comprehensive, leaving space for the development of more holistic metrics.

### 5.1 Limitations

Several important limitations must be noted. First, due to resource constraints, we were unable to test larger more complex models. As a result, our findings are limited to smaller model architectures, and it remains unclear whether the observed trends hold for larger models, which could exhibit different behaviors with respect to dimensionality and robustness.

Additionally, the results are based purely on empirical analysis, and while we observed consistent patterns, these findings do not establish a direct causal relationship between effective dimensionality and adversarial robustness. For example, there were clear outliers present in Section 4.3 and Section 4.1. This points to the need for further theoretical exploration to better understand the underlying mechanisms and interactions between dimensionality and other factors, such as the loss landscape or boundary geometry.

### 5.2 Future Work

We conducted an extensive investigation into how model scale and different adversarial training methods affect effective dimensionality, and how this relates to the models’ adversarial performance. However, it will be worthwhile to conduct an evaluation into how model quantization and distillation affect this measure in relation to robustness. A future branch of work would also involve conducting a similar exploration for other architectures, such as vision transformers, and on different domains, such as reinforcement learning environments.

Given the observed trends, future research could explore the integration of multiple robustness metrics, including effective dimensionality, boundary geometry, and loss flatness, to develop a more unified and accurate predictor of robustness. It would also be worthwhile to explore why certain models (such as ResNet and VGG, mentioned in Section 4.1) are outliers from this observed trend.

## 6 Conclusion

In this paper, we presented an extensive empirical investigation into the relationship between effective dimensionality and adversarial robustness in deep neural networks. Through our experiments on production-scale models, including YOLO and ResNet architectures, we demonstrated that effective dimensionality serves as a strong predictor of robustness to adversarial examples. Specifically, we found a linear inverse correlation between the two, showing that models with lower effective dimensionality tend to exhibit greater robustness. This trend holds both within architecture families, where larger models generally possess lower complexity, and under adversarial training techniques, which further reduce effective dimensionality in line with improved robustness.

These findings suggest that effective dimensionality can serve as a useful criterion for model selection and robustness evaluation, providing a more nuanced and effective metric than parameter count or existing flatness- and boundary-based measures. However, our study is limited to empirical observations, and further theoretical work is required to fully understand the mechanisms driving the relationship between complexity and robustness. Nonetheless, our work lays a foundation for future research on the role of effective dimensionality in adversarial robustness and model optimization.

#### Acknowledgments

David Khachaturov is supported by the University of Cambridge Harding Distinguished Postgraduate Scholars Programme.

## Appendix A Appendix

### A.1 Adversarial Performance vs Model Size

Figure: Figure 5: Relative adversarial performance, plotted against the respective model’s size, measured in number of trainable parameters. A description of the performance metric is given in Section 3. We report the top-5 accuracy for AutoAttack and the top-1 accuracy for PGD and GN.
Refer to caption: x19.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: ## Questions
* 149: Calculating the hessian is generally super expensive. Is there some trick here to make it scale? Or does this only work on small models?
* 215: "clear polynomial trend" I think this is significantly too generous -- there appears to be a lot of bumpiness in the plots. Do you know what's going on there? Can you get more seeds?
* 247: it seems much too strong to say that "models across all datasets" show a clear inverse trend when several of the lines go up and not down in the right column of figure 2 plots. Am I misunderstanding something?
* can you get more seeds? it's hard to tell what's going on with only one seed.
* I don't really understand what's going on with the adversarial training setup and plots (figure 3). Maybe you can give more details on the specific of adversarial training? I think there could be interesting details to look through here.

## Specific things:
* line 34: "has been a myriad of" -> "have been myriad"
* line 35: isn't postprocessing an important thing too?
* 39: delete "recently"
* 48: it's -> its
* 53: what does "most of which are often used in production" mean?
* 66: please make it obvious you're in the computer vision setting
* 68: delete the argmax
* 72: use \ell instead of l
* 74-99: why these three methods? there are many many adversarial example finding techniques. why did you choose these specifically? why no FGSM as a baseline?
* 103: please define what is in the equation; people won't have necessarily read the previous papers
* 106: I think it's generous to say adversarial training is SOTA. it's a go-to technique and important tool, but it alone will not get you SOTA robustness
* 129: it's -> its
* 173: exact -> specific
* 212: demonstrated -> show
* 215: the ":-" is not something I've seen before and I don't know what it means
* 233: graphs -> plots
* 258-263: writing it out like this is hard to follow. make a table and highlight one important result from it if you want?
* 265: "we removed outliers" maybe you can find a different way to include the outliers (maybe make a scatter but don't include them in the fit and make it obvious by using a different color?). Removing data (which I'm grateful you mentioned in the text) raises suspicions of cherry-picking.
* 269: delete "above"
* 323: slightly mixed -> mixed
* 380: extensive investigation -> investigation
* 393: same
* 395: you mention production-scale models here, but previously you just said you had to do only small models because of computational constraints. I would expect either that these models are production-scale and that's ok, or that you don't have the compute for production-scale and that's ok, but not both?

## General note
With 3 seeds per model (possibly even only for one family of models and one adversarial attack method) this could be a great workshop paper. As is, I can't recommend it for main conference. I could imagine increasing my rating to a 5 if more seeds are run, all the data are shown, clearer justification is given to attack methods, and the results look less noisy OR are talked about in a less superlative way. Going deeper into the adversarial training direction could be quite interesting and would start to look more like a main conference paper to me, though I imagine that might be infeasible to do in the rebuttal time-window.
soundness: 3
strengths: The paper is relatively clearly written.

The question being asked and answered is compelling.
summary: The authors explore the relationship between "effective dimensionality" and robustness for a collection of computer vision classification models.
weaknesses: * the paper is quite short and sweet: some experiments are conducted, on small models, on a single seed.
* there is no "novel" contribution from the paper. In itself, this is not a problem at all. But without a novel contribution, I would expect the analysis to be more in-depth (not being able to run larger models is understandable; perhaps running more than one seed and more models is feasible? Or some of the other directions that the authors mention could be explored: looking at other architectures or boundary geometry or loss flatness. Or more exploration of what happens during adversarial training.
* the paper is too superlative in its writing, almost to make up for interesting but not the strongest-looking results (which is also ok, just means more investigation is needed).
* overall it seems like valuable work to be presented at a computer vision or robustness workshop, but the contribution is not enough to meet the ICLR main conference bar in my current opinion
* handful of typos

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: See weaknesses.
soundness: 1
strengths: * The paper is generally well-written and clear. 
* The paper addresses and important topic: identifying factors that either are causally related to, or correlate very strongly with adversarial robustness is an important area of research, for cheap model selection or for understanding how to train adversarially robust models.
* The ideas proposed by the paper are novel, to my knowledge.
summary: The authors evaluate adversarial robustness and effective dimension on an array of models of different architectures, trained with different datasets, and trained with various adversarial training algorithms. The authors plot the relationship between robustness and effective dimension and make four claims: (1) Effective dimension sometimes decreases with model size; (2) adversarial robustness is correlated with model size within the same model class, but not outside; (3) adversarial robustness and effective dimension are negative correlated; (4) adversarial training reduces effective dimension.
weaknesses: The primary weakness of this paper is major: the evidence presented in this paper does not sufficiently support the claims made by the authors. The primary premise of the paper is built by the following claims made by the authors: (1) adversarial robustness and effective dimension are negatively correlated; (2) effective dimension decreases with adversarial training.
1. The negative correlation between adversarial robustness and effective dimension is weak at best, and often yields the opposite trend. The evidence for this claim is given in Figure 2. In this figure, ResNet appears to clearly have the claimed trend in 1 out of 9 plots (bottom right only). ShuffleNetV2 has the opposite trend in 2 out of 9 plots, and appears to have no correlation in another 2 out of 9, leaving only 5 out of 9 plots with the claimed trend. VGG seems to have the opposite trend or appears noisy in all plots in which it appears.  RepVGG has the opposite trend in the bottom right trend. MobileNetV2 seems to have the opposite trend, or noise in 4 out of the 6 plots in which it appears (all but the bottom left two plots). In total, the only model that actually observes the claimed trend is YOLO, which only appears in three of the plots. 
2. The fact that adversarial training decreases effective dimensionality has slightly stronger, but still not sufficient evidence: the adversarial training in Figure 3 left has greater effective dimensionality than the standard training. Further, the relationship between effective dimensionality and robustness does not appear correlated when considered training without AWP in two of the three plots in Figure 4 (center and right, considering the circle points). It seems, therefore, that perhaps AWP leads to both substantially smaller effective dimension and also better robustness, but the claim made by the authors, that robustness in general is negative correlated with effective dimensionality, appears false.

Related work:
* AWP is closely related to Sharpness-Aware Minimization, and so it would be beneficial to include some discussion of the relationship between Sharpness-Aware Minimization and effective dimension. For example, see [1].

Minor:
* The clarity of your plots could be improved by making the legends not overlap with the plotted lines, and by keeping colors consistent across nearby plots (e.g., the models are given different colors in different plots in Figure 2)

[1] Andriushchenko, Maksym, et al. "Sharpness-aware minimization leads to low-rank features." Advances in Neural Information Processing Systems 36 (2023): 47032-47051.

---

confidence: 4
contribution: 1
initial_rating: 3
presentation: 2
questions: **Question 1:** Any conjecture to explain the clear correlations between the effective dimensionality and the model robustness
- Would you explain why clear relationships are observed, which is not shown in the prior metrics?

**Question 2:** Relationships between the effective dimensionality and the model flatness
- In Eq. 3, the effective dimensionality seems to be strongly related to the eigenvalues of Hessian. When revisiting the traditional measurements of model flatness on the parameter space, i.e., the maximum or summation of the eigenvalues of Hessian, the effective dimensionality increases when the model flatness decreases. Thus, it is hard to figure out why the effective dimensionality is particularly showing the more clear relationships to the model robustness. Would you share your thoughts on the relationships between the effective dimensionality and the model flatness?
soundness: 3
strengths: **Strength 1:** This paper provides the empirical findings to understand the robustness capability of various deep model architectures.
- Evaluating and understanding the robustness of deep models, or the generalization capability with a wide viewpoint, are long-lasting issues crucial in the theoretical understanding of the parametric model's capability and in realizing robust AI models in applications. From this perspective, the main strength of this paper is that it provides a novel empirical finding to understand model robustness.
summary: In this paper, the authors have investigated the relationship between the effective dimensionality and the model robustness, where the effective dimensionality shows a negative linear correlation with the model robustness.
weaknesses: **Weakness 1:** Limited theoretical understanding of the effective dimensionality
- This paper only investigates the empirical results to show the relationships between the effective dimensionality and the model robustness, but its theoretical understanding is not yet explored.
- Specifically, to further provide in-depth understanding, it would be better to explain why the effective dimensionality correlates more clearly with the model robustness than other measurements.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See above.
soundness: 1
strengths: 1. The paper investigates the relationship between two metrics, models' effective dimensionality and adversarial robustness. Various empirical results are presented, which are helpful for readers to get the relationship of two these results.

2. The paper is generally easy to read.
summary: The paper links adversarial robustness to the models' effective dimensionality, using empirical experiment to investigate the relationship between these two aspect of the model.  The paper perform various experiments on different datasets (e.g. CIFAR10, CIFAR100, ImageNet) across different model architectures (e.g. ResNet, ShuffleNet, Yolo) and tries the prove the negative relationship between effective dimensionality and adversarial robustness. However, empirical results cannot validate such claim.
weaknesses: 1. The definition of effective dimensionality is highly correlated to the adversarial accuracy itself. Actually, it seems to be a tautology of adversarial robustness. effective dimensionality measures the flatness of loss landscape on the test data using Hessian Matrix while adversarial robustness reflects the flatness of loss landscape using the accuracy. 

2. The empirical result cannot prove the claim of the paper. As shown in Figure 2, the relationship between effective dimensionality and adversarial robustness is chaotic without clear trend.