Post-mortem on a deep learning contest: a Simpson’s paradox and the complementary roles of scale metrics versus shape metrics
Authors: Michael W. Mahoney
charles h martin
Venue: Submitted to ICLR 2023 (2023)
Keywords: model diagnostics
heavy-tailed self regularization
Simpson's paradox

Abstract: To understand better good generalization performance in state-of-the-art neural network (NN) models, and in particular the success of the AlphaHat metric based on Heavy-Tailed Self-Regularization (HT-SR) theory, we analyze of a corpus of models that was made publicly-available for a contest to predict the generalization accuracy of NNs. These models include a wide range of qualities and were trained with a range of architectures and regularization hyperparameters. We break AlphaHat into its two subcomponent metrics: a scale-based metric; and a shape-based metric. We identify what amounts to a Simpson’s paradox: where “scale” metrics (from traditional statistical learning theory) perform well in aggregate, but can perform poorly on subpartitions of the data of a given depth, when regularization hyperparameters are varied; and where “shape” metrics (from HT-SR theory) perform well on each subpartition of the data, when hyperparameters are varied for models of a given depth, but can perform poorly overall when models with varying depths are aggregated. Our results highlight the subtlety of comparing models when both architectures and hyperparameters are varied; the complementary role of implicit scale versus implicit shape parameters in understanding NN model quality; and the need to go beyond one-size-fits-all metrics based on upper bounds from generalization theory to describe the performance of NN models. Our results also clarify further why the AlphaHat metric from HT-SR theory works so well at predicting generalization across a broad range of CV and NLP models.

Under review as a conference paper at ICLR 2023
POST-MORTEM ON A DEEP LEARNING CONTEST: A SIMPSON'S S PARADOX AND THE COMPLEMENTARY ROLES OF SCALE METRICS VERSUS SHAPE METRICS
Anonymous authors Paper under double-blind review
ABSTRACT
To understand better good generalization performance in state-of-the-art neural network (NN) models, and in particular the success of the AlphaHat metric based Rell-eeularizatinn (HT-SR) theory, we analyze of a corpus of
on Heavy-Tailed models that was made aoolicly-available for contest predict the generaliza- tion accuracy of NNs. These models include a wide range of qualities and were trained with range of architectures and regularization hyperparameters. We break A AlphaHat into its two subcomponent metrics: a scale-based metric; and shape- based metric. We identify what amounts Simpson's 's paradox: where "scale" metrics (from traditional statistical learning theory) perform well in aggregate, but can perform poorly on subpartitions of the data of given depth, when regulariza- tion hyperparameters are varied; and where "shape" metrics (from HT-SR theory) perform well on each subpartition of the data, when hyperparameters are varied for models of a given depth, but can perform poorly overall when models with varying depths are aggregated. Our results highlight the subtlety of comparing models when both architectures and hyperparameters are varied; the complementary role of implicit scale versus implicit shape parameters in understanding NN model quality; and the need to go beyond one-size- sizesii-fitt-all metrics based upper bounds from generalization theory to describe the performance of NN models. Our results also clarify further why the A phaHat. metric from HT-SR theory works well predicting generalization across broad range of CV and NLP models.
INTRODUCTION
is of increasing interest to develop metries to measure and monitor the quality of Deep Neural Net- work (DNN) models, especially in production environments, where data pipelines can unexpectedly fail, training data can become corrupted, and errors can be difficult to detect. There are few good methods which can readily diagnose problems at layer-by-layer level and in automated way.
Motivated by this, recent work introduced the A AlphaHat metric, (i.e., â), showing that it can predict trends in the quality, generalization capacity, of state-of-the-an (SOTA) DNN models without access to any training or testing data (Martin et al., 20220022222 outprrrorrrooiiiigg other metrics from statistical learning theory (SI.T) in large meta-analysis of hundreds of SOTA models from computer vision (CV) and natural language processing (NLP). The â metric is based on the recently-develope Heavy-Tailes Self- Regulaarraiii (HT-SR) theory (Martin & Mahoney, 2021; 2019; 2020), which is based on statistical mechanics and Heavy- -aalled (HT) random matrix theory. Further, being a weighted average of layer metrics, understanding why AlphaHat works will help practitioners to diagnose potential problems layer-by-layer.
In this paper. we evaluate the AlphaHat (â) metric (and its subcomponents) on a series of pre- trained DNN models from recent contest ("the Contest") to predict generalization in deep learn- ing (Jiang et al., 2020a;b). The Contest was interested in metrics that were "causally informative of generalization," and it wanted participants to propose "robust and general complexity mea- sure" (Jiang et al., 2020a;b). These Contest models were smaller and more narrow than those analyzed in the large-scale meta-analysis (Martin et al., 2021). However, for that narrower class of models, the Contest data was more detailed. There were models with a wider range of test accura- cies, including models that generalize well, generalize poorly, and even models which appear to be overtrained. The models are partitioned into sub-groups of fixed depth, where regularization hyperpa- rameters (and width) are varied. This more fine- fininng set of pre-trained models lets us evaluate

---

Under review as a conference paper at ICLR 2023
the á metric, and its subeomponents, across the opposing dimensions of depth and hyperparameter changes, and more finely than analyzed previously on SOTA models.
Our analysis here provides new insights on how theories for generalization perform well-trained versus poorly-trained models; on how this depends in subtle ways on what can be interpreted as implicit scale versus implicit shape parameters of the models learned by these DNNs; and how model quality metrics depend on architectural parameters versus solver parameters. Most importantly, this work helps clarify why the AlphaHat metric performs so well across so many models.
Background: Heavy-TTailed Self- ceff-Regularization (HT-SR) Theory. HT-SR theory is a phe- nomenology, based on Random Matrix Theory (RMT), and motivated by the statistical mechanies of learning, that explains empirical results the spectral (cigenvalue) properties of SOTA DNNs (Mar- tin & Mahoney, 2021). (A detailed discussion of HT-ST theory can be found in Martin & Mahoney (2021); here we can only summarize the basics.) Empirical results (Martin & Mahoney, 2021; 2019; 2020) show that, for nearly all well-trained DNN models (in CV and NLP), the layer Correlation Matrices X N W W are HT, in the sense of being well-fit to a Power Law (PL) or truncated PL distribution (even though individual matrices W are not HT elementwise). Moreover, HT-SR theory indicates that as training proceeds and/or regularization is increased, the HTness of the correlations (however it is measured) generally increases (Martin & Mahoney, 2021). Using these facts, HT-SR theory allows one construct various generalization capacity metrics for DNNs (which are imple- mented in the We ght cher open source tool (wei, 2018)) that measure the model average HTness (A 1hap, AlphaHat, LogSpect ra 1N rm, etc.) as a proxy for generalization capacity.
For moderately and very HT weight matrices, it is possible to quantify the HTness using a standard PL fit of the ESD. In this case, smaller PL exponents (a) correspond to heavier tails. Given a DNN weight matrix W, (N x M, N > M), let À be an eigenvalue of the correlation matrix X = N W' W. The Empirical Spectral Density (ESD), p()), is the just an empirical fit of the histogram of the M cigenvalues. In looking at hundreds of models and thousands of weight matrices, the tail of the ESDs of well trained DNNs can nearly always be well fit to a PI. distribution:
Piaa x Imin X S Imaz-
Here, Tmaz Amar is the maximum eigenvalue of the ESD, and Zmin is the start, fit using the procedure of Clauset et al. (2009). The best fit is given by the KS-distance Dks (denoted qualityOfAlphate
below). For models that generalize well, the fitted a 2 2.0 for (nearly) every layer (Martin & Mahoney, 2021; 2019; 2020). Models that generalize better generally have smaller (weighted) average a, A1 phaHat (7), when compared within an architecture series (VGG, ResNet, ...) or with different size data sets (GPT vs GPT2) (Martin al., 2021).
Still one may ask, "Why are the layer correlations matrices X HT, but the weight matrices W themselves are not ?" For the ESD of W to be HT, either W must be HT elementwise, having many spuriously large elements Wij. and/or X simply has many large eigenvalues. It is well known that the well-regularized models should not contain spuriously large elements Waj. and methods like L2-regularization attempt to ensure this. In contrast, the largest eigenvalues of X will correspond to eigenvectors with the most important non-random information. Consequently, Amaz will become larger, not smaller, as more information is learned. (This is well known from RMT. However, it is also known in machine learning, as it is the basis for methods like Latent Semantic Analysis (LSA).) The HT-SR theory exploits these facts to build theory of generalization for DNNs.
Using layer PL fits, one can define the HT-based AlphaHat metric for an entire DNN model. This metric can predict trends in the quality of SOTA DNNs, even without access to training testing data (Martin et al., 2021). 1). HaphaHat (â) is a weighted average over L layers of two complementary metrics, the PL exponent and the logarithm maximum eigenvalue log Amas.
à
aj log Xmaz
AlphaHat is both weighted- Alpha, weighted by the Scale of the layer ESD (log X X and a weighted-average LogSpect ralNorr., weighted by the Shape of the layer ESD(a). We therefore evaluate how these two subcomponents, the Alpha and LogSpectralNd rm metrics, individually perform when varying the opposing dimensions of depth (number of layers L) and regularization hyperparameters (f), such as dropout, momentum, weight decay, etc.
In investigating "why AlphaHat works," we discovered that the Alpha and LogSpect x a lloorm metrics frequently display Simpson's 's paradox. Being aware of challenges with designing good
2

---

Under review as a conference paper at ICLR 2023
contests and with extracting causality from correlation (Pearl, 2009), we did not use the causal metric provided by the Contest. Instead, we adopted a different approach: we identified Simpson's s paradoxes within the Contest data; and we used this to understand better the AlphaHat metric from HT-SR theory. A Simpson's 's paradox can arise when looking for trends in a large data set containing of multiple sub-groups; it occurs when trend for the total data set reverses-or disappears- when considering the sub-groups individually (Simpson, 1951; Bickel et al., 1975; Robinson, 2009; Kievit al., 2013). Simpson's 's paradoxes are particularly relevant when one wants to understand the data and attribute causal interpretations to the underlying data or model.
Because A AlphaHat combines two complementary metrics, the LogSpect and Alpha, we consider two classes of metrics that have been used to predict DNN model quality: norm- based metrics from SLT (TogSpect. 1 Norm); and HT--based metrics from HT-SR theory (A pha). The norm-based metrics describe the Scale associated with the model implicitly-leamer by the training process, and they have been used to provide generalization theory upper bound on simple models. Empirically, they can perform in strange and counter-intuitivy ways even moderately- realistic models (Jiang et al., 2019). The HT (or other PL) metrics describe the HTness of the layer correlations using the Shape of the implicitly-learnee model, are related to HT-SR theory (Martin & Mahoney, 2021; 2019; 2020). The Shape-based Alpha metric has been conjectured to predict test accuracy when only varying regularization hyperparameters, and the A AlphaHat metric was developed to improve A pha by correcting for the Scale of different layers when comparing similar models of differing depth (i.c., VGGII VGG199 (Martin & Mahoney, 2021; Martin et 2021).
Our main results. We apply these SLT and HT-SR metries to predict generalization accuracies of the pre-trained models provided in the Contest, as a function of both model depth I. and regularization hyperparameters 0. Our main contributions are the following.
New shape-based metries. Based on our preliminary analysis of the data, we introduce two new data-free quality metrics- -Alpha and QualityOfAlphaa
motivated by the HT-SR theory.
Existence of Simpson's 's paradox in Contest data. We examine models provided by the Contest, and we identify Simpson's 's paradoxes. Depending on the specific Contest task and model sub-group, LogSpect ra Norm and A Alpha are either strongly anti-correlated correlated with each other. For both Contest tasks (Task1 and Task2), and each model sub-group, the LogSpect ra ralNorm trend (at best) increases with increasing model quality (in disagreement with SLT). For Taskl, it increases when considering all models together, however, for Task2, it decreases when grouping all models, exhibiting a clear Simpson's paradox. Also, for both Contest tasks, and for each model subgroup, Alpha decreases with increasing model quality (in agreement with HT-SR theory). For Task1, it decreases when considering all models together; however, for Task2, it increases when aggregating all models, again exhibiting a clear Simpson's 's paradox.
modestly to weakly anti-
That is: (1) for both LogSpect ra rm and Alpha, the Task2 models exhibit a Simpson's 's para- dox; (2) the predictions of paa.Sppeetaaaooooo Nm disagree with (SLT) theory, except when aggregated when a Simpson's 's paradox is present; and (3) the predictions of Alpha (and LogSpect r No) agree agree with (HT-SR) theory, except when aggregated when Simpson's 's paradox is present.
Shape parameters and hyperparameter variation. The Scale based LogSpect ra aaappettraNaonnn LNorm can do
well when the data aggregated, but it does very poorly when the data are segmented by architecture type (here, depth). More generally, this metric predicts test accuracy with large architectural changes, e.g., depth. However, it may behave oppositely that suggested by bounding theorems, meaning that it can be anti-correlated with test accuracies, when varying the regularization hyperparameters. This confirms unexplained observations made in a different setting (Jiang et al., 2019).
The Shape based Alpha from HT-SR theory is predictive of test accuracy as hyperparameters vary, with large-scale architectural changes held fixed. Our results are the first to demonstrate that Alpha- a Shape metric- vetric-prediecs test accuracy, as hyperparameters 9 are varied independently. Also, and importantly, the A Alpha metric performs better for higher quality models; this evident comparing results for the Task2 2xx versus 6xx models. For models with better test accuracies, the A Alpha metric predicts them better. This is seen visually in Figure 5, and also Table 5, which compares both linear and rank correlation metrics, for model sub-groups with fixed number of layers L.
Extracting scale and shape metrics from pre-trained DNN models. While computing norm-based Scale metrics is straightforward, computing HT-based Shape metries is much more subtle. From HT-SR theory (Martin & Mahoney, 2021), we want to fit the tail of ESD of layer weight matrices to PL. distribution, in Eqn. (1). This is accomplished with the We ghtWatcher tool) (wei,
3

---

Under review as a conference paper at ICLR 2023
2018). The parameters Zmaz (the largest eigenvalue) and a (the PL exponent) in Equ. (1) (below) have a natural interpretation in terms of the scale and shape of the PL distribution, so we interpret the corresponding rogSpecttraNoooo and Alpha as Scale and Shape DNN model quality metries.
2 PRELIMINARIES AND RELATED WORK
Predicting trends in model quality. There is a large body of (older (Bartlett, 1997) and more recent (Neyshabur et al., 2015; Bartlett et al., 2017; Arora et al., 2018)) work from SLT on providing upper bounds on generalization quality of models; and there is a smaller body of (older (Engel & den Broeck, 2001) and more recent (Zdeborová & Krzakala, 2016; Martin & Mahoney, 2021; Bahri et al., 2020)) work using ideas from statistical mechanics to predict performance of models. Most relevant for are the recent results on "Predicting trends in the quality of state-of oft--h--rtttttttt neural networks without access to training or testing data" (Martin et al., 2021) and "Fantastic generalization measures and where find them" (Jiang et al., 2019). The former work (Martin et al., 2021) considered very broad range of CV and NLP models, including nearly every aavilalee pre- te-irained model, totaling to hundreds of SOTA models; and it focused on metrics that perform well on SOTA, without any access to training and/or testing data. The latter work (Jiang et al., 2019) considered larger number models drawn from a much narrower range of CV models; and it considered broad range of metrics, most of which require access to the training and/or testing data.
The Contest used a casual measure, similar to that in Jiang et al. (2019), to evaluate all the models, for both tasks, in aggregate, opposed to evaluating the individual sub-groups, as we do here. This causal metric is highly suspect. In particular, we point to Dziugaite et al. (2020), who used very different methods than ours to obtain conclusions consistent with ours. They noted that the kind of causal measures proposed by Jiang et al. (2019) can obscure failures and successes; and they argued that generalization studies should be evaluated using an aggregate measure of "distributional robustness." There is also work that attempts to identify reliable generalization metrics, but using different evaluation methods than ours (Dziugaite et al., 2020; Liao al., 2018; Thomas et al., 2019).
None of the works above, however, have considered that Simpson's paradoxes can arise in this kind of analysis. Pearl has argued for some time that causal relationships can not be inferred directly from observational data because of the presence of potential Simpson's 's paradoxes (Pearl, 2009). Simpson's ' paradoxes often arise in social science and biomedicine, since understanding causal relationships is particularly important in those areas. To our knowledge, this is the first application of these ideas in this area. Importantly, we would not have been able to observe the tradeoff between Scale and Shape metrics if we had chosen to use an aggregate, or, worse, causal measure, here.
Models from a contest. The Contest (Jiang et al., 2020a;b) covered a rather narrow class of CV models, but it provided multiple versions of each, trained with different numbers of layers (depths) and regularization hyperparameters. The approximately 150 CV models were organized into two architectural types, VGG-like models and Wetwork-in-Netwoor models, each with several sub- sub-groups of fixed depth (number of layers L), and trained with different batch sizes, dropout, and weight decay.
See Table 4 in Appendix B for a summary of the models and aub-groups. There are two model groups: Taskl and Task2. In every model sub-group (each row in Table 4), all models have the same depth (number of layers). Taskl contains 2 Conv2D widths in each subgroup, whereas in Task2, only the the regularization is varied. Briefly, the models are:
Task1 ("taskl_v4") 96 VGG-like models, trained on CIFAR1O, with 4 subgroups having 24 models each: Oxx, 1xx, 2xx, 5xx, 6xx, and 7xx.
Task2 ("task2_y1 1)):: 54 models, with stacked Dense layers, trained on SVHN, with subgroups having 18 models each: 2xx, 6xx, 9xx, and 10xx.
3
EXTRACTING SCALE / SHAPE PARAMETERS FROM PRE-TRAINED MODELS
3.1 COMPUTING SCALE AND SHAPE PARAMETERS WITH NORMS AND PL EXPONENTS
For completeness, we have evaluated several quality metries, including: two from SLT (including LogSpect r INorm rm and LogF robeni usNorm) (Jiang et al., 2019; 2020a;b;; two from statistical mechanics and HT-SR Theory (including A AlphaHat and LogglpphShatten tenoorm (Martin & Mahoney, 2021; 2019; 2020); and two (Alpha and Qualitt
that we introduce in this paper. See Table 3 and the discussion in Appendix A for a summary of the best performing.
All metrics reported are averages over the individual layer- metrics, as computed using the publi- cally available and open-source We Wei gettaattttthhheeee (wei, 2018). Both computing the norm
4

---

Under review as a conference paper at ICLR 2023
109 (og FS0 VSS16
Un.un ESD VG3
[gg
ESD VOGTS
KS Ditance (Pyd vs
107
6.8
0.75
0.6
10"2
c.4
002
0.2
0.25
2008
40
200:2
(a) ESD (log-log plot)
(b) ESD (lin-lin plot)
(c) ESD (log- lin plot)
(d) PL fit quality Zmin
Figure 1: Illustration of the role of the ESD shape in determining the PL parameter a in the PL. fit.
Shown is VGGI6, FC3 (a nearly "ideal" example). (See Appendix C for less ideal examples.)
metrics and fitting HT distributions has subtleties (Clauset et al., 2009; Newman, 2005), and WeightWatcher encaspulates the process, making it straightforward and reproducible.
"Scale" parameters. The LegSpect ra INorm and LogF roofriobeiusNorn are layer averages over the logarithm of the corresponding norm (Jiang et al., 2019). (See Table 4 for details.) These norms have natural interpretation as layer "scale" implicitly learned by the NN during training. The LogSpectza persspeeczaloonn is also measure of average HTness, and is anti-correlated with Alpha in many cases.
"Shape" parameters. The A Alpha metric takes the layer average of the PL exponent o, as fit to the tail of the layer ESD (A). The parameter that is the same one that arises in the recently- developed HT-SR theory (Martin & Mahoney, 2021; 2019; 2020). The QualityOfi wottyyffffaaraaa Satnngorov-Smmmmmo test (or KS-test),
metric is the average Goodness of Fit, determined using denoted Dss. Neither Alpha nor QualityOfAlphaf it have been considered previously.
Combining shape and scale. The AlphaHat (Martin et al., 2021) and LooAllphaShattenNNoz (Martin & Mahoney, 2020) combine Shape and Scale. AlphaHat which is a weighted average of layer a, weighted by the layer log Amaz- LogAlphaShattennooo. is logarithm of the Shatten norm of layer weight matrices, where the Shatten parameter for each layer is the a from the PL. fit of that layer. The AlphaHat metric approximates the LogalphaShattenl rm for the tail of the ESD, but both give similar results in practice.
3.2 FITTING ESDS TO PLS
In practice, these PL fits require some care to obtain consistent, reliable results; and, for this reason, here we describe some of these issues. Among other things, is important to understand the behavior of the PL fit Imin is varied. See Figure for illustrative (very "nice") example; and see Appendix for more discussion of these fitting issues, including dealing with non-ideal ESDs. (These figures can be generated by the WeightWatcher tool.)
In Figure 1, we consider the third fully connected (FC) layer from VGG16 (which was studied previously (Martin & Mahoney, 2021; Martin et al., 2021)), which is nearly "ideal" example that we use to illustrate the method. In 1(a), 1(b), and 1(c), respectively, we show the ESD in log-log plot, linear-linear plot, and lin-linear plot; and in each of those plots, we mark the position of Zmaas the Emin found by the fitting procedure, and a suboptimal value of Zmin chosen by hand.
The log-log plots, in I(a) highlight the (well-known but often finicky) linear trend on log-log plot. To aid the eye, in 1(a), we show the slopes on a log-log plot, as determined by the a fit for that value of mmin- The linear-linear plots, in 1(b), the usual so-called Scree plots, are not particularly informative. The log-linear plots, in I(c), shows that the optimal value of Tmin is near the peak of the distribution, and that the distribution is clearly not log-normal, having strong right-ward bias, with spreading out of larger eigenvalues as Amaz is approached.
In (d)) we show the quality of the fit, measured by the KS distance, as a function of Imin- The interpretation of these PL fits is that, above that value of Zmin and below the value of Zmazz the ESD fit to a line, with slope -a, in a log-log plot. In (dd), we see that the suboptimal value of x Z'án was chosen to be a local minimum in the the KS distance plot, and that the fit quality gradually degrades as Tmin increases. Looking at (c)), we see that there is a very slight "shelf" in the ESD probability mass; and that the suboptimal value of Eman corresponds fitting much smaller portion of the ESD
5

---

Under review as a conference paper at ICLR 2023
with PL fit. For high-quality models, like this one, smaller values of corresponds to better models, and decreasing a is well-correlated with increasing Amas-
3.3 COMPARING SCALE VERSUS SHAPE PARAMETERS
Here, we compare Shape versus Scale parameters, illustrating that they capture different infor- mation about models, as task, depth, and regularization / solver hyperparameters are varied. Alpha vs. LogSpectralNorm.
See Figure 2, which compares Alpha and LogSpectzalNoxn for mod- els from Table 4, segmented into sub- groups corresponding to models with
Alpha vs. LogSpectra Norm
40g Alpha cn
Aur Alrhe
the same depth. See also Table I for more detailed numerical results. For some model sub-groups, Alpha and LegSpect ra NNormm are strongly anti-correlated; while for other sub- groups, they are modestly to weakly anti-correlated, at best.
Avg LagSpectralNorm (109://W12)
Aug LogSpecralNonn (loS1o
(a) Task1 models.
(b) Task2 models.
For example, for Task1, in the 2xæ and 9ææ models, the two metrics are strongly anti-correlated metrics, with
Figure 2: Comparison of the Logspect and
Al Alpha metrics, for Task and Task2 models.
Ra 0.6 and a Kendall-r rank correlation metric of 7 0.75:² in the lxx models, they are mod- estly so, with R² 0.4 and т 0.4; and in the remaining model sub- groups, R² which we identify as weakly cor- related. Similarly, in Task2, we can identify model sub-group 9rz has having well anti-correlated av- erage metrics, with large R² and Kendall-r; model sub-groups 6az and 10xz exhibit intermediate behavior, as R² < 0.3 for each; and model sub-group 2ræ shows substantial correlation at all between A. Alpha and LogSpectralNorm. See Ap- pendix D for more details on two il- lustrative pairs of examples.
R
Kendall-r
Correlation
Taski- 0xx
0.162
0,29
Weak
Taski- 1xx
0,405
0.394
Modest
Taski- 2xx
0.803
0.788
Strong
Taskl- 5xx
0.124
0.117
Weak
Taski 6xx
0.124
0,263
Weak
0.2 and т S 0.3.
Taski 7xx
0.64
0.909
Strong
Taskl- AVG
0.38
0,46
Task2- 2xx
0.113
0.0327
None
Task2- 6xx
0.282
0.451
Modest
Task2- 9xx
0.754
0.600
Strong
Task2- 10xx
0.273
0.636
Modest
Task2- AVG
0.36
0.43
Table
the Alpha and LogSpectralNorm metrics, for Taskl and Task2 models. R² and Kendall-7 for Taskl and Task2, both aggregated and partitioned into model sub-groups.
Comparison
4
A SIMPSON'S S PARADOX: ARCHITECTURE VERSUS SOLVER KNOBS
4.1
BASIC PROPERTIES OF THE DATA
The obvious Basel ine for model quality is the training accuracy. If we have access only pre- trained models and no data (as in prior work (Martin et al., 2021) and as assumed with Alpha and LegSpect r 1Nrr)), then cannot check this baseline. Similarly, if the training error is exactly zero, then this is not a useful baseline. Otherwise, we can check against it, and we expect testing accuracy to improve training accuracy improves.
In a practical setting, for a properly trained model, one expects that test accuracy will increase with increasing training accuracy, it fixed.3 If the test accuracy does decrease, this indicates that the model may be somewhat
least not decrease, when the model capacity and dataset
"Tii is predicted by HT-SR Theory (Martin & Mahoney, 2021; Martin et al., 2021). but can be surprising from perspective of SLT (Jiang al., 2019; 2020abb).
'Likewise, a strong linear correlation, not just good rank correlation, is predicted for good models by HT-SR theory (Martin & Mahoney, 2021), and this has been observed previously for SOTA CV and NLP models (Martin al., 2021).
'hee the model capacity or data set size is changed, then test accuracy not track the training accuracy due the Double Descent phenomena (Belkin et al., 2019), and the fact that NNs can exhibit complex phase hehavior (Martin & Mahoney, 2017). While unlikely, we can totally discount this for the Task models.

---

Under review as a conference paper at ICLR 2023
overtrained and therefore will not generalize well. To examine this, we look at different model- groups which have a fixed depth (L). Figure 3 plots the relationship between training and testing accuracies for the Taskl and Task2 models, color-coded by model number (0xx, 1xx, ...).
See also Table for more detailed nu- merical results. By looking the both the trendlines and the distribution of points in Figure 3, we can identify such overtrained models here: almost all of the Task1 models (except per- haps the very highly accurate 5xx and 6xx models, and the moderately accu- rate Oxx and 1xx models); the Task2 2 9xx model; and to a lesser extent the Task2 2xx model.
Training vs. Test Accuracy
Training vs. Accurecy
Doc
C200
10cl
Trairing Accuracy
Trining Accuracy
(a) Taskl models.
(b) Task2 models.
For both Taskl and Task2, there is very large gap between the train- ing and testing accuracies. A posi- tive correlation in Figure 3 indicates that improving the training accuracy, even marginally, would lead to im- proved testing accuracy. Instead, we see that for Taskl, for most model sub-groups (1xx, 2xx, 6xx, and 7xx),
Figure 3: Relationship between training accuracy and testing accuracy for Taskl and Task2 models. One expects positive correlation or (if the training error near zero) at least not negative correlation. Observe, however, that in many cases they are strongly anti-correlated. See also Table 2.
training and testing accuracies are (very) anti-correlated. For other two (0xx and 5xx), the trendline is only weakly anti-correlated. For all models, however, the training accuracy approaches 1, the test accuracy plummets (except for a few outliers the 5xx and 6xx groups). In this sense, these Taskl models are over-trained; they clump into two sub-groups. those for this R² x O and those for which H² > 0.1. Similarly, for Task2, for model number 9xx, they are (very) anti-correlated, and otherwise they (slightly) anti-correlated for 2xx. The other two models show essentially no discernible trend. In all cases, improving training accuracy leads either to no noticeable improvement in test accuracy,
worse testing performance.
Overall, many of the Taskl mod- els, as well as model Task2 9xx, be- have qualitatively and quantitatively differently than the reasonably well- trained Task2 models. The Taskl models are generally of lower qual- ity. Some interpolate exactly to 0 test error while others do not-a signif- icant difference. We suspect some may be overtrained. Notably, only the Task2 models display the Simp- son's 's paradoxes we identify here. is those where we understand why the Al AlphaHat metric works.
R
RMSE
Kendall-r
Correlation
Taski Oxx
0.01
0.02
0.10
Weak
Taski Ixx
0.12
0.02
0.33
Modest
Task1 2xx
0.22
0.02
0.54
Strong
Task1 5xx
0.01
0.04
0.01
Weak
Task1 6xx
0.16
0.05
0.31
Modest
Task1 7xx
0.78
0.02
0.60
Strong
Taskl AVG
0.22
0.03
0.31
Task2 2xx
0.24
0.02
0.25
Modest
Task2 6xx
0.02
0.02
-0.05
Weak
Task2 9xx
0.83
0.01
0.73
Strong
Task2 10xx
0.02
0.02
0,02
Weak
Task2 AVG
0.28
0.02
0.24
Table 2: Quality metrics (for Rt, larger is better; for RMSE, smaller is better; and for Kendall-r rank correlation, larger magnitude is better) for the relationship between training error and testing error, as illustrated in Figure 3.
4.2 VISUALIZING THE SIMPSON'S PARADOX IN DNN MODELS
We now consider how the test accuracy varies against the Log and Alpha. respectively, overall and broken down for each model sub-group, for all models from Table 4. See Figure 4 and Figure 5 for summary. In our analysis, we first look each model sub-group (0xx, 1xx, 2xx, ...) individually, which corresponds to specific depth L, and we measure the regression and rank correlation metrics on the test accuracy as the hyperparameters vary.
7

---

Under review as a conference paper at ICLR 2023
Simpsons Plot for task]
Simpsons Plat for task?
From Figure 4, for each model sub-group, the CogSpectralNcrat increases with increasing model
Test Accuracy
Test Accurecy 200
quality. For Task1, the
LogSpect 1Norm increases
when all models are considered
300
together; but for Task2, the
0:00
SesSpectralNorss when all models are considered together, exhibiting clear Simpson's paradox.
decreases
DE
14
as 13 12 Avg LogSpertralNomm (logia WWE:
Avg LagSpoctralfform (log://[))
(a) Taskl models.
(b) Task2 models.
Figure 4: Test accuracy versus LogSpect ra 1 NNrmm, for Task1 1 and Task2, overall and segmented by model sub- group. Observe the Simpson's 's paradox for Task2 models.
From Figure 5, for each model sub- group, the Al Alpha decreases with in- creasing model quality. For Taskl, the Alpha decreases when all mod- els are considered together; but for Task2, the Alpha increases when all models considered together, exhibiting a clear Simpson's 's paradox.
Both metries exhibit clear Simpson's paradox for Task2.
Simpsons Plot for lask]
Simpsons Plot for task2
2ox
Based bounding theorems from SLT, one might expect that smaller val- would
Test Accuracy
300
Test Acturacy
of SogSpectralNor correspond to better models. Sim-
255
ilarly, based on HT-SR theory, one would expect that smaller values of Alpha would correspond to better models. With respect to these refer- ences, the LogSpectra ra 1Norm be- haves the opposite of what SLT would suggest, except when considering the aggregated data (all models) when a Simpson's 's paradox is present (i.e., in Task2). On the other hand, the A Alpha behaves precisely as what the HT-SR theory predicts, except when considering the aggregated data when Simpson's ' paradox is present. As shown below, however, by combining both into the AlphaHat metric, the Simpson's 's paradox in Alpha for the Task2 models can be alleviated.
331
Avg Alpha
Alpha
(a) Task1 models.
(b) Task2 models.
Figure 5: Test accuracy versus Al Alpha, for Taskl and Task2, overall and segmented by model sub-group. Observe the Simpson's 's paradox, in particular for Task2 models.
4.3 CHANGING ARCHITECTURES VERSUS CHANGING SOLVER HYPERPARAMETERS
Here, we provide more detailed analysis of the main results presented in Figure 4 and Fig- ure 5. See Figure 6 for histograms summarizing some of these results, in particular for Alpha and LogSpectra l Norm. See also Table 5 and Table 6 in Appendix E for detailed statistics, including R² and Kendall-r statistics, for metrics from Table 3.
From these, we see that Alpha (the mean PL exponent, averaged over all layers, which corresponds to a Shape parameter) is correlated with the test accuracy for each DNN, when changing just the hyperparameters, While it does not always exhibit the strongest correlation, it the most consistent. We also see that LogSpectro peppttallltee ranoooo rm (the mean log spectral norm, also averaged over all layers, which corresponds to a Scale parameter) is correlated with the test accuracy, when changing the number of layers, but it is anti-correlated when changing the hyperparumeters, Thus, it performs quite poorly when trying to identify more fine-scale structure.
For completeness, we have also included LogF www.eroooeeiio.oeo showing that it is often but not always correlated with test accuracy, and AnalityOfAlphaat (the KS distance), showing that correlated with test accuracy in about half the cases.
Finally, consider Figure 7, which shows the test accuracy versus AlphaHat, for Taskl and Task2, segmented by model sub-group, and compare this with Figure (for LogSpectraaNorm) and Figure (for Alpha). Recall that AlphaHat may be viewed either as a weighted Alpha (weighted by the layer Xmax) or weighted LogSpect Norm rm (weighted by the layer oz).
8

---

Under review as a conference paper at ICLR 2023
14
(a) Taskl, R
(b) Task2, R²
(c) Task1, Kendall-r
(d) Task2, Kendall-r
Figure 6: Comparison of predictions for test accuracies by A Alpha (a) and LogSpect ra No rm 10 [W]] [)). as measured using R² and Kendall-T correlation and rank correlation metrics, respec- tively, for Taskl and Task2, for each model sub-group, (See Table 5 for more details.)
As a weighted Alpha, the spectral norm weighting corrects for the fact that Alphe is scale- invariant, accounting for the variation in the scale of each weight matrix across different layers. As weighted LogSpect r Norrom, the a weighting corrects for the fact that the LogSpect aaspecttrlNoorn anti-correlated with the test accuracy when varying the regularization hyperparameters.
In either case, this combination "cor- rects for" the Simpson's 's paradoxes observed in Figures 4 and 5. This is the explanation for the previously- observed success of this metric at pre- dicting the quality of SOTA DNN models (Martin et al., 2021).
Stmpsons Rist for task) vi
mmoonn flet 2aok vl
035
0.2
0190
020
0:35
0:00
dox
Scr
035
5 CONCLUSION
030
0.75
0.00
A goal of the HT-SR theory is to develop model quality metrics, like Alpha and AlphaHat, that can be applied to so-called pre-trained DNN models, by only watching the weights, i.e, without the need for access to training/testing data; and, in doing so, to also provide layer-by-lyeer quality metrics. Previous work demonstrated the utility of the AlphaHat metric when applied to a wide range of large scale and production, pre-trained CV and NLP models (Martin et al., 2021).
(a) Taskl.
(b) Task2.
Figure 7: Test accuracy versus A phat for Task1 and Task2, segmented by model sub-group. (Similar results are seen for CogglphaShattenNNo.
Here, we seek to better understand why the AlphaHat model quality metric works so well. We also present, for the first time, evidence that the Al pha metric correlates well with changes in regularization hyperparameters, and, in fact, works better with better performing models.
To accomplish this, we evaluated the AlphaHat metric and its subcomponent metries, Alpha on a set of aublicly-available pre-trained models made available from recent machine learning contest aimed at understanding causes of good generalization. To our initial surprise, we identified a clear Simpson's 's paradox in the data. From our exploration of that, we discovered the complementary roles of Scale metrics versus Shape metrics in evaluating model quality. Overall, our analysis explains the success of the We Weight ght cherA phaHat metrie; combines Scale and Shape information (Martin et al., 2021), and also shed light on previously- observed peculiarities of norm-based metries (Jiang al., 2019). Our results also highlight the need to go beyond one-size- une-size-fits-all. fils- -ilps-llls causal metrics- to describe the performance SOTA NNs.
and LogSpectraaNooe
Based on our findings, we expect that LogSpect (and related Scale-based metrics) can capture coarse model trends due to changes in depth / number of layers whereas Alpha (and related Shape-based metries) can capture fine-scale model properties (e.g., changes in regularization and other hy typeraarameters including batch size, step scale, etc.) more generally, providing data-free diagnostics for DNNs on layer-by-layer basis. (See Appendix F for some additional discussion.)

---

Under review as a conference paper at ICLR 2023
REFERENCES
WeightWatcher, 2018. http:///////ppp...oooo//////////
Alstott, E. Bullmore, and D. Plenz. powerlaw: A python package for analysis of heavy-tailed distributions, PLoS ONE, 91)xe85777, 2014.
S. Arora, R. Ge, B. Neyshabur, and Y. Zhang. Stronger generalization bounds for deep via compression approach. Technical Report Preprint: arXiv:1802.05296, 2018.
Y. Bahri, J. Kadmon, J. Pennington, S. Schoenholz, J. Sohl-Dickstein, and S. Ganguli. Statistical mechanics of deep learning. Annual Review of Condensed Matter Physies, 11:501-528, 2020.
P. Bartlett, D. J. Foster, and M. Telgarsky. Spectrally Technical Report Preprint: arXiv: 106.089006 2017.
margin bounds for neural networks.
P. L. Bartlett. For valid generalization, the size of the weights is more important than the size of the network. In Anmual Advances in Neural Information Processing Systems 9: Proceedings of the 1996 Conference, pp. 134-140, 1997.
M. Beggs and N. Timme. Being critical of criticality in the brain. Frontiers in Physiology. 3(163), 2012.
M. Belkin, D. Hsu, Ma, and Mandal. Reconciling modern machine-learning practice and the classical bias variane trade-off. Proc. Natl. Acad. Sci. USA, 116:15849-1585. 2019.
J. Bickel, E. A. Hammel, and J. W. O'Connell. Sex bias in graduate admissions: Data from Berkeley. Science, 187(4175)3398- 404, 1975.
J. Bouchaud and M. Potters. Theory of Financial Risk and Derivative Pricing: From Staristical Physics to Risk Management. Cambridge University Press, 2003.
A. Clauset, C. R. Shalizi, and M. E. J. Newman. Power-law distributions in empirical data. SIAM Review, 51(4):661-705 2009.
G. K. Dziugaite, A. Drouin, B. Neal, N. Rajkumar, Caballero, L. Wang, Mitliagkas, and D. M. Roy, In search of robust measures of generalization. Technical Report Preprint: 11X1v20010.11944, 2020.
A. Engel and C. P. L. Van den Broeck. Statistical mechanics of learning. Cambridge University Press. New York, NY. USA, 2001.
Y. Jiang, B. Neyshabur, H. Mobahi, D. Krishnan, and S. Bengio. Fantastic generalization measures and where to find them. Technical Report Preprint: arXiv:1912.021 2019.
Y. Jiang, P. Foret, S. Yak, D. M. Roy, H. Mobahi, G. K. Dziugaite, S. Bengio, S. Gunasekar, I. Guyon, and Neyshabur. NeurIPS 2020 competition: Predicting generalization in deep learning (version 1.0). Technical Report Manuscript, June 28, 2020, 2020a.
Y. Jiang, P. Foret, S. Yak, D. M. Roy, H. Mobahi, G. Dziugaite, S. Bengio, S. Gunasekar, Guyon, and B. Neyshabur. NeurIPS 2020 competition: Predicting generalization in deep learning (version 1.1). Technical Report Preprint: December 16, 2020: arXiv:2012.07976v/ 2020b.
R. A. Kievit, W. E. Frankenhuis, J. Waldorp, and D. Borsboom. Simpson's 's paradox in psychological science: practical guide. Frontiers in Psychology, 4(513):1-14, 2013.
Z. Liao, T. Drummond, I. Reid, and G. Carneiro. Approximate Fisher information matrix to characterise the training of deep neural networks. Technical Report Preprint: arXiv:1810.06767. 2018.
N. Marshall, N. M. Timme, N. Bennett, M. Ripp, E. Lautzenhiser, and J. M. Beggs. Analysis of power laws, shape collapses, and neural complexity: New techniques and MATLAB support via the NCC toolbox. Frontiers in Physiology, 7(50)): 1-18, 2005.
10

---

Under review as a conference paper at ICLR 2023
C. H. Martin and M. W. Mahoney. Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior. Technical Report Preprint: arXiv:1710.09553, 2017.
C. H. Martin and M. W. Mahoney. Traditional and heavy-tailed self regularization in neural network models. In Proceedings of the 36th International Conference on Machine Learning, pp. 4284-4293, 2019.
C. Martin and M. Mahoney. Heavy-tailed Universality predicts trends test accuracies for very large pre-trained deep neural networks. In Proceedings of the 20th SIAM International Conference on Data Mining, 2020.
C. H. Martin and M. W. Mahoney. Implicit reffreegularzzation in deep neural networks: Evidence from random matrix theory and implications for learning. Journal of Machine Learning Research, 22(165):1-73, 2021.
C. H. Martin, T. S. Peng, and M. W. Mahoney. Predicting trends in the quality of state-of-the-ar neural networks without access training or testing data. Nature Communications, 12(4122): 1-13, 2021.
M. E. J. Newman. Power laws, Pareto distributions and Zipf's law. Contemporary Physics, 46: 323-351, 2005.
B. Neyshabur, R. Tomioka, and N. Srebro. Norm-based capacity control in neural networks. In Proceedings of the 28th Annual Conference on Learning Theory, pp. 1376-1401, 2015.
Pearl. Causality: Models, Reasoning and Inference. Cambridge University Press, 2009.
W. S. Robinson. Ecological correlations and the behavior of individuals. International Journal of Epidemiology, 38:337-341, 2009.
E. H. Simpson. The interpretation of interaction in contingency tables. Journal of the Royal Statistical Society. Series B (Methodological), 13(2):23-2411 1951.
D. Sornette. Probability distributions in complex systems. Technical Report Preprint: arXiv:0707.2199. 2007.
Thomas, F. Pedregosa, B. van Merrienboer, P.-A. Mangazol, Y. Bengio, and N. Le Roux. interplay between noise and curvature and its effect optimization and generalization. Technical Report Preprint: arXiv: 19Xv:19966.07774 2019.
L. Zdeborová and E. Krzakala. Statistical physics inference: thresholds and algorithms. Advances in Physics, 65(5):453-555 2016.

---

Under review as a conference paper at ICLR 2023
A METRICS CONSIDERED IN OUR ANALYSIS
As mentioned in Section we considered a range metrics our analysis. See Table 3 for summary of the best performing. This includes both average PL metrics, from HT-SR Theory, and average log- log-norm metrics, from SLT, as well as metrics that combine the two approaches. (Other metrics performed worse, exhibited similar qualitative trends, or required access to training and testing data.)
Need initial
Scale or Shape?
Need data?
Need GPUs?
Complexity Metric
Average
Ref.
weights?
rogSpect Norm
(log10
W
(Jiang al. (2019))
Scale
No
No
No
logFroben usNorm
(logıg
(Jiang ct al. (2019))
Scale
No
No
Alpha
{a)
(this paper)
Shape
No
No
No
Qua phaiit
DKS
(this paper)
Shape
No
No
No
Alphafat Logk_phaShattenNNomm
(Martin al. (2021))
Both
No
No
(log10
W
(Martin et al. (2021))
Both
No
No
No
Table 3: Overview of model quality metrics. Based on initial analysis of Contest models,
propose and evaluate Alpha and Quali wawllittOfApppaFii. The metrics in this table do not need access to training/testing data; they do need information such the initial weight distribution; and they do not require training/retraining (and thus access to GPUs).
Average Power-Law (PL) metrics. Given a (pre-)trained DNN with layer weight matrices W, PL metrics are computed by fitting the ESD of the correlation matrix X W W of each layer to PL, and then averaging over all layers." 4 In more detail, the tail of each layer ESD is fit to a PL of the form:
Proil (A)
Emin
Tmaz-
The fitting procedure selects the optimal PL exponent a, and the adjustable parameter Imin- The PL exponent a characterizes the tail of the ESD; it measures what may be interpreted as the "shape" of most important part of the spectrum.
Alpha: (a) = Σ, og. This is simple average of fitted over all layers L. From the perspective
of statistical mechanics, Al Alpha quantifies amount of correlation in the layer weight matrices. From the perspective of statistics, Alpha may be viewed shape parameter.
Qua i tyoffl PhaFit: (Dxs). For the given set of parameters (a, Tmin; Xmar), the quality (KS) distance (Dxs) between the
of the PI. fit can be measured by the Kolmogorov-Smimno empirical and theoretical distributions. The smaller Dks is, the better the fit.
We describe the PL fitting procedure in more detail (in Section 3.1), and we give examples of both high and low quality fits (in Appendix C). To our knowledge, we are the first to use Alpha and QualityofAlphal to gauge model complexity."
Average Log-Norm metrics. Log-Norm metrics are related to product-norm measures of the model complexity C. Given a (pre- (trined DNN with layers, and layer weight matrices W,, we define the C as the product over norm of the layer weight matrices
C = || x |2 x ... |
(3)
*For Conv2D(k; N, M) layers, extract N & matrices of shape N x M: typically. A 1 or 3. 'Prior work (Martin Mahoney, 2021) has argued that NNs resemble the strongly correlated systems, e.g.. in electronic structure theory and statistical mechanics, which was the origin of early work in heavy-tailed random matrix theory (Bouchaud & Potters, 2003).
'Prior work (Martin et al., 2021) used a weighted version of this metric (AlphaHat). There, PL exponents were computed, but they were not evaluated as a measure of test accuracy, and they were not shown to correlate with variations in solver hyperparameters (such as batch size, dropout, weight decay, etc.), as we do for the first time here.
12

---

Under review as a conference paper at ICLR 2023
where | denotes some arbitrary matrix norm for layer 1.7 If we take the logarithm of both side, we can express this complexity as an average over all layers (L), in log-units units
1 Σ logio |ill-
1og ~ (logio [W])
(4)
Metrics of this form provide measure of the "scale" (in log units) of model.
LogSpect
(logıo | This is the (average) of the log of the layer Spectral norms.
The layer Spectral Norm is just maximum eigenvalue Amar of the correlation matrix X.
LogFroben usNorm: 10 ( This the average of the log the Frobenius norm for cach layer weight matrix. It is included for completeness.
Importantly, these metrics take a layer average, and not a sum, since otherwise these metrics will trivially depend on the depth L of the network.
Combining PL and Log-Norm metrics. Previous work considered metrics that combine "scale" and "shape" ideas (Martin & Mahoney, 2020; Martin et al., 2021), two of which we consider here.
AlphaHat: á= = Σ, on logio Amaz, This has been used previously as a complexity measure for a large number of pre-trained NNs of varying depth and hyperparameters (Martin & Ma- honcy, 2020; Martin et al., 2021). AlphaHat may be viewed in one of two complementary ways: either weighted Alpha (weighted by the layer A/maz); or, equivalently, as weighted ragSpectraaNoos
(weighted by the layer at). As a weighted Alpha, the spectral norm weight- ing corrects for the fact that Al pha is scale-invariant accounting for the variation in the scale of each weight matrix across different layers. As weighted LogSpect ralNorm, the a weighting corrects for the fact that the LogSpect NNoom anti-correlated with the test accuracy when varying the regularization hyperparameters.
LogAlphaShat tenNorm: (log10 | [
[ This has been used previ- ously (Martin & Mahoney, 2020), and it is the average of the Log of the standard Shatten norm, defined with the value of 2a from the PI. fit of the ESD of each layer weight matrix W. The AlphaHat metric approximates the Loga www.ppattheeetrrrrrerr under certain conditions (Martin Mahoney, 2020).
(log10
Table 3 provides a summary of the metrics we considered. There many other metrics that examined but that we do not describe here: the path norm and Fisher-Rao metrics (Jiang et al., 2020a;b) are more expensive, perform worse, and/or don't ton't add new insight; the Jacobian Norm is also too expensive to compute; and other metrics described in the Contest (Jiang et al., 2020a;b) either were uninteresting (c.g., the sum not average of layer norms, which is proxy for depth) or performed very poorly."
B MODELS CONSIDERED IN OUR ANALYSIS
See Table 4 for a summary of the models we considered in our analysis. These are described in more detail in Section 2. See also Jiang et al. (2020a;b) for more details.
C MORE ON DETERMINING SHAPE PARAMETERS (FITTING ESDs TO PLs), FROM SECTION 3.2
Fitting data to PLs is very finicky (Newman, 2005; Someetee 2007; Clauset et al., 2009; Beggs & Timme, 2012; Alstott et al., 2014; Marshall et al., 2005). We have found it best to proceed with a combination of visual inspection and analysis with the WeightMatcher tool (wei, 2018).
We drop the layer subscript when it is clear from the context. 'Prior work has shown that using corm-based metrics in log-scale tends be superior working with them non-log scale (Martin et al., 2021). Even when taking averages, however, norm-hased metrics, unlike Al Alpha are not scale invariant.
For the Conv2D layers, extracting the layer weight matrices some arbitrary choices; see Martin & Mahoney (2020) for discussion of how the we ghtwatcher tool does this
13

---

Under review as a conference paper at ICLR 2023
Scries
#
L
Batch Size
Dropout
Weight Decay
Conv Width
Dense
(k)
Taskl
Oxx
4
8,32, 512
0.0, 0.5
0.0, 0.001
256,
1
"task1 v4" (VGG-like)
1xx
5
8,32, 512
0.0,
0.001
512
1
2xx
8.32, 512
0.0, 0.5
0.0
256. 512
1
5xx
8
8.32. 512
0.0, 0.5
0.001
256, 512
1
3
6xx
8
8. 512
0,0, 0.5
0.0
256. 512
2
3
7xx
9
512
0.0, 0,5
0.0
256, 512
2
3
Task2 "task2 v1"
2xx
13
512. 1024
0.0, 0.25, 0.5
0,0, 0.001
512
6xX
7
32, 512, 1024
0.0, 0.25, 0.5
0.0, 0.001
512
(Network-
9xx
10
1024
0.0, 0.25, 0.5
0.0, 0.001
512
in-Network)
10xx
10
512
0.0, 0.25, 0.5
0.0, 0.001
512
Table 4: Overview of models we considered in each task and sub-group, including variations in depth (L., regularization hyperparameters (Batch Size, amount of Dropout, and Weight Decay), and architectural changes (Width of selected Convolutional Layers, number of selected Dense layers, and kernel-size (k 1, for selected Convolutional layers). See Jiang et al. (2020a;b) for complete details.
Visual inspection of ESDs. The following advice, taken directly from Somettte (Sornette, 2007), is particularly helpful for visual inspection ESDs: "we recommend preliminary visual exploration by plotting the survival and density distributions in (i) finear-linears coordinates, (ii) log- ling-linerrraa coordinates (linear abscissa and logarithmic ordinate) and (iii) log-log coordinates (logarithmic abscissa and logarithmic ordinate). The visual comparison between these three plots provides fast and intuitive view of the nature of the data.
A power law distribution will appear as a convex curve in the Linear-lineal and log-linear plots and as straight line in the log-log plot.
A Gaussian distribution will appear as a bell-shaped curve in the linear-linear plot, as an inverted parabola in the log-linear plot and as strongly concave sharply falling curve in the log-log plot.
An exponential distribution will appear as a convex curve in the Ainear-lineal plot, straight line the log-linear plot and as a concave curve in the log-logp lot.
Having in mind the shape of these three reference distributions in these three representations provides fast and useful reference points to classify the unknown distribution under study." It is helpful to examine ESDs such as those in Figure (in Section 3.2) and Figure (here) light of these comments.
Of course, such a visual inspection is just a first step to more detailed analysis, since by itself visual analysis of HT data can be misleading. Sornette (Sornette, 2007) goes on to say: "While we recommend first visual inspection, it is only first indication, not proof. It is a necessary step to convince oneself (and the reviewers and journal editors) but certainly not sufficient condition. is standard rule of thumb that power law scaling is thought to be meaningful if it holds over at least two to three decades on both axes and bracketed by deviations on both sides whose origins be understood (for instance, due to insufficient sampling and/or finite-size effects)." It is for these reasons that understanding the behavior of the ESDs near Tmin and Imaa (of Eqn. (1)) is so important.
Fitting to non-ideal ESDs. In Figure 1, we illustrated how PL fits of ESDs perform nearly "ideal" example. Here, we discuss how it performs on less-than than-ihaaaa examples (that occurred in the Contest data). See Figure 8.
In Figure 8, we see several examples of layers that are less well-fit by PL. these cases, we see that the linear-linear plots are non-informative; the log-linear plots show that the distributions have a strong left-ward bias, indicating a tail of increasingly small eigenvalues; and there is not broad range of large eigenvalues on the right of the distribution. This relative paucity of large eigenvalues is seen on the log-lineat plots by a (more or less aggressive) truncation in probability mass for larger cigenvalues, and the log-log plots by steeper downward slope the right of the ESD. (In these cases, compare with Figure I.) The first row (Taskl, model 152, layer I) illustrates meompletely-develoon right tail (alternatively "bulk-plus-spikes" model may be
14

---

Under review as a conference paper at ICLR 2023
14
(a) ESD (log-log plot)
(b) ESD (lin-lin plot)
(c) ESD (log-lin plot)
(d) PL fit quality vs Xmin
16-1
12
(c) ESD (log-log plot)
(f) ESD (lin-lin plot)
(g) ESD (log-lin plot)
(h) PL fit quality vs Zmin
Figure 8: Illustration of the role of the ESD shape in determining the PL parameter a in the PL fit. Rows correspond to different layers in different models. Columns correspond viewing the same ESD in different ways on log-log plot (first column), a linear-linear plot (second column). and a log-linear plot (third)-an and the KS Distance of the PL fit function of I'án (final column). Row 1: Task1, model 152, layer 1. Row 2: Task2, model 1006, layer 10.
more appropriate fit than a PL fit), meaning that the spectral norm is somewhat smaller and that the fitting procedure has difficulty choosing Tmin near the peak of the distribution. The second row (Task2, model 1006, layer 10) illustrates an aggressively-shore shortennd (effectively truncated, i.e., not even spikes) right tail, which leads to much smaller spectral norm (Zmax) and thus much larger Tmin (since there such small range over which a linear appropriate). as well as a broad range of (large) Umin values over which low-quality fits are obtained. In each of these of these cases, the KS distance plots have less well-defined minimum as a function of æmin-
Additional discussion. In simple cases, scale and shape parameters do capture similar information. (MP) distribution or
For models whose ESDs are very well-approximated by a Marchenko-Pasee MP bulk-plus-spike distribution (R [RADOOM-LIKE and BULK SULKESS phases from Martin & Mahoney (2021)), visual inspection of ESD plots often yields insight, and there is a strong correspondence between norm-bor norm-based scale metrics and random Ma-based shape metrics. Similarly, for models whose ESDs are very well-approximated by PI. distributions (HEAVY-TAILED phase from Martin & Mahoney (2021), where the Tmar Xmaz truncation is not significant (Martin & Mahoney, 2021) to the fit), there is a strong correspondence between norm-based scale metries and PL-based shape metrics, in the sense that smaller a values correspond closely to larger Amaz values. For realistic models, however, and for the models from Table 4, these two metrics can be very different and can reveal very different information.
The broad range of behavior seen in the Contest data arises since ESDs look like ones in the rows of Figure 8 (rather than the more "ideal" case shown in Figure 1) where the linear fit to the ESD on log-scale is not very good. For models where Alpha against LogSpect lNrrm behave more similarly, the ESDs look more like Figure 1. For the Contest models shown in Figure 8 (and others), for simple mnonttruncated, long tail) PL distribution, smaller exponents a correspond to longer tail, which corresponds to a larger Amaz- However, since the tail of the ESDs are typically best fit by PL often with exponents a much larger than expected, since many of the models from Table 4 are of lower quality-thie the situation is not so simple. This simple connection only holds for few model sub-gr grobggr Generally speaking, these scale and shape metries (LogSpect ra No rm and A pha) characterize the model ESDs differently and capture different properties of the models.
D ILLUSTRATIVE EXAMPLES: COMPARING SCALE VERSUS SHAPE PARAMETERS
We saw in Figure 2 a comparison of the Alpha and LogSpect ralNorm metrics, for Taskl and Ta Task2 models. To get more detailed insight, Figure 9 plots A Aha versus ra for
15

---

Under review as a conference paper at ICLR 2023
Alpha vs. No
Alpha vs. LegSpeciralNem
Alpha LogSpectraNoom
Alpha vs. LogSpectraliom
AND Nipre 60
Avg Appne Grf
Awo Po
Aug
tepspectelhomm (loguo
tesspactelhom
(a) Taskl, Zrx
Task1, 5zx
(c) Task2,
(d) Task2, 10rr
Figure 9: Comparison of Al ilpha and LogSpect ra No for selected model sub-groups from both tasks. Lines (simply to guide the eye) shows a linear regression on the data.
illustrative pairs of examples, from each of Taskl and Task2. Consider, as baseline example, Taskl, model sub-group 2rz. in Figure (a). Here, the two metries are strongly anti-correlated, with linear correlation metric R 0.803 and Kendall-r rank correlation metric 7 0.788. In contrast, for Task1, 1, model sub-group 5 shown in 9(b), the two metrics are (at best) only weakly correlated, with R² 0.124 and r 0.177. This is typical; some model sub-groups exhibit large rank and/or linear correlations, others virtually none at all. For Task2, model sub sub-group 2zæ, versus Task2, model sub-group 10rz, shown in 9()) and 9(d), we see an example where the two plots look similar visually, and both have small(ish) linear correlation R2 but they have very different Kendall-F rank correlation metrics.
E ADDITIONAL DETAILS ON CHANGING ARCHITECTURES VERSUS CHANGING SOLVER HYPERPARAMETERS
LogSpectra
LogF robeuuuoen
Alpha
R²
Kendall-r
R
Kendall-7
R²
Kendall-r
Task1- Oxx
0.34
0.41
0.00
-0.03
0.55
-0.55
Taskl- 1xx
0.68
0.73
0.29
-0.30
0.88
-0.48
Taskl- 2xx
0.59
0.73
0.42
0.55
0.69
-0.52
Taskl-
0.58
0.60
0.19
-0.25
0.09
-0.10
Task 6xx
0.22
0.59
0.05
-0.10
0.67
-0.32
Taskl- 7xx
0.58
0.61
0.38
0.58
0.45
-0.51
Taskl- AVG
0.50
0.61
0.22
0.07
0.55
-0.41
T-sk2- 2xx
0.05
0.01
0.67
-0.84
0.69
-0.74
Task2- 10xx
0.32
0.60
0.60
-0.53
0.62
-0.67
Task2- 6xx
0.21
0.36
0.01
-0.05
0.36
-0.25
Task2- 9xx
0.17
0.47
0.65
-0.87
0.64
-0.87
Task2- AVG
0.19
0.36
0.48
-0.57
0.58
-0.63
Table 5: Model quality for different metries (all those mentioned in Table 3), for Taskl and Task2, both overall and by model sub-group. (This table part of 2; see also Table 6.) Bar plots for Al Alpha and LogSpectra Norm data from here shown in Figure 6.
See Table and Table for additional details on changing architectures versus changing solver hyperparameters, from Section 4.3.
F ADDITIONAL DISCUSSION
We conclude with a few more general thoughts on our results.
As is well known, trying to extract causality from correlation is difficult- difficult-precisely since there may be Simpson's like paradoxes present in the data, depending on how the data are partitioned. When confronted with Simpson's paradox, one is tempted to ask whether the marginal associations
16

---

Under review as a conference paper at ICLR 2023
AlphaHat
Loga.phaShattenNonn
Qualityo-Alphar
R²
Kendall-r
R²
Kendall-r
R²
Kendall-r
Taskl- Oxx
0.00
-0.06
0.00
-0.02
0.02
-0.10
Taskl- 1xx
0.01
0.00
0.00
-0.03
0.00
0.15
Taskl- 2xx
0.49
0.70
0.48
0.61
0.67
-0.82
Taskl- 5xx
0.01
0.10
0.00
0.08
0.04
0.18
Taskl- 6XX
0.00
-0.04
0.01
-0.03
0.10
-0.13
Taskl- 7xx
-0.06
0.07
-0.03
0.60
-0.58
Taskl- AVG
0.09
0.11
0.09
0.10
0.24
-0.22
Task2- 2xx
0.77
-0.75
0.62
-0.73
0.89
-0.83
T'ask2- 10xx
0.41
-0.38
0.46
-0.42
0.70
-0.67
Task2- 6xx
0.12
-0.14
0.12
-0.08
0.35
-0.25
Task2- 9xx
0.59
-0.87
0.65
-0.87
0.91
-0.87
Task2- AVG
0.47
-0.53
0.46
-0.53
0.71
-0.66
Table Model quality for different metries (all those mentioned in Table 3), for Taskl and Task2, both overall and by model sub-group. (This table is part 2 of 2; see also Table 5.) Bar plots for Alpha and LogSpectralNer data from here are shown in Figure 6.
partial associations are correct. Often, the answer is that both are correct, depending on what is of interest. In particular, simple statistical analysis does not provide any guidance as causal rela- tionships and whether the marginal association or the partial association is the spurious relationship. For that reason, rather than trying to extract causality, we took a different approach: we looked for Simpson's s paradox; and when we found it, we tried interpret it in terms of scale versus shape metrics from SLT and HT-SR theory.
One might wonder "why" LogSpectralNe restricted to changes the model depth and/or solver hyperparameters, respectively. Establishing such causal explanation, of course, requires going beyond the data at hand and requires some sort of counterfactual analysis. This is beyond the scope of this paper. A plausible hypothesis, however, the following. Since upper bounds from SLT suggest that models with smaller norms have less capacity, these norms are used (either explicitly as regularizer, or implicitly by adjusting large matrix alements/colummss/owww during the training process, particular when one varies coarse model parameters such as depth. On the other hand, coming from HT-SR theory, Alpha is not used explicitly implicitly during the training process. Instead, the training process extracts correlations over many size scales from the data, and it is these correlations that are captured by smaller A Alpha values, consistent with HT-SR theory and practice (Martin & Mahoney, 2021; 2019). This hypothesis consistent with "why" fitted PL metrics from HT-SR theory-in in particular the fitted AlphaHat metric-perform so well, both for the models considered this contest, as well as for much wider range of availale SOTA DNN Models (Martin al., 2021). Testing this hypothesis is an important question raised by our results.
and Alpha perform as well as they do, when
17


Reviews:
[{"review_id": "k_N0uD2JkQ", "reviewer": "Reviewer_E8e4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an analysis of why a particular model quality metric named AlphaHat works well. The AlphaHat metric is used to predict model performance without accessing the training and testing datasets. It only looks at the model weights and has two subcomponents, Alpha and LogSpectralNorm. The authors identify that there is a clear Simpson's paradox in the data and they found that Scale and Shape play two complementary roles as metrics for evaluating the model quality.\n", "summary_of_the_review": "Overall, it is a solid paper with extensive experiments for an important research question. The presentation can be improved to make the paper more accessible to people who are not familiar with AlphaHat and the Contest. Using more figures with examples can make it better. The novelty is good and I think it presents useful insights for the community to continue this line of research. \n", "strength_and_weaknesses": "\nStrengths\n\nThe analysis is very extensive and provides an insightful and convincing explanation of why AlphaHat is powerful for predicting model quality. It has both extensive theoretical analysis and empirical results to support the hypothesis. The discussion on the Shape vs Scale fills the gap in the current literature. I particularly like the discussion on why the generalization across a broad range of CV and NLP models.\n\nWeakness\n\nI think the current presentation assumes that readers have enough background knowledge. Also, it will be better if the authors also introduce the motivation with real-world applications for illustration. \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "g8U8r1YXtLm", "reviewer": "Reviewer_9hRU", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Disclaimer: This is a rather unusual and unusually written (meta-)paper that falls outside my core area of expertise. I still tried to review the paper in good faith.\n\nThe work combines two lines of work: L1. first line aims to analyse generalisation capacity of neural networks (NNs) -- pretrained models -- directly without having access to any training or test data using and extending a particular metric from the recently-developed\nHeavy-Tailed Self-Regularization (HT-SR) theory; L2. second line of work offered a contest with a series of pretrained Computer Vision models for a similar purpose (analysing and predicting generalisability). The idea is to run the basic framework from L1 on the available models from the L2 Contest and improve some mechanisms (e.g., the AlphaHat metric) typically used in L1.\n\nThe main contribution of this work is then the extension of the HT-SR theory which is at heart  of L1, based on the findings and analyses on the Contest (L2). The authors, among other things, show the disadvantage of the currently used single metric and decompose it into two metrics, and empirically validated the existence of a specific phenomenon (i.e., Simpson's paradox) in the Contest data. ", "summary_of_the_review": "While my review should be taken with a few grains of salt, my impression is that this work is too tied to a very narrow idea, and will result in very limited impact; the presentation should be reorganised, and the paper should also provide suggestions on how to connect its findings to more application-based ML (which it doesn't do at the moment): how can it inform our model selection in CV or NLP research?", "strength_and_weaknesses": "Strengths:\n- An atypical piece of (meta-)research that aims at an ambitious goal: analysing and comparing neural models without having access to training or test data, offering new insights on a particular phenomenon: Simpson's paradox.\n\nWeaknesses:\n- The whole work is basically an incremental piece of research heavily focused on one specific theory built in the work of Martin & Mahoney, and I am unsure how much impact this type of research will have on a wider ML audience. I am not convinced that the HT-SR theory is the right way to go in the first place when it comes to analysing and comparing different models. And it seems that the main goal of this particular work aims to improve one very specific metric (AlphaHat). The whole work seems a bit like a technical addendum to the source AlphaHat paper, as also stated by the authors themselves: \"Most importantly, this work helps clarify why the AlphaHat metric performs so well across so many models.\"\n- The work is also slightly detached from 'real-world' experiments: what consequences this type of research will have on ML models in practice and in production? What are the main take-home messages for researchers and practitioners working outside this niche area and not interested in in-vitro model analyses directly?\n- The whole Contest (and the subsequent analyses) are based on 'simple/smaller-arhictecture) NNs in computer vision - the authors 'over-claim' that similar findings will hold for NLP models as well, but this is never empirically verified - they make conclusions 'by proxy', and I am unsure how much the findings hold across NNs with different architectures (e.g., CNN-based versus Transformer-based versus LSTM-based) and with different parameter budgets and depths. This again brings me to the question of relevance of this research.\n- The presentation requires much more work - it is really difficult for someone who is not strictly in-domain (and not knowledgeable in the HT-SR) to follow the paper. First, the actual theory has to be defined with all the important preliminaries, the Simpson's paradox also needs a clear definition, and the Contest should also be described in more detail. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dqMk_dXs1n", "reviewer": "Reviewer_WuLE", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a large number of experiments relating the AlphaHat measure to generalization performance of DNNs.", "summary_of_the_review": "The goals of this work are important, and I think there are some interesting ideas here.\n\nThe writing is unfortunately very very hard to follow though. There are few technical details, and a great deal of opinion and prose. \n\nAs one example, take the end of page 2, the phrase \"In investigating, why AlphaHat works...\" The AlphaHat metric is defined just before this, but absolutely no context is given for what it would mean for the metric \"to work\". This kind of vagueness leads the reader guessing at many crucial points in the paper.\n\nThe paper goes on \"Being aware of challenges with designing good contests and with extracting causality from correlation (Pearl, 2009), we did not use the causal metric provided by the Contest. Instead, we adopted a different approach: we identified Simpson’s paradoxes within the Contest data; and we used this to understand better the AlphaHat metric from HT-SR theory\"\n\nI found this impossible to follow. What was \"the causal metric provided by the Contest\"? It would be simple enough to describe this, it seems crucial to the general argument. What does it mean to identify Simpson's paradoxes in this setting? It's very hard for the reader to understand what is going on.\n\nOverall I think there are probably some interesting ideas here, but I would strongly encourage the authors to better explicate their work, and the context for their work.", "strength_and_weaknesses": "Strengths: understanding generalization is of great importance, the authors have some novel ideas here.\n\nWeaknesses: the arguments made in the paper are extremely hard to follow. A serious round of editing and reorganization would greatly improve the readability of the paper.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Y4ZM_jgqsA", "reviewer": "Reviewer_tNyg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper explains the success of the alphahat metric [1] in predicting model generalisation: the alphahat metric consists of two components, (i) alpha, the power law exponent in the empirical spectral density, and (ii) the log spectral norm. The paper shows that these components result in a Simpson's paradox in the analysis of their prediction of generalizability: the log spectral norm is scale based, and does well in aggregated data, while alpha is shape based, and does well in sub partitions of the data, but less well in aggregation. The authors argue that alphahat works well by combining these two components into a single metric. \n\n[1] Martin, Charles H., and Michael W. Mahoney. \"Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning.\" J. Mach. Learn. Res. 22.165 (2021): 1-73.", "summary_of_the_review": "I propose to reject the paper as \n- the contribution as an empirical analysis of results published in 2021 does not qualify it for ICLR\n- the paper has been referenced in 2021 by a publication by the same authors.", "strength_and_weaknesses": "Strengths:\n- The paper analyses the success of the alphahat metric, and showed that its two components separately capture scale and shape.\n\nWeaknesses:\n- The paper simply analyses the failures of alpha and the log spectral norm. There is not really any theoretical justification in this paper on why combining the two components in alphahat should result in its good performance. \n- this submission has been referenced in [1] published in 2021. I think it could serve as an appendix to the journal paper, but the contribution is insufficient for a full paper at ICLR.\n\n\n[1] Martin, Charles H., and Michael W. Mahoney. \"Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning.\" J. Mach. Learn. Res. 22.165 (2021): 1-73.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject