Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents an analysis of why a particular model quality metric named AlphaHat works well. The AlphaHat metric is used to predict model performance without accessing the training and testing datasets. It only looks at the model weights and has two subcomponents, Alpha and LogSpectralNorm. The authors identify that there is a clear Simpson's paradox in the data and they found that Scale and Shape play two complementary roles as metrics for evaluating the model quality.

strength_and_weaknesses: 
Strengths

The analysis is very extensive and provides an insightful and convincing explanation of why AlphaHat is powerful for predicting model quality. It has both extensive theoretical analysis and empirical results to support the hypothesis. The discussion on the Shape vs Scale fills the gap in the current literature. I particularly like the discussion on why the generalization across a broad range of CV and NLP models.

Weakness

I think the current presentation assumes that readers have enough background knowledge. Also, it will be better if the authors also introduce the motivation with real-world applications for illustration. 


clarity,_quality,_novelty_and_reproducibility: Typo: "a *casual* measure"

I suggest the authors talk more about why the dataless assumption is used for this paper. 
summary_of_the_review: Overall, it is a solid paper with extensive experiments for an important research question. The presentation can be improved to make the paper more accessible to people who are not familiar with AlphaHat and the Contest. Using more figures with examples can make it better. The novelty is good and I think it presents useful insights for the community to continue this line of research.
review_title: null

final_score: rating: 1: strong reject
review_content: summary_of_the_paper: The paper describes a large number of experiments relating the AlphaHat measure to generalization performance of DNNs.
strength_and_weaknesses: Strengths: understanding generalization is of great importance, the authors have some novel ideas here.

Weaknesses: the arguments made in the paper are extremely hard to follow. A serious round of editing and reorganization would greatly improve the readability of the paper.
clarity,_quality,_novelty_and_reproducibility: Clarity: very hard to follow unfortunately.
summary_of_the_review: The goals of this work are important, and I think there are some interesting ideas here.

The writing is unfortunately very very hard to follow though. There are few technical details, and a great deal of opinion and prose. 

As one example, take the end of page 2, the phrase "In investigating, why AlphaHat works..." The AlphaHat metric is defined just before this, but absolutely no context is given for what it would mean for the metric "to work". This kind of vagueness leads the reader guessing at many crucial points in the paper.

The paper goes on "Being aware of challenges with designing good contests and with extracting causality from correlation (Pearl, 2009), we did not use the causal metric provided by the Contest. Instead, we adopted a different approach: we identified Simpson’s paradoxes within the Contest data; and we used this to understand better the AlphaHat metric from HT-SR theory"

I found this impossible to follow. What was "the causal metric provided by the Contest"? It would be simple enough to describe this, it seems crucial to the general argument. What does it mean to identify Simpson's paradoxes in this setting? It's very hard for the reader to understand what is going on.

Overall I think there are probably some interesting ideas here, but I would strongly encourage the authors to better explicate their work, and the context for their work.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: Disclaimer: This is a rather unusual and unusually written (meta-)paper that falls outside my core area of expertise. I still tried to review the paper in good faith.

The work combines two lines of work: L1. first line aims to analyse generalisation capacity of neural networks (NNs) -- pretrained models -- directly without having access to any training or test data using and extending a particular metric from the recently-developed
Heavy-Tailed Self-Regularization (HT-SR) theory; L2. second line of work offered a contest with a series of pretrained Computer Vision models for a similar purpose (analysing and predicting generalisability). The idea is to run the basic framework from L1 on the available models from the L2 Contest and improve some mechanisms (e.g., the AlphaHat metric) typically used in L1.

The main contribution of this work is then the extension of the HT-SR theory which is at heart  of L1, based on the findings and analyses on the Contest (L2). The authors, among other things, show the disadvantage of the currently used single metric and decompose it into two metrics, and empirically validated the existence of a specific phenomenon (i.e., Simpson's paradox) in the Contest data. 
strength_and_weaknesses: Strengths:
- An atypical piece of (meta-)research that aims at an ambitious goal: analysing and comparing neural models without having access to training or test data, offering new insights on a particular phenomenon: Simpson's paradox.

Weaknesses:
- The whole work is basically an incremental piece of research heavily focused on one specific theory built in the work of Martin & Mahoney, and I am unsure how much impact this type of research will have on a wider ML audience. I am not convinced that the HT-SR theory is the right way to go in the first place when it comes to analysing and comparing different models. And it seems that the main goal of this particular work aims to improve one very specific metric (AlphaHat). The whole work seems a bit like a technical addendum to the source AlphaHat paper, as also stated by the authors themselves: "Most importantly, this work helps clarify why the AlphaHat metric performs so well across so many models."
- The work is also slightly detached from 'real-world' experiments: what consequences this type of research will have on ML models in practice and in production? What are the main take-home messages for researchers and practitioners working outside this niche area and not interested in in-vitro model analyses directly?
- The whole Contest (and the subsequent analyses) are based on 'simple/smaller-arhictecture) NNs in computer vision - the authors 'over-claim' that similar findings will hold for NLP models as well, but this is never empirically verified - they make conclusions 'by proxy', and I am unsure how much the findings hold across NNs with different architectures (e.g., CNN-based versus Transformer-based versus LSTM-based) and with different parameter budgets and depths. This again brings me to the question of relevance of this research.
- The presentation requires much more work - it is really difficult for someone who is not strictly in-domain (and not knowledgeable in the HT-SR) to follow the paper. First, the actual theory has to be defined with all the important preliminaries, the Simpson's paradox also needs a clear definition, and the Contest should also be described in more detail. 
clarity,_quality,_novelty_and_reproducibility: See under Weaknesses:
- The work is largely tied to particular ideas from a particular theory, and it feels like an addendum to a well-known existing work from specific researchers, so novelty is quite limited, and quite narrow.
- The paper is quite difficult to follow for non-experts, and it remains unclear what impact it will have on wider research beyond the HT-SR theory. It is also unclear if the HT-SR theory is really the right theory - I am not convinced by it given how the paper was motivated. The presentation should be improved.
summary_of_the_review: While my review should be taken with a few grains of salt, my impression is that this work is too tied to a very narrow idea, and will result in very limited impact; the presentation should be reorganised, and the paper should also provide suggestions on how to connect its findings to more application-based ML (which it doesn't do at the moment): how can it inform our model selection in CV or NLP research?
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper explains the success of the alphahat metric [1] in predicting model generalisation: the alphahat metric consists of two components, (i) alpha, the power law exponent in the empirical spectral density, and (ii) the log spectral norm. The paper shows that these components result in a Simpson's paradox in the analysis of their prediction of generalizability: the log spectral norm is scale based, and does well in aggregated data, while alpha is shape based, and does well in sub partitions of the data, but less well in aggregation. The authors argue that alphahat works well by combining these two components into a single metric. 

[1] Martin, Charles H., and Michael W. Mahoney. "Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning." J. Mach. Learn. Res. 22.165 (2021): 1-73.
strength_and_weaknesses: Strengths:
- The paper analyses the success of the alphahat metric, and showed that its two components separately capture scale and shape.

Weaknesses:
- The paper simply analyses the failures of alpha and the log spectral norm. There is not really any theoretical justification in this paper on why combining the two components in alphahat should result in its good performance. 
- this submission has been referenced in [1] published in 2021. I think it could serve as an appendix to the journal paper, but the contribution is insufficient for a full paper at ICLR.


[1] Martin, Charles H., and Michael W. Mahoney. "Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning." J. Mach. Learn. Res. 22.165 (2021): 1-73.
clarity,_quality,_novelty_and_reproducibility: Clarity:
- the paper is sufficiently clear, although there are acronyms that are defined after their use (e.g., ESD), and graphs where the x-axis variables are not visible (e.g., Figure 1). 

Quality:
- the analysis in the paper is purely empirical, and lacks theoretical justification on why alphahat (previously published in 2021) should do well.

Novelty:
- the materials in the paper is probably novel, but is publicly available and referenced (by the same authors) in their 2021 journal paper.

Reproducibility:
- the weight watcher code is publicly available, and so the implementations of the subcomponents should be in that code.


summary_of_the_review: I propose to reject the paper as 
- the contribution as an empirical analysis of results published in 2021 does not qualify it for ICLR
- the paper has been referenced in 2021 by a publication by the same authors.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper aims to compare and analyse neural models without having access to training or test data, offering new insights on a particular phenomenon, namely Simpson's paradox. The reviewers found the paper slightly difficult to follow for non-experts and the contribution incremental and somewhat niche. The authors contested the reviewers claims, however, reviewers were not convinced that the paper will have  any impact on wider research beyond the HT-SR theory. 
justification_for_why_not_higher_score: See metareview above. 
justification_for_why_not_lower_score: See metareview above.
Decision: Reject