Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper empirically measures correlations between various generalization metrics and model quality (BLEU score) on a machine translation dataset across a range of hyperparameters. Compare to prior work, the main differences are:
- Focusing on NLP, rather than computer vision
- Focusing on metrics that predict test error rather than generalization error
- Focusing on a class of metrics based on whether the empirical spectral density (ESD) of the weight matrix follows a power law.
strength_and_weaknesses: **Strengths**
- This paper extends the study of generalization metrics to NLP. This is a valuable contribution, because unlike in computer vision, NLP models are not typically trained to zero training error. As the authors note, even if a generalization metric is rank-correlated with the generalization gap, it may not be correlated with the test error due to differences in training error. There are also differences in neural network architecture (transformers vs. CNNs).
- The paper considers a fairly wide range of metrics and hyperparameters.

**Weaknesses**
- There are two main components in the paper (1) Evaluating natural language processing models with generalization metrics, and (2) advocating the use of a certain class of generalization metrics based on heavy-tail self-regularization theory (HT-SR), including several new metrics based on fitting exponentially truncated power laws to the ESD of the weight matrix. In my opinion, the second point is largely orthogonal to the first, and it's difficult to assess whether these HT-SR metrics are preferable to existing metrics due to methodological differences with prior work [1, 2]. I think the paper would be stronger if it placed less emphasis on shape metrics and focused more on evaluating generalization metrics for NLP, using a similar evaluation methodology to prior work.
- Interpretation of Figure 4 (comparing generalization metrics): Figure 4 indicates that scale-based metrics are positively correlated with generalization gap but have a strong negative correlation with model quality, meaning a lower value of the generalization metric is associated with higher value of model quality. Isn’t this to be expected? We would expect that a low generalization gap is correlated with a high quality score. From this point of view, the metrics at the bottom of Figure 4a seem better. The other evaluations in the main paper only compare shape metrics, or compare correlations with hyperparameters rather than quality (see next point), so overall I am not convinced that shape metrics are preferable to other metrics.
- The evaluation setting in 3.3 is puzzling to me. From what I can understand, this section does not measure the correlation between generalization metrics and model quality, but instead measures correlation with architecture parameters like depth and hidden dimension. It's not clear how this section supports the argument of the paper. This section might be more convincing if it compared models that were trained on the same data and then evaluated on a consistent quality metric.
- Justification for HT-SR metrics: In particular, is there any theoretical justification for these metrics? The authors propose several different shape metrics (based on PLs and E-TPLs, different parameters, and goodness-of-fit metrics). Which of these should be preferred?
- Methodological differences from prior work: Prior work [1, 2] has explored methodological questions about comparing generalization measures and proposed metrics and methodologies aimed at controlling for noise and unwanted correlations---in particular using Kendall's tau correlation and controlling for Monte Carlo noise across environments. This paper reports Spearman's rank correlation and does not explicitly address the questions raised by [1, 2] about how to robustly evaluate generalization metrics.
- "Generalization metrics that do not need access to any training or testing data": The authors argue that shape metrics are preferable "without access to training or testing data". It would be valuable to be more clear about how existing metrics depend on the training data: to my understanding, metrics based on generalization bounds depend only the number of samples in the training data, which is a relatively mild requirement (you need to know the size of the data, but don't need access to the data itself). Also, none of the other generalization metrics considered use the testing data, so that should not be considered a special property of shape metrics.
- The authors note that "NLP pretraining datasets are typically web-scale and are challenging to access", but their evaluation is focused on a machine translation dataset (where the data is available), so the argument about NLP pretraining is not entirely relevant.
- Correlation evaluated on optimally trained models: It's not clear to me that this is a reasonable evaluation setting. How are you supposed to determine which model is optimally trained without access to testing data? If you can determine whether a model is optimally trained, why do you need a generalization metric?

- The paper only considers one dataset and quality metric (BLEU score). It would be valuable to consider a wider range of commonly used NLP benchmarks, such as the GLUE benchmark.
- It's not entirely clear to me which weight matrices are being considered in these metrics, and how the scores of different weight matrices are aggregated. More generally, it would be valuable to discuss another difference from prior work, which is that the models considered here are based on the transformer architecture, while prior work has mainly studied CNNs (and most theoretical metrics have been developed for feed-forward networks, but see [3].)

[1] Jiang et al., 2019. Fantastic generalization measures and where to find them.
[2] Dziugaite et al., 2020. In search of robust measures of generalization.
[3] Long and Sedghi, 2019. Size-free generalization bounds for convolutional neural networks.
clarity,_quality,_novelty_and_reproducibility: *Quality*: The paper conducts an extensive evaluation of generalization metrics, but I have some doubts about the choice of evaluation and the interpretation of the conclusions. In particular, the first set of evaluations (figures 2 and 3) only only compares shape metrics; the second set of evaluations (figure 4) may indicate that scale-based metrics are actually preferable (see above); and final set of evaluations (figure 5) may have limited relevance to the paper, because it is comparing correlation with model hyperparameters rather than a quality score.

*Clarity/reproducibility*: The paper is clearly written for the most part, but I have methodological questions about how the scores from different weight matrices are combined to calculate the PL metrics.

*Novelty*: To my knowledge, evaluating generalization metrics for NLP has not been addressed in prior work, and this is a valuable new contribution.
summary_of_the_review: My recommendation is to reject this paper. The paper addresses an interesting problem, evaluating generalization metrics for NLP, but it is difficult to interpret the results due to the secondary focus of the paper on advocating the use of a certain class of metrics, and I find that some of the analysis does not seem to support the main claims of the paper. I think the paper would be stronger if it placed less emphasis on shape metrics and focused more on evaluating generalization metrics for NLP, using a similar evaluation methodology to prior work.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors provide the first systematic empirical study on various generalization metrics in NLP, including 400 transformers trained with varying hyperparameters. The authors measure the correlation between 28 generalization metrics and the model quality. The authors find that shape metrics consistently perform better than scale metrics for predicting model quality.


strength_and_weaknesses: Strength:
1. The authors have done widely exploration of generalization metrics on machine translation tasks.
2. The metric can be used for machine translation model evaluation. 

Weakness:
1. The paper over-claimed the contribution. The authors only work on machine translation task but claim "Evaluating natural language processing models " in the title. I would suggest authors work on GLUE/SuperGLUE/AdvGLUE dataset to further explore the robustness.
2. Not sure why the authors want to emphasize "do not need access to any training or testing data". It is ambiguous. Some scores like BERTScore (Evaluating Text Generation with BERT), BARTScore (BARTScore: Evaluating Generated Text as Text Generation) also don't need. Can the metric evaluate model only based on the model initialization?
clarity,_quality,_novelty_and_reproducibility: Transferring the metric from CV to machine translation. The novelty looks a bit limited, but experiments are solid.
summary_of_the_review: Overall, I think the authors over-claimed too much on the contribution. I would suggest the authors either add more NLP tasks in the experiments or change the title. I appreciate the large-scale experiments on machine translation tasks, while the metric mainly comes from CV and the novelty is little bit limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper provides a very comprehensive empirical analysis of predicting the test set performance on NLP tasks, and reveals several interesting properties like shape based metrics are generally better than scale based ones, and some fitting strategies (for empirical spectral densities of a weight matric) like E-TPL could alleviate the issues of other typical fitting methods like pure PL based. 
strength_and_weaknesses: **Strengths**

1. The problem at hand is indeed meaningful: getting the full-set dev set performance for real NLP tasks, especially that web-scale ones, are very time and resource consuming, hence having a data-free surrogate metrics for model selection would be quite useful, especially for real industry use cases;

2. The study here is quite solid which delivers convincing insights for the community. 

3. The paper is generally written in a clear way.

**Weaknesses**

1. The raw innovation of this paper seems limited given its natural of being an empirical study focused work.
clarity,_quality,_novelty_and_reproducibility: **Clarity**

It is impressive that this paper is stated in such a clear way that I feel easy to absorb quite a few information it wants to deliver.  In particular it gives a good literature survey which lays down the overall background.

I only have one question which puzzled me: I cannot figure out why *Then even if we have i) access to both models’ training errors, and ii) a metric which is guaranteed to perfectly rank correlate with the generalization gap, then we still cannot determine which model as smaller test error.*

**Quality**

The study here is quite solid. For example different settings of NLP experiments are studied like across different well trained models, across different hyper-parameters and across different training phases/time. I believe it could be very useful for NLP researchers who care a lot about real-world performances.

**Novelty**

While this paper proposes several new metrics to study, its technical novelty seems not comparable with several previous works. On the other hand there is a certain degree of novelty in terms of problem itself, that is driving some data free metric to predict test set performances in NLP which would be very useful in real-world.

**Reproducibility**

Good.

 
summary_of_the_review: As stated above, despite that lacking of technical novelty, this paper is generally good given its solid empirical study which could be of good/common interest to NLP researchers.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a metric to measure generalization ability for transformer models that is not data-dependent. Their proposed metric is based entirely on a spectral analysis of the weight matrices, and the authors seek to show that this metric correlates directly with test accuracy (as a proxy for model generalization) rather than the 'generalization gap' considered by other works. They demonstrate the effectiveness of their generalization metric on transformers trained with a wide variety of hyperparameter settings on WMT '14, as well as analyzing its predictions on a number of pretrained models from the literature such as BERT, GPT-2 and T5.
strength_and_weaknesses: **Strengths**

The idea of producing a comprehensive study of generalization metrics for NLP models is an interesting and valuable contribution, and the proposed metric does indeed perform well in the thorough empirical studies they performed. The authors tested a wide variety of hyperparameter settings and convincingly showed strong performance on predicting generalization ability. The analysis of existing pretrained models also led to some interesting results, and considered a wide variety of models from the literature with different variants.
	
**Weaknesses**

My primary criticism of this paper is that almost all this analysis was performed only on a single dataset:  the WMT '14 english to german translation dataset. As such, it is difficult to be entirely confident that the results shown in the paper are truly general, and not specific to either this dataset or machine translation models on the whole. An analysis of multiple datasets with diverse task settings and dataset sizes that could show consistent results would be significantly more convincing. I certainly understand that this must already have been an enormous computational undertaking, but surely some of the resources used in the broad hyperparameter search might be better spent on replicating these results with other datasets. This is particularly critical given that this paper's primary contribution seems to be the broad scope of empirical analysis, rather than technical novelty.

My one other criticism is that I believe it is extremely important that papers relying on large-scale experiments accurately report the computational cost of their experiments and particularly the approximate greenhouse gas emissions of said experiments (there are a number of calculators online for estimating this). It is important for us to be cognizant of the environmental implications of our research and I believe transparency about the environmental cost of our experiments is ethically significant.

Questions/Minor errors:
1) I don’t believe ESD is defined anywhere in the introduction of the paper outside of the abstract
2) Figure 2 is a little bit unclear. You state that "the shape metrics always correctly predict the model quality, i.e., the BLEU scores should be higher when the metric values are smaller". Is this true only for the black starred models (the optimal configurations)? The wording makes it seem like this holds more generally, but this is not visually obvious and is contradicted by the caption.
3) In Figure 4 the 'scale' metrics seem near-perfectly anticorrelated with model quality. Shouldn't that mean these metrics are actually *more* predictive of model quality than the shape metrics, as a perfect negative indicator should be just as valuable as a perfect positive one?

clarity,_quality,_novelty_and_reproducibility: The work is generally clear and well-written (with a few small exceptions), and makes a significant contribution to the field. It is not wholly original as it does not propose any technically novel methods, but the scope and thoroughness of the study nevertheless constitute valuable and novel contributions. My only qualm with this is the limitation of focus to only a single dataset, which may make the results of questionable value outside the scope of that one dataset and task.
summary_of_the_review: Overall I think the paper makes a valuable contribution in thoroughly cataloguing generalization metrics for NLP models and demonstrating the effectiveness of HT-SR methods at this task in a data-agnostic way. If these results had been demonstrated on a broader array of natural language tasks and datasets I would have no qualms about recommending it be accepted. As is, I believe this is a significant enough limitation for a work whose primary novelty is in the scope of its empirical studies that I view it as more of a borderline paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper provides a comprehensive empirical study of the correlations between various generalization metrics and test performance of NLP models. 

This paper is a borderline paper and has been discussed between AC and reviewers.

Most reviewers appreciate the thorough studies that have been done in this paper, and the paper is well-written and condenses a lot of information of background. The experiments are also generally solid.

There were several major concerns initially:
* The results were only reported on a single dataset WMT’14, while the scope of the paper targets “Evaluating NLP models..”
* Whether the paper offers a robust, comprehensive comparison of quality measures across hyperparameters (the standard needs to be high given the paper is a purely empirical paper, and some reviewers questioned the technical novelty of the work)

The authors added new experiments on MNLI, WikiText-103, and Reddit during the rebuttal phase. However, several reviewers still believe the new results are not convincing enough (e.g., only comparing the correlation between a single shape metric and quality across different amounts of training data, whether they represent a broad range of NLP tasks).

In summary, due to the above concerns and there isn’t any (strong) support for the paper, I can’t recommend acceptance of the paper in its current form.

justification_for_why_not_higher_score: This is a harder decision to make because the paper seems to be well-written and has clear merits.
However, several reviewers feel that the empirical results are not convincing enough in the current draft.
Decision: Reject