Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper addresses the issue of pre-training loss cannot fully explain downstream performance, they instead claim that the flatness of the model is well-correlated with downstream performance. The author showed that at the same level of pretraining loss, large models have better downstream performance, because of the flatness of large models. The authors conducted experiments with PCFG/HMM/OPT-generated data as downstream tasks. The authors also provided theories to support their claims.

strength_and_weaknesses: Strength:
- The paper is well-written and easy to follow.
- The paper is supported by many experiments to verify the claims that the authors proposed.
- Besides empirical results, the author also provided theoretical points of view to support their claim.

Weaknesses:
- The claim that flatness can decide the downstream performance is not well-supported. There are many factors that can affect the downstream performance, for example, the model size. One possible explanation is that the flatness could be a consequence of scaling up the model size, and the good performance is brought by the large model size, too. If that is the case, then flatness is not the reason for good downstream performance. The authors need to have more evidence to prove that flatness is the main reason that leads to good performance, otherwise, they may be both the consequence of another factor (like model size). 
- When computing Hessian, I assume that you are using the loss of pretraining datasets, not downstream datasets. Then how can the flatness on the pretraining task reflect the situation on downstream tasks? This claim is not very intuitive. It may need more explanations.
clarity,_quality,_novelty_and_reproducibility: Clarity&Quality: The paper is clear, well-written and easy to follow.
Novelty: As far as I know, there is no existing paper discussing the same topic.
Reproducibility: The authors didn't provide the code. I think it's hard to reproduce the results by ourselves.
summary_of_the_review: The paper is good in terms of its qualities, but there are some problems that need to be answered to further prove the correctness of their claims. I would tend to accept this paper if they can give good answers to my questions. For now, I will give a borderline score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Practitioners tend to gauge the downstream performance of large language models by comparing their pretraining losses. However, this paper shows that when the pretraining loss is near convergence, downstream performance can vary depending on the training methods, despite near identical pretraining loss. While there isn’t necessarily a correlation between the pretraining loss and downstream performance, the “flatness” of the solution, as characterized by the trace of the Hessian, does correlate with downstream performance. This observation is then formalized and proved on a synthetic language task.
strength_and_weaknesses: This work seeks to better understand an important question in representation learning: how does pretraining performance correlate with downstream performance?

This question is adequately answered in controlled settings where we observe the non-correlation between pretraining loss and downstream performance and the correlation between solution flatness and downstream performance. The investigation is both well-motivated and nicely executed. In addition, a theoretical result is provided in support of the empirical observations.

It is, however, unclear what the practical implications of this work are. First of all, current large language models are not in the saturation regime, and it is hard to estimate when they will be as datasets grow with model size in tandem. Second, while this paper points that pretraining loss is not a reliable indicator of downstream performance, a simple remedy is to evaluate on downstream tasks during pretraining and compare models accordingly, which is likely already done in practice. Finally, this paper does not demonstrate if the insight gleaned in this work can lead to additional “flatness regularization” that induces better downstream performance on real datasets. It is understandable that large-scale experiments are expensive and are not expected, but given the rather empirical motivation of the paper, some validation on real data seems desirable.

clarity,_quality,_novelty_and_reproducibility: The paper is well-written and presents novel results.
summary_of_the_review: Interesting empirical observations and theoretical result. However, more empirical results on real data, especially on how the discovered insight can enable better downstream performance, would strength this work given the rather empirical motivation.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: the submission conducted experiments to study the impact of configurations on the transferability of a pre-trained model through the lens of Hessian matrices, and the studied configurations include the number of training iterations, the training objectives, and the size of a model. Unsurprisingly, the conclusion is that more training iterations, standard training objectives, and larger models would lead to better performance on the downstream tasks. 
strength_and_weaknesses: ----Strength----

1. the submission found a correlated trend between the decay of the trace of the Hessian matrix evaluated on the training data and the increase of performance on the downstream tasks.

2. given the above observation, it becomes straightforward to infer that a pre-trained model with parameters at a flat minimum would lead to decent performance on the downstream tasks.

----Weaknesses----


1. the trace of a Hessian matrix in the formulation from this paper is very easy to obtain and one doesn't even need to materialise the matrix. The issue is that the Hessian matrix is evaluated around the global minimum of a model being trained on the pre-training data, and, without any knowledge of the pre-training data, it becomes very difficult to compute the Hessian matrix. The other issue is that it does require many samples to obtain a reasonable estimate of the trace.

2. the trace of the Hessian matrix is only an overview of the magnitude of all eigenvalues. A smaller trace could mean that the minimum might be flatter than other minima with the same loss, but it could also mean that the minimum might be steep in only a few directions, and completely non-informative in others. The latter case wouldn't necessarily give us a strong transferable model, and yet the trace of the Hessian matrix wouldn't be able to tell.

3. maybe I am missing the point and I am happy to be corrected, but the arguments made in this submission or the observations obtained are already well-known empirically, especially in the case of zero-shot and few-shot learning with very large pre-trained transformer models. In addition, the theorems mentioned in the submission don't seem necessary, or, in any case, support the arguments or the observations.
clarity,_quality,_novelty_and_reproducibility: clear presentation, but not very original
summary_of_the_review: I don't recommend accepting the submission since the empirical results are well-known and the theorems don't provide theoretical justifications to the observations.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper questions the common sense that correlation between pre-training validation loss such as masked language modeling loss and generalization performance of downstream tasks after fine-tuning. It empirically shows that the models with the same pre-training loss can achieve varying test accuracy on downstream tasks. Instead, the models converged to flat minima obtain better test accuracy.
strength_and_weaknesses: 
## Strength
- The paper explores the interesting and practical research question --- correlation between pre-training loss and generalization performance of downstream tasks.


- In the controlled experiments, the authors empirically show the correlation between flatness and generalization. Even though language model achieves optimal pre-training loss, we can improve generalization performance of downstream task as we continue pre-training, which results in flatter minima.

- Authors provide some theoretical analysis that connects flatter solution to the pre-training loss and generalization performance of downstream tasks in the limited settings. 
--- 
Please address my following concerns. If they are properly addressed, I am willing to raise my score.
## Weakness

- My expertise not learning theory, but in my humble opinion, what the authors claims is too bold. They claim that they "prove that SGD with standard mini-batch noise implicitly prefers flatter minim in language models", but it is limited to the special case ---  Dyck language experiment which is far from practical scenario.  It would be better to tone down the claim.

- I am a bit confused about the correlation between flatness and generalization. For example, Dinh et al., [1] show that we can build equivalent models but with sharper minima, which might contradict to  what the authors claim, specifically theorem 5.1. However, the authors did not properly tackle this issue even in the related work section. 


- It is not clear how the size of models and flat minima is related to each other. The authors claim that smaller transformer architecture is a subset of the larger transformer architecture family, but it does not make sense. If we increase the width of transformer or increase the vocabulary size, the smaller model cannot be a special case of larger model. It would be better to explain why larger model converges to flatter minima than smaller one.

- In the experiments, it is unclear why the authors compare vanilla sgd training with adversarial training. I think adversarial weight perturbation [2] is more relevant baseline since it explicitly regularize the flatness of weight loss landscape.

- Minor typo: In page 4, (2) training for different number of steps -> (1) training for different number of steps


## Questions

- Regarding the flat minima, what happens if we explicitly enforce the language model to converge flatter minima? For example, we can explicitly regularize the trace of Hessian to be small or use SAM [3] optimizer to enforce such regularization.

- Why is the cross-entropy loss is typically non-zero at the global minimizers?

- Why do you assume that we only mask a single token in a sentence for masked language model? In practice, we usually mask 15% of tokens  of a sentence (e.g. BERT [4], RoBERTA [5]).

## References
[1] Dinh, Laurent, et al. "Sharp minima can generalize for deep nets." International Conference on Machine Learning. PMLR, 2017.

[2] Wu, Dongxian, Shu-Tao Xia, and Yisen Wang. "Adversarial weight perturbation helps robust generalization." Advances in Neural Information Processing Systems 33 (2020): 2958-2969.

[3] Foret, Pierre, et al. "Sharpness-aware Minimization for Efficiently Improving Generalization." International Conference on Learning Representations. 2021.

[4] Devlin, Jacob, et al. "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding." Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019.


[5] Liu, Yinhan, et al. "Roberta: A robustly optimized bert pretraining approach." arXiv preprint arXiv:1907.11692 (2019).


clarity,_quality,_novelty_and_reproducibility: Although this paper tackles interesting research question, it is unclear that flatter minima actually leads to better generalizations since it does not compare relevant baselines and lack of justification. Please see detain in strength and weakness section.
summary_of_the_review: As previously mentioned, the authors do not properly tackle the open question --- flat minima and generalization which might still seem to be controversial topic. Moreover, they need to tone down their claim since their theoretical analysis is limited to very specific controlled experiment which is far from realistic scenario. Thus, I am inclined to reject.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work shows that pretraining loss does not fully explain downstream performance and claim that the flatness of the model is well-correlated with downstream performance. The authors conduct experiments with synthetically generated data as downstream tasks and tie that together with theories to show how flatness improves downstream performance.

Strengths:
* The paper is well written.
* This work tackles an important question in representation learning, which is the question of how pretraining performance affects downstream performance.
* The work found a correlation between the decay of the trace of the Hessian matrix evaluated on the training data and the improved performance on downstream tasks.
* The work provides some theoretical analysis that connects flatter solutions to the pretraining loss and generalization performance of downstream tasks.
* This could be a new metric for language models other than perplexity.

Weaknesses:
* The work is unable to show a causal link between flatness and downstream performance. In particular, the work does not attempt to perform any interventions, such as regularising the trace of the Hessian or using the SAM optimizer to try to improve the flatness directly. This means the common causal factor for flatness and downstream performance could still be things such as the model size.
* It's unclear if this link between flatness and downstream performance generalise to more tasks.
* The theoretical analysis is too constrained for general practical applicability.
* The work shows that flatter minima leads to better generalization but only in the special case of the Dyck language experiment.
justification_for_why_not_higher_score: While this work tackles an interesting problem and raises an interesting finding, without theoretical backing it's unclear if this finding generalizes to other downstream tasks. Interventions on model flatness and theoretical analyses of flatness on a wider range of tasks would make this a stronger paper.
Decision: Reject