The Hyperfitting Phenomenon: Sharpening and Stabilizing LLMs for Open-Ended Text Generation

Authors: Fredrik Carlsson, Fangyu Liu, Daniel Ward, Murathan Kurfali, Joakim Nivre
Primary area: foundation or frontier models, including LLMs
Keywords: llm, nlp, fine-tuning, open-ended text generation, hyperfitting, phenomenon, neural networks, early-stopping, overfitting

Abstract:
This paper introduces the counter-intuitive generalization results of overfitting pre-trained large language models (LLMs) on very small datasets. In the setting of open-ended text generation, it is well-documented that LLMs tend to generate repetitive and dull sequences, a phenomenon that is especially apparent when generating using greedy decoding. This issue persists even with state-of-the-art LLMs containing billions of parameters, trained via next-token prediction on large datasets. We find that by further fine-tuning these models to achieve a near-zero training loss on a small set of samples -- a process we refer to as hyperfitting -- the long-sequence generative capabilities are greatly enhanced.
Greedy decoding with these Hyperfitted models even outperform Top-P sampling over long-sequences, both in terms of diversity and human preferences. This phenomenon extends to LLMs of various sizes, different domains, and even autoregressive image generation. We further find this phenomena to be distinctly different from that of Grokking and double descent. Surprisingly, our experiments indicate that hyperfitted models rarely fall into repeating sequences they were trained on, and even explicitly blocking these sequences results in high-quality output. All hyperfitted models produce extremely low-entropy predictions, often allocating nearly all probability to a single token.

Decision: accept (poster)
Reviewer scores: 6, 8, 5, 6

Reviews and discussion:
<thread>
creation date:
1730280581041

thread_type:
official_review

<thread_object>
creator: reviewer_p1nv

<content>
summary:
This paper focuses on the problem of repetitive generation results with large language models (LLMs) under greedy decoding strategy. Specially, this paper introduces a newly-observed phenomenon called *hyperfitting*, which helps to eliminate the repetition problem. This is achieved by finetuning the LLM on a separated small dataset untill achieving minimal loss. This paper runs abundant experiments to show that, hyperfitting can efficiently increase the token diversity as well as human preference for greedily generated text, with a side effect that the conditional token probability distributions are generally sharpened. Also, such hyperfitting phenomenon is observed in popular LLMs such as Llama 3.1, and even in image generation models (ImageGPT).

strengths:
* The introduced hyperfitting phenomenon is interesting, and there may be some potential application scenarios.
* There are abundant analytic experiments in this paper to support the widespread existence of the hyperfitting phenomenon in LLMs, as well as demonstrating the side effects of hyperfitting.

weaknesses:
The main weakness of this paper lies in its weak contribution. Though the introduced hyperfitting phenomenon is attractive, this paper failed to convey crucial conclusions concerned by readers. Neither did this paper reveal the key reason why such phenonmenon exists and how it works, nor did it validate its advantages in practical downstream tasks. The experiments only answers how the hyperfitted model behaves, but did not answer why. I noticed there are some results for downstream tasks in Tab. 6, however, I found the results are even worse for hyperfitted models, and the experimental setup has its limitations (no comparison with other random decoding strategies, which are more practical in my point of view). 

Either of the following suggestions can help for improving this paper:
1. Explore the key reasons why hyperfitting works for eliminating the repetition problem, as well as why hyperfitting results in sharped predictions.
2. Demonstrate the advantages of the hyperfitting technique under a variety of realistic downstream tasks, and compare with non-greedy decoding strategies.

questions:
1. There seems to be a contradiction between line284 "never appear in the hyperfitting dataset" and line301 "neither occur in the training data", which is right?
2. Is the finetuning dataset a subset of the training datasets?
3. What if one apply random decoding with hyperfitted model? Will the generation results be similar to greedy decoding (because of low entropy)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730608082742

thread_type:
official_review

<thread_object>
creator: reviewer_mz78

<content>
summary:
The paper introduces the hyperfitting phenomenon, where overfitting pre-trained LLMs on a small dataset until it achieves near-zero training loss enhances the LLMs' long-form generation capabilities, yielding higher-quality texts that are preferred by humans, despite the models achieving significantly worse validation losses. In particular, the paper finds this phenomenon across models hyperfitted on Fiction-Stories dataset, these models rarely repeat training sequences and produce low-entropy predictions. The paper takes an approach to explain this phenomenon with the concept of top-rank encouragement, where hyperfitting prioritizes desirable tokens in the top ranks of predictions, resulting in improved text quality.

strengths:
1. The paper is well written and easy to follow.
2. The discovery of the hyperfitting phenomenon is interesting, and the paper includes sufficient experiment to support this.
3. The explanation on the hyperfitting phenomenon is convincing.

weaknesses:
1. I'm not quite sure what specific applications this phenomenon has, because LLM practitioners rarely use pretrained base models directly for generation tasks. Generally, they first perform SFT (supervised fine-tuning) and then use chat models for generation. However, this paper does not compare the generation quality between hyperfitted models and chat models.
2. Although this paper shows hyperfitting can improve the generation quality, its impact on the model's internal knowledge, hallucination, and other factors has not been studied in the article. I suggest that the authors test the effects of hyperfitting on model performance using datasets like MMLU and GSM8K.

questions:
Please refer to weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730667666718

thread_type:
official_review

<thread_object>
creator: reviewer_gnwy

<content>
summary:
This work studies a surprising phenomenon that occurs when a modern LLM is overfit to a small text corpus finding that a model's greedy decoding capabilities are actually improved in various ways rather than degenerating as one might expect. They observe that after training a model to near zero loss on the small target corpus, while these models become poor "language models" for other held out text (poor validation loss), in _generative_ evaluation scenarios, "hyperfitted" models produce better outputs according to human judges and exhibit higher diversity in their completions as compared to the base models from which they are derived. They also perform similar experiments with autoregressive image generation models and observe similar effects. They present and analyze an explanation for these observations focusing on the "sharpening" of the model's predictive distribution and ablate the effect of the dataset and training curricula on these phenomena.

strengths:
1. Work explores an interesting phenomenon and presents it as "curious" without resulting to hyperbolic/"hype"-y language.
2. Contextualization in prior work is relatively complete.
3. A variety of diversity measures are used to examine the difference in output distributions caused by "hyperfitting" in a wholistic manner.
4. The human preference evaluation is a strong test for impact of hyperfitting, and the results for the hyperfitted models are promising -- the recovery of the 7B and 8B models to near equivalence in preference to ground truth completions is surprising.
5. In the event one wanted to use a hyperfitted model in practice, the "citation-blocking" mechanism is a simple and practical solution to the increased generation overlap with the small training corpus that hyperfitting causes.

weaknesses:
1. There is only a focus on the impact to open-ended generation (continuation/completion) and not enough focus on utility. There are no conversational benchmark scores like AlpacaEval or MT-bench, nor knowledge intensive benchmark scores such as for MMLU or other tasks available in suites like the lm-eval-harness. 
2. Use of base models only in this particular case is an issue since it might reveal more about the observation and hypothesized mechanism being presented if instruction tuned models were included in the analysis.
3. Diversity and length are not equal to output quality or utility, and this equivalence both implicit in the analysis as well as explicitly stated a few times.

Some of these issues may be addressable within the review window and thereby enable a recommendation of acceptance, see questions.

questions:
Primary:
1. Relating to the weakness about lack of benchmark scores, what do the differences in MMLU or other leaderboard tasks look like? I know Tinyllama scores too poorly to measure on many tasks, but the 7B and 8B models should have non trivial base performance. I suspect that since hyperfitting seems to destroy the ability of the model to properly assign loss values to val data (eg. no longer is a good "language model" in the technical sense of the term), this will impact these types of benchmarks. If benchmark scores after hyperfitting are near chance or even just significantly degraded, try reformulating the tasks as a generative evaluation and checking the hyperfitted model again.
2. Relating to the weakness about only including base models, did the authors run any experiments with instruction tuned versions of these same base models? I hypothesize that hyperfitting to these stories or news datasets effectively seems to do some of the work that all of the post training process normally accomplishes. Whatever this unqiue style of training does, seems to sort of take the model out of LM p(x) mode and into generator f(x)->y mode. So... what happens when you try to "hyperfit" an already post-trained model? Does it get worse/better/remain unchanged wrt the diversity analyses presented and the benchmark scores noted in prior comments?

Minor:

3.  In Table 3, what do "@1/3/5 Prob" mean? They are not defined anywhere and this table has no descriptive caption.

4. TTR seems to be a measure from linguistics and child language acquisition research, please define this inline in the sentence in which it is first used. Alternately use some self-evident description like "ratio of unique n-grams" for n=1 and other values. Also, Self-BLEU considers up to what n-value in this analysis? Generally, discussion and analysis of n-gram diversity and n-gram copying rate from training data might be more interpretable.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681504500

thread_type:
official_review

<thread_object>
creator: reviewer_kad8

<content>
summary:
The paper discovers a new phenomenon in fine-tuning LLMs on tiny datasets, which they term Hyperfitting as it's closely connected to overfitting. Surprisingly, while overfitting indeed increases validation perplexity, the model performs much better in open-text generation as rated by human evaluation. They find that models even perform comparably to those with 10x larger parameters in open-ended generation tasks. The phenomenon is demonstrated across three different models and three validation datasets across different domains in text generation. Additionally, they also conduct preliminary experiments in autoregressive image generation and observe similar effects. The authors also find that the fine-tuned models tend to generate a more diverse context, which may help mitigate the common repetition issue in long-text generation.

strengths:
- The paper is well-written and easy to follow
- The phenomenon is novel and quite surprising, especially its implication in reducing the repetition issues. The paper opens up a new perspective in understanding repetition in text generation. 
- I enjoyed reading the experiment results presented in the paper. E.g., the experiments in Section 6.2 present an interesting finding that the model fine-tuned on News performs better in human evaluation than the ones fine-tuned on Fiction and Wiki. This result is surprising given that Fiction, with its inherently diverse and creative language, might be expected to enhance performance more. This finding invites further exploration into how different training domains impact model generation quality.

weaknesses:
- The experiments focus on relatively small models, with the largest at 8B parameters. Since Hyperfitting is studied on small datasets, extending the analysis to larger models could provide insights into whether this phenomenon scales and is influenced by model capacity.
- A related issue is that the models are only evaluated in a limited range of datasets, which may not fully capture the phenomenon’s applicability across domains and tasks. For example, it would be useful to see how Hyperfitting behaves in specialized domains such as legal summarization [1], medical transcription [2], or dialogue summarization [3]
- Some implementation details appear to be missing. E.g. are training and evaluation results averaged over different random seeds to ensure consistency? Moreover, are all parameters updated during fine-tuning? If so, does the phenomenon also exist in parameter-efficient fine-tuning, e.g. LoRA [4]?

[1] https://huggingface.co/datasets/lighteval/legal_summarization

[2] https://huggingface.co/datasets/rungalileo/medical_transcription_4

[3] Chen, Yulong, Yang Liu, and Yue Zhang. "DialogSum challenge: Summarizing real-life scenario dialogues." Proceedings of the 14th International Conference on Natural Language Generation. 2021.

[4] Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685 (2021).

questions:
Please refer to the weaknesses part.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a finding denoted as the "hyperfitting phenomenon," which demonstrates that overfitting pre-trained Large Language Models (LLMs) on a small dataset can enhance their open-ended text generation capabilities. This approach mitigates the issue of repetitions in long-text generation. This is a counterintuitive phenomenon and it is explored and analyzed using various models and datasets, supported by a relatively large-scale human evaluation. More precisely, the study finds that hyperfitting leads to worse validation loss, deteriorating the language model's overall performance, but results in generated text that aligns better with human preferences and exhibits greater diversity according to established metrics. The authors propose some explanations for this phenomenon, including the sharpening of predictive distributions and improved top-rank prediction. Beyond text, preliminary experiments and qualitative evaluations are also conducted for image generation.

The reviewers acknowledge the novelty and significance of the phenomenon, as well as the experimental validation, including the extensive human evaluation. They raised questions regarding the practical implications, the explanations provided for the phenomenon, and the need for additional experiments with larger LLMs and more diverse datasets. During the rebuttal, the authors conducted experiments with larger LLMs, instruction-tuned models, and a new dataset (MMLU), addressing many of the concerns raised in the initial review. Following the rebuttal, the reviewers concluded that the novelty of the finding, along with the enhanced experimental contributions validating the observation across a range of models and tasks, represents a meaningful contribution. I recommend acceptance.