From Loops to Oops: Fallback Behaviors of Language Models Under Uncertainty

Authors: Maor Ivgi, Ori Yoran, Jonathan Berant, Mor Geva
Primary area: foundation or frontier models, including LLMs
Keywords: Hallucinations, Epistemic uncertainty, Degenerate text, Fallback behaviors

Abstract:
Large language models (LLMs) often exhibit undesirable behaviors, such as hallucinations and sequence repetitions.
We propose to view these behaviors as fallbacks that models exhibit under epistemic uncertainty, and investigate the connection between them.
We categorize fallback behaviors — sequence repetitions, degenerate text, and hallucinations — and extensively analyze them in models from the same family that differ by the amount of pretraining tokens, parameter count, or the inclusion of instruction-following training.
Our experiments reveal a clear and consistent ordering of fallback behaviors, across all these axes: 
the more advanced an LLM is (i.e., trained on more tokens, has more parameters, or instruction-tuned), 
its fallback behavior shifts from sequence repetitions, to degenerate text, and then to hallucinations.
Moreover, the same ordering is observed during the generation of a single sequence, even for the best-performing models; as uncertainty increases, models shift from generating hallucinations to producing degenerate text and finally sequence repetitions. 
Lastly, we demonstrate that while common decoding techniques, such as random sampling, alleviate unwanted behaviors like sequence repetitions, they increase harder-to-detect hallucinations.

Decision: reject
Reviewer scores: 5, 5, 5, 8

Reviews and discussion:
<thread>
creation date:
1730460462959

thread_type:
official_review

<thread_object>
creator: reviewer_rirk

<content>
summary:
The paper studies different fallback behaviors of LLMs that occur under uncertainty,
For this, the authors construct settings where the model is expected to be uncertain.
They evaluate various models differing in both size and how long they are trained and compare greedy vs. sampling-based decoding methods.
The authors find that "stronger" models (trained longer, more parameters) exhibit more complex fallback behaviors which means that more hallucinations are produced.

strengths:
- The idea of "epistemic uncertainty by construction" is an interesting way of circumventing problems with quantification of uncertainty.

- Overall, the paper seems methodologically sound.

- The paper is very well-written and easy to follow.

- The ideas presented in the paper can help guide further works on combating (unwanted) fallback behaviors of LLMs

weaknesses:
- Epistemic uncertainty as a whole is a quite rich concept that might warrant a larger discussion. In particular, the authors focus mainly on uncertainty about facts but mostly neglect other forms / sources of uncertainty that are also often seen as contributing to epistemic uncertainty. An example is L417-419: verbatim recollection is hypothesized to be more present in OOD settings than factual uncertainty but OOD settings are often also thought to give rise to epistemic uncertainty, because not enough training data is observed in the domain, leading to a lack of knowledge. For example, the authors could discuss the relation of epistemic uncertainty and aleatoric uncertainty [1] and the sources of epistemic uncertainty further [2] 

- The main dataset is comparably small with only 95 items but also highly curated. Similarly, the other datasets are also fairly small, usually around 100 examples

- Some statements are a bit imprecise, for example: what is meant with internal capability to avoid hallucinations (L303) and what backs it up experimentally?

questions:
Below are some questions and comments:

- Footnote 2 is a bit hard to understand without having read later parts of the paper

- it's not immediately clear to me why Datasets 1-3 have different levels of epistemic uncertainty, could you clarify this?

References

[1] Hüllermeier, Eyke, and Willem Waegeman. "Aleatoric and epistemic uncertainty in machine learning: An introduction to concepts and methods." Machine learning 110.3 (2021): 457-506.

[2] Baan, Joris, et al. "Uncertainty in natural language generation: From theory to applications." arXiv preprint arXiv:2307.15703 (2023).
</content>
</thread_object>

</thread>
<thread>
creation date:
1730638707286

thread_type:
official_review

<thread_object>
creator: reviewer_pl8x

<content>
summary:
This paper mainly analyzes the fallback behaviors of large language models (LLMs), which include sequence repetitions, degenerate text, and hallucinations. This paper conduct various experiments to explore the fallback behaviors in models from the same family that differ by  the amount of pretraining tokens, parameter count, or the inclusion of instruction-following training. It further discuss factors influencing the fallback behaviors of a LLM, such as sampling methods, simplifying the question and prompt engineering. The experiments reveal the following phenomenon:
1. Increasing model strength through extra pretraining, more parameters, or instruction tuning shifts fallback behaviors from simple to complex, i.e., from repetitions to hallucinations.
2. While LLMs have some internal capability to avoid hallucinations, this fallback behavior is inherent to their generation scheme and is likely unavoidable with current decoding methods. 
3. As models generate longer texts, they shift in their fallback behavior, first generating hallucinations and eventually producing degenerate text.

strengths:
•	This paper provides a controlled analysis of LLM undesired behaviors, showing that strengthening models increases hallucinations over degenerate text and repetitions.
•	This paper demonstrates that during generation, LLMs undergo a phase shift from correct answers to hallucinations and then repetitions.
•	This paper indicates that methods to reduce text degeneration, like sampling, increase errors and hallucinations.

weaknesses:
•	This paper seems to overclaim its contribution and conclusion. According to the datasets this paper used, the LLMs need to recall multiple facts that are related to the same topic. The other QA circumstances are ignored, including the simple QA like Natural Questions, multi-hop question answering like HotpotQA, open-ended long-formed question answering but beyond biography like ELI5. Also many other knowledge intensive tasks are also not discussed, which is also related with the epistemic uncertainty.
•	According to the paper, the fallback behaviors include sequence repetitions, degenerate text, and hallucinations. But degeneration is not discussed in the metric and experiment results.
•	The experiment setting of FQAMP seems not reasonable. If forcing uncertainty is essential, it is more natural to replace the subjects with a low popularity names. If replacing with made-up names and requiring LLMs to abstain, it is an OOD problem but not an uncertainty problem.

questions:
•	If this paper focuses on “epistemic uncertainty” and do not explore other types of uncertainty, why this paper is titles “under epistemic uncertainty”.
•	Is the TRIVIAFACTS dataset evaluated? The question mentioned in the paper, which is “The following 25 colors are in the Olympic rings.” seems to mislead LLMs to repeat and hallucinate.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716897379

thread_type:
official_review

<thread_object>
creator: reviewer_txwn

<content>
summary:
The paper studies how LLMs behave in the face of epistemic uncertainty, claiming that the LLMs “fall back” to various strategies, such as hallucinations, degenerate text, and more when this occurs. The study explores these fallback strategies across several model parameters, including model size, pretraining tokens, instruction tuning, and decoding algorithms.

The paper presents some interesting findings such as:

1. Larger models tend to be more factual but are more likely to hallucinate as a fallback strategy rather than generate repetitive or poorly formatted text.
2. As the number of pretraining tokens increases, the number of factual responses also increases, though the model may hallucinate more frequently.
3. Instruction tuning makes the model's fallback strategies more complex without significantly changing the percentage of factual answers.
4. Increasing the sampling temperature slightly reduces factuality but decreases repetitiveness and occasionally enhances creativity.


The authors conduct experiments on several different models, checkpoints, and datasets.

strengths:
The paper analyzes fallback mechanisms when the model encounters uncertainty—an important and timely issue. The experimentation is extensive, with tests on multiple models across various datasets to validate its conclusions. The findings are interesting and hold significant value for other researchers working on LLM hallucinations. Some of the conclusions seem less surprising (although empirical verification of them is still useful)

While the paper has notable strengths, there are some concerns (see weaknesses). I am open to raising my score once these issues have been addressed.

weaknesses:
1. **Presentation Improvements**

   - The paper’s presentation could be refined. In its current form, it may take considerable time for readers to extract the main takeaways.
   - There are multiple task settings, and results can sometimes be difficult to interpret due to the variety of datasets used for each claim. Standardizing these settings could improve clarity.
   - In some cases, important results are presented only in the appendix, which can be frustrating for readers. Additionally, the graph captions could be more informative (for instance, specifying the dataset used for Figures 3 and 4).
   - Terminology is sometimes inconsistent—for example, the term "stronger models" is used, but it’s unclear what this means.

2. **Experimentation Concerns**

   - Why are instruction-tuned models tested only on TriviaFacts? Although list-based questions may simplify evaluation, this limits conclusions to a single, small, self-created task (hence weakening the claim) 
   - Some parameters are analyzed independently, even though they often co-occur. For example, model size could be varied specifically within instruction-tuned models. This approach sometimes leads to inconsistencies. In Figure 21, llama-3-8B-inst is shown to be more factual (% factual) and hallucinates significantly more than llama-3-70B-inst—contradicting the earlier finding that larger models tend to produce more factual answers while hallucinating more. This also appears inconsistent with broader findings, such as in the Llama-3 report, where llama-3-70B is generally noted as more accurate.
   
   These outliers and inconsistencies warrant further discussion.

questions:
See weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730898971213

thread_type:
official_review

<thread_object>
creator: reviewer_cg5v

<content>
summary:
This paper looks at “fallback” behaviors in large language models in the face of epistemic uncertainty, specifically focusing on repetitive text, degenerate phrasing, and hallucinations. They set up scenarios to control the presence of epistemic uncertainty and characterise the outputs of various LLMs, evaluating how the number of parameters, amount of pretraining data, instruction-following training, and decoding impact the type of fallback behavior exhibited. They observe what they call a “fallback hierarchy” where as models become more uncertain (or if they are weaker to begin with), they rely more on basic degenerate behaviors. They posit that fallback behaviors are interconnected, based on the observation that models transition from one behavior to another as uncertainty decreases. Importantly, they find that larger, more sophisticated models also exhibit fallback behaviors in the face of uncertainty, even if they may be more sophisticated fallback behaviors and that some techniques claimed to mitigate degenerate behaviors have other adverse effects or only a limited ability to help

strengths:
* The work offers a thorough analysis across various LLM families of different types of frequently-observed degenerate behavior
* The experimental setup seems like a reasonable way to control for epistemic uncertainty
* There are interesting empirical findings about the prevalence and characteristics of degenerate outputs from modern LLMs

weaknesses:
* The contributions of the work are limited: 
    * empirical observations are made but no insights about what the source of these behaviors are or how they can be fixed are given
    * The analysis is limited to the presence of epistemic uncertainty, but there are other scenarios in which degenerate behaviors occur (e.g. when adversarial prompts are given or in the face of aleatoric uncertainty) so we’re still far from a comprehensive empirical analysis of degenerate behaviors 
* The setups are rather contrived and since the work's results are only empirical, its not clear what the practical implications of them are
* While the authors claim to explore the impact of decoding on these fallback behaviors, they seem to only look at greedy decoding and temperature sampling. They also seem to only explore one prompt as a “mitigation” technique
* Some claims don’t feel adequately supported. E.g., “fallback behaviors… emerge…unavoidably under uncertainty” Even though a comprehensive set of prevention techniques weren’t explored

questions:
I don’t understand what is meant in the sentence “Moreover, introducing randomness reduces the number of correct facts, which could be attributed to the random skipping of correct facts that have low confidence in the model.” Could you clarify?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732504560086

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revision updates and general response

comment:
We sincerely thank all reviewers for their thoughtful and constructive feedback, which has helped us significantly improve the clarity and quality of the paper. Below, we address the changes made in response to the reviewers’ suggestions, indicating where these revisions are reflected in the updated manuscript.

1. Clarifications on Key Points:
   - In response to Reviewer Rirk’s comment about footnote 2 being difficult to understand, we revised its phrasing for clarity. It now appears as footnote 3 in the new version.
   - Reviewer CG5v raised a question about the impact of randomness on factual answers. We have added footnote 7 to clarify why randomness reduces the correctness of answers.
2. Discussion Enhancements:
   - As suggested by Reviewer Rirk, we expanded our discussion of other types of uncertainties (e.g., aleatoric uncertainty) and their relationship to epistemic uncertainty. This is now included in Section 7 (related work, _Uncertainty in language modeling_, lines 503-519), along with citations to relevant literature, including Hüllermeier et al. (2021) and Baan et al. (2023).
   - Following Reviewer CG5v’s comments about the practical implications of our work, we added a discussion of these points to Section 8 (Conclusion and Discussion, lines 532-539).
3. Dataset and Experimental Justifications:
   - Addressing Reviewer PL8x’s concerns regarding FQAMP, we added a justification for its importance and its alignment with natural scenarios in Section 3, where the dataset is defined.
   - Reviewer tXwN noted inconsistencies in terminology such as “stronger models.” We have now provided a formal definition of this term in footnote 2 in the introduction, ensuring consistency throughout the paper.
4. Improved Presentation:
   - Reviewer tXwN suggested improving the clarity of figures by specifying the dataset used in the captions. We have updated captions for Figures 2, 3, 4, 5, and 7 accordingly.
   - To make the paper easier to follow, we verified that all significant results are now presented in the main body, even if full figures are still in the appendix. We thank Reviewer tXwN for highlighting this, and we agree that the revised structure is more reader-friendly.
   - To address Reviewer PL8x’s feedback on degeneration metrics, we made the discussion of degeneration explicit from the beginning of Section 6.2.
5. Additional Experiments:
   - Following Reviewer tXwN’s request, we conducted additional experiments with instruction-tuned models across more datasets. These results have been added to Sections 4.3 and 5.3, further strengthening our findings.
   - As requested by Reviewer PL8x, we performed new experiments with top-p sampling, alongside additional models for both top-p and temperature sampling. These results are included in Section 5.1, providing more robust evidence for our conclusions.
6. Addressing LLaMA Concerns: Reviewer PL8x and Reviewer tXwN raised concerns about inconsistencies in LLaMA results. We added detailed explanations of how models were loaded and potential issues with some LLaMA checkpoints weights. These are now included in Section 3 and footnote 5, with further discussion in Appendix E.1.
7. Structural Improvements: Reviewer tXwN suggested adding an outline of the paper to the introduction. We have now included a paragraph at the end of the introduction that outlines the structure of the paper and explicitly mentions where key topics, such as degenerate text, are discussed (e.g., Section 6).

We deeply appreciate the reviewers’ efforts to engage with our work and provide actionable suggestions. We believe these changes have strengthened the paper and made it more accessible and informative. As we have addressed the reviewers’ concerns and conducted the experiments they requested, we would sincerely appreciate it if they could reconsider their evaluation and consider increasing their scores. Thank you for your time and valuable input.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733208125132

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to once again thank the reviewers for their comments and suggestions. We have made considerable efforts to address the concerns raised, including running additional experiments and revising the paper accordingly. As the discussion period is nearing its conclusion and we have not received further comments, we would highly appreciate it if the reviewers can take our responses into account and reevaluate their assessment of the paper.
Thank you again for your time and constructive feedback.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper investigates fallback behaviours in large language models (LLMs) under epistemic uncertainty, categorizing them into sequence repetitions, degenerate text, and hallucinations, and proposes a "fallback hierarchy" that connects these behaviours. Experiments are performed across various model configurations, including variations in pretraining data, parameter sizes, and decoding methods, demonstrating consistent patterns in how LLMs exhibit fallback behaviours as uncertainty increases. 

Despite the relevance of the topic and the thoroughness of some experiments, the paper has several shortcomings. While the authors provide an interesting characterization of fallback behaviours, the claims about their inevitability and possible mitigation remain insufficiently supported. The experimental setups rely heavily on small, somewhat contrived datasets that limit generalizability, and key findings are weakened by inconsistent terminology, a lack of clarity in experimental descriptions, and an overreliance on appendices for important results. Although the authors conducted additional experiments and added discussions on related uncertainties during the rebuttal period, these revisions did not adequately address concerns about the paper’s scope, generality, and practical implications. Given the limited generality of the findings, the unresolved methodological and presentation issues, and the overstated contributions, I recommend rejecting the paper.