Critical Phase Transition in Large Language Models

Authors: Kai Nakaishi, Yoshihiko Nishikawa, Koji Hukushima
Primary area: foundation or frontier models, including LLMs
Keywords: natural language processing, large language models, statistical physics, phase transitions, critical phenomena, large-scale numerical experiments

Abstract:
Large Language Models (LLMs) have demonstrated impressive performance. To understand their behaviors, we need to consider the fact that LLMs sometimes show qualitative changes. The natural world also presents such changes called phase transitions, which are defined by singular, divergent statistical quantities. Therefore, an intriguing question is whether qualitative changes in LLMs are phase transitions. In this work, we have conducted extensive analysis on texts generated by LLMs and suggested that a phase transition occurs in LLMs when varying the temperature parameter. Specifically, statistical quantities have divergent properties just at the point between the low-temperature regime, where LLMs generate sentences with clear repetitive structures, and the high-temperature regime, where generated sentences are often incomprehensible. In addition, critical behaviors near the phase transition point, such as a power-law decay of correlation and slow convergence toward the stationary state, are similar to those in natural languages. Our results suggest a meaningful analogy between LLMs and natural phenomena.

Decision: reject
Reviewer scores: 3, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730398815688

thread_type:
official_review

<thread_object>
creator: reviewer_nkyn

<content>
summary:
This paper identifies a phase transition around temperature 1.0 when increasing the temperature parameter for LLM text generation. They focus on POS tags of generated sequences, finding a phase change in (1) the correlation of POS tags at two time points (for some time change $\Delta t$) and (2) periodic behavior. Natural text seems to correspond to roughly temperature 1.0.

strengths:
1. Drawing a connection between statistical physics and LLM text generation is very interesting.
2. The paper identifies a phase transition for multiple metrics (correlation between two tags, power spectra).

weaknesses:
1. The results for periodic behavior are nicely quantified, but slightly unsurprising (degeneration into repeated sequences for low temperature, but not high temperature; e.g. https://arxiv.org/pdf/1904.09751). Looking at the specific types of repetitive loops that the model falls into could be more informative.
2. The result that GPT-2 is closest to natural language text at T=1.0 is a bit unsurprising given that this leaves the probabilities unchanged from how they were trained. For example, we would likely expect lowest perplexity over a large corpus for T=1.0, because the model is trained to minimize loss (log-perplexity) for T=1.0.

questions:
1. What is the motivation behind the POS tag correlation on p. 5? It's not necessarily intuitive why a proper noun appearing at time t should be highly correlated with a proper noun appearing at time $t+\Delta t$ for large $\Delta t$ (e.g. 100 tokens later; Figure 2). What do these correlations tell us about the actual text that gets generated?
2. Consider including the word temperature in the title; phase changes often refer to transitions over the course of pretraining (https://arxiv.org/abs/2209.11895) rather than varying the temperature parameter, so it would be helpful to clarify that this paper focuses on varying temperature.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730708317437

thread_type:
official_review

<thread_object>
creator: reviewer_hr6f

<content>
summary:
This paper investigates whether phase transitions occur in Large Language Models (LLMs). The authors conduct a statistical analysis of sequences generated by GPT-2 at various temperatures and found the phase transition occurs.

strengths:
1. The project fosters the understanding of LLMs from an interesting angle of phase transition.
2. The experiments are well designed and cleared documented.

weaknesses:
1. While LLMs develop fast over the past few years, this work tests on GPT-2, instead of newer generations of models.

questions:
1. In what ways will the understanding of phase transition impact future LLM works?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730740899143

thread_type:
official_review

<thread_object>
creator: reviewer_7ukk

<content>
summary:
This paper argues that LM generations undergo a phase transition at temperature $T = 1$. Specifically, they generate texts from GPT-2 using different temperatures (conditioned on only the BOS token), and map each text into a sequence of POS tags (though the focus only on PROPN), and study three features of the POS tags.

1. **Correlation between two tags:** The authors study whether PROPN tags are correlated at particular intervals in the text. They find the integrated correlation $\\tau$ between PROPN and PROPN saturates to a finite value for $T>1.0$, which means correlation decays faster than critical decay, whereas for $T<1.0$, $\\tau$ diverges, which means the correlation converges to a finite value. In other words, sequences with $T<1$ have long-range correlation between POS tags, and sequences above $T>1$ converge to zero correlation.
2. **Power spectra of POS sequences**: The authors study whether there is periodic appearance of the PROPN tag via the power spectrum. Again, there appears to be long-range order at $T<1.0$, but at $T>1.0$ the spectrum is featureless.
3. **Time evolution of POS sequences:** The authors study the probability of the PROPN tag at time $t$. They find that this probability converges quickly to a limiting value for $T<1.0$ and $T>1.0$, but converges much more slowly for values close to $T\approx 1.0$.

strengths:
This paper investigates whether LLMs go through phase transition with the temperature parameter. This (I believe) contrasts with much of the literature, which is about the possibility of a phase transition with the size of the model. The statistics that the authors introduce may have value for future work that aims to study structural features of LLM generations.

The writing and organization are clear, and the figures are clearly explained.

weaknesses:
- There are obvious empirical concerns: namely, in the main paper, the authors only study GPT-2 small, and all of the analysis is specifically about the structure of where the proper noun PROPN tag occurs in generated text. This must be quite narrow, and it would be more convincing if the authors could summarize the results for other models and other POS tags in the main paper (they are referenced as being in the Appendix, which I did not read).
- This is very far from my area of expertise, but I am not convinced that there is truly *singularity* at $T=1.0$. In their analysis in §4.2 (power spectra) and §4.3 (time evolution), the model seems to demonstrate continuous behavior around $T=1.0$. In particular, I am looking at Figure 3 (A), where the peak in integrated correlation actually occurs at $T=1.1$, and Figure 5, where the values neighboring $T=1.0$ (from $0.9$ to $1.2$) all behave qualitatively similarly to $T=1.0$. This makes me worried that the paper is just showing that generations have structurally different qualities for small $T$ and large $T$, which is somewhat obviously true because $T$ directly controls the peakiness of the LM prediction, such that low $T$ will be highly redundant and high $T$ will be quite degenerate (esp. for GPT-2 era models).
- It is not clear to me how important the authors' findings are for the community. They mention their results suggest "a meaningful analogy between LLMs and natural phenomena," but the analogy was not clear to me from reading the paper. Are there practical insights about what LMs are learning or how we should build future LMs?
- There seems to be [related work](https://community.wolfram.com/groups/-/m/t/2958851) from 2023 that also studies phase transition at $T=1.0$, which the authors mention, and the distinction in their contributions is not clear to me. This may lessen the novelty of their work.

questions:
- Can you help me interpret Equation (1)? My understanding is that $a$ and $b$ are POS tags, and $y_t$ is the particular POS tag at position $t$. Therefore, this is measuring whether there is a predictable relationship that when $y_t=a$, then $y_{t+\\Delta t}=b$. What is the purpose of the second term? Does it serve as some kind of normalization, to account for the base frequency of $y_t=a$ and $y_{t+\\Delta t}=b$? Why are there $18 \\times 17/2$ possible pairs $a$ and $b$? I would have assumed $18^2$?
- L. 240: "The plateau value increases with the position $t$ of the former tag." Is this true? It looks like the green line ($t=15$) is ordered between purple ($t=0$) and blue ($t=3$).
- L. 244: What is "the prefactor" of the decay? I am again failing to see a reliable trend for $t$, since it seems like $t=0$, $t=63$, and $t=225$ all have extreme error bars, while $t=3$ and $t=15$ do not.
- L. 301: What does it mean for a spectrum to be featureless?
- For the power spectra analysis, I can see how the plots at different temperatures look different, but why is there *singular* behavior at $T=1.0$?
- In Figure 5, doesn't $v(t)$ immediately reach its limiting value for $T=1.0$, contrary to what's written in the paper?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730966399088

thread_type:
official_review

<thread_object>
creator: reviewer_lsmv

<content>
summary:
This paper proposes that qualitative changes in language model outputs can be analyzed as phase transitions—an analogy drawn from statistical physics. The authors empirically explore the shifts in characteristics of model-generated natural language that appear with sampling temperature scaling, which they posit may resemble phase transitions observed in physical systems. They observe a shift from what they characterize as an ordered to a disordered phases (i.e., the temporal structures in the different sets of sequences are statistically distinctive) as a result of changes in temperature. They conclude that LMs exhibit phase transition-like behavior around critical parameter values. They compare several quantitive attributes of the sequences generated at different temperatures to those of human generated text and find notable differences.

strengths:
* The motivation behind the paper is nice: the application of a well-studied concept from physics can perhaps allow us to use knowledge about/properties of that concept to better understand natural language and language models
* The work provides a comparison of quantitative aspects of human- and machine-generated language, an approach that is more objective than the qualitative comparisons that are often done and thus perhaps a better ground from which to draw conclusions 
* The finding that the statistical properties of language at different sampling temperatures is interesting (anecdotally, at least)

weaknesses:
* The paper is generally difficult to follow:
    * There is confusing terminology that isnt defined/contextualized before it is used, e.g., “long-range correlation” in the introduction). This will likely confuse most readers (myself included)
    * The implications of the observed critical properties are incredibly unclear. See subsequent questions for the parts that felt particularly unclear to me, although this is not comprehensive. As such, it is difficult to draw meaningful conclusions from the paper
* The work only looks at changes in model behavior as a result of changing the temperature parameter. This is a very narrow exploration and also somewhat contrived since it's an external change to the model that doesn’t have any implications about what happens when internal model attributes vary
* Only rather small models are used (on the scale of 100M parameters) and so results are not relevant for modern LLMs

questions:
* In the introduction, the motivation that “the decay of correlation in natural languages follows a power law” is used. This is an incredibly vague statement. What are some concrete correlations in natural language that this applies to?
* Terminological clarifications: 
    * What are “changes with a singularity”, both in a physics and POS sequence context?
    * What is a concrete example of a ‘critical phenomenon’ in a physical system and what might be an example of one in natural language? 
    * What does it mean that “the high-temperature phase is not simply disordered”? What does this look like qualitatively in language? Does it tell us something about human-generated natural language? And what does it mean about a language model?
* What are the implications of causality being broken for the conclusions that can be drawn? It seems as though this might invalidate results. I don’t see why the effect “shouldn’t matter”. I also don’t think it’s accurate to use the example of next characters as a situation where there are not dependency relationships. Anecdotally: language modeling can also be successful at the character-level. Perhaps I am not understanding something.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732002843001

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers for their careful reading of our manuscript and their valuable feedback. In response to the comments, we have revised our manuscript to address specific concerns. We have removed potentially confusing terminology, added explanations to make it clearer for a broader range of readers, and highlighted the significance of our research. Furthermore, we modified the title to include _temperature-induced_ to clarify our focus. If there are any other concerns, we would be happy to further revise our manuscript. 

In the original manuscript, we observed the phase transition in models of sizes 70M, 124M, and 361M. To further support our claims, we have performed additional numerical analyses on Pythia of size 1B, showing that this model also exhibits a similar phase transition (see App. J for the numerical results). These indicate that the phase transition occurs in even larger models.

Finally, we would like to highlight the following key points:
- Based on the formal definition of phase transitions, we present convincing evidence to support the presence of a phase transition.
- We expect that the phase transition can be observed robustly, as discussed in Sec. 6. Indeed, we have observed phase transitions in other models with different mappings, as shown in Apps. I and J.
- Our finding of the phase transition allows us to approach the understanding of LLMs from the physical point of view. For example,
    - The power-law decay in correlation is necessary for texts to be natural. Studying phase transitions and critical phenomena in LLMs would provide a clear understanding of how and what characteristics of natural languages LLMs recognize.
    - Existing statistical-mechanical models do not have phase transitions with the qualitatively same properties observed in this study. This highlights the nature of LLMs and natural languages beyond typical physical systems.
    - The phase transition in every LLM should belong to a _universality class_, as discussed in Sec. 6. Dimensionality and symmetry of the system and the emerging order at the phase transition control its universality class. Thus, identifying the universality classes of LLMs directly classifies the models and should tell us how much a given LLM has mimicked natural language.
- Our analysis can be extended to examine changes induced by not only the temperature parameter but also any other parameters.

In summary, we have proposed a novel framework for characterizing and analyzing the behavior of LLMs. We believe that it will be crucial in advancing future practical developments in LLMs.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper looks at the range of temperature values at which LLM-generated text mimics the statistical properties of natural language, framing it through the lens of phase transitions in physical systems. The authors hypothesize the existence of a singularity point on the temperature scale where LLM-generated text aligns with natural language properties, with deviations occurring at temperatures above or below this point.

While the perspective is interesting, the experimental setup and evidence supporting the hypothesis are limited. The primary finding is that LLM-generated text preserves the statistical properties of natural text at a temperature of 1. This is not really surprising and also doesn't lead to a deeper understanding of LLMs through the concept of phase transitions. This concern was shared by the reviewers.

The statistical properties examined in the study include long-range correlations of POS tags (specifically, PROPN) and the power spectra of POS tags. However, the rationale behind selecting these properties in drawing parallels with phase transitions is insufficiently justified. The reviewers also raised concerns about older and smaller LLMs being included in the experiments (models: GPT-2, pythia etc., size: <1B), essentially, not providing a complete view of the current state of LLMs. Lastly, the clarity of the writing was a recurring concern among reviewers, making the paper difficult to follow.

Overall, the study is further by its reliance on older and smaller LLMs for experiments, its narrow focus on two statistical properties, and the lack of novelty in its findings. After the author-reviewer discussion, two reviewers expressed weak acceptance (scores of 6 and 6), while the remaining two reviewers recommended rejection (scores of 3 and 3). Based on these considerations, I lean towards rejecting the paper.