Phase Transitions in the Output Distribution of Large Language Models

## Abstract

Abstract In a physical system, changing parameters such as temperature can induce a phase transition: an abrupt change from one state of matter to another. Analogous phenomena have recently been observed in large language models. Typically, the task of identifying phase transitions requires human analysis and some prior understanding of the system to narrow down which low-dimensional properties to monitor and analyze. Statistical methods for the automated detection of phase transitions from data have recently been proposed within the physics community. These methods are largely system agnostic and, as shown here, can be adapted to study the behavior of large language models. In particular, we quantify distributional changes in the generated output via statistical distances, which can be efficiently estimated with access to the probability distribution over next-tokens. This versatile approach is capable of discovering new phases of behavior and unexplored transitions – an ability that is particularly exciting in light of the rapid development of language models and their emergent capabilities.

## 1 Introduction

Colloquially, the term phase transition refers to a change among the basic phases of matter. For example, in response to changes in external conditions such as temperature or pressure, water can transition to a solid, liquid, or gaseous state. More broadly, in physics a phase transition refers to an abrupt change in the macroscopic behavior of a large-scale system of interacting constituents [, ]. Notable examples include transitions in the magnetic properties of materials [], transitions from a normal conducting state to a superconductor [], transitions in the entanglement properties of quantum circuit [], or the collective motion of active matter such as a flock of birds [].

In the context of artificial intelligence, “phase transition”-like phenomena have also been observed in the learning behavior of neural networks (NNs) [, , , , , , , , , , ]. For example, during training, AlphaZero [] underwent periods of rapid knowledge acquisition in which increasingly sophisticated chess openings were favored by the engine []. Large language models (LLMs) have been observed to make sudden improvements in their inductive abilities during training which is related to the formation of special circuitry (so-called induction heads) []. Similar abrupt improvements in specific capabilities, often referred to as breakthroughs, have been observed for a variety of different models and tasks [, , , , , , , , , , ]. Moreover, phenomena such as double descent [, ] or grokking [, , , , , ] are also reminiscent of phase transitions in physics.

The detection of phase transitions(^1^11In the following, we adopt a more general definition of a phase transition as a sudden shift in the qualitative behavior of a system as a function of a control parameter [, , ].) in deep learning systems may improve our understanding and eventually enable better model training. For example, an in-depth analysis of the grokking transition [, ] led to a way for accelerating generalization []. Moreover, it has been shown that models are highly sensitive to perturbations, such as data corruptions, at critical points [, ]. Being able to predict the behavior of models is also crucial for ensuring safe model deployment [] as well as for projecting the performance of future model versions and optimally allocating resources for their training [].

The characterization of phase transitions in physics is difficult because the state of the systems to be studied typically lives in a very high-dimensional space and is probabilistic in nature, meaning that for given values of the tuning parameters we can find the system in various states. Physicists solve this problem by finding a suitable set of a few low-dimensional quantities, called order parameters [], which capture the essence of each phase of the system. For example, even though water is a highly complex system, we can detect the liquid-gas transition by looking at the density which shows a sudden jump at the boiling point and, in this case, serves as an order parameter. However, finding such a suitable set of order parameters is “considered an art” [], as it requires a great deal of human intuition as well as prior understanding.

Faced with the task of characterizing phase transitions in learning systems based on large NNs, similar issues are encountered. NNs contain an enormous amount of trainable parameters and their state space, as characterized by their neural activations, is huge. This problem is exacerbated in generative models such as LLMs where also the output space is large, i.e., the high dimensionality cannot be foregone by treating the inside of the NN as a black box and focusing solely on its output characteristics. Understanding LLMs from first principles has been notoriously hard []. Theories capturing their microscopic and macroscopic behavior, for instance based on mechanistic interpretability [, , , , ] or neural scaling laws [, , , , , , , ], are still nascent. In particular, the definition of appropriate low-dimensional quantities that facilitate the detection of transitions has been done manually, for example through the extraction of appropriate circuitry [, , ]. Due to this human-in-the-loop, transitions can be easily missed [] or spuriously induced [].

In physics, these problems have been tackled using statistical methods for the detection of phase transitions from data, which requires minimal prior system knowledge and human input [, , , , , , , , , , , , , , , , , , , , , , ]. Inspired by this body of work, we here adapt such an approach for the automated detection of phase transitions in LLMs. The method is based on measuring changes in the distribution of the text output of LLMs via generic statistical distances belonging to the family of $f$-divergences, making it a versatile all-purpose tool for objectively and automatically mapping out phase diagrams of generative models. Such an approach has the potential to characterize unexplored phase transitions and potentially discover new phases of behaviors. This is crucial in light of the rapid development of LLMs [, , ] and their emergent capabilities [, , , , , , , , , , , , ].

As a demonstration, we characterize transitions occurring as a function of three different control parameters in Pythia [], Mistral (7B) [], and Llama3 (8B) [] language models: an integer occurring in the input prompt, the temperature hyperparameter for text generation, and the model’s training epoch.

#### Specific Findings.

We find that

- •
the instruction-tuned Llama and Mistral models seem to have the capability to order integers whereas all considered base models do not.
- •
changes in integer tokenization can be visible in the text output as sharp transitions.
- •
three distinct phases of behavior as a function of an LLM’s temperature can be mapped out: a deterministic “frozen” phase near zero temperature, an intermediate “coherent” phase, and a “disordered” phase at high temperatures.
- •
an LLM’s “heat capacity” with respect to the temperature can be negative, i.e., the LLM’s mean energy can decrease as its temperature is increased.
- •
rapid changes in the distribution of weights during training can coincide with transitions in the text output that are present across many prompts.
- •
different prompts result in different transition times during training, suggesting that distinct type of behavior can be learned rapidly at distinct times in training.

## 2 Methodology

### 2.1 Quantifying Dissimilarity between Distributions

In this work, we view phase transitions as rapid changes in the probability distribution $P(\cdot|T)$ governing the state of the system $\bm{x}\sim P(\cdot|T)$ as the control parameter $T$ is varied.(^2^22This definition encompasses phase transitions in physics, i.e., abrupt changes in the distribution governing large-scale systems of interacting constituents.)
That is, values of the parameter at which the distribution changes strongly are considered critical points where phase transitions occur. While it is possible to generalize our approach to distributions conditioned on multiple control parameters (see [, ]), for simplicity we consider the one-dimensional scenario in the following.

We quantify the rate of change using $f$-divergences [], as they have particularly nice properties, such as satisfying the data processing inequality. Given a convex function $f:\mathbb{R}_{\geq 0}\rightarrow\mathbb{R}$ with $f(1)=0$, the corresponding $f$-divergence is a statistical distance defined as

$$ $D_{f}[p,q]=\sum_{\bm{x}}q(\bm{x})f\left(\frac{p(\bm{x})}{q(\bm{x})}\right)\geq 0.$ (1) $$

Prominent examples of $f$-divergences include the Kullback-Leibler (KL) divergence, the Jensen-Shannon (JS) divergence, which corresponds to a symmetrized and smoothened version of the KL divergence, as well as the total variation (TV) distance. Ideally, we would also like the statistical distance we choose to be symmetric $D[p,q]=D[q,p]$. This condition is only satisfied by the TV distance and the JS divergence among the examples above.

Hence, in this work, we will focus on the TV distance

$$ $D_{\rm TV}[p,q]=\frac{1}{2}\sum_{\bm{x}}|p(\bm{x})-q(\bm{x})|$ (2) $$

corresponding to $f(x)=\frac{1}{2}|1-x|$, as well as the JS divergence

$$ $D_{\rm JS}[p,q]=\frac{1}{2}D_{\rm KL}\left[p,\frac{p+q}{2}\right]+\frac{1}{2}D _{\rm KL}\left[q,\frac{p+q}{2}\right]$ (3) $$

corresponding to $f(x)=\frac{1}{2}\left[x\log(\frac{2x}{1+x})+\log(\frac{2}{1+x})\right]$, where $D_{\rm KL}$ is the KL divergence.

The TV distance and the JS divergence have also had tremendous success in detecting phase transitions in physical systems without prior system knowledge under the name of “learning-by-confusion” [, , , , , , , , , , , , , , , , , , , , , ].(^3^33Note that both the TV distance and the JS divergence form lower bounds to the KL divergence and other $f$-divergence, such as the $\chi^{2}$ divergence: $D_{\rm JS}[p,q]\leq D_{\rm TV}[p,q]\leq\sqrt{D_{\rm KL}[p,q]}\leq\sqrt{D_{\chi ^{2}}[p,q]}$ []. In this sense, detecting a large dissimilarity in terms of the TV distance or the JS divergence also signals a large dissimilarity in other measures.)

### 2.2 Detecting Phase Transitions

Having defined appropriate notions of distance between probability distributions, we now describe their use to detect phase transitions: Consider a sampled set $\mathcal{T}$ of control parameter values $T$, forming a uniform one-dimensional grid. For each $T^{*}$ lying halfway in between grid points, we assess whether it is a critical point by computing a dissimilarity score
$D(T^{*})=D\left[P_{\mathrm{left}}(\cdot|T^{*}),P_{\mathrm{right}}(\cdot|T^{*})\right]$
between the distributions underlying the segments $\sigma_{\mathrm{left}}(T^{*})$ to the left and $\sigma_{\mathrm{right}}(T^{*})$ to the right of $T^{*}$. Denoting the cardinality as $|\cdot|$, we can write these probabilities as $P_{i}(\cdot|T^{*})=\frac{1}{|\sigma_{i}(T^{*})|}\sum_{T\in\sigma_{i}(T^{*})}P(
\cdot|T)$ for $i\in\{\mathrm{left},\mathrm{right}\}$. Critical points where phase transitions occur can then be identified as local maxima in $D$.

For the sake of simplicity, we proceed with segments of equal length for the rest of this article, and define the length $L=|\sigma_{\rm left}|=|\sigma_{\rm right}|$ as the number of parameter values $T$ to the left or right of $T^{*}$ that characterize the segment. We are free to adjust it according to the problem, as $L$ sets a natural length scale on which changes in the distributions are assessed. Examples will be discussed in Sec. 3. In particular for $L=1$ and neighboring parameter points separated by $\delta T$, $D_{f}(T^{*})=\tfrac{1}{2}{f^{\prime\prime}(1)}\mathcal{F}(T^{*})\delta T^{2}+
\mathcal{O}(\delta T^{3})$ where $\mathcal{F}$ is the Fisher information []. That is, local changes in a distribution as measured by any $f$-divergence reduce to the Fisher information in the limit $\delta T\to 0$.
Having the Fisher information as a limiting case is a desirable property: It is a well-known, generic statistical measure for quantifying how sensitive probability distributions are to changes in their parameters and its behavior is well-understood when used to detect phase transitions in physical systems [, , , ].

### 2.3 Application to Language Models and Numerical Implementation

In the case of language models, $\bm{x}$ is the sampled text and $T$ is any variable that influences the sampling probability. Because of the autoregressive structure of language models, we can efficiently sample text $\bm{x}$ for a given prompt and evaluate its probability $P(\bm{x}|T)$. Thus, we can obtain an unbiased estimate $\hat{D}(T^{*})$ of $D(T^{*})$ by replacing expected values with sample means where samples correspond to text generated with language models conditioned on different parameter settings $T$, see Appendix A for details on implementation.

For numerical stability and efficient sampling, we express our dissimilarity measures as parameterized by a function $g$ acting on the probability $P(\sigma_{i}|\bm{x})=\frac{P_{i}(\bm{x})}{P_{\mathrm{left}}(\bm{x})+P_{\mathrm
{right}}(\bm{x})}$ for $\bm{x}$ to stem from segment $\sigma_{i}$. Specifically, we consider

$$ $D_{g}=\frac{1}{2L}\sum_{i\in\{{\rm left},{\rm right}\}}\sum_{T\in\sigma_{i}} \mathbb{E}_{\bm{x}\sim P(\cdot|T)}\biggl{[}g\left[P(\sigma_{i}|\bm{x})\right] \biggr{]}.$ (4) $$

These $g$-dissimilarities and the $f$-divergences [Eq. (1)] defined above correspond to each other in the following sense: any $g$-dissimilarity $D_{g}$ can be rewritten in the form of an $f$-divergence $D_{f}[P_{\rm left},P_{\rm right}]$ with

$$ $f(x)=\frac{x}{2}\cdot g\left(\frac{x}{1+x}\right)+\frac{1}{2}\cdot g\left( \frac{1}{1+x}\right),$ (5) $$

see Appendix B for the derivation and further discussion. In particular, for the choice $g(x)=\log(x)+\log(2)$, $D_{g}$ corresponds to the JS divergence [Eq. (3)]. For $g(x)=1-2\min\{x,1-x\}$, $D_{g}$ corresponds to the TV distance [Eq. (2)].

A natural choice for $g$ is any linear function in $x$. In particular, setting $g(x)=2x-1$ results in a dissimilarity measure that quantitifies the ability of an optimal classifier to tell whether a sample $\bm{x}$ has been drawn in the left or right sector. This measure is 0 if the two distributions are completely indistinguishable and 1 if the two distributions are perfectly distinguishable. Moreover, $g(x)=2x-1$ has the property of being bounded between 1 and -1, where the edge values are attained for the certain predictions 0 and 1, and the value 0 corresponds to uncertain predictions at 0.5. This results in a low variance and favorable convergence properties for $\hat{D}_{g(x)=2x-1}$, which we will refer to as linear dissimilarity in what follows. This quantity is a valid $f$-divergence and reduces to the Fisher information in lowest non-vanishing order(^4^44In fact, any $g$-dissimilarity with $g(1/2)=0$ and a twice-differentiable $g$-function can be shown to be proportional to the Fisher information in lowest order.), as shown in Appendix B.

### 2.4 Utilized Large Language Models

In this work, we study transitions in models of the Pythia, Mistral, and Llama family.

#### Pythia

Pythia [] is suite of 16 LLMs released in 2023 that were trained on public data in the same reproducible manner ranging from 70 million (M) to 12 billion (B) parameters in size. We consider every second model, i.e. the models with 70M, 410M, 1.4B, and 6.9B parameters.

#### Mistral

From the Mistral family, we consider the base model Mistral-7B-v0.1 with 7.3B parameters and the corresponding fine-tuned Mistral-7B-Instruct model [] released in 2023.

#### Llama

Llama 3 [] from Meta AI was released in 2024. We consider both the Llama-3 8B parameter base model and NVIDIA’s chat-tuned Llama3-ChatQA-1.5-8B []. For the chat model we use accordingly formatted inputs.

## 3 Results

In the following, we will explore all three fundamental ways in which a parameter $T$ may influence the output distribution of a language model: $(i)$ As a variable within the input prompt, we scan through integers injected to the prompt in Sec. 3.1. $(ii)$ As a hyperparameter controlling how a trained language model is applied, we vary the temperature in Sec. 3.2. $(iii)$ As a training hyperparameter of the language model, we vary the number of training epochs in Sec. 3.3.

### 3.1 Transitions as a Function of a Variable in the Prompt

Figure: Figure 1: Mistral model applied to the integer ordering prompt. (a) Different $g$-dissimilarities with $L=3$. (b) Linear dissimilarity for different $L$-values. [Number of text outputs generated per parameter value $T$: 10280. Number of generated output tokens: 10. Error bars indicate standard error of the mean over 4 batches, each with batch size 2056.]
Refer to caption: x1.png

As an introduction, we start with the simplest case: The parameter $T$ to be varied is a particular part of the prompt, and all parameters of the language model itself are fixed. As a first such prompt, consider “$T$ is larger than 42. True or False?” with an integer $T$ as the control parameter. An LLM that understands the order of integers should output very different answers for $T<42$ versus $T>42$, i.e., its distribution over outputs should change drastically around $T=42$. Thus, in such a case we expect the dissimilarities to show a clear peak around $T=42$.

Figure 1(a) shows dissimilarities based on various $g$-functions for the Mistral-7B-Instruct model []. All dissimilarities show a clear peak around $T=42$, whereas they are relatively flat otherwise. This is a clear example of an abrupt transition between two distinct phases of behaviors of an LLM as a function of a tunable parameter. As compared to the linear dissimilarity, the logarithm-based JS divergence is arguably a bit sharper in that it decays more rapidly to baseline 0. The TV distance’s peak is the broadest due to the $\min$ function appearing in its $g$-function. In the following, we will focus on the linear dissimilarity as a compromise between sensitivity and numerical stability.

The transition is also clearly visible using different $L$ settings, see Fig. 1(b). Smaller $L$ values are closer to the Fisher information limit, while larger values generally lead to higher distinguishability of distributions and therefore larger peaks at transition points. As we will see in more detail in Sec. 3.3, they can also be less susceptible to outliers due to the averaging over several parameter points.

Figure: Figure 2: Benchmarking of various models using the linear dissimilarity with $L=3$. (a) Test of ability to compare integers in value. (b) Bare integers as prompt reveals transition in tokenizer encoding. [Same numerical settings as in Fig 1.]
Refer to caption: x3.png

Interestingly, when performing the same analysis on base models such as the Llama3-8B and Mistral base models, as well as Pythia models [] of various sizes, the resulting linear dissimilarity is flat, signaling the absence of any transition [see Fig. 2(a)]. In contrast to Mistral-7B-Instruct and NVIDIA’s chat-tuned Llama3-8B, these models do not show a clear peak around $T=42$.

A transition of a different origin can be observed in Fig. 2(b), where the LLMs are probed using the prompt “$T$” with $T$ again being an integer.
Interestingly, all Pythia models show a peak between $T=2020$ and $T=2021$. This behavioral transition may originate from a transition in the tokenizers of these models, which encode numbers in a range below $T=2021$ with a single token and numbers in a range at and above $T=2021$ with two. This explanation is corroborated by the absence of the transition around $T=2021$ for the Llama and Mistral models, whose tokenizers translate a number into tokens following rules that are independent of the number’s frequency.

The Mistral models and the base Llama3-8B model show a smaller peak around $T=2023/2024$. Both models have only encountered training data from before and around that time given their release date in 2023/2024, which may explain the peak. This transition is absent in the Pythia models.

### 3.2 Transitions as a Function of the Model’s Temperature

Next, we consider transitions as a function of the temperature hyperparameter $T$ controlling how the logits $\bm{z}$ are converted to probabilities

$$ $p_{i}=\frac{e^{z_{i}/T}}{\sum_{j}e^{z_{j}/T}}$ (6) $$

for next-token prediction where the sum runs over all possible tokens. Per construction, at $T=1$ language models predict probabilities $p_{i}$ to approximate the distribution to be learned. In the limit $T\rightarrow 0$, the model deterministically picks the most likely next token in each step. For $T\rightarrow\infty$ the model samples the next token uniformly.

This scenario somewhat resembles a system of a one-dimensional lattice of spins that are coupled via long-range interactions, i.e., the one-dimensional Ising model [, ], which has an order-disorder phase transition. In our case, the tokens take the role of the spins, and the coupling is mediated via the transformer’s attention mechanism.

In Fig. 3, the dissimilarity shows two distinct peaks corresponding to two transition points: one at a very low temperature $T_{1}^{*}\approx 0.02$ and one at an intermediate temperature $T_{2}^{*}\approx 0.5$. Intuitively, these two points mark transitions between three distinct phases of behavior: “frozen” at low temperatures $T<T_{1}^{*}$, “unfrozen and sensible” at intermediate temperatures $T_{1}^{*}<T<T_{2}^{*}$, and “random” at high temperatures $T_{2}^{*}<T$. The transition at low temperatures has recently been investigated in Ref. [] for GPT-2 using physics-inspired quantities. Moreover, they speculated on the existence of a phase transition at higher temperatures.

Figure: Figure 3: Temperature transitions of Pythia 70M model in response to the prompt “There’s measuring the drapes, and then there’s measuring the drapes on a house you haven’t bought, a” – an excerpt from OpenWebText []. Linear dissimilarity measure ($L=5$) is shown in black. Heat capacity is shown in red. Dashed lines indicate local maxima, i.e., predicted critical points. Shaded regions indicate the error bands. (a) Temperature range $[10^{-4},2]$. (b) Zoomed-in range $[10^{-4},0.2]$ near $T=0$. [Number of text outputs generated per parameter value $T$: 20480. Number of generated output tokens: 10. Error bars indicate standard error of the mean over 4 batches, each with batch size 5120.]
Refer to caption: x5.png

We perform an analysis independent of the dissimilarity-based indicators by taking inspiration from statistical mechanics, where the state of thermal systems is governed by the Boltzmann distribution (see Appendix C for details). We view the LLM as such a thermal system at varying temperature where the negative logarithmic probability at $T=1$, $-\log P(\bm{x}|T=1)$, takes on the role of the energy $E$ of a given text output $\bm{x}$. In physical systems governed by Boltzmann distributions, thermal phase transitions can be detected as peaks in the heat capacity $C(T)=\partial\mathbb{E}_{\bm{x}\sim P(\cdot|T)}\left[E(\bm{x})\right]/\partial
T$, i.e., by looking at the temperature derivative of the mean total energy [].

Figure 3 shows that the locations of peaks (i.e., dips) in these quantities are close to the critical points highlighted by our method. Note that in the LLM case, the text outputs are not truly sampled from a Boltzmann distribution governed by the total energy. Instead, each individual token is drawn from a Boltzmann distribution for its individual energy conditioned on the previous tokens only. This procedure corresponds to a greedy sampling strategy. The resulting sampling mismatch can lead to the counterintuitive phenomenon of the mean energy of the system increasing with decreasing temperature corresponding to a negative “heat capacity”, cf. Fig. 3(b).

The intermediate temperature transition at $T_{2}^{*}$ may be reminiscent of the Schottky anomaly [] occurring in systems with a finite number of energy levels. As such, this phenomenon is perhaps better described as a crossover rather than a phase transition in the Ehrenfest sense. In particular, we also observed such a transition for a basic language model that samples words according to their overall frequency without taking into account any word-to-word interaction.

In Fig. 3 we have investigated the output distributions corresponding to a specific prompt. While we find that the temperature behavior is strongly dependent on the prompt, there seems to be a trend: many distinct prompts lead to a transition at $T\approx 1$ (i.e., on the order of the natural temperature scale), at $T\ll 1$, or both.

### 3.3 Transitions as a Function of the Training Epoch

Finally, we search for transitions as a function of the training epoch, i.e., we compare the output distributions of models at different stages during training and see whether there are certain epochs at which these statistics change drastically. Such temporal analyses are rare given that they require access to models at checkpoints during training [, , ]. Here, we analyze the Pythia suite of models for which such checkpoints are publicly available.

Ref. [] analyzed the weight distribution of the Pythia models, and similar weight-based analyses of other NNs during training have also been performed in previous works [, , ]. In order to study the previously observed transitions [],
we analyze changes in the weight distributions in the same manner as for the output distributions (see Sec. 2), i.e., to characterize phase transitions using dissimilarities. The lists of model weights are converted to distributions via histogram binning (10000 bins for the range -3 to 3).

The results for $L=6$ are shown in Fig. 4(a) as colored lines, each corresponding to the distribution of the weights of a particular QKV layer. Different layers show transitions at roughly 20K (layer 5), 40K (layers 3), 50K (layer 4), and 80K (layer 4) epochs. We also observe a large peak around epoch 0, i.e., at the start of the training, highlighting that the LLM learns most rapidly at the beginning stages. In the long run, the dissimilarity curves approach 0, signaling that overall the weight distributions become less and less distinguishable.

Figure: Figure 4: Linear dissimilarity by epoch, with checkpoints taken every 1000 epochs. (a) Computed at $L=6$ for both weights and responses to 20 random prompts from OpenWebText (gray) and 7 short prompts (black) shown in panel (b). (b) Computed at $L=1$ for several prompts. For reference, the mean linear dissimilarity over short prompts and OpenWebText prompts with $L=1$ is also shown. [Number of text outputs generated per parameter value $T$ and prompt: 1536. Number of generated output tokens: 10. Error bars indicate the standard error of the mean over all corresponding prompts. Error bars for the individual prompts in panel (b) are almost negligible and thus omitted to avoid visual clutter.]
Refer to caption: x7.png

Complementarily, in the same plot, we show dissimilarities derived from the LLM output distributions. The grey line corresponds to an average of the dissimilarities obtained by using entries from OpenWebText [] (which serves as a proxy for the Pythia training dataset) as prompts. The black line corresponds to the average of results obtained from a selection of single-token prompts [see also panel (b)]. Both dissimilarity curves show a peak around epoch 0 as well as a peak around 80K epochs that is potentially related to the rapid change of layer 4 around the same time.

Figure 4(b) shows dissimilarities as a function of the training epoch for models queried with short, generic prompts (“ ”, “0”, “I”, “?”, “1”, “You”, and “!”) at $L=1$. These short prompts were selected to be as general as possible and the associated output distributions seem more sensitive as compared to the long examples from OpenWebText: their mean dissimilarity shows clear peaks near epochs 20K, 40K, and 80K. These correspond to outliers where the output distribution changes severely only at a single point and returns back (close) to its original behavior immediately after. As such, these peaks do not mark transitions between two macroscopic phases of behavior. We further verified this outlier behavior by inspecting the dissimilarity between the points directly to the left and right of the potential outlier. It remains an open question if these outliers are linked to the transitions observed in the layer weights shown in panel (a).

The larger $L$ value used in panel (a) averages out the signal stemming from these outliers. Such a reduced susceptibility to outliers can be an advantage of using $L\gg 1$ when searching for macroscopic transitions in particular.

Some peak locations in the dissimilarity curves are prompt-dependent, indicating that learning progresses differently for different types of behavior.
Here we have used rather generic prompts, resulting in an analysis of the LLM’s general behavior during training. However, in principle, conditioning on the prompt allows one to analyze whether and when specific knowledge emerges [, ]. As an outlook, one can imagine automatically monitoring changes across a multitude of prompts on different topics and testing different abilities at scale, without the need to design individual metrics for each prompt.

## 4 Related Works

Before concluding, let us discuss how our method relates to other approaches for studying transitions in LLMs.

Generic performance-based analysis. Many previous works found transitions in LLM behavior by locating sharp changes in generic performance measures, such as sudden drops in training loss [, ]. While this may capture transitions in the overall behavior, such an approach cannot resolve transitions in specific LLM behavior. In particular, it may miss algorithmic transitions where the same performance is reached but by different means [].

Prompt-specific success metrics. Other works have found transitions by looking at success metrics tailored toward specific prompts [, , , , , , ]. Recalling the example studied in Sec. 3.1, this would correspond to assigning a score of 1 if the LLM provided the correct answer to the question $x<42$ and 0 otherwise. Similarly, one could compute such a score in a temporal analysis (Sec. 3.3) or for detecting transitions as a function of another hyperparameter (Sec. 3.2). A downside of this approach is that it is restricted to prompts that allow for a clear score to be assigned. In particular, choosing an appropriate scoring function may require lots of human engineering. Moreover, discontinuous metrics can artificially induce transitions where the underlying behavior varies smoothly []. Similarly, they may miss transitions where the same performance is reached but by different means [].

Measures based on model internals. The aforementioned approaches are based on the model output. Many works have also detected transitions based on changes in the internal structure of models, such as its trainable parameters [, ] (similar to the weight-based analysis we have performed in Sec. 3.3). However, access to model internals may not always be available. Moreover, the design of measures that capture specific transitions in behavior requires lots of human input [, , ], e.g., using insights from the field of mechanistic interpretability.

## 5 Conclusion and Outlook

We have proposed a method for automating the detection of phase transitions in LLMs, and demonstrated that it successfully reveals a variety of transitions. Leveraging access to the LLMs’ next-token probability distributions, the proposed dissimilarity measures can efficiently quantify distribution shifts without fine-tuning or adaption to the specific scenario at hand. Because the method is solely based on analyzing a model’s output distribution and access to the model weights is not required, it enables black-box interpretability studies.

The proposed method is not only applicable to language models, but can be straightforwardly adapted to any generative model with an explicit, tractable density [, ]. If one can draw samples from the output distribution but does not have explicit access to the underlying probabilities, then the dissimilarity measures can still be approximated using NN-based classifiers [, ] tailored toward the particular data type, such as natural language.

#### Limitations.

Future large-scale investigations are needed to fully understand how the uncovered transitions depend on variables such as the specific prompt, the number of generated output tokens, or the selected model. In particular, due to computational resource constraints, the size of the studied language models has been limited.

#### Broader Impact.

Our method has the potential to enhance the development of future AI systems due to an improved understanding of their behavior. The dual-use nature of such systems carries inherent risks, which requires one to proceed with caution and implement mechanisms to ensure they are used safely and ethically.

## Acknowledgments and Disclosure of Funding

We thank Christoph Bruder for stimulating discussions and helpful suggestions on the manuscript. J.A. and N.L. acknowledge financial support from the Swiss National Science Foundation individual grant (grant no. 200020 200481). This material is based upon work supported by the National Science Foundation under grant no. OAC-1835443, grant no. OAC-2103804, and grant no. DMS-2325184. The authors acknowledge the MIT SuperCloud and Lincoln Laboratory Supercomputing Center for providing HPC resources that have contributed to the research results reported within this paper.

## Appendix A Implementation details

#### Estimating dissimilarity measures.

Our method for detecting phase transitions is based on estimating $D_{g}$ across the entire parameter range. Starting with a fixed set of points on a uniform grid $\mathcal{T}$, let us denote the set of in-between points at least $L$ points away from the border of the range as $\bar{\mathcal{T}}$ (note that $|\bar{\mathcal{T}}|=|\mathcal{T}|-2L$). For each trial point $T^{*}\in\bar{\mathcal{T}}$, we obtain an unbiased estimate $\hat{D}_{g}=(\hat{J}_{\rm left}+\hat{J}_{\rm right})/2$, where

$$ $\hat{J}_{i}=\frac{1}{|\sigma_{i}|}\sum_{T\in\sigma_{i}}\frac{1}{\mathcal{D}(T) }\sum_{\bm{x}\in\mathcal{D}(T)}g\left[P(\sigma_{i}|\bm{x})\right].$ (A1) $$

Recall that $\sigma_{\rm left}$ and $\sigma_{\rm right}$ denote the $L$ closest points to the left or right of the trial point. Here, $\mathcal{D}(T)$ denotes a set of output texts $\bm{x}$ generated via the LLM at point $T\in\mathcal{T}$. In this work, we choose the number of generated text samples to be the same for all $T\in\mathcal{T}$, i.e., $|\mathcal{D}(T)|=|\mathcal{D}|$.

We perform the computation of $\hat{D}_{g}(T^{*})\;\forall\,T^{*}\in\bar{\mathcal{T}}$ in two stages. In a first stage, we go through each grid point $T\in\mathcal{T}$ and generate text outputs that are $N_{\rm tokens}$ in length via the LLM. The associated computation time scales as $|\mathcal{T}|\cdot N_{\rm tokens}\cdot t_{\rm LLM,eval}(|\mathcal{D}|)$, where $t_{\rm LLM,eval}(|\mathcal{D}|)=\mathcal{O}(|\mathcal{D}|)$ corresponds to the time it takes the LLM to generate $|\mathcal{D}|$ different outputs (single token in length). In a second stage, for a given trial point $T^{*}\in\bar{\mathcal{T}}$, we evaluate the probability of each text output generated in its vicinity $\{\bm{x}\in\mathcal{D}(T)|T\in\sigma_{\rm left}(T^{*})\cup\sigma_{\rm right}(T
^{*})\}$ to come from the left or right segment. That is, we compute $P(\bm{\sigma}_{\rm left}|\bm{x})$ and $P(\bm{\sigma}_{\rm right}|\bm{x})$, i.e., a term in the sum of Eq. (A1). The computation time associated with the second stage scales as $|\bar{\mathcal{T}}|\cdot N_{\rm tokens}\cdot t_{\rm LLM,eval}(|\mathcal{D}|)
\cdot 2L$. Note that in practice, one can embarrasingly parallelize over the different grid and trial points. Moreover, one generates and evaluates text outputs batchwise.

#### Compute Resources.

For our computations, we used an NVIDIA RTX 3090 GPU as well as NVIDIA Tesla V100 (32 GB) and NVIDIA A100 (40GB) GPUs.

#### Code availability.

A Python implementation of our method is available at [github.com/llmtransitions/llmtransitions](https://github.com/llmtransitions/llmtransitions).

#### Utilized assets.

Our code is implemented in Python and internally uses NumPy \citeSIharris:2020, PyTorch \citeSIpytorch:2019, and transformers \citeSIwolf:2020. For the presented examples, we make use of the following additional packages: pandas \citeSImckinney:2010, SciPy \citeSIscipy:2020, Matplotlib \citeSIhunter:2007, and seaborn \citeSIwaskom:2021.

The [Pythia models](https://huggingface.co/collections/EleutherAI/pythia-scaling-suite-64fb5dfa8c21ebb3db7ad2e1), the Mistral models [Mistral-7B-Instruct model](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) and
[Mistral-7B-Base model](https://huggingface.co/mistralai/Mistral-7B-v0.1), are available under the Apache-2.0 license on Hugging Face. The [Meta-Llama-3-8B](https://huggingface.co/meta-llama/Meta-Llama-3-8B) model and NVIDIA’s [Llama3-ChatQA-1.5-8B](https://huggingface.co/nvidia/Llama3-ChatQA-1.5-8B) are available on Hugging Face under the Meta Llama 3 community license.

## Appendix B Theoretical background on g 𝑔 g italic_g -dissimilarities

#### Correspondence Between f 𝑓 f italic_f -divergences and g 𝑔 g italic_g -dissimilarities.

Let us establish a correspondence between $f$-divergences [Eq. (1)] and $g$-dissimilarities [Eq. (4)]. We can write any $g$-dissimilarity as

$$ $\displaystyle D_{g}$ $\displaystyle=\frac{1}{2}\left(\mathbb{E}_{\bm{x}\sim P_{\rm left}}\biggl{[}g \left[P(\sigma_{\rm left}|\bm{x})\right]\biggr{]}+\mathbb{E}_{\bm{x}\sim P_{ \rm right}}\biggl{[}g\left[P(\sigma_{\rm right}|\bm{x})\right]\biggr{]}\right)$ $\displaystyle=\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}\frac{1}{2}\frac{P _{\rm left}(\bm{x})}{P_{\rm right}(\bm{x})}g\left[P(\sigma_{\rm left}|\bm{x}) \right]\biggr{]}+\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}\frac{1}{2}g \left[P(\sigma_{\rm right}|\bm{x})\right]\biggr{]}$ $\displaystyle=\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}\frac{1}{2}\frac{P _{\rm left}(\bm{x})}{P_{\rm right}(\bm{x})}g\left[P(\sigma_{\rm left}|\bm{x}) \right]+\frac{1}{2}g\left[P(\sigma_{\rm right}|\bm{x})\right]\biggr{]}$ $\displaystyle=\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}\frac{1}{2}\frac{P _{\rm left}(\bm{x})}{P_{\rm right}(\bm{x})}g\left[\frac{P_{\rm left}(\bm{x})}{ P_{\mathrm{left}}(\bm{x})+P_{\mathrm{right}}(\bm{x})}\right]+\frac{1}{2}g\left [\frac{P_{\rm right}(\bm{x})}{P_{\mathrm{left}}(\bm{x})+P_{\mathrm{right}}(\bm {x})}\right]\biggr{]}$ $\displaystyle=\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}\frac{1}{2}\frac{P _{\rm left}(\bm{x})}{P_{\rm right}(\bm{x})}g\left[\frac{\frac{P_{\rm left}(\bm {x})}{P_{\mathrm{right}}(\bm{x})}}{\frac{P_{\mathrm{left}}(\bm{x})}{P_{\mathrm {right}}(\bm{x})}+1}\right]+\frac{1}{2}g\left[\frac{1}{\frac{P_{\mathrm{left}} (\bm{x})}{P_{\mathrm{right}}(\bm{x})}+1}\right]\biggr{]}.$ $$

Thus, any $g$-dissimilarity $D_{g}$ can be rewritten in the form of an $f$-divergence $D_{f}[P_{\rm left},P_{\rm right}]$ with

$$ $f(x)=\frac{x}{2}\cdot g\left(\frac{x}{1+x}\right)+\frac{1}{2}\cdot g\left( \frac{1}{1+x}\right).$ (B1) $$

Note, however, that not any choice of $g$-function will lead to a proper $f$-divergence in the sense that the resulting $f$-function may not be convex and $f(1)$ may not be zero (recall the definition of an $f$-divergence in Sec. 2.1).

#### JS divergence.

Using the correspondence above, we have that $D_{g(x)=\log(x)+\log(2)}$ is equivalent to an $f$-divergence $D_{f}[P_{\rm left},P_{\rm right}]$ with

$$ $f(x)=\frac{x}{2}\cdot\log\left(\frac{2x}{1+x}\right)+\frac{1}{2}\cdot\log\left (\frac{2}{1+x}\right)$ (B2) $$

which corresponds to the JS divergence, see Eq. (3).

#### TV distance.

Let us further prove that the $D_{g(x)=1-2\min\{x,1-x\}}$ corresponds to the TV distance $D_{\rm TV}[P_{\rm left},P_{\rm right}]$. We have

$$ $\displaystyle D_{g(x)=1-2\min\{x,1-x\}}=1$ $\displaystyle-\mathbb{E}_{\bm{x}\sim P_{\rm left}}\biggl{[}\min\{P(\sigma_{\rm left }|\bm{x}),P(\sigma_{\rm right}|\bm{x})\}\biggr{]}$ $\displaystyle-\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}\min\{P(\sigma_{ \rm left}|\bm{x}),P(\sigma_{\rm right}|\bm{x})\}\biggr{]}$ (B3) $$

Using the identity

$$ $\min\{P(\sigma_{\rm left}|\bm{x}),P(\sigma_{\rm right}|\bm{x})\}=\frac{1}{2} \left(1-\left|P(\sigma_{\rm left}|\bm{x})-P(\sigma_{\rm right}|\bm{x})\right| \right),$ $$

Eq. (B) can be rewritten as

$$ $\displaystyle D_{g(x)=1-2\min\{x,1-x\}}$ $\displaystyle=\frac{1}{2}\mathbb{E}_{\bm{x}\sim P_{\rm left}}\biggl{[}\left|P( \sigma_{\rm left}|\bm{x})-P(\sigma_{\rm right}|\bm{x})\right|\biggr{]}$ $\displaystyle\quad+\frac{1}{2}\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[} \left|P(\sigma_{\rm left}|\bm{x})-P(\sigma_{\rm right}|\bm{x})\right|\biggr{]}$ $\displaystyle=\frac{1}{2}\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}\left(1 +\frac{P_{\rm left}(\bm{x})}{P_{\rm right}(\bm{x})}\right)\left|P(\sigma_{\rm left }|\bm{x})-P(\sigma_{\rm right}|\bm{x})\right|\biggr{]}$ $\displaystyle=\frac{1}{2}\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}P( \sigma_{\rm right}|\bm{x})\left(1+\frac{P_{\rm left}(\bm{x})}{P_{\rm right}( \bm{x})}\right)\left|1-\frac{P(\sigma_{\rm left}|\bm{x})}{P(\sigma_{\rm right} |\bm{x})}\right|\biggr{]}.$ $$

Noting that $\frac{P(\sigma_{\rm left}|\bm{x})}{P(\sigma_{\rm right}|\bm{x})}=\frac{P_{\rm
left
}(\bm{x})}{P_{\rm right}(\bm{x})}$ and $P(\sigma_{\rm left}|\bm{x})+P(\sigma_{\rm right}|\bm{x})=1$, we finally obtain

$$ $D_{g(x)=1-2\min\{x,1-x\}}=\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}\frac{ 1}{2}\left|1-\frac{P_{\rm left}(\bm{x})}{P_{\rm right}(\bm{x})}\right|\biggr{]}$ $$

which corresponds to an $f$-divergence $D_{f}[P_{\rm left},P_{\rm right}]$
with $f(x)=\frac{1}{2}|1-x|$, i.e., the TV distance.

#### Freedom in Choice of g 𝑔 g italic_g -function.

Note that the choice of $g$-function leading to a particular $g$-dissimilarity is not unique. In particular, we have that $D_{\tilde{g}}=D_{g}$ for any $\tilde{g}(x)=g(x)+c(\frac{1}{x}-2)$ where $c\in\mathbb{R}$ is some constant:

$$ $\displaystyle D_{\tilde{g}}$ $\displaystyle=D_{g}+\frac{c}{2}\left(\mathbb{E}_{\bm{x}\sim P_{\rm left}} \biggl{[}\frac{1-P(\sigma_{\rm left}|\bm{x})}{P(\sigma_{\rm left}|\bm{x})}-1 \biggr{]}+\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}\frac{1-P(\sigma_{\rm right }|\bm{x})}{P(\sigma_{\rm right}|\bm{x})}-1\biggr{]}\right)$ $\displaystyle=D_{g}+\frac{c}{2}\left(\mathbb{E}_{\bm{x}\sim P_{\rm left}} \biggl{[}\frac{P(\sigma_{\rm right}|\bm{x})}{P(\sigma_{\rm left}|\bm{x})}-1 \biggr{]}+\mathbb{E}_{\bm{x}\sim P_{\rm right}}\biggl{[}\frac{P(\sigma_{\rm left }|\bm{x})}{P(\sigma_{\rm right}|\bm{x})}-1\biggr{]}\right)$ $\displaystyle=D_{g}+\frac{c}{2}\left(\mathbb{E}_{\bm{x}\sim P_{\rm left}}[1]+ \mathbb{E}_{\bm{x}\sim P_{\rm right}}[1]-2\right)=D_{g}.$ $$

#### Relation to the Fisher information.

In the following, we prove that any $g$-dissimilarity with $g(\frac{1}{2})=0$ and a twice-differentiable $g$-function reduces to the Fisher information in lowest order.

For this, consider the case with $L=1$ where we compare the distributions at two points in parameter space that are separated by $\delta T$. The corresponding $g$-dissimilarity is equivalent to an $f$-divergence $D_{f}[P(\bm{x}|T),P(\bm{x}|T+\delta T)]$. Note that $D_{f}[P(\bm{x}|T),P(\bm{x}|T)]=0$ if $f(1)=0$ and $\left.\partial D_{f}[P(\bm{x}|T),P(\bm{x}|B)]/\partial B\right|_{B=T}=f^{
\prime}(1)\partial\mathbb{E}_{\bm{x}\sim P(\cdot|T)}[1]/\partial T=f^{\prime}(
1)\partial 1/\partial T=0$. The second-order derivative corresponds to

$$ $\displaystyle\left.\frac{\partial^{2}D_{f}[P(\bm{x}|T),P(\bm{x}|B)]}{\partial B ^{2}}\right|_{B=T}$ $\displaystyle=f^{\prime}(1)\mathbb{E}_{\bm{x}\sim P(\cdot|T)}\biggl{[}\frac{1} {P(\bm{x}|T)}\frac{\partial^{2}P(\bm{x}|T)}{\partial T^{2}}\biggr{]}$ $\displaystyle\quad+f^{\prime\prime}(1)\mathbb{E}_{\bm{x}\sim P(\cdot|T)}\biggl {[}\left(\frac{\partial\log P(\bm{x}|T)}{\partial T}\right)^{2}\biggr{]}$ $\displaystyle=f^{\prime\prime}(1)\mathcal{F}(T)$ $$

assuming $f^{\prime}(1)=0$, where $\mathcal{F}$ is the Fisher information. Thus, for $L=1$ and parameter values separated by $\delta T$, we can express any $g$-dissimilarity with $g(\frac{1}{2})=0$ as $D_{g}=\frac{g^{\prime\prime}(\frac{1}{2})}{32}\mathcal{F}(T)\delta T^{2}+
\mathcal{O}(\delta T^{3})$. The fact that $f^{\prime}(1)$ must be zero translates into the condition that $g^{\prime}(\frac{1}{2})=0$. This is not a fundamental restriction since we have some freedom in the choice of $g$ function. That is, we can replace $g\mapsto\tilde{g}$, where $\tilde{g}(x)=g(x)+c(\frac{1}{x}-2)$ with $c=\frac{1}{6}g^{\prime}(\frac{1}{2})$, retaining $D_{g}=D_{\tilde{g}}$ and ensuring that $\tilde{g}^{\prime}(\frac{1}{2})=0$.

## Appendix C Details on Energy-Based Analysis of Temperature Transition

Let $\bm{x}=(x_{1},\dots,x_{N})$ be a sequence of $N$ tokens generated for a fixed prompt from an autoregressive LLM such as the ones considered in this article. The distribution of $\bm{x}$ is given by

$$ $P(\bm{x}|T)=Q_{T}(x_{N}|x_{1},\dots,x_{N-1})Q_{T}(x_{N-1}|x_{1},\dots,x_{N-2}) \cdots Q_{T}(x_{2}|x_{1})Q_{T}(x_{1}),$ $$

denoting the fact that the tokens are sampled sequentially. In each step, a token is sampled from a Boltzmann distribution $Q_{T}$,

$$ $Q_{T}(x_{i}|x_{1},\dots,x_{i-1})=e^{-E(x_{i}|x_{1},\dots,x_{i-1})/T}/Z_{i}(T).$ $$

Here, $Z_{i}(T)=\sum_{x_{i}}e^{-E(x_{i}|x_{1},\dots,x_{i-1})/T}$ is a normalization factor with the sum running over all possible $i$th tokens. The conditional energies, $E(x_{i}|x_{1},\dots,x_{i-1})$ are typically referred to as logits and learned from data. Note that while the distribution over individual tokens can be expressed as a Boltzmann distribution at varying temperature, the overall distribution $P(\bm{x}|T)$ cannot.(^5^55In order for a quantity to be a valid energy of a system, it cannot itself depend on temperature, i.e., change with temperature.)

Nevertheless, we can define an energy scale for the entire system by viewing the overall probability distribution at $T=1$ as a Boltzmann

$$ $P(\bm{x}|T=1)=e^{-E(\bm{x})}/Z,$ (C1) $$

where $Z$ is a normalization constant independent of $\bm{x}$ (also referred to as partition function). Recall that any valid probability distribution can be written in the form of Eq. (C1) with a suitably chosen energy function. Taking the logarithm of Eq. (C2) and reordering, we have

$$ $E(\bm{x})=-\log P(\bm{x}|T=1)-\log Z.$ (C2) $$

Using Eq. (C2), we can compute the total energy up to the constant $-\log Z$ which serves as our reference point for the energy scale. In the main text, we use $-\log P(\bm{x}|T=1)$ as the total energy to compute and compare energy statistics at various temperatures. In particular, the heat capacity remains the same under the mapping $-\log P(\bm{x}|T=1)-\log Z\mapsto-\log P(\bm{x}|T=1)$.

unsrt
\bibliographySIrefs_appendix

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: See Weaknesses: How do we know the method captures something interesting? Is there external validation?
soundness: 2
strengths: - The paper is innovative, linking physics concepts with language models
- The paper studies multiple open LLMs
- The proposed metric is theoretically grounded in a link to the Fisher information
- I found the execution to be rigorous and writing to be clear
- Findings can be of potential interest, e.g. it might be surprising that there phase transitions are identified at some training epochs
summary: This paper investigates phase transitions in the output distributions of language models – that is, situations where the output distribution of an LM abruptly changes when a numerical parameter crosses some value. Changes are measured using f-divergences, where f is a specific rational function. The paper investigates three example parameters: (i) a number appearing in the context [concretely, a number “T” as in “T is larger than 42”], (ii) the softmax temperature in the output, (iii) the number of training epochs.
weaknesses: From my perspective, the primary weakness of the paper is the lack of any external validation of the proposed approach. While the paper proposes a method for measuring phase transitions, it remained unclear to me how to validate the findings. The findings might in principle depend a lot on the choice of the divergence. Is there any independent way of validating that the outcomes of the proposed method are useful or interesting? Is there a clear use case? Is there a principled relation to a theoretical property of language, or of transformer networks? Without this, it is not clear to me how to (i) substantiate the claim that the proposed metric captures transitions in an interesting way, (ii) make the paper of interest to the community. I do believe that such things are possible, and that the proposed method can be of substantial interest, but it would take further work to do this, and to validate the proposed approach specifically, compared to various other ways one could measure phase transitions (e.g., with other f-divergences)

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. What advantages does linear dissimilarity offer compared to total variation distance and Jensen-Shannon divergence? Figure 1 shows that all types of divergence can capture the dissimilarities.
2. It would be interesting to examine how phase transitions occur given combinations of input and temperature, similar to how water's phase transitions depend on both pressure and temperature.
3. What are the different properties of phases before and after transition? Can these phases be analogous to physical states like 'solid' or 'gas'? Is there a human-interpretable explanation for each phase, particularly regarding temperature?
4. What is the definition of a phase transition? In Figure 2, the linear dissimilarity of 0.4 for the Llama3 base model appears quite high. Can this be considered a phase transition?
5. Is panel (b) in Figure 4 truly a zoomed-in plot of panel (a)? The black line and y-axis values appear to be different.
6. In Figure 5(b), why did you mention that the clear peaks near epochs 20K, 40K, and 80K are outliers?"
soundness: 2
strengths: The connection between phase transitions in physics and large language models is interesting!
summary: The authors propose statistical methods to automatically detect phase transitions in large language models. They provide empirical evidence for three types of phase transitions that occur when varying the input, temperature, and training epoch.
weaknesses: The paper's main challenge and contribution are unclear. While the authors cite several previous works that have analyzed phase transitions, their contribution appears to be proposing new statistical detection methods. However, if this is their primary contribution, the experimental validation is insufficient - they use only a limited number of prompts and test cases. More extensive experiments with diverse prompts would be needed to validate their methods.

Alternatively, if the paper aims to establish theoretical connections to physics, it requires more accessible and thorough explanations. For example, while they discuss the link between the Ising model and language models, readers unfamiliar with physics may struggle to follow the comparison without mathematical formalization. Also, the paper would be more effective if it included parallel visualizations comparing phase transitions in physical systems (e.g., heat capacity plots for water) with their language model findings.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Questions:
Which of the Pythia models is used for the graphs in figure 5? Why are you only plotting layer 0-5?

EDIT: Thanks for your answer.

Suggestions:

Perhaps interesting work to engage with related to case study 1: Hanna, M., Liu, O., & Variengien, A. (2024). How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language model. Advances in Neural Information Processing Systems, 36. This paper reveals a circuitry involved in building up a numerical prediction; the emergence of such a circuitry during training, or the switching on or off of such a circuitry due to contextual cues (in-context learning) would be an actually interesting phase transition.

Related to case study 3: Nanda, N., Chan, L., Lieberum, T., Smith, J., & Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. arXiv preprint arXiv:2301.05217. This paper shows you can predict the *sudden* emergence of a specific numerical ability (a phase transition) by tracking the *gradual* evaluation of specific "progress measures".
soundness: 3
strengths: This paper addresses an interesting problem: whether changes in output distributions of LLMs, as one varies an underlying parameter, can be used to detect interesting changes in the underlying mechanisms. It reviews some potentially useful measures from statistical physics, that place widely used measures in NLP and ML such as the KL divergence in a broader context.

The authors have looked at many different models and checkpoints, and obtaining the numbers reported in this paper / creating the graphs must have been quite a lot of work.
summary: This paper looks at LLMs from the perspective of statistical physics. It proposes several ways of measuring changes in probability distributions, where large changes are interpreted as signatures for "phase transitions". It applies these measures in 3 case studies. In the first, change is measured as a simple prompt, containing a numerical value, is varied. In the second, the temperature parameter in the softmax is varied. In the third, changes across checkpoints during training are recorded. For each of these case studies, graphs are produced, showing that there are ranges for the varied parameter where changes are larger.
weaknesses: The first half of the paper is devoted to reviewing mathematical tools to measure distances between distributions. In the second half of the paper, the distance measures are used in some simple case studies. The main weakness of the paper is that each of these case studies stops where it starts to become interesting, such that we learn nothing new about the behaviour or underlying mechanisms of LLMs, and that the usefulness of the mathematical tools from the first half is never shown.

I don't have a lot of comments on the distance measures from the first 3-4 pages. Countless papers in ML and NLP already measure distances between probability distributions (usually with KL and computationally tractable approximations such as ELBO). It would be surprising and potentially important if tools developed for studying phase transitions in physics offer a new light on these issues. I cannot judge how novel the framework the authors offer is for ML, but novel or not, I will just note that it is problematic that the authors do not in any way show that their favourite tools are in fact more useful than better known tools.

This usefulness would have to be established in the case studies in the second half, but unfortunately isn't. In case study (1), figure 1 and 2 reveals that LLMs build up internal representations for the numerical value of integers, that some do worse than others and that tokenization sometimes messes up the representations. None of that is news for people that have studied LLMs, and especially for people that have read papers on numerosity representations. And, moreover, it has nothing to do with phase transitions, unless that term is given such a broad definition that it looses its value. The fact that output distributions change significantly around the point where the input necessitates a different response is simply a consequence of designing and training networks to predict categorical output.

In case study (2), we get some graphs of how sensitive output distributions are for different values of the temperature parameter in the output softmax. The graphs look pretty, but what have we learned? My conclusions from reading this section is that for high temperature, distributions do not change (because they're random anyway), and that for low temperatures the exact value might have a quite big impact on the behaviour of the LLM (but that we don't know beforehand at which value and which effect). But all of that is pretty much the starting point of the section.

In case study (3), the paper studies training dynamics in the Pythia model suite. Figure 5 looks interesting, especially the different timing of transitions in layer 0-2, 3, 4 and 5 (although it is not clear to me which Pythia model this is exactly, and why are you only plotting layer 0-5). But then the text (line 449) explains that some of the peaks (in 5b at least) might just be noise, and not reflecting any macroscopic phase changes. So again, we are presented some graphs that may or may not show that some interesting changes in the underlying representations might be happening, here at specific moments during training. But the paper presents no effort to actually confirm that these changes are indeed "phase transitions" with a different method, or in any way characterize the representation before and after the "phase transition".

All in all, there is clearly a lot of effort and technical know-how that has gone into this project, but it suffers from a lack of engagement with the extensive literature on LLMs, their learned representations and probability distributions, the qualitative changes in these representations, and the potential usefulness of concepts like "phase transitions" and "emergence" in understanding those changes. In section 4, the authors cite a lot of relevant papers (e.g., Wei et al 22, Schaeffer et al. 24, Chen et al. 23), but without actually engaging with them. For this project to really have some impact on ML/NLP, more is required than plotting the values of a measure for lots of different models and lots of different parameters: you need to show that high values on these measures indeed reveal something important, by showing this importance with a different method yourself or by linking it to a result from an existing paper.

EDIT: I have read the general response and the responses to my and other reviews. Although the direct response to my review was brief, overall I appreciate that the authors have put quite a bit of work into answering questions and concerns from the reviewers. It's clear that the authors are knowledgable about work of phase transitions in physics; however, their responses don't take away my concern that the main work -- to demonstrate the usefulness of porting this framework to ML -- still needs to be done. I will therefore maintain my score (as well as my recommendation for the future to engage with some existing work in ML, and investigate whether new tools can shed new light on it!).

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: *I would like clarification on how the authors calculated the heat capacity. In connection with this, it would be valuable if the authors could provide concrete evidence or specific analyses regarding the occurrence of negative heat capacity. In typical thermodynamic systems, heat capacity should be positive; however, systems with long-range interactions or non-equilibrium properties can exhibit negative values. Have the authors investigated such properties in LLMs, for instance, through analyses of long-range interactions such as the attention mechanism, or do they have evidence supporting this phenomenon in LLMs?
* The advantage of the method used by the authors is that it can detect phase transitions from the output distributions. However, what kind of phase can the authors say it represents? Additionally, can the authors specify the type of phase transition, such as whether it is first-order or second-order?
*Minor points.
** It would be better to write the number "-1" in line 166 in math mode. 
** The caption for Fig.1 seems incorrect regarding the reference to the Appendix. 
** I initially thought Fig.4(b) was an enlarged view of Fig.4(a). While specific heat naturally connects, linear dissimilarity appears not to connect. Could the authors explain the reason?
soundness: 2
strengths: * This study is significant for applying a phase transition detection method from physics to the output distribution of LLMs. In physics, order parameters are often known, making the method less critical. However, in LLMs, the order parameter is entirely unknown, allowing this method to show its full advantages. 

* Considering the complex internal structure of LLMs, often regarded as a black box, this approach provides a potentially valuable tool for understanding LLM properties. As the authors point out, this is an advantage as it can be implemented even without detailed knowledge of an LLM's internal structure.
summary: This paper examines phase transition in the output distributions of large language models(LLMs) and proposes a novel approach that applies a method used in physical systems for detecting phase transitions to LLMs. In physical systems, phase transitions occur as control parameters, such as temperature and pressure, change. These transitions often depend only on macroscopic parameters and show universal properties. However, phase transitions in LLMs have been shown to depend on detailed ``microscopic" inputs, such as prompts, with phase transitions occurring at different temperatures depending on the prompt. This difference suggests that a distinct mechanism may be at work in LLMs compared to physical systems, and the fact that the "phase of behavior" in LLMs is induced by prompts is particularly interesting. This study provides a new method for automatically detecting the phase transitions in LLMs through black-box analysis based on output distribution, yielding new insights into the learning behaviors of LLMs.
weaknesses: * Although multiple phase transitions are identified, it remains unclear how they are distinguished from crossover, as mentioned in Appendix. For example, the study suggests three phases when temperature is changed, but there is a lack of explanation of what each phase represents and its significance from a linguistic perspective.

* This point slightly contradicts the above strength: what does it mean for phase transitions to depend on prompts? For example, if the temperature at which water freezes changes depending on the shape of the glass, can we truly say water undergoes a phase transition? While I understand the significance of the authors' extensive numerical experiments, it is difficult to understand their relevance to LLM research.

Meta-review: This paper applies mathematical ideas about phase transitions to study language models. Though reviewers found this interesting in the abstract, there was a general confusion about what question is actually being addressed here, and what these methods really bring to the table. It seems that more clarity is required before this paper is ready for publication. See in particular the discussion from reviewer mqhi

Decision: Reject