Learnability on the Information-Theoretic Continuum: Inductive Bias for Information Locality in Neural Language Models
Authors: Taiga Someya, Anej Svete, Brian DuSell, Mario Giulianelli, Timothy J. O'Donnell, Ryan Cotterell
Accepted at: acl25

Information Locality as an Inductive Bias for Neural Language Models

Taiga Someya1* Anej Svet2 Brian DuSell2
Timothy J. O’Donnell3 Mario Giulianelli2 Ryan Cotterell2
1The University of Tokyo 2ETH Zürich 3McGill University
taiga98-0809@g.ecc.u-tokyo.ac.jp timothy.odonnell@mcgill.ca
{asvete,brian.dusell,mario.giulianelli,ryan.cotterell}@inf.ethz.ch

Abstract

Inductive biases are inherent in every machine learning system, shaping how models generalize from finite data. In the case of neural language models (LMs), debates persist as to whether these biases align with or diverge from human processing constraints. To address this issue, we propose a quantitative framework that allows for controlled investigations into the nature of these biases. Within our framework, we introduce \( m \)-local entropy—an information-theoretic measure derived from average lossy-context surprisal—that captures the local uncertainty of a language by quantifying how effectively the \( m - 1 \) preceding symbols disambiguate the next symbol. In experiments on both perturbed natural language corpora and languages defined by probabilistic finite-state automata (PFSA), we show that languages with higher \( m \)-local entropy are more difficult for Transformer and LSTM LMs to learn. These results suggest that neural LMs, much like humans, are highly sensitive to the local statistical structure of a language.

https://github.com/rycolab/lm-inductive-bias

1 Introduction

Every machine learning system has some form of inductive bias: given a finite training sample with potentially infinitely many compatible hypotheses, its architecture and learning algorithm predispose it to favor certain generalizations over others (Mitchell, 1980; Rawski and Heinz, 2019). The concept of inductive bias is central to the growing discussion of whether the inductive biases of neural network LMs align with the cognitive pressures that shape human language learning. In a 2023 New York Times article, Chomsky et al. argued that neural LMs possess inductive biases fundamentally different from human cognitive constraints, a claim that has motivated theoretical rebuttals (Piantadosi, 2024) as well as empirical research to test the extent of this divergence (Kallini et al., 2024; Ahuja et al., 2024).

In particular, Kallini et al. (2024) demonstrated that perturbing natural language corpora to alter their sequential structure, making the languages less human-like, renders languages harder for neural LMs to learn. While their findings suggest that disrupting local structure (e.g., through local shuffling transformations) impacts learnability, they do not isolate the specific property responsible for this effect. To rigorously assess whether a neural LM’s inductive biases align with human constraints, we must identify *quantifiable properties* of language that affect human learning difficulty and *systematically manipulate* these properties in controlled experiments with neural LMs. Information-theoretic models of language

![KL divergence (Transformer LM) as a function of the 3-local entropy of the language generated from a PFSA in Experiment 2. LMs perform better at languages with lower local entropy.](page_1016_370_355_355.png)

Figure 1: KL divergence (Transformer LM) as a function of the 3-local entropy of the language generated from a PFSA in Experiment 2. LMs perform better at languages with lower local entropy.
processing, which view the structure of languages as shaped by language users’ joint optimization of informativity and complexity, provide a promising framework for identifying these properties.

In this paper, we turn to the principles of information locality, which suggest that language is structured to minimize linear distance between linguistic elements with high mutual information (Gibson, 1998, 2001; Futrell, 2019; Futrell et al., 2020). Information locality is thought to arise from the memory limitations of human processors, which make it challenging to integrate long-range linguistic dependencies (Hahn et al., 2022). The influence of these cognitive constraints has been observed across multiple timescales, in both language comprehension and production, and across diverse languages of the world (Hahn et al., 2021; Futrell, 2023; Futrell and Hahn, 2024). Demonstrating that a neural LM’s learnability of a language is influenced by its local predictability would reveal an inductive bias that aligns with the functional pressures shaping human language processing.

As a first step in this direction, we propose using \( m \)-local entropy, an information-theoretic measure designed to quantify a linear notion of local predictability which can be derived from the principles of information locality (Futrell et al., 2020).\footnote{More specifically, \( m \)-local entropy can be derived from the lossy-context surprisal theory of language comprehension (Futrell et al., 2020), a theory belonging to the expectation-based family of theories of language processing (Hale, 2001; Levy, 2008). See §2.2.2 for details.} We study how \( m \)-local entropy affects learnability in two sets of experiments: one where we perturb natural language corpora, and another where we randomly generate probabilistic finite-state automata (PFSAs). We train LSTM and Transformer LMs on these languages and examine whether neural LMs’ difficulty in learning a language is predicted by the language’s \( m \)-local entropy, to see if neural LMs and humans share an inductive bias for information locality. Our experiments demonstrate that \( m \)-local entropy negatively correlates with the ability of a neural LM to learn a probabilistic language. Specifically, our experiment with an English natural language corpus—and its perturbed variants—reveals that LSTM and Transformer LMs show systematic degradation in performance as \( m \)-local entropy increases, even when global and next-symbol entropy are held constant. Furthermore, in experiments using PFSAs, we manipulate the properties of languages more systematically and show that this trend is not an artifact of the corpora or particular perturbation functions used in our experiments.

2 Formal Background

2.1 Languages and Language Models

An alphabet \( \Sigma \) is a finite, non-empty set of symbols. The Kleene closure \( \Sigma^* \) is the set of all strings with symbols from \( \Sigma \). We use \( |y| \) to denote the length of \( y \in \Sigma^* \). A language is a subset of \( \Sigma^* \).

A language model \( p \) is a probability distribution over \( \Sigma^* \). The prefix probability \( \overrightarrow{p}(y) \) is the probability that a string begins with \( y \in \Sigma^* \):

\[
\overrightarrow{p}(y) \overset{\text{def}}{=} \sum_{y' \in \Sigma^*} p(y y')
\] (1)

The conditional prefix probability of a string \( y' \in \Sigma^* \) given another string \( y \) is given by

\[
\overrightarrow{p}(y' \mid y) = \frac{\overrightarrow{p}(y y')}{\overrightarrow{p}(y)}.
\] (2)

Using the notion of a conditional prefix probability, one can factorize a language model \( p \) as\footnote{Modern neural LMs (e.g., LSTMs, Transformers) directly parameterize the *next-symbol* probability distribution \( q'(y \mid y) \) over \( \Sigma \), which coincides with the conditional prefix probability \( \overrightarrow{p}(y \mid y) \). Training therefore maximizes the log-likelihood of observed symbols, and the full string probability \( q(y) \) is recovered via the product in (3).}

\[
p(y) = \overrightarrow{p}(\text{EOS} \mid y) \prod_{t=1}^{|y|} \overrightarrow{p}(y_t \mid y_{<t}),
\] (3)

where each \( \overrightarrow{p}(y_t \mid y_{<t}) \) is a distribution over \( \Sigma \overset{\text{def}}{=} \Sigma \cup \{ \text{EOS} \} \), where EOS \( \notin \Sigma \) is a distinguished end-of-string symbol, and where we define

\[
\overrightarrow{p}(\text{EOS} \mid y) \overset{\text{def}}{=} \frac{p(y)}{\overrightarrow{p}(y)}.
\] (4)

We define \( p \)'s infix probability \( \overleftarrow{p} \) as

\[
\overleftarrow{p}(y) \overset{\text{def}}{=} \sum_{y' \in \Sigma^*} \sum_{y'' \in \Sigma^*} p(y' y y'').
\] (5)

Note that, despite denoting the probability of an event, \( \overrightarrow{p} \) and \( \overleftarrow{p} \) are *not* probability distributions over \( \Sigma^* \). In order to convert them into probability distributions, we have to renormalize them. However, the sums \( \overrightarrow{Z} \overset{\text{def}}{=} \sum_{y \in \Sigma^*} \overrightarrow{p}(y) \) and \( \overleftarrow{Z} \overset{\text{def}}{=} \sum_{y \in \Sigma^*} \overleftarrow{p}(y) \) may diverge. A necessary
and sufficient condition for \( \overrightarrow{Z} \) to be finite is that the expected length \( \mu \overset{\mathrm{def}}{=} \mathbb{E}_{y \sim p}|y| \) under \( p \) is finite.\footnote{See Opedal et al. (2024, Prop. 1).} Thus, in the following, we assume that \( \mu \overset{\mathrm{def}}{=} \mathbb{E}_{y \sim p}|y| < +\infty \), which implies that we can normalize the prefix probability function to form probability distributions. We do not require that \( \overrightarrow{Z} \) be finite in this paper. However, one can show that \( \overrightarrow{Z} \) is finite if and only if \( \mathbb{E}_{y \sim p}|y|^2 < +\infty \).

**Random Variables.** We will use the following random variables in our exposition. First, let \( \mathbf{Y} \) be a \( \Sigma^* \)-valued random variable distributed as

\[
\Pr(\mathbf{Y} = y) \overset{\mathrm{def}}{=} p(y).
\] (6)

Then, let \( \overrightarrow{\mathbf{Y}} \) be a \( \Sigma^* \)-valued random variable distributed according to

\[
\Pr(\overrightarrow{\mathbf{Y}} = y) \overset{\mathrm{def}}{=} \frac{\overrightarrow{p}(y)}{Z}.
\] (7)

Next, let \( \overline{Y} \) be a \( \Sigma \)-valued random variable jointly distributed with \( \overrightarrow{Y} \) according to

\[
\Pr(\overline{Y} = y \mid \overrightarrow{Y} = y) \overset{\mathrm{def}}{=} \overrightarrow{p}(y \mid y).
\] (8)

Finally, let \( C \) be a \( \Sigma^{m-1} \)-valued random variable distributed according to \( \overrightarrow{p} \) normalized over \( \Sigma^{m-1} \):

\[
\Pr(C = c) \overset{\mathrm{def}}{=} \frac{\overrightarrow{p}(c)}{\sum_{c' \in \Sigma^{m-1}} \overrightarrow{p}(c')}.
\] (9)

\( \Pr(C = c) \) can be interpreted as observing \( c \) as a length-\((m-1)\) substring of a string sampled from \( p \). Additionally, the random variable \( \overline{Y} \) is conditionally distributed, given \( C \), according to

\[
\Pr(\overline{Y} = y \mid C = c) = \sum_{y' \in \Sigma^*} \overrightarrow{p}(y \mid y'c) \overrightarrow{p}(y'),
\] (10)

i.e., as the next symbol given that the previous \( m-1 \) symbols were \( c \).

**2.2 Global Entropy and \( M \)-local Entropy**

The concept of *entropy*, as introduced by Shannon (1948), provides a foundational framework for quantifying uncertainty in language. Depending on how one defines the underlying probability distribution over linguistic units, entropy can capture different aspects of language complexity. In this paper, we discuss two versions of entropy—*global* entropy (i.e., the Shannon entropy) and *\( M \)*-*local* entropy—each capturing different facets of language complexity.

**2.2.1 Global Entropy**

The global entropy of \( p \) is defined as

\[
H(Y) \overset{\mathrm{def}}{=} - \sum_{y \in \Sigma^*} p(y) \log p(y).
\] (11)

Note that Eq. (11) may be infinite for some \( p \). However, for the remainder of our paper, we will assume that \( H(Y) < +\infty \). Eq. (11) reflects the uncertainty in the distribution over all possible strings \( y \in \Sigma^* \): Higher global entropy indicates that \( p \) distributes probability mass more uniformly across strings.\footnote{Because \( \Sigma^* \) is countably infinite, there is no uniform distribution over \( \Sigma^* \).}

We further define the (global) next-symbol entropy for finite-mean-length language models as the weighted average of the entropy of all local next-symbol distributions, averaging over all possible contexts \( y \in \Sigma^* \), and weighted by the normalized prefix probability of \( y \).\footnote{The weighting cannot be uniform, as \( \Sigma^* \) is infinite.} First, for a specific context \( y \), we define the context-specific next-symbol entropy as follows:

\[
\begin{align}
H(\overline{Y} \mid \overrightarrow{Y} = y) &\overset{\mathrm{def}}{=} - \sum_{y' \in \Sigma} \overrightarrow{p}(y' \mid y) \log \overrightarrow{p}(y' \mid y).
\end{align}
\] (12)

Then, using Eq. (12), we construct the **next-symbol entropy** as

\[
\begin{align}
H(\overline{Y} \mid \overrightarrow{Y}) &= \sum_{y \in \Sigma^*} \Pr(\overrightarrow{Y} = y) H(\overline{Y} \mid \overrightarrow{Y} = y) \tag{13a}\\
&= \sum_{y \in \Sigma^*} \frac{\overrightarrow{p}(y)}{Z} H(\overline{Y} \mid \overrightarrow{Y} = y) \tag{13b}\\
&= \frac{1}{\mu + 1} \sum_{y \in \Sigma^*} \overrightarrow{p}(y) H(\overline{Y} \mid \overrightarrow{Y} = y) \tag{13c}\\
&= \frac{1}{\mu + 1} H(Y). \tag{13d}
\end{align}
\]

where we exploit the identity \( \mu + 1 = \overrightarrow{Z} \) and where the last equality follows from Malagutti et al. (2024, Thm. 2.2). What Eq. (13d) tells us is that the next-symbol entropy is *proportional* to the global entropy. So, if both were used as predictors in a linear model, they would yield identical predictive power.

**Invariance of global and next-symbol entropy.**

Our goal in this paper is to isolate how local uncertainty of a language affects the performance of neural LMs in learning the language. Global entropy
and its length-normalized variant, next-symbol entropy, are obvious benchmarks, yet neither is sensitive to the local statistical structure we want to manipulate. Global entropy is invariant to bijective transformations of \( \Sigma^* \); that is, for any bijection \( f : \Sigma^* \to \Sigma^*, \mathrm{H}(Y) = \mathrm{H}(f(Y)) \). Next-symbol entropy is only slightly more rigid: it is, in general, not conserved under bijection, but it remains constant under any length-preserving bijection. To capture the aspect of local uncertainty that these measures miss, we introduce \( m \)-local entropy in the next section. Unlike global entropy and next-symbol entropy, \( m \)-local entropy does change under the length-preserving, bijective perturbations used in our experiment (\S3.1), which makes it possible to create a set of counterfactual corpora where each corpus has the same global and next-symbol entropy but has different \( m \)-local entropy.

2.2.2 \( M \)-local Entropy
Next-symbol entropy measures uncertainty over next-symbol predictions conditioned on the full available context, averaged across all possible contexts. This can be seen as the limit of a local quantification of uncertainty, which captures the unpredictability of the next symbol given a fixed amount of preceding context. We term this fixed-context uncertainty measure \( m \)-local local entropy.

Given any \( c \in \Sigma^{m-1} \), we can compute
\[
\mathrm{H}(\overline{Y} \mid C = c) = - \sum_{y \in \Sigma} \Pr(\overline{Y} = y \mid C = c) \\
\log \Pr(\overline{Y} = y \mid C = c).
\]
(14)

This captures the unpredictability of a symbol \( y \) after observing a given local context \( c \). We can then say that the \( m \)-local entropy of \( p \) is an expectation over possible contexts \( c \in \Sigma^{m-1} \), with each context weighted by \( \Pr(C = c) \):
\[
\mathrm{H}(\overline{Y} \mid C) = \sum_{c \in \Sigma^{m-1}} \Pr(C = c) \ \mathrm{H}(\overline{Y} \mid C = c).
\]
(15)

This yields a measure of local complexity that can differ from global entropy by more than a multiplicative constant. Even when two languages have identical global entropy, their \( m \)-local entropies reflect differences in how reliably the local context predicts the next symbol. Importantly, unlike global entropy, local entropy is not necessarily preserved under bijective transformations of \( \Sigma^* \), which enables us to assess the impact of such transformations on learnability. Our experiments show that transformations that disrupt local statistical structure are associated with how well neural LMs are able to learn a probabilistic language.

\( M \)-local entropy and lossy-context surprisal.
As a generalization of the surprisal model of language processing difficulty (Hale, 2001; Levy, 2008), Futrell et al. (2020) propose lossy-context surprisal. In this framework, the predicted difficulty of an upcoming word is proportional to the word’s expected log probability given a lossy memory representation of the preceding context. A memory encoding function specifies a distribution over such representations. If that function keeps only the \( m-1 \) symbols immediately preceding the target word, the metric collapses to the familiar \( m \)-gram surprisal. Averaging this quantity over all possible contexts with language-specific weights yields the average (lossy-context) surprisal of a language (Futrell, 2019; Hahn et al., 2021). When those weights are the contexts’ infix probabilities (see Eq. (5)), the average coincides with our definition of \( m \)-local entropy (Eq. (15)). To our knowledge, no prior work has linked lossy-context surprisal directly to language model learnability—a connection that our work aims to explore.

2.3 Probabilistic Finite-State Automata

Definition 2.1. A probabilistic finite-state automaton (PFSA) is a 5-tuple \( (\Sigma, Q, \delta, \lambda, \rho) \) where
• \( \Sigma \) is an alphabet,
• \( Q \) is a finite set of states,
• \( \delta \subseteq Q \times \Sigma \times [0, 1] \times Q \) is a finite set of weighted transitions, rendered as \( q \xrightarrow{y/w} q' \) with \( y \in \Sigma \) and \( w \in [0, 1] \),
• \( \lambda, \rho : Q \to [0, 1] \) are the initial and final weighting functions,
• \( \lambda \) satisfies \( \sum_{q \in Q} \lambda(q) = 1 \), and
• for all \( q \in Q \), \( \sum_{q' \in \delta(q)} w + \rho(q) = 1 \).

A path \( \pi \) in a PFSA \( \mathcal{A} \) is a sequence of consecutive transitions \( q_0 \xrightarrow{y_1/w_1} \cdots \xrightarrow{y_N/w_N} q_N \). We define its scan as \( s(\pi) \overset{\text{def}}{=} y_1 \cdots y_N \). \( \Pi(\mathcal{A}, y) \) denotes the set of all paths in \( \mathcal{A} \) that scan \( y \in \Sigma^* \). The inner path weight of \( \pi \) is \( \overline{w}(\pi) = \prod_{n=1}^N w_n \), and its path weight is \( w(\pi) = \lambda(q_0) \overline{w}(\pi) \rho(q_N) \).

A PFSA \( \mathcal{A} \) induces a language model \( p_{\mathcal{A}} \) as
\[
p_{\mathcal{A}}(y) \overset{\text{def}}{=} \sum_{\pi \in \Pi(\mathcal{A}, y)} w(\pi).
\]
(16)

Studying PFSAs not only allows us to perform controlled experiments but also enables us to com-
pute many quantities of interest exactly. App. A contains a collection of closed-form solutions for computing various quantities of interest, including the string (prefix and infix) probabilities and the \( m \)-local entropy of the induced language model.

3 Experiment 1: LM Performance along the M-local Entropy Continuum

In the first experiment, we investigate the relationship between local entropy and LM performance using a natural language corpus. We hypothesize that *local entropy* is a key factor in determining how easily an LM learns a language. To test this hypothesis, we apply a length-preserving, bijective perturbation function to a natural language corpus that alters its local structure. As we will see in the following section, this results in a counterfactual perturbed corpus (cf. Kallini et al., 2024), where language models have different \( m \)-local entropy but the same global entropy and next-symbol entropy. We then train neural LMs on the naturally occurring corpus and the perturbed one and study how local entropy affects the neural LMs’ performance.

3.1 Constructing Languages with Different Local Complexity

Here, we detail several specific transformations implemented in our experiments, refining the perturbation functions of Kallini et al. (2024). Note that all of the perturbation functions defined below are both bijective and length-preserving.

**DeterministicShuffle.** Given a string of length \( T \), \( y = y_1 y_2 \cdots y_T \), the function applies a length-specific permutation \( \sigma_T \) of the positions \( \{1, \ldots, T\} \). Each \( \sigma_T \) is sampled *once* at the start of the experiment (using a fixed pseudorandom seed) and then reused. Therefore, two strings of the same length are shuffled in exactly the same way, whereas strings of different lengths are transformed according to different permutations. By construction, every \( \sigma_T \) is a bijection on \( \{1, \ldots, T\} \), so the string length is left unchanged.

**Reverse.** This function reverses the entire sequence of symbols. Formally, given a string \( y = y_1 y_2 \cdots y_T \), the REVERSE mapping produces \( y_T y_{T-1} \cdots y_1 \). It is trivially invertible (by applying the same operation again), making it a bijection.

**EvenOddShuffle/OddEvenShuffle.** Let \( y = y_1 y_2 \cdots y_T \) be a string of length \( T \). Define two subsequences \( O(y) = y_1 y_3 y_5 \cdots \) and \( E(y) = y_2 y_4 y_6 \cdots \) that collect all symbols in \( y \) at *even* positions and *odd* positions, respectively. We then define **EvenOddShuffle**(\( y \)) = \( E(y)O(y) \) and **OddEvenShuffle**(\( y \)) = \( O(y)E(y) \).

**K-localDeterministicShuffle.** Let \( y = y_1 y_2 \cdots y_T \) be a string in \( \Sigma^* \), which we partition into consecutive windows of size \( k \). For the \( i \)th window, \( y_{(i-1)k+1} \cdots y_{ik} \), we apply a fixed permutation \( \pi_i^k \) determined by window size \( k \), window index \( i \) and a global random seed. Formally, the K-LOCALDETERMINISTICSHUFFLE of \( y \) produces \( (\pi_1^k(y_1 \cdots y_k), \pi_2^k(y_{k+1} \cdots y_{2k}), \ldots) \).\footnote{If the string length \( T \) is not a multiple of \( k \), then the final window, which contains \( \ell (< k) \) symbols, is permuted by applying the length-\( \ell \) permutation \( \pi_i^k \) to all the available symbols in that window.}

3.2 Estimating the M-local Entropy

Unfortunately, when we only observe a corpus, the \( m \)-local entropy of the data-generating distribution is not known. In this experiment, we estimate it using an \( n \)-gram language model implemented with KenLM (Heafield, 2011). Given a corpus \( \mathbb{D} \), we train an \( n \)-gram model on \( \mathbb{D} \) to get the estimated conditional probability distribution \( \hat{p}(y \mid c) \) for \( c \in \Sigma^{m-1} \). Plugging this estimated probability distribution into Eq. (14), we can compute

\[
\hat{H}(\overline{Y} \mid C = c) = -\frac{1}{N(c)} \sum_{y \in \mathbb{D}} \log \hat{p}(y \mid c),
\]

where \( N(c) \) is the number of times \( c \) appears in \( \mathbb{D} \). The normalized infix probability is estimated as

\[
\hat{p}(C = c) = \frac{N(c)}{N_{\text{total}}},
\]

where \( N_{\text{total}} = \sum_{c' \in \Sigma^{m-1}} N(c') \).

Given these and Eq. (15), we can approximate the \( m \)-local entropy as

\[
\begin{align*}
\mathrm{H}(\overline{Y} \mid C) \\
&= \sum_{c \in \Sigma^{m-1}} \hat{p}(C = c) \; \hat{H}(\overline{Y} \mid C = c) \tag{19a} \\
&= -\frac{1}{N_{\text{total}}} \sum_{c y \in \mathbb{D}} \log \hat{p}(y \mid c). \tag{19b}
\end{align*}
\]

This estimator is a practical proxy for the quantity in Eq. (15). The \( m \)-local entropy of each corpus is estimated by an \( n \)-gram model with order \( m - 1 \) trained on the concatenation of the training, validation, and test set of the corpus.
<table>
  <tr>
    <th rowspan="2">m</th>
    <th colspan="2">50K samples</th>
    <th colspan="2">200K samples</th>
  </tr>
  <tr>
    <th>MAE</th>
    <th>MRE (%)</th>
    <th>MAE</th>
    <th>MRE (%)</th>
  </tr>
  <tr>
    <td>2</td>
    <td>0.0015 ± 0.0009</td>
    <td>0.04 ± 0.02</td>
    <td>0.0007 ± 0.0005</td>
    <td>0.02 ± 0.01</td>
  </tr>
  <tr>
    <td>3</td>
    <td>0.0132 ± 0.0058</td>
    <td>0.36 ± 0.14</td>
    <td>0.0046 ± 0.0021</td>
    <td>0.13 ± 0.05</td>
  </tr>
  <tr>
    <td>4</td>
    <td>0.1267 ± 0.0615</td>
    <td>3.63 ± 1.53</td>
    <td>0.0522 ± 0.0267</td>
    <td>1.49 ± 0.67</td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.4702 ± 0.1953</td>
    <td>13.82 ± 4.78</td>
    <td>0.2599 ± 0.1229</td>
    <td>7.61 ± 3.10</td>
  </tr>
</table>

Table 1: Mean absolute error (MAE) and mean relative error (MRE) of the \( m \)-gram estimator of \( m \)-local entropy (averaged over 16 PFSAs).

3.2.1 Validating \( M \)-local Entropy Estimation
To ensure the reliability of our \( m \)-local entropy estimation with \( n \)-gram models, we quantify the discrepancy between the \( m \)-local entropy estimated with an empirical \( n \)-gram model with order \( m-1 \) and the true \( m \)-local entropy that can be calculated analytically for PFSAs. We sample 16 PFSAs covering two values for the number of states (\(|Q| \in \{8, 16\}\)), two alphabet sizes (\(|\Sigma| \in \{32, 48\}\)), two different random topologies, and two different random transition weights (see App. B for the details). For each PFSA, we generate two corpora, with 50K and 200K strings respectively, and then estimate \( m \)-local entropy using \( n \)-gram models with order \( m-1 \) for \( m = 2, \ldots, 5 \). For every \((m, \text{PFSA})\) combination, we compute the absolute (AE) and relative (RE) error between the estimated and the true \( m \)-local entropy, then average the errors across the 16 PFSAs. Table 1 summarizes our results. With 200K strings, the estimator achieves relative error below one percent up to \( m = 4 \) and remains under 8% even for \( m = 5 \), the most challenging setting. Increasing the corpus size from 50K to 200K consistently reduces both absolute and relative error, confirming that additional data yield further improvements. In sum, the \( m \)-gram estimator provides a faithful approximation to the true \( m \)-local entropy for all \( m \) studied.

3.3 Experimental Setup

3.3.1 Neural Language Models
We investigate how varying \( m \)-local entropy in a language model impacts the performance of two widely used neural LM architectures: the LSTM (Hochreiter and Schmidhuber, 1997) and the Transformer (Vaswani et al., 2017). We use a single-layer LSTM with 512-dimensional hidden units and a 4-layer causally-masked Transformer encoder with 768-dimensional representations, 3072-dimensional feedforward layers, and 12 attention heads. Both are implemented in PyTorch (Paszke et al., 2019). Both architectures are trained on the training set via the standard language modeling objective across 5 random training seeds. See Appendices C and D for more details.

3.3.2 Dataset
We conduct our experiments on a subset of the Brown Laboratory for Linguistic Information Processing 1987–89 Corpus Release 1 (BLLIP; Charniak et al., 2000).7 Specifically, we adopt the same training, development, and test splits as BLLIP-sm in Hu et al. (2020), which comprise roughly 200K sentences, totaling around 5M tokens. Starting from this original corpus, we apply the perturbation functions in §3.1 to produce perturbed corpora. For both DETERMINISTICSHUFFLE and K-LOCALDETERMINISTICSHUFFLE, we use 20 random seeds. In the case of K-LOCALDETERMINISTICSHUFFLE, we vary the parameters \( k \) over the set \(\{3, 4, 5, 6, 7\}\), yielding 20 perturbed corpora for each \( k \). This produces a total of 124 distinct corpora, including other perturbed corpora and the BASE (original) corpus.

3.3.3 Evaluating Learning Difficulty
In this experiment, we rely on next-symbol cross-entropy as a measure of how well a trained language model \( q \) approximates the target distribution. If \( p \) is the ground-truth language model, and \( q \) is any learned neural LM, we can estimate the next-symbol cross-entropy as:

\[
\hat{H}_q(\overline{Y} \mid \overline{Y'})
= - \frac{1}{S} \sum_{y \in \mathbb{D}} \left[ \log \ q\left( \overline{Y} = \text{EOS} \mid \overline{Y'} = y \right) \right.
+ \left. \sum_{t=1}^{|y|} \log \ q\left( Y = y_t \mid \overline{Y'} = y_{<t} \right) \right],
\]

7We also conduct the same set of experiments using the BabyLM corpus (Choshen et al., 2024); results in App. F.
<table>
  <tr>
    <th></th>
    <th>2-local entropy</th>
    <th>3-local entropy</th>
    <th>4-local entropy</th>
    <th>5-local entropy</th>
  </tr>
  <tr>
    <th>BASE</th>
    <td>6.67</td>
    <td>4.27</td>
    <td>2.92</td>
    <td>2.45</td>
  </tr>
  <tr>
    <th>REVERSE</th>
    <td>6.98</td>
    <td>4.39</td>
    <td>2.98</td>
    <td>2.51</td>
  </tr>
  <tr>
    <th>EVENODDSHUFFLE</th>
    <td>7.91</td>
    <td>5.10</td>
    <td>3.76</td>
    <td>3.43</td>
  </tr>
  <tr>
    <th>ODDEVENSHUFFLE</th>
    <td>7.87</td>
    <td>5.07</td>
    <td>3.74</td>
    <td>3.41</td>
  </tr>
  <tr>
    <th>LOCALSHUFFLE (k=3)</th>
    <td>8.12 ± 0.08</td>
    <td>5.05 ± 0.06</td>
    <td>3.68 ± 0.07</td>
    <td>3.39 ± 0.07</td>
  </tr>
  <tr>
    <th>LOCALSHUFFLE (k=4)</th>
    <td>8.25 ± 0.07</td>
    <td>5.17 ± 0.04</td>
    <td>3.80 ± 0.04</td>
    <td>3.56 ± 0.05</td>
  </tr>
  <tr>
    <th>LOCALSHUFFLE (k=5)</th>
    <td>8.33 ± 0.07</td>
    <td>5.25 ± 0.04</td>
    <td>3.88 ± 0.04</td>
    <td>3.64 ± 0.05</td>
  </tr>
  <tr>
    <th>LOCALSHUFFLE (k=6)</th>
    <td>8.43 ± 0.07</td>
    <td>5.31 ± 0.05</td>
    <td>3.97 ± 0.06</td>
    <td>3.72 ± 0.06</td>
  </tr>
  <tr>
    <th>LOCALSHUFFLE (k=7)</th>
    <td>8.47 ± 0.07</td>
    <td>5.36 ± 0.05</td>
    <td>4.03 ± 0.06</td>
    <td>3.78 ± 0.07</td>
  </tr>
  <tr>
    <th>DETERMINISTICSHUFFLE</th>
    <td>8.77</td>
    <td>5.73</td>
    <td>4.60</td>
    <td>4.41</td>
  </tr>
</table>

Table 2: \( M \)-local entropy values for the BASE (original) corpus and the different perturbed corpora. LOCAL SHUFFLE refers to the K-LOCALDETERMINISTICSHUFFLE. Values are shown as mean ± standard deviation (averaged over different random seeds).

where \( \mathbb{D} = \{ y^{(n)} \}_{n=1}^N \) is a set of i.i.d. draws from \( p \), and \( S = \sum_{y \in \mathbb{D}} |y| + 1 \). In this experiment, we evaluate each LM using the estimated next-symbol cross-entropy on the test set.

When comparing the ability of a neural LM to learn two different target language models, it is essential to account for the inherent entropy of each target language model. In the statistical setting, learning a language means estimating its probability distribution as closely as possible. Consequently, cross-entropy without taking into account the entropy can lead to incorrect conclusions; see, for example, NONDETERMINISTICSHUFFLE in Kallini et al., 2024. To address this, our experiments ensure that all corpora have the same inherent (global) entropy; see §3.1, which allows us to safely compare cross-entropy results across different languages.

3.4 Results

How do different perturbations affect \( m \)-local entropy? Table 2 reports the \( m \)-local entropy values (\( m \in \{2, 3, 4, 5\} \)) for the BASE corpus and the various perturbed corpora. REVERSE barely changes the \( m \)-local entropy, whereas EVENODDSHUFFLE and ODDEVENSHUFFLE increase it somewhat more. In contrast, K-LOCALDETERMINISTICSHUFFLE yields progressively higher entropy as the window size \( k \) grows, indicating a greater disruption of local ordering. Finally, DETERMINISTICSHUFFLE produces the highest \( m \)-local entropies among all transformations. Our results confirm that the bijective transformations we defined in §3.1 effectively generate new language models with different \( m \)-local entropies from the original one, while preserving the global entropy by design. This yields a continuum of languages along a specific measurable axis of complexity rather than a qualitative notion of possibility as in Kallini et al. (2024).

\( M \)-local Entropy and LM Performance. Figure 2 shows the relationship between the \( m \)-local entropy (estimated by \( m \)-gram models; §3.2) and the next-symbol cross-entropy of each neural LM on the test set. We observe a strong positive correlation between \( m \)-local entropy and next-symbol cross-entropy for both neural architectures. For example, with \( m = 4 \), the coefficient of determination \( R^2 \) reaches 0.922 for the LSTM LM and 0.915 for the Transformer LM, indicating that higher local ambiguity (as measured by \( m \)-local entropy) generally leads to decreased performance (i.e., higher next-symbol cross-entropy) under both models. Furthermore, since our transformations are designed to preserve global entropy and global next-symbol entropy, these results highlight the crucial role of local entropy in the learnability of a language by neural LMs. This suggests that neural LMs inherently possess an inductive bias toward languages with lower local entropy.

4 Experiment 2: Controlled Learnability Tests with PFSAs

Experiment 1 only focused on a specific English corpus and a specific set of perturbation functions. To confirm that the results are not just an artifact of this experimental design but a fundamental property of neural LMs, we conduct a controlled experiment using PFSAs. This also enables us to
Figure 2: Scatter plots of next-symbol cross-entropy (y-axis) versus m-local entropy (x-axis) for \( m \in \{2, 3, 4, 5\} \), for both LSTM (top row) and Transformer LM (bottom row). Each marker type/color corresponds to a different perturbation (e.g., Reverse, DeterministicShuffle, K-LOCALDETERMINISTICSHUFFLE with various window sizes, etc.). The red star indicates the unperturbed Base condition (original corpus). The dashed line in each panel is a linear fit, with \( R^2 \) indicating the coefficient of determination.

compute quantities of interest exactly, especially the \( m \)-local entropy of the induced language model.

4.1 Experimental Setup

We use the same neural LMs and training configurations as in §3, but we generate datasets using PFSAs (\S4.1.1) and evaluate neural LMs while controlling for global entropy (\S4.1.2).\footnote{Recall that in Experiment 1 it was unnecessary to control for global entropy.}

4.1.1 Generating Datasets Using PFSAs

We construct random PFSAs with alphabet sizes \( |\Sigma| \in \{32, 48, 64\} \) and numbers of states \( |Q| \in \{16, 24, 32\} \). For each of the nine configurations, we randomly generate 25 automata. We do this by generating five random PFSA topologies (the underlying multi-graph) and five random weightings for each. See Algorithm 1 in App. B for details. We sample 20K strings for the training set, 5K for the validation set, and 5K for the test set from \( p_A \) for each PFSA \( \mathcal{A} \).

4.1.2 Evaluating Learning Difficulty

Using PFSAs allows us to compute a range of entropy-related values, including the inherent next-symbol entropy (\S2.3), which enables us to evaluate LMs based on KL divergence \( D_{KL} \). Specifically, the estimated \( \hat{D}_{KL} \) is given by subtracting the next-symbol entropy of the PFSA from the estimated next-symbol cross-entropy of the LM (Eq. (20)): \( \hat{D}_{KL} = \hat{H}_q(\overline{Y} \mid \overline{Y}) - H(\overline{Y} \mid \overline{Y}) \). In this second experiment, we evaluate each LM using \( \hat{D}_{KL} \) on the test set.

4.2 \( m \)-local Entropy and LM Performance

Figure 3 shows the relationship between the \( m \)-local entropy of PFSAs (calculated analytically; App. A) and the KL divergence of each neural LM on the test set; see Table 3 for Pearson correlation coefficients. The experimental results reveal a clear positive correlation between \( m \)-local entropy and \( \hat{D}_{KL} \) across both architectures and all values of \( m = 2, 3, 4, 5 \), indicating that neural LMs find it more challenging to model distributions with higher local uncertainty. The Transformer LM consistently shows higher \( \hat{D}_{KL} \) compared to the LSTM within each topological cluster, suggesting that LSTMs are more effective at modeling these particular probability distributions (Weiss et al., 2018; Borenstein et al., 2024). Additionally, when \( |\Sigma| \) is constant, \( \hat{D}_{KL} \) is higher for PFSAs with larger \( |Q| \), consistent with Borenstein et al. (2024).

5 Discussion and Conclusion

By proposing \( m \)-local entropy as a predictor of learning difficulty grounded in lossy-context surprisal theory and information locality principles, we provide a formal information-theoretic perspective that connects the inductive biases of LMs and
Figure 3: Scatter plots of symbol-level KL divergence (y-axis) versus \( m \)-local entropy (x-axis) for \( m \in \{2, 3, 4, 5\} \), for both LSTM (top row) and causally-masked Transformer encoder (Transformer; bottom row) models. Each marker type/color corresponds to a different combination of number of states (\(|Q|\)) and symbols (\(|\Sigma|\)). The dashed line is a linear fit for each cluster.

the statistical properties of language thought to be shaped by functional pressures in humans (Gibson, 2001; Futrell et al., 2020). Through two sets of experiments—one on perturbations of a natural language corpus and another using PFSAs for the controlled generation of synthetic languages—we consistently find that both LSTM and Transformer architectures model languages with lower \( m \)-local entropy more effectively. The shared sensitivity to information locality between artificial and human learners suggests a common inductive bias shaping both systems, possibly because both systems process language incrementally.

Our findings open several promising directions for future research. One avenue is to explore inductive biases beyond information locality, such as sensitivity to hierarchical structure or structure dependence (Chomsky, 1957; Everaert et al., 2015), in order to better understand the full range of factors influencing language learnability in both humans and machines. Additionally, incorporating local entropy into model evaluation or as a regularization signal during training could lead to more robust and cognitively plausible language models (Timkey and Linzen, 2023; De Varda and Marelli, 2024).

In summary, our study presents new evidence of the strong sensitivity of neural LMs to a language’s local statistical structure, advancing our understanding of their inductive biases and establishing a foundation for future research on assessing and improving the alignment between artificial and human language processors.

Limitations

While our study reveals a strong correlation between \( m \)-local entropy and LM performance, it is important to note that our analysis remains correlational. We have not yet pinpointed the precise mechanisms by which variations in local uncertainty impact the learning dynamics of neural language models. Additionally, our controlled experiments relied on PFSAs to generate languages with varied \( m \)-local entropy. Although PFSAs provide a tractable framework for such investigations, they capture only a limited set of possible language models. It is plausible that employing more expressive formalisms, such as pushdown automata or even more powerful models, might reveal different relationships between local entropy and model performance. In fact, empirical work has repeatedly suggested that some neural LM architectures do not have human-like inductive biases (McCoy et al., 2020; Yedetore et al., 2023, inter alia).

Furthermore, our focus on information locality, as measured by \( m \)-local entropy, does not preclude the influence of other inductive biases that may also play significant roles in learning. Future work will need to disentangle these factors to fully understand their individual and combined effects on neural LMs.
Ethical considerations

We employed AI-based tools (ChatGPT and GitHub Copilot) for writing and coding assistance. These tools were used in compliance with the ACL Policy on the Use of AI Writing Assistance.

Acknowledgments

AS was supported by an ETH AI Center Doctoral Fellowship. MG was supported by an ETH Zürich Postdoctoral Fellowship.

References

Kabir Ahuja, Vidhisha Balachandran, Madhur Panwar, Tianxing He, Noah A. Smith, Navin Goyal, and Yulia Tsvetkov. 2024. Learning syntax without planting trees: Understanding when and why transformers generalize hierarchically. ArXiv, abs/2404.16367.

Nadav Borenstein, Anej Svete, Robin Chan, Josef Valvoda, Franz Nowak, Isabelle Augenstein, Eleanor Chodroff, and Ryan Cotterell. 2024. What languages are easy to language-model? A perspective from learning probabilistic regular languages. Preprint, arXiv:2406.04289.

Alexandra Butoi, Ghazal Khalighinejad, Anej Svete, Josef Valvoda, Ryan Cotterell, and Brian DuSell. 2025. Training neural networks as recognizers of formal languages. In The Thirteenth International Conference on Learning Representations.

Eugene Charniak, Don Blaheta, Niyu Ge, Keith Hall, John Hale, and Mark Johnson. 2000. BLLIP 1987-89 WSJ Corpus Release 1.

Noam Chomsky. 1957. Syntactic Structures.

Noam Chomsky, Ian Roberts, and Jeffrey Watumull. 2023. Noam chomsky: The false promise of ChatGPT. The New York Times.

Leshem Choshen, Ryan Cotterell, Michael Y. Hu, Tal Linzen, Aaron Mueller, Candace Ross, Alex Warstadt, Ethan Wilcox, Adina Williams, and Chengxu Zhuang. 2024. The 2nd BabyLM challenge: Sample-efficient pretraining on a developmentally plausible corpus. Preprint, arXiv:2404.06214.

Andrea De Varda and Marco Marelli. 2024. Locally biased transformers better align with human reading times. In Proceedings of the Workshop on Cognitive Modeling and Computational Linguistics, pages 30–36, Bangkok, Thailand. Association for Computational Linguistics.

Martin B.H. Everaert, Marinus A.C. Huybregts, Noam Chomsky, Robert C. Berwick, and Johan J. Bolhuis. 2015. Structures, not strings: Linguistics as part of the cognitive sciences. Trends in Cognitive Sciences, 19(12):729–743.

Richard Futrell. 2019. Information-theoretic locality properties of natural language. In Proceedings of the First Workshop on Quantitative Syntax (Quasy, SyntaxFest 2019), pages 2–15, Paris, France. Association for Computational Linguistics.

Richard Futrell. 2023. Information-theoretic principles in incremental language production. Proceedings of the National Academy of Sciences of the United States of America, 120.

Richard Futrell, Edward Gibson, and Roger P. Levy. 2020. Lossy-context surprisal: An information-theoretic model of memory effects in sentence processing. Cognitive Science, 44(3):e12814.

Richard Futrell and Michael Hahn. 2024. Linguistic structure from a bottleneck on sequential information processing. ArXiv, abs/2405.12109.

Edward Gibson. 1998. Linguistic complexity: Locality of syntactic dependencies. Cognition, 68(1):1–76.

Edward Gibson. 2001. The dependency locality theory: A distance-based theory of linguistic complexity. In Image, Language, Brain: Papers from the First Mind Articulation Project Symposium. The MIT Press.

Michael Hahn, Judith Degen, and Richard Futrell. 2021. Modeling word and morpheme order in natural language as an efficient trade-off of memory and surprisal. Psychological Review, 128(4):726–756.

Michael Hahn, Richard Futrell, Roger Levy, and Edward Gibson. 2022. A resource-rational model of human processing of recursive linguistic structure. Proceedings of the National Academy of Sciences, 119(43):e2122602119.

John Hale. 2001. A probabilistic Earley parser as a psycholinguistic model. In Second Meeting of the North American Chapter of the Association for Computational Linguistics.

Kenneth Heafield. 2011. KenLM: Faster and smaller language model queries. In Proceedings of the Sixth Workshop on Statistical Machine Translation, pages 187–197, Edinburgh, Scotland. Association for Computational Linguistics.

Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735–1780.

Jennifer Hu, Jon Gauthier, Peng Qian, Ethan Wilcox, and Roger Levy. 2020. A systematic assessment of syntactic generalization in neural language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1725–1744, Online. Association for Computational Linguistics.

Julie Kallimi, Isabel Papadimitriou, Richard Futrell, Kyle Mahowald, and Christopher Potts. 2024. Mission: Impossible language models. In Proceedings of the 62nd Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pages 14691–14714, Bangkok, Thailand. Association for Computational Linguistics.

Diederik P. Kingma and Jimmy Lei Ba. 2015. Adam: A method for stochastic optimization. In The Third International Conference for Learning Representations, San Diego, California, USA.

Roger Levy. 2008. Expectation-based syntactic comprehension. Cognition, 106(3).

Luca Malagutti, Andrius Buinovskij, Anej Sveti, Clara Meister, Afra Amini, and Ryan Cotterell. 2024. The role of n-gram smoothing in the age of neural networks. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6882–6899, Mexico City, Mexico. Association for Computational Linguistics.

R. Thomas McCoy, Robert Frank, and Tal Linzen. 2020. Does syntax need to grow on trees? sources of hierarchical inductive bias in sequence-to-sequence networks. Transactions of the Association for Computational Linguistics, 8:125–140.

Tom M. Mitchell. 1980. The need for biases in learning generalizations. Technical Report CBM-TR 5-110, Rutgers University, New Brunswick, New Jersey, USA.

Andreas Opedal, Eleanor Chodroff, Ryan Cotterell, and Ethan Wilcox. 2024. On the role of context in reading time prediction. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 3042–3058, Miami, Florida, USA. Association for Computational Linguistics.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.

Steven Piantadosi. 2024. Modern language models refute Chomsky’s approach to language. LingBuzz Published In: Edward Gibson & Moshe Poliak (eds.), From fieldwork to linguistic theory: A tribute to Dan Everett (Empirically Oriented Theoretical Morphology and Syntax 15), 353–414. Berlin: Language Science Press. https://doi.org/10.5281/zenodo.12665933.

Jonathan Rawski and Jeffrey Heinz. 2019. No free lunch in linguistics or machine learning: Response to pater. Language, 95:e125 – e135.

C. E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal, 27(3):379–423.

William Timkey and Tal Linzen. 2023. A language model with limited memory capacity captures interference in human sentence processing. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8705–8720, Singapore. Association for Computational Linguistics.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.

Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 740–745, Melbourne, Australia. Association for Computational Linguistics.

Aditya Yedetore, Tal Linzen, Robert Frank, and R. Thomas McCoy. 2023. How poor is the stimulus? Evaluating hierarchical generalization in neural networks trained on child-directed speech. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 9370–9393, Toronto, Canada. Association for Computational Linguistics.
A  Probabilistic Finite-State Automata

Before listing a number of useful results for computing quantities of interest in PFSAs, we list a few relevant definitions.

Definition A.1. Let \( \mathcal{A} = (\Sigma, Q, \delta, \lambda, \rho) \) be a PFSA. We define the transition matrix \( M \in \mathbb{R}^{|Q| \times |Q|} \) of \( \mathcal{A} \) as the matrix containing the probabilities of transitioning from state \( q_i \in Q \) to state \( q_j \in Q \) in \( \mathcal{A} \) with any \( y \in \Sigma \):

\[
M_{i,j} \overset{\text{def}}{=} \sum_{y \in \Sigma} \sum_{q_i \xrightarrow{y/w} q_j \in \delta} w,
\]

where we fix some arbitrary enumeration of states \( (q_1, \ldots, q_{|Q|}) \). We also define the symbol-specific transition matrix \( M^{(y)} \) where \( M^{(y)}_{i,j} \) is the probability of transitioning from state \( q_i \in Q \) to state \( q_j \in Q \) in \( \mathcal{A} \) with a \( y \)-labeled transition:

\[
M^{(y)}_{i,j} \overset{\text{def}}{=} \sum_{q_i \xrightarrow{y/w} q_j \in \delta} w.
\]

We naturally extend this definition to strings and define for \( y = y_1 \cdots y_T \):

\[
M^{(y)} \overset{\text{def}}{=} M^{(y_1)} \cdots M^{(y_T)}.
\]

Furthermore, we define Kleene closure of \( M \) as

\[
M^* \overset{\text{def}}{=} \sum_{n=0}^{\infty} M^n,
\]

where the series above converges when the spectral radius satisfies \( \rho(M) < 1 \). Additionally, when the \( (I - M) \) is invertible, this is simply calculated as

\[
M^* = (I - M)^{-1}.
\]

Remark 1. It is a standard exercise to show that \( M^{(y)}_{i,j} \) equals the sum of the weights of \( y \)-scanning strings from \( q_i \) to \( q_j \).

Definition A.2. Let \( \mathcal{A} = (\Sigma, Q, \delta, \lambda, \rho) \) be a PFSA. The emission matrix \( E \in \mathbb{R}^{|Q| \times |\Sigma|} \) is defined by

\[
E_{i,k} \overset{\text{def}}{=} \sum_{q_i \xrightarrow{y_k/w} q' \in \delta} w.
\]

For a PFSA \( \mathcal{A} \) and a path \( \pi = q_0 \xrightarrow{y_1/w_1} \cdots \xrightarrow{y_N/w_N} q_N \in \Pi(\mathcal{A}) \), we write \( \iota(\pi) \overset{\text{def}}{=} q_0 \) for the initial state of the path and \( \varphi(\pi) \overset{\text{def}}{=} q_N \) for its final state. We define the path prefix random variable \( \overrightarrow{\Pi} \) distributed as

\[
\Pr\left( \overrightarrow{\Pi} = \pi \right) \propto \lambda(\iota(\pi)) \overline{w}(\pi).
\]

This is analogous to prefix string probabilities, and the distribution is normalizable exactly when prefix probabilities are. Similarly, we define \( \overrightarrow{\Pi}' \), which is distributed as

\[
\Pr\left( \overrightarrow{\Pi}' = \pi \right) \propto \sum_{\substack{\pi' \in \Pi(\mathcal{A}) \\ \varphi(\pi') = \iota(\pi)}} \lambda(\iota(\pi')) \overline{w}(\pi') \overline{w}(\pi),
\]

which is analogous to string infix probabilities.

PFSAs are particularly attractive to study since they allow us to exactly compute many interesting quantities efficiently. In the following section, we describe how one can compute the \( m \)-local entropy of the language model defined by a PFSA.
A.1 Useful Properties of Probabilistic Finite-State Automata

The following lemmata hold for a general PFSA \( \mathcal{A} = (\Sigma, Q, \delta, \lambda, \rho) \) and the language model \( p_{\mathcal{A}} \) induced by it. None of the results are novel, but we include full proofs for completeness.

Lemma A.1 (Computing the probability of a string with a PFSA). *The probability of \( y \in \Sigma^* \) is:*

\[
p_{\mathcal{A}}(y) = \lambda^\top M^{(y)} \rho.
\] (29)

*Proof.* We know from Remark 1 that \( M^{(y)}_{i,j} \) corresponds to the sum of the path weights from \( q_i \) to \( q_j \). Multiplying each entry with the source state’s initial weight and the target state’s final weight, we arrive at the result. \( \square \)

Lemma A.2 (Computing the prefix probability of a string). *The prefix probability of \( y \in \Sigma^* \) is:*

\[
\overrightarrow{p}_{\mathcal{A}}(y) = \lambda^\top M^{(y)} M^* \rho.
\] (30)

*Proof.*

\[
\begin{align*}
\overrightarrow{p}_{\mathcal{A}}(y) &= \sum_{y' \in \Sigma^*} p_{\mathcal{A}}(yy') \\
&= \sum_{y' \in \Sigma^*} \lambda^\top M^{(yy')} \rho & \text{(Lemma A.1, 31b)} \\
&= \sum_{y' \in \Sigma^*} \lambda^\top M^{(y)} M^{(y')} \rho & \text{(31c)} \\
&= \lambda^\top M^{(y)} \left( \sum_{y' \in \Sigma^*} M^{(y')} \right) \rho & \text{(31d)} \\
&= \lambda^\top M^{(y)} M^* \rho & \text{(31e)}
\end{align*}
\]

\( \square \)

Lemma A.3 (Computing the next-symbol distribution). *Let \( y \in \Sigma^* \). The distribution over the next symbols after observing \( y \) is*

\[
\Pr(y_k \mid s(\vec{\Pi}) = y) = \frac{(\lambda^\top M^{(y)} E)_k}{\overrightarrow{p}_{\mathcal{A}}(y)}.
\] (32)

*Proof.*

\[
\begin{align*}
\Pr(y_k \mid s(\vec{\Pi}) = y) &= \frac{\Pr\left(y_k, s(\vec{\Pi}) = y\right)}{\Pr\left(s(\vec{\Pi}) = y\right)} \\
&= \frac{1}{\overrightarrow{p}_{\mathcal{A}}(y)} \sum_{\pi \in \Pi(\mathcal{A}, y)} \lambda(\iota(\pi)) \overline{w}(\pi) \Pr\left(y_k \mid \varphi(\pi)\right) & \text{(Summing over all \( y \)-yielding paths, 33b)} \\
&= \frac{1}{\overrightarrow{p}_{\mathcal{A}}(y)} \sum_{j=1}^{|Q|} \Pr\left(y_k \mid q_j\right) \sum_{\substack{\pi \in \Pi(\mathcal{A}, y) \\ \varphi(\pi) = q_j}} \lambda(\iota(\pi)) \overline{w}(\pi) & \text{(33c)} \\
&= \frac{1}{\overrightarrow{p}_{\mathcal{A}}(y)} \sum_{j=1}^{|Q|} \Pr\left(y_k \mid q_j\right) \sum_{i=1}^{|Q|} \lambda(q_i) \sum_{\substack{\pi \in \Pi(\mathcal{A}, y) \\ d(\pi) = q_i, \varphi(\pi) = q_j}} \overline{w}(\pi) & \text{(33d)}
\end{align*}
\]
= \frac{1}{\overrightarrow{p}_A(\mathbf{y})} \sum_{j=1}^{|Q|} \Pr(y_k \mid q_j) \sum_{i=1}^{|Q|} \lambda(q_i) M^{(y)}_{i,j} \tag{33e}
= \frac{1}{\overrightarrow{p}_A(\mathbf{y})} \sum_{j=1}^{|Q|} \Pr(y_k \mid q_j) \left( \lambda^\top M^{(y)} \right)_j \tag{33f}
= \frac{1}{\overrightarrow{p}_A(\mathbf{y})} \sum_{j=1}^{|Q|} \left( \lambda^\top M^{(y)} \right)_j w \tag{q_i \xrightarrow{y_k/w} q' \in \delta, 33g}
= \frac{1}{\overrightarrow{p}_A(\mathbf{y})} \left( \lambda^\top M^{(y)} E \right)_k \tag{(Eq. (26), 33h)}

Lemma A.4 (Computing the infix probability of a string). The infix probability of \( y \in \Sigma^* \) is:
\[
\overrightarrow{p}_A(y) = \lambda^\top M^* M^{(y)} M^* \rho.
\] (34)

Proof.

\[
\overrightarrow{p}_A(y) = \sum_{y' \in \Sigma^*} \overrightarrow{p}_A(y'/y) \tag{35a}
= \sum_{y' \in \Sigma^*} \lambda^\top M^{(y'y)} M^* \rho \tag{Lemma A.2, 35b}
= \lambda^\top \left( \sum_{y' \in \Sigma^*} M^{(y')} \right) M^{(y)} M^* \rho \tag{35c}
= \lambda^\top M^* M^{(y)} M^* \rho \tag{35d}

Lemma A.5 (Computing the infix next-symbol distribution). Let \( c \in \Sigma^{m-1} \). The distribution over the next symbols after observing \( c \) as the last \( m-1 \) symbols is
\[
\Pr(y_k \mid s(\overrightarrow{\Pi}) = c) = \frac{(\lambda^\top M^* M^{(c)} E)_k}{\overrightarrow{p}_A(c)}. \tag{36}
\]

Proof.

\[
\Pr(y_k \mid s(\overrightarrow{\Pi}) = c) = \frac{\Pr\left( y_k, s(\overrightarrow{\Pi}) = c \right)}{\Pr\left( s(\overrightarrow{\Pi}) = c \right)} \tag{37a}
= \frac{1}{\overrightarrow{p}_A(c)} \sum_{y' \in \Sigma^*} \sum_{\pi \in \Pi(A, y'c)} \lambda(\iota(\pi)) \overline{w}(\pi) \Pr\left( y_k \mid \varphi(\pi) \right) \tag{Summing over all c-ending paths, 37b}
= \frac{1}{\overrightarrow{p}_A(c)} \sum_{j=1}^{|Q|} \Pr(y_k \mid q_j) \sum_{y' \in \Sigma^*} \sum_{\substack{\pi \in \Pi(A, y'c) \\ \varphi(\pi) = q_j}} \lambda(\iota(\pi)) \overline{w}(\pi) \tag{37c}
= \frac{1}{\overrightarrow{p}_A(c)} \sum_{j=1}^{|Q|} \Pr(y_k \mid q_j) \sum_{i=1}^{|Q|} \sum_{y' \in \Sigma^*} \lambda(q_i) \sum_{\substack{\pi \in \Pi(A, y'c) \\ \iota(\pi) = q_i, \varphi(\pi) = q_j}} \overline{w}(\pi) \tag{37d}
\[
= \frac{1}{\overrightarrow{p}_A(c)} \sum_{j=1}^{|Q|} \Pr(y_k \mid q_j) \sum_{i=1}^{|Q|} \sum_{y' \in \Sigma^*} \lambda(q_i) \left( M^{(y')} M^{(c)} \right)_{i,j} \tag{37e}
\]
\[
= \frac{1}{\overrightarrow{p}_A(c)} \sum_{j=1}^{|Q|} \Pr(y_k \mid q_j) \sum_{i=1}^{|Q|} \lambda(q_i) \sum_{y' \in \Sigma^*} \left( M^{(y')} M^{(c)} \right)_{i,j} \tag{37f}
\]
\[
= \frac{1}{\overrightarrow{p}_A(c)} \sum_{j=1}^{|Q|} \Pr(y_k \mid q_j) \sum_{i=1}^{|Q|} \lambda(q_i) \left( M^* M^{(c)} \right)_{i,j} \tag{37g}
\]
\[
= \frac{1}{\overrightarrow{p}_A(c)} \sum_{j=1}^{|Q|} \Pr(y_k \mid q_j) \left( \lambda^\top M^* M^{(y)} \right)_j \tag{37h}
\]
\[
= \frac{1}{\overrightarrow{p}_A(c)} \sum_{j=1}^{|Q|} \left( \lambda^\top M^* M^{(y)} \right)_j w \quad (q_i \xrightarrow{y_k/w} q' \in \delta, 37i)
\]
\[
= \frac{1}{\overrightarrow{p}_A(c)} \left( \lambda^\top M^* M^{(y)} E \right)_k \tag{Eq. (26), 37j}
\]

Lemma A.6 (Computing the m-local entropy of a DPFSA). *The m-local entropy of \( p_A \) can be computed in time \( O\left(m|Q|^3|\Sigma|^{m-1}\right) \).*

*Proof.* The m-local entropy of \( p_A \) can be computed as
\[
\mathrm{H}_m\left(p_A\right) = \mathbb{E}_{c \sim \Pr(C=c)} \left[ \mathrm{H}(\overline{Y} \mid C = c) \right] \tag{38a}
\]
\[
= \frac{1}{Z} \sum_{c \in \Sigma^{m-1}} \overrightarrow{p}_A(c) \mathrm{H}(\overline{Y} \mid C = c) \tag{38b}
\]
The terms \( \overrightarrow{p}_A(c) \) and \( \mathrm{H}(\overline{Y} \mid C = c) \) can be computed in time \( O\left(m|Q|^3\right) \) as per Lemmata A.4 and A.5 for each \( c \in \Sigma^{m-1} \). Computing this for each \( c \) individually, we arrive at the claimed complexity. \( \Box \)
B  Generating Random PFSA

Algorithm 1 and the subprocedure in Algorithm 2 describe our PFSA generation procedure.

Algorithm 1 Generate a Random DPFSA.
Input: \(|Q|\) (number of states), \(|\Sigma|\) (number of symbols), \(\mu\) (target mean string length), \(R_T\) (topology random generator), and \(R_W\) (weight random generator).
Output: A PFSA \(A\) with randomly assigned transitions and normalized weights with \(|Q|\) states and \(|\Sigma|\) symbols.
Note: CHOICE\((R, S, 1)\) denotes selecting one element uniformly at random from the set \(S\) using the random generator \(R\).
unused is initialized as \(Q\), and out-arcs is a mapping that assigns to each state a subset of \(\Sigma\) (the allowed outgoing symbols).
For exponential sampling, we write \(w \sim \mathrm{Exp}(0.1)\) to denote that \(w\) is drawn from an exponential distribution with rate 0.1, i.e., with density \(f(w) = 0.1 e^{-0.1w}\) for \(w \geq 0\).

function RANDOMDPFSA(|Q|, |\Sigma|, \mu, R_T, R_W)
    q_t \leftarrow \mathrm{CHOICE}(R_T, Q, 1)
    Initialize \(\mathcal{A} \leftarrow (\Sigma, Q, \delta, \lambda, \rho)\)
    Initialize \(\lambda \leftarrow 0_{|Q|}\) and set \(\lambda(q_t) \leftarrow 1\)
    Initialize \(M^{(y)}\) to a \(|Q| \times |Q|\) matrix of zeros for \(y \in \Sigma\)
    unused \leftarrow Q
    state-outgoing-symbols \leftarrow \mathrm{GetOutgoingSymbols}(Q, \Sigma, R_T)
    for \(q \in Q\) do
        for \(y \in \Sigma\) do
            if unused \neq \emptyset then
                \(q' \leftarrow \mathrm{CHOICE}(R_T, \text{unused}, 1)\)
                Remove \(q'\) from unused
            else
                \(q' \leftarrow \mathrm{CHOICE}(R_T, Q, 1)\)
                Let \(w \sim \mathrm{Exp}(0.1)\)
                \(M^{(y)}_{q,q'} \leftarrow w \cdot 1\{y \in \text{state-outgoing-symbols}[q]\} + 0.001\)
        end for
        for \(q \in Q\) do      // Set final weights and normalize outgoing weights for each state.
            \(t \leftarrow \sum_{y=0}^{|\Sigma|-1} \mathrm{sum}(M^{(y)}_{q,:})\)
            \(\rho(q) \leftarrow t / \mu\)
            \(s \leftarrow t + \rho(q)\)
            for \(y \in \{0, \ldots, |\Sigma| - 1\}\) do
                \(M^{(y)}_{q,:} \leftarrow M^{(y)}_{q,:}/s\)
            end for
            \(\rho(q) \leftarrow \rho(q)/s\)
        end for
    end for
return \(\mathcal{A}\)

C  Details of Neural Language Models

C.1 Transformer
We use a 4-layer causally-masked Transformer with 768-dimensional embeddings, 3,072-dimensional feedforward layers, and 12 attention heads, implemented in PyTorch. Following Vaswani et al. (2017), we map input symbols to vectors of size 768 with a scaled embedding layer and add sinusoidal positional encodings. We use pre-norm instead of post-norm and apply layer norm to the output of the last layer. We use the same dropout rate throughout the Transformer. We apply it in the same places as Vaswani et al. (2017), and, as implemented by PyTorch, we also apply it to the hidden units of feedforward sublayers
Algorithm 2 Generate Outgoing Symbols for Each State.
Input: \(|Q|\) (number of states), \(|\Sigma|\) (number of symbols), \(R\) (random generator), and \(s_{min}\) (min. unique symbols per state; default: 2).
Output: A list \(S\) of \(|Q|\) sets, each containing outgoing symbols.
Note: CHOICE\((R, X, k)\) selects \(k\) distinct elements uniformly at random from \(X\) using \(R\), and INTEGERS\((R, a, b)\) returns a random integer in \([a, b)\).

1: function GetOutgoingSymbols\((Q, \Sigma, R, s_{min})\)
2: Initialize state-outgoing-symbols \(←\) an array of \(|Q|\) empty sets
3: for \(q \in Q\) do
4:   s \(←\) CHOICE\((R, \mathcal{N}, \min(s_{min}, |\Sigma|))\) \hspace{1cm} ▷ Assign each state at least \(s_{min}\) symbols.
5:   state-outgoing-symbols\[q\] \(←\) state-outgoing-symbols\[q\] \(\cup\) s
6: for \(y \in \Sigma\) do \hspace{1cm} ▷ Ensure each symbol appears in at least one set.
7:   q \(←\) CHOICE\((R, Q, 1)\)
8:   Add \(y\) to state-outgoing-symbols[q]
9:   \(M \leftarrow \sum_{q=0}^{|Q|-1}\) INTEGERS\((R, 0, \max(1, |\Sigma|/2 - s_{min}))\)
10: for \(j \leftarrow 1\) to \(M\) do \hspace{1cm} ▷ Add random transitions.
11:   y \(←\) CHOICE\((R, \Sigma, 1)\)
12:   q \(←\) CHOICE\((R, Q, 1)\)
13:   Add \(y\) to state-outgoing-symbols[q]
14: return state-outgoing-symbols

and to the attention probabilities of scaled dot-product attention operations. We always use BOS as the first input symbol to the Transformer.

C.2 LSTM
We use a single-layer LSTM (Hochreiter and Schmidhuber, 1997) with 512-dimensional hidden units, implemented in PyTorch with some modifications as in Butoi et al. (2025).

\[
\begin{align*}
h_t^{(0)} &\overset{\mathrm{def}}{=} x_t = E_{w_t} \tag{39a} \\
h_0^{(\ell)} &\overset{\mathrm{def}}{=} \tanh(w_0^{(\ell)}) \tag{39b} \\
h_t^{(\ell)} &\overset{\mathrm{def}}{=} \text{DROPOUT}(h_t^{(\ell-1)}) \tag{39c} \\
i_t^{(\ell)} &\overset{\mathrm{def}}{=} \sigma(W_i^{(\ell)} \begin{bmatrix} h_t^{(\ell-1)} \\ h_{t-1}^{(\ell)} \end{bmatrix} + b_i^{(\ell)}) \tag{39d} \\
f_t^{(\ell)} &\overset{\mathrm{def}}{=} \sigma(W_f^{(\ell)} \begin{bmatrix} h_t^{(\ell-1)} \\ h_{t-1}^{(\ell)} \end{bmatrix} + b_f^{(\ell)}) \tag{39e} \\
g_t^{(\ell)} &\overset{\mathrm{def}}{=} \tanh(W_g^{(\ell)} \begin{bmatrix} h_t^{(\ell-1)} \\ h_{t-1}^{(\ell)} \end{bmatrix} + b_g^{(\ell)}) \tag{39f} \\
o_t^{(\ell)} &\overset{\mathrm{def}}{=} \sigma(W_o^{(\ell)} \begin{bmatrix} h_t^{(\ell-1)} \\ h_{t-1}^{(\ell)} \end{bmatrix} + b_o^{(\ell)}) \tag{39g} \\
c_t^{(\ell)} &\overset{\mathrm{def}}{=} f_t^{(\ell)} \odot c_{t-1}^{(\ell)} + i_t^{(\ell)} \odot g_t^{(\ell)} \tag{39h} \\
h_t^{(\ell)} &\overset{\mathrm{def}}{=} o_t^{(\ell)} \odot \tanh(c_t^{(\ell)}) \tag{39i} \\
c_0^{(\ell)} &\overset{\mathrm{def}}{=} 0 \tag{39j} \\
h_t &\overset{\mathrm{def}}{=} h_t^{(L)} \tag{39k}
\end{align*}
\]
Here, \( E \) is an embedding matrix to map each symbol \( w_t \) of the input string to an embedding \( x_t = E_{w_t} \). The size of the embeddings is always \( d \) (the size of the hidden vectors), and we denote the number of layers in the model as \( L \). Also, \( \odot \) denotes elementwise multiplication, and DROPOUT(\( \cdot \)) indicates the application of dropout. Here, \( w_0^{(\ell)} \in \mathbb{R}^d \) is a learned parameter, making the initial hidden state \( h_0^{(\ell)} \) of each layer learned. A modification is made from the original PyTorch implementation: each pair of \( b_{ii} \) and \( b_{hi} \), \( b_{ij} \) and \( b_{hf} \), \( b_{io} \) and \( b_{ho} \), and \( b_{io} \) and \( b_{ho} \) is replaced with a single bias parameter per layer.

D  Hyperparameters for Neural Language Model Training

Wherever dropout is applicable, we use a dropout rate of 0.1. For layer norm, we initialize weights to 1 and biases to 0. We initialize all other parameters by sampling uniformly from \([−0.1, 0.1]\).

For each epoch, we randomly shuffle the training set and group strings of similar lengths into the same minibatch, enforcing an upper limit of 2,048 symbols per batch, including padding, BOS, and EOS symbols. We train each model by minimizing cross-entropy on the validation set using Adam (Kingma and Ba, 2015). We clip gradients with a threshold of 5 using \( L^2 \) norm rescaling. We take a checkpoint every 10K examples, at which point we evaluate the model on the validation set and update the learning rate and early stopping schedules. We multiply the learning rate by 0.5 after 5 checkpoints of no decrease in cross-entropy on the validation set, and we stop early after 10 checkpoints of no decrease. We select the checkpoint with the lowest cross-entropy on the validation set when reporting results. We train for a maximum of 1K epochs.

E  Pearson Correlation Coefficients between \( M \)-local Entropy and KL Divergence

Table 3 reports the Pearson correlation coefficients between the \( m \)-local entropy of PFSA and the estimated KL divergence (\( \tilde{D}_{KL} \); §4.1.2) in §4.

<table>
  <tr>
    <th rowspan="2">ARCHITECTURE</th>
    <th rowspan="2">M</th>
    <th colspan="2">16</th>
    <th colspan="2">24</th>
    <th colspan="2">32</th>
  </tr>
  <tr>
    <th>Q</th>
    <th>|Σ|</th>
    <th>Q</th>
    <th>|Σ|</th>
    <th>Q</th>
    <th>|Σ|</th>
  </tr>
  <tr>
    <td rowspan="5">LSTM</td>
    <td>2</td>
    <td>0.433</td>
    <td>0.501</td>
    <td>0.137</td>
    <td>0.291</td>
    <td>0.583</td>
    <td>0.121</td>
    <td>0.423</td>
    <td>0.311</td>
    <td>0.623</td>
  </tr>
  <tr>
    <td>3</td>
    <td>0.396</td>
    <td>0.532</td>
    <td>0.230</td>
    <td>0.291</td>
    <td>0.488</td>
    <td>0.119</td>
    <td>0.460</td>
    <td>0.274</td>
    <td>0.702</td>
  </tr>
  <tr>
    <td>4</td>
    <td>0.412</td>
    <td>0.546</td>
    <td>0.236</td>
    <td>0.311</td>
    <td>0.477</td>
    <td>0.122</td>
    <td>0.474</td>
    <td>0.283</td>
    <td>0.702</td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.415</td>
    <td>0.554</td>
    <td>0.234</td>
    <td>0.338</td>
    <td>0.472</td>
    <td>0.120</td>
    <td>0.466</td>
    <td>0.283</td>
    <td>0.686</td>
  </tr>
  <tr>
    <td rowspan="5">TRANSFORMER</td>
    <td>2</td>
    <td>0.740</td>
    <td>0.679</td>
    <td>0.455</td>
    <td>0.290</td>
    <td>0.658</td>
    <td>0.844</td>
    <td>0.551</td>
    <td>0.622</td>
    <td>0.709</td>
  </tr>
  <tr>
    <td>3</td>
    <td>0.743</td>
    <td>0.728</td>
    <td>0.569</td>
    <td>0.374</td>
    <td>0.735</td>
    <td>0.859</td>
    <td>0.693</td>
    <td>0.832</td>
    <td>0.820</td>
  </tr>
  <tr>
    <td>4</td>
    <td>0.737</td>
    <td>0.674</td>
    <td>0.549</td>
    <td>0.389</td>
    <td>0.727</td>
    <td>0.844</td>
    <td>0.668</td>
    <td>0.848</td>
    <td>0.799</td>
  </tr>
  <tr>
    <td>5</td>
    <td>0.717</td>
    <td>0.614</td>
    <td>0.516</td>
    <td>0.333</td>
    <td>0.705</td>
    <td>0.830</td>
    <td>0.625</td>
    <td>0.833</td>
    <td>0.770</td>
  </tr>
</table>

Table 3: Pearson correlation coefficients between \( m \)-local entropy and KL divergence for different architectures, number of states \( |Q| \), and alphabet sizes \( |\Sigma| \).

F  Additional Experiments with BabyLM Corpus

We also conducted the same set of experiments using the BabyLM corpus (Choshen et al., 2024). Table 4 and Figure 4 show our experimental results. They show the same trends as in our main experiment (§3), but with slightly different tendencies for the REVERSE language.

G  Computational Resources

Across all experiments, we used a total of approximately 717.5 GPU hours. Training was conducted on NVIDIA GeForce RTX 4090 24GB and NVIDIA Quadro RTX 6000 24GB GPUs.

H  License of the Data

The BLLIP corpus (Charniak et al., 2000) is used under the terms of the BLLIP 1987-89 WSJ Corpus Release 1 License Agreement.
<table>
  <tr>
    <th></th>
    <th>2-local entropy</th>
    <th>3-local entropy</th>
    <th>4-local entropy</th>
    <th>5-local entropy</th>
  </tr>
  <tr>
    <th>BASE</th>
    <td>5.78</td>
    <td>3.72</td>
    <td>2.69</td>
    <td>2.32</td>
  </tr>
  <tr>
    <th>REVERSE</th>
    <td>6.50</td>
    <td>3.87</td>
    <td>2.78</td>
    <td>2.43</td>
  </tr>
  <tr>
    <th>EVENODDSHUFFLE</th>
    <td>6.82</td>
    <td>4.47</td>
    <td>3.36</td>
    <td>3.14</td>
  </tr>
  <tr>
    <th>ODDEVENSHUFFLE</th>
    <td>6.94</td>
    <td>4.45</td>
    <td>3.39</td>
    <td>3.14</td>
  </tr>
  <tr>
    <th>LOCALSHUFFLE (k=3)</th>
    <td>6.99 ± 0.16</td>
    <td>4.27 ± 0.08</td>
    <td>3.21 ± 0.07</td>
    <td>2.97 ± 0.08</td>
  </tr>
  <tr>
    <th>LOCALSHUFFLE (k=4)</th>
    <td>7.09 ± 0.15</td>
    <td>4.35 ± 0.06</td>
    <td>3.25 ± 0.03</td>
    <td>3.06 ± 0.04</td>
  </tr>
  <tr>
    <th>LOCALSHUFFLE (k=5)</th>
    <td>7.15 ± 0.13</td>
    <td>4.42 ± 0.06</td>
    <td>3.29 ± 0.04</td>
    <td>3.08 ± 0.03</td>
  </tr>
  <tr>
    <th>LOCALSHUFFLE (k=6)</th>
    <td>7.25 ± 0.11</td>
    <td>4.47 ± 0.07</td>
    <td>3.35 ± 0.06</td>
    <td>3.14 ± 0.05</td>
  </tr>
  <tr>
    <th>LOCALSHUFFLE (k=7)</th>
    <td>7.28 ± 0.12</td>
    <td>4.50 ± 0.08</td>
    <td>3.39 ± 0.07</td>
    <td>3.19 ± 0.08</td>
  </tr>
  <tr>
    <th>DETERMINISTICSHUFFLE</th>
    <td>7.41</td>
    <td>4.69</td>
    <td>3.59</td>
    <td>3.40</td>
  </tr>
</table>

Table 4: M-local entropy values for BASE (original) corpus and different transformed corpora. “Local shuffle” refers to the K-LOCALDETERMINISTICSHUFFLE. Values are shown as mean ± standard deviation (averaged over different random seeds).

![Scatter plots of next-symbol cross-entropy (y-axis) versus m-local entropy (x-axis) for m ∈ {2, 3, 4, 5}, for both LSTM (top row) and causally-masked Transformer encoder (Transformer; bottom row) models. Each marker type/color corresponds to a different perturbation (e.g., Reverse, DeterministicShuffle, K-LOCALDETERMINISTICSHUFFLE with various window sizes, etc.). The red star indicates the unperturbed BASE condition (original corpus). The dashed line in each panel is a linear fit, with R^2 indicating the coefficient of determination.](page_120_1042_1208_410.png)

Figure 4: Scatter plots of next-symbol cross-entropy (y-axis) versus m-local entropy (x-axis) for \( m \in \{2, 3, 4, 5\} \), for both LSTM (top row) and causally-masked Transformer encoder (Transformer; bottom row) models. Each marker type/color corresponds to a different perturbation (e.g., Reverse, DeterministicShuffle, K-LOCALDETERMINISTICSHUFFLE with various window sizes, etc.). The red star indicates the unperturbed BASE condition (original corpus). The dashed line in each panel is a linear fit, with \( R^2 \) indicating the coefficient of determination.

Metareview score: 4
Metareview: This paper shows that sequence prediction models (LSTMs and Transformers) have a learning bias towards formal languages with strong local predictability. This is formalized using the measure of $m$-local entropy. A controlled setup allows varying this, while controlling global entropy. The results suggest that these sequence models have a fundamental learning bias that shows some alignment to human language processing.
Summary of reasons to publish: * The paper addresses a question foundational to natural language, to NLP, and to modern ML architectures used in NLP, linking their inductive biases to recent results from psycholinguistics 
* Results are shown for two architectures (LSTMs and Transformers) [Gmha]
* The paper is well written [Rw6P]
* Experiments are well-controlled; in particular, keeping global entropy invariant is a key part here [JsJk]