What Languages are Easy to Language-Model? A Perspective from Learning Probabilistic Regular Languages (2024)

Abstract: What can large language models learn? By definition, language models (LM) are distributions over strings. Therefore, an intuitive way of addressing the above question is to formalize it as a matter of learnability of classes of distributions over strings. While prior work in this direction focused on assessing the theoretical limits, in contrast, we seek to understand the empirical learnability. Unlike prior empirical work, we evaluate neural LMs on their home turf—learning probabilistic languages—rather than as classifiers of formal languages. In particular, we investigate the learnability of regular LMs (RLMs) by RNN and Transformer LMs. We empirically test the learnability of RLMs as a function of various complexity parameters of the RLM and the hidden state size of the neural LM. We find that the RLM rank, which corresponds to the size of linear space spanned by the logits of its conditional distributions, and the expected length of sampled strings are strong and significant predictors of learnability for both RNNs and Transformers. Several other predictors also reach significance, but with differing patterns between RNNs and Transformers.

heading: 1 Introduction
text: Language models are, definitionally, distributions over strings. However, not all neural LMs are capable of learning—or even representing—all possible distributions. This raises two natural questions: What classes of distributions can neural LMs represent and what can they learn from training examples? In terms of the first question, the relationship between recurrent neural networks and more symbolic computational models has been subject to study for over three decades (McCulloch and Pitts, 1943; Kleene, 1956; Siegelmann and Sontag, 1992; Hao et al., 2018; Korsky and Berwick, 2019; Merrill, 2019; Merrill et al., 2020; Hewitt et al., 2020; Chung and Siegelmann, 2021; Merrill et al., 2022; Merrill and Tsilivis, 2022; Svete and Cotterell, 2023; Nowak et al., 2023). Moreover, the
prevalence of Transformer-based LMs has led to a recent body of work investigating their representational capacity (e.g., Hahn, 2020; Ebrahimi et al., 2020; Bhattamishra et al., 2020; Merrill and Sabharwal, 2023b). However, almost all of this work is theoretical, i.e., researchers seek theorems that give exact limitations on the capacity of specific neural LMs. While such work provides a good characterization of what neural LMs could, in principle, learn, it does not speak to what LMs can learn in practice.
In contrast to the above, a more empirically minded researcher might prefer to run a series of controlled experiments to determine what neural LMs can and cannot learn. Their goal is to empirically characterize what classes of formal LMs, e.g., probabilistic finite-state automata, are learnable with neural LMs in practice, using current best practices. Such work can inform our understanding of what types of languages larger LMs trained on human-written text might represent—specifically, what grammatical structures they can recognize, and how efficiently they can do so. All of the above is crucial for quantifying the practical capabilities, and limits, of neural LMs. While plenty of empirical work has provided insights into the linguistic capabilities of modern LMs using linguistically annotated datasets (e.g., Linzen et al., 2016; Hewitt and Manning, 2019; Jawahar et al., 2019; Liu et al., 2019; Icard, 2020; Manning et al., 2020; Rogers et al., 2021; Belinkov, 2022),
15115
human-annotated datasets give us limited insight into the types of distributions a neural LM can learn because the true distribution that the neural LMs is modeling is often unclear. For instance, fitting an LM to Wikipedia leaves open to interpretation exactly which probability distribution over strings the neural LM is modeling. In contrast, learning a probabilistic formal language in a controlled experiment offers an unparalleled level of control.
A closer look at the existing work on the empirical learnability of formal languages (see App. A for an overview) reveals a categorical mismatch between what LMs are, i.e., probability distributions over strings, and what learning a formal language means, i.e., classifying strings as members of a specific language, i.e., a set of strings (Ebrahimi et al., 2020; Delétang et al., 2023; Wang and SteinertThrelkeld, 2023). In response, we propose to investigate the practical representation capacity of neural LMs by testing their ability to learn distributions over strings. By sampling training corpora from probabilistic finite-state automata (PFSA), we can ask precise questions about the learnability.
We offer an empirical study, sampling datasets of 20k strings from 2100 randomly generated PFSAs, and training 15k RNN and Transformer language models with a varying hidden state size on datasets. Our study is informed by various theoretical results on the representational capacity of RNNs concerning probabilistic finite-state automata. We assess the learnability by approximating the KL divergence between PFSAs and neural LMs. In a regression analysis, we find that a large number of properties of the automaton, e.g., the number of states, the number of transitions, the rank of its emission matrix, and its entropy all contribute to learnability. In particular, the rank of the emission matrix and expected length of the sampled strings are strong predictors of learnability for both RNN and Transformer LMs. Several other predictors also demonstrate relevance in accounting for learnability, but have differing magnitudes and patterns of significance for the RNN and Transformer LMs. Similar to Delétang et al. (2023), we find that RNNs are better suited to modeling formal languages than Transformers.

heading: 2 Preliminaries
text: We begin with an introduction of the relevant mathematical preliminaries, based on Cotterell et al. (2024). An alphabet Σ is a finite, non-empty set of
symbols. The Kleene closure Σ˚ of the alphabet Σ is the set of all strings of the symbols in Σ. The length of the string y “ y1 . . . yT P Σ˚, denoted by |y| “ T , is the number of symbols the string contains. A language model p is a probability distribution over Σ˚. Two LMs p and q are equivalent if p pyq “ q pyq for all y P Σ˚. Most modern LMs define p pyq as a product of conditional probability distributions:
p pyq def“ p pEOS | yq |y|ź
t“1 p pyt | yătq , (1)
where EOS R Σ is a special end of sequence symbol. We denote Σ def“ Σ Y tEOSu and y an element of Σ.

heading: 2.1 Neural Language Models
text: Representation-based neural LMs define the conditional distributions p pyt | yătq through a linearly transformed hidden state. In this paper, we focus on softmax-normalized, representationbased neural LMs, where a language encoder h : Σ˚ Ñ RD (Chan et al., 2024) computes a representation, and the conditional distributions of the neural LMs are defined as follows
p pyt | yătq def“ softmaxpEhpyătqqyt (2a) def“ exp pEhpyătqqytř
yPΣ exp pEhpyătqqy . (2b)
We will call E P R|Σ|ˆD the output matrix. Representation-based neural LMs differ in how h is computed as a function of yăt. In this paper, we consider the two most popular modern language modeling architectures: recurrent neural networks (Elman, 1990), specifically the LSTM variant (Hochreiter and Schmidhuber, 1997), where h is computed recurrently, and Transformers (Vaswani et al., 2017), where h is computed with self-attention.

heading: 2.2 Regular Language Models
text: A classic formalism for defining LMs is probabilistic finite-state automata (PFSAs), a probabilistic version of finite-state automata that defines string probabilities. Intuitively, a PFSA defines a finite number of conditional next-symbol distributions p py | qq based on a finite number of states q P Q that summarize string prefixes analogous to how the hidden state h of an RNN summarizes the prefix y1 . . . yt.
A PFSA moves between its states based on the input symbols according to the transitions defined by a transition relation. It accepts a string with the probability equal to the product of the transition weights along the string’s path in the automaton and the last state’s final weight (or the sum over all paths if there are multiple paths accepting the string).1 The PFSA is deterministic (a DPFSA) if the transition relation is a function of the current state and symbol, i.e., if, for all q P Q and y P Σ, there exists at most one q1 P Q such that p pq1 | q, yq ą 0. A DPFSA is minimal if it has the smallest number of states among all its equivalent DPFSAs.2 The minimal DPFSA is unique up to the naming of the states.
Definition 2.1. The LM p is regular (an RLM) if there exists an equivalent PFSA.
Fig. 1 shows an example of an RLM defining a distribution over ta, bu˚ with p pabnabmq “ 1¨0.6¨ 0.1n ¨0.9¨0.7m ¨0.3 and p pbbmq “ 1¨0.4¨0.7m ¨0.3.

heading: 3 Representing RLMs with Neural LMs
text: Neural LMs have demonstrated an ability to model human language well. This raises the question of what enables their success in capturing human language and how we can improve them further. However, neural LMs are notoriously challenging to analyze, making it difficult to state any formal claims on what they are (in)capable of modeling. To amend this, a large body of work has linked neural LMs to formal models of computation. DPFSAs feature prominently in this line of research (Merrill, 2019; Merrill et al., 2020; Svete and Cotterell, 2023). To facilitate a detailed inspection of how neural LMs can represent DPFSAs, we now formalize DPFSAs in a way that is particularly easy to connect to softmax-normalized representation-based neural LMs, see App. B for the definition of DPFSA.

heading: 3.1 Softmax-Normalized DPFSAs
text: The conditional distributions p py | qq defined by a DPFSA can, in general, be arbitrary distributions over Σ—a DPFSA therefore defines |Q| distributions, each with |Σ| ´ 1 degrees of freedom. As we will see, such a parameterization makes the connection to neural LMs, which define conditional distributions in terms of shared parameters of the neural
1Final weights of states are analogous to the EOS symbol which signals the end of string generation in neural LMs.
2See App. B for more formal details.
network and the output matrix E, straightforward. To make this connection clear, consider a DPFSA A with support over Σ˚. Because A is deterministic, there exists a function hA : Σ˚ Ñ t0, 1u|Q| that maps every string that the unique state A enters to a one-hot encoding after reading in the string. Encoding the final weight as EOS, we end up with the following autoregressive formulation of A,
p pyt | yătq def“ softmaxpThApyătqqyt . (3)
where T P R|Σ|ˆ|Q|. Note that it is because we sought to use a softmax that we require support over all of Σ˚. Also, despite its deceiving notation, Eq. (3), is not, in this form, obviously a neural LM as we did not give a neural network that computes hA—its computation is performed by a DPFSA. Written as in Eq. (3), we say that a DPFSA is of rank R if the output matrix T is of rank R.
Minimal representations. If A is a minimal DPFSA, then the linear subspace spanned by thApyq | y P Σ˚u will be a subspace of R|Q| of dimensionality |Q| due to the one-hot nature of the state encodings. However, if T is of rank R, then the linear subspace tThApyq | y P Σ˚u cannot span a subspace of dimensionality larger than R. Therefore, for a neural LM to be equivalent to the DPFSA, the hidden state’s dimension must be large enough. If that is not the case, the neural LM will naturally not be able to match all the conditional distributions defined by the states of the DPFSA, which leads us to the following result.
Theorem 3.1. Let p be a language model induced by a minimal DPFSA A with full support. Furthermore, let R be the rank of p when expressed as a representation-based LM (Eq. (3)). Then, any equivalent representation-based LM q must have a hidden state of size at least R.
Proof. See App. C. ■
Given a rank-R DPFSA A, Thm. 3.1 says that an equivalent neural LM needs a hidden state of size at least R, establishing a general lower bound on an LM’s hidden state size for equivalence with a DPFSA. A hidden state of size R, however, does not guarantee that the neural LM can implement the transitions between DPFSA’s states. The neural LM’s ability to do so depends on the particular architecture implementing the neural LM. In other words, Thm. 3.1 furnishes us with a necessary condition. In the rest of the section,
we briefly recapitulate some known results on the relationship between DPFSAs and the two neural architectures in question—recurrent neural networks and Transformers.

heading: 3.2 DPFSAs and Recurrent Neural LMs
text: RNNs’ connection to DPFSAs is particularly well understood for recurrent neural LMs (e.g., Peng et al., 2018; Merrill, 2019; Merrill et al., 2020; Svete and Cotterell, 2023; Svete et al., 2024). The following theorem from Svete and Cotterell (2023) very concretely summarizes the relationship between DPFSAs and finite-precision RNNs, i.e., ones where the representations h fall into some finite subset of RD. Theorem 3.2 (Svete and Cotterell (2023), Thm. 4.1). The classes of finite-precision RNN LMs and DPFSA are equivalent.
While Thm. 3.2 ensures the existence of a finite-precision RNN LM weakly equivalent to a given DPFSA, it does not describe the size of the hidden state required for doing so. The following theorem makes the relationship more precise: It states that, in general, the size of the hidden state of RNN LMs must scale linearly with the number of states if the DPFSA is full-rank, i.e., of rank |Q|. Theorem 3.3 (Svete and Cotterell (2023), Thms. 5.1 and 5.2). A family of DPFSAs exists such that the hidden states sizes of equivalent RNN LMs scale linearly with |Q| and |Σ|.
Our Thm. 3.1 can be seen as a more precise version of Thm. 3.3, which does not consider the rank of the DPFSA.3

heading: 3.3 DPFSAs and Transformer LMs
text: Statements similar to Thm. 3.2 are more difficult to make for Transformer LMs due the their parallelizable nature (Merrill and Sabharwal, 2023b; Svete and Cotterell, 2024); unlike DPFSAs, Transformers do not compute a sequential inner state. Existing work has connected Transformers to sequential automata like DPFSAs with the empirically successful framework of chain-of-thought reasoning (Pérez et al., 2021; Feng et al., 2023; Merrill and Sabharwal, 2023a) or by assuming a growing number of layers with increasing string length (Liu
3The linear scaling with respect to |Q| is unique to weighted FSAs; more efficient constructions simulating unweighted FSAs with RNNs exist (Indyk, 1995; Svete and Cotterell, 2023), highlighting the importance of considering probabilistic models separately.
et al., 2023). However, these results concern binary language recognition rather than the task of language models. To the best of our knowledge, the only work that directly involves language modeling setting connects Transformers to the simple n-gram LMs, a special, well-structured class of DPFSAs.
Theorem 3.4 (Svete and Cotterell (2024), Thm 3.1). For any n-gram LM p, there exists an equivalent Transformer LM pT with hard attention.
Unlike Thm. 3.2, Thm. 3.4 does not provide a complete characterization of Transformer LMs but rather a (loose) lower bound on their capabilities, in line with their weaker connection to sequential automata. We are not aware of any claims analogous to Thm. 3.3 for Transformer LMs; this makes the generally applicable Thm. 3.1 that much more interesting, as it lower-bounds the size of the vectorial representations regardless of the neural architecture used to compute them. This distinction is also mirrored in the empirical part of the paper.

heading: 3.4 Beyond Representational Capacity
text: The theoretical results presented and summarized here capture the fact that the size of the hidden representations in neural LMs must inevitably scale with the size of the DPFSA being simulated or learned (either with its rank, its number of states, or the size of the alphabet). In that sense, the representational capacity of neural LMs can be theoretically described relatively comprehensively in terms of formal models of computation. However, existing theoretical work only considers the question of which distributions can be represented by a neural LM. This leaves us with a large gap in understanding which distributions are learnable by neural LMs. Compared to pure representational capacity results, formal claims about learning are much more difficult to make due to the dependence on factors such as the learning algorithm and aspects of the training data. Therefore, we test the learnability of DPFSAs empirically, allowing us to overcome the above-mentioned difficulties and gain valuable insights into the problem.

heading: 4 Practical Learnability of RLMs
text: Our main goal is to provide a principled study of the ability of neural LMs to learn RLMs. We now describe and justify our experimental design.

heading: 4.1 A Critique of Learning Formal Languages
text: As discussed in §1, much empirical work has investigated the ability of neural language models to learn formal languages such as those described by finite-state automata, i.e., how well a neural language model can be used to assess membership of individual strings in a set. There are multiple workarounds for this discrepancy. Most solutions involve measuring some sort of accuracy of next-symbol prediction. For example, Suzgun et al. (2019a,b) and Bhattamishra et al. (2020) evaluate neural LMs on the next-symbol prediction task, which, intuitively, measures whether all allowed continuations of the string under the formal model achieve a large enough probability under the neural LM. Delétang et al. (2023) evaluate the models by the proportion of correctly predicted tokens (where the approximate argmax of the neural LM prediction has to match the ground-truth label). Unfortunately, all these approaches inevitably shoehorn a neural LM into a sort of classifier, mismatching the type of an LM—a probability distribution—and a formal language—a set, as illustrated in Fig. 1. Ideally, we would like to measure precisely how the neural LM has learned the distribution induced by a formal LM. In this section, we outline and motivate a possible way to approach this challenge.
At a high level, we test the learnability of random representation-based RLMs by training neural LMs on strings sampled from randomly generated DPFSAs and measuring the distance between the neural LM and the DPFSA. Crucially, unlike most existing work, we do not have to rely on classification or next-symbol prediction accuracy-based metrics, but rather we directly measure the similarity of distributions which presents a much cleaner way of evaluating model similarity. Concretely, given an DPFSA p and a neural LM q, we measure the KL divergence between the DPFSA and the
neural LM:
DKL pp || qq def“ ÿ yPΣ˚ p pyq log p pyq q pyq (4a)
“ Hpp, qq ´ Hppq. (4b)
The KL divergence is an established and wellunderstood measure of the distance4 between two distributions. As such, it lends itself naturally to evaluating the difference between LMs; in our case, measuring how well the neural LM has captured the distribution of the DPFSA. Such a holistic treatment of the difference between two LMs gives us a tangible and interpretable way of understanding how they differ. To compute DKL pp || qq, we use Eq. (4b). We estimate the first term H pp, qq by computing pH pp, qq, the empirical cross-entropy between p and q. The second term can be computed exactly by dynamic programming (Eisner, 2002; Zmigrod et al., 2021). See App. D.4 for further details on the computation of these evaluation metrics.

heading: 4.2 Generating Random DPFSAs
text: We evaluate neural LMs’ ability to learn random representation-based DPFSAs, which we construct as follows. First, we sample the number of states |Q| P t2, 4, 6, 8, 10, 12, 16u and the number of symbols |Σ| P t2, 4, 6, 8, 10, 12, 16u, both uniformly at random. Then, we select the destination for the outgoing arcs of each of the states—one for each y P Σ, i.e., for each q P Q and y P Σ we randomly choose q1 P Q and add the transition q yÝÑ q1 to A. Again, the sampling is done uniformly at random. We add weights to the transition function of A as follows. Finally, we generate a random matrix T P R|Σ|ˆ|Q|.
4Note, that KL divergence is not a true distance, as it is not symmetric and does not fulfill the triangle inequality.
For each R P t1, 2, 4, 6, 8, 10, 12, 16u satisfying R ď min `|Q|, |Σ|˘, we define TR to be the best rank-R approximation of T, i.e., TR “ argminM ∥M ´ T∥2F subject to the constraint rankpMq “ R. This is an example of the Procrustes problem and can be efficiently computed using SVD. We then set the transition probability of q yÝÑ q1 to wq,y “ softmax ` TR:,q ˘ y . Similarly,
we set the final weight ρ pqq “ softmax`TR:,q ˘ EOS .
As the Transformer LMs have a limited context length, it is undesirable to sample DPFSAs that generate strings that are longer than the context length with high probability. This would introduce an artificial discrepancy between the entropy of the DPFSAs and the empirical entropy of the truncated strings used to train and evaluate the Transformer models. Therefore, we filter out DPFSAs with high expected sequence length. In practice, we filter out DPFSAs with an expected length larger than the median value of expected lengths, i.e., half of the DPFSAs. See App. D.4 for details about calculating the expected string length of a DPFSA.
This process results in the generation of up to eight random DPFSAs, 5 all sharing the same Q,Σ, and underlying transition function. They differ, however, in the rank of the matrix TR that defines the weights of the transitions. Furthermore, the construction of exactly one transition for each q and y ensures that the DPFSA mirrors the nature of a neural LM, which also defines full-support nextsymbol probabilities for any prefix of the string. Altogether, this allows us to precisely control the quantities from Tab. 1 and thus the complexity of the DPFSA. See App. D.1 for additional details.

heading: 5 Results

heading: 5.1 Statistical Evaluation
text: Following the experimental setup outlined in App. D, we obtained DKL results for 15k RNN and 15k Transformer LMs of differing hidden state dimensions, trained on strings sampled from 2100 random DPFSAs with specific sets of complexity
5We have that |tr | r ď min p|Q|, |Σ|q , r P t1, 2, 4, 6, 8, 10, 12, 16uu| ď 8
parameters. The complexity parameters of DPFSAs correspond to the number of states, number of symbols, and the DPFSA rank; additional derivative metrics can also capture overall complexity, including the number of transitions, the expected string length, the upper bound of the DPFSA rank, and the DPFSA entropy (see Tab. 1). We expect the difficulty of learning, quantified as KL divergence, to increase with each complexity parameter and to decrease with the hidden state size.
As shown in Fig. 2, as the number of states and DPFSA rank increase, overall learning difficulty also empirically increases for both RNNs and Transformers. To demonstrate that the observed effect is not wholly reducible to differences in entropy, the relationship between the combined number of states and rank with DPFSA entropy is also presented. (Note that the KL divergence is the LM loss minus the DPFSA entropy.) A more nuanced picture, however, arises in the empirical relationship of alphabet size and DPFSA rank with overall learning difficulty (Fig. 3). Finally, Fig. 4 demonstrates that learning difficulty numerically increases with rank, though a large hidden state size of the neural LM reduce this difficulty to some degree.
To assess the influence of such effects on KL divergence, we implemented a linear regression model for the output of each neural LM. The independent variables were the DPFSA complexity parameters, as well as the neural LM’s hidden state size D. The full list of predictors is shown in Tab. 1. Prior to model fitting, each predictor was standardized using a z-score transformation for an interpretable comparison of the estimated coefficients.

heading: 5.2 RNN Findings
text: As shown in Tab. 2, the KL divergence was significantly influenced by the number of states, DPFSA rank, expected string length, DPFSA entropy, and RNN hidden state size (|Q|, R, Exp.
len., HpAq, D). The number of symbols, number of transitions, and the upper bound of R did not reach significance (|Σ|, |Q||Σ|, minp|Q|, |Σ|q). Of the significant effects, the number of states, expected length, and rank were positive in direction, indicating an increase in KL divergence with an increase in the predictor of interest. Perhaps surprisingly, the DPFSA entropy was negative in influence, indicating a decrease in KL with an increase in DPFSA entropy. Unsurprisingly, the RNN hidden state size was also negative, indicating a decrease in KL with an increase in hidden state size. Overall, the DPFSA rank had the strongest influence on KL divergence, followed by the expected length, the DPFSA entropy, number of states, and RNN hidden state size. The remaining predictors were smaller in influence.

heading: 5.3 Transformer Findings
text: The linear regression model of the Transformers data revealed significant effects of all included predictors on KL divergence (see Tab. 4). Of these, the number of states, number of symbols, number of transitions, DPFSA rank, and expected string length were positive in influence, indicating an increase in KL divergence as the predictor of interest increased (|Q|, |Σ|, |Q||Σ|, R, Exp. len.). The upper bound of R, DPFSA entropy, and the Transformer hidden state size were negative in influence, indicating a decrease in KL divergence with an increase in the predictor of interest (HpAq, D). Of the positive relationships, the expected string length had the largest influence, followed in order by DPFSA rank (R), then number of symbols (|Σ|). Of the negative relationships, DPFSA entropy had the largest influence, followed by the Transformer hidden state size, then by the upper bound of R.

heading: 5.4 Comparing RNN and Transformer LMs
text: The linear models revealed an overall similar pattern of effects between RNNs and Transformers, albeit with few notable disparities. First, note the overall performance of each neural LM, as indicated by the model intercept for KL divergence— see Tab. 2, Tab. 4, and Fig. 6 in App. E. The model intercept reflects the predicted KL divergence when all other (z-transformed) predictors are equal to 0. RNNs tend to outperform Transformers in this task (RNNs: pβ0 “ 8.72, Transformers: pβ0 “ 13.5), demonstrating lower average loss. This difference in performance could be attributed to two main factors: 1) As previous research has shown, RNNs are
better suited to modeling formal languages (Delétang et al., 2023), and 2) Transformers necessitate careful training involving language-specific hyperparameter tuning, which poses a severe computational challenge. Nevertheless, we expect that the trend observed here would persist even if the Transformers were trained under optimal conditions.
While different in magnitude, the number of states, rank, expected string length, DPFSA entropy, and hidden state size all shared the same direction in their influence on KL divergence for each of the RNN and Transformer outputs. Of these, the biggest disparity in magnitude is in the influence of the expected string length and DPFSA entropy, which were significantly larger for Transformers compared to RNNs. Finally, not all predictors reached significance for both RNNs and Transformers. For instance, the number of symbols, number of transitions, and the upper bound of R were significant predictors of KL divergence for Transformers, but not for RNNs.
As shown in Figs. 2, 3, and 4, the observed trends in the DPFSAs’ entropy, visualized alongside the neural LMs DKL, suggest that the entropies alone cannot solely account for the observed effect in DKL. In other words, larger DKL values are not fully explained by greater entropy values, highlighting the significance of the rank R on learnability even more clearly. Particularly notable is Fig. 4, which presents similar patterns to those observed in Tab. 2 and Tab. 4. The KL divergence of both model types increases with an increase in the PFSA rank and decreases with an increase in the hidden state size. While somewhat noisier, Fig. 2 and Fig. 3 further demonstrate the weaker influence of the number of states and number of symbols on the KL divergence.

heading: 6 Discussion
text: Implications of Thm. 3.1. Thm. 3.1 concretely quantifies the minimum size of the representation space of any neural LM required for the correct representation of regular LMs. While this is special case of the so-called softmax bottleneck principle (Chang and McCallum, 2022), it is, to the best of our knowledge, the first result connecting the principle to formal models of computation. Practical implementations of RLMs might use state spaces and alphabets of sizes ranging from tens of thousands to hundreds of thousands (Mohri and Riley, 1999), which is much larger than the representa-

heading: 21.81 26.65 33.18

heading: 48.39 69.24

heading: 23.26 29.83 28.04

heading: 57.28 59.08
text: tions used by most modern neural LMs, which tend to be in the order of a few thousand dimensions (Groeneveld et al., 2024). The high performance of much smaller neural LMs on similar datasets indicates that those LMs are indeed low-rank and can thus be approximated well using smaller hidden representations. Nevertheless, Thm. 3.1 provides an interesting limitation on what distributions neural LMs of finite size can represent and points out the limitations of parameter sharing in representing formal models of computation; while neural LMs are good at approximating such models of computation, their inability to represent them exactly implies that, with increasing string lengths, their errors will unavoidably accumulate. This leads to poor length generalization often observed in prior work (Weiss et al., 2018; Suzgun et al., 2019b; Bhattamishra et al., 2020; Delétang et al., 2023).
Takeaways from the empirical results. The empirical results in §5.2 complement the theoretical discussion from §3 and the growing field of literature characterizing the representational capacity of neural LMs. In line with the theoretical setting and in contrast to related work, our approach directly evaluates the KL divergence between neural LMs and DPFSAs, instead of relying on classification or next-token prediction accuracy measures. Comparing distributions over strings offers a more holistic view of a neural LM’s overall ability to emulate DPFSAs allowing us to provide compelling insights into what aspects of distributions affect the learnability of formal LMs by controlling for various properties of the DPFSAs being learned. Neatly, the observed effects of the rank and the neural LM hidden state size on KL divergence align with the theoretical results derived in Thm. 3.1: Namely, as the rank of a DPFSA grows, a larger hidden state is required in the neural LM to model the DPFSA’s language appropriately. The dependence of the performance on the rank of the DPFSA demonstrates the utility of formal language theory in providing interpretable insights into the learning abilities of neural LMs.
Another noteworthy observation is the negative influence of the DPFSA entropy on the KL divergence, indicating that neural LMs tend to better learn RLMs with higher intrinsic randomness. A possible explanation for this counter-intuitive result is that when a DPFSA is truly random, its simulation is trivial. However, it is more challenging to accurately learn RLMs with more
underlying latent structures.
Extensions. We focus on the learnability of deterministic PFSAs. This makes the theoretical results from §3 particularly interpretable. Extensions to the non-deterministic automata, however, are an interesting next step. Note that in this case, the PFSA rank analysis is slightly more nuanced. A non-deterministic PFSA can, at any point, be in any of the |Q| states (with a probability distribution over them), meaning that the probability of the next symbol is a convex combination of the individual conditional probability distributions (not their logits). This makes the analysis trickier and less interpretable; we leave it for future work to make the current exposition more concise. A further interesting follow-up is also the study of the learnability of (deterministic) context-free LMs represented by probabilistic pushdown automata (PPDAs). PPDAs augment PFSAs by implementing a stack that gives the automaton infinitely many configurations. Despite the infinitely many configurations, controlling for their rank analogously to the rank of a PFSA could elucidate how efficiently they are representable by neural LMs.

heading: 7 Conclusion
text: We provide a comprehensive empirical study of the learnability of DPFSAs by neural LMs. More concretely, we investigate how well RNN and Transformer LMs learn to match the distributions over strings generated by DPFSAs of varying complexity. To this end, we operationalized learning difficulty using the KL divergence between such distributions over strings; this measure holistically captures the similarity between the neural LM and the targeted regular LM. In an regression analysis of KL divergence variation, several complexity parameters of the DPFSA reached significance, along with the hidden state size of the neural LM. Of note were the DPFSA rank and the expected length of sampled strings, which corresponded to significant increases in KL divergence across the RNN and Transformer results. We establish that for equivalence, a neural LM’s hidden state size is theoretically lower-bounded by the DPFSA’s rank. This is consistent with the results of our controlled experiments. Our results demonstrate that using probabilistic formal language theory can help us generate insights into what neural LMs can and cannot learn. Nevertheless, our findings call for further theoretical investigations closer to practical applications.
Limitations
We point out some limitations of the presented study. First and foremost, the finite-state automata considered in this paper are limited by their bounded memory. Additionally, to keep our work concise and results self-contained, we focus only on deterministic DPFSAs. Similar and more comprehensive investigations could of course include non-deterministic automata and languages higher up on the Chomsky hierarchy, such as context-free LMs, or even context-sensitive LMs. Our experiments also omit the effect of training dataset size, which might be an interesting quantity to consider when training neural LMs. We leave those considerations to future work.
Moreover, due to computational constraints and the substantial computation load imposed by our experiments, we could not fine-tune our models with language-specific hyperparameters, which are particularly important for transformers. For the same reason, we had to refrain from optimizing larger and more capable models. However, we believe that this should not impair the validity of our results, as the trend we observed would hold even with optimal training.

heading: Acknowledgements
text: Ryan Cotterell acknowledges support from the Swiss National Science Foundation (SNSF) as part of the “The Forgotten Role of Inductive Bias in Interpretability” project. Anej Svete is supported by the ETH AI Center Doctoral Fellowship. Josef Valvoda is funded by the Nordic Programme for Interdisciplinary Research Grant 105178 and the Danish National Research Foundation Grant no. DNRF169. This research was partially funded by a DFF Sapere Aude research leader grant under grant agreement No 0171-00034B, the DanishIsraeli Study Foundation in Memory of Josef and Regine Nachemsohn, and the Privacy Black & White project, a UCPH Data+ Grant. This work was further supported by the Pioneer Centre for AI, DNRF grant number P1.

heading: A Additional Related Work
text: A.1 Representational Capacity of Neural LMs
Much of the theoretical work has investigated the representational capacity of various neural LM architectures (Merrill, 2023; Strobl et al., 2023). Regular languages (and, to a lesser extent, regular LMs) have been linked to neural LMs particularly often, especially to recurrent neural LMs, but similar connections have also been made for Transformers (Merrill, 2019; Merrill et al., 2020; Liu et al., 2023). Distinctively interesting are the bounds on the space requirements for emulating FSAs (Dewdney, 1977; Indyk, 1995; Hewitt et al., 2020; Svete and Cotterell, 2023). This work bridges the theoretical work with practice, tests its applicability, and uses its insights for an informed study of the practical representational capacity of neural LMs.
A.2 Learning Formal Languages
Work similar to ours in spirit is that of Jumelet and Zuidema (2023), where the authors train and evaluate neural LMs with probabilistic context-free grammars. They use the underlying data-generating distribution (the probabilistic grammar) to evaluate how well the model has learned the distribution. Moreover, the knowledge of grammar allows them to probe the model for the encodings of individual constituents, similar to how we probe for the states of the automaton. In contrast to our work, however, Jumelet and Zuidema (2023) focus on learning human-language-based grammars, which do not provide a holistic picture of the representability of general formal LMs by neural LMs.
Delétang et al. (2023) provide a comprehensive survey of the learnability of diverse formal languages. Unlike us, they focus on learning discrete languages, particularly from the perspective of learning algorithms and investigating LMs’ inductive biases. They formulate this as a transduction—a stringto-string mapping. They arrive at interesting results showing that popular neural LMs are hard to place on the standard Chomsky hierarchy of languages. This can partly be explained by the mismatch of the training task—transduction—and the probabilistic nature of a neural LM, since the probabilistic Chomsky hierarchy is known to differ from the discrete one (Icard, 2020). In contrast to our work, Delétang et al. (2023) also only consider a limited set of hand-picked languages which, while providing algorithmic insights into how LMs work, do not extensively probe the learnability of the language classes.
Testing the compositional generalization of NNs, Valvoda et al. (2022) sample an infinite number of finite languages. Thereby they can draw conclusions about the learnability of an entire class of languages—sub-regular ones encoded by subsequential finite state transducers. Their work connects Montague’s theory of compositional generalization (Montague, 1970) with the popular SCAN benchmark of compositional behavior (Lake and Baroni, 2018). Unlike our work, they investigate deterministic transducers and seq2seq models.
Another similar work is that of White and Cotterell (2021), who use artificial languages to identify the biases of neural LMs. By modifying a base grammar, they experiment with the learnability of 64 languages. Unlike us, their work focuses solely on topological aspects of the language, which limits their findings to observations over the word order.
In a different line of work, Akyürek et al. (2024) evaluate neural LMs’ abilities to learn regular languages in context. Rather than learning one particular distribution from the training dataset, they train neural LMs to model the language of any finite-state automaton given a number of samples from it—that is, to infer the generating mechanism from the context. They consider only discrete languages (even though their generative setup is probabilistic) and due to the in-context learning setting, they do not analyze the dynamics of the neural LM implementing individual languages.

heading: B Probabilistic Finite-state Automata
text: We begin by more formally defining the notion of probabilistic finite-state automata (PFSAs), which were only informally introduced in §2.
Definition B.1. A probabilistic finite-state automaton (PFSA) is a 5-tuple pΣ, Q, δ, λ, ρq where Σ is an alphabet, Q a finite set of states, δ Ď Q ˆ Σ ˆ r0, 1s ˆ Q a finite set of weighted transitions and λ, ρ : Q Ñ r0, 1s the initial and final weighting functions. Moreover, δ, λ and ρ are required to satisfy
that ř qPQ λ pqq “ 1, and, for all q P Q, ř
pq,y,w,q1qPδ w ` ρ pqq “ 1. We denote pq, y, w, q1q P δ with q
y{wÝÝÑ q1. Definition B.2. A path π in a PFSA A is a string of consecutive transitions q1
y1{w1ÝÝÝÑ q2, ¨ ¨ ¨ , qN yN {wNÝÝÝÝÑ qN`1. A path π’s length |π| is the number of transitions in it and its scan s pπq the concatenation of the symbols on them. We denote with ΠpAq the set of all paths in A and with ΠpA,yq the set of all paths that scan y P Σ˚.
The weights of the transitions along a path are multiplicatively combined to form the weight of the path. The weights of all the paths scanning the same string are combined additively to form the weight of that string. Definition B.3. The path weight of π P ΠpAq is w pπq “ λ pq1q ”śN n“1wn ı ρ pqN`1q. The stringsum of y P Σ˚ is A pyq def“ řπPΠpA,yq w pπq. It is easy to see that the final weights ρ pqq play an analogous role to the EOS symbol in the context of autoregressive LMs—they both correspond to the probabilities of ending the generation of the string.
Definition B.4. A PFSA A “ pΣ, Q, δ, λ, ρq is deterministic if |tq | λ pqq ą 0u| “ 1 and, for every q P Q, y P Σ, there is at most one q1 P Q such that q y{wÝÝÑ q1 P δ with w ą 0.
In general, there can be infinitely many PFSAs that define a given FSLM. However, in the deterministic case, there is a unique minimal DPFSA.
Definition B.5. A DPFSA A “ pΣ, Q, δ, λ, ρq is minimal for the FSLM p if there is no equivalent DPFSA A1 “ pΣ, Q1, λ1, ρ1, δ1q with |Q1| ă |Q|.

heading: C Proofs of Theoretical Results
text: Theorem 3.1. Let p be a language model induced by a minimal DPFSA A with full support. Furthermore, let R be the rank of p when expressed as a representation-based LM (Eq. (3)). Then, any equivalent representation-based LM q must have a hidden state of size at least R.
Proof. This proof is effectively a restatement of the softmax bottleneck (Yang et al., 2018). Because the language model p has full support, we may transform it into a softmax-normalized, representation-based LM. By supposition, after this transformation, p’s output matrix T, as in §3.1, is of rank R. For q to be equivalent to p, it has to hold that
softmaxpEhpyqq “ softmaxpThApyqq, (5) for all y P Σ˚ where E’s q’s output matrix. Next, define
Z1pyq def“ ÿ
yPΣ expppEhpyqqyq, (6a)
Z2pyq def“ ÿ
yPΣ expppThApyqqyq. (6b)
For all y P Σ, due to the additive invariance property of the softmax, we have the following softmaxpEhpyq ´ logZ1pyq ¨ 1qy “ exppEhpyq ´ logZ1pyq ¨ 1qy (7a)
“ exppThApyq ´ logZ2pyq ¨ 1qy (7b) “ softmaxpThApyq ´ logZ2pyq ¨ 1qy, (7c)
where 1 is the vector of all ones P R|Σ|. It follows that, by taking logs, we have Ehpyq ´ logZ1pyq ¨ 1 “ ThApyq ´ logZ2pyq ¨ 1, (8)
and, furthermore, that
Ehpyq “ ThApyq ` log Z1pyq Z2pyq ¨ 1. (9)
From Eq. (9), it follows that
span ˆ tThApyq ` log Z1pyq
Z2pyq ¨ 1 | y P Σ ˚u
˙ “ span ptEhpyq | y P Σ˚uq . (10)
Noting that span ´ tlog Z2pyqZ1pyq ¨ 1 | y P Σ˚u ¯ “ spanpt1uq, we can write
span ptThApyq | y P Σ˚uq ‘ span pt1uq “ span ptEhpyq | y P Σ˚uq , (11)
where ‘ is the direct sum, and we note that dimpspan ptThApyq | y P Σ˚uqq “ R exactly because A’s minimality implies dimpspan pthApyq | y P Σ˚uqq “ |Q|. Next Eq. (11), in turn, implies
span ptThApyq | y P Σ˚uqloooooooooooooooomoooooooooooooooon dimensionality R Ď span ptEhpyq | y P Σ˚uq . (12)
Thus, we arrive at a lower bound on the rank of E. Why is it a lower bound? Because for an arbitrary representation-based LM, we do not know dim pspan pthpyq | y P Σ˚uqq. However, we do know it is lower-bounded by dim pspan ptEhpyq | y P Σ˚uqq. Thus, rank pEq ě R. Because E’s rank is bounded above by D, we have D ě R, as desired. ■

heading: D Experimental Details
text: D.1 Sampling DPFSAs of varying complexity
The DPFSAs we used in our experiments were sampled with |Q| P t2, 4, 6, 8, 10, 12, 16u over alphabets alphabets of sizes |Σ| P t2, 4, 6, 8, 10, 12, 16u. Given a sampled DPFSA A with |Q| states over an alphabet Σ, we randomly set its unweighted transition function. That is, for each q P Q and y P Σ we randomly choose q1 P Q and add the transition q yÝÑ q1 to A.
We add weights to the transition function of A as follows. We generate a random matrix T P R|Σ|ˆ|Q|,Ti,j „ N pµ “ 0, σ2 “ 4q , and define Rmax “ rank pTq (note that Rmax ď minp|Q|, |Σ|q). For each R P tr | r ď min p|Q|, |Σ|q , r P t1, 2, 4, 6, 8, 10, 12, 16uu, we define TR def“ SVD pT,Rq, where SVD pT,Rq is the operation of reducing the rank of T to R using SVD. Next, we add weights to the transition function of A by replacing each unweighted transition q yÝÑ q1 with q y{wq,yÝÝÝÝÑ q1, where wq,y “ softmax ` TR:,q ˘ y . Finally, we set ρ pqq “ softmax`TR:,q ˘ EOS
. This process results with the generation of up to eight6 random DPFSAs, all sharing the same Q,Σ and underlying transition function. They differ, however, in the rank of the matrix TR that defines the weights of the transitions.
As the Transformer LMs have a limited context length, it is undesirable to sample DPFSAs that are likely to generate strings longer than the context length. This may unfairly hinder the Transformer LMs ability to learn such languages. We, therefore, filter out DPFSAs with an expected string length larger than the median value of expected string lengths (in practice, 46 symbols), i.e., half of the DPFSAs. See App. D.4 for details about how the expected generation length of DPFSAs is calculated.
D.2 Generating the Data
For a given DPFSA A, we sample 20k random strings, terminating the generation process of each string when EOS is sampled. We divide the dataset into train and test splits, such that no string is shared between the sets, and the test set has at least 2k strings. We truncate the strings to 256 symbols to accommodate the limited context length of the Transformer model we used. Fig. 5 shows a histogram of the average length of strings generated for each DPFSA.
6Note that we have | tr | r ď min p|Q|, |Σ|q , r P t1, 2, 4, 6, 8, 10, 12, 16uu | ď 8.
D.3 Training the Neural LMs
We train neural LMs on our dataset using the following procedure, repeated 2100 times:
1. Sample a random DPFSA A with |Q|, |Σ| and rank R using the process described in App. D.1. 2. Sample 20k strings from A and split them to train set and test set using the process described in
App. D.2.
3. For each D P t2, 4, 6, 8, 10, 12, 16u train an RNN on the train set strings. 4. For each D P t2, 4, 6, 8, 10, 12, 16u train a Transformer model on the train set strings. 5. Compute the DKL between A and the trained neural LMs on the test set strings.
We train the RNN and Transformer models using the following hyperparameters:
• RNNs: We use a unidirectional LSTM with four hidden layers, each with 64-dimensional hidden states and an embedding size of 64. We trained each model for two epochs using a batch size of 32 and a learning rate of 0.001, an Adam optimizer with default settings, and a standard cross-entropy loss (Kingma and Ba, 2014). We did not tie the weights of the word embeddings.
• Transformers: We use the GPT-2 model architecture (Radford et al., 2019) with six attention layers, each with four attention heads and 128-dimensional representations. We use an embedding size of 64 and an input context length of 256. We trained each model for two epochs using a batch size of 32 and a learning rate of 0.001, an AdamW (Loshchilov and Hutter, 2018) optimizer with default settings, and a standard cross-entropy loss. We did not tie the weights of the word embeddings.
D.4 Derivations
D.4.1 Entropy of a DPFSA Let A “ pΣ, Q, δ, λ, ρq be a softmax-normalized DPFSA with the parameters matrix T P R|Σ|ˆ|Q|. The entropy of A is defined as
H pAq def“ ´ ÿ
yPΣ˚ A pyq logA pyq . (13)
Grenander (1967, Thm. 4.2) provides an efficient method of computing H pAq, which we summarize in the following, based on Sánchez et al.’s (2018) exposition.
In deterministic PFSAs, Eq. (13) can equivalently be computed by summing over the paths in A, ΠpAq, rather than the strings y P Σ˚; as A is deterministic, there exists a single path π in ΠpA,yq for every
y P Σ˚. We can then derive
H pAq “ ´ ÿ
yPΣ˚ A pyq logA pyq (14a)
“ ´ ÿ
πPΠpAq w pπq logw pπq . (14b)
Now, let M P R|Q|ˆ|Q| be the matrix of transition probabilities between each two states. That is, Mi,j is the probability of transitioning from state qi P Q to state qj P Q in A, and it is computed as
Mi,j def“
ÿ yPΣ 1
" qi y{wÝÝÑ qj P δ * ¨ softmaxpT:,qiqy, (15)
where 1 " qi y{wÝÝÑ qj P δ * is the indicator function of whether the transition qi y{wÝÝÑ qj exists in A.7
Similarly, let α P R|Q| be defined such that αi def“ λpqiq, i.e., the probability of starting a generation at state qi. Finally, let ξ P R|Q| be defined as
ξi def“ ´
ÿ yPΣ softmaxpT:,qiqy log softmaxpT:,qiqy. (16)
This allows us to restate Grenander (1967, Thm. 4.2).
Theorem D.1 (Entropy of a DPFSA). Given the definitions in Eqs. (15) and (16), H pAq can be computed using the following expression
H pAq “ αJ pI ´ Mq´1 ξ. (17)
We use Eq. (17) to compute the entropy of all our regular LMs.
D.4.2 Expected String Length Under a DPFSA To compute the expected length of a string under a DPFSA, we rely on the following lemma.
Lemma D.1 (Expected string length under a DPFSA). The expected length of strings generated by a DPFSA A “ pΣ, Q, δ, λ, ρq can be computed using the following identity
E πPΠ rNpπqs “
ÿ
qPQ
´ αJ pI ´ Mq´1 ¯ q ´ 1. (18)
Proof. The following derivation proves the result
ÿ
qPQ
´ αJ pI ´ Mq´1 ¯ q “ ÿ
qPQ
˜ αJ ˜ 8ÿ
n“0 Mn
¸¸
q
(19a)
“ ÿ
qPQ
« 8ÿ
n“0
` αJMn ˘ q ff (19b)
“ 8ÿ
n“0
« ÿ
qPQ
` αJMn ˘ q ff . (19c)
Notice that ` αJMn ˘ q
is the total weight of paths in A reaching the state q, but does not necessarily terminate there due to the absence of final weights ρ, after exactly n transitions, starting in initial states
7M is in fact the stochastic matrix describing A.
according to their initial weights (Malagutti et al., 2024). It follows that ř iPQ ` αJMn ˘ i
equals the weights of all paths (visiting in any state P Q) that generate a string of length of at least n:
ÿ
iPQ
` αJMn ˘ i “ A pty | |y| ě kuq . (20)
Let ppNpπq “ k | Aq be the probability of the A generating a string of length exactly k, terminating the generation after k transitions. In other words, ppNpπq “ k | Aq corresponds to the probability mass of the set ty | |y| “ ku under the DPFSA. By the reasoning above,
ÿ
qPQ
` αJMn ˘ q “ 8ÿ
k“n ppNpπq “ k | Aq, (21)
and
8ÿ
n“0
« ÿ
qPQ
` αJMn ˘ q ff (22a)
“ 8ÿ
n“0
8ÿ
k“n ppNpπq “ k | Aq (22b)
“ 8ÿ
k“0 ppNpπq “ k | Aq `
8ÿ k“1 ppNpπq “ k | Aq ` 8ÿ k“2 ppNpπq “ k | Aq ` . . . (22c)
“ ppNpπq “ 0 | Aq ` ppNpπq “ 1 | Aq ` ppNpπq “ 2 | Aq ` ppNpπq “ 3 | Aq ` ¨ ¨ ¨ (22d) ` ppNpπq “ 1 | Aq ` ppNpπq “ 2 | Aq ` ppNpπq “ 3 | Aq ` ¨ ¨ ¨
` ppNpπq “ 2 | Aq ` ppNpπq “ 3 | Aq ` ¨ ¨ ¨ ` ppNpπq “ 3 | Aq ` ¨ ¨ ¨
` ¨ ¨ ¨
“ 8ÿ
n“0 ppNpπq “ n | Aq `
8ÿ n“0 n ¨ ppNpπq “ n | Aq (22e)
“ 1 ` 8ÿ
n“0 n ¨ ppNpπq “ n | Aq (22f)
“ 1 ` E πPΠ rNpπqs . (22g)
■
D.4.3 Cross Entropy Between a DPFSA and a Neural LM To stochastically approximate the cross entropy between a ground-truth DPFSA and a neural LM, we compute
pH pppDtestq, qpDtestqq “ ´ 1 N
Nÿ
n“1
¨ ˝log qpEOS | ypnqq ` |yn|ÿ
t“1 log qpypnqt | yătpnqq
˛ ‚, (23)
where qpyt | yătpnqq def“ softmaxpEht´1qyt is given by the neural LM. The strings ypnq are sampled i.i.d. from p, language model induced by the DPFSA.

heading: E Additional Results
text: This section includes figures presenting the results of additional experiments augmenting and supporting the claims in the main paper.
Fig. 6. Overall performance of the neural LM models as a function of the “total complexity” of the PFSA they were optimised for, which we define as the sum of |Σ|, |Q|, and R. Performance is measured as the cross-entropy of the neural model on a held-out test set. We compute the loss by summing it over symbols and dividing this sum by the number of strings in the test set. We can see that RNNs tend to overperform Transformers, especially for more complex PFSAs.
Fig. 7. The DKL of RNNs and Transformers as a function of the average string length of the DPFSA. Similarly to Tab. 4, we see that Transformers are much more sensitive to string length compared to RNNs.

heading: Future Work
text: We leave those considerations to future work. Moreover, due to computational constraints and the substantial computation load imposed by our experiments, we could not fine-tune our models with language-specific hyperparameters, which are particularly important for transformers. For the same reason, we had to refrain from optimizing larger and more capable models. However, we believe that this should not impair the validity of our results, as the trend we observed would hold even with optimal training