On Efficiently Representing Regular Languages as RNNs (2024)

Abstract: Recent work by Hewitt et al. (2020) provides an interpretation of the empirical success of recurrent neural networks (RNNs) as language models (LMs). It shows that RNNs can efficiently represent bounded hierarchical structures that are prevalent in human language. This suggests that RNNs’ success might be linked to their ability to model hierarchy. However, a closer inspection of Hewitt et al.’s (2020) construction shows that it is not inherently limited to hierarchical structures. This poses a natural question: What other classes of LMs RNNs can efficiently represent? To this end, we generalize Hewitt et al.’s (2020) construction and show that RNNs can efficiently represent a larger class of LMs than previously claimed— specifically, those that can be represented by a pushdown automaton with a bounded stack and a specific stack update function. Altogether, the efficiency of representing this diverse class of LMs with RNN LMs suggests novel interpretations of their inductive bias. https://github.com/rycolab/bpdas

text: Findings of the Association for Computational Linguistics ACL 2024, pages 4118–4135 August 11-16, 2024 ©2024 Association for Computational Linguistics
https://github.com/rycolab/bpdas

heading: 1 Introduction
text: Neural LMs have demonstrated a human-level grasp of grammar and linguistic nuance. Yet, a considerable gap remains between our empirical observations and our theoretical understanding of their capabilities. One approach to bridge this gap is to study what classes of formal languages neural LMs can efficiently represent. The rationale behind this object of study is that a neural LM’s ability to represent a language efficiently suggests the presence of an inductive bias in the model architecture that prefers that language over others, and may make it specifically easier to learn, e.g., due to Occam’s razor.
Exploring LMs’ ability to model formal languages has garnered significant interest in recent years; see, e.g., the surveys by Merrill (2023) and Strobl et al. (2023). Investigation into this area has a particularly long history in the context of RNNs (McCulloch and Pitts, 1943; Minsky, 1954; Siegel-
mann and Sontag, 1992).1 For example, a classic result (Minsky, 1954) states that RNNs are equivalent to finite-state automata (FSAs). How efficiently an RNN can encode an FSA was studied by Indyk (1995), who showed that an FSA with statesQ over an alphabet Σ can be simulated by an RNN with O ´ |Σ|a|Q| ¯ neurons. This construction is opti-
mal in the sense that there exist FSAs that require this many neurons to be emulated by an RNN.
Indyk’s (1995) bound presents a worst-case analysis: It reflects the number of neurons needed for an adversarially selected FSA. However, it is easy to construct FSAs that can be encoded with exponentially fewer neurons than the number of states. Fig. 2 exhibits an n-gram LM that, when encoded as an FSA, has |Σ|n´1 states, but can still be represented with O pn log |Σ|q neurons. Building on this insight, Hewitt et al. (2020) show that an entire class of languages—Dpb, nq, i.e., the Dyck language over b parentheses types with nesting up to depth n—can be encoded in logarithmic space. Specifically, even though an FSA that accepts the
1In terms of empirical performance on statistical language modeling, RNNs constituted the empirical state of the art until recently (Qiu et al., 2020; Orvieto et al., 2023), and have, despite the prominence of transformer-based LMs, seen a resurgence of late (Peng et al., 2023; Orvieto et al., 2023; Zhou et al., 2023).
4118
Dpb, nq language requires O pbnq states, it can be encoded by an RNN with O pn log bq neurons, which is memory-optimal. Attractively, Dpb, nq languages capture, to some extent, the bounded nested nature of human language.2
Hewitt et al.’s (2020) result poses an interesting question: Are all languages that RNNs can implement efficiently hierarchical in nature? We show that this is not the case. We revisit Hewitt et al.’s (2020) construction and demonstrate that the same optimal compression can be achieved for a more general class of languages, which do not necessarily exhibit hierarchical structure. Stated differently, RNNs do not necessarily encode hierarchical languages any more compactly than non-hierarchical ones. To show this, we introduce probabilistic bounded pushdown automata (BPDAs), probabilistic pushdown automata (Abney et al., 1999) whose stack is bounded. The probabilistic nature of BPDAs furthermore allows us to go beyond the binary recognition and reason about classes of language models—probability distributions over strings—that RNN LMs can efficiently represent. This is particularly interesting
2Bounded Dyck languages offer a useful framework for modeling the hierarchical and recursive aspects of human language syntax with limited memory. However, they fall short of capturing the full complexity of human language, which often includes context sensitivity and ambiguity that may go beyond the capabilities of a deterministic model of computation.
3For simplicity, the final weights and the binary representations in the hidden state are omitted.
because it requires the RNN to not only efficiently encode the transition dynamics of the automaton but also the string probabilities.4 Thus, we generalize the result by Hewitt et al. (2020) and give sufficient conditions for a probabilistic BPDA to be optimally compressible into an RNN.
The non-hierarchical nature of BPDAs leads us to contend that the reason some LMs lend themselves to exponential compression has little to do with their hierarchical structure. This offers two intriguing insights. First, it shows that the inductive biases of RNN LMs might be difficult to link to a hierarchical structure. Second, it provides a new perspective on which languages RNNs efficiently encode, i.e., those representable by sequential machines with a bounded stack. Such a machine is illustrated in Fig. 1.

heading: 2 Preliminaries
text: We begin by introducing some core concepts. An alphabet Σ is a finite, non-empty set of symbols. Its Kleene closure Σ˚ is the set of all strings of symbols in Σ. The length of the string y “ y1 . . . yT P Σ˚, denoted by |y| “ T , is the number of symbols it contains. A language model p is a probability distribution over Σ˚. Two LMs p and q are weakly equivalent if p pyq “ q pyq for all y P Σ˚ and two families of LMs P and Q are weakly equivalent if, for any p P P , there exists a weakly equivalent q P Q and vice versa.
Most modern LMs define p pyq as a product of conditional probability distributions:
p pyq def“ p pEOS | yq |y|ź
t“1 p pyt | yătq , (1)
where EOS R Σ is a distinguished end-of-string symbol. We denote Σ def“ Σ Y tEOSu and use the notation y P Σ whenever y can also be EOS. Such a definition is without loss of generality; any LM can be factorized in this form (Cotterell et al., 2024). However, not all models expressible as Eq. (1) constitute LMs, i.e., a probabilistic model of the form given in Eq. (1) may leak probability mass to infinite sequences (Du et al., 2023). In this paper, we assume all autoregressive models are tight, i.e., they place probability 1 on Σ˚.
A historically important class of LM are those that obey the n-gram assumption.
4Note the same lower bounds for FSAs do not apply to the probabilistic case (Svete and Cotterell, 2023b).
Assumption 2.1. The n-gram assumption states that the probability p pyt | yătq only depends on n ´ 1 previous symbols yt´1, . . . , yt´n`1:
p pyt | yătq “ p pyt | yt´n`1 ¨ ¨ ¨ yt´1q . (2) Fig. 2 shows an example of an 2-gram LM. The weights on the transitions denote the conditional probabilities of the next symbol given the previous n ´ 1 “ 2 symbols encoded by the current state.

heading: 2.1 Recurrent Neural Language Models
text: The conditional distributions of recurrent neural LMs are given by a recurrent neural network. Hewitt et al. (2020) present results for both Elman RNNs (Elman, 1990) as well as those derived from the LSTM architecture (Hochreiter and Schmidhuber, 1997). Our paper focuses on Elman RNNs, as they are easier to analyze and suffice to present the main ideas, which also easily generalize to the LSTM case. Definition 2.1. An Elman RNN R “ pΣ, σ,D,U,V,b,ηq is an RNN with the hidden state recurrence
h0 “ η pt “ 0q (3a) ht “ σ pUht´1 ` Vrpytq ` bq pt ą 0q, (3b)
where ht P RD is the hidden state at time step t, η P RD is an initialization parameter, yt P Σ is the input symbol at time step t, r : Σ Ñ RR is a symbol representation function, U P RDˆD,V P RDˆR are parameter matrices, b P RD is a bias vector, and σ : RD Ñ RD is an element-wise non-linear activation function. We refer to the dimensionality of the hidden state, D, as the size of the RNN, and to each entry of the hidden state as a neuron.
Because ht represents the string consumed by the RNN, we also use the evocative notation h pyq to denote the result of the application of Eq. (3b) over the string y “ y1 ¨ ¨ ¨ yt. An RNN can be used to specify an LM by using the hidden states to define the conditional distributions over y given yăt. Definition 2.2. Let R be an Elman RNN, E P R|Σ|ˆD and u P R|Σ|. An RNN LM is an LM whose conditional distributions are defined as
ppy | yătq def“ softmax pEhpyătq ` uqy (4) for y P Σ,yăt P Σ˚.5 We term E the output matrix and u the bias vector.
5Throughout the paper, we index vectors and matrices directly with symbols from Σ. This is possible because of a trivial bijective relationship between Σ and “|Σ|‰.

heading: 2.1.1 Activation Functions and Precision
text: An important consideration when analyzing RNN LM’s ability to compute the probability of a string y P Σ˚ is the number of bits required to represent the entries in ht and how the number of bits scales with the length of the string, |y|. This depends both on the dynamics of the RNN and the activation function used (Merrill, 2019) and motivates the following definition of precision. Definition 2.3. The precision of an RNN is the number of bits required to represent h pyq: ψR pyq def“ max
dPrDs min p,qPN,
p q “hpyqd rlog2 ps ` rlog2 qs. (5)
We say that an Elman RNN is of constant precision if ψR pyq “ O p1q, i.e., if ψR pyq ď C for all y P Σ˚ and some C P R. It is of unbounded precision if ψR pyq cannot be bounded by a function of |y|.
Common choices for the nonlinear function σ in Eq. (3b) are the Heaviside function H pxq def“ 1 tx ą 0u, its continuous approximation, the sigmoid function σpxq def“ 11`expp´xq , and ReLU def“ maxp0, xq. Hewitt et al. (2020) focus on sigmoid activations as originally presented by Elman (1990). However, to simply the analysis, they assume that |x| " 0, such that σ pxq « 0 or σ pxq « 1 since limxÑ´8 σ pxq “ 0 and limxÑ8 σ pxq “ 1. Concretely, they define a parameter β P R` and assume σ pxq “ 0 for x ă ´β and σ pxq “ 1 for x ą β. Restricting to values of x with |x| ą β results in a constant-precision RNN, equivalent to one with the Heaviside activation function. Its hidden states live in t0, 1uD, meaning that they can be interpreted as binary vectors. The update rule Eq. (3b) can then be interpreted as a logical operation on the hidden state and the input symbol (Svete and Cotterell, 2023b). We simplify the exposition by working directly with H-activated RNNs, which additionally allows for easy analysis of the RNN’s precision.6

heading: 3 Bounded Pushdown Automata
text: In this section, we present a broad class of LMs that provide a convenient framework for analyzing the sufficient conditions for efficient representation by RNNs: bounded pushdown automata. They maintain a stack, albeit one that contains at most a fixed number of symbols.
6Since the Heaviside function can be implemented as a difference of two ReLU functions (over the integers), all our constructions work with the ReLU activation function as well, albeit with networks of twice the size.
Definition 3.1. Anm-bounded pushdown automaton (BPDA) is a tuple pΣ,Γ,m, µ, λ, ρq wherem P N, Σ is an alphabet of input symbols, Γ is an alphabet of stack symbols, µ : Γďm ˆΣˆΓďm Ñ r0, 1s is a weighted transition function, λ : Γďm Ñ r0, 1s is the initial weight function, and ρ : Γďm Ñ r0, 1s is the final weight function.
The string γ currently stored in the stack is the BPDA’s configuration. We read γ bottom to top, e.g., in the stack γ “ γ1γ2 ¨ ¨ ¨ γℓ, γ1 is at the bottom of the stack, while γℓ is at the top. We say that the m-bounded stack is empty if |γ| “ 0 (equivalently, γ “ ε) and full if |γ| “ m. Sometimes, it will be useful to think of the bounded stack as always being full—in that case, if there are ℓ ă m elements on the stack, we assume that the other m´ ℓ elements are occupied by a special placeholder symbol ι R Γ; see Fig. 3 for an illustration. We denote Γι
def“ Γ Y tιu. We call a BPDA probabilistic if the following
two equations hold
ÿ
γPΓďm λ pγq “ 1 (6a)
ÿ
yPΣ γ1PΓďm
µ ` γ, y,γ 1 ˘ ` ρ pγq “ 1,@γ P Γďm. (6b)
We call a BPDA P “ pΣ,Γ,m, µ, λ, ρq deterministic if there exists exactly one γ P Γďm with λ pγq ą 0 and, for any γ P Γďm and y P Σ, there is at most one γ 1 P Γďm with µ pγ, y,γ 1q ą 0. For deterministic BPDAs, we also define:
• ϕ pγ, yq def“ γ 1 for µ pγ, y,γ 1q ą 0 as the function returning the (deterministic) next configuration of the BPDA,
• ω pγ, yq def“ w for w “ µ pγ, y, ϕ pγ, yqq as the weight of the only y-labeled transition from γ, and
• φ pyq as the (unique) stack configuration reached upon reading y.
Runs of a BPDA. A BPDA processes a string y “ y1 ¨ ¨ ¨ yT P Σ˚ left to right by reading its symbols and changing its configurations accordingly. It starts with an initial configuration γ0 according to λ. Then, it updates the stack according to µ for each symbol in y, resulting in a sequence of stacks π “ γ0,γ1, . . . ,γT where µ pγt´1, yt,γtq ą 0. Each such sequence of stacks π is called a run and
we will denote all runs of P on y as Π py;Pq. A probabilistic BPDA P assigns y the probability
P pyq def“ (7) ÿ
πPΠpy;Pq, π“γ0,γ1,...,γT
λ pγ0q « Tź
t“1 µ pγt´1, yt,γtq
ff ρ pγT q.
In this sense, probabilistic BPDAs induce distributions over strings and generalize Hewitt et al.’s (2020) thresholded string acceptance to the probabilistic setting.7 This is illustrated in Fig. 3.
Pushing and popping. We define the transition function µ as a general function of the stack configuration and the input symbol. As important special cases, µ can define the standard POP and PUSH operations. Popping the top of stack τ is performed by transitions of the form pγτ , y,γq with µ pγτ , y,γq ą 0;
POP pγτ , y,γq def“ µ pγτ , y,γq. (8) Pushing definitionally increases the size of the stack, which raises the question of how to handle stack overflows. Rather than simply rejecting PUSHs that would result in a stack overflow, we allow a BPDA to discard the bottom of the stack when pushing; this is easy to specify with the general transition function µ. Given the current stack configuration γ “ γ1 ¨ ¨ ¨ γℓ, we define the weight of pushing the symbols τ “ γ11 ¨ ¨ ¨ γ1r P Γďm as PUSH pγ, y,γτ q def“ (9)# µ pγ, y,γτ q if ℓ` r ď m µ pγ, y, γℓ`r´m`1 ¨ ¨ ¨ γℓτ q otherwise .
7Hewitt et al.’s (2020) construction focuses on binary recognition of languages through their notion of truncated language recognition (cf. Def. A.1 in App. A.1). Specifically, they show that RNN LMs can assign high enough probabilities to correct continuations of strings while assigning low probabilities to incorrect continuations.
This definition is without loss of generality; we can always define BPDAs that assign transitions resulting in a stack overflow probability 0. PUSH is illustrated in Fig. 4.

heading: 3.1 Efficiently Representable BPDAs
text: It is easy to see that there are O p|Γι|mq possible m-bounded stacks over Γ, defining the number of configurations a BPDA can be in. Interpreting the configurations as states of a (probabilistic) FSA, general constructions of RNN simulating FSAs mentioned in §1 would require Ω ´ |Σ||Γι|m2 ¯ neurons, exponentially many in the size of the stack (Minsky, 1954; Dewdney, 1977; Indyk, 1995; Svete and Cotterell, 2023a,b). One might, however, hope to exploit the special structure of the state space induced by the stack configurations to come up with more efficient representations. This motivates the following definition.
Definition 3.2. A BPDA P “ pΣ,Γ,m, µ, λ, ρq is called C-efficiently representable if there exists a weakly equivalent RNN LM of size
D ď Crlog2 |Γι|ms “ Cmrlog2 |Γι|s. (10)
Intuitively, Def. 3.2 characterizes efficiently representable BPDAs as those that can be defined by an RNN LM with logarithmically many neurons in the number of configurations. This is a natural generalization of (efficient) thresholded acceptance defined by Hewitt et al. (2020) to the probabilistic (language modeling) setting.8
Def. 3.2 paves the way to a formalization of LMs efficiently representable by RNN LMs with
8Thresholded acceptance defines string recognition based on their conditional probabilities. See App. A.1 for details.
BPDAs. However, BPDAs are too rich to be efficiently representable in general. The following theorem shows that BPDA LMs are weakly equivalent to LMs induced by probabilistic finite-state automata, a classic family of computational models with a well-described relationship to RNNs.
Theorem 3.1. The family of LMs induced by PFSAs is weakly equivalent to the family of LMs induced by BPDAs.
Proof. See App. B. ■
Because BPDAs define the same class of LMs as probabilistic FSAs, they cannot, in general, be represented more efficiently than with Ω p|Σ||Γι|mq neurons—a more efficient simulation of general BPDAs would contradict the known lower bounds on simulating FSAs (Indyk, 1995; Svete and Cotterell, 2023b). This implies that we inevitably must restrict the class of BPDAs to ensure their efficient representability. The following definitions will help us characterize efficiently representable BPDAs.
Vectorial representation of a stack. When connecting BPDAs to RNNs, it is useful to think of the stack as a vector. Setting G def“ rlog2 |Γι|s, we define the stack vector function χ : Γďm Ñ t0, 1umG as
χpγ1 ¨ ¨ ¨ γmq def“
¨ ˚̋
Binpγmq ... Binpγ1q
˛ ‹‚P t0, 1umG. (11)
Here, Bin : Γι Ñ t0, 1uG is the binary encoding function, where we assume that Binpιq “ 0G, the G-dimensional vector of zeros. More precisely, let c : Γι Ñ t0, 1, . . . , |Γ|u be a bijection between Γι and t0, 1, . . . , |Γ|u such that c pιq “ 0. Then Binpγq is the binary representation of c pγq. Definition 3.3. A function f : Γďm Ñ Γďm is stack-affine if there exists a matrix M P RmGˆmG and a vector v P RmG such that, for all γ P Γďm, it holds that χpf pγqq “ Mχpγq ` v. Definition 3.4. A function ζ : Γďm ˆ Σ Ñ Γďm is K-varied in Σ if there exists a partition Σ “ Σ1 \ ¨ ¨ ¨ \ ΣK such that, for all k “ 1, . . . ,K, it holds that ζ pγ, yq “ ζ pγ, y1q def“ ζk pγq for all y, y1 P Σk and γ P Γďm. Definition 3.5. A function α : Γďm ˆ Σ Ñ Γďm is Σ-determined if there exists a function s : Σ Ñ Γ and a family of partitions
prms “ J y1 \ J y2 \ J y3 qyPΣ such that, for all γ P Γďm, it holds that
α pγ, yqj “ $ ’& ’% γj if j P J y1 s pyq if j P J y2 ι if j P J y3 . (12)
In words, a stack-affine function can be implemented by an affine transformation of the vectorial representation of the stack, aK-varied function can be decomposed into K different functions that are invariant to the input symbol, and a Σ-determined function changes the stack in a manner that only depends on the input symbol: it either keeps a symbol the same, replaces it with s pyq, or empties the slot (inserting the placeholder symbol ι). Importantly, a Σ-determined function acts independent of the stack γ.
Lastly, we consider the efficient representation of next-symbol probabilities. For a deterministic BPDA P “ pΣ,Γ,m, µ, λ, ρq, we define
p py | γq def“ ω pγ, yq (13a) p pEOS | γq def“ ρ pγq, (13b)
for γ P Γďm and y P Σ and p py | yq def“ p py | φ pyqq , (14)
for y P Σ˚ and y P Σ. Definition 3.6. A deterministic BPDA is representation-compatible if there exists a matrix E P R|Σ|ˆmG and a vector u P R|Σ| such that, for every γ P Γďm, it holds for all y P Σ that
log p py | γq “ softmaxpEχpγq ` uqy. (15)

heading: 4 Efficiently Representing BPDAs
text: The introduced technical machinery allows us to present our main result.
Theorem 4.1. Let P “ pΣ,Γ,m, µ, λ, ρq be a deterministic representation-compatible (cf. Def. 3.6) BPDA where
µ ` γ, y,γ 1 ˘ “ # ω pγ, yq if γ 1 “ α pζ pγ, yqq 0 otherwise
(16) for a Σ-determined function α (cf. Def. 3.5) and a K-varied function (cf. Def. 3.4) ζ where all ζk are stack-affine (cf. Def. 3.3). Then, P is K-efficiently representable (cf. Def. 3.2).
This generalizes Hewitt et al.’s (2020) result by considering the more general class of BPDA LMs, which includes Dpb, nq LMs as a special case, and by incorporating the probabilistic nature of LMs. The requirement for the BPDA to be representation compatible is crucial for Thm. 4.1; there exist BPDAs that fulfill all the criteria of Thm. 4.1 but the one on representation compatibility that are not efficiently representable by RNN LMs.
Theorem 4.2. There exist deterministic BPDAs whose transition function µ conforms to the structure in Eq. (16), but which are not efficiently representable for any K independent of |Σ|. Proof intuition. Consider a BPDA where Γ “ Σ.9 Intuitively, this holds because a BPDA LM defines exponentially many possible (independently specified) conditional distributions over the next symbol given the current stack configuration. For an RNN LM to be weakly equivalent, it would need to, after affinely transforming the hidden state (applying the output matrix and the output bias vector), match the logits of these distributions. However, as the logits can in general span the entire |Σ|dimensional space, the hidden state h would have to be of size Ω p|Σ|q to be able to do that, rather than Km log2 |Σ|, as we have in the case of efficient simulation. See App. C for details. ■

heading: 4.1 Known Efficiently Representable Families
text: Thm. 4.1 offers a very abstract characterization of (the components of) efficiently representable BPDAs. Here, we frame this characterization in the context of two well-known LM families: bounded Dyck and n-gram LMs.
Proposition 4.1. Representation-compatible LMs over Dpb, nq are 2-efficiently representable with m “ n and Γ “ txi| i P rbsu. Proof. LMs over Dpb, nq define distributions over strings of well-nested parentheses of b up do depth n. They work over the alphabet Σ “ txi| i P rbsuY tyi | i P rbsu. A BPDA modeling an LM over Dpb, nq only has to define POP and PUSH operations. Specifically, xi corresponds to a push operation PUSH pγ, xi,γxiq while yi corresponds to a pop operation POP pγxi, yi,γq for all i P rbs.10 Since popping can be performed by shifting all entries
9Examples of such BPDAs are those defining n-gram LMs.
10This is true because the input-to-stack-symbol function s here is taken to be the identity function.
on the stack one position up while pushing can be performed by shifting all entries one position down (before inserting a new symbol), the BPDA modeling a Dpb, nq language is stack-affine and 2- varied with the partition Σ “ ΣPOP \ ΣPUSH where ΣPOP “ tyi | i P rbsu, ΣPUSH “ txi| i P rbsu. Additionally, since each input symbol modifies the stack in a deterministic way—either it is discarded (in case of popping) or is added to the top position of the stack (in case of pushing)—the BPDA is Σ-determined. Moreover, since the stack only ever needs to store the b opening brackets, only rlog2 pb` 1qs bits are needed to represent each entry. Altogether, this means that Dpb, nq LMs are efficiently representable by RNN LMs of size D “ 2nrlog2 pb` 1qs. ■
Note that the construction presented by Hewitt et al. (2020) results in RNNs of size 2 ¨ 2nrlog2 pb` 1qs. This is because they rely on stack representations that contain the complements of the symbol encodings. This is not strictly required, which is interesting since Hewitt et al. (2020) note a difference in the constant factor between Elman RNNs and LSTMs. Our construction does away with this difference; see App. C for details.
Proposition 4.2. Representation-compatible ngram LMs are 1-efficiently representable with m “ n ´ 1 and Γ “ Σ. Proof. Representing an n-gram LM requires computing the probability of every symbol given the previous n ´ 1 symbols.11 This can be performed by, at step t, (1) storing the previous n ´ 1 symbols in the bounded stack of size m “ n ´ 1 and (2) checking the probability of the symbol yt given the n ´ 1 stored symbols. The required updates to the bounded stack can easily be performed by the stack-affine operation of shifting all symbols one position downward. Since all symbols perform that action, the function is 1-varied. Furthermore, since each input symbol induces the same update to the stack—the insertion of the symbol at the top of the stack—the BPDA is Σ-determined. This makes ngram LMs efficiently representable by RNN LMs of size D “ pn ´ 1q rlog2 p|Σ| ` 1qs. ■

heading: 5 Discussion
text: This work was motivated by the question of what classes of LMs beyond those over Dpb, nq can be
11The prefixes at the beginning of the string have to be appropriately padded.
efficiently represented by RNN LMs, a generalization of an open question posed by Hewitt et al. (2020). We address this with Thm. 4.1, whose implications we discuss next.
Analyzing LMs with general models of computation. This work puts the results by Hewitt et al. (2020) in a broader context of studying the representational capacity of RNN LMs not limited to human language phenomena. Restricting the analysis to isolated phenomena might not provide a holistic understanding of the model, its capabilities, and the upper bounds of its representational capacity. General models of computation provide a framework for such holistic analysis; besides being able to provide concrete lower and upper bounds on the (efficient) representational capacity, the thorough understanding of their relationship to human language also provides apt insights into the model’s linguistic capabilities. For example, the simulation of n-gram LMs shows that RNNs can efficiently represent representation-compatible strictly local LMs (Jäger and Rogers, 2012), a simple and wellunderstood class of LMs. This provides a concrete (albeit loose) lower bound on the efficient representational capacity of RNNs. Further, we directly study the efficient probabilistic representational capacity of RNN LMs rather than the binary acceptance of strings. This allows for a more natural and immediate connection between the inherently probabilistic neural LMs and probabilistic formal models of computation such as BPDAs.
Inductive biases of RNN LMs. Understanding neural LMs in terms of the classes of LMs they can efficiently represent allows us to reason about their inductive biases. When identifying the best model to explain the data, we suspect an RNN would prefer to learn simple representations that still effectively capture the underlying patterns, rather than more complex ones. We note that this is a form of inductive bias inherent to the model architecture; we do not address other defining aspects of inductive biases, such as the learning procedure itself. Understanding such inductive biases can then, as argued by Hewitt et al. (2020), lead to architectural improvements. One can, for example, design better architectures or training procedures that exploit these inductive biases, enforce them, or loosen them if they are too restrictive. In this light, Thm. 4.1 substantiates that there is nothing inherent in RNN LMs that biases them towards hierarchi-
cal languages since non-hierarchical ones can be modeled just as efficiently. Encouraging RNNs to learn and model cognitively plausible mechanisms for modeling language might, therefore, require us to augment them with additional mechanisms that explicitly model hierarchical structure, such as a stack (DuSell and Chiang, 2023). Thm. 4.1 also indicates that hierarchical languages might not be the most appropriate playground for studying the inductive biases of RNNs. For example, isolating the hidden state recurrence to the POP and PUSH operations disregards that, unlike a stack, an RNN can look at and modify the entire hidden state when performing the update; POP and PUSH operations are limited to modifying of the top of the stack.

heading: Inductive biases and learnability. A defining
text: aspect of an inductive bias is its effect on the learning behavior. While we do not discuss the learnability of bounded stack LMs by RNN LMs, we underscore the importance of this factor for a full understanding of inductive biases. Both theoretical and empirical insights are required; they provide an exciting avenue for future work, one which we see as deserving of its own treatment. More broadly, the inductive biases of a particular neural model rely on various aspects beyond the architecture, including the learning objective, the training algorithm, and features of the training data such as its ordering and size. With this in mind, we note that our results in no way suggest that RNNs are any worse at modeling hierarchical languages than nonhierarchical ones. The results merely provide a first step towards a more thorough understanding of the inductive biases and suggest that studies striving to understand RNNs’ learning behavior should look beyond hierarchical languages.
On the connection to human language. Our paper was motivated by the question of whether RNNs can efficiently represent languages beyond Dpb, nq, which models hierarchical structures prominent in human language. Interestingly, our exploration of a broader class of efficiently representable LMs revealed that n-gram LMs, another class of models useful for studying human language processing, are also efficiently representable (Bickel et al., 2005; Shain et al., 2020; Wang et al., 2024). This suggests a compelling interpretation: While RNNs may not naturally prefer hierarchical languages, our extended framework associates them with a more extensive range of
human-relevant languages. This opens new avenues for understanding the inductive biases of RNNs and their connection to human language, encouraging further theoretical and empirical studies on the specific facets of human language that efficiently representable BPDAs can represent and that RNNs are adept at modeling.

heading: A new interpretation of Elman RNN recurrence.
text: As detailed by the construction in the proof of Thm. 4.1, the core principle behind the efficient representation lies in the utilization of different parts of the hidden state as placeholders for relevant symbols that have occurred in the context; see also Fig. 1. This suggests that a natural interpretation of the languages efficiently representable by Elman RNNs: Those recognized by an automaton that keeps a memory of a fixed number of elements that have occurred in the string so far.
Relation to other classes of formal languages. Striving to make the results as general as possible motivates the connection of bounded stack languages to other well-understood classes of languages. Since bounded stack languages are a particular class of finite-state languages, a natural question is how they relate to the existing subregular languages. Those have in the past been connected to various aspects of human language (Jäger and Rogers, 2012) and the representational power of convolutional neural LMs (Merrill, 2019) and transformers (Yao et al., 2021). At first glance, efficiently representable BPDAs do not lend themselves to a natural characterization in terms of known classes of sub-regular languages, but we plan on investigating this further in future work.

heading: 6 Conclusion
text: We build on Hewitt et al.’s (2020) results and show that RNNs can efficiently represent a more general class of LMs than those over bounded Dyck languages. Concretely, we introduce bounded stack LMs as LMs defined by automata that keep m selected symbols that have occurred in the string so far and update the memory using simple update mechanisms. We show that instances of such LMs can be represented by RNN LMs in optimal space. This provides a step towards a more holistic grasp of the inductive biases of RNN LMs.

heading: Limitations
text: We conclude by discussing some limitations of our theoretical investigation and point out some ways these limitations can be addressed. We first touch on the universality of the result implied by Thm. 4.1. While we provide a new, more general, class of languages efficiently representable by RNNs and discuss the implications of this result, we do not provide an exact characterization of RNNs’ efficient representational capacity. In other words, we do not provide tight lower and upper bounds on the complexity of the languages that can be efficiently represented by RNNs. The lower bound of strictly local languages that can be encoded using parameter sharing is relatively loose (Dpb, nq languages are much more complex than strictly local languages) and the upper bound remains evasive. This is because we do not characterize the relation of bounded stack languages to other classes of languages, such as sub-regular languages. Determining precise bounds is further complicated by the unavoidable fact that the efficient representational capacity of RNNs also depends on the parameterization of the specific probability distributions formal models of computation can represent, as made clear by Thm. 4.2. Establishing more concrete bounds is therefore a challenging open problem that is left for future work.
We note that all our results are also specifically tailored to Elman RNNs with the specific update rule from Eq. (3b). However, the results naturally generalize to the LSTM architecture in the same way that Hewitt et al.’s (2020) original constructions do.

heading: Ethics Statement
text: The paper provides a way to theoretically analyze language models. To the best of the authors’ knowledge, this paper has no ethical implications.

heading: Acknowledgements
text: Ryan Cotterell acknowledges support from the Swiss National Science Foundation (SNSF) as part of the “The Forgotten Role of Inductive Bias in Interpretability” project. Anej Svete is supported by the ETH AI Center Doctoral Fellowship. We thank the reviewers for their insightful comments and suggestions.

heading: A Related Work
text: This work is part of the ongoing effort to better apprehend the theoretical representational capacity of LMs, particularly those implemented with RNNs (Merrill, 2023). The study of the representational capacity of RNNs has a long history (see, e.g., McCulloch and Pitts, 1943; Minsky, 1954; Kleene, 1956; Siegelmann and Sontag, 1992; Hao et al., 2018; Korsky and Berwick, 2019; Merrill, 2019; Merrill et al., 2020; Hewitt et al., 2020; Merrill et al., 2022; Merrill and Tsilivis, 2022; Svete and Cotterell, 2023b; Svete et al., 2024, inter alia). Minsky (1954), for example, showed that binary-activated RNNs are equivalent to (deterministic) finite-state automata: They can represent any finite-state language and can be simulated by deterministic finite-state automata. Minsky’s construction of an RNN simulating a general deterministic FSA was extended to the probabilistic—language modeling—setting by Svete and Cotterell (2023b).12
Recently, increased interest has been put on the efficient representational capacity of LMs and their inductive biases. To emulate a deterministic probabilistic FSA A with states Q over the alphabet Σ,13 Minsky’s (1954) construction requires an RNN of size O p|Σ||Q|q. This was improved by Dewdney (1977), who established that a general FSA can be simulated by an RNN of size O ´ |Σ||Q|3{4 ¯ , and further by Indyk (1995), who lowered this bound to O ´ |Σ|a|Q| ¯
. The latter was also shown to be optimal for general (adversarial) FSAs. The starting point of this work, Hewitt et al. (2020), was the first to show the possibility of exponentially compressing a specific family of finite-state languages, namely the bounded Dyck languages. This provides an exponential improvement over the general FSA simulation results. In this work, we generalize this result to a more general class of languages, aiming to gain a more thorough understanding of the mechanisms that allow exponential compression with RNNs. As shown by Thm. 4.1, a possible candidate for the mechanism enabling exponential compression might be the notion of a bounded stack.

heading: A.1 Relation to Hewitt et al.’s (2020) Notion of Language Recognition
text: A crucial difference in our approach to that of Hewitt et al. (2020) is the direct treatment of language models rather than binary languages. The notion of recognition of binary languages by RNNs (or any other neural LM) can be somewhat tricky, since the LM assigns a probability to each string in the language, not just a binary decision of whether the string is in the language. To reconcile the discrepancy between the discrete nature of formal languages and the probabilistic nature of (RNN) LMs, Hewitt et al. (2020) define the so-called truncated recognition of a formal language. Definition A.1. Let Σ be an alphabet and α P R`. An LM p is said to recognize the language L Ď Σ˚ with the threshold α if for every y “ y1 . . . yT P L, it holds for all t P rT s that
p pyt | yătq ą α and p pEOS | yq ą α. (17) In words, a language L is recognized by p if p assigns sufficiently high probability to all allowed continuations of (sub)strings in L.14 The notion of truncated recognition allows Hewitt et al. (2020) to talk about the recognition of binary languages by RNNs and design RNNs efficiently representing bounded Dyck languages. In contrast to our results, they do not consider the exact probabilities in their constructions—they simply ensure that the probabilities of valid continuations in their constructions are bounded from below by a constant and those of invalid continuations are bounded from above by a constant. While this results in interesting insights and reusable mechanisms, as we showcase here, it is not clear how to generalize this approach to languages more general than the structured Dyck languages. Taking into account the exact probabilities
12Unlike binary FSAs, where non-determinism does not add any expressive power, non-deterministic probabilistic FSAs are more expressive than their deterministic counterparts. This is why, although Minsky’s (1954) construction only considers deterministic FSAs, it shows the equivalence of RNNs to all FSAs. The same cannot be said about the probabilistic setting, where the equivalence holds only for the deterministic case. The relationship between RNN LMs and general probabilistic FSAs is addressed in Svete et al. (2024).
13See App. B.1 for a formal definition of a PFSA. 14In contrast to p pyq, which necessarily diminishes with the string length, the conditional probabilities of valid continuations
are bounded from below.
allows us to consider a more general class of languages, for example, n-gram LMs. When applicable, thresholding the exact probabilities from our construction in the manner described by Def. A.1 allows us to reconstruct the construction of Hewitt et al. (2020), meaning that our results provide a convenient generalization of theirs.

heading: B Finite-state Automata and Bounded Stack Languages
text: The main part of the paper discussed the connection of LMs representable by BPDAs and RNNs. BPDA LMs, however, are a particular class of finite-state LMs, and we discuss this connection in more detail here.

heading: B.1 Probabilistic Finite-state Automata
text: We begin by more formally defining the notion of probabilistic finite-state automata (PFSAs). Probabilistic finite-state automata are a well-understood real-time computational model.
Definition B.1. A probabilistic finite-state automaton (PFSA) is a 5-tuple pΣ, Q, δ, λ, ρq where Σ is an alphabet, Q is a finite set of states, δ Ď Qˆ Σ ˆ Rě0 ˆQ is a finite set of weighted transitions where we write transitions pq, y, w, q1q P δ as q y{wÝÝÑ q1,15 and λ, ρ : Q Ñ Rě0 are functions that assign each state its initial and final weight, respectively. Moreover, for all states q P Q, δ, λ and ρ satisfy řqPQ λ pqq “ 1, and ř
q y{wÝÝÑq1Pδ
w ` ρ pqq “ 1.
We next define some basic concepts. A PFSA A “ pΣ, Q, δ, λ, ρq is deterministic if |tq | λ pqq ą 0u| “ 1 and, for every q P Q, y P Σ, there is at most one q1 P Q such that q
y{wÝÝÑ q1 P δ with w ą 0. Any state q where λ pqq ą 0 is called an initial state, and if ρ pqq ą 0, it is called a final state. A path π of length N is a sequence of subsequent transitions in A, denoted as
q1 y1{w1ÝÝÝÑ q2 y2{w2ÝÝÝÑ q3¨ ¨ ¨qN yN {wNÝÝÝÝÑ qN`1. (18)
The yield of a path is s pπq def“ y1 . . . yN . The prefix weight rw of a path π is the product of the transition and initial weights, whereas the weight of a path additionally has the final weight multiplied in. In symbols, this means
rwpπq def“ Nź
n“0 wn, (19) wpπq def“
N`1ź n“0 wn, (20)
with w0 def“ λpq1q and wN`1 def“ ρpqN`1q. We write ΠpAq for the set of all paths in A and we write ΠpA,yq for the set of all paths in A with yield y. The sum of weights of all paths that yield a certain string y P Σ˚ is called the stringsum, given in the notation below
A pyq def“ ÿ
πPΠpA,yq w pπq . (21)
The stringsum gives the probability of the string y.

heading: B.2 Bounded-stack Probabilistic Finite-state Automata
text: The main text presented bounded stack LMs as LMs defined by BPDAs. While the specification with respect to bounded stacks is enough to connect them to RNN LMs, we can also investigate weakly equivalent PFSAs, establishing an explicit connection to this well-studied class of computational models. This relationship is characterized by the following theorem.
Theorem 3.1. The family of LMs induced by PFSAs is weakly equivalent to the family of LMs induced by BPDAs.
Proof. ( ðù ). It is easy to see that any BPDA defines a PFSA: Since the set of possible bounded stack configurations is finite and µ simply defines transitions between the finitely many configurations,
15We further assume a pq, y, q1q triple appears in at most one element of δ.
Bounded stack RNN
one can think of the BPDA as defining transitions between the finitely many states represented by the configurations.
( ùñ ). Let A “ pΣ, Q, δ, λ, ρq be a PFSA. We define the weakly equivalent BPDA P “ pΣ,Γ,m, µ, λP , ρPq with Γ “ Q, m “ 1, λP “ λ, ρP “ ρ, µ pq, y, q1q “ δ pq, y, q1q for all q, q1 P Q and y P Σ. By noting that there is a trivial bijection between the stack configurations and the states of the PFSA as well as between the two transition functions, it is easy to see that the BPDA P is weakly equivalent16 to the PFSA A. ■
The interpretation of BPDA LMs in terms of three mechanisms—a stack modification function, an RNN, and a bounded-stack PFSA—is illustrated in Fig. 5, which shows the three update mechanisms in action on the same string.

heading: C Proofs
text: This section contains the proofs of the theorems stated and used in the paper.

heading: C.1 Emulating Stack Updates
text: This section contains the proof of Thm. 4.1. In many ways, it resembles the original exposition by Hewitt et al. (2020) but is presented in our notation and with the additional generality of BPDAs. Some aspects are also simplified in our framework (for example, we do not have to scale the inputs to the activation functions).
On the use of the Heaviside activation function. The proof below relies on the use of the Heaviside activation function. As mentioned in §2.1, due to the relationship between ReLU and H, the same results hold for ReLU-activated RNNs as well. More precisely, it is easy to show that
H piq “ ReLU piq ´ ReLU pi´ 1q (22) for all i P Z.17 All our results therefore map to the setting of ReLU-activated RNNs, but might require hidden states of twice the size to store the results of ReLU piq and ReLU pi´ 1q.18 Interestingly, the
16In fact, one could also prove strong equivalence. 17The restriction to the integers is enough since we are considering finite-precision RNNs. 18These two values can then be combined at the next step of the computation before being used as H piq.
duplication of the size of the hidden state is not required for the same reason as in Hewitt et al. (2020); they use hidden states of size 2KmG (in our notation) to store the logarithmic encodings and their complements. This is required since it allows for an easier explicit determination of next-symbol probabilities. We do not require that. Notice that the binary complement x of a vector x P t0, 1uD is its affine transformation x “ 1D ´ x. Thus, the transformation x ÞÑ x can be absorbed into the affine transformation Ex ` u.19 Theorem 4.1. Let P “ pΣ,Γ,m, µ, λ, ρq be a deterministic representation-compatible (cf. Def. 3.6) BPDA where
µ ` γ, y,γ 1 ˘ “ # ω pγ, yq if γ 1 “ α pζ pγ, yqq 0 otherwise
(16)
for a Σ-determined function α (cf. Def. 3.5) and a K-varied function (cf. Def. 3.4) ζ where all ζk are stack-affine (cf. Def. 3.3). Then, P is K-efficiently representable (cf. Def. 3.2).
Proof. The proof of Thm. 4.1 requires us to show that the RNN recurrence (cf. Eq. (3b)) can (i) implement the update mechanisms implementing the stack update mechanism of the BPDA and (ii) encode the same next-symbol probabilities as the BPDA. We do that by defining the appropriate recurrence and input matrices U and V, the bias vector b, and the output parameters E and u. We also use one-hot encodings of input symbols y, which we denote with JyK. More precisely, for a bijection h : Σ Ñ r|Σ|s, we define the entries of JyK P t0, 1u|Σ| as
JyKi def“ 1 ti “ h pyqu (23)
for i P r|Σ|s The condition Eq. (16) can be equivalently expressed as the requirement that (the deterministic) P defines the next-stack configuration function ϕ of the form
ϕ pγ, yq “ α pζ pγ, yq , yq (24)
for a Σ-determined function α and a K-varied function ζ where all ζk are stack-affine. Definitionally, it thus holds for all ζk that χpζk pγ, yqq “ χpζk pγqq “ Mkχpγq ` vk (25) for some matrix Mk P RmGˆmG and vk P RmG. We now define κ : Σ Ñ rKs as the function that maps a symbol y P Σ to the index of the set y belongs to in the partition of Σ. That is, we define
κ pyq def“ k if y P Σk, (26)
where Σ “ Σ1 \ . . .\ ΣK is the partition defined by the K-varied function ζ. Then, we have that
ϕ pγ, yq “ α `ζκpyq pγq , y ˘ “ α ´ Mκpyqχpγq ` vκpyq, y ¯ (27)
This motivates the following implementation of ϕ with the RNN recurrence:
1. Divide the hidden state h into K copies, one of those containing the vectorial representation of the actual stack (cf. Eq. (11)).
2. Depending on the input symbol y P Σ, perform the appropriate affine transformation h ÞÑ Mκpyqh` vκpyq.
3. Apply the Σ-determined function α.
19Interestingly, the recognition of bounded Dyck languages does not require the duplication of the hidden state size even when we use ReLU activation function. This is because whenever the network inserts a symbol on the stack (the only place where the Heaviside function behaves differently compared to the ReLU in the proof of Thm. 4.1), the slot where the symbol is inserted is already empty. Thus, the values of the stack encoding stay P t0, 1u, which means that they do not require the clipping performed by the ReLU function.
We thus define, for the (single) initial configuration of P , γ0,
η “ h0 def“ ¨ ˚̊ ˚̋ χpγ0q 0mG
... 0mG
˛ ‹‹‹‚P t0, 1u KmG, (28)
which encodes the initial configuration. In general, we will write
h “
¨ ˚̋ h1
... hK
˛ ‹‚P t0, 1uKmG. (29)
We will say that the hidden state h satisfies the single-copy invariance if at most one of the components h1, . . . ,hK is non-zero. This completes step 1.
To implement steps 2 and 3 we proceed as follows. To perform the K different stack-affine functions, we define the parameters
U def“
¨ ˚̋
M1 ¨ ¨ ¨ M1 ... . . . ... MK ¨ ¨ ¨ MK
˛ ‹‚P RKmGˆKmG (30a)
b def“
¨ ˚̋ v1
... vK
˛ ‹‚ P RKmG. (30b)
We also define the input matrix V as
V def“ `rpy1q ¨ ¨ ¨ rpy|Σ|q ˘ P RKmGˆ|Σ| (31) where, for y P Σ, we write
rpyq “
¨ ˚̋
rpyq1 ... rpyqK
˛ ‹‚P RKmG, (32)
and define
rpyqk def“ # z pyq if k “ κ pyq ´1mG otherwise , (33)
where finally
z pyqj “ $ ’& ’% 0G if j P J y1 2Binps pyqq ´ 1G if j P J y2 ´1G if j P J y3 , (34)
for j P rms. Here, 1G is the G-dimensional vector of ones. We now show that the parameters defined above simulate the stack update function ϕ correctly. Let y P Σ˚, γ def“ φ pyq, and y P Σ. Furthermore, assume (by an inductive hypothesis) that h satisfies the single-copy invariance and that the non-zero component of h contain χpγq. We want to show that h1 def“ H pUh ` VJyK ` bq (1) satisfies the single-copy invariance, and (2) that the non-zero copy in h1 contains exactly the encoding χpϕ pγ, yqq. Because of the single-copy invariance of h and the definition of U, we see that
Uh ` b “
¨ ˚̋
M1h ` v1 ...
MKh ` vK
˛ ‹‚. (35)
We now note that, for ζk to be a valid stack-affine function, it has to hold that Mkh ` vk P t0, 1umG. Now, let k P rKs. We distinguish two cases:
• k “ κ pyq. Then VJyK “ rpyqκpyq “ z pyq . (36)
This results in the entries
H ´ Mκpyqh ` rpyqκpyq ` vκpyq ¯ j “ H ´ Mκpyqh ` z pyq ` vκpyq ¯ j
(37a)
“ H ˆ´ Mκpyqh ` vκpyq ¯ j ` z pyqj ˙ . (37b)
We further consider three cases
– j P J y1 . Then
H ˆ´ Mκpyqh ` vκpyq ¯ j ` z pyqj ˙ “ H ˆ´ Mκpyqh ` vκpyq ¯ j ` 0G ˙
(38a)
“ ´ Mκpyqh ` vκpyq ¯ j
(38b)
– j P J y2 . Then
H ˆ´ Mκpyqh ` vκpyq ¯ j ` z pyqj ˙ “ H ˆ´ Mκpyqh ` vκpyq ¯ j ` 2Binps pyqq ´ 1G ˙ (39a)
“ Binps pyqq (39b) This follows from the fact that 2Binps pyqq ´ 1G contains the value 1 wherever Binps pyqq is 1 and the value ´1 elsewhere, masking out the entries in the vector that are not active in Binps pyqq.
– j P J y3 . Then
H ˆ´ Mκpyqh ` vκpyq ¯ j ` z pyqj ˙ “ H ˆ´ Mκpyqh ` vκpyq ¯ j ´ 1G ˙
(40a)
“ 0G (40b) • k ‰ κ pyq. Then
VJyK “ rpyqk “ ´1mG, (41) resulting in Mkh ` rpyqk ` vk “ Mkh ´ 1mG ` vk, (42) whose entries are ď 0. This directly implies that
H ´ Mkh ` rpyqk ` vk ¯ “ 0mG, (43)
masking the kth component of h1.
Examining Eqs. (38b), (39b) and (40b) reveals that these results cover the three conditions of Σ-determined function α. Moreover Eq. (43) shows that the update rule preserves the single-copy invariance: All but the κ pyqth component of the hidden state are masked to 0. Summarizing, this shows that the RNN parametrized with the parameters U,V, and b correctly implements the stack update function ϕ.
To extend this to the probabilistic setting, we use the assumption that P is representation-compatible. By definition, P then defines next-symbol probabilities p py | yq where
log p py | φ pyqq “ softmax`E1χpφ pyqq ` u1˘ y
(44)
for some matrix E1 P R|Σ|ˆmG and u1 P R|Σ|. The first part of the proof shows that h pyq contains exactly one copy of χpφ pyqq. We use that fact and define
E def“ `E1 ¨ ¨ ¨ E1˘ P R|Σ|ˆKmG (45a)
u def“ u1 P R|Σ|, (45b)
which will result in the softmax-normalized RNN computing identical next-symbol probabilities to those computed by P . This means that R and P are weakly equivalent.
■

heading: C.2 On the Impossibility of Efficiently Representing All BPDA LMs
text: Theorem 4.2. There exist deterministic BPDAs whose transition function µ conforms to the structure in Eq. (16), but which are not efficiently representable for any K independent of |Σ|. Proof. The reason behind this is intuitive—the |Σ|m next-symbol probability distributions defined by a BPDA LM can be completely arbitrary and might not lend themselves to a compact parametrization with matrix multiplication (cf. Def. 3.6). This is why the proof of Thm. 4.1 relies heavily on specific families of BPDA LMs that are particularly well-suited for efficient representation by RNN LMs through parameter sharing. More formally, this is a special case of the softmax bottleneck (Yang et al., 2018; Chang and McCallum, 2022; Borenstein et al., 2024): The notion that the representations h pyq P RD defining an LM whose conditional logits span a d-dimensional subspace of R|Σ| must be of size D ě d. Because BPDAs can in general define full-rank distributions whose logits span R|Σ|, there exist BPDAs for which it has to hold that D ě |Σ|. Any such BPDA is not efficiently representable; |Σ| ď D ď Cm log2 |Σ| would require C ě |Σ|m log2 |Σ| , which is not constant in |Σ|. ■ Connection to the result by Hewitt et al. (2020). The impossibility result from Thm. 4.2 of course includes distributions over bounded Dyck languages as well. That is, a general distribution over a bounded Dyck language may not be efficiently represented by an RNN LM. This does not contradict the results from Hewitt et al. (2020)—the LMs considered by Hewitt et al. (2020) form a particular family of LMs that are efficiently representable by Elman RNN LMs. Intuitively, this is because of two reasons:
1. In their construction, the probability of the next symbol only depends on the top of the stack. As such, many stack configurations (and thus RNN hidden states) result in the same next-symbol probability distribution, allowing for more efficient encoding.
2. Hewitt et al. (2020) are only interested in recognizing binary languages, which means that they only consider LMs that assign sufficiently large probabilities to the correct continuations of the input string. Exact probabilities under the language model are not important.

heading: Future Work
text: Establishing more concrete bounds is therefore a challenging open problem that is left for future work. We note that all our results are also specifically tailored to Elman RNNs with the specific update rule from Eq. (3b). However, the results naturally generalize to the LSTM architecture in the same way that Hewitt et al. ’s (2020) original constructions do