Venue: NAACL 2024

# Lower Bounds on the Expressivity of Recurrent Neural Language Models

Anej Svate\({}^{*}\)   Franz Nowak\({}^{*}\)  Anisha Mohamed Sahabdeen  Ryan Cotterell

{asvete, fnowak, amohame, ryan.cotterell}@ethz.ch

Equal contribution.

###### Abstract

The recent successes and spread of large neural language models (LMs) call for a thorough understanding of their computational ability. Describing LMs' computational abilities through their representational capacity is a lively area of research. However, investigation into the representational capacity of neural LMs has predominantly focused on their ability to _recognize_ formal languages. For example, recurrent neural networks (RNN) with Heaviside activation functions are tightly linked to regular languages, i.e., languages defined by finite-state automata (FSAs). Such results, however, fall short of describing the capabilities of RNN _language models_ (LMs), which are definitionally _distributions_ over strings. We take a fresh look at the representational capacity of RNN LMs by connecting them to _probabilistic_ FSAs and demonstrate that RNN LMs with linearly bounded precision can express arbitrary regular LMs.

+
Footnote †: _Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 6820–6844 June 16-21, 2024 02024 Association for Computational Linguistics

[https://github.com/rycolab/nondeterministic-rnns](https://github.com/rycolab/nondeterministic-rnns)

## 1 Introduction

Neural language models (LMs) excel at many NLP tasks, e.g., machine translation, conversational AI, text classification, and natural language inference Cho et al. (2014); Albalak et al. (2022); Sun et al. (2023); Schick and Schutze (2021). Their strong empirical performance suggests the need for theory to explain what LMs can and cannot do in a formal sense. Such theory should serve to better understand as well as alleviate the practical limitations of neural LMs. However, the most common paradigm to perform such an analysis of neural LMs revolves around proving which formal languages common LM architectures can express Ackerman and Cybenko (2020); Icard (2020); Merrill et al. (2020); Perez et al. (2021), _inter alia_), i.e., to characterize the representational capacity of neural LMs with tools from the theory of computation.

Formally, a language model is a probability distribution over \(\Sigma^{*}\), the set of all strings from some alphabet \(\Sigma\).1 We say that two language models are weakly equivalent if they express the same distribution over strings. PFSAs (both deterministic and non-deterministic) and recurrent neural LMs, e.g., those derived from Elman recurrent neural networks (RNNs; Elman (1990), however, are instances of _families_ of language models, i.e., sets of distributions over strings that can be represented under the specific formalism. For instance, if one changes the parameters of a PFSA, one arrives at a different language model. Therefore, going beyond the formal comparison of individual LMs, we may also desire to make formal statements about families of models. One natural way to construct an (upper) bound on the representational capacity of an entire family of language models \(\mathcal{F}_{1}\), e.g., those expressible by RNNs, is to find another family of language models \(\mathcal{F}_{2}\) and show that, for every language model \(q\in\mathcal{F}_{2}\), there exists a weakly equivalent \(p\in\mathcal{F}_{1}\). Our goal in this paper is to show that, for every non-deterministic PFSA, there exists a weakly equivalent RNN LM.

Footnote 1: An alphabet is a finite, non-empty set of symbols.

The approach to studying language models presented in the previous paragraph differs fundamentally from most work that uses formal language theory to study the representational capacity of language models. Indeed, most papers try to relate language models to language _acceptors_Siegelmann and Sontag (1992); Merrill et al. (2020); Perez et al. (2021); Merrill et al. (2022), _inter alia_). E.g., finite-state automata have been linked to RNNs extensively before Alon et al. (1991); Indyk (1995); Merrill (2019); Merrill and Tsilivis (2022). However, acceptors are fundamentally different objects than LMs, as they define _sets_ of strings rather than distributions over them. Thus, analyzing acceptors, does not address our desire to understand recurrent neural LMs as probability distributions over \(\Sigma^{*}\).

In previous work, Svate and Cotterell (2023) demonstrate a particularly close relationship between recurrent neural language models and _deterministic_ PFSAs. However, their theory does not extend to the family of non-deterministic

PFSAs because it is a basic fact of probabilistic formal language theory that not all PFSAs can be determinized [15]. To date, no corresponding result exists for the non-deterministic case. To fill this void, we give a construction that shows real-time \({\rm ReLU}\)-activated Elman RNNs (Elman LMs) with linearly bounded precision can exactly simulate or arbitrary approximate _non_-deterministic PFSAs depending on the components employed in the network. Concretely, we show that _(1)_ Elman LMs with the sparsemax function [23] can represent any regular LM exactly; and, _(2)_ Elman LMs with the softmax projection function and a nonlinear output function can approximate a regular LM arbitrarily well. The roadmap of the paper is presented in Fig. 1.

## 2 Preliminaries

We begin by introducing some preliminaries.

### Language Modeling

Most modern neural LMs define the probability \(p\left(\boldsymbol{y}\right)\) of a string \(\boldsymbol{y}\in\Sigma^{*}\) as a product of conditional probability distributions \(p\), i.e.,

\[p\left(\boldsymbol{y}\right)\stackrel{{\mbox{\tiny def}}}{{=}} p\left(\textsc{cos}~{}|~{}\boldsymbol{y}\right)\prod_{t=1}^{|\boldsymbol{ y}|}p\left(y_{t}~{}|~{}\boldsymbol{y}_{<t}\right), \tag{1}\]

where EOS\(\notin\Sigma\) is a special end-of-sequence symbol. A language model expressed as in Eq. (1) is called **autoregressive**. Let \(\overline{\Sigma}\stackrel{{\mbox{\tiny def}}}{{=}}\Sigma\cup \left\langle\textsc{eos}\right\rangle\). Then, each \(p\left(\overline{y}_{t}~{}|~{}\boldsymbol{y}_{<t}\right)\) is a distribution over \(\overline{\Sigma}\). Additionally, we may also consider \(\varepsilon\)-augmented language models where each \(p\) is a distribution over \(\overline{\Sigma}\cup\left\{\varepsilon\right\}\) where \(\varepsilon\notin\Sigma\) is a special symbol not in the alphabet that represents the empty string. This allows the language model to perform computations _longer_ than the length of the string it generates. An autoregressive language model is called **real-time** if each \(p\) is _only_ a distribution over \(\overline{\Sigma}\), i.e., not over \(\overline{\Sigma}\cup\left\{\varepsilon\right\}\). Throughout this paper, we construct and investigate the expressivity of real-time autoregressive language models based on RNNs.

At a high level, we are interested in encoding real-time LMs as RNN LMs. To do so, we need a notion of equivalence between language models. In this paper, we will work with weak equivalence.

**Definition 2.1**.: _Two LMs \(p\) and \(q\) over \(\Sigma^{*}\) are **weakly equivalent** if \(p\left(\boldsymbol{y}\right)=q\left(\boldsymbol{y}\right)\) for all \(\boldsymbol{y}\in\Sigma^{*}\)._

### Regular Language Models

Probabilistic finite-state automata are a well-understood real-time computational model.

**Definition 2.2**.: _A **probabilistic finite-state automaton** (PFSA) is a 5-tuple \(\left(\Sigma,Q,\tau,\lambda,\rho\right)\) where \(\Sigma\) is an alphabet, \(Q\) is a finite set of states, \(\tau\subseteq Q\times\Sigma\times\mathbb{Q}_{\geq 0}\times Q\) is a finite set of weighted transitions where we write transitions \(\left(q,y,w,q^{\prime}\right)\in\tau\) as \(q\xrightarrow[]{y/w}q^{\prime}\),2 and \(\lambda,\rho\colon Q\to\mathbb{Q}_{\geq 0}\) are functions that assign each state its initial and final weight, respectively. Moreover, for all states \(q\in Q\), \(\tau,\lambda\) and \(\rho\) satisfy \(\sum_{q\in Q}\lambda\left(q\right)=1\), and \(\sum\limits_{q\xrightarrow[]{y/w}q^{\prime}\in\tau}w+\rho\left(q\right)=1\)._

Footnote 2: We further assume a \(\left(q,y,q^{\prime}\right)\) triple appears in at most _one_ element of \(\tau\).

We next define some basic concepts. A PFSA \(\mathcal{A}=\left(\Sigma,Q,\tau,\lambda,\rho\right)\) is **deterministic** if \(\left|\left\{q~{}|~{}\lambda\left(q\right)>0\right\}\right|=1\) and, for every \(q\in Q,y\in\Sigma\), there is at most one \(q^{\prime}\in Q\) such that \(q\xrightarrow[]{y/w}q^{\prime}\in\tau\) with \(w>0\).3 Any state \(q\) where \(\lambda\left(q\right)>0\) is called an **initial state**, and if \(\rho\left(q\right)>0\), it is called a **final state**. A **path \(\boldsymbol{p}\)** of length \(N\) is a sequence of subsequent transitions in \(\mathcal{A}\), denoted as

Footnote 3: In this paper, we do _not_ distinguish between a transition for a given symbol with weight \(w=0\) and the absence of a transition on that symbol. That is, we assume there always exists a transition \(q\xrightarrow[]{y/w}q^{\prime}\in\tau\) for any \(q,q^{\prime}\in Q\) and \(y\in\Sigma\), albeit possibly with \(w=0\). Such a choice turns out to be useful in our technical exposition; see Alg. 1.

\[q_{1}\xrightarrow[]{y_{1}/w_{1}}q_{2}\xrightarrow[]{y_{2}/w_{2}} q_{3}\cdots q_{N}\xrightarrow[]{y_{N}/w_{N}}q_{N+1}. \tag{2}\]

The **yield** of a path is \(\mathbf{s}\left(\boldsymbol{p}\right)\stackrel{{\mbox{\tiny def}}}{{=}} y_{1}\ldots y_{N}\). The **prefix weight**\(\widehat{\mathrm{w}}\) of a path \(\boldsymbol{p}\) is the product of

Figure 1: A roadmap of the results in this paper describing the representational capacity of Elman LMs. We start with an intuitive result showing how a sparsemax RNN with an \(\lVert\cdot\rVert_{1}\)-normalized hidden state can exactly simulate a regular LM. We then bring these modeling choices closer to practical implementations, leading us to the final result, which shows how sparsemax Elman LMs with an MLP-transformed hidden state can approximate arbitrary regular LMs.

 

the transition and initial weights, whereas the **weight** of a path additionally has the final weight multiplied in. In symbols, this means

\[\widetilde{\mathbf{w}}(\boldsymbol{p})\stackrel{{\text{def}}}{{=}} \prod_{n=0}^{N}w_{n},\quad\eqref{eq:w_1}\quad\mathrm{w}(\boldsymbol{p}) \stackrel{{\text{def}}}{{=}}\prod_{n=0}^{N+1}w_{n}, \tag{4}\]

with \(w_{0}\stackrel{{\text{def}}}{{=}}\lambda(q_{1})\) and \(w_{N+1}\stackrel{{\text{def}}}{{=}}\rho(q_{N+1})\). We write \(P(\mathcal{A})\) for the set of all paths in \(\mathcal{A}\) and we write \(P(\mathcal{A},\boldsymbol{y})\) for the set of all paths in \(\mathcal{A}\) with yield \(\boldsymbol{y}\). The sum of weights of all paths that yield a certain string \(\boldsymbol{y}\in\Sigma^{*}\) is called the **stringsum**, given in the notation below

\[\mathcal{A}\left(\boldsymbol{y}\right)\stackrel{{\text{def}}}{{= }}\sum_{\boldsymbol{p}\in P(\mathcal{A},\boldsymbol{y})}\mathrm{w}\left( \boldsymbol{p}\right). \tag{5}\]

The stringsum gives the probability of the string \(\boldsymbol{y}\). We say a state \(q\in Q\) is **accessible** if there exists a path with non-zero weight from an initial state to \(q\). A state \(q\in Q\) is **co-accessible** if there exists a path with non-zero weight from \(q\) to a final state. An automaton in which all states are accessible and co-accessible is called **trim**.

#### PFSAs as Autoregressive Language Models.

We now show how a PFSA \(\mathcal{A}\) induces a LM \(p_{\mathcal{A}}\) over strings \(\boldsymbol{y}\in\Sigma^{*}\). By Def. 2.2, the weights of all available transitions of a PFSA in state \(q\), together with the final weight, define a probability distribution over the next action, i.e., taking a transition or halting. We can translate this into a distribution over \(\overline{\Sigma}\) where eos corresponds to halting. In the following, we use the notation \(\overline{y}\in\overline{\Sigma}\) and \(y\in\Sigma\) to distinguish between symbols that could be EOS and those that cannot, respectively. Specifically, we can define the probability over \(\overline{\Sigma}\) as follows

\[p_{\mathcal{A}}\left(\overline{y}_{t}~{}|~{}q,\boldsymbol{y}_{<t}\right)= \begin{cases}\sum_{\frac{y_{l}/w}{q^{\prime}}\neq q^{\prime}}w&\textbf{if}\; \overline{y}_{t}\in\Sigma\\ \rho\left(q\right)&\textbf{if}\;\overline{y}_{t}=\textrm{eos}.\end{cases} \tag{6}\]

Moreover, in a PFSA, the probability of \(\overline{y}\) is conditionally independent of \(\boldsymbol{y}_{<t}\) given the state \(q\), i.e.,

\[p_{\mathcal{A}}\left(\overline{y}_{t}~{}|~{}q,\boldsymbol{y}_{<t}\right)=p_{ \mathcal{A}}\left(\overline{y}_{t}~{}|~{}q\right). \tag{7}\]

Finally, using the law of total probability, we can define an autoregressive language model as

\[p_{\mathcal{A}}\left(\overline{y}_{t}~{}|~{}\boldsymbol{y}_{<t} \right)\stackrel{{\text{def}}}{{=}}\sum_{q\in Q}p_{\mathcal{A}} \left(\overline{y}_{t}~{}|~{}q\right)p_{\mathcal{A}}\left(q~{}|~{}\boldsymbol{ y}_{<t}\right)\] \[=\sum_{q\in Q}p_{\mathcal{A}}\left(\overline{y}_{t}~{}|~{}q \right)\frac{p_{\mathcal{A}}\left(q,\boldsymbol{y}_{<t}\right)}{p_{\mathcal{A }}\left(\boldsymbol{y}_{<t}\right)} \tag{8a}\] \[=\sum_{q\in Q}p_{\mathcal{A}}\left(\overline{y}_{t}~{}|~{}q \right)\frac{p_{\mathcal{A}}\left(q,\boldsymbol{y}_{<t}\right)}{\sum_{q^{ \prime}\in Q}p_{\mathcal{A}}\left(q^{\prime},\boldsymbol{y}_{<t}\right)}, \tag{8b}\]

where \(p_{\mathcal{A}}\left(q,\boldsymbol{y}_{<t}\right)\) can be written as

\[p_{\mathcal{A}}\left(q,\boldsymbol{y}_{<t}\right)=\left(\overset{\top}{ \lambda}\overset{\top}{\underset{s=1}{\sum}}\mathbf{T}^{(y_{s})}\right)_{q}, \tag{9}\]

where \(\overset{\rightarrow}{\lambda}\) and \(\mathbf{T}^{(y_{s})}\) refer to the vectorized initial function and symbol-specific transition matrices of the PFSA \(\mathcal{A}\), respectively; see App. A for a full specification. Eq. (8a) shows that a PFSA induces a language model \(p_{\mathcal{A}}\) as in Eq. (1) through the conditional probabilities defined above.

Definition 2.3 ().: _An LM \(p\) is a **regular** language model if there exists a PFSA \(\mathcal{A}\) whose induced language model \(p_{\mathcal{A}}\) is weakly equivalent to \(p_{\mathcal{A}}\)._

See Fig. 2 for examples of PFSAs that induce regular language models over \(\Sigma=\{\mathtt{a},\mathtt{b}\}\).

#### PFSAs and FSAs.

Although PFSAs share many properties with unweighted (boolean-weighted) finite-state automata, one important difference relates to determinization. In the unweighted case, the class of deterministic and non-deterministic FSAs are equivalent, i.e., any non-deterministic FSA has an equivalent deterministic FSA that accepts the same language. This result, however, does not hold for PFSAs: There exist PFSAs that admit no deterministic equivalent Mohri (1997); Buchsaum et al. (2000), meaning that non-deterministic PFSAs are strictly more expressive than deterministic ones. For example, the PFSA in Fig. 1(b) is non-deterministic and does not admit a deterministic equivalent, since \(p(\mathtt{ab}^{n})\) cannot be expressed as a single term for arbitrary \(n\in\mathbb{N}_{\geq 0}\).

### Recurrent Neural Language Models

Recurrent neural LMs are LMs whose conditional probabilities are given by a recurrent neural network. Throughout this paper, we will focus on Elman RNNs Elman (1990) as they are the easiest to analyze and special cases of more common networks, e.g., those based on long short-term memory 

(LSTM; Hochreiter and Schmidhuber, 1997) and gated recurrent units (GRUs; Cho et al., 2014a).

**Definition 2.4**.: _An **Elman RNN** (ERNN) \({\mathcal{R}}=\left(\mathbb{Q}^{D},\Sigma,\sigma,\mathbf{U},\mathbf{V},\mathbf{b },\boldsymbol{\eta}\right)\) is an RNN with the following hidden state recurrence:_

\[\mathbf{h}_{0} =\boldsymbol{\eta} (t=0) \tag{10a}\] \[\mathbf{h}_{t} =\boldsymbol{\alpha}\left(\mathbf{U}\mathbf{h}_{t-1}+\mathbf{V} \mathbf{r}(y_{t})+\mathbf{b}\right)~{}(t>0), \tag{10b}\]

_where \(\mathbf{h}_{t}\in\mathbb{Q}^{D}\) is the hidden state vector4 at time step \(t\), \(\boldsymbol{\eta}\in\mathbb{Q}^{D}\) is an initialization parameter, \(y_{t}\in\Sigma\) is the input symbol at time step \(t\), \(\mathbf{r}\colon\Sigma\to\mathbb{Q}^{R}\) is a symbol representation function, \(\mathbf{U}\in\mathbb{Q}^{D\times D}\) and \(\mathbf{V}\in\mathbb{Q}^{D\times R}\) are parameter matrices, \(\mathbf{b}\in\mathbb{Q}^{D}\) is a bias vector, and \(\boldsymbol{\alpha}\colon\mathbb{Q}^{D}\to\mathbb{Q}^{D}\) is an element-wise non-linear activation function._

Footnote 4: Throughout this paper all vectors are column vectors.

Because \(\mathbf{h}_{t}\) hides the string that was consumed by the Elman RNN, we also use the evocative notation \(\mathbf{h}(\boldsymbol{y})\) to denote the result of the application of Eq. (10b) over the string \(\boldsymbol{y}=y_{1}\cdots y_{t}\). Common examples of \(\boldsymbol{\alpha}\) include the element-wise application of \(\mathrm{ReLU}\), i.e., \(\mathrm{ReLU}(x)\stackrel{{\text{def}}}{{=}}\max(0,x)\), the Heaviside function \(H(x)\stackrel{{\text{def}}}{{=}}\mathbbm{1}\left\{x>0\right\}\) and the sigmoid function \(\sigma(x)\stackrel{{\text{def}}}{{=}}\frac{1}{1+\exp(-x)}\). The \(\mathrm{ReLU}\) function is the most common choice of activation in modern deep learning and the focus of our analysis in this paper Goodfellow et al. (2016). To define an autoregressive language model, an Elman RNN constructs a distribution over the next symbol in \(\overline{\Sigma}\) by transforming the hidden state using some function \(\mathbf{F}\colon\mathbb{R}^{D}\to\mathbb{R}^{|\overline{\Sigma}|}\).

**Definition 2.5**.: _Let \({\mathcal{R}}\) be an ERNN and \(\mathbf{F}\colon\mathbb{R}^{D}\to\mathbb{R}^{|\overline{\Sigma}|}\) a differentiable function. An **Elman LM** is an LM whose conditional distributions are defined by projecting \(\mathbf{F}\left(\mathbf{h}_{t-1}\right)\) onto the probability simplex \(\boldsymbol{\Delta}^{|\overline{\Sigma}|-1}\) using a projection function \(\boldsymbol{\pi}\colon\mathbb{R}^{|\overline{\Sigma}|}\to\boldsymbol{\Delta} ^{|\overline{\Sigma}|-1}\):_

\[p_{\mathcal{R}}\left(\overline{y}_{t}\mid\boldsymbol{y}_{<t}\right)\stackrel{{ \text{def}}}{{=}}\boldsymbol{\pi}\left(\mathbf{F}\left(\mathbf{h}_{t-1} \right)\right)_{\overline{y}_{t}}. \tag{11}\]

_We call \(\mathbf{F}\) an **output function**._

The vector \(\mathbf{F}\left(\mathbf{h}_{t-1}\right)\in\mathbb{R}^{|\overline{\Sigma}|}\) thus represents a vector of \(|\overline{\Sigma}|\) values that are later projected onto \(\boldsymbol{\Delta}^{|\overline{\Sigma}|-1}\) to define \(p_{\mathcal{R}}\left(\overline{y}_{t}\mid\boldsymbol{y}_{<t}\right)\). The most common choice for the projection function \(\boldsymbol{\pi}\) is the **softmax** defined for \(\mathbf{x}\in\mathbb{R}^{N},N\in\mathbb{N}\), and \(n\in[N]\) as

\[\boldsymbol{\sigma}(\mathbf{x})_{n}\stackrel{{\text{ def}}}{{=}}\frac{\exp\left(\lambda\,x_{n}\right)}{\sum_{n^{\prime}=1}^{ N}\exp\left(\lambda\,x_{n^{\prime}}\right)}, \tag{12}\]

where \(\lambda\in\mathbb{R}\) is called the **inverse temperature** parameter. Softmax may be viewed as the solution to the following convex optimization problem:

\[\boldsymbol{\sigma}(\mathbf{x})=\operatorname*{argmax}_{\mathbf{z}\in \boldsymbol{\Delta}^{N-1}}\mathbf{z}^{\top}\mathbf{x}+\mathrm{H}(\mathbf{z}), \tag{13}\]

where \(\mathrm{H}(\mathbf{z})\stackrel{{\text{def}}}{{=}}-\sum_{n=1}^{N} z_{n}\log z_{n}\) is the Shannon entropy.5 An important limitation of the softmax is that it implies the LM has full support, i.e., an Elman LM with a softmax projection assigns positive probability to all strings in \(\Sigma^{*}\). To construct Elman LMs without full support, we also consider the **sparsemax**Martins and Astudillo (2016) as our projection function:

Footnote 5: See Boyd and Vandenberghe (2004, Ex. 25) for a derivation.

\[\operatorname*{sparsemax}(\mathbf{x})\stackrel{{\text{ def}}}{{=}}\operatorname*{argmin}_{\mathbf{z}\in\boldsymbol{\Delta}^{N-1}}\lVert\mathbf{z}- \mathbf{x}\rVert_{2}^{2}. \tag{14}\]

Note that this, too, is a convex optimization problem. In contrast to the softmax function, sparsemax is the **identity** on \(\boldsymbol{\Delta}^{N-1}\), i.e., we have \(\operatorname*{sparsemax}(\mathbf{x})=\mathbf{x}\) for \(\mathbf{x}\in\boldsymbol{\Delta}^{N-1}\).

Figure 2: Examples of PFSAs inducing probability distributions over \(\left\{\mathbf{a},\mathbf{b}\right\}^{*}\). Numbers in green signify initial weights, and numbers in red signify final weights.

 

Common output functions.We now outline three special cases of \({\bf F}\) which will be useful in our exposition.

* **Affine output Elman LMs**. A common way to implement \({\bf F}\) is as an affine transformation. In that case, \({\bf F}\) takes the form of \({\bf F}\left({\mathbf{h}}\right)\stackrel{{\text{def}}}{{=}}{\bf E h +d}\) for some \({\bf E}\in\mathbb{R}^{[\Sigma]\times D},{\bf d}\in\mathbb{R}^{[\Sigma]}\). We call \({\bf E}\) the **output matrix**.
* **Non-linear output Elman LMs**. In this case, \({\bf F}\) is an arbitrary non-linear function.
* **Deep output Elman LMs**. \({\bf F}\) is often implemented as a multi-layer perceptron in practice Pascanu et al. (2014). In that case, we call \({\bf F}\) a deep output Elman LM.

To concretely discuss the deep output Elman LMs, we now give the definition of a multi-layer perceptron, a common non-linear model we will deploy as the output function of the Elman RNN. Our definition is precise since we will later require an application of a universal approximation theorem.

**Definition 2.6**.: _A **multi-layer perceptron** (MLP) \({\bf F}\colon\mathbb{R}^{N}\to\mathbb{R}^{M}\) is a function defined as the composition of elementary functions \({\mathbf{f}}_{1},\cdots,{\mathbf{f}}_{L}\)_

\[{\bf F}\left({\bf x}\right)\stackrel{{\text{def}}}{{=}}{\mathbf{f}}_ {L}\circ{\mathbf{f}}_{L-1}\circ\dots\circ{\mathbf{f}}_{1}\left({\bf x}\right), \tag{15}\]

_where each function \({\mathbf{f}}_{\ell}\) for \(\ell\in[L]\) is defined as_

\[{\mathbf{f}}_{\ell}({\bf x}) \stackrel{{\text{def}}}{{=}}{\mathbf{\beta}}\left({\bf W }_{\ell}{\bf x}+{\bf b}_{\ell}\right)\quad\ell\in[L-1] \tag{16a}\] \[{\mathbf{f}}_{L}({\bf x}) \stackrel{{\text{def}}}{{=}}{\bf W}_{L}{\bf x}+{\bf b }_{L}, \tag{16b}\]

_where \({\bf W}_{\ell}\in\mathbb{R}^{N_{\ell}\times M_{\ell}}\) is a weight matrix with dimensions \(N_{\ell}\) and \(M_{\ell}\) specific to layer \(\ell\), \({\bf b}_{\ell}\in\mathbb{R}^{M_{\ell}}\) is a bias vector, and \({\mathbf{\beta}}\) is an element-wise nonlinear activation function. The function \({\mathbf{f}}_{1}\) is called the **input layer**, the function \({\mathbf{f}}_{L}\) is called the output layer, and the function \({\mathbf{f}}_{\ell}\) for \(\ell=2,\cdots,L-1\) are called **hidden layers**.6_ Footnote 6: Note that we refer to MLPs by the number of hidden layers, e.g., a one-layer-MLP is an MLP with one _hidden_ layer.

Besides their common application in neural LMs, MLPs with various activation functions also possess well-understood approximation abilities Cybenko (1989); Funahashi (1989); Hornik et al. (1989); Pinkus (1999). This makes their application to our goal--approximating language models--natural. In our constructions, we concretely focus on the result by Pinkus (1999) due to its generality. We restate it below in our own words, slightly adapting it to our use case:7

Footnote 7: In his original theorem, Pinkus (1999) only proves the approximation for real-valued functions \({\bf G}\), i.e., \(M=1\). Our restatement trivially extends the original theorem to a range of \(\mathbb{R}^{M}\) noting that one can use one large single-layer MLP consisting of \(M\) separate single-layer MLPs in parallel to approximate \(M\) individual functions. Furthermore, the original theorem also proves the converse direction that for polynomial \({\mathbf{\beta}}\) no such MLP exists, which we drop for brevity.

**Theorem 2.1**Pinkus (1999, Theorem 3.1).: _For any continuous, non-polynomial element-wise activation function \({\mathbf{\beta}}\), any continuous function \({\mathbf{\mathrm{G}}}:\mathbb{R}^{N}\to\mathbb{R}^{M}\), any compact subset \(\mathcal{K}\subset\mathbb{R}^{N}\), and any \(\epsilon>0\), there exists a single-layer MLP \({\bf F}:\mathbb{R}^{N}\to\mathbb{R}^{M}\) with activation function \({\mathbf{\beta}}\) such that \(\max\limits_{{\bf x}\in\mathcal{K}}\|{\bf G}({\bf x})-{\bf F}({\bf x})\|_{ \infty}<\epsilon\)._ Bounded Precision.We assume the hidden states \({\bf h}_{t}\) to be rational vectors. An important consideration in processing strings \({\mathbf{y}}\in\Sigma^{*}\) is then the number of bits required to represent the entries in \({\bf h}_{t}\) and how the number of bits scales with the length of the string, \(|{\mathbf{y}}|\). This motivates the definition of precision.

**Definition 2.7**.: _The **precision**\(\psi\left({\mathbf{y}}\right)\) of an Elman RNN is the number of bits required to represent the entries of \({\bf h}\left({\mathbf{y}}\right)\):_

\[\psi\left({\mathbf{y}}\right)\stackrel{{\text{def}}}{{=}}\max\limits_ {d\in[D]}\min\limits_{\begin{subarray}{c}p,q\in\mathbb{N},\\ p=n\left({\mathbf{y}}\right)_{d}\end{subarray}}\lceil\log_{2}p\rceil+\lceil\log_{2 }q\rceil. \tag{17}\]

_We say that an Elman RNN is of_

* _constant precision if_ \(\psi\left({\mathbf{y}}\right)=\mathcal{O}\left(1\right)\)_, i.e., if_ \(\psi\left({\mathbf{y}}\right)\leq C\) _for all_ \({\mathbf{y}}\in\Sigma^{*}\) _and some_ \(C\in\mathbb{R}\)_,_
* _logarithmically bounded precision_ _if_ \(\psi\left({\mathbf{y}}\right)=\mathcal{O}\left(\log|{\mathbf{y}}|\right)\)_, i.e., if there exist_ \(T_{0}\in\mathbb{N}\) _and_ \(C\in\mathbb{R}\) _such that for all_ \({\mathbf{y}}\in\Sigma^{*}\) _with_ \(|{\mathbf{y}}|\geq T_{0}\)_,_ \(\psi\left({\mathbf{y}}\right)\leq C\log_{2}|{\mathbf{y}}|\)_,_
* _linearly bounded precision_ _if_ \(\psi\left({\mathbf{y}}\right)=\mathcal{O}\left(|{\mathbf{y}}|\right)\)_, i.e., if there exist_ \(T_{0}\in\mathbb{N}\) _and_ \(C\in\mathbb{R}\) _such that for all_ \({\mathbf{y}}\in\Sigma^{*}\) _with_ \(|{\mathbf{y}}|\geq T_{0}\)_,_ \(\psi\left({\mathbf{y}}\right)\leq C|{\mathbf{y}}|\)_, and_
* _unbounded precision_ _if_ \(\psi\left({\mathbf{y}}\right)\) _cannot be bounded by a function of_ \(|{\mathbf{y}}|\)_._

The constructions in the existing literature range from constant to unbounded precision. The RNNs considered by Svete and Cotterell (2023), encoding of deterministic PFSAs, for example, are of constant precision. Weiss et al. (2018); Merrill (2019); Merrill et al. (2020), etc., consider models 

with logarithmically bounded precision. In contrast, Siegelmann and Sontag (1992) and Nowak et al. (2023) require unbounded precision to be able to represent possibly infinite running times required for the Turing completeness of RNNs.

## 3 Exact Simulation

This section presents our results on the exact simulation (that is, weak equivalence) of regular LMs with both sparsemax as well as softmax-normalized Elman LMs. The results rely on Thm. 3.1, which provides the basis for all our subsequent results by showing that Elman RNNs with linear precision can compute the state-string distributions \(p_{\mathcal{A}}\left(q,\mathbf{y}\right)\) (cf. Eq. (9)) of any PFSA \(\mathcal{A}\).

A note on notation.Throughout the paper, we implicitly index vectors and matrices directly with symbols for conciseness. Concretely, we assume an implicit ordering of the symbols and a corresponding ordering of the vector and matrix entries. This ordering is effected through a bijection \(n\colon\Sigma\rightarrow[|\Sigma|]\) (or any other set in place of \(\Sigma\)). We then define, for \(\mathbf{x}\in\mathbb{R}^{|\Sigma|}\), \(\mathbf{x}_{y}\stackrel{{\text{def}}}{{=}}\mathbf{x}_{n(y)}\) and, for \(\mathbf{X}\in\mathbb{R}^{|\Sigma|\times|\Sigma|}\), \(\mathbf{X}_{y,y^{\prime}}\stackrel{{\text{def}}}{{=}}\mathbf{ x}_{n(y),n(y^{\prime})}\) for \(y,y^{\prime}\in\Sigma\).

**Theorem 3.1**.: _Let \(\mathcal{A}\) be a PFSA with the state-string distribution \(p_{\mathcal{A}}\left(q,\mathbf{y}_{\leq t}\right)\) (cf. Eq. (9)). Then, there exists an Elman RNN \(\mathcal{R}\) with linearly bounded precision such that for all \(q\in Q\) and all \(\mathbf{y}_{\leq t}=y_{1}\ldots y_{t}\in\Sigma^{*}\)_

\[\mathbf{h}(\mathbf{y}_{\leq t})_{\left(q,y\right)}=\mathbbm{1}\left\{y=y_{t} \right\}p_{\mathcal{A}}\left(q,\mathbf{y}_{<t}\right). \tag{18}\]

Proof intuition.The proof translates the computation of the state-string probabilities under the PFSA (cf. Eq. (9)) into the recurrence of the Elman RNN. See App. B for the details. 

### Sparsemax-normalized Elman LMs

We now present the first result on the representational capacity of Elman LMs of linearly bounded precision: A lower bound showing that they can implement any regular LM, including those induced by non-deterministic PFSAs. This is formally captured by the following corollary of Thm. 3.1.

**Corollary 3.1**.: _Let \(\mathcal{A}=\left(\Sigma,Q,\tau,\lambda,\rho\right)\) be a trim PFSA inducing the tight LM \(p_{\mathcal{A}}\). Then, there exists a weakly equivalent, sparsemax-normalized nonlinear output Elman LM._

Proof intuition.Considering Eq. (8a), \(\mathbf{h}_{t-1}\) contains all the information needed to compute \(p_{\mathcal{A}}\left(\overline{y}_{t}\mid\mathbf{y}_{<t}\right)\). In accordance with Eq. (8b), the hidden state is \(\left\lVert\cdot\right\rVert_{1}\)-normalized and then multiplied by the output matrix \(\mathbf{E}\) containing the values \(p_{\mathcal{A}}\left(\overline{y}\mid q\right)\). See App. C for the full derivation. 

### Softmax-normalized Elman LMs

Cor. 3.1 shows that sparsemax-normalized Elman LMs can simulate any non-deterministic PFSA after normalizing the hidden states to unit \(\left\lVert\cdot\right\rVert_{1}\) norm. The sparsemax function, which is used in place of softmax, constitutes a crucial part of our construction. However, it is more standard to use the softmax in a practical implementation of an Elman RNN, but the softmax does not, unlike the sparsemax, equal the identity function on \(\mathbf{\Delta}^{N-1}\); \(\operatorname{softmax}\left(\mathbf{x}\right)\neq\mathbf{x}\) for \(\mathbf{x}\in\mathbf{\Delta}^{N-1}\). Luckily, by using a different output function \(\mathbf{F}\), we can achieve weak equivalence using the softmax projection function as well.

**Corollary 3.2**.: _Let \(\mathcal{A}=\left(\Sigma,Q,\tau,\lambda,\rho\right)\) be a trim PFSA inducing the tight LM \(p_{\mathcal{A}}\). Then, there exists a weakly equivalent, softmax-normalized nonlinear output Elman LM with an \(\overline{\mathbb{R}}\)-valued \(\mathbf{F}\), where \(\overline{\mathbb{R}}\stackrel{{\text{def}}}{{=}}\mathbb{R}\cup \left\{-\infty,\infty\right\}\) are the extended reals._

Proof intuition.The idea behind this construction is similar to that of Cor. 3.1. Rather than the explicit \(\left\lVert\cdot\right\rVert_{1}\)-normalization, however, this construction relies on an (\(-\infty\)-augmented) log transformation of the hidden state before the softmax normalization, which computes the appropriate probabilities from Eq. (8a). See App. C for the details. 

## 4 Approximate Simulation

SS3 concerns itself with the exact simulation of regular LMs with Elman LMs. The softmax-normalized result from Cor. 3.2, particularly, shows that a non-linear output Elman LM with an extended-real-valued function \(\mathbf{F}\) and a softmax projection function can perfectly simulate any PFSA. The construction, however, relies on the use of the extended reals. Restricting the output function \(\mathbf{F}\) to the \(\mathbb{R}\) induces approximation errors due to the full support of the softmax on real vectors. To bring the results on the representational capacity of Elman LM even closer to practical implementations, we now discuss how an \(\mathbb{R}\)-vector-valued output function \(\mathbf{F}\) can be used to _approximate_ regular LMs with Elman RNNs with vanishingly small approximation error. We first consider \(\mathbf{F}\) implemented by a custom non-linear function (the \(\log\)) and later as a common multi-layer perceptron. We use total 

variation distance to measure the approximation error between two language models:

**Definition 4.1**.: _Let \(p\) and \(q\) be two probability distributions over \(\Sigma^{*}\). Then the **total variation distance** between \(p\) and \(q\) is given by_

\[\textsc{tvd}(p,q)\stackrel{{\text{def}}}{{=}}\frac{1}{2}\sum_{ \boldsymbol{y}\in\Sigma^{*}}|p(\boldsymbol{y})-q(\boldsymbol{y})|. \tag{19}\]

### Approximation with \(\log\)

We first show how regular LMs can be approximated with \(\mathbb{R}\)-valued output functions. The following theorem states that the \(-\infty\) from Cor. 3.1 can be avoided--all outputs can be \(\mathbb{R}\)-valued--if we allow for an (arbitrarily small) approximation error.

**Theorem 4.1**.: _For any PFSA \(\mathcal{A}\) with induced LM \(p_{\mathcal{A}}\) and any \(\epsilon>0\), there exists a \(\mathrm{ReLU}\)-activated non-linear output Elman LM \(p_{\mathcal{R}}\) with an \(\mathbb{R}\)-vector-valued function \(\mathbf{F}\colon\mathbb{R}^{D}\to\mathbb{R}^{D}\) such that \(\textsc{tvd}(p_{\mathcal{A}},p_{\mathcal{R}})<\epsilon\)._

Proof intuition.: The proof relies on approximating the arbitrary PFSA \(\mathcal{A}\) with a _full support_ PFSA \(\mathcal{A}^{\prime}\) which approximates the LM induced by \(\mathcal{A}\) arbitrarily well in terms of the total variation distance. Since \(\mathcal{A}^{\prime}\) has full support, it can be represented exactly by a softmax-normalized Elman RNN with the same mechanism as in Cor. 3.2, resulting in a recurrent LM approximating the original PFSA \(\mathcal{A}\). See App. D for the full proof. 

This provides an approximation analog to Cor. 3.1. Due to the absence of infinities, this result is more representative of practical implementations. Nevertheless, the use of the \(\log\) output function is non-standard. We address this in the next section.

### Approximation with Pinkus (1999)

SS4.1 brings the results from SS3 to a more practical setting by considering the approximation of string probabilities. The usage of the non-standard \(\log\) transformation, however, is still unsatisfactory. In this section, we extend the result from the previous section to the most practically relevant setting, in which we consider the approximation of string probabilities under a regular LM using an Elman LM with an output function in the form of an MLP. Since such models are common practice in modern LMs Pascanu et al. (2014), we consider this to be the most impactful result of our investigation.

Concretely, we show that the approximation abilities of \(\mathrm{ReLU}\)-activated deep neural networks allow us to approximate the conditional probabilities of a PFSA arbitrarily well, providing theoretical backing for the strong performance of softmax-normalized ERNNs on regular languages.8

Footnote 8: Our construction applies an approximation result by Pinkus (1999), which characterizes wide fixed-depth neural networks. An analogous result could naturally be stated for fixed-width deep ReLU networks by applying any of the approximation theorems that cover such cases (e.g., Montufar et al., 2014; Yarotsky, 2017; Raghu et al., 2017; Hanin, 2019).

**Theorem 4.2**.: _Let \(p_{\mathcal{A}}\) be a regular LM induced by a PFSA \(\mathcal{A}\) and let \(\epsilon>0\). Then, there exists a \(\mathrm{ReLU}\)-activated, softmax-normalized deep output Elman LM \(p_{\mathcal{R}}\) such that \(\textsc{tvd}(p_{\mathcal{A}},p_{\mathcal{R}})<\epsilon\)._

Proof intuition.: The final construction relies on approximating two functions: _(1)_ the LM induced by the arbitrary \(\mathcal{A}\) with a full support PFSA \(\mathcal{A}^{\prime}\) and _(2)_ the \(\log\) output function. Thm. 4.1 solves step _(1)_ while the application of Thm. 2.1 to the approximation of \(\log\) solves step _(2)_. Importantly, the full support assumption provides the necessary compact domain over which \(\log\) is approximated. See App. D for the full proof. 

## 5 Discussion

SSSS 3 and 4 present a series of results describing the probabilistic representational capacity of Elman LMs. The results range from theorems that are simple, but detached from practical implementations (Cors. 3.1 and 3.2), to more elaborate theorems that describe the _approximation_ abilities of Elman LMs with modeling choices very close to those used in practice (Thm. 4.1). All results lower-bound the probabilistic representational capacity of Elman LMs, i.e., they state what Elman LM _can_ do. We discuss the implications of these lower bounds here.

Approximation abilities of softmax-normalized Elman LMs.The most practically relevant result in our paper is the approximation result of Thm. 4.2. Indeed, other results in this paper can be seen as a step-by-step build-up, each stripping away unpractical assumptions leading to this final approximation result. Thm. 4.2 shows that Elman LMs can approximate not only the individual conditional probabilities but rather any tight regular LM arbitrarily well as measured by total variation distance. Note that while arbitrarily good approximation of the _conditional_ probabilities does not necessarily imply the arbitrarily good approximation of full string probabilities (string probabilities 

are computed by multiplying the conditional probabilities of the individual symbols arbitrarily many times, resulting in a potentially large error), the assumption of tightness nevertheless allows us to make the total variation arbitrarily small--since the probability mass of a sufficiently long string under a tight language model diminishes with the string length, the strings with accumulating error only contribute marginally to the total variation distance between the true regular LM and its neural approximation.

Non-determinism and neural LMs.One of the main motivations for this investigation was capturing the non-determinism of regular LMs with neural LMs. Non-determinism is a prevalent notion in formal language theory as it allows both for more expressive formalisms as well as a more concise and intuitive construction of formal models of computation Hopcroft et al. (2006); Ch. 2). However, existing work has not addressed the implications of non-determinism on the link between regular LMs and neural LMs. To resolve this, the constructions presented here propose direct ways of implementing non-deterministic PFSAs with Elman LMs by means of linearly increasing the precision of hidden states, which is able to model all possible decisions of the PFSA directly. This allows us to lower-bound the representational capacity of real-time Elman LMs (cf. Cor. 3.1). This lower bound implies Elman LMs with linearly bounded precision are strictly more expressive than their counterparts with finite precision, which are only able to express _deterministic_ PFSAs Svete and Cotterell (2023). To the best of our knowledge, our construction is the first connection that tackles non-determinism directly by emulating the computation of string probabilities in real time.9 Moreover, addressing non-determinism also allows us to _compress_ otherwise determinizable automata into weakly equivalent non-deterministic ones and, thus, implement them in possibly exponentially smaller Buchsbaum et al. (2000) by directly simulating the smaller non-deterministic automaton with an Elman LM, a substantial improvement over existing work.10

Footnote 9: Alternatively, one could define neural LMs that simulate all accepting paths individually, which would necessitate the introduction of some form of non-determinism into the neural architecture Nowak et al. (2023, 2024).

Footnote 10: While such an LM would still require a hidden state linear in the number of states (like in the deterministic case), the number of states of the non-deterministic automaton would be smaller.

On the role of real-time processing.We limit ourselves to the real-time processing of strings by Elman LMs, as this most realistically captures how most modern LMs work Weiss et al. (2018). Moreover, real-time Elman LMs are not _upper_-bounded by regular LMs. Using the same stack encoding function as Nowak et al. (2023), one can show that Elman LMs with linearly bounded precision can probabilistically generate a weighting of a context-free Dyck language of nested parentheses because simulating the required stack can be performed in real-time.11 Moreover, by simulating multiple stacks, Elman LMs can simulate real-time Turing machines with arbitrarily many tapes Nowak et al. (2023), resulting in computational power exceeding that of PFSAs Rabin (1963); Rosenberg (1967). Going beyond real-time processing, a classical result by Siegelmann and Sontag (1992), and extended to the probabilistic case by Nowak et al. (2023), establishes that RNNs with unbounded precision can simulate Turing machines. However, the requirement of unbounded precision (and thereby computation time) is a departure from how RNNs function in practice. Nowak et al. (2023) place upper and lower bounds on the types of LMs ReLU-activated Elman RNNs can represent. Our real-time results stand in contrast to such Turing completeness theorems.

Footnote 11: Note that all (probabilistic) context-free languages can be recognized in real-time Greibach (1965); Abney et al. (1999).

Space complexity of emulating regular LMs.Our constructions show that hidden states of size \({\cal O}\left(|Q||\Sigma|\right)\) are sufficient to simulate arbitrary regular LMs. While we do not address the _lower bounds_ on the space required for the simulation of regular LMs, we conjecture that existing lower bounds for deterministic PFSAs apply here Svete and Cotterell (2023); Theorems 5.1 and 5.2). Concretely, the size of the Elman LM is lower-bounded by \(\min\left(|Q|,|\overline{\Sigma}|\right)\) due to the **softmax bottleneck** principle Yang et al. (2018); Svate et al. (2024); Borenstein et al. (2024). Because the PFSA inducing a regular LM can, in general, define \(|Q|\) probability distributions over \(\overline{\Sigma}\), a weakly equivalent Elman LM needs to model a \(\Omega\left(\min\left(|Q|,|\overline{\Sigma}|\right)\right)\)-dimensional space of logits to match the conditional probability distributions. This, in general, requires \(\Omega\left(\min\left(|Q|,|\overline{\Sigma}|\right)\right)\)-dimensional hidden states Chang and McCallum (2022).

 

Probabilistic Regular Languages.The relationship between RNNs and FSAs has attracted a lot of attention, resulting in a well-understood connection between RNNs and binary regular languages (Kleene, 1956; Cleeremans et al., 1989; Korsky and Berwick, 2019; Merrill, 2019; Merrill et al., 2020), as well as between RNNs and _deterministic_ probabilistic regular languages (Svete and Cotterell, 2023). To the best of our knowledge, Peng et al. (2018) are the first to discuss recurrent neural models that can simulate the computation of string probabilities under a set of distinct PFSAs. In particular, each dimension of the network's hidden state represents the probability of the input string under a specific PFSA. Their construction, however, does not consider the _lower bound_ of representational capacity of standard RNN architectures such as the Elman RNN, and, importantly, despite being weighted, is not connected to the language modeling setting. While some aspects of our construction resemble theirs, we notably extend it to the language modeling setting, where the generalization to the probabilistic case (Icard, 2020) is especially relevant.

## 6 Conclusion

We study a practically relevant formulation of Elman LMs with real-time processing and linear precision with respect to the string length and describe its probabilistic representational capacity in terms of regular LMs. Concretely, we show that Elman LMs with non-standard normalization and activation functions can simulate arbitrary non-deterministic PFSAs (Cors. 3.1 and 3.2). We then extend these exact simulation results to _approximation-based_ ones, where we show that Elman LMs as used in practice can approximate regular LMs arbitrarily well as measured by total variation distance (Thm. 4.2). The intuitive constructions provide a better understanding of the representational capacity of Elman LMs and showcase the probabilistic representational capacity of Elman LMs in a new light.

### Limitations

Since we only present lower bounds on the representational capacity of Elman LMs and no upper bounds, we do not provide a complete picture of their representational capacity. We also do not consider the _learnability_ of regular LMs with Elman LMs. This crucial part of language modeling is a much harder problem that remains to be tackled. However, many existing results from formal language theory show that learning regular LMs based on only positive examples (as done in modern approaches to language modeling) is hard or even impossible (Gold, 1967; Kearns and Valiant, 1994). The same naturally holds for the more expressive models of computation that Elman LMs can also emulate. This puts hard bounds on what can be learned in the language modeling framework from data alone, irrespective of the concrete parameterization of the model--be it RNNs, transformers, or any other model.

The restriction to the consideration of regular LMs of course fails to consider the non-regular phenomena of human language (Chomsky, 1957). While limits of human cognition might suggest that large portions of human language can be modeled by regular mechanisms (Hewitt et al., 2020; Svate et al., 2024), such formalisms lack the structure and interpretability of some mechanisms higher on the Chomsky hierarchy. However, the possibility of simulating many other real-time models of computation suggests that linearly bounded precision Elman LMs might indeed be able to capture more than just regular aspects of human language.

### Ethics Statement

The paper advances our theoretical understanding of language models. The authors do not foresee any ethical implications of this paper.

#### Acknowledgements

Ryan Cotterell acknowledges support from the Swiss National Science Foundation (SNSF) as part of the "The Forgotten Role of Inductive Bias in Interpretability" project. Anej Svete is supported by the ETH AI Center Doctoral Fellowship.

## References

* Abney et al. (1999) Steven Abney, David McAllester, and Fernando Pereira. 1999. Relating probabilistic grammars and automata. In _Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics_, pages 542-549, College Park, Maryland, USA. Association for Computational Linguistics.
* Ackerman and Cybenko (2020) Joshua Ackerman and George Cybenko. 2020. A survey of neural networks and formal languages. _arXiv preprint arXiv:2006.01338_.
* Albalak et al. (2020) Alon Albalak, Yi-Lin Tuan, Pegah Jandaghi, Connor Pryor, Luke Yoffe, Deepak Ramachandran, Lisse 

Getoor, Jay Pujara, and William Yang Wang. 2022. FETA: A benchmark for few-sample task transfer in open-domain dialogue. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Allauzen and Mohri (2003) Cyril Allauzen and Mehryar Mohri. 2003. Efficient algorithms for testing the twins property. Journal of Automata, Languages and Combinatorics8:117-144.
* Alon et al. (1991) Noga Alon, A. K. Dewdney, and Teunis J. Ott. 1991. Efficient simulation of finite automata by neural nets. Journal of the ACM38(2):495-514.
* Borenstein et al. (2024) Nadav Borenstein, Anej Svate, Robin Shing Moon Chan, Josef Valvoda, Franz Nowak, Isabelle Augenstein, Eleanor Chodroff, and Ryan Cotterell. 2024. What languages are easy to language-model? a perspective from learning probabilistic regular languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bangkok, Thailand. Association for Computational Linguistics.
* Boyd and Vandenberghe (2004) Stephen Boyd and Lieven Vandenberghe. 2004. _Convex Optimization_. Cambridge University Press.
* Buchsbaum et al. (2000) Adam L. Buchsbaum, Raffaele Giancarlo, and Jeffery R. Westbrook. 2000. On the determinization of weighted finite automata. SIAM Journal on Computing30(5):1502-1531.
* Chang and McCallum (2022) Haw-Shiuan Chang and Andrew McCallum. 2022. Softmax bottleneck makes language models unable to represent multi-mode word distributions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland. Association for Computational Linguistics.
* Cho et al. (2014a) Kyunghyun Cho, Bart van Merrienboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: Encoder-decoder approaches. In Proceedings of SSST-8, Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, Doha, Qatar. Association for Computational Linguistics.
* Cho et al. (2014b) Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1724-1734, Doha, Qatar. Association for Computational Linguistics.
* Chomsky (1957) Noam Chomsky. 1957. _Syntactic Structures_. De Gruyter Mouton, Berlin, Boston.
* Cleeremans et al. (1989) Axel Cleeremans, David Servan-Schreiber, and James L. McClelland. 1989. Finite state automata and simple recurrent networks. Neural Computation1(3):372-381.
* Cybenko (1989) G. Cybenko. 1989. Approximation by superpositions of a sigmoidal function. Mathematics of Control, Signals and Systems2(4):303-314.
* Du et al. (2023) Li Du, Lucas Torroba Hennigen, Tiago Pimentel, Clara Meister, Jason Eisner, and Ryan Cotterell. 2023. A measure-theoretic characterization of tight language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada. Association for Computational Linguistics.
* Elman (1990) Jeffrey L. Elman. 1990. Finding structure in time. Cognitive Science14(2):179-211.
* Funahashi (1989) Ken-Ichi Funahashi. 1989. On the approximate realization of continuous mappings by neural networks. Neural Networks2(3):183-192.
* Gao and Pavel (2018) Bolin Gao and Lacra Pavel. 2018. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprint arXiv:1704.00805.
* Gold (1967) E Mark Gold. 1967. Language identification in the limit. Information and Control10(5):447-474.
* Goodfellow et al. (2016) Ian Goodfellow, Yoshua Bengio, and Aaron Courville. 2016. Deep Learning. MIT Press. [http://www.deeplearningbook.org](http://www.deeplearningbook.org).
* Greibach (1965) Sheila A. Greibach. 1965. A new normal-form theorem for context-free phrase structure grammars. J. ACM12(1):42-52.
* Hanin (2019) Boris Hanin. 2019. Universal function approximation by deep neural nets with bounded width and relu activations. Mathematics7(10):992.
* Hewitt et al. (2020) John Hewitt, Michael Hahn, Surya Ganguli, Percy Liang, and Christopher D. Manning. 2020. RNNs can generate bounded hierarchical languages with optimal memory. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. Neural Computation9(8):1735-1780.
* Hopcroft et al. (2006) John E. Hopcroft, Rajeev Motwani, and Jeffrey D. Ullman. 2006. _Introduction to Automata Theory, Languages, and Computation (3rd Edition)_. Addison-Wesley Longman Publishing Co., Inc., USA.
* Hornik et al. (1989) Kurt Hornik, Maxwell Stinchcombe, and Halbert White. 1989. Multilayer feedforward networks are universal approximators. Neural Networks2(5):359-366.
* Icard (2020) Thomas F. Icard. 2020. Calibrating generative models: The probabilistic Chomsky-Schutzenberger hierarchy. Journal of Mathematical Psychology95:102308 

P. Indyk (1995)Optimal simulation of automata by neural nets. In STACS 95, Berlin, Heidelberg, pp. 337-348. External Links: ISBN 0261-4503 Cited by: SS1.
* M. Kearns and L. Valiant (1994)Crypographic limitations on learning boolean formulae and finite automata. Journal of the ACM41 (1), pp. 67-95. External Links: ISSN 0021-4503, Link, Document Cited by: SS1.
* S. C. Kleene (1956)Representation of events in nerve nets and finite automata. In CME, Chamon and J. McCarthy, Automata Studies, Vol. 34, pp. 3-42. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* S. A. Korsky and R. C. Berwick (2019)On the computational power of RNNs. arXiv preprint arXiv:1906.06349. Cited by: SS1.
* Volume 48, ICML'16, pp. 1614-1623. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* W. Merrill (2019)Sequential neural networks as automata. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges, Florence, pp. 1-13. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* W. Merrill, A. Sabharwal, and N. A. Smith (2022)Saturated transformers are constant-depth threshold circuits. Transactions of the Association for Computational Linguistics10, pp. 843-856. External Links: ISSN 0021-4503, Link, Document Cited by: SS1.
* W. Merrill and N. Tsilivis (2022)Extracting finite automata from rn using state merging. arXiv preprint arXiv:2201.12451. Cited by: SS1.
* W. Merrill, G. Weiss, Y. Goldberg, R. Schwartz, N. A. Smith, and E. Yahav (2020)A formal hierarchy of rnn architectures. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 443-459. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* M. Mohri (1997)Finite-state transducers in language and speech processing. Computational Linguistics23 (2), pp. 269-311. External Links: ISSN 0021-4503, Link, Document Cited by: SS1.
* Volume 2, NIPS'14, Cambridge, MA, USA, pp. 2924-2932. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* F. Nowak, A. Vete, Alexandra Butoi, and R. Cotterell (2024)On the representational capacity of neural language models with chain-of-thought reasoning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Bargkok, Thailand, pp. 830-838. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* F. Nowak, A. Svate, L. Du, and R. Cotterell (2023)On the representational capacity of recurrent neural language models. arXiv preprint arXiv:2310.12942. Cited by: SS1.
* R. Pascanu, C. Gulcehre, K. Cho, and Y. Bengio (2014)How to construct deep recurrent neural networks: proceedings of the second international conference on learning representations (iclr 2014). In 2nd International Conference on Learning Representations, ICLR 2014, Cited by: SS1.
* H. Peng, R. Schwartz, Sam Thomson, and Noah A. Smith (2018)Rational recurrences. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, pp. 1203-1214. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* Allan Pinkus (1999)Approximation theory of the mlp model in neural networks. Acta Numerica8, pp. 143-195. External Links: ISSN 0021-4503, Link, Document Cited by: SS1.
* J. Perez, P. Barcelo, and J. Marinkovic (2021)Attention is turing-complete. Journal of Machine Learning Research22 (75), pp. 1-35. External Links: ISSN 0021-4503, Link, Document Cited by: SS1.
* M. O. Rabin (1963)Real time computation. Israel Journal of Mathematics1 (4), pp. 203-211. External Links: ISSN 0021-4503, Link, Document Cited by: SS1.
* M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. Sohl-Dickstein (2017)On the expressive power of deep neural networks. In Proceedings of the 34th International Conference on Machine Learning, Proceedings of Machine Learning Research, PMLR. org, pp. 2847-2854. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* A. L. Rosenberg (1967)Real-time definable languages. Journal of the ACM14 (4), pp. 645-662. External Links: ISSN 0021-4503, Link, Document Cited by: SS1.
* Timo Schick and Hinrich Schutze (2021)Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, pp. 255-269. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* H. T. Siegelmann and E. D. Sontag (1992)On the computational power of neural nets. In Proceedings of the Fifth Annual Workshop on Computational Learning Theory, COLT '92, New York, NY, USA, pp. 440-449. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* X. Sun, X. Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang (2023)Text classification via large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, pp. 8990-9005. External Links: ISBN 0261-4503, Link, Document Cited by: SS1.
* A. Svete, R. Shing Moon Chan, and Ryan Cotterell (2024)A theoretical result on the inductive bias of rnn language models. arXiv preprint arXiv:2402.15814.

 

Anej Svate and Ryan Cotterell. 2023. Recurrent neural language models as probabilistic finite-state automata. In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_, pages 8069-8086, Singapore. Association for Computational Linguistics.
* Weiss et al. (2018) Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 740-745, Melbourne, Australia. Association for Computational Linguistics.
* Yang et al. (2018) Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bottleneck: A high-rank RNN language model. In _International Conference on Learning Representations_.
* Yarotsky (2017) Dmitry Yarotsky. 2017. Error bounds for approximations with deep relu networks. _Neural Networks_, 94:103-114.

 

Computation of String Probabilities Under PFSAs

In this section, we describe how a general PFSA computes string probabilities using standard linear algebra notation. This will allow for a concise connection to the Elman recurrence (cf. Eq.10b) later. Let \({\cal A}=(\Sigma,Q,\tau,\lambda,\rho)\) be a PFSA and let \(\mathbf{y}\in\Sigma^{*}\) be a string. Let \(\left\{\mathbf{T}^{(y)}~{}|~{}y\in\Sigma\right\}\) be the set of symbol-specific transition matrices of \({\cal A}\), i.e., the matrices with entries

\[\mathbf{T}_{(q,q^{\prime})}^{(y)}\stackrel{{\text{\tiny def}}}{{=}} \begin{cases}w&\textbf{if}\;q\xrightarrow{y/w}q^{\prime}\in\tau\\ 0&\textbf{otherwise}\end{cases}. \tag{20}\]

That is, \(\mathbf{T}_{(q,q^{\prime})}^{(y)}\) denotes the probability of transitioning from state \(q\) to state \(q^{\prime}\) with the symbol \(y\). Then, the probability of a string \(\mathbf{y}=y_{1}\ldots y_{T}\in\Sigma^{*}\) under \({\cal A}\) is given by:

\[p_{\mathcal{A}}\left(\mathbf{y}\right)=\overset{\rightarrow}{\lambda}^{\top} \mathbf{T}^{(y_{1})}\mathbf{T}^{(y_{2})}\cdots\mathbf{T}^{(y_{T})}\overset{ \rightarrow}{\rho}, \tag{21}\]

where \(\overset{\rightarrow}{\lambda}\stackrel{{\text{\tiny def}}}{{=}} \left(\lambda(q)\right)_{q\in Q}\) and \(\overset{\rightarrow}{\rho}\stackrel{{\text{\tiny def}}}{{=}} \left(\rho(q)\right)_{q\in Q}\) are vectors whose elements are the initial and final weights of the states of \({\cal A}\), respectively Peng et al. (2018).

## Appendix B Construction of an RNN to Simulate PFSAs

**Theorem 3.1**.: _Let \({\cal A}\) be a PFSA with the state-string distribution \(p_{\mathcal{A}}\left(q,\mathbf{y}_{\leq t}\right)\) (cf. Eq.9). Then, there exists an Elman RNN \({\cal R}\) with linearly bounded precision such that for all \(q\in Q\) and all \(\mathbf{y}_{\leq t}=y_{1}\ldots y_{t}\in\Sigma^{*}\)_

\[\mathbf{h}(\mathbf{y}_{\leq t})_{(q,y)}=\mathbbm{1}\left\{y=y_{t}\right\}p_{ \mathcal{A}}\left(q,\mathbf{y}_{<t}\right). \tag{18}\]

Proof.: To prove Thm.3.1 we must, for any (possibly non-deterministic) PFSA \({\cal A}=(\Sigma,Q,\tau,\lambda,\rho)\), construct an Elman RNN that keeps track of the state-string probabilities in its hidden states. At a high level, we want to show that the equality Eq.18 holds for all string prefixes \(\mathbf{y}_{\leq t}\in\Sigma^{*}\) for all time steps \(t\in\mathbb{N}\), that is, it is an invariance that holds for all \(t\in\mathbb{N}\). We provide the full construction of the RNN in multiple steps. The construction uses a hidden dimension of \(D=|Q||\Sigma|\).

Initial condition.The initial hidden state, \(\mathbf{h}_{0}=\mathbf{\eta}\) encodes the initial probability distribution over \(Q\)--the one defined by the initial weighting function \(\lambda\)--but only for positions corresponding to a single arbitrary dummy input symbol \(y_{0}\in\Sigma\):12

Footnote 12: Throughout the text, we assume that all non-specified entries of the parameters are set to \(0\).

\[\mathbf{\eta}_{(q,y_{0})}\stackrel{{\text{\tiny def}}}{{=}}\lambda( q),\qquad\quad\forall q\in Q=p_{\mathcal{A}}\left(q,\varepsilon\right). \tag{22}\]

Encoding the transition function.The Elman recurrence (cf. Eq.10b) will effectively simulate the dynamics of the PFSA. Intuitively, the updates to the hidden state will be analogous to parts of the computation Eq.21; the individual updates to the hidden state (through the multiplication with the recurrence matrix \(\mathbf{U}\)) are analogous to the multiplications with the matrices \(\mathbf{T}^{(y)}\). However, since the RNN cannot _choose_ among the possible transition matrices \(\left\{\mathbf{T}^{(y)}~{}|~{}y\in\Sigma\right\}\) when updating the hidden state (unlike the PFSA), the hidden state update rule will perform _all possible_ transitions in parallel and then mask out the transitions that are not applicable to the current symbol.

The recurrence matrix \(\mathbf{U}\in\mathbb{R}^{D\times D}\) thus encodes the transition weights. By having access to the hidden state \(\mathbf{h}_{t}\) through the matrix-vector multiplication, \(\mathbf{U}\) performs all possible transitions of the PFSA, i.e., the transitions for all symbols, in parallel. That is, for each symbol \(y\in\Sigma\), it computes a new distribution over states \(p_{\mathcal{A}}\left(q,\mathbf{y}_{\leq t}y\right)\).

The entries of the recurrence matrix are set as follows, for all \(y^{\prime}\in\Sigma\):

\[\mathbf{U}_{(q^{\prime},y^{\prime}),(q,y)}\stackrel{{\text{\tiny def }}}{{=}}\begin{cases}w&\mid\textbf{if}\;q\xrightarrow{y/w}q^{\prime}\in\tau\\ 0&\mid\textbf{otherwise}\end{cases}.\] (23) 

Since \(y^{\prime}\) is free, the product \(\mathbf{U}\mathbf{h}_{t-1}\) copies \(p_{\mathcal{A}}\left(q~{}|~{}\mathbf{y}_{<t}\right)\) into every entry associated with \(q\). Visually, the recurrence matrix \(\mathbf{U}\) consists of the transition matrices of each symbol, stacked vertically and copied horizontally, e.g., if \(\Sigma=\{\mathsf{a},\mathsf{b},\cdots,\mathsf{z}\}\):

\[\mathbf{U}=\begin{bmatrix}\mathbf{U}^{(\mathsf{a})}&\mathbf{U}^{(\mathsf{b})}& \cdots&\mathbf{U}^{(\mathsf{z})}\\ \mathbf{U}^{(\mathsf{a})}&\mathbf{U}^{(\mathsf{b})}&\cdots&\mathbf{U}^{( \mathsf{z})}\\ \vdots&\vdots&\vdots&\vdots\\ \mathbf{U}^{(\mathsf{a})}&\mathbf{U}^{(\mathsf{b})}&\cdots&\mathbf{U}^{( \mathsf{z})}\end{bmatrix}\quad\in\mathbb{R}^{D\times D}, \tag{24}\]

where each transition matrix \(\mathbf{U}^{(y)}\stackrel{{\mathrm{def}}}{{=}}(\mathbf{T}^{( y)})^{\top}\) has entries

\[\mathbf{U}^{(y)}_{q^{\prime},q}\stackrel{{\mathrm{ def}}}{{=}}\begin{cases}w&|\textbf{if}\ q\xrightarrow{y/w}q^{\prime}\in\tau\\ 0&|\textbf{otherwise}\end{cases}. \tag{25}\]

This way, multiplication with the recurrence matrix preserves the invariance in Eq. (18): equationparentequation

\[(\mathbf{U}\,\mathbf{h}_{t})_{(q^{\prime},y)} =\sum_{q^{\prime}\succeq q^{\prime}\in\tau}w\cdot p_{\mathcal{ A}}\left(q,\mathbf{y}_{\leqslant t}\right) \tag{26a}\] \[=\sum_{q\in Q}p_{\mathcal{A}}\left(y,q^{\prime}~{}|~{}q\right) \cdot p_{\mathcal{A}}\left(q,\mathbf{y}_{\leqslant t}\right)\] (26b) \[=\sum_{q\in Q}p_{\mathcal{A}}\left(y,q^{\prime}~{}|~{}q,\mathbf{y}_{ \leqslant t}\right)\cdot p_{\mathcal{A}}\left(q,\mathbf{y}_{\leqslant t}\right)\] (26c) \[=\sum_{q\in Q}p_{\mathcal{A}}\left(y,q^{\prime}~{}|~{}q,\mathbf{y}_{ \leqslant t}\right)\cdot p_{\mathcal{A}}\left(q~{}|~{}\mathbf{y}_{\leqslant t} \right)\cdot p_{\mathcal{A}}\left(\mathbf{y}_{\leqslant t}\right)\] (26d) \[=\Bigg{(}\sum_{q\in Q}p_{\mathcal{A}}\left(y,q^{\prime}~{}|~{}q, \mathbf{y}_{\leqslant t}\right)\cdot p_{\mathcal{A}}\left(q~{}|~{}\mathbf{y}_{ \leqslant t}\right)\Bigg{)}\cdot p_{\mathcal{A}}\left(\mathbf{y}_{\leqslant t}\right)\] (26e) \[=p_{\mathcal{A}}\left(y,q^{\prime}~{}|~{}\mathbf{y}_{\leqslant t} \right)\cdot p_{\mathcal{A}}\left(\mathbf{y}_{\leqslant t}\right)\] (26f) \[=p_{\mathcal{A}}\left(y,q^{\prime},\mathbf{y}_{\leqslant t}\right)\] (26g) \[=p_{\mathcal{A}}\left(q^{\prime},\mathbf{y}_{\leqslant t}y\right). \tag{26h}\]

We now have a distribution over states and symbols given the history. However, at time step \(t\) we also observe the actual input symbol, meaning that we want to only retain the distribution over states for the observed symbol.

We use the input matrix \(\mathbf{V}\) and the bias vector \(\mathbf{b}\) to zero out parts of the hidden state so that it only retains the probabilities of states reachable by reading the current symbol \(y_{t}\) and sets the rest to \(0\). Since the \(\operatorname{ReLU}\) activation function zeroes out negative values, it will be sufficient to subtract an appropriate bias term from the entries associated with unobserved symbols. This can be achieved by defining, for any \(q\in Q\) and \(y\in\Sigma\): equationparentequation

\[\mathbf{V}_{(q,y),(y)} \stackrel{{\mathrm{def}}}{{=}}1, \tag{27a}\] \[\mathbf{b} \stackrel{{\mathrm{def}}}{{=}}-\mathbf{1}_{D}, \tag{27b}\]

where \(\mathbf{1}_{D}\) is a \(D\)-dimensional vector of ones. This ensures that the negative bias values are balanced by the product \(\mathbf{V}\mathbf{r}(y_{t})\) exactly for the tuple entries associated with symbol \(y_{t}\): equationparentequation

\[\mathbf{h}\left(\mathbf{y}_{\leqslant t}y\right)_{(q^{\prime},y^{ \prime})} =\operatorname{ReLU}\big{(}\mathbf{U}_{(q^{\prime},y^{\prime})} \mathbf{h}\left(\mathbf{y}_{\leqslant t}\right)+\mathbf{V}_{(q^{\prime},y^{\prime}) }[\![y]\!]+\mathbf{b}_{(q^{\prime},y^{\prime})}\big{)} \tag{28a}\] \[=\operatorname{ReLU}\big{(}p_{\mathcal{A}}\left(q^{\prime},\mathbf{y} _{\leqslant t}y\right)+\mathbbm{1}\left\{y^{\prime}=y\right\}-1\big{)}\] (28b) \[=\mathbbm{1}\left\{y^{\prime}=y\right\}p_{\mathcal{A}}\left(q^{ \prime},\mathbf{y}_{\leqslant t}y\right). \tag{28c}\]

Since probability scores cannot be greater than 1, the values corresponding to symbols that were not observed will be zeroed out by the activation function 

Linearly Bounded Precision.Finally, we want to show that the RNN requires linearly bounded precision, i.e., there exist \(T_{0}\in\mathbb{N}\) and \(C\in\mathbb{R}_{\geq 0}\) such that for all \(\mathbf{y}\in\Sigma^{*}\) with \(|\mathbf{y}|\geq T_{0}\), \(\psi\left(\mathbf{y}\right)\leq C|\mathbf{y}|\). In the following, we write the precision required by a rational \(x\in\mathbb{Q}\) as

\[\psi(x)\stackrel{{\text{\tiny def}}}{{=}}\min_{ \begin{subarray}{c}p,q\in\mathbb{N}\\ \frac{p}{q}=x\end{subarray}}[\log_{2}p]+\lceil\log_{2}q\rceil. \tag{29}\]

Note that \(\psi(xy)\leqslant\psi(x)+\psi(y)\) for all \(x,y\in\mathbb{Q}\).

Two remarks will be important for the proof:

1. The number of bits required to represent any dimension of \(\mathbf{Vr}(y_{t})+\mathbf{b}\) is upper bounded by a constant \(C^{\prime}\) independent of \(t\), given by \[C^{\prime}\stackrel{{\text{\tiny def}}}{{=}}\max_{ y\in\Sigma}\max_{d\in[D]}\psi([\mathbf{Vr}(y)+\mathbf{b}]_{d}).\] (30)
2. The \(\mathrm{ReLU}\) either leaves a dimension of the hidden state unchanged or sets it to 0, meaning it never increases the required precision of its argument. We show linear precision by induction over \(|\mathbf{y}|\) for \(\mathbf{y}\in\Sigma^{*}\) with \(|\mathbf{y}|\geqslant T_{0}\stackrel{{\text{\tiny def}}}{{=}}1\), choosing \(C\) as \[C\stackrel{{\text{\tiny def}}}{{=}}\psi(D)+\max_{i,j \in[D]}\psi(\mathbf{U}_{ij})+\max_{d}\psi(\mathbf{\eta}_{d})+C^{\prime}.\] (31)

* **Base case:** For \(\mathbf{y}=y\) for some \(y\in\Sigma\), \[\psi\left(y\right) =\max_{d\in[D]}\psi(\mathbf{h}_{1,d})\] (32a) \[=\max_{d\in[D]}\psi(\mathrm{ReLU}(\mathbf{U}\mathbf{\eta}+\mathbf{ Vr}(y)+\mathbf{b})_{d})\] (32b) \[\leqslant\max_{d\in[D]}\psi((\mathbf{U}\mathbf{\eta})_{d})+C^{\prime}\] (32c) \[\leqslant\max_{d,i,j,i\in[D]}\psi(D\mathbf{U}_{i,j}\mathbf{\eta}_{d}) +C^{\prime}\] (32d) \[\leqslant\psi(D)+\max_{i,j\in[D]}\psi(\mathbf{U}_{ij})+\max_{d \in[D]}\psi(\mathbf{\eta}_{d})+C^{\prime}\] (32e) \[=C.\] (32f)
* **Induction step:** Assume that \(\psi\left(\mathbf{y}\right)\leqslant C|\mathbf{y}|\) for all \(\mathbf{y}\in\Sigma^{*}\) with \(|\mathbf{y}|=T\). Let \(\mathbf{y}\in\Sigma^{*}\) with \(|\mathbf{y}|=T\) be such a string. Then, for \(\mathbf{y}^{\prime}\stackrel{{\text{\tiny def}}}{{=}}\mathbf{y}y^{\prime}\) with \(y^{\prime}\in\Sigma\) (a string of length \(T+1\)), we have \[\psi\left(\mathbf{y}^{\prime}\right) =\max_{d\in[D]}\psi(\mathbf{h}_{t+1,d})\] (33a) \[=\max_{d\in[D]}\psi(\mathrm{ReLU}\left(\mathbf{U}\mathbf{h}_{t}+ \mathbf{Vr}\big{(}y^{\prime}\big{)}+\mathbf{b}\right)_{d})\] (33b) \[\leqslant\max_{d\in[D]}\psi((\mathbf{U}\mathbf{h}_{t})_{d})+C^{\prime}\] (33c) \[\leqslant\psi(D)+\max_{i,j\in[D]}\psi(\mathbf{U}_{ij})+\underbrace{ \max_{d\in[D]}\psi(\mathbf{h}_{t-1_{d}})}_{=\psi(\mathbf{y})}+C^{\prime}\] (33d) \[\leqslant C+C|\mathbf{y}|\] (33e) \[=C(\mathbf{y}+1)\] (33g) \[=C\mathbf{y}^{\prime}.\] Therefore, any \(\mathrm{ReLU}\)-activated Elman RNN over the rationals can be represented with linear precision. Note that for \(t=0\), \(\mathbf{h}_{0}=\mathbf{\eta}\), which requires constant precision depending only 

on the choice of \({\mathbf{\eta}}\) which is included in \(C\), so above we actually showed a stronger bound of \(\psi\left(t\right)\leqslant C(t+1)\) that applies to all time steps. Observe also that we use the fact that each computational step in our model corresponds to reading a symbol in \(\Sigma\); this is in contrast to models such as those investigated in Nowak et al. (2023), which are allowed to consume the empty string \(\varepsilon\).

## Appendix C Proofs of Exact Simulation

**Corollary 3.1**.: _Let \(\mathcal{A}=\left(\Sigma,Q,\tau,\lambda,\rho\right)\) be a trim PFSA inducing the tight LM \(p_{\mathcal{A}}\). Then, there exists a weakly equivalent, sparsemax-normalized non-linear output Elman LM._

Proof.: Let \(p_{\mathcal{A}}\left(q,{\mathbf{y}}\right)\) be \(\mathcal{A}\)'s state-string distribution (cf. Eq. (9)). By Thm. 3.1, there exists an Elman RNN \(\mathcal{R}=\left(\mathbb{Q}^{D},\Sigma,\sigma,\mathbf{U},\mathbf{V},\mathbf{ b},{\mathbf{\eta}}\right)\) with the hidden states

\[\mathbf{h}({\mathbf{y}}_{<t})_{\left(q,y\right)}=\mathbbm{1}\left\{y=y_{t-1} \right\}p_{\mathcal{A}}\left(q,{\mathbf{y}}_{<t}\right) \tag{34}\]

for all \({\mathbf{y}}_{<t}\in\Sigma^{*}\).

We now define a simple non-linear transformation \(\mathbf{F}\) that transforms \(\mathbf{h}({\mathbf{y}}_{<t})\) into values that are later normalized to \(p_{\mathcal{A}}\left(y_{t}~{}|~{}{\mathbf{y}}_{<t}\right)\) by the sparsemax. The function \(\mathbf{F}\) is a composition of two functions: _(1)_\(\left\lVert\cdot\right\rVert_{1}\)-normalization of the hidden state and _(2)_ linear transformation of the normalized hidden state that sums the outgoing transitions from \(q\in Q\) that emit \(\overline{y}_{t}\). \(\mathbf{F}\) will this take the form

\[\mathbf{F}\left(\mathbf{h}\right)\stackrel{{\text{def}}}{{=}} \mathbf{E}\ \frac{\mathbf{h}}{\left\lVert\mathbf{h}\right\rVert_{1}} \tag{35}\]

for some \(\mathbf{E}\in\mathbb{R}^{\left\lvert\overline{\Sigma}\right\rvert\times D}\) that we define below.

Defining \(\mathbf{h}\stackrel{{\text{def}}}{{=}}\mathbf{h}_{t}=\mathbf{h} \left({\mathbf{y}}_{<t}\right)\) and applying \(\left\lVert\cdot\right\rVert_{1}\) normalization to \(\mathbf{h}\) as \(\mathbf{h}^{\prime}\stackrel{{\text{def}}}{{=}}\frac{\mathbf{h}}{ \left\lVert\mathbf{h}\right\rVert_{1}}\) results in

\[\mathbf{h}^{\prime}_{\left(q,y\right)} =\frac{\mathbbm{1}\left\{y=y_{t-1}\right\}p_{\mathcal{A}}\left( q,{\mathbf{y}}_{<t}\right)}{\sum_{d^{\prime}}\left\lvert\mathbf{h}_{d^{\prime}}\right\rvert} \tag{36a}\] \[=\frac{\mathbbm{1}\left\{y=y_{t-1}\right\}p_{\mathcal{A}}\left( q,{\mathbf{y}}_{<t}\right)}{\sum_{q\in Q,y^{\prime}\in\Sigma}\left\lvert\mathbf{ h}_{\left(q^{\prime},y^{\prime}\right)}\right\rvert}\] (36b) \[=\frac{\mathbbm{1}\left\{y=y_{t-1}\right\}p_{\mathcal{A}}\left(q,{ \mathbf{y}}_{<t}\right)}{\sum_{q^{\prime}\in Q,y^{\prime}\in\Sigma}\mathbbm{1} \left\{y^{\prime}=y_{t-1}\right\}p_{\mathcal{A}}\left(q^{\prime},{\mathbf{y}}_{< t}\right)}\] (36c) \[=\frac{\mathbbm{1}\left\{y=y_{t-1}\right\}p_{\mathcal{A}}\left(q,{ \mathbf{y}}_{<t}\right)}{\sum_{q^{\prime}\in Q}p_{\mathcal{A}}\left(q^{\prime},{ \mathbf{y}}_{<t}\right)}\] (36d) \[=\mathbbm{1}\left\{y=y_{t-1}\right\}p_{\mathcal{A}}\left(q~{}|~{} {\mathbf{y}}_{<t}\right), \tag{36e}\]

i.e., the \(\mathbf{h}^{\prime}\) holds a distribution over states given all the input symbols up to time step \(t-1\). We define \(\mathbf{E}\) for \(q\in Q,y\in\Sigma,\overline{y}\in\overline{\Sigma}\) as

\[\mathbf{E}_{\overline{y},\left(q,y\right)}\stackrel{{\text{def}}}{{= }}\begin{cases}\sum_{q\xrightarrow{y/w}_{q^{\prime}\in\tau}q^{\prime}\in\tau} w&\textbf{if}\ \overline{y}\in\Sigma\\ \rho\left(q\right)&\textbf{if}\ \overline{y}=\textsc{eos}\end{cases}= \begin{cases}p_{\mathcal{A}}\left(y~{}|~{}q\right)&\textbf{if}\ \overline{y}\in\Sigma\\ p_{\mathcal{A}}\left(\textsc{eos}~{}|~{}q\right)&\textbf{if}\ \overline{y}=\textsc{ eos}\end{cases}.\] (37) 

Computing the matrix-vector product \({\bf E}\,{\bf h}^{\prime}\left({\mathbf{y}_{<t}}\right)\) reveals that \(\left({\bf E}\,{\bf h}^{\prime}\left({\mathbf{y}_{<t}}\right)\right)_{\overline{ y}}=p_{\mathcal{A}}\left(\overline{y}~{}|~{}{\mathbf{y}_{<t}}\right)\):

\[\left({\bf E}\,{\bf h}^{\prime}\left({\mathbf{y}_{<t}}\right)\right)_{ \overline{y}} =\sum_{d=1}^{D}{\bf E}_{\overline{y},d}{\bf h}^{\prime}\left({\bm {y}_{<t}}\right)_{d} \tag{38a}\] \[=\sum_{q\in Q,y\in\Sigma}{\bf E}_{\overline{y},\left(q,y\right)}{ \bf h}^{\prime}\left({\mathbf{y}_{<t}}\right)_{\left(q,y\right)}\] (38b) \[=\sum_{q\in Q,y\in\Sigma}p_{\mathcal{A}}\left(\overline{y}~{}|~{} q\right)\mathbbm{1}\left\{y=y_{t-1}\right\}p_{\mathcal{A}}\left(q~{}|~{}{\mathbf{y}_{< t}}\right)\] (38c) \[=\sum_{q\in Q}p_{\mathcal{A}}\left(\overline{y}~{}|~{}q\right)p_ {\mathcal{A}}\left(q~{}|~{}{\mathbf{y}_{<t}}\right)\] (38d) \[=\sum_{q\in Q}p_{\mathcal{A}}\left(\overline{y}~{}|~{}{\mathbf{y}_{<t }}\right)p_{\mathcal{A}}\left(q~{}|~{}{\mathbf{y}_{<t}}\right)\] (38e) \[=p_{\mathcal{A}}\left(\overline{y}~{}|~{}{\mathbf{y}_{<t}}\right). \tag{38f}\]

The identity of the sparsemax on \(\boldsymbol{\Delta}^{|\overline{\Sigma}|-1}\) then gives us

\[p_{\mathcal{R}}(\overline{y}~{}|~{}{\mathbf{y}_{\leq t}}) \stackrel{{\text{\tiny def}}}{{=}}\operatorname*{sparsemax} \left({\bf E}\,{\bf h}({\mathbf{y}_{\leq t}})\right)_{\overline{y}} \tag{39a}\] \[=p_{\mathcal{A}}(\overline{y}~{}|~{}{\mathbf{y}_{\leq t}})\] (39b) for all \[{\mathbf{y}_{<t}}\in\Sigma^{*},\overline{y}\in\overline{\Sigma}\] . This implies by Eq. ( 1 ) that \[p_{\mathcal{R}}\] and \[p_{\mathcal{A}}\] are weakly equivalent. \[\blacksquare\]

**Corollary 3.2**.: _Let \(\mathcal{A}=\left(\Sigma,Q,\tau,\lambda,\rho\right)\) be a trim PFSA inducing the tight LM \(p_{\mathcal{A}}\). Then, there exists a weakly equivalent, softmax-normalized non-linear output Elman LM with an \(\overline{\mathbb{K}}\)-valued \({\bf F}\), where \(\overline{\mathbb{K}}\stackrel{{\text{\tiny def}}}{{=}}\mathbb{ R}\cup\left\{-\infty,\infty\right\}\) are the extended reals._

Proof.: Like in the proof of Cor. 3.1, we construct the function \({\bf F}\) that transforms the hidden state \({\bf h}_{t-1}\) into \(|\overline{\Sigma}|\) values that are later normalized into \(p_{\mathcal{A}}\left(y_{t}~{}|~{}{\bf h}_{t}\right)\), this time with the softmax. Concretely, we want to compute the autoregressive next-symbol probabilities \(p_{\mathcal{A}}\left(y_{t}~{}|~{}{\mathbf{y}_{<t}}\right)\). Assuming that all values are positive (we relax this later), we can write:

\[p_{\mathcal{A}}\left(\overline{y}_{t}~{}|~{}{\mathbf{y}_{<t}}\right) =\frac{p_{\mathcal{A}}\left({\mathbf{y}_{<t}}\overline{y}_{t}\right)} {p_{\mathcal{A}}\left({\mathbf{y}_{<t}}\right)} \tag{40a}\] \[=\frac{\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}_{<t}} \overline{y}_{t}\right)}{\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}_{<t}} \right)}\] (40b) \[=\frac{\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}_{<t}} \right)p_{\mathcal{A}}\left(\overline{y}_{t}~{}|~{}q,{\mathbf{y}_{<t}}\right)}{ \sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}_{<t}}\right)}\] (40c) \[=\frac{\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}_{<t}} \right)p_{\mathcal{A}}\left(\overline{y}_{t}~{}|~{}q\right)}{\sum_{q\in Q}p_{ \mathcal{A}}\left(q,{\mathbf{y}_{<t}}\right)}\] (40d) \[=\exp\left(\log\left(\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y} _{<t}}\right)p_{\mathcal{A}}\left(\overline{y}_{t}~{}|~{}q\right)\right)-\log \left(\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}_{<t}}\right)\right)\right). \tag{40e}\]

Intuitively, \(p_{\mathcal{A}}\left(y_{t}~{}|~{}{\mathbf{y}_{<t}}\right)\) can thus be computed by normalizing the values \(\log\left(\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}_{>t}}\right)p_{\mathcal{ A}}\left(\overline{y}_{t}~{}|~{}q\right)\right)-\log\left(\sum_{q\in Q}p_{ \mathcal{A}}\left(q,{\mathbf{y}_{<t}}\right)\right)\) for \(\overline{y}_{t}\in\overline{\Sigma}\) with the softmax. Defining the vector \({\mathbf{v}_{\overline{y}}}\stackrel{{\text{\tiny def}}}{{=}}\log \left(\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}_{<t}}\right)p_{\mathcal{A}} \left(\overline{y}_{t}~{}|~{}q\right)\right)-\log\left(\sum_{q\in Q}p_{ \mathcal{A}}\left(q,{\mathbf{y}_{<t}}\right)\right)\) for \(\overline{y}_{t}\in\overline{\Sigma}\), we therefore have

\[p_{\mathcal{A}}\left(y_{t}~{}|~{}{\mathbf{y}_{<t}}\right)={\mathbf{\sigma}}({\mathbf{ v}})_{\overline{y}}. \tag{41}\]

However, due to the invariance of the softmax to the addition of vectors of the form \(c\cdot{\bf 1}\), for any \(c\in\mathbb{R}\), we also have that

\[p_{\mathcal{A}}\left(y_{t}~{}|~{}{\mathbf{y}_{<t}}\right)={\mathbf{\sigma}}\!\left({ \mathbf{v}+\log\left(\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}_{<t}}\right) \right)\cdot{\bf 1}}\right)_{\overline{y}}.\] (42) 

Entries of \({\bf v}^{\prime}\stackrel{{\text{\tiny def}}}{{=}}{\bf v}+\log\left( \sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}}_{<t}\right)\right)\cdot{\bf 1}\) equal

\[{\bf v}^{\prime}_{\overline{y}}\stackrel{{\text{\tiny def}}}{{=}}{ \bf v}_{\overline{y}}+\log\left(\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}}_{ <t}\right)\right)=\log\left(\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}}_{<t} \right)p_{\mathcal{A}}\left(\overline{y}_{t}\mid q\right)\right). \tag{43}\]

The goal, then, is to compute the values \(\sum_{q\in Q}p_{\mathcal{A}}\left(q,{\mathbf{y}}_{<t}\right)p_{\mathcal{A}}\left( \overline{y}_{t}\mid q\right)\), logarithmically transform them, and then normalize them with the softmax function. To achieve this, we take the same output matrix \({\bf E}\in\mathbb{R}^{|\overline{\Sigma}|\times D}\) as in the proof of Cor. 3.1 (cf. Eq. (37)). The matrix-vector product \({\bf Eh}\) contains the values

\[\left({\bf Eh}\right)_{y} =\sum_{d=1}^{D}{\bf E}^{\prime}_{\overline{y},d}{\rm h}\left({ \mathbf{y}}_{<t}\right)_{d} \tag{44a}\] \[=\sum_{q\in Q,y\in\Sigma}{\bf E}^{\prime}_{\overline{y},\left(q, y\right)}{\rm h}\left({\mathbf{y}}_{<t}\right)_{\left(q,y\right)}\] (44b) \[=\sum_{q\in Q,y\in\Sigma}p_{\mathcal{A}}\left(\overline{y}\mid q \right){\mathbbm{1}}\left\{y=y_{t-1}\right\}p\left(q,{\mathbf{y}}_{<t}\right)\] (44c) \[=\sum_{q\in Q}p_{\mathcal{A}}\left(\overline{y}\mid q\right)p \left(q,{\mathbf{y}}_{<t}\right). \tag{44d}\]

These are the values we are interested in (from Eq. (40e)).

To allow for zero-valued arguments to the \(\log\), we now define the non-linear transformation \(\overline{\log}\) as

\[\overline{\log}\left({\bf x}\right)_{d}\stackrel{{\text{\tiny def }}}{{=}}\begin{cases}\log\left({\bf x}\right)_{d}&\textbf{if ${\bf x}_{ d}>0$}\\ -\infty&\textbf{otherwise}\end{cases}. \tag{45}\]

Note that the use of \(-\infty\) here is what necessitates us to require the extended reals, in contrast to in Cor. 3.1. This motivates the definition of \({\bf F}\colon\mathbb{R}^{\mathbf{h}}\to\overline{\mathbb{R}}^{|\Sigma|}\) as

\[{\bf F}\left({\bf h}\right)\stackrel{{\text{\tiny def}}}{{=}} \overline{\log}\left({\bf E}^{\prime}{\bf h}\right). \tag{46}\]

The application of the softmax to \({\bf F}\left({\bf h}\right)\) results in

\[p_{\mathcal{R}}\left(\overline{y}\mid{\mathbf{y}}_{<t}\right) =\mathbf{\sigma}({\bf F}\left({\bf h}\right))_{\overline{y}} \tag{47a}\] \[=\frac{\exp\left(\overline{\log}\left(p_{\mathcal{A}}\left({\mathbf{ y}}_{<t}\overline{y}\right)\right)\right)}{\sum_{\overline{y}^{\prime}\in\Sigma} \exp\left(\overline{\log}\left(p_{\mathcal{A}}\left({\mathbf{y}}_{<t}\overline{y} ^{\prime}\right)\right)\right)}\] (47b) \[=\frac{p_{\mathcal{A}}\left({\mathbf{y}}_{<t}\overline{y}\right)}{ \sum_{\overline{y}^{\prime}\in\Sigma}p_{\mathcal{A}}\left({\mathbf{y}}_{<t} \overline{y}^{\prime}\right)}\] (47c) \[=p_{\mathcal{A}}\left(\overline{y}\mid{\mathbf{y}}_{<t}\right), \tag{47d}\]

implying that \(p_{\mathcal{A}}\) and \(p_{\mathcal{R}}\) are weakly equivalent, as desired. 

## Appendix D Proof of Approximate Simulation by Elman LMs with Softmax over the Real Numbers

Recall that we assume, without loss of generality, that for each pair of states \(q,q^{\prime}\) and each symbol \(y\in\Sigma\) there exists a transition \(q\xrightarrow{y/w}q^{\prime}\), albeit potentially with zero weight, \(w=0\). For any PFSA, we define the following corresponding fully connected[13] automaton:

**Definition D.1**.: _Given a PFSA \(\mathcal{A}=\left(\Sigma,Q,\tau,\lambda,\rho\right)\) and \(\delta\in\mathbb{R}\), \(\mathcal{A}\)'s \(\delta\)**-perturbed PFSA \(\mathcal{A}_{\delta}=\left(\Sigma,Q,\tau_{\delta},\lambda,\rho_{\delta}\right)\)** is a fully connected PFSA derived from \(\mathcal{A}\) by_

* _adding_ \(\delta\) _to all transition weights and final weights and_
* _locally re-normalizing the outgoing transition weights and the final weight of each state such that they sum to_ \(1\) 

See Alg. 1 for the pseudocode of transforming a PFSA to a \(\delta\)-perturbed PFSA.

**Definition D.2**.: _Given two LMs \(p\) and \(q\) over \(\Sigma^{*}\) and a subset \(S\subset\Sigma^{*}\), the **restricted tvd** on \(S\) is defined as_

\[\textsc{tvd}_{S}(p,q)\stackrel{{\text{def}}}{{=}}\frac{1}{2}\sum_ {\boldsymbol{y}\in S}|p(\boldsymbol{y})-q(\boldsymbol{y})|. \tag{48}\]

**Lemma D.1**.: _Let \(\mathcal{A}=(\Sigma,Q,\tau,\lambda,\rho)\) be a PFSA inducing the LM \(p_{\mathcal{A}}\) and \(\mathcal{A}_{\delta}\)\(\mathcal{A}\)'s \(\delta\)-perturbed PFSA inducing the LM \(p_{\mathcal{A}_{\delta}}\) for some \(\delta\in\mathbb{R}\).15 For any finite subset of strings \(S\subset\Sigma^{*}\), we have that the restricted total variation distance \(\textsc{tvd}_{S}(p_{\mathcal{A}},p_{\mathcal{A}_{\delta}})\) is continuous at \(\delta=0\)._

Footnote 15: Fully connected here means the PFSA has nonzero transition probability for all symbols between all pairs of strings, and nonzero final weights for all states.

Proof.: Let \(P(\boldsymbol{y})\) be the set of all paths in \(\mathcal{A}\) that accept \(\boldsymbol{y}\). Because \(\mathcal{A}\) and \(\mathcal{A}_{\delta}\) have the same topology, we will also use \(P(\boldsymbol{y})\) for the set of all strings in \(\mathcal{A}_{\delta}\) that accept \(\boldsymbol{y}\). Note that \(\varphi(\boldsymbol{p})\) is a finite set since both \(\mathcal{A}\) and \(\mathcal{A}_{\delta}\) is a real-time PFSA. Now, to show that \(\textsc{tvd}(p_{\mathcal{A}},p_{\mathcal{A}_{\delta}})\) is continuous at \(\delta=0\), we need to show \(\lim_{\delta\to 0}\textsc{tvd}_{S}(p_{\mathcal{A}},p_{\mathcal{A}_{\delta}})\) exists. This follows directly from the following manipulations: equationparentequation

\[\lim_{\delta\to 0}\textsc{tvd}_{S}(p_{\mathcal{A}},p_{\mathcal{ A}_{\delta}}) =\lim_{\delta\to 0}\frac{1}{2}\sum_{\boldsymbol{y}\in S}|p_{ \mathcal{A}}(\boldsymbol{y})-p_{\mathcal{A}_{\delta}}(\boldsymbol{y})|\] (49a) \[=\lim_{\delta\to 0}\frac{1}{2}\sum_{\boldsymbol{y}\in S}\left| \sum_{\boldsymbol{p}\in P(\boldsymbol{y})}\prod_{n=0}^{|\boldsymbol{y}|+1}w_{ n}-\sum_{\boldsymbol{p}\in P(\boldsymbol{y})}\prod_{n=0}^{|\boldsymbol{y}|+1} w_{n}^{\delta}\right|\] (49b) \[=\lim_{\delta\to 0}\frac{1}{2}\sum_{\boldsymbol{y}\in S}\left| \sum_{\boldsymbol{p}\in P(\boldsymbol{y})}\prod_{n=0}^{|\boldsymbol{y}|+1}w_{ n}-\sum_{\boldsymbol{p}\in P(\boldsymbol{y})}\prod_{n=0}^{|\boldsymbol{y}|+1} \frac{w_{n}+\delta}{1+\delta(|\Sigma||Q|+1)}\right|\] (49c) \[=\frac{1}{2}\sum_{\boldsymbol{y}\in S}\left|\sum_{\boldsymbol{ p}\in P(\boldsymbol{y})}\prod_{n=0}^{|\boldsymbol{y}|+1}w_{n}-\sum_{\boldsymbol{p}\in P (\boldsymbol{y})}\prod_{n=0}^{|\boldsymbol{y}|+1}\frac{w_{n}+\lim_{\delta \to 0}\delta}{1+\lim_{\delta\to 0}\delta(|\Sigma||Q|+1)}\right|\] (49d) \[=\frac{1}{2}\sum_{\boldsymbol{y}\in S}\left|\sum_{\boldsymbol{p} \in P(\boldsymbol{y})}\prod_{n=0}^{|\boldsymbol{y}|+1}w_{n}-\sum_{\boldsymbol{ p}\in P(\boldsymbol{y})}\prod_{n=0}^{|\boldsymbol{y}|+1}\frac{w_{n}+0}{1+0}\right|\] (49e) \[=\frac{1}{2}\sum_{\boldsymbol{y}\in S}\left|\sum_{\boldsymbol{p} \in P(\boldsymbol{y})}\prod_{n=0}^{|\boldsymbol{y}|+1}w_{n}-\sum_{\boldsymbol{ p}\in P(\boldsymbol{y})}\prod_{n=0}^{|\boldsymbol{y}|+1}w_{n}\right|\] (49f) \[=0,\] (49g) 

where the step from Eq. (49d) to Eq. (49c) follows from the basic limit laws (continuous functions preserve limits), as desired. 

**Lemma D.2**.: _Let \({\cal A}\) be a trim PFSA inducing the LM \(p_{\mathcal{A}}\). For any \(\epsilon>0\), we can partition the set of all strings \(\Sigma^{*}\) into a finite set \(S\) and its complement \(\Sigma^{*}\backslash S\) such that \(p_{\mathcal{A}}\left(\Sigma^{*}\backslash S\right)=\sum\limits_{\mathbf{y}\in \Sigma^{*}\setminus S}p_{\mathcal{A}}(\mathbf{y})<\epsilon\)._

Proof.: By Du et al. (2023, Thm 5.3), a PFSA induces a tight language model if and only if every accessible state is also co-accessible. Thus, because we assume \({\cal A}\) is trim, the above-stated condition holds and \(p_{\mathcal{A}}\) is tight. Moreover, because \(p_{\mathcal{A}}\) is tight, we have

\[1=\sum\limits_{\mathbf{y}\in\Sigma^{*}}p_{\mathcal{A}}(\mathbf{y})=\sum\limits_{n=0}^ {\infty}\sum\limits_{\mathbf{y}\in\Sigma^{n}}p_{\mathcal{A}}(\mathbf{y}) \tag{50}\]

Since the infinite sum converges, we have that for any \(\epsilon>0\), there exists an \(M<\infty\) such that

\[\sum\limits_{n=M+1}^{\infty}\sum\limits_{\mathbf{y}\in\Sigma^{n}}p_{\mathcal{A}}( \mathbf{y})<\epsilon \tag{51}\]

and thus

\[\left|1-\sum\limits_{n=0}^{M}\sum\limits_{\mathbf{y}\in\Sigma^{n}}p_{\mathcal{A}}( \mathbf{y})\right|<\epsilon. \tag{52}\]

Defining \(S\stackrel{{\text{\tiny def}}}{{=}}\bigcup_{n=0}^{M}\Sigma^{n}\) completes the proof since \(\sum_{n=0}^{M}\sum_{\mathbf{y}\in\Sigma^{n}}p_{\mathcal{A}}(\mathbf{y})=\sum_{\mathbf{y} \in S}p_{\mathcal{A}}(\mathbf{y})\geqslant 1-\epsilon\) and \(\sum_{\mathbf{y}\in\Sigma^{*}\backslash S}p_{\mathcal{A}}(\mathbf{y})<\epsilon\). 

**Theorem D.1**.: _For any PFSA \({\cal A}\) inducing the LM \(p_{\mathcal{A}}\) and any \(\epsilon>0\), there exists a rational \(\eta>0\) such that \(\textsc{tvd}(p_{\mathcal{A}},p_{\mathcal{A}_{\eta}})<\epsilon\), where \(p_{\mathcal{A}_{\eta}}\) is the LM of \(\mathcal{A}_{\eta}\), the \(\eta\)-perturbed \({\cal A}\)._

Proof.: Let \(\epsilon>0\). Choose a _finite_ set \(S\subset\Sigma^{*}\) such that \(p(S)>1-\frac{\epsilon}{2}\) and \(p(\Sigma^{*}\backslash S)<\frac{\epsilon}{2}\). By Lem. D.2, we can always find such a set \(S\). Next, by Lem. D.1, \(\textsc{tvd}_{S}(p_{\mathcal{A}},p_{\mathcal{A}_{\delta}})\colon\mathbb{R} \to\mathbb{R}\) is continuous at \(\delta=0\). Thus, there exists a \(\delta^{\prime}\) such that \(|\eta-0|<\delta^{\prime}\) implies \(\textsc{tvd}_{S}(p_{\mathcal{A}},p_{\mathcal{A}_{\eta}})<\frac{\epsilon}{4}\). Choose a rational \(\eta\) positive such that \(\eta<\delta^{\prime}\). Next, we partition \(\textsc{tvd}(p_{\mathcal{A}},p_{\mathcal{A}_{\eta}})\) into three additive terms and bound each term individually

\[\textsc{tvd}(p_{\mathcal{A}},p_{\mathcal{A}_{\eta}}) =\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}}|p_{\mathcal{A}}(\mathbf{y})- p_{\mathcal{A}_{\eta}}(\mathbf{y})| \tag{53a}\] \[=\frac{1}{2}\sum_{\mathbf{y}\in S}|p_{\mathcal{A}}(\mathbf{y})-p_{ \mathcal{A}_{\eta}}(\mathbf{y})|+\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}\backslash S }|p_{\mathcal{A}}(\mathbf{y})-p_{\mathcal{A}_{\eta}}(\mathbf{y})|\] (53b) \[\leqslant\frac{1}{2}\sum_{\mathbf{y}\in S}|p_{\mathcal{A}}(\mathbf{y})-p_ {\mathcal{A}_{\eta}}(\mathbf{y})|+\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}\backslash S }|p_{\mathcal{A}}(\mathbf{y})|+\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}\backslash S}|p_ {\mathcal{A}_{\eta}}(\mathbf{y})|\] (53c) \[=\underbrace{\frac{1}{2}\sum_{\mathbf{y}\in S}|p_{\mathcal{A}}(\mathbf{y} )-p_{\mathcal{A}_{\eta}}(\mathbf{y})|}_{<\epsilon/4}+\underbrace{\frac{1}{2}\sum_{ \mathbf{y}\in\Sigma^{*}\backslash S}p_{\mathcal{A}}(\mathbf{y})}_{<\epsilon/2}+\underbrace {\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}\backslash S}p_{\mathcal{A}_{\eta}}(\mathbf{ y})}_{<\epsilon}\] (53d) \[<\frac{1}{4}\epsilon+\frac{1}{4}\epsilon+\frac{1}{2}\epsilon\] (53e) \[=\epsilon, \tag{53f}\]

where, in the step from Eq. (53d) to Eq. (53e), the first two terms are bounded by the choice of \(S\) while the bound on the third term follows from the claim below. This proves the result. 

**Claim D.1**.: _We claim that \(\epsilon>p_{\mathcal{A}_{\eta}}(\Sigma^{*}\backslash S)\)._ 

Proof.: The proof follows from simple manipulations:

\[\frac{\epsilon}{4}>\textsc{tvd}_{S}(p_{\mathcal{A}},p_{\mathcal{ A}_{9}}) =\frac{1}{2}\sum_{\mathbf{y}\in S}\left|p_{\mathcal{A}}(\mathbf{y})-p_{ \mathcal{A}_{9}}(\mathbf{y})\right|\geqslant\frac{1}{2}\left|\sum_{\mathbf{y}\in S} \left(p_{\mathcal{A}}(\mathbf{y})-p_{\mathcal{A}_{9}}(\mathbf{y})\right)\right| \tag{54a}\] \[=\frac{1}{2}\left|p_{\mathcal{A}}(S)-p_{\mathcal{A}_{9}}(S) \right|=\frac{1}{2}\left|1-p_{\mathcal{A}}(\Sigma^{*}\backslash S)-1-p_{ \mathcal{A}_{9}}(\Sigma^{*}\backslash S)\right|\] (54b) \[=\frac{1}{2}\left|p_{\mathcal{A}}(\Sigma^{*}\backslash S)-p_{ \mathcal{A}_{9}}(\Sigma^{*}\backslash S)\right|. \tag{54c}\]

This results in \(\tfrac{\epsilon}{2}>\left|p_{\mathcal{A}}(\Sigma^{*}\backslash S)-p_{ \mathcal{A}_{9}}(\Sigma^{*}\backslash S)\right|\). Additionally, we have \(\tfrac{\epsilon}{2}>\left|p_{\mathcal{A}}(\Sigma^{*}\backslash S)\right|\). Summing these together, we arrive at

\[\epsilon >\left|p_{\mathcal{A}}(\Sigma^{*}\backslash S)-p_{\mathcal{A}_{ 9}}(\Sigma^{*}\backslash S)\right|+\left|p_{\mathcal{A}}(\Sigma^{*}\backslash S )\right| \tag{55a}\] \[\geqslant\left|p_{\mathcal{A}}(\Sigma^{*}\backslash S)-p_{ \mathcal{A}_{9}}(\Sigma^{*}\backslash S)+p_{\mathcal{A}}(\Sigma^{*}\backslash S )\right|=\left|p_{\mathcal{A}_{9}}(\Sigma^{*}\backslash S)\right| \tag{55b}\]

Thus, we arrive at \(\epsilon>p_{\mathcal{A}_{9}}(\Sigma^{*}\backslash S)\). 

Next, we define the notion of Lipschitz continuity.

**Definition D.3**.: _Let \(\mathcal{K}\subseteq\mathbb{R}^{N}\) be a subset of \(\mathbb{R}^{N}\). A mapping \(f\colon\mathcal{K}\to\mathbb{R}^{M}\) is Lipschitz (or L-Lipschitz) with respect to a norm \(\lVert\cdot\rVert_{p}\) if there exists a constant \(L>0\) such that for all \(\mathbf{x},\mathbf{y}\in\mathcal{K}\),_

\[\lVert f(\mathbf{x})-f(\mathbf{y})\rVert_{2}\leqslant L\|\mathbf{x}-\mathbf{ y}\rVert_{2}. \tag{56}\]

**Proposition D.1** (Gao and Pavel (2018)).: _The softmax function \(\mathbf{\sigma}\) is \(\lambda\)-Lipschitz with respect to \(\lVert\cdot\rVert_{2}\) i.e., for all \(\mathbf{x},\mathbf{x}^{\prime}\in\mathbb{R}^{N}\),_

\[\lVert\mathbf{\sigma}(\mathbf{x})-\mathbf{\sigma}(\mathbf{x}^{\prime})\rVert_{2} \leqslant\lambda\|\mathbf{x}-\mathbf{x}^{\prime}\|_{2}, \tag{57}\]

_where \(\lambda\) is the inverse temperature parameter as defined in Eq. (12)._

Softmax-normalized Elman LMs (cf. Def. 2.5) define the conditional next-symbol probabilities \(p\left(y_{t}\mid\mathbf{y}_{<t}\right)\) by normalizing \(\mathbf{F}\left(\mathbf{h}\left(\mathbf{y}_{<t}\right)\right)\). To abstract away the particular hidden states and their transformations, we now introduce the following definition.

**Definition D.4**.: _Let \(\mathbf{y}\in\Sigma^{*}\) be a string of length \(T\). Let \(\mathbf{X}\in\mathbb{R}^{D\times(T+1)}\) be a matrix of \(T+1\) column vectors of size \(D\):_

\[\mathbf{X}\stackrel{{\text{\tiny def}}}{{=}}\begin{bmatrix}|&|&|\\ \mathbf{x}(1)&\mathbf{x}(2)&\cdots&\mathbf{x}(T+1)\\ |&|&&|\end{bmatrix}. \tag{58}\]

_Define the function \(f_{\mathbf{y}}\colon\mathbb{R}^{D\times(T+1)}\to[0,1]\) that induces an autoregressive LM \(p\):_

\[f_{\mathbf{y}}(\mathbf{X}) =p(\textsc{cos}\mid\mathbf{y})\prod_{t=1}^{T}p(\overline{y}_{t}\mid \mathbf{y}_{<t}) \tag{59a}\] \[=\mathbf{\sigma}(\mathbf{x}(T+1))_{\mathrm{EoS}}\prod_{t=1}^{T}\mathbf{ \sigma}(\mathbf{x}(t))_{y_{t}}\] (59b) \[=\prod_{t=1}^{T+1}\mathbf{\sigma}(\mathbf{x}(t))_{\overline{y}_{t}}\] (59c) \[\stackrel{{\text{\tiny def}}}{{=}}\prod_{t=1}^{T+1} \frac{\exp\mathbf{x}(t)_{\overline{y}_{t}}}{\sum_{\overline{y}\in\Sigma}\exp \mathbf{x}(t)_{\overline{y}}}, \tag{59d}\]

_where \(\overline{y}_{T+1}\stackrel{{\text{\tiny def}}}{{=}}\textsc{cos}\)._

**Lemma D.3**.: _The function \(f_{\mathbf{y}}\) from Def. D.4 is Lipschitz continuous with respect to \(||\cdot||_{\infty}\) with Lipschitz constant \(L_{\mathbf{y}}=\mathcal{O}\left(|\mathbf{y}|\right)\), which, as the notation suggests, depends on the string \(\mathbf{y}\)._

Proof.: The function \(f_{\mathbf{y}}(\mathbf{X})\) is a product of softmax functions. Therefore, we divide the proof into two Steps. In Step 1, we start by showing that a single softmax \(\mathbf{\sigma}\) is Lipschitz continuous with respect to \(||\cdot||_{\infty}\). In Step 2, we show that products of softmax functions, e.g., \(f_{\mathbf{y}}(\mathbf{X})\), are Lipschitz continuous by induction.

 

Step 1: Lipschitz continuity of \(\mathbf{\sigma}_{\mathbf{\cdot}}\)Prop. D.1 states that \(\mathbf{\sigma}\colon\mathbb{R}^{N}\to\mathbb{R}^{N}\) is Lipschitz with respect to the norm \(||\cdot||_{2}\), with Lipschitz constant \(\lambda\). This implies the following inequality

\[|\mathbf{\sigma}(\mathbf{x})_{n}-\mathbf{\sigma}(\mathbf{y})_{n}|\leqslant\lambda|| \mathbf{x}-\mathbf{y}||_{2}\quad\forall n\in\mathbb{N}. \tag{60}\]

We generalize this result to \(||\cdot||_{\infty}\) (and, indeed, all norms on \(\mathbb{R}^{N}\)) by noting the equivalence of norms on finite-dimensional vector spaces, i.e., there exists a constant \(C\) such that \(||\mathbf{x}||_{2}\leqslant C||\mathbf{x}||_{\infty}\) for all \(\mathbf{x}\in\mathbb{R}^{N}\). Thus, we get

\[|\mathbf{\sigma}(\mathbf{x})_{n}-\mathbf{\sigma}(\mathbf{y})_{n}|\leqslant\lambda|| \mathbf{x}-\mathbf{y}||_{2}\leqslant\underbrace{C\lambda}_{\stackrel{{ \text{def}}}{{=}}L_{\mathbf{\sigma}}}||\mathbf{x}-\mathbf{y}||_{\infty}. \tag{61}\]

Taking a max over the left-hand side of Eq. (61), we get

\[||\mathbf{\sigma}(\mathbf{x})-\mathbf{\sigma}(\mathbf{y})||_{\infty}\leqslant L_{\bm {\sigma}}||\mathbf{x}-\mathbf{y}||_{\infty}. \tag{62}\]

Step 2: Lipschitz continuity of \(f_{\mathbf{y}}(\mathbf{X})\).The function \(f_{\mathbf{y}}(\mathbf{X})\), as defined in Eq. (59c) is a product of \(T+1\) softmax functions. By definition, each component of the softmax \(\mathbf{\sigma}_{d}\) for \(d\in[D]\) has a bounded range \([0,1]\). We now prove by induction that the product of \(T\) softmax functions is Lipschitz continuous. Define \(f_{N}\colon\mathbb{R}^{D\times T+1}\to\mathbb{R}\) as \(f_{N}(\mathbf{X})\stackrel{{\text{def}}}{{=}}\prod_{t=1}^{N}\| \mathbf{\sigma}(\mathbf{x}(t))\|_{\infty}\), where \(N\leqslant T+1\) and \(\mathbf{x}(t)\) is the \(t^{\text{th}}\) column vector of \(\mathbf{X}\). Clearly, \(f_{T}(\mathbf{X})\) upper bounds \(f_{\mathbf{y}}(\mathbf{X})\).

* **Base case:** For \(N=1\), \(f_{1}(\mathbf{X})=\|\mathbf{\sigma}(\mathbf{x}(1))\|_{\infty}\). Therefore, \[|f_{1}(\mathbf{X})-f_{1}(\mathbf{Y})|=\|\mathbf{\sigma}(\mathbf{x}(1)-\mathbf{y}( 1))\|_{\infty}\leqslant L_{\mathbf{\sigma}}\|\mathbf{x}(1)-\mathbf{y}(1)\|,\] (63) where \(L_{\mathbf{\sigma}}\) is the Lipschitz constant as shown in Step 1. Note that \(\mathbf{x}(1)\) and \(\mathbf{y}(1)\) are submatrices of \(\mathbf{X}\) and \(\mathbf{Y}\), respectively, so \(\|\mathbf{x}(1)-\mathbf{y}(1)\|_{\infty}\leqslant\|\mathbf{X}-\mathbf{Y}\|_{\infty}\). Therefore, \[|f_{1}(\mathbf{X})-f_{1}(\mathbf{Y})|\leqslant L_{\mathbf{\sigma}}\|\mathbf{X}- \mathbf{Y}\|_{\infty}.\] (64)
* **Induction step:** Assume that for \(N\), \(f_{N}(\mathbf{X})=\prod_{t=1}^{N}\|\mathbf{\sigma}(\mathbf{x}(t))\|_{\infty}\) is Lipschitz with constant \(NL_{\mathbf{\sigma}}\). We show that \(f_{N}(\mathbf{X})=\prod_{t=1}^{N+1}\|\mathbf{\sigma}(\mathbf{x}(t))\|_{\infty}\) is Lipschitz with constant \((N+1)L_{\mathbf{\sigma}}\). Consider \(\mathbf{X},\mathbf{Y}\in\mathbb{R}^{D\times T+1}\). By the definition of \(f\), \[|f_{N+1}(\mathbf{X})-f_{N+1}(\mathbf{Y})|\] (65a) \[=|f_{N}(\mathbf{X})\|\mathbf{\sigma}(\mathbf{x}(N+1))\|_{\infty}-f_{ N}(\mathbf{Y})\|\mathbf{\sigma}(\mathbf{y}(N+1))\|_{\infty}|\] (65b) \[=|f_{N}(\mathbf{X})(\|\mathbf{\sigma}(\mathbf{x}(N+1))\|_{\infty}-\| \mathbf{\sigma}(\mathbf{y}(N+1))\|_{\infty})+(f_{N}(\mathbf{X})-f_{N}(\mathbf{Y})) \|\mathbf{\sigma}(\mathbf{x}(N+1))\|_{\infty}|\] (65c) \[\leqslant|f_{N}(\mathbf{X})|||\mathbf{\sigma}(\mathbf{x}(N+1))\|_{ \infty}-\|\mathbf{\sigma}(\mathbf{y}(N+1))\|_{\infty}|+|f_{N}(\mathbf{X})-f_{N}( \mathbf{Y})||\mathbf{\sigma}(\mathbf{x}(N+1))\|_{\infty}\] (65d) \[\leqslant|\|\mathbf{\sigma}(\mathbf{x}(N+1))\|_{\infty}-\|\mathbf{\sigma}( \mathbf{y}(N+1))\|_{\infty}|+|f_{N}(\mathbf{X})-f_{N}(\mathbf{Y})|\] (65e) \[\leqslant|\|\mathbf{\sigma}(\mathbf{x}(N+1))-\mathbf{\sigma}(\mathbf{y}(N+1 ))\|_{\infty}|+|f_{N}(\mathbf{X})-f_{N}(\mathbf{Y})|\] (65f) \[\leqslant L_{\mathbf{\sigma}}\|\mathbf{x}(N+1)-\mathbf{y}(N+1))\|_{ \infty}+|f_{N}(\mathbf{X})-f_{N}(\mathbf{Y})|\] (65g) \[\leqslant L_{\mathbf{\sigma}}\|\mathbf{x}(N+1)-\mathbf{y}(N+1))\|_{ \infty}+NL_{\mathbf{\sigma}}\|\mathbf{X}-\mathbf{Y}\|_{\infty}\] (65h) \[\leqslant L_{\mathbf{\sigma}}\|\mathbf{X}-\mathbf{Y}\|_{\infty}+NL_{\mathbf{ \sigma}}\|\mathbf{X}-\mathbf{Y}\|_{\infty}\] (65i) \[=(N+1)L_{\mathbf{\sigma}}\|\mathbf{X}-\mathbf{Y}\|_{\infty},\] (65j) where in Eq. (65e) we use the fact that \(\|\mathbf{\sigma}(\cdot)\|_{\infty}\leqslant 1\), and hence also \(f_{N}(\mathbf{X})\leq 1\) for all \(N\in\mathbb{N}\). In Eq. (65f), we use the inverse triangle inequality, in Eq. (65g) we use the result from Step 1, and in Eq. (65h), we use the induction hypothesis. Finally, in Eq. (65i), we use the fact that the infinity norm of a matrix bounds the infinity norm of a subset of that matrix.

Thus, for \(\mathbf{y}\in\Sigma^{*}\), we have \(f_{\mathbf{y}}\) is Lipschitz continuous with Lipschitz constant \(L_{\mathbf{y}}=|T+1|L_{\mathbf{\sigma}}=|T+1|C\lambda=\mathcal{O}\left(|\mathbf{y}|\right)\) 



**Theorem 4.1**.: _For any PFSA \({\cal A}\) with induced LM \(p_{\cal A}\) and any \(\epsilon>0\), there exists a \(\mathrm{ReLU}\)-activated nonlinear output Elman LM \(p_{\cal R}\) with an \(\mathbb{R}\)-vector-valued function \(\mathbf{F}\colon\mathbb{R}^{D}\to\mathbb{R}^{D}\) such that \(\textsc{tvd}(p_{\cal A},p_{\cal R})<\epsilon\)._

Proof.: Let \({\cal A}\) be a PFSA with induced LM \(p_{\cal A}\), and be the LM induced by the \({\cal A}\) and \(\epsilon>0\). By Thm. D.1, we can construct an \(\eta\)-perturbed PFSA \({\cal A}_{\eta}\) inducing the full-support LM \(p_{{\cal A}_{\eta}}\) such that \(\textsc{tvd}(p_{\cal A},p_{A_{\eta}})<\epsilon\). Using the same construction as that in the proof of Cor. 3.2, except that we replace \(\overline{\log}\) in Eq. (45) with \(\log\), we can construct a non-linear output Elman LM \(p_{\cal R}\) (the Elman RNN identical to the one in Thm. 3.1) that is _weakly equivalent_ to the full-support \(p_{{\cal A}_{9}}\). This is because, since \(p_{{\cal A}_{9}}\) has full support, all outputs of \(\mathbf{F}\) are positive, and for any \(x>0,\log(x)=\overline{\log}(x)\). We then have that

\[\textsc{tvd}(p_{\cal A},p_{\cal R})\leq\underbrace{\textsc{tvd}( p_{\cal A},p_{A_{\eta}})}_{<\varepsilon}+\underbrace{\textsc{tvd}(p_{{\cal A}_{ \eta}},p_{\cal R})}_{=0}<\varepsilon, \tag{66}\]

which finishes the proof. 

## Appendix E Proof of Approximate Simulation by Softmax-normalized, Deep Output Elman LMs

See 4.2

Proof.: Fix \(\epsilon>0\) and let \(p_{{\cal A}_{\delta}}\) be the language model induced from a \(\delta\)-perturbation of a PFSA \({\cal A}\) weakly equivalent to \(p_{\cal A}\); we will specify \(\delta\) at a later point in the proof. Our goal is to show that there exists a \(\mathrm{ReLU}\)-activated, softmax-normalized deep output Elman LM \(p_{\cal R}\) such that \(\textsc{tvd}(p_{\cal A},p_{\cal R})=\frac{1}{2}\sum_{y\in\Sigma*}|p_{\cal A}( y)-p_{\cal R}(y)|\). The RNN \({\cal R}\) will be the same as the one in Thm. 3.1; we only seek to approximate the _output_ function of the Elman LM. To do so, we first partition \(\textsc{tvd}(p_{\cal A},p_{\cal R})\) into four additive terms and bound each term individually:

\[\textsc{tvd}(p_{\cal A},p_{\cal R})=\frac{1}{2}\sum_{\mathbf{y}\in \Sigma^{*}}|p_{\cal A}(\mathbf{y})-p_{\cal R}(\mathbf{y})| \tag{67a}\] \[=\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}}|p_{\cal A}(\mathbf{y})-p_{{ \cal A}_{\delta}}(\mathbf{y})+p_{{\cal A}_{\delta}}(\mathbf{y})-p_{\cal R}(\mathbf{y})|\] (67b) \[\leq\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}}|p_{\cal A}(\mathbf{y})-p_{{ \cal A}_{\delta}}(\mathbf{y})|+\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}}|p_{{\cal A}_{ \delta}}(\mathbf{y})-p_{\cal R}(\mathbf{y})|\] (67c) \[=\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}}|p_{\cal A}(\mathbf{y})-p_{{ \cal A}_{\delta}}(\mathbf{y})|+\frac{1}{2}\sum_{\mathbf{y}\in S}|p_{{\cal A}_{\delta}} (\mathbf{y})-p_{\cal R}(\mathbf{y})|+\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}\setminus S}|p _{{\cal A}_{\delta}}(\mathbf{y})-p_{\cal R}(\mathbf{y})|\] (67d) \[\leq\underbrace{\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}}|p_{\cal A}( \mathbf{y})-p_{{\cal A}_{\delta}}(\mathbf{y})|}_{<\frac{\epsilon}{4}\text{ by Step 1}}+\underbrace{\frac{1}{2}\sum_{\mathbf{y}\in S}|p_{{\cal A}_{ \delta}}(\mathbf{y})-p_{\cal R}(\mathbf{y})|}_{<\frac{\epsilon}{4}\text{ by Step 2}}+\underbrace{\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*} \setminus S}|p_{{\cal K}_{\delta}}(\mathbf{y})|}_{<\frac{\epsilon}{4}\text{ by Step 4}}\] \[<\epsilon. \tag{67e}\]

The bounds are shown in the corresponding steps below.

_Caution_: Steps 1, 2, 3, and 4 all depend on \(\delta\), and Steps 2, 3, and 4 all depend on \(S\). To decouple the individual steps (for expository purposes), in each step, we will choose a separate \(\delta_{i}\) and then assign \(\delta=\min(\delta_{1},\delta_{2},\delta_{3},\delta_{4})\) and a separate set \(S_{i}\) and then assign \(S=S_{2}\cup S_{3}\cup S_{4}\).

Step 1.By Thm. D.1, we can choose a \(\delta_{1}\) such that \(\frac{1}{2}\sum_{\mathbf{y}\in\Sigma^{*}}|p_{\cal A}(\mathbf{y})-p_{{\cal A}_{\delta_{1} }}(\mathbf{y})|<\frac{\epsilon}{4}\).



Step 2.Let \(S_{2}\) be _any_ finite subset of \(\Sigma^{*}\) and choose \(\delta_{2}\in(0,1)\) arbitrarily. We first note that the \(\delta_{2}\)-perturbed language model \(p_{\mathcal{A}_{\delta_{2}}}\) can be written as follows

\[p_{\mathcal{A}_{\delta_{2}}}(\boldsymbol{y}) =p_{\mathcal{A}_{\delta_{2}}}(\textsc{eos}~{}|~{}\boldsymbol{y}) \prod_{t=1}^{T}p_{\mathcal{A}_{\delta_{2}}}(y_{t}~{}|~{}\boldsymbol{y}_{<t})\] (68a) \[=\boldsymbol{\sigma}(\log(\mathbf{Eh}(\boldsymbol{y})))_{\mathrm{ EOS}}\prod_{t=1}^{T}\boldsymbol{\sigma}(\log(\mathbf{Eh}(\boldsymbol{y}_{< t})))_{y_{t}}.\] (68b) Moreover, since \[p_{\mathcal{A}_{\delta_{2}}}\] is induced from an \[\delta_{2}\] -perturbed PFSA, \[\mathbf{h}(\boldsymbol{y})_{q}>0\] for all \[q\in Q\] and all \[\boldsymbol{y}\in\Sigma^{*}\] . Next, define the following constants \[\xi_{1}\stackrel{{\text{ def}}}{{=}} \min_{\boldsymbol{y}\in S_{2}}\min_{q\in Q}\mathbf{h}( \boldsymbol{y})_{q}>0\] (69a) \[\xi_{2}\stackrel{{\text{ def}}}{{=}} \min_{\boldsymbol{y}\in S_{2}}\min_{\overline{y}\in\overline{z}} \log(\mathbf{Eh}(\boldsymbol{y}))_{\overline{y}}<0\] (69b) Finally, we will define \[L=\max_{\boldsymbol{y}\in S_{2}}L_{\boldsymbol{y}}\] (70) where \[L_{\boldsymbol{y}}\] is the Lipschitz constant of the function \[f_{\boldsymbol{y}}\] (cf. Def. D.4) given in Lem. D.3. Our goal is to find a \[\operatorname{ReLU}\] MLP **F** that is an arbitrarily good approximator of \[\mathbf{h}\mapsto\log(\mathbf{Eh})\colon[\xi_{1},1]^{|Q|}\to[\xi_{2},0]^{| \Sigma|}\] where the \[\log\] is applied element-wise. Because \[[\xi_{1},1]^{|Q|}\ni\mathbf{h}\mapsto\log(\mathbf{Eh})\] is continuous and its domain is compact, Thm. 2.1 tells us there does exist a one-layer MLP \[\textbf{F}_{1}\] such that the following bound holds \[\max_{\mathbf{h}[\xi_{1},1]^{|Q|}}||\log(\mathbf{Eh})-\textbf{ F}_{1}(\mathbf{h})||_{\infty}<\frac{\epsilon}{2L|S|}\] (71) In order to make sure that the one-layer MLP **F** only receives inputs in its domain \[[\xi_{1},1]^{|Q|}\] , we compose \[\textbf{F}_{1}\] with another \[\operatorname{ReLU}\] layer \[\textbf{F}_{2}\] defined element-wise as follows \[\textbf{F}_{2}(\mathbf{h})_{i}\stackrel{{\text{ def}}}{{=}} \operatorname{ReLU}(0,\mathbf{h}_{i}-\xi_{1})+\xi_{1}\] (72) which computes the function \[\max(\xi_{1},x)\] . Because of the definition of \[\xi_{i}\] in Eq. ( 69a ), the composition of \[\textbf{F}_{1}\] with \[\textbf{F}_{2}\] has no effect on arguments taken from the set \[\{\mathbf{h}(\boldsymbol{y})~{}|~{}\boldsymbol{y}\in S_{2}\}\] . Now, define \[\textbf{F}(\mathbf{h})\stackrel{{\text{ def}}}{{=}} \textbf{F}_{1}(\textbf{F}_{2}(\mathbf{h}))\] . Then, making use of the finiteness of \[S_{2}\] , for any \[\boldsymbol{y}\in S_{2}\] , we have that \[f_{\boldsymbol{y}}\] is \[L_{\boldsymbol{y}}\] -Lipschitz continuous and a fortiori \[L\] -Lipschitz continuous by Lem. D.1 . Combining the result with the Eq. ( 71 ), we have that \[|f_{\boldsymbol{y}}(\log\mathbf{Eh}(\boldsymbol{y}))-f_{\boldsymbol{ y}}(F(\mathbf{h}(\boldsymbol{y})))|<\frac{\epsilon}{2L|S_{2}|}L=\frac{\epsilon}{2|S_{2}|}\] (73) which, with no more than an adjustment of notation (see Def. D.4), tells us that \[|p_{\mathcal{A}_{\delta_{2}}}(\boldsymbol{y})-p_{\mathcal{R}}(\boldsymbol{y})| <\frac{\epsilon}{2L|S_{2}|}L=\frac{\epsilon}{2|S_{2}|}\] (74) Because we chose a finite set \[S_{2}\] of strings, we arrive at the following bound \[\frac{1}{2}\sum_{\boldsymbol{y}\in S_{2}}|p_{\mathcal{A}_{\delta_{2}}}( \boldsymbol{y})-p_{\mathcal{R}}(\boldsymbol{y})| <\frac{1}{2}\sum_{\boldsymbol{y}\in S_{2}}\frac{\epsilon}{|S_{2}|}=\frac{1}{2 }|S_{2}|\frac{\epsilon}{2|S_{2}|}=\frac{\epsilon}{4}\] (75) as desired.

Step 3.Again by Thm. D.1, we can choose \(\delta_{3}\) such that \(\frac{1}{2}\sum_{\boldsymbol{y}\in\Sigma^{*}}|p_{\mathcal{A}}(\boldsymbol{y})- p_{\mathcal{A}_{\delta_{3}}}(\boldsymbol{y})|<\frac{\epsilon}{8}\). The proof of Thm. D.1 yields a set \(S_{3}\) such that \(p_{\mathcal{A}}(S_{3})=1-\frac{\epsilon}{16}\). Then, we can apply Claim D.1, which says \(\frac{\epsilon}{2}>p_{\mathcal{A}_{\delta_{3}}}(\Sigma^{*}\backslash S_{3})\).

 



**Step 4.** By Step 2, for any finite set \(S_{4}\subset\Sigma^{*}\), we can choose a \(\delta_{4}>0\) such that \(\frac{\epsilon}{8}>\frac{1}{2}\sum_{\mathbf{y}\in S_{4}}\left|p_{\mathcal{A}_{\delta _{4}}}(\mathbf{y})-p_{\mathcal{R}}(\mathbf{y})\right|\). Then, consider the following chain of inequalities

\[\frac{\epsilon}{8} >\frac{1}{2}\sum_{\mathbf{y}\in S_{4}}\left|p_{\mathcal{A}_{\delta_{4} }}(\mathbf{y})-p_{\mathcal{R}}(\mathbf{y})\right| \tag{76a}\] \[\geq\frac{1}{2}\left|\sum_{\mathbf{y}\in S_{4}}\left(p_{\mathcal{A}_{ \delta_{4}}}(\mathbf{y})-p_{\mathcal{R}}(\mathbf{y})\right)\right|\] (76b) \[=\frac{1}{2}\left|p_{\mathcal{A}_{\delta_{4}}}(S_{4})-p_{ \mathcal{R}}(S_{4})\right)\right|\] (76c) \[=\frac{1}{2}\left|1-p_{\mathcal{A}_{\delta_{4}}}(\Sigma^{*} \backslash S_{4})-(1-p_{\mathcal{R}}(\Sigma^{*}\backslash S_{4}))\right|\] (76d) \[=\frac{1}{2}\left|p_{\mathcal{A}_{\delta_{4}}}(\Sigma^{*} \backslash S_{4})-p_{\mathcal{R}}(\Sigma^{*}\backslash S_{4})\right|. \tag{76e}\]

This results in \(\frac{\epsilon}{4}>\left|p_{\mathcal{A}_{\delta_{4}}}(\Sigma^{*}\backslash S_{ 4})-p_{\mathcal{R}}(\Sigma^{*}\backslash S_{4})\right|\).

So far, we only required \(S_{4}\) to be finite but did not fix its value. Now, by Claim D.1, we can choose \(S_{4}\) such that we have \(\frac{\epsilon}{4}>\left|p_{\mathcal{A}_{\delta_{4}}}(\Sigma^{*}\backslash S_{ 4})\right|\). Summing both of these together, we arrive at

\[\frac{\epsilon}{2} >\left|p_{\mathcal{A}_{\delta_{4}}}(\Sigma^{*}\backslash S_{4}) -p_{\mathcal{R}}(\Sigma^{*}\backslash S_{4})\right|+\left|p_{\mathcal{A}_{ \delta_{4}}}(\Sigma^{*}\backslash S_{4})\right| \tag{77a}\] \[\geq\left|p_{\mathcal{A}_{\delta_{4}}}(\Sigma^{*}\backslash S_{4}) -p_{\mathcal{R}}(\Sigma^{*}\backslash S_{4})+p_{\mathcal{A}_{\delta_{4}}}( \Sigma^{*}\backslash S_{4})\right|=\left|p_{\mathcal{R}}(\Sigma^{*}\backslash S _{4})\right|. \tag{77b}\]

Thus, we have \(\frac{\epsilon}{2}>p_{\mathcal{R}}(\Sigma^{*}\backslash S_{4})\) because probabilities are non-negative.