Venue: NAACL 2018

# Recurrent Neural Networks as Weighted Language Recognizers

Yining Chen

Dartmouth College

yining.chen.18@dartmouth.edu &Sorcha Gilroy

ILCC

University of Edinburgh

s.gilroy@sss.ed.ac.uk &Andreas Maletti

Institute of Computer Science

Universitat Leipzig

andreas.maletti@uni-leipzig.de &Jonathan May

Information Sciences Institute

University of Southern California

jonmay@isi.edu &Kevin Knight

Information Sciences Institute

University of Southern California

knight@isi.edu

###### Abstract

We investigate the computational complexity of various problems for simple recurrent neural networks (RNNs) as formal models for recognizing weighted languages. We focus on the single-layer, ReLU-activation, rational-weight RNNs with softmax, which are commonly used in natural language processing applications. We show that most problems for such RNNs are undecidable, including consistency, equivalence, minimization, and the determination of the highest-weighted string. However, for consistent RNNs the last problem becomes decidable, although the solution length can surpass all computable bounds. If additionally the string is limited to polynomial length, the problem becomes NP-complete. In summary, this shows that approximations and heuristic algorithms are necessary in practical applications of those RNNs.

## 1 Introduction

Recurrent neural networks (RNNs) are an attractive apparatus for probabilistic language modeling Mikolov and Zweig (2012). Recent experiments show that RNNs significantly outperform other methods in assigning high probability to held-out English text Jozefowicz et al. (2016).

Roughly speaking, an RNN works as follows. At each time step, it consumes one input token, updates its hidden state vector, and predicts the next token by generating a probability distribution over all permissible tokens. The probability of an input string is simply obtained as the product of the predictions of the tokens constituting the string followed by a terminating token. In this manner, each RNN defines a _weighted language_; i.e. a total function from strings to weights. Siegelmann and Sontag (1995) showed that single-layer rational-weight RNNs with saturated linear activation can compute any computable function. To this end, a specific architecture with 886 hidden units can simulate any Turing machine in real-time (i.e., each Turing machine step is simulated in a single time step). However, their RNN encodes the whole input in its internal state, performs the actual computation of the Turing machine when reading the terminating token, and then encodes the output (provided an output is produced) in a particular hidden unit. In this way, their RNN allows "thinking" time (equivalent to the computation time of the Turing machine) after the input has been encoded.

We consider a different variant of RNNs that is commonly used in natural language processing applications. It uses ReLU activations, consumes an input token at each time step, and produces softmax predictions for the next token. It thus immediately halts after reading the last input token and the weight assigned to the input is simply the product of the input token predictions in each step.

Other formal models that are currently used to implement probabilistic language models such as finite-state automata and context-free grammars are by now well-understood. A fair share of their utility directly derives from their nice algorithmic properties. For example, the weighted languages computed by weighted finite-state automata are closed under intersection (pointwise product) and union (pointwise sum), and the corresponding unweighted languages are closed under intersection, union, difference, and complementation Droste et al. (2013). Moreover, toolkits like OpenFST Allauzen et al. (2007) and Carmel1 implement efficient algorithms on automata like minimization, intersection, finding the highest-weighted path and the highest-weighted string.

Footnote 1: [https://www.isi.edu/licensed-sw/carmel/](https://www.isi.edu/licensed-sw/carmel/)

RNN practitioners naturally face many of these same problems. For example, an RNN

based machine translation system should extract the highest-weighted output string (i.e., the most likely translation) generated by an RNN, Sutskever et al. (2014); Bahdanau et al. (2014). Currently this task is solved by approximation techniques like heuristic greedy and beam searches. To facilitate the deployment of large RNNs onto limited memory devices (like mobile phones) minimization techniques would be beneficial. Again currently only heuristic approaches like knowledge distillation Kim and Rush (2016) are available. Meanwhile, it is unclear whether we can determine if the computed weighted language is consistent; i.e., if it is a probability distribution on the set of all strings. Without a determination of the overall probability mass assigned to all finite strings, a fair comparison of language models with regard to perplexity is simply impossible.

The goal of this paper is to study the above problems for the mentioned ReLU-variant of RNNs. More specifically, we ask and answer the following questions:

* Consistency: Do RNNs compute consistent weighted languages? Is the consistency of the computed weighted language decidable?
* Highest-weighted string: Can we (efficently) determine the highest-weighted string in a computed weighted language?
* Equivalence: Can we decide whether two given RNNs compute the same weighted language?
* Minimization: Can we minimize the number of neurons for a given RNN?

## 2 Definitions and notations

Before we introduce our RNN model formally, we recall some basic notions and notation. An _alpha-bet_\(\Sigma\) is a finite set of symbols, and we write \(|\Sigma|\) for the number of symbols in \(\Sigma\). A _string_\(s\) over the alphabet \(\Sigma\) is a finite sequence of zero or more symbols drawn from \(\Sigma\), and we write \(\Sigma^{*}\) for the set of all strings over \(\Sigma\), of which \(\epsilon\) is the empty string. The length of the string \(s\in\Sigma^{*}\) is denoted \(|s|\) and coincides with the number of symbols constituting the string. As usual, we write \(A^{B}\) for the set of functions \(\{f~{}|~{}f\colon B\to A\}\). A _weighted language_\(L\) is a total function \(L\colon\Sigma^{*}\rightarrow\mathbb{R}\) from strings to real-valued weights. For example, \(L(a^{n})=e^{-n}\) for all \(n\geq 0\) is such a weighted language.

We restrict the weights in our RNNs to the rational numbers \(\mathbb{Q}\). In addition, we reserve the use of a special symbol $ to mark the start and end of an input string. To this end, we assume that $\notin\Sigma\) for all considered alphabets, and we let \(\Sigma_{\$}=\Sigma\cup\{\$\}\).

**Definition 1**.: _A single-layer RNN \(R\) is a \(7\)-tuple \(\langle\Sigma,N,h_{-1},W,W^{\prime},E,E^{\prime}\rangle\), in which_

* \(\Sigma\) _is an_ input alphabet__,_
* \(N\) _is a finite set of_ neurons_,_
* \(h_{-1}\in\mathbb{Q}^{N}\) _is an_ initial activation vector_,_
* \(W\in\mathbb{Q}^{N\times N}\) _is a_ transition matrix_,_
* \(W^{\prime}=(W^{\prime}_{a})_{a\in\Sigma_{\$}}\) _is a_ \(\Sigma_{\$}\)_-indexed family of_ bias vectors_ \(W^{\prime}_{a}\in\mathbb{Q}^{N}\)_,_
* \(E\in\mathbb{Q}^{\Sigma_{\$}\times N}\) _is a_ prediction matrix_, and_
* \(E^{\prime}\in\mathbb{Q}^{\Sigma_{\$}}\) _is a_ prediction bias vector_._

Next, let us define how such an RNN works. We first prepare our input encoding and the effect of our activation function. For an input string \(s=s_{1}s_{2}\cdots s_{n}\in\Sigma^{*}\) with \(s_{1},\ldots,s_{n}\in\Sigma\), we encode this input as $\(s\)$ and thus assume that \(s_{0}=\$\) and \(s_{n+1}=\$\). Our RNNs use ReLUs (Rectified Linear Units), so for every \(v\in\mathbb{Q}^{N}\) we let \(\sigma\langle v\rangle\) (the ReLU activation) be the vector \(\sigma\langle v\rangle\in\mathbb{Q}^{N}\) such that

\[\sigma\langle v\rangle(n)=\max\bigl{(}0,v(n)\bigr{)}\quad\text{ for every } n\in N.\]

In other words, the ReLUs act like identities on nonnegative inputs, but clip negative inputs to \(0\). We use softmax-predictions, so for every vector \(p\in\mathbb{Q}^{\Sigma_{\$}}\) and \(a\in\Sigma_{\$}\) we let

\[\operatorname{softmax}\langle p\rangle(a)=\frac{e^{p(a)}}{\sum_{a^{\prime}\in \Sigma_{\$}}e^{p(a^{\prime})}}\enspace.\]

RNNs act in discrete time steps reading a single letter at each step. We now define the semantics of our RNNs.

**Definition 2**.: _Let \(R=\langle\Sigma,N,h_{-1},W,W^{\prime},E,E^{\prime}\rangle\) be an RNN, \(s\) an input string of length \(n\) and \(0\leq t\leq n\) a time step. We define_

* _the_ hidden state vector__\(h_{s,t}\in\mathbb{Q}^{N}\) _given by_ \[h_{s,t}=\sigma\langle W\cdot h_{s,t-1}+W^{\prime}_{st}\rangle\enspace,\] _where_ \(h_{s,-1}=h_{-1}\) _and we use standard matrix product and point-wise vector addition,_
* _the_ next-token prediction vector__\(E_{s,t}\in\mathbb{Q}^{\Sigma_{\$}}\)___\[E_{s,t}=E\cdot h_{s,t}+E^{\prime}\]
* _the_ next-token distribution__\(E^{\prime}_{s,t}\in\mathbb{R}^{\Sigma_{\$}}\)___\[E^{\prime}_{s,t}=\operatorname{softmax}\langle E_{s,t}\rangle\enspace.\] 

_Finally, the RNN \(R\) computes the weighted language \(R\colon{\Sigma^{*}}\to\mathbb{R}\), which is given for every input \(s=s_{1}\cdots s_{n}\) as above by_

\[R(s)=\prod_{t=0}^{n}E^{\prime}_{s,t}(s_{t+1})~{}~{}.\]

In other words, each component \(h_{s,t}(n)\) of the hidden state vector is the ReLU activation applied to a linear combination of all the components of the previous hidden state vector \(h_{s,t-1}\) together with a summand \(W^{\prime}_{s_{t}}\) that depends on the \(t\)-th input letter \(s_{t}\). Thus, we often specify \(h_{s,t}(n)\) as linear combination instead of specifying the matrix \(W\) and the vectors \(W^{\prime}_{a}\). The semantics is then obtained by predicting the letters \(s_{1},\ldots,s_{n}\) of the input \(s\) and the final terminator SS and multiplying the probabilities of the individual predictions.

Let us illustrate these notions on an example. We consider the RNN \(\langle\Sigma,N,h_{-1},W,W^{\prime},E,E^{\prime}\rangle\) with \(\gamma\in\mathbb{Q}\) and

* \(\Sigma=\{a\}\) and \(N=\{1,2\}\),
* \(h_{-1}=(-1,0)^{T}\) and \[W=\begin{pmatrix}1&0\\ 1&0\end{pmatrix}\quad\text{and}\quad W^{\prime}_{8}=W^{\prime}_{a}=\begin{pmatrix} 1\\ 0\end{pmatrix}\]
* \(E(\$,\cdot\,)=(M+1,\,-(M+1))\) and \(E(a,\,\cdot\,)=(1,\,-1)\) and
* \(E^{\prime}(\$)=-M\) and \(E^{\prime}(a)=0\).

In this case, we obtain the linear combinations

\[h_{s,t}=\sigma\Bigg{<}\!\!\!\begin{array}{c}h_{s,t-1}(1)+1\\ h_{s,t-1}(1)\end{array}\!\!\!\Bigg{>}\]

computing the next hidden state components. Given the initial activation, we thus obtain \(h_{s,t}=\sigma\langle t,t-1\rangle\). Using this information, we obtain

\[E_{s,t}(\$) =(M+1)\cdot(t-\sigma\langle t-1\rangle)-M\] \[E_{s,t}(a) =t-\sigma\langle t-1\rangle~{}~{}.\]

Consequently, we assign weight \(\frac{e^{-M}}{1+e^{-M}}\) to input \(\varepsilon\), weight \(\frac{1}{1+e^{-M}}\cdot\frac{e^{1}}{e^{1}+c^{1}}\) to \(a\), and, more generally, weight \(\frac{1}{1+e^{-M}}\cdot\frac{1}{2^{m}}\) to \(a^{n}\).

Clearly the weight assigned by an RNN is always in the interval \((0,1)\), which enables a probabilistic view. Similar to weighted finite-state automata or weighted context-free grammars, each RNN is a compact, finite representation of a weighted language. The softmax-operation enforces that the probability \(0\) is impossible as assigned weight, so each input string is principally possible. In practical language modeling, smoothing methods are used to change distributions such that impossibility (probability \(0\)) is removed. Our RNNs avoid impossibility outright, so this can be considered a feature instead of a disadvantage.

The hidden state \(h_{s,t}\) of an RNN can be used as scratch space for computation. For example, with a single neuron \(n\) we can count symbols in \(s\) via:

\[h_{s,t}(n)=\sigma\langle h_{s,t-1}(n)+1\rangle~{}~{}.\]

Here the letter-dependent summand \(W^{\prime}_{a}\) is universally \(1\). Similarly, for an alphabet \(\Sigma=\{a_{1},\ldots,a_{m}\}\) we can use the method of SiegelmannSontag (1995) to encode the complete input string \(s\) in base \(m+1\) using:

\[h_{s,t}(n)=\sigma\langle(m+1)h_{s,t-1}(n)+c(s_{t})\rangle~{}~{},\]

where \(c\colon{\Sigma_{\$}}\to\{0,\dots,m\}\) is a bijection. In principle, we can thus store the entire input string (of unbounded length) in the hidden state value \(h_{s,t}(n)\), but our RNN model outputs weights at each step and terminates immediately once the final delimiter SS is read. It must assign a probability to a string _incrementally_ using the chain rule decomposition \(p(s_{1}\cdots s_{n})=p(s_{1})\cdot\ldots\cdot p(s_{n}\mid s_{1}\cdots s_{n-1})\).

Let us illustrate our notion of RNNs on some additional examples. They all use the alphabet \(\Sigma=\{a\}\) and are illustrated and formally specified in Figure 1. The first column shows an RNN \(R_{1}\) that assigns \(R_{1}(a^{n})=2^{-(n+1)}\). The next-token prediction matrix ensures equal values for \(a\) and SS at every time step. The second column shows the RNN \(R_{2}\), which we already discussed. In the beginning, it heavily biases the next symbol prediction towards \(a\), but counters it starting at \(t=1\). The third RNN \(R_{3}\) uses another counting mechanism with \(h_{s,t}=\sigma\langle t-100,t-101,t\rangle\). The first two components are ReLU-thresholded to zero until \(t>101\), at which point they overwhelm the bias towards \(a\) turning all future predictions to SS.

## 3 Consistency

We first investigate the consistency problem for an RNN \(R\), which asks whether the recognized weighted language \(R\) is indeed a probability distribution. Consequently, an RNN \(R\) is _consistent 

if \(\sum_{s\in\Sigma^{*}}R(s)=1\). We first show that there is an inconsistent RNN, which together with our examples shows that consistency is a nontrivial property of RNNs.2

Footnote 2: For comparison, all probabilistic finite-state automata are consistent, provided no transitions exit final states. Not all probabilistic context-free grammars are consistent; necessary and sufficient conditions for consistency are given by Booth and Thompson (1973). However, probabilistic context-free grammars obtained by training on a finite corpus using popular methods (such as expectation-maximization) are guaranteed to be consistent Nederhof and Satta (2006).

We immediately use a slightly more complex example, which we will later reuse.

**Example 3**.: Let us consider an arbitrary RNN

\[R=\langle\Sigma,N,h_{-1},W,W^{\prime},E,E^{\prime}\rangle\]

with the single-letter alphabet \(\Sigma=\{a\}\), the neurons \(\{1,2,3,n,n^{\prime}\}\subseteq N\), initial activation \(h_{-1}(i)=0\) for all \(i\in\{1,2,3,n,n^{\prime}\}\), and the following linear combinations:

\[h_{s,t}(1)=\sigma\langle h_{s,t-1}(1)+h_{s,t-1}(n)-h_{s,t-1}(n^{\prime})\rangle\]

\[h_{s,t}(2)=\sigma\langle h_{s,t-1}(2)+1\rangle\]

\[h_{s,t}(3)=\sigma\langle h_{s,t-1}(3)+3h_{s,t-1}(1)\rangle\]

\[E_{s,t}(\$)=h_{s,t}(3)-h_{s,t}(2)\]

\[E_{s,t}(a)=h_{s,t}(2)\]

Now we distinguish two cases:

**Case 1: If \(h_{s,t}(n)-h_{s,t}(n^{\prime})=0\) for all \(t\in\mathbb{N}\), then \(h_{s,t}(1)=0\) and \(h_{s,t}(2)=t+1\) and \(h_{s,t}(3)=0\). Hence we have \(E_{s,t}(\$)=-(t+1)\) and \(E_{s,t}(a)=t+1\). In this case the termination probability**

\[E^{\prime}_{s,t}(\$)=\frac{e^{-(t+1)}}{e^{-(t+1)}+e^{t+1}}=\frac{1}{1+e^{2(t+1)}}\]

(i.e., the likelihood of predicting $) shrinks rapidly towards \(0\), so the RNN assigns less than 15% of the probability mass to the terminating sequences (i.e., the finite strings), so the RNN is inconsistent (see Lemma 15 in the appendix).

**Case 2: Suppose that there exists a time

Figure 1: Sample RNNs over single-letter alphabets, and the weighted languages they recognize. \(M\) is some positive rational number which depends on the desired error margin. If we want to express the second and the third languages with error margin \(\delta\), \(M\) is chosen so that \(M>-\ln\frac{\delta}{1-\delta}\) in column \(2\), and chosen so that \((1+e^{-M})^{100}<\frac{1}{1-\delta}\) in column \(3\).

 

point \(T\in{\mathbb{N}}\) such that for all \(t\in{\mathbb{N}}\)

\[h_{s,t}(n)-h_{s,t}(n^{\prime})=\begin{cases}1&\text{if $t=T$}\\ 0&\text{otherwise.}\end{cases}\]

Then \(h_{s,t}(1)=0\) for all \(t\leq T\) and \(h_{s,t}(1)=1\) otherwise. In addition, we have \(h_{s,t}(2)=t+1\) and \(h_{s,t}(3)=\sigma\langle 3(t-T-1)\rangle\). Hence we have

\[E_{s,t}(\$) =\sigma\langle 3(t-T-1)\rangle-(t+1)\] \[=\begin{cases}-(t+1)&\text{if $t\leq T$}\\ 2t-3T-4&\text{otherwise}\end{cases}\] \[E_{s,t}(a) =t+1~{}~{},\]

which shows that the probability

\[E^{\prime}_{s,t}(\$) =\begin{cases}\frac{1}{1+e^{2(t+1)}}&\text{if $t\leq T$}\\ \frac{e^{t-3T-5}}{1+e^{t-3T-5}}&\text{otherwise}\end{cases}\]

of predicting \(\$\) increases over time and eventually (for \(t\gg 3T\)) far outweighs the probability of predicting \(a\). Consequently, in this case the RNN is consistent (see Lemma 16 in the appendix).

We have seen in the previous example that consistency is not trivial for RNNs, which takes us to the consistency problem for RNNs:

Consistency:Given an RNN \(R\), return "yes" if \(R\) is consistent and "no" otherwise.

We recall the following theorem, which, combined with our example, will prove that consistency is unfortunately undecidable for RNNs.

**Theorem 4** (Theorem 2 of Siegelmann and Sontag (1995)).: _Let \(M\) be an arbitrary deterministic Turing machine. There exists an RNN_

\[R=\langle\Sigma,N,h_{-1},W,W^{\prime},E,E^{\prime}\rangle\]

_with saturated linear activation, input alphabet \(\Sigma=\{a\}\), and \(1\) designated neuron \(n\in N\) such that for all \(s\in\Sigma^{*}\) and \(0\leq t\leq|s|\)_

* \(h_{s,t}(n)=0\) _if_ \(M\) _does not halt on_ \(\varepsilon\)_, and_
* _if_ \(M\) _does halt on empty input after_ \(T\) _steps, then_ \[h_{s,t}(n)=\begin{cases}1&\text{if $t=T$}\\ 0&\text{otherwise.}\end{cases}\]

In other words, such RNNs with saturated linear activation can semi-decide halting of an arbitrary Turing machine in the sense that a particular neuron achieves value \(1\) at some point during the evolution if and only if the Turing machine halts on empty input. An RNN with saturated linear activation is an RNN following our definition with the only difference that instead of our ReLU-activation \(\sigma\) the following saturated linear activation \(\sigma^{\prime}\colon{\mathbb{Q}}^{N}\to{\mathbb{Q}}^{N}\) is used. For every vector \(v\in{\mathbb{Q}}^{N}\) and \(n\in N\), let

\[\sigma^{\prime}\langle v\rangle(n)=\begin{cases}0&\text{if $v(n)<0$}\\ v(n)&\text{if $0\leq v(n)\leq 1$}\\ 1&\text{if $v(n)>1$}~{}~{}.\end{cases}\]

Since \(\sigma^{\prime}\langle v\rangle=\sigma\langle v\rangle-\sigma\langle v- \vec{1}\rangle\) for all \(v\in{\mathbb{Q}}^{N}\), and the right-hand side is a linear transformation, we can easily simulate saturated linear activation in our RNNs. To this end, each neuron \(n\in N\) of the original RNN \(R=\langle\Sigma,N,h_{-1},U,U^{\prime},E,E^{\prime}\rangle\) is replaced by two neurons \(n_{1}\) and \(n_{2}\) in the new RNN \(R^{\prime}=\langle\Sigma,N^{\prime},h^{\prime}_{-1},V,V^{\prime},F,F^{\prime}\rangle\) such that \(h_{s,t}(n)=h^{\prime}_{s,t}(n_{1})-h^{\prime}_{s,t}(n_{2})\) for all \(s\in\Sigma^{*}\) and \(0\leq t\leq|s|\), where the evaluation of \(h^{\prime}_{s,t}\) is performed in the RNN \(R^{\prime}\). More precisely, we use the transition matrix \(V\) and bias function \(V^{\prime}\), which is given by

\[V(n_{1},n^{\prime}_{1}) =V(n_{2},n^{\prime}_{1})=U(n,n^{\prime})\] \[V(n_{1},n^{\prime}_{2}) =V(n_{2},n^{\prime}_{2})=-U(n,n^{\prime})\] \[V^{\prime}_{a}(n_{1}) =U^{\prime}_{a}(n)\] \[V^{\prime}_{a}(n_{2}) =U^{\prime}_{a}(n)-1\] \[h^{\prime}_{-1}(n_{1}) =h_{-1}(n)\] \[h^{\prime}_{-1}(n_{2}) =0\]

for all \(n,n^{\prime}\in N\) and \(a\in\Sigma\cup\{\$\}\), where \(n_{1}\) and \(n_{2}\) are the two neurons corresponding to \(n\) and \(n^{\prime}_{1}\) and \(n^{\prime}_{2}\) are the two neurons corresponding to \(n^{\prime}\) (see Lemma 17 in the appendix).

**Corollary 5**.: _Let \(M\) be an arbitrary deterministic Turing machine. There exists an RNN_

\[R=\langle\Sigma,N,h_{-1},W,W^{\prime},E,E^{\prime}\rangle\]

_with input alphabet \(\Sigma=\{a\}\) and \(2\) designated neurons \(n_{1},n_{2}\in N\) such that for all \(s\in\Sigma^{*}\) and \(0\leq t\leq|s|\)_

* \(h_{s,t}(n_{1})-h_{s,t}(n_{2})=0\) _if_ \(M\) _does not halt on_ \(\varepsilon\)_, and_
* _if_ \(M\) _does halt on empty input after_ \(T\) _steps, then_ \[h_{s,t}(n_{1})-h_{s,t}(n_{2})=\begin{cases}1&\text{if $t=T$}\\ 0&\text{otherwise.}\end{cases}\] 

We can now use this corollary together with the RNN \(R\) of Example 3 to show that the consistency problem is undecidable. To this end, we simulate a given Turing machine \(M\) and identify the two designated neurons of Corollary 5 as \(n\) and \(n^{\prime}\) in Example 3. It follows that \(M\) halts if and only if \(R\) is consistent. Hence we reduced the undecidable halting problem to the consistency problem, which shows the undecidability of the consistency problem.

**Theorem 6**.: _The consistency problem for RNNs is undecidable._

As mentioned in Footnote 2, probabilistic context-free grammars obtained after training on a finite corpus using the most popular methods are guaranteed to be consistent. At least for 2-layer RNNs this does not hold.

**Theorem 7**.: _A two-layer RNN trained to a local optimum using Back-propagation-throughput (BPTT) on a finite corpus is not necessarily consistent._

Proof.: The first layer of the RNN \(R\) with a single alphabet symbol \(a\) uses one neuron \(n^{\prime}\) and has the following behavior:

\[h_{-1}(n^{\prime}) =0\] \[h_{s,t}(n^{\prime}) =\sigma\langle h_{s,t-1}(n^{\prime})+1\rangle\]

The second layer uses neuron \(n\) and takes \(h_{s,t}(n^{\prime})\) as input at time \(t\):

\[h_{s,t}(n) =\sigma\langle h_{s,t}(n^{\prime})-2\rangle\] \[E_{s,t}(a) =h_{s,t}(n) E_{s,t}(\$)=0\] \[E^{\prime}_{s,t}(a) =\begin{cases}\frac{1}{2}&\text{if }t\leq 1\\ \frac{e^{(t-1)}}{1+e^{(t-1)}}&\text{otherwise.}\end{cases}\]

Let the training data be \(\{a\}\). Then the objective we wish to maximize is simply \(R(a)\). The derivative of this objective with respect to each parameter is \(0\), so applying gradient descent updates does not change any of the parameters and we have converged to an inconsistent RNN. 

It remains an open question whether there is a single-layer RNN that also exhibits this behavior.

## 4 Highest-weighted string

Given a function \(f\colon\Sigma^{*}\to\mathbb{R}\) we are often interested in the highest-weighted string. This corresponds to the most likely sentence in a language model or the most likely translation for a decoder RNN in machine translation.

For deterministic probabilistic finite-state automata or context-free grammars only one path or derivation exists for any given string, so the identification of the highest-weighted string is the same task as the identification of the most probable path or derivation. However, for nondeterministic devices, the highest-weighted string is often harder to identify, since the weight of a string is the sum of the probabilities of all possible paths or derivations for that string. A comparison of the difficulty of identifying the most probable derivation and the highest-weighted string for various models is summarized in Table 1, in which we marked our results in bold face.

We present various results concerning the difficulty of identifying the highest-weighted string in a weighted language computed by an RNN. We also summarize some available algorithms. We start with the formal presentation of the three studied problems.

1. **Best string:** Given an RNN \(R\) and \(c\in(0,1)\), does there exist \(s\in\Sigma^{*}\) with \(R(s)>c\)?
2. **Consistent best string:** Given a consistent RNN \(R\) and \(c\in(0,1)\), does there exist \(s\in\Sigma^{*}\) with \(R(s)>c\)?
3. **Consistent best string of polynomial length:** Given a consistent RNN \(R\), polynomial \(\mathcal{P}\) with \(\mathcal{P}(x)\geq x\) for \(x\in\mathbb{N}^{+}\), and \(c\in(0,1)\), does there exist \(s\in\Sigma^{*}\) with \(|s|\leq\mathcal{P}(|R|)\) and \(R(s)>c\)? As usual the corresponding optimization problems are not significantly simpler than these decision problems. Unfortunately, the general problem is also undecidable, which can easily be shown using our example.

\begin{table}
\begin{tabular}{|c|c|} \hline  & Best-path & Best-string \\ \hline General RNN & **Undecidable** \\ Consistent RNN & **NP-c**3  \\ \hline Det. PFSA/PCFG & P 4  \\ Nondet. PFSA/PCFG & NP-c5  \\ \hline \end{tabular}
\end{table}
Table 1: Comparison of the difficulty of identifying the most probable derivation (Best-path) and the highest-weighted string (Best-string) for various models.

 



**Theorem 8**.: _The best string problem for RNNs is undecidable._

Proof.: Let \(M\) be an arbitrary Turing machine and again consider the RNN \(R\) of Example 3 with the neurons \(n\) and \(n^{\prime}\) identified with the designated neurons of Corollary 5. We note that \(R(\varepsilon)=\frac{1}{1+e^{2}}<0.12\) in both cases. If \(M\) does not halt, then \(R(a^{n})\leq\frac{1}{1+e^{2(n+1)}}\leq\frac{1}{1+e^{2}}<0.12\) for all \(n\in\mathbb{N}\). On the other hand, if \(M\) halts after \(T\) steps, then

\[R(a^{3T-5})\] \[=\Big{(}\prod_{t=0}^{T}\frac{e^{2(t+1)}}{1+e^{2(t+1)}}\Big{)} \cdot\big{(}\prod_{t=T+1}^{3T-6}\frac{1}{1+e^{t-3T-5}}\Big{)}\cdot\frac{1}{2}\] \[\geq\frac{2}{(-1,e^{-2})_{\infty}}\cdot\Big{(}\prod_{t=T+1}^{3T- 6}\frac{e^{3T+5-t}}{e^{3T+5-t+1}}\Big{)}\cdot\frac{1}{2}\] \[\geq\frac{2}{(-1,e^{-2})_{\infty}}\cdot(-1,e^{-1})_{\infty}\geq 0 .25\]

using Lemma 14 in the appendix. Consequently, a string with weight above \(0.12\) exists if and only if \(M\) halts, so the best string problem is also undecidable. 

If we restrict the RNNs to be consistent, then we can easily decide the best string problem by simple enumeration.

**Theorem 9**.: _The consistent best string problem for RNNs is decidable._

Proof.: Let \(R\) be the RNN over alphabet \(\Sigma\) and \(c\in(0,1)\) be the bound. Since \(\Sigma^{*}\) is countable, we can enumerate it via \(f\colon\mathbb{N}\to\Sigma^{*}\). In the algorithm we compute \(S_{n}=\sum_{i=0}^{n}R(f(i))\) for increasing values of \(n\). If we encounter a weight \(R(f(n))>c\), then we stop with answer "yes." Otherwise we continue until \(S_{n}>1-c\), at which point we stop with answer "no."

Since \(R\) is consistent, \(\lim_{i\to\infty}S_{i}=1\), so this algorithm is guaranteed to terminate and it obviously decides the problem. 

Next, we investigate the length \(|w_{R}^{\text{max}}|\) of the shortest string \(w_{R}^{\text{max}}\) of maximal weight in the weighted language \(R\) generated by a consistent RNN \(R\) in terms of its (binary storage) size \(|R|\). As already mentioned by Siegelmann and Sontag (1995) and evidenced here, only small precision rational numbers are needed in our constructions, so we assume that \(|R|\leq c\cdot|N|^{2}\) for a (reasonably small) constant \(c\), where \(N\) is the set of neurons of \(R\). We show that no computable bound on the length of the best string can exist, so its length can surpass all reasonable bounds.

**Theorem 10**.: _Let \(f\colon\mathbb{N}_{+}\to\mathbb{N}\) be the function with_

\[f(n)=\max_{\begin{subarray}{c}consistent\leavevmode\nobreak\ RNN\\ |R|\leq n\end{subarray}}|w_{R}^{\text{max}}|\]

_for all \(n\in\mathbb{N}_{+}\). There exists no computable function \(g\colon\mathbb{N}\to\mathbb{N}\) with \(g(n)\geq f(n)\) for all \(n\in\mathbb{N}\)._

Proof.: In the previous section (before Theorem 6) we presented an RNN \(R_{M}\) that simulates an arbitrary (single-track) Turing machine \(M\) with \(n\) states. By Siegelmann and Sontag (1995) we have \(|R_{M}|\leq c\cdot(4n+16)\). Moreover, we observed that this RNN \(R_{M}\) is consistent if and only if the Turing machine \(M\) halts on empty input. In the proof of Theorem 8 we have additionally seen that the length \(|w_{R}^{\text{max}}|\) of its best string exceeds the number \(T_{M}\) of steps required to halt.

For every \(n\in\mathbb{N}\), let \(BB(n)\) be the \(n\)-th "Busy Beaver" number (Rado, 1962), which is

\[BB(n)=\max_{\begin{subarray}{c}normalized\leavevmode\nobreak\ n-\text{state Turing machine $M$ with $2$ tape symbols that halts on empty input}\end{subarray}}T_{M}\]

It is well-known that \(BB\colon\mathbb{N}_{+}\to\mathbb{N}\) cannot be bounded by any computable function. However,

\[BB(n) \leq\max_{\begin{subarray}{c}normalized\leavevmode\nobreak\ n-\text{ state Turing machine $M$ with $2$ tape symbols that halts on empty input}\end{subarray}}|w_{R}^{\text{ max}}|\] \[\leq\max_{\begin{subarray}{c}consistent\leavevmode\nobreak\ RN\\ |R|\leq c\cdot(4n+16)\end{subarray}}|w_{R}^{\text{max}}|\] \[=f(4nc+16c)\enspace,\]

so \(f\) clearly cannot be computable and no computable function \(g\) can provide bounds for \(f\). 

Finally, we investigate the difficulty of the best string problem for consistent RNN restricted to solutions of polynomial length.

**Theorem 11**.: _Identifying the best string of polynomial length in a consistent RNN is NP-complete._

Proof.: To show NP-hardness, we reduce from the 3-SAT problem. Let \(x_{1},\ldots,x_{m}\) be \(m\) Boolean variables and

\[F=\bigwedge_{i=1}^{k}\Bigl{(}\ell_{i1}\vee\ell_{i2}\vee\ell_{i3}\Bigr{)}\enspace,\]

be a formula in conjunctive normal form, where \(\ell_{ij}\in\{x_{1},\ldots,x_{m},\neg x_{1},\ldots,\neg x_{m}\}\). 3-SAT asks whether there is a setting of \(x_{i}\)s that makes \(F\) true

We initialize \(h_{-1}(n)=0,\forall~{}n\in N=\{x_{1},\ldots,x_{m},c_{1},\ldots,c_{k},c^{\prime}_{1} ,\ldots,c^{\prime}_{k},F,r_{1},n_{2},n_{3},\star\}\). Let \(s\in\{0,1\}^{*}\) be the input string. Denote the value of \(F\) when \(x_{j}=s_{j}\) for all \(j\in[m]\) as \(F(s)\). Let \(t\in\mathbb{N}\) with \(t\leq|s|\). Set \(h_{s,t}(x_{m})=\sigma\langle I(s_{t})\rangle\), where \(I(0)=I(\$)=0\) and \(I(1)=1\). This stores the current input symbol in neuron \(x_{m}\), so \(h_{s,t}(x_{m})=I(s_{t})\). In addition, we let \(h_{s,t}(x_{j})=\sigma\langle h_{s,t-1}(x_{j+1})\rangle\) for all \(j\in[m-1]\). Consequently, for all \(j\in[m]\)

Next, we evaluate the clauses. For each \(i\in[k]\), we use two neurons \(c_{i}\) and \(c^{\prime}_{i}\) such that

\[h_{s,t}(c_{i}) =\sigma\langle f_{s,t}(\ell_{i1})+f_{s,t}(\ell_{i2})+f_{s,t}(\ell _{i3})\rangle\] \[h_{s,t}(c^{\prime}_{i}) =\sigma\langle f_{s,t}(\ell_{i1})+f_{s,t}(\ell_{i2})+f_{s,t}(\ell _{i3})-1\rangle,\]

where \(f_{s,t}(x_{m})=I(s_{t})\), \(f_{s,t}(\neg x_{m})=1-I(s_{t})\), and \(\forall j\in[m-1],f_{s,t}(x_{j})=h_{s,t-1}(x_{j+1})\), \(f_{s,t}(\neg x_{j})=1-h_{s,t-1}(x_{j+1})\). Note that \(h_{s,t}(c_{i})-h_{s,t}(c^{\prime}_{i})\) contains the evaluation of the clause \(\ell_{i1}\vee\ell_{i2}\lor\ell_{i3}\). Let

so \(h_{s,t}(F)=F(s)\) contains the evaluation of the formula \(F\) using the values in neurons \(x_{1},\ldots,x_{m}\).

We use three counters \(n_{1},n_{2},n_{3}\) to ensure that the only relevant inputs are of length \(m+2\):

\[h_{s,t}(n_{1}) =\sigma\langle h_{s,t-1}(n_{3})-(m+2)\rangle\] \[h_{s,t}(n_{2}) =\sigma\langle h_{s,t-1}(n_{3})-(m+1)\rangle\] \[h_{s,t}(n_{3}) =\sigma\langle h_{s,t-1}(n_{3})+1\rangle~{},\]

which yields \(h_{s,t}(n_{3})=t+1\), \(h_{s,t}(n_{2})=\sigma\langle t-(m+1)\rangle\), and \(h_{s,t}(n_{1})=\sigma\langle t-(m+2)\rangle\).

Our goal neuron is \(\star\), which we set to

\[h_{s,t}(\star)=\sigma\langle h_{s,t-1}(F)-h_{s,t-1}(n_{1})+h_{s,t-1}(n_{2})-1\rangle\]

so that

\[h_{s,t}(\star)=\begin{cases}h_{s,t-1}(F)&\text{if $t=m+2$}\\ 0&\text{otherwise,}\end{cases}\]

so \(h_{s,t}(\star)=1\) if and only if \(t=m+2\) and \(F(s)=1\).

Let \(m^{\prime}=m+4\). The output is set as follows:

\[E_{s,t}(0) =E_{s,t}(1)=m^{\prime}\big{(}1-2h_{s,t}(\star)\big{)}\] \[E_{s,t}(\$)=-m^{\prime}\big{(}1-2h_{s,t}(\star)\big{)}~{}~{},\]

This yields \(E_{s,t}(0)=S_{s,t}(1)=-E_{s,t}(\$)=-m^{\prime}\) if \(t=m+2\) and \(F(s)=1\), and \(m^{\prime}\) otherwise. For \(a\in\{0,1\}\),

\[E^{\prime}_{s,t}(a)\!=\!\left\{\begin{array}{ll}\frac{e^{-m^{ \prime}}}{2e^{-m^{\prime}}+e^{m^{\prime}}}&\text{if $t\!=\!m\!+\!2$ and $ F(s)\!=\!1$}\\ \frac{e^{m^{\prime}}}{2e^{m^{\prime}}+e^{-m^{\prime}}}&\text{otherwise}\end{array}\right.\] \[E^{\prime}_{s,t}(\$)\!=\!\left\{\begin{array}{ll}\frac{e^{ m^{\prime}}}{2e^{-m^{\prime}}+e^{m^{\prime}}}&\text{if $t\!=\!m\!+\!2$ and $F( s)\!=\!1$}\\ \frac{e^{-m^{\prime}}}{2e^{m^{\prime}}+e^{-m^{\prime}}}&\text{otherwise}.\end{array}\right.\]

Finally, we set the threshold \(\xi=3^{-m^{\prime}}\). When \(|s|\neq m+2\), \(s_{m+3}\not=\$\), so the weight of \(s\) contains the factor \(\frac{e^{-m^{\prime}}}{2e^{-m^{\prime}}+e^{m^{\prime}}}=\frac{1}{2+e^{2m^{ \prime}}}\) and thus is upper-bounded by \(\frac{1}{2+e^{2m^{\prime}}}<\xi\). Hence no input of length different from \(m+2\) achieves a weight that exceeds \(\xi\). A string \(s\) of length \(m+2\) achieves the weight \(w_{s}\) given by

When \(F(s)=0\), \(w_{s}<\frac{e^{-m^{\prime}}}{2e^{m^{\prime}}+e^{-m^{\prime}}}<\xi\), so if \(F\) is unsatisfiable, no input string achieves a weight above the threshold \(\xi\). When \(F(s)=1\), \(w_{s}=\frac{e^{m^{\prime}}}{2e^{-m^{\prime}}+e^{m^{\prime}}}\cdot\Big{(}\frac{ e^{en^{\prime}}}{2e^{m^{\prime}}+e^{-m^{\prime}}}\Big{)}^{m+2}>\xi\). An input string with weight above \(\xi\) exists if and only if \(F\) is satisfiable. Obviously, the reduction can be computed in polynomial time since all constants can be computed in logarithmic space. The constructed RNN is consistent, since the output prediction is constant after \(m+3\) steps. 

## 5 Equivalence

We prove that equivalence of two RNNs is undecidable. For comparison, equivalence of two deterministic WFSAs can be tested in time \(O(|\Sigma||(Q_{A}|+|Q_{B}|)^{3})\), where \(|Q_{A}|\), \(|Q_{B}|\) are the number of states of the two WFSAs and \(|\Sigma|\) is the size of the alphabet Cortes et al. (2007); equivalence of nondeterministic WFSAs are undecidable Griffiths (1968). The decidability of language equivalence for deterministic probabilistic push-downtown automata (PPDA) is still open Forejit et al. (2014), although equivalence for deterministic unweighted push-downtown automata (PDA) is decidable Senizergues (1997) 

The equivalence problem is formulated as follows:

Equivalence:Given two RNNs \(R\) and \(R^{\prime}\), return "yes" if \(R(s)=R^{\prime}(s)\) for all \(s\in\Sigma^{*}\), and "no" otherwise.

**Theorem 12**.: _The equivalence problem for RNNs is undecidable._

Proof.: We prove by contradiction. Suppose Turing machine \(M\) decides the equivalence problem. Given any deterministic Turing Machine \(M^{\prime}\), construct the RNN \(R\) that simulates \(M^{\prime}\) on input \(\epsilon\) as described in Corollary 5. Let \(E_{s,t}(a)=0\) and \(E_{s,t}(\$)=h_{s,t}(n_{1})-h_{s,t}(n_{2})\). If \(M^{\prime}\) does not halt on \(\epsilon\), for all \(t\in\mathbb{N}\), \(E^{\prime}_{s,t}(a)=E^{\prime}_{s,t}(\$)=1/2\); if \(M^{\prime}\) halts after \(T\) steps, \(E^{\prime}_{s,T}(a)=1/(e+1)\), \(E_{s,T}(\$)=e/(e+1)\). Let \(R^{\prime}\) be the trivial RNN that computes \(\{a^{n}:P(a^{n})=2^{-(n+1)},n\geq 0\}\). We run \(M\) on input \(\langle R,R^{\prime}\rangle\). If \(M\) returns "no", \(M^{\prime}\) halts on \(x\), else it does not halt. Therefore the Halting Problem would be decidable if equivalence is decidable. Therefore equivalence is undecidable. 

## 6 Minimization

We look next at minimization of RNNs. For comparison, state-minimization of a deterministic PFSA is \(O(|E|\log|Q|)\) where \(|E|\) is the number of transitions and \(|Q|\) is the number of states [3]. Minimization of a non-deterministic PFSA is PSPACE-complete [4].

We focus on minimizing the number of hidden neurons (\(|N|\)) in RNNs:

Minimization:Given RNN \(R\) and non-negative integer \(n\), return "yes" if \(\exists\) RNN \(R^{\prime}\) with number of hidden units \(|N^{\prime}|\leq n\) such that \(R(s)=R^{\prime}(s)\) for all \(s\in\Sigma^{*}\), and "no" otherwise.

**Theorem 13**.: _RNN minimization is undecidable._

Proof.: We reduce from the Halting Problem. Suppose Turing Machine \(M\) decides the minimization problem. For any Turing Machine \(M^{\prime}\), construct the same RNN \(R\) as in Theorem 12. We run \(M\) on input \(\langle R,0\rangle\). Note that an RNN with no hidden unit can only output constant \(E^{\prime}_{s,t}\) for all \(t\). Therefore the number of hidden units in \(R\) can be minimized to \(0\) if and only if it always outputs \(E^{\prime}_{s,t}(a)=E^{\prime}_{s,t}(\$)=1/2\). If \(M\) returns "yes", \(M^{\prime}\) does not halt on \(\epsilon\), else it halts. 

## 7 Conclusion

We proved the following hardness results regarding RNN as a recognizer of weighted languages:

1. Consistency: 1. Inconsistent RNNs exist. 2. Consistency of RNNs is undecidable.
2. Highest-weighted string: 1. Finding the highest-weighted string for an arbitrary RNN is undecidable. 2. Finding the highest-weighted string for a consistent RNN is decidable, but the solution length can surpass all computable bounds. 3. Restricting to solutions of polynomial length, finding the highest-weighted string is NP-complete.
3. Testing equivalence of RNNs and minimizing the number of neurons in an RNN are both undecidable.

Although our undecidability results are upshots of the Turing-completeness of RNN [8], our NP-completeness result is original, and surprising, since the analogous hardness results in PFSA relies on the fact that there are multiple derivations for a single string [9]. The fact that these results hold for the relatively simple RNNs we used in this paper suggests that the case would be the same for more complicated models used in NLP, such as long short term memory networks (LSTMs; Hochreiter and Schmidhuber 1997).

Our results show the non-existence of (efficient) algorithms for interesting problems that researchers using RNN in natural language processing tasks may have hoped to find. On the other hand, the non-existence of such efficient or exact algorithms gives evidence for the necessity of approximation, greedy or heuristic algorithms to solve those problems in practice. In particular, since finding the highest-weighted string in RNN is the same as finding the most-likely translation in a sequence-to-sequence RNN decoder, our NP-completeness result provides some justification for employing greedy and beam search algorithms in practice.

### Acknowledgments

This work was supported by DARPA (W911NF-15-1-0543 and HR0011-15-C-0115). Andreas Maletti was financially supported by DFG Graduiertenkolleg 1763 (QuantLA) 



## References

* Aho et al. (1974) Alfred V. Aho, John E. Hopcroft, and Jeffrey D. Ullman. 1974. _The design and analysis of computer algorithms_. Addison-Wesley.
* Allauzen et al. (2007) Cyril Allauzen, Michael Riley, Johan Schalkwyk, Wojciech Skut, and Mehryar Mohri. 2007. _OpenFst: A General and Efficient Weighted Finite-State Transducer Library_, Springer Berlin Heidelberg, Berlin, Heidelberg, pages 11-23.
* Bahdanau et al. (2014) D. Bahdanau, K. Cho, and Y. Bengio. 2014. Neural machine translation by jointly learning to align and translate. In _Proc. ICLR_.
* Booth and Thompson (1973) T. L. Booth and R. A. Thompson. 1973. Applying probability measures to abstract languages. _IEEE Transactions on Computers_ C-22(5):442-450. [https://doi.org/10.1109/t-c.1973.223746](https://doi.org/10.1109/t-c.1973.223746).
* Casacuberta and de la Higuera (2000) Francisco Casacuberta and Colin de la Higuera. 2000. Computational complexity of problems on probabilistic grammars and transducers. _Grammatical Inference: Algorithms and Applications Lecture Notes in Computer Science_ pages 15-24. [https://doi.org/10.1007/978-3-540-45257-7_2](https://doi.org/10.1007/978-3-540-45257-7_2).
* Cortes et al. (2007) Corinna Cortes, Mehryar Mohri, and Ashish Rastogi. 2007. \({\rm L}_{p}\) distance and equivalence of probabilistic automata. _International Journal of Foundations of Computer Science_ 18(04):761-779. [https://doi.org/10.1142/s0129054107004966](https://doi.org/10.1142/s0129054107004966).
* Droste et al. (2013) Manfred Droste, Werner Kuich, and Heiko Vogler. 2013. _Handbook of Weighted Automata_. Springer Berlin.
* Forejt et al. (2014) Voitch Forejt, Petr Janar, Stefan Kiefer, and James Worrell. 2014. Language equivalence of probabilistic pushdown automata. _Information and Computation_ 237:1-11. [https://doi.org/10.1016/j.ic.2014.04.003](https://doi.org/10.1016/j.ic.2014.04.003).
* Griffiths (1968) T. V. Griffiths. 1968. The unsolvability of the equivalence problem for \(\wedge\)-free nondeterministic generalized machines. _Journal of the ACM_ 15(3):409-413. [https://doi.org/10.1145/321466.321473](https://doi.org/10.1145/321466.321473).
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural Comput._ 9(8):1735-1780. [https://doi.org/10.1162/neco.1997.9.8.1735](https://doi.org/10.1162/neco.1997.9.8.1735).
* Jiang and Ravikumar (1993) Tao Jiang and B. Ravikumar. 1993. Minimal NFA problems are hard. _SIAM Journal on Computing_ 22(6):1117-1141. [https://doi.org/10.1137/0222067](https://doi.org/10.1137/0222067).
* Jozefowicz et al. (2016) Rafal Jozefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, and Yonghui Wu. 2016. Exploring the limits of language modeling. [https://arxiv.org/pdf/1602.02410.pdf](https://arxiv.org/pdf/1602.02410.pdf).
* Kim and Rush (2016) Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Austin, Texas, pages 1317-1327. [https://aclweb.org/anthology/D16-1139](https://aclweb.org/anthology/D16-1139).
* Knuth (1977) Donald E. Knuth. 1977. A generalization of Dijkstra's algorithm. _Information Processing Letters_ 6(1):1-5. [https://doi.org/10.1016/0020-0190](https://doi.org/10.1016/0020-0190)(77)90002-3.
* Mikolov and Zweig (2012) T. Mikolov and G. Zweig. 2012. Context dependent recurrent neural network language model. In _2012 IEEE Spoken Language Technology Workshop (SLT)_. pages 234-239. [https://doi.org/10.1109/SLT.2012.6424228](https://doi.org/10.1109/SLT.2012.6424228).
* Nederhof and Satta (2006) Mark-Jan Nederhof and Giorgio Satta. 2006. Estimation of consistent probabilistic context-free grammars. In _Proceedings of the Human Language Technology Conference of the NAACL, Main Conference_. Association for Computational Linguistics, New York City, USA, pages 343-350. [http://www.aclweb.org/anthology/N/N06-1044](http://www.aclweb.org/anthology/N/N06-1044).
* Radg (1962) Tibor Radg. 1962. On non-computable functions. _Bell System Technical Journal_ 41:877-884.
* Senizergues (1997) Geraud Senizergues. 1997. The equivalence problem for deterministic pushdown automata is decidable. In _Proc. Automata, Languages and Programming: 24th International Colloquium_, Springer Berlin Heidelberg, pages 671-681.
* Siegelmann and Sontag (1995) Hava T. Siegelmann and Eduardo D. Sontag. 1995. On the computational power of neural nets. _Journal of Computer and System Sciences_ 50(1):132-150. [https://doi.org/10.1006/jcss.1995.1013](https://doi.org/10.1006/jcss.1995.1013).
* Simaan (1996) Khalil Simaan. 1996. Computational complexity of probabilistic disambiguation by means of treegrammars. In _Proc. COLING_. pages 1175-1180. [https://doi.org/10.3115/993268.993392](https://doi.org/10.3115/993268.993392).
* Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, _Advances in Neural Information Processing Systems 27_. Curran Associates, Inc., pages 3104-3112.

## Appendix

**Lemma 14**.: _For every \(k\in\mathbb{N}_{+}\)_

\[\prod_{t\in\mathbb{N}}\frac{e^{k(t+1)}}{e^{k(t+1)}+1}=\frac{2}{(-1;\, e^{-k})_{\infty}}~{}~{},\]

_where \((-1;e^{-k})_{\infty}\) is the infinite \(e^{-k}\)-Pochhammer symbol._

Proof.: \[\prod_{t\in\mathbb{N}}\frac{e^{k(t+1)}}{e^{k(t+1)}+1}=\prod_{t\in \mathbb{N}_{+}}\Big{(}\frac{e^{kt}}{e^{kt}+1}\cdot\frac{e^{-kt}}{e^{-kt}}\Big{)}\] \[=\prod_{t\in\mathbb{N}_{+}}\frac{1}{1+e^{-kt}}=\Bigg{(}\Big{(}\prod _{t\in\mathbb{N}_{+}}\frac{1}{1+e^{-kt}}\Big{)}^{-1}\Bigg{)}^{-1}\]

\[= \Big{(}\prod_{t\in\mathbb{N}_{+}}(1+e^{-kt})\Big{)}^{-1}=\bigg{(} \frac{1}{2}\prod_{t\in\mathbb{N}}(1+e^{-kt})\bigg{)}^{-1}\] \[= \;\frac{2}{(-1;\,e^{-k})_{\infty}}\qed\]

**Lemma 15**.: _Reconsider the RNN of Example 3 and suppose that \(h_{s,t}(n)-h_{s,t}(n^{\prime})=0\) for all \(t\in\mathbb{N}\). Then_

\[\sum_{s\in\Sigma^{*}}R(s)=1-\frac{2}{(-1;\,e^{-2})_{\infty}}\approx 0.14\]

Proof.: \[\quad\sum_{s\in\Sigma^{*}}R(s)=\sum_{n\in\mathbb{N}}R(a^{n})\] \[= \sum_{n\in\mathbb{N}}\Bigl{(}\frac{e^{-(n+1)}}{e^{n+1}+e^{-(n+1) }}\cdot\prod_{t=0}^{n-1}\frac{e^{t+1}}{e^{t+1}+e^{-(t+1)}}\Bigr{)}\] \[= \;1-\prod_{t\in\mathbb{N}}\frac{e^{2(t+1)}}{e^{2(t+1)}+1}=1- \frac{2}{(-1;\,e^{-2})_{\infty}}\] \[\approx \;0.14\enspace,\]

where the final equality utilizes Lemma 14. 

**Lemma 16**.: _Reconsider the RNN of Example 3 and suppose that there exists a time point \(T\in\mathbb{N}\) such that for all \(t\in\mathbb{N}\)_

\[h_{s,t}(n)-h_{s,t}(n^{\prime})=\begin{cases}1&\text{if $t=T$}\\ 0&\text{otherwise.}\end{cases}\]

_Then_

\[\sum_{s\in\Sigma^{*}}R(s)=1\]

Proof.: \[\quad\sum_{s\in\Sigma^{*}}R(s)=\sum_{n\in\mathbb{N}}R(a^{n})\] \[= \;\Bigl{(}\sum_{n=0}^{T}R(a^{n})\Bigr{)}+\Bigl{(}\sum_{n=T+1}^{ \infty}R(a^{n})\Bigr{)}\] \[= \;\sum_{n=0}^{T}\Bigl{(}\frac{e^{-(n+1)}}{e^{n+1}+e^{-(n+1)}} \cdot\prod_{t=0}^{n-1}\frac{e^{t+1}}{e^{t+1}+e^{-(t+1)}}\Bigr{)}\] \[\quad+\sum_{n=T+1}^{\infty}\frac{e^{2n-3T-4}}{e^{n+1}+e^{2n-3T-4}}\] \[\cdot\Bigl{(}\prod_{t=0}^{T}\frac{e^{t+1}}{e^{t+1}+e^{-(t+1)}} \Bigr{)}\cdot\Bigl{(}\prod_{t=T+1}^{n-1}\frac{e^{t+1}}{e^{t+1}+e^{2t-3T-4}} \Bigr{)}\] \[= \;\sum_{n=0}^{T}\Bigl{(}\frac{1}{e^{2(n+1)}+1}\cdot\prod_{t=0}^{ n-1}\frac{e^{2(t+1)}}{e^{2(t+1)}+1}\Bigr{)}\] \[\quad+\sum_{n=T+1}^{\infty}\frac{e^{n-3T-5}}{1+e^{n-3T-5}}\cdot \Bigl{(}\prod_{t=0}^{T}\frac{e^{2(t+1)}}{e^{2(t+1)}+1}\Bigr{)}\] \[\qquad\cdot\Bigl{(}\prod_{t=T+1}^{n-1}\frac{1}{1+e^{t-3T-5}} \Bigr{)}\] \[= \;1-\Bigl{(}\prod_{t=0}^{T}\frac{e^{2(t+1)}}{e^{2(t+1)}+1}\Bigr{)} \cdot\Bigl{(}\prod_{t=T+1}^{\infty}\frac{1}{1+e^{t-3T-5}}\Bigr{)}\] \[\geq \;1-\Bigl{(}\prod_{t=0}^{T}\frac{e^{2(t+1)}}{e^{2(t+1)}+1}\Bigr{)} \cdot\Bigl{(}\prod_{t\in\mathbb{N}}\frac{1}{1+e^{t}}\Bigr{)}\] \[= \;1\qed\]

**Lemma 17**.: __

Proof.: We set \(h_{s,-1}(n)=h_{-1}(n)\) for all \(n\in N\) and \(h^{\prime}_{s,-1}(n^{\prime})=h^{\prime}_{-1}(n^{\prime})\) for all \(n^{\prime}\in N^{\prime}\). Then trivially \(h^{\prime}_{s,-1}(n_{1})-h^{\prime}_{s,-1}(n_{2})=h_{-1}(n)-0=h_{s,-1}(n)\). Moreover, \(h^{\prime}_{s,t}(n)=\sigma\langle V\cdot h^{\prime}_{s,t-1}+V^{\prime}_{s[t]} \rangle(n_{1})\)

\[= \;\sigma\langle\sum_{n^{\prime}\in N^{\prime}}V(n_{1},n^{\prime} )\cdot h^{\prime}_{s,t-1}(n^{\prime})\] \[+ V^{\prime}_{s[t]}(n_{1})\rangle\] \[= \;\sigma\langle\sum_{n^{\prime}\in N}\bigl{(}V(n_{1},n^{\prime} _{1})\cdot h^{\prime}_{s,t-1}(n^{\prime}_{1})\] \[+ V(n_{1},n^{\prime}_{2})\cdot h^{\prime}_{s,t-1}(n^{\prime}_{2}) \bigr{)}+V^{\prime}_{s[t]}(n_{1})\rangle\] \[= \;\sigma\langle\sum_{n^{\prime}\in N}U(n,n^{\prime})\cdot\big{(}h ^{\prime}_{s,t-1}(n^{\prime}_{1})-h^{\prime}_{s,t-1}(n^{\prime}_{2})\bigr{)}\] \[+ U^{\prime}_{s[t]}(n)\rangle\] \[= \;\sigma\langle\sum_{n^{\prime}\in N}U(n,n^{\prime})\cdot h_{s, t-1}(n^{\prime})+U^{\prime}_{s[t]}(n)\rangle\] \[\cdot\Bigl{(}\prod_{t=0}^{T}\frac{e^{t+1}}{e^{t+1}+e^{-(t+1)}} \Bigr{)}\cdot\Bigl{(}\prod_{t=T+1}^{n-1}\frac{e^{t+1}}{e^{t+1}+e^{2t-3T-4}} \Bigr{)}\] \[= \;\sum_{n=0}^{T}\Bigl{(}\frac{1}{e^{2(n+1)}+1}\cdot\prod_{t=0}^{n-1 }\frac{e^{2(t+1)}}{e^{2(t+1)}+1}\Bigr{)}\] \[+ \sum_{n=T+1}^{\infty}\frac{e^{n-3T-5}}{1+e^{n-3T-5}}\cdot\Bigl{(} \prod_{t=0}^{T}\frac{e^{2(t+1)}}{e^{2(t+1)}+1}\Bigr{)}\] \[+ \sum_{n=T+1}^{\infty}\frac{e^{n-3T-5}}{1+e^{n-3T-5}}\cdot\Bigl{(} \prod_{t=0}^{T}\frac{e^{2(t+1)}}{e^{2(t+1)}+1}\Bigr{)}\] \[= \;1-\Bigl{(}\prod_{t=0}^{T}\frac{e^{2(t+1)}}{e^{2(t+1)}+1}\Bigr{)} \cdot\Bigl{(}\prod_{t=T+1}^{\infty}\frac{1}{1+e^{t-3T-5}}\Bigr{)}\] \[\geq \;1-\Bigl{(}\prod_{t=0}^{T}\frac{e^{2(t+1)}}{e^{2(t+1)}+1}\Bigr{)} \cdot\Bigl{(}\prod_{t\in\mathbb{N}}\frac{1}{1+e^{t}}\Bigr{)}\] \[= \;1\qed\]

**Lemma 18**.: __

Proof.: We set \(h_{s,-1}(n)=h_{-1}(n)\) for all \(n\in N\) and \(h^{\prime}_{s,-1}(n^{\prime})=h^{\prime}_{-1}(n^{\prime})\) for all \(n^{\prime}\in N^{\prime}\). Then trivially \(h^{\prime}_{s,-1}(n_{1})-h^{\prime}_{s,-1}(n_{2})=h_{-1}(n)-0=h_{s,-1}(n)\). Moreover, \(h^{\prime}_{s,t}(n)=\sigma\langle V\cdot h^{\prime}_{s,t-1}+V^{\prime}_{s[t]} \rangle(n_{1})\)