On the Computational Power of RNNs	Recent neural network architectures such as the basic recurrent neural network (RNN) and Gated Recurrent Unit (GRU) have gained prominence as end-to-end learning architectures for natural language processing tasks. But what is the computational power of such systems? We prove that finite precision RNNs with one hidden layer and ReLU activation and finite precision GRUs are exactly as computationally powerful as deterministic finite automata. Allowing arbitrary precision, we prove that RNNs with one hidden layer and ReLU activation are at least as computationally powerful as pushdown automata. If we also allow infinite precision, infinite edge weights, and nonlinear output activation functions, we prove that GRUs are at least as computationally powerful as pushdown automata. All results are shown constructively.

However, just as PDAs can lazily be described as a DFA with an associated stack, it is well-known that Turing machines are equally as powerful as DFAs with associated queues, which can be simulated with two stacks. Such an approach using two counters was used in proofs in [6], [8] to establish that RNNs with arbitrary precision can emulate Turing machines. We believe that an approach related to this fact could ultimately prove successful, but it would be more useful if set up as in the proofs above in a way that is faithful to the architecture of the neural networks. Counter automata of this sort are also quite unlike the usual implementations found for context-free languages or their extensions for natural languages. Work described in [10] demonstrates that in practice, LSTMs cannot really generalize to recognize the Dyck language INLINEFORM1 . It remains to investigate whether any recent neural network variation does in fact readily generalize outside its training set to “out of sample” examples. This would be an additional topic for future research.

Merge the INLINEFORM4 nodes in the input layer corresponding to the input and merge the single output nodes of both RNNs. Stack the two hidden layers, and add no new edges. There were INLINEFORM5 hidden nodes in the first RNN and INLINEFORM6 in the second, so altogether the new RNN has INLINEFORM7 hidden nodes. The output of the new RNN is equal to the summed output of the two original RNNs, and from the proofs of Theorems 1.1 and 1.10 these outputs are always nonnegative. Thus the output of the new RNN is INLINEFORM0 if and only if the outputs of both old RNNs were INLINEFORM1 , immediately proving the theorem. INLINEFORM2  Discussion 1.12. This result shows that simple RNNs with arbitrary precision are at least as computationally powerful as PDAs.

Furthermore, the authors admit that were the saturated-linear activation to be replaced with the similar and more common sigmoid or hyperbolic tangent activation functions, their methodology would fail. More recent work [7] suggests that single-layer RNNs with rectified linear unit (ReLU) activations and softmax outputs can also be simulated as universal Turing Machines, but this approach again suffers from the assumption that the entire input is read before computation occurs. Motivated by these earlier theoretical results, in this report we seek to show results about the computational power of recurrent architectures actually used in practice - namely, those that read tokens one at a time and that use standard rather than specially chosen activation functions. In particular we will prove that, allowing infinite precision, RNNs with just one hidden layer and ReLU activation are at least as powerful as PDAs, and that GRUs are at least as powerful as deterministic finite automata (DFAs). Furthermore, we show that using infinite edge weights and a non-standard output function, GRUs are also at least as powerful as PDAs.

Introduction	Recent work [1] suggests that recurrent “neural network" models of several types perform better than sequential models in acquiring and processing hierarchical structure. Indeed, recurrent networks have achieved state-of-the-art results in a number of natural language processing tasks, including named-entity recognition [2], language modeling [3], sentiment analysis [4], natural language generation [5], and beyond. The hierarchical structure associated with natural languages is often modeled as some variant of context-free languages, whose languages may be defined over an alphabet INLINEFORM0 . These context-free languages are exactly those that can be recognized by pushdown automata (PDAs). Thus it is natural to ask whether these modern natural language processing tools, including simple recurrent neural networks (RNNs) and other, more advanced recurrent architectures, can learn to recognize these languages. The computational power of RNNs has been studied extensively using empirical testing. Much of this research [8], [9] focused on the ability of RNNs to recognize simple context-free languages such as INLINEFORM0 and INLINEFORM1 , or context-sensitive languages such as INLINEFORM2 .

Now because the INLINEFORM6 are clearly linearly independent, we can find a INLINEFORM7 such that INLINEFORM8  for all INLINEFORM0 and it's clear that the INLINEFORM1 -language of the resulting GRU will be INLINEFORM2 , as desired. INLINEFORM3  Discussion 2.2. In the above proof, we are implicitly assuming that the activation functions of the GRU are not actually the sigmoid and hyperbolic tangent functions but rather finite precision analogues for which the equations we solved are all consistent. However, for the remainder of this section we can drop this assumption. If we remove the finite precision restriction, we again wish to prove that Gated RNNs are as powerful as PDAs. To do so, we emulate the approach from Section 1. Immediately we encounter difficulties - in particular, our previous approach relied on maintaining the digits of a state INLINEFORM0 in base INLINEFORM1 very carefully. With outputs now run through sigmoid and hyperbolic tangent functions, this becomes very hard. Furthermore, updating the state INLINEFORM2 occasionally requires multiplication by INLINEFORM3 (when we read a closing parenthesis).

Suggestions for Further Research	We recognize two main avenues for further research. The first is to remove the necessity for infinite edge weights in the proof of Theorem 2.13, and the second is to extend the results of Theorems 1.11 and 2.13 to Turing recognizable languages. In the proof of Lemma 2.11, edge weights of INLINEFORM0 are necessary for determining whether a hidden node ever becomes negative. Merely using large but finite weights does not suffice, because the values in the hidden state that they will be multiplied with are rapidly decreasing. Their product will vanish, and thus we would not be able to utilize the squashing properties of common activation functions as we did in the proof of Lemma 2.11. Currently we believe that it is possible to prove that GRUs are as computationally powerful as PDAs without using infinite edge weights, but are unaware of a method to do so. Because to the our knowledge there is no analogue of the Chomsky-Sch INLINEFORM0 tzenberger Theorem for Turing recognizable languages, it seems difficult to directly extend our methods to prove that recurrent architectures are as computationally powerful as Turing machines.

Gated RNNs	In practice, architectures more complicated than the simple RNNs studied above - notably gated RNNs, including the Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) - perform better on many natural language tasks. Thus we are motivated to explore their computational capabilities. Here we focus on the GRU, described by the equations below: INLINEFORM0   for some INLINEFORM0 where INLINEFORM1 has dimension INLINEFORM2 and INLINEFORM3 is the sigmoid function and INLINEFORM4 is the hyperbolic tangent function, and the INLINEFORM5 symbol represents element-wise multiplication. Usually the hidden state INLINEFORM6 is initialized to be INLINEFORM7 , but we will ignore that restriction. Some literature switches the placements of the INLINEFORM8 and INLINEFORM9 , but since INLINEFORM10 this is immaterial. We begin this section by again limiting our architecture to use finite precision, and also assume INLINEFORM0 for some INLINEFORM1 . We can prove an analogue of Theorem 1.1: Theorem 2.1.

For the RNN, the input layer will have exactly two INLINEFORM13 s, namely the node with label INLINEFORM14 and the node with label INLINEFORM15 . Since all edges have weight INLINEFORM16 , that means that before adding INLINEFORM17 or applying INLINEFORM18 the maximum value a node in the hidden layer can take on is INLINEFORM19 . For this to occur it must be connected to both the nodes in the input layer with value INLINEFORM20 , and thus by definition its label must be INLINEFORM21 . By integrality every other node in the hidden layer will take on a value of at most INLINEFORM22 , so after adding INLINEFORM23 and applying INLINEFORM24 we easily see that the invariant is maintained. Utilizing this invariant it is clear that upon reading a word INLINEFORM0 the RNN will output INLINEFORM1 , and upon reading a word INLINEFORM2 it will output INLINEFORM3 . Thus INLINEFORM4 is precisely the INLINEFORM5 -language of the RNN and the theorem is proven. INLINEFORM6  Discussion 1.2. This result shows that simple RNNs with finite precision are exactly as computationally powerful as DFAs.

For every language INLINEFORM0 , INLINEFORM1 is regular if and only if INLINEFORM2 is the INLINEFORM3 -language of some finite precision GRU. Proof. The “if" direction can be shown in the same manner as in Theorem 1.1. So, here we focus on the “only if" direction. Suppose we have a DFA INLINEFORM0 with corresponding language INLINEFORM1 . We will construct a GRU whose inputs are one-hotted symbols from INLINEFORM2 with INLINEFORM3 hidden nodes whose INLINEFORM4 -language is INLINEFORM5 . For convenience, for all INLINEFORM0 let INLINEFORM1 denote the corresponding one-hot vector for INLINEFORM2 . Furthermore, let INLINEFORM3 . First set INLINEFORM0 and INLINEFORM1 and INLINEFORM2 , so the simplified GRU is given by: INLINEFORM3   Now, define an arbitrary bijective map INLINEFORM0 .

Related works [10], [11], [12] focus instead on Dyck languages of balanced parenthesis, which motivates some of our methods. Gated architectures such as the Gated Recurrent Unit (GRU) and Long Short-Term Memory (LSTM) obtain high accuracies on each of these tasks. While simpler RNNs have also been tested, one difficulty is that the standard hyperbolic tangent activation function makes counting difficult. On the other hand, RNNs with ReLU activations were found to perform better, but suffer from what is known as the “exploding gradient problem" and thus are more difficult to train [8]. Instead of focusing on a single task, many researchers have studied the broader theoretical computational power of recurrent models, where weights are not trained but rather initialized to recognize a desired language. A celebrated result [6] shows that a simple recurrent architecture with 1058 hidden nodes and a saturated-linear activation INLINEFORM0 is a universal Turing Machine, with: INLINEFORM1  However, their architecture encodes the whole input in its internal state and the relevant computation is only performed after reading a terminal token. This differs from more common RNN variants that consume tokenized inputs at each time step.

Simple RNNs	Let a simple RNN be an RNN with the following architecture: INLINEFORM0   where INLINEFORM0 for all INLINEFORM1 , for some chosen activation function INLINEFORM2 , usually the ReLU or the hyperbolic tangent functions. We assume that the inputs are one-hots of a given set of symbols INLINEFORM3 , vectors of length INLINEFORM4 where each element but one is INLINEFORM5 and the remaining element is INLINEFORM6 . Say that an RNN accepts an input INLINEFORM0 of length INLINEFORM1 if after passing INLINEFORM2 through the RNN, its final output INLINEFORM3 belongs to a predetermined set INLINEFORM4 , for which membership can be tested in INLINEFORM5 time. Let the INLINEFORM6 -language of an RNN consist exactly of all inputs that it accepts given set INLINEFORM7 . In practice, the inputs and hidden nodes of an RNN are stored as numbers with finite precision. Including this restriction, we show the following result: Theorem 1.1. For every language INLINEFORM0 , INLINEFORM1 is regular if and only if INLINEFORM2 is the INLINEFORM3 -language of some finite precision simple RNN.

In terms of reducing the size of the hidden layer constructed in the proof of the “only if" direction, it seems likely that INLINEFORM0 is optimal since INLINEFORM1 is defined on INLINEFORM2 inputs and needs to be captured fully by the RNN. Removing the finite precision stipulation unsurprisingly increases the capabilities of RNNs. It is natural to now ask whether these simple RNNs can recognize more complicated INLINEFORM0 -languages, and indeed the answer is affirmative. Thus we shift our focus to context-free languages. We begin with some preliminaries: The Dyck language INLINEFORM0 consists of all words over the size INLINEFORM1 alphabet INLINEFORM2 that correspond to a balanced string of INLINEFORM3 types of parentheses. We also define the set of proper prefixes INLINEFORM4  so that any word in INLINEFORM0 is the prefix of a word in INLINEFORM1 but is itself unbalanced. We proceed with a motivating theorem: Theorem 1.3 (Chomsky-Sch INLINEFORM0 tzenberger Theorem). Any context-free language INLINEFORM1 can be written as INLINEFORM2 for some INLINEFORM3 and regular language INLINEFORM4 after a suitable relabeling.

Thus to upper bound INLINEFORM0 it suffices to consider only words that do not contain the substring INLINEFORM1 , which are words in the form INLINEFORM2  with INLINEFORM0 open parentheses followed by INLINEFORM1 closing parentheses. Furthermore, adding extra closing parenthesis where suitable clearly increases the final INLINEFORM2 so we can assume INLINEFORM3 . We can then exactly calculate INLINEFORM4 as INLINEFORM5  Considering each sum separately we have for sufficiently large INLINEFORM0 that INLINEFORM1   and INLINEFORM0   And therefore INLINEFORM0 is an upper bound on INLINEFORM1 . Thus INLINEFORM2  for all INLINEFORM0 as desired. INLINEFORM1  Corollary 2.6. For any input word INLINEFORM0 , if INLINEFORM1 contains INLINEFORM2 open parentheses and INLINEFORM3 closing parentheses then INLINEFORM4  with INLINEFORM0 for all INLINEFORM1 . Proof. This follows directly from the computations in the proof of Lemma 2.5 and the recurrence for INLINEFORM0 . INLINEFORM1  Now, set INLINEFORM0 .

This immediately follows from the definition of a balanced string. Indeed, if INLINEFORM0 is the state associated with INLINEFORM1 then this unique INLINEFORM2 is given by INLINEFORM3   INLINEFORM0  We are now ready to show the following: Lemma 1.7. Given an input word INLINEFORM0 , we have that INLINEFORM1 . Proof. We first restrict our attention to INLINEFORM0 . Note that INLINEFORM1  for any INLINEFORM0 , which follows from the definition of INLINEFORM1 and INLINEFORM2 . Then using Corollary 1.5 we find INLINEFORM3  Now using the inequality in the proof of Lemma 1.6 we immediately obtain INLINEFORM0 as desired. Considering now INLINEFORM0 we notice INLINEFORM1  and doing an analysis similar to that for INLINEFORM0 , we obtain INLINEFORM1 as desired. INLINEFORM2  Applying Lemma 1.6 allows us to make the following statement: Lemma 1.8. Given a word INLINEFORM0 , consider the unique INLINEFORM1 such that INLINEFORM2 . Then with respect to a word INLINEFORM3 with INLINEFORM4 , we have INLINEFORM5 .

But because INLINEFORM4 and INLINEFORM5 for all INLINEFORM6 , this is impossible to do with the GRU architecture. To account for both of these issues, instead of keeping track of the state INLINEFORM0 as we read a word, we will instead keep track of the state INLINEFORM1 of a word INLINEFORM2 defined by INLINEFORM3   for all INLINEFORM0 , for some predetermined sufficiently large INLINEFORM1 . We have the following relationship between INLINEFORM2 and INLINEFORM3 : Lemma 2.3. For any word INLINEFORM0 we have INLINEFORM1 for all INLINEFORM2 . Proof. Multiplying the recurrence relationship for INLINEFORM0 by INLINEFORM1 we recover the recurrence relationship for INLINEFORM2 in Section 1, implying the desired result. INLINEFORM3  Thus the state INLINEFORM0 allows us to keep track of the old state INLINEFORM1 without having to multiply by any constant greater than INLINEFORM2 . Furthermore, for large INLINEFORM3 , INLINEFORM4 will be extremely small, allowing us to abuse the fact that INLINEFORM5 for small values of INLINEFORM6 .

However, INLINEFORM3 edge weights make sense in the context of allowing infinite precision, and simple nonlinear functions over the hidden nodes are often used in practice, like the common softmax activation function.

If INLINEFORM6 then from the proof of Lemma 2.8 we have that INLINEFORM7 and so INLINEFORM8 . Since INLINEFORM9 this means that INLINEFORM10 . If INLINEFORM11 then from the analogue of the proof of Lemma 2.8 for INLINEFORM12 , we obtain INLINEFORM13 . This completes the proof. INLINEFORM14  We are now ready to combine these lemmas to prove an important result, the analogue of Theorem 1.10 for GRUs: Theorem 2.12. The INLINEFORM0 -language of the GRU described earlier in the section is INLINEFORM1 . Proof. Consider any input word INLINEFORM0 into the GRU. We consider four cases: In this case, we clearly have INLINEFORM0 and INLINEFORM1 from the proof of Corollary 2.7, so by Lemmas 2.9 and 2.10 we have that INLINEFORM2  with INLINEFORM0 . Furthermore from Lemma 2.11 we have that INLINEFORM1 so since INLINEFORM2 we must have INLINEFORM3  for sufficiently large INLINEFORM0 , as desired.

But since INLINEFORM5 by Lemma 1.9 this means that either INLINEFORM6 or INLINEFORM7 , so INLINEFORM8 . Thus INLINEFORM0 if and only if INLINEFORM1 , completing the proof of the theorem. INLINEFORM2  Now recall in the proof of Theorem 1.1 we showed that any regular language INLINEFORM0 was the INLINEFORM1 -language of some simple RNN, and moreover that for any input not in INLINEFORM2 the output of that RNN is positive. This allows us to provide a simple proof of the main theorem of this section: Theorem 1.11. For any context-free language INLINEFORM0 , suppose we relabel and write INLINEFORM1 for some regular language INLINEFORM2 , whose corresponding minimum-size DFA has INLINEFORM3 states. Then there exists a simple RNN with a hidden layer of size INLINEFORM4 whose INLINEFORM5 -language is INLINEFORM6 . Proof. Consider the simple RNN with INLINEFORM0 as its INLINEFORM1 -language described in the proof of Theorem 1.1 and the simple RNN with INLINEFORM2 as its INLINEFORM3 -language constructed to prove Theorem 1.10.

Each of the hidden nodes are initialized to INLINEFORM0 except a single hidden node with label INLINEFORM1 for a randomly chosen INLINEFORM2 , which is initialized to INLINEFORM3 . To complete the description of the RNN, we set INLINEFORM4 and INLINEFORM5 . We claim that the following invariant is maintained: after reading some word, suppose the current state of INLINEFORM6 is INLINEFORM7 . Then after reading the same word, the hidden nodes of the RNN would all be equal to INLINEFORM8 except for one node with label INLINEFORM9 for some INLINEFORM10 , which would equal INLINEFORM11 . We prove the claim by induction on the length of the inputted word INLINEFORM0 . The base case of INLINEFORM1 is trivial. Now assume that after reading a word of length INLINEFORM2 the current state of INLINEFORM3 is INLINEFORM4 , and after reading that same word all hidden nodes of the RNN are equal to INLINEFORM5 except one node with label INLINEFORM6 for some INLINEFORM7 , which is equal to INLINEFORM8 . If the next symbol is INLINEFORM9 , then the current state of INLINEFORM10 would be INLINEFORM11 where INLINEFORM12 .

The INLINEFORM0 -language of the simple RNN described earlier in the section is INLINEFORM1 . Proof. Consider any input INLINEFORM0 into the RNN. For the remainder of the proof, remember that INLINEFORM1 for all INLINEFORM2 because of the ReLU activation. We consider three cases: In this case by Corollary 1.5 we have INLINEFORM0 . Furthermore, by Lemma 1.7 we have INLINEFORM1 . By combining Lemmas 1.7 and 1.9, we have INLINEFORM2 . Thus INLINEFORM3 which, given that INLINEFORM4 , equals INLINEFORM5 precisely when INLINEFORM6 , by the inequality from the proof of Lemma 1.6. In this case we clearly must have INLINEFORM0 for some INLINEFORM1 and thus by Lemma 1.8 we have that either INLINEFORM2 or INLINEFORM3 , so INLINEFORM4 . Suppose INLINEFORM0 is the minimal index such that INLINEFORM1 . Then by minimality INLINEFORM2 so again by Lemma 1.8 we have that either INLINEFORM3 or INLINEFORM4 .

Proof. We begin with the “if" direction. Suppose we are given some simple RNN and set INLINEFORM0 . It suffices to show that there exists a DFA that accepts the INLINEFORM1 -language of this RNN. Assume that the RNN has INLINEFORM2 hidden nodes, and that these hidden nodes are precise up to INLINEFORM3 bits. Then there are exactly INLINEFORM4 possible hidden states for the RNN. Construct the following DFA with: It's clear that after reading the first INLINEFORM0 inputs of a word INLINEFORM1 , the current state of this DFA is INLINEFORM2 , which immediately completes the proof of this direction. For the “only if" direction, suppose we have a DFA INLINEFORM0 with corresponding language INLINEFORM1 . We will construct a simple RNN whose inputs are one-hotted symbols from INLINEFORM2 , with ReLU activation function INLINEFORM3 , and with INLINEFORM4 hidden nodes whose INLINEFORM5 -language is INLINEFORM6 .

INLINEFORM3  We may now proceed to show the main theorem of this section, an analogue of Theorem 1.11 for GRUs: Theorem 2.13. For any context-free language INLINEFORM0 suppose we relabel and write INLINEFORM1 for some regular language INLINEFORM2 , whose corresponding minimum DFA has INLINEFORM3 states. Then there exists a GRU with a hidden layer of size INLINEFORM4 whose INLINEFORM5 -language is INLINEFORM6 . Proof. This follows by combining the GRUs from the proofs of Theorems 2.1 and 2.12, as we did for simple RNNs in the proof of Theorem 1.11. INLINEFORM0  Discussion 2.14. A critical idea in this section was to use the fact that INLINEFORM0 near INLINEFORM1 , and in fact this idea can be used for any activation function with a well-behaved Taylor series expansion around INLINEFORM2 . Discussion 2.15. We “cheated" a little bit by allowing INLINEFORM0 edge weights and by having INLINEFORM1 where INLINEFORM2 wasn't quite linear.

Proof. The interested reader may find a proof in [13]. INLINEFORM0  Thus it makes sense to focus on constructing sets INLINEFORM0 and simple RNNs whose INLINEFORM1 -language is INLINEFORM2 . Indeed, since INLINEFORM3 for some homomorphism INLINEFORM4 , we start by focusing on INLINEFORM5 , in some sense the “hardest" context-free language. The critical idea is to “memorize" an input in the binary representation of some rational number, simulating a stack. Indeed, consider associating with any word INLINEFORM0 a state INLINEFORM1 , defined as follows: INLINEFORM2   Consider the word INLINEFORM0 . The evolution of the state as the word is read symbol by symbol is given by INLINEFORM1  This example makes it clear that this notion of state accurately captures all the relevant information about words in INLINEFORM0 . The difficulty in capturing this notion of state in a RNN is that the constant to multiply INLINEFORM0 by changes depending on the input (it can be either INLINEFORM1 or INLINEFORM2 in our example above). Thus storing INLINEFORM3 in a single hidden node is impossible.

anchor=south east] INLINEFORM16 (q3); (r2) – node [pos=0.75, anchor=north west] INLINEFORM17 (q4); (r2) – node [pos=0.25, anchor=south west] INLINEFORM18 (h1); (r2) – node [pos=0.5, anchor=south west] INLINEFORM19 (h2); Thus we have found an efficient way to store INLINEFORM0 . Now it's clear that for any INLINEFORM1 we have INLINEFORM2 and for any INLINEFORM3 we have INLINEFORM4 , so it is tempting to try and add a simple output layer to this RNN and claim that its INLINEFORM5 -language is INLINEFORM6 . However, this is most likely impossible to accomplish. Indeed, consider the word INLINEFORM0 . We have that INLINEFORM1 for this word, but INLINEFORM2 . Furthermore, consider the word INLINEFORM3 . We have that INLINEFORM4 for all INLINEFORM5 and INLINEFORM6 for this word, yet INLINEFORM7 .

Instead, we use two hidden nodes. Below, we generalize from INLINEFORM4 to INLINEFORM5 . Ignoring the output layer for now, consider the simple RNN defined by INLINEFORM0   where the inputs INLINEFORM0 are INLINEFORM1 one-hots of the symbols in INLINEFORM2 (the alphabet of INLINEFORM3 ) in the order INLINEFORM4 and the hidden states have dimension INLINEFORM5 where INLINEFORM6   As before, associate with each word INLINEFORM0 a state INLINEFORM1 now satisfying INLINEFORM2   for all INLINEFORM0 . This is similar to the state we defined before, though now generalized to INLINEFORM0 and also with intentionally present blank space inserted between the digits in base INLINEFORM1 . We will show the following invariant: Lemma 1.4. Given an input word INLINEFORM0 , we have INLINEFORM1 or INLINEFORM2 for all INLINEFORM3 . Proof. We proceed by induction on INLINEFORM0 . The base case of INLINEFORM1 is trivial. Now, suppose INLINEFORM2 for some INLINEFORM3 and assume without loss of generality that INLINEFORM4 .

Then INLINEFORM5  Now, since INLINEFORM0 we have that INLINEFORM1 for any INLINEFORM2 , which follows immediately from the stack interpretation of the base INLINEFORM3 representation of INLINEFORM4 . Thus INLINEFORM5 and so INLINEFORM6  as desired. Alternatively, suppose INLINEFORM0 for some INLINEFORM1 . Again, assume without loss of generality that INLINEFORM2 . Then INLINEFORM3  The fact that INLINEFORM0 clearly implies that INLINEFORM1 and so we have that INLINEFORM2  which completes the induction. INLINEFORM0  A pictorial example of this RNN is depicted below for INLINEFORM0 : vertex=[circle, draw] [transform shape] vertex](r1) at (-2, 2) INLINEFORM0 ; vertex](r2) at (2, 2) INLINEFORM1 ; vertex](q1) at (-7,-2) INLINEFORM2 ; vertex](q2) at (-5,-2) INLINEFORM3 ; vertex](q3) at (-3,-2) INLINEFORM4 ; vertex](q4) at (-1,-2) INLINEFORM5 ; vertex](h1) at (3,

-2) INLINEFORM6 ; vertex](h2) at (7,-2) INLINEFORM7 ; [every path/.style=-, every node/.style=inner sep=1pt] (r1) – node [pos=0.5, anchor=south east] INLINEFORM8 (q1); (r1) – node [pos=0.5, anchor=south east] INLINEFORM9 (q2); (r1) – node [pos=0.7, anchor=north west] INLINEFORM10 (q3); (r1) – node [pos=0.5, anchor=north east] INLINEFORM11 (q4); (r1) – node [pos=0.75, anchor=south west] INLINEFORM12 (h1); (r1) – node [pos=0.65, anchor=south west] INLINEFORM13 (h2); (r2) – node [anchor=south east, pos=0.8] INLINEFORM14 (q1); (r2) – node [anchor=south east, pos=0.8] INLINEFORM15 (q2); (r2) – node [pos=0.5,

A.2. GRU D 2 \displaystyle D_2 Examples	Consider the GRU described in the proof of Theorem 2.12 for INLINEFORM0 and INLINEFORM1 . We will show the evolution of its hidden state as it reads various inputs: For this example we obtain INLINEFORM0  For this example we obtain INLINEFORM0  For this example we obtain INLINEFORM0

Hence we must be able to flag when an inappropriate closing parenthesis appears in an input and retain that information while reading the rest of the input. To that end, consider the following simple RNN, an example of which can be found in Appendix A.1: INLINEFORM8   where again the inputs INLINEFORM0 are INLINEFORM1 one-hots of the symbols in INLINEFORM2 (the alphabet of INLINEFORM3 ) in the order INLINEFORM4 and the hidden states have dimension INLINEFORM5 where INLINEFORM6   Because the last four elements of the first two rows of INLINEFORM0 are all equal to INLINEFORM1 and otherwise the first two rows of INLINEFORM2 and INLINEFORM3 are the same as before, it is clear that Lemma 1.4 still applies in some form for the new simple RNN. Indeed, denoting INLINEFORM4  we have Corollary 1.5. With respect to a word INLINEFORM0 , we have INLINEFORM1 or INLINEFORM2 for all INLINEFORM3 . We proceed with an important lemma: Lemma 1.6. For any word INLINEFORM0 , there is a unique INLINEFORM1 such that INLINEFORM2 . Proof.

Similarly, with respect to a word INLINEFORM6 with INLINEFORM7 , we have INLINEFORM8 . Proof. First suppose INLINEFORM0 . As in the proof of Lemma 1.7, we use INLINEFORM1  where we again use Corollary 1.5 and the fact that INLINEFORM0 from Lemma 1.7. But from the proof of Lemma 1.6, since INLINEFORM1 we know that INLINEFORM2  and since INLINEFORM0 we have that INLINEFORM1 since INLINEFORM2 and INLINEFORM3 are integral. Thus INLINEFORM4 as desired. Now assume INLINEFORM0 . As in the previous case we obtain INLINEFORM1  again using Corollary 1.5 and Lemma 1.7. And again using the inequality from the proof of Lemma 1.6 and the fact that INLINEFORM0 we obtain INLINEFORM1 , completing the proof. INLINEFORM2  Thus we have constructed the desired “flags."

Indeed, hidden nodes INLINEFORM0 and INLINEFORM1 remain equal to INLINEFORM2 while the currently read input lies in INLINEFORM3 , but one of these nodes becomes positive the moment the currently read input does not lie in this set. However, there are still difficulties. It is possible for INLINEFORM0 or INLINEFORM1 to become positive and later return to INLINEFORM2 . Indeed, running the simple RNN on the word INLINEFORM3 , we compute INLINEFORM4 . However, clearly INLINEFORM5 . Therefore we need to add architecture that retains the information as to whether the hidden nodes INLINEFORM6 or INLINEFORM7 ever become positive, and below we show that hidden nodes INLINEFORM8 and INLINEFORM9 respectively are sufficient. Lemma 1.9. For any input INLINEFORM0 we have INLINEFORM1 INLINEFORM2  Proof. From the definition of INLINEFORM0 and INLINEFORM1 we have INLINEFORM2 INLINEFORM3  and since INLINEFORM0 for all INLINEFORM1 (because of the ReLU) we immediately have the result by induction or direct expansion. INLINEFORM2  We are now ready to combine these lemmas and accomplish our original goal: Theorem 1.10.

A.1. Simple RNN D 2 \displaystyle D_2 Examples	Consider the RNN described in the proof of Theorem 1.10 for INLINEFORM0 . We will show the evolution of its hidden state as it reads various inputs: For this example we obtain INLINEFORM0  For this example we obtain INLINEFORM0  For this example we obtain INLINEFORM0

Then construct INLINEFORM1 vectors INLINEFORM2  where for all INLINEFORM0 and INLINEFORM1 we set INLINEFORM2  Our goal will be to find INLINEFORM0 and INLINEFORM1 such that if INLINEFORM2 for some INLINEFORM3 , and INLINEFORM4 is the one-hot encoding of some INLINEFORM5 , then INLINEFORM6 where if INLINEFORM7 for some INLINEFORM8 then INLINEFORM9 . If this is possible, then we could set INLINEFORM10 and be able to track the current state of the DFA effectively. The strategy for accomplishing this is essentially to pick a simple INLINEFORM0 , and then solve a system of equations to produce the desired INLINEFORM1 . For convenience, define the natural map INLINEFORM0 where INLINEFORM1 if and only if the INLINEFORM2 th element of INLINEFORM3 is equal to INLINEFORM4 . Let INLINEFORM0  where INLINEFORM0  for all INLINEFORM0 and INLINEFORM1 . Now consider the INLINEFORM2 equations INLINEFORM3  where INLINEFORM0 , for every INLINEFORM1 and INLINEFORM2 . Let INLINEFORM3   for all INLINEFORM0 and INLINEFORM1 and INLINEFORM2 .

Letting INLINEFORM3  The INLINEFORM0 earlier equations can now be combined as a single matrix equation given by INLINEFORM1  Now it is easy to see that INLINEFORM0  where INLINEFORM0 is a INLINEFORM1 matrix for each INLINEFORM2 . In particular, we have that INLINEFORM3  for each INLINEFORM0 . Using basic row operations it is easy to see that INLINEFORM0 for all INLINEFORM1 , so INLINEFORM2  and thus INLINEFORM0 is well-defined. Furthermore, since INLINEFORM1 for each INLINEFORM2 , the inputs into all inverse hyperbolic tangents in INLINEFORM3 lie in INLINEFORM4 and so INLINEFORM5 is well-defined as well. Thus our expression for INLINEFORM6 is well-defined. Now, given our choices for the INLINEFORM0 , and INLINEFORM1 , after reading any input INLINEFORM2 , if INLINEFORM3 is the current state of the DFA associated with INLINEFORM4 , then INLINEFORM5 .

In terms of the stack of digits interpretation of INLINEFORM7 , INLINEFORM8 is the same except between every pop or push we add INLINEFORM9 zeros to the top of the stack. Again we wish to construct a GRU from whose hidden state we can recover INLINEFORM0 . Ignoring the output layer for now, consider the GRU defined by INLINEFORM1   where INLINEFORM0 will be determined later, the inputs INLINEFORM1 are again INLINEFORM2 one-hots of the symbols in INLINEFORM3 in the order INLINEFORM4 and the hidden states have dimension INLINEFORM5 where INLINEFORM6   where INLINEFORM0 is the inverse of the sigmoid function. For sufficiently large INLINEFORM1 , clearly our use of INLINEFORM2 is well-defined. We will show the following invariant: Lemma 2.4. Given an input word INLINEFORM0 , if INLINEFORM1 then we have INLINEFORM2 for all INLINEFORM3 . Proof. As in Section 1, let INLINEFORM0 and INLINEFORM1 and INLINEFORM2 . First, we will show INLINEFORM3 for all INLINEFORM4 by induction on INLINEFORM5 .

The base case is trivial, so note INLINEFORM6   so by induction INLINEFORM0 as desired. Similarly, we obtain INLINEFORM1 for all INLINEFORM2 . Now we restrict our attention to INLINEFORM0 . Note that INLINEFORM1   and so using the definition of INLINEFORM0 we obtain INLINEFORM1   If we removed the INLINEFORM0 from the above expression, it would simplify to INLINEFORM1  which is exactly the recurrence relation satisfied by INLINEFORM0 . Since the expressions inside the hyperbolic tangents are extremely small (on the order of INLINEFORM1 ), this implies that INLINEFORM2 is a good approximation for INLINEFORM3 as desired. This will be formalized in the next lemma. INLINEFORM4  Lemma 2.5. For any input word INLINEFORM0 , if INLINEFORM1 then we have INLINEFORM2 for all INLINEFORM3 . Proof. Let INLINEFORM0 for all INLINEFORM1 . Then we easily find that INLINEFORM2  Now define INLINEFORM0 by the recurrence INLINEFORM1  with INLINEFORM0 . Because INLINEFORM1 for all INLINEFORM2 it is easy to see that INLINEFORM3 for all INLINEFORM4 .

Now by a Taylor expansion, INLINEFORM0 , so we have that INLINEFORM1  for INLINEFORM0 . Thus we obtain the bound INLINEFORM1  Since INLINEFORM0 and INLINEFORM1 we also have INLINEFORM2  Similarly we obtain the bound INLINEFORM0  Since again INLINEFORM0 and INLINEFORM1 we also have INLINEFORM2  Thus if we define INLINEFORM0 by the recurrence INLINEFORM1  with INLINEFORM0 , then INLINEFORM1 for all INLINEFORM2 . Now we wish to upper bound INLINEFORM0 . Since INLINEFORM1 is not present in the recurrence for INLINEFORM2 , assume without loss of generality that all parenthesis in an input word INLINEFORM3 lie in INLINEFORM4 . Suppose that INLINEFORM5 was a substring of INLINEFORM6 , so that INLINEFORM7 . Then we would have INLINEFORM8   However, for the word INLINEFORM0 (which would clearly still lie in INLINEFORM1 ) we would have INLINEFORM2   which is larger.

We then have the following useful analogues of Lemmas 1.7 and 1.8: Corollary 2.7. For any input word INLINEFORM0 we have INLINEFORM1 . Proof. This follows immediately from Corollary 2.6 and the fact that INLINEFORM0 . INLINEFORM1  Lemma 2.8. Given a word INLINEFORM0 , consider the unique INLINEFORM1 such that INLINEFORM2 . Then for an input word INLINEFORM3 with INLINEFORM4 , we have INLINEFORM5 . Note that INLINEFORM0  so multiplying both sides by INLINEFORM0 and using the inequality from the proof of Lemma 2.5 we have INLINEFORM1  Now by Corollary 2.6 we have that INLINEFORM0  where we used the inequality from the proof of Lemma 1.6 and the fact that INLINEFORM0 . Therefore INLINEFORM1  Since INLINEFORM0 we have that INLINEFORM1 and so for sufficiently large INLINEFORM2 we then have INLINEFORM3  as desired.

INLINEFORM0  With these results in hand, consider the larger GRU, an example of which can be found in Appendix A.2, defined by INLINEFORM0   where the inputs INLINEFORM0 are again INLINEFORM1 one-hots of the symbols in INLINEFORM2 in the order INLINEFORM3 and the hidden states have dimension INLINEFORM4 where INLINEFORM5   As before, with respect to a word INLINEFORM0 define INLINEFORM1 by INLINEFORM2   for all INLINEFORM0 and all INLINEFORM1 . Similarly define INLINEFORM2 by INLINEFORM3   For our new GRU, let INLINEFORM0 . We then have the following results: Lemma 2.9. For any input word INLINEFORM0 we have INLINEFORM1 . Proof. This follows immediately from the proof of Lemma 2.4. INLINEFORM0  Lemma 2.10. For any input word INLINEFORM0 , if INLINEFORM1 contains INLINEFORM2 open parentheses and INLINEFORM3 closing parenthesis then INLINEFORM4 INLINEFORM5  with INLINEFORM0 for all INLINEFORM1 . Proof.

This follows immediately from the proof of Corollary 2.6 and the new INLINEFORM0 , since INLINEFORM1 behaves exactly like INLINEFORM2 if each input INLINEFORM3 or INLINEFORM4 were INLINEFORM5 or INLINEFORM6 respectively, instead. INLINEFORM7  Lemma 2.11. For any input word INLINEFORM0 we have INLINEFORM1 and INLINEFORM2 if and only if INLINEFORM3 . Proof. From our chosen INLINEFORM0 we see that INLINEFORM1 INLINEFORM2  Since INLINEFORM0 and since the fourth and eighth rows of INLINEFORM1 are identically INLINEFORM2 , the equation INLINEFORM3  implies that INLINEFORM0 INLINEFORM1  which immediately implies that INLINEFORM0 . Now, suppose INLINEFORM1 . Then from Corollary 2.7 and its analogue for INLINEFORM2 we see that INLINEFORM3 for all INLINEFORM4 , so INLINEFORM5 as desired. Otherwise, there exists some minimal INLINEFORM0 such that INLINEFORM1 . Then INLINEFORM2 for some INLINEFORM3 . Consider the unique INLINEFORM4 such that INLINEFORM5 .

As in Case 1 we have that INLINEFORM0 and so by Lemmas 2.9 and 2.10 we have that INLINEFORM1  with INLINEFORM0 . Furthermore from Lemma 2.11 we have that INLINEFORM1 so here INLINEFORM2  for sufficiently large INLINEFORM0 , since the minimum value of INLINEFORM1 is clearly INLINEFORM2 . Suppose INLINEFORM0 for some unique INLINEFORM1 . If INLINEFORM2 for some INLINEFORM3 then from Lemmas 2.9 and 2.10 and the proof of Lemma 2.8 we obtain INLINEFORM4  for sufficiently large INLINEFORM0 . If instead INLINEFORM1 then the same technique with the inequality INLINEFORM2 can be used to show INLINEFORM3  if INLINEFORM0 is sufficiently large. As before using Lemma 2.11 we have that INLINEFORM1 and combining these bounds we find that INLINEFORM2  In this case we know that INLINEFORM0 by Lemma 2.9, so we have INLINEFORM1  and by Lemma 2.11 we know that INLINEFORM0 so INLINEFORM1  Thus INLINEFORM0 if INLINEFORM1 and INLINEFORM2 otherwise, as desired.

The RNN has three layers: the first layer (input layer) has INLINEFORM0 nodes; the second layer (hidden layer) has INLINEFORM1 nodes; and the third layer (output layer) has one node. For the INLINEFORM2 nodes in the input layer associated with the one-hot of the current symbol, label each node with its corresponding symbol from INLINEFORM3 . Label the INLINEFORM4 hidden nodes (in both the first and second layers) with all INLINEFORM5 symbol-state combinations INLINEFORM6 for INLINEFORM7 and INLINEFORM8 . For every INLINEFORM0 , connect the node in the input layer with label INLINEFORM1 to all nodes in the hidden layer with labels INLINEFORM2 for any INLINEFORM3 with edges with weight INLINEFORM4 . For all INLINEFORM5 , connect the node in the input layer with label INLINEFORM6 to all nodes in the hidden layer with labels INLINEFORM7 where INLINEFORM8 with edges also of weight INLINEFORM9 . Finally, for all INLINEFORM10 , connect the node in the hidden layer with label INLINEFORM11 to the single node in the output layer with an edge of weight INLINEFORM12 .

Question: How do they prove that RNNs with arbitrary precision are as powerful as a pushdown automata?
Answer: Theorem 1.11. For any context-free language INLINEFORM0 , suppose we relabel and write INLINEFORM1 for some regular language INLINEFORM2 , whose corresponding minimum-size DFA has INLINEFORM3 states. Then there exists a simple RNN with a hidden layer of size INLINEFORM4 whose INLINEFORM5 -language is INLINEFORM6 .