SlowGuess/ACL_2020_OCR huggingface dataset:
William Merrill* Roy Schwartz*§
Gail Weiss† Noah A. Smith*§
Yoav Goldberg*‡  
Eran Yahav†
* Allen Institute for AI † Technion ‡ Bar Ilan University § University of Washington {willm, yoavg, roys, noah}@allenai.org {sgailw, yahave}@cs.technion.ac.il
We develop a formal hierarchy of the expressive capacity of RNN architectures. The hierarchy is based on two formal properties: space complexity, which measures the RNN's memory, and rational recurrence, defined as whether the recurrent update can be described by a weighted finite-state machine. We place several RNN variants within this hierarchy. For example, we prove the LSTM is not rational, which formally separates it from the related QRNN (Bradbury et al., 2016). We also show how these models' expressive capacity is expanded by stacking multiple layers or composing them with different pooling functions. Our results build on the theory of "saturated" RNNs (Merrill, 2019). While formally extending these findings to unsaturated RNNs is left to future work, we hypothesize that the practical learnable capacity of unsaturated RNNs obeys a similar hierarchy. Experimental findings from training unsaturated networks on formal languages support this conjecture.
While neural networks are central to the performance of today's strongest NLP systems, theoretical understanding of the formal properties of different kinds of networks is still limited. It is established, for example, that the Elman (1990) RNN is Turing-complete, given infinite precision and computation time (Siegelmann and Sontag, 1992, 1994; Chen et al., 2018). But tightening these unrealistic assumptions has serious implications for expressive power (Weiss et al., 2018), leaving a significant gap between classical theory and practice, which theorems in this paper attempt to address.
Recently, Peng et al. (2018) introduced rational RNNs, a subclass of RNNs whose internal state can be computed by independent weighted finite automata (WFAs). Intuitively, such models have a computationally simpler recurrent update than
Figure 1: Hierarchy of state expressiveness for saturated RNNs and related models. The  $y$  axis represents increasing space complexity.  $\emptyset$  means provably empty. Models are in bold with qualitative descriptions in gray.
conventional models like long short-term memory networks (LSTMs; Hochreiter and Schmidhuber, 1997). Empirically, rational RNNs like the quasi-recurrent neural network (QRNN; Bradbury et al., 2016) and unigram rational RNN (Dodge et al., 2019) perform comparably to the LSTM, with a smaller computational budget. Still, the underlying simplicity of rational models raises the question of whether their expressive power is fundamentally limited compared to other RNNs.
In a separate line of work, Merrill (2019) introduced the saturated  $\mathbf{RNN}^1$  as a formal model for analyzing the capacity of RNNs. A saturated RNN is a simplified network where all activation functions have been replaced by step functions. The saturated network may be seen intuitively as a "stable" version of its original RNN, in which the in
Originally referred to as the asymptotic RNN.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 443-459 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
ternal activations act discretely. A growing body of work—including this paper—finds that the saturated theory predicts differences in practical learnable capacity for various RNN architectures (Weiss et al., 2018; Merrill, 2019; Suzgun et al., 2019a).
We compare the expressive power of rational and non-rational RNNs, distinguishing between state expressiveness (what kind and amount of information the RNN states can capture) and language expressiveness (what languages can be recognized when the state is passed to a classifier). To do this, we build on the theory of saturated RNNs.
State expressiveness We introduce a unified hierarchy (Figure 1) of the functions expressible by the states of rational and non-rational RNN encoders. The hierarchy is defined by two formal properties: space complexity, which is a measure of network memory, $^{2}$  and rational recurrence, whether the internal structure of the RNN can be described by WFAs. The hierarchy reveals concrete differences between LSTMs and QRNNs, and further separates both from a class containing convolutional neural networks (CNNs, Lecun and Bengio, 1995; Kim, 2014), Elman RNNs, and gated recurrent units (GRU; Cho et al., 2014).
We provide the first formal proof that LSTMs can encode functions that rational recurrences cannot. On the other hand, we show that the saturated Elman RNN and GRU are rational recurrences with constant space complexity, whereas the QRNN has unbounded space complexity. We also show that an unrestricted WFA has rich expressive power beyond any saturated RNN we consider—including the LSTM. This difference potentially opens the door to more expressive RNNs incorporating the computational efficiency of rational recurrences.
Language expressiveness When applied to classification tasks like language recognition, RNNs are typically combined with a "decoder": additional layer(s) that map their hidden states to a prediction. Thus, despite differences in state expressiveness, rational RNNs might be able to achieve comparable empirical performance to non-rational RNNs on NLP tasks. In this work, we consider the setup in which the decoders only view the final hidden state of the RNN. We demonstrate that
a sufficiently strong decoder can overcome some of the differences in state expressiveness between different models. For example, an LSTM can recognize  $a^n b^n$  with a single decoding layer, whereas a QRNN provably cannot until the decoder has two layers. However, we also construct a language that an LSTM can recognize without a decoder, but a QRNN cannot recognize with any decoder. Thus, no decoder can fully compensate for the weakness of the QRNN compared to the LSTM.
Experiments Finally, we conduct experiments on formal languages, justifying that our theorems correctly predict which languages unsaturated recognizers trained by gradient descent can learn. Thus, we view our hierarchy as a useful formal tool for understanding the relative capabilities of different RNN architectures.
Roadmap We present the formal devices for our analysis of RNNs in Section 2. In Section 3 we develop our hierarchy of state expressiveness for single-layer RNNs. In Section 4, we shift to study RNNs as language recognizers. Finally, in Section 5, we provide empirical results evaluating the relevance of our predictions for unsaturated RNNs.
2 Building Blocks
In this work, we analyze RNNs using formal models from automata theory—in particular, WFAs and counter automata. In this section, we first define the basic notion of an encoder studied in this paper, and then introduce more specialized formal concepts: WFAs, counter machines (CMs), space complexity, and, finally, various RNN architectures.
2.1 Encoders
We view both RNNs and automata as encoders: machines that can be parameterized to compute a set of functions  $f:\Sigma^{*}\to \mathbb{Q}^{k}$ , where  $\boldsymbol{\Sigma}$  is an input alphabet and  $\mathbb{Q}$  is the set of rational reals. Given an encoder  $M$  and parameters  $\theta$ , we use  $M_{\theta}$  to represent the specific function that the parameterized encoder computes. For each encoder, we refer to the set of functions that it can compute as its state expressiveness. For example, a deterministic finite state acceptor (DFA) is an encoder whose parameters are its transition graph. Its state expressiveness is the indicator functions for the regular languages.
2.2 WFAs
Formally, a WFA is a non-deterministic finite automaton where each starting state, transition, and
$^2$ Space complexity measures the number of different configurations an RNN can reach as a function of input length. Formal definition deferred until Section 2.
3This is common, but not the only possibility. For example, an attention decoder observes the full sequence of states.
final state is weighted. Let  $Q$  denote the set of states,  $\Sigma$  the alphabet, and  $\mathbb{Q}$  the rational reals.4
This weighting is specified by three functions:
1. Initial state weights  $\lambda :Q\to \mathbb{Q}$
2. Transition weights  $\tau :Q\times \Sigma \times Q\to \mathbb{Q}$
3. Final state weights  $\rho :Q\to \mathbb{Q}$
The weights are used to encode any string  $x \in \Sigma^{*}$ :
Definition 1 (Path score). Let  $\pi$  be a path of the form  $q_0 \to_{x_1} q_1 \to_{x_2} \dots \to_{x_t} q_t$  through WFA  $A$ .
The score of  $\pi$  is given by
$$
A [ \pi ] = \lambda (q _ {0}) \left(\prod_ {i = 1} ^ {t} \tau (q _ {i - 1}, x _ {i}, q _ {i})\right) \rho (q _ {t}).
$$
By  $\Pi (x)$ , denote the set of paths producing  $x$ .
Definition 2 (String encoding). The encoding computed by a WFA  $A$  on string  $x$  is
$$
A [ x ] = \sum_ {\pi \in \Pi (x)} A [ \pi ].
$$
Hankel matrix Given a function  $f:\Sigma^{*}\to \mathbb{Q}$  and two enumerations  $\alpha ,\omega$  of the strings in  $\Sigma^{*}$  , we define the Hankel matrix of  $f$  as the infinite matrix
$$
[ H _ {f} ] _ {i j} = f \left(\alpha_ {i} \cdot \omega_ {j}\right). \tag {1}
$$
where  $\cdot$  denotes concatenation. It is sometimes convenient to treat  $H_{f}$  as though it is directly indexed by  $\Sigma^{*}$ , e.g.  $[H_f]_{\alpha_i,\omega_j} = f(\alpha_i\cdot \omega_j)$ , or refer to a sub-block of a Hankel matrix, row- and column-indexed by prefixes and suffixes  $P,S\subseteq \Sigma^{*}$ . The following result relates the Hankel matrix to WFAs:
Theorem 1 (Carlyle and Paz, 1971; Fliess, 1974). For any  $f: \Sigma^{*} \to \mathbb{Q}$ , there exists a WFA that computes  $f$  if and only if  $H_{f}$  has finite rank.
Rational series (Sakarovitch, 2009) For all  $k \in \mathbb{N}$ ,  $\mathbf{f} : \Sigma^* \to \mathbb{Q}^k$  is a rational series if there exist WFAs  $A_1, \dots, A_k$  such that, for all  $x \in \Sigma^*$  and  $1 \leq i \leq k$ ,  $A_i[x] = f_i(x)$ .
2.3 Counter Machines
We now turn to introducing a different type of encoder: the real-time counter machine (CM; Merrill, 2020; Fischer, 1966; Fischer et al., 1968). CMs are deterministic finite-state machines augmented with finitely many integer counters. While processing a string, the machine updates these counters, and may use them to inform its behavior.
We view counter machines as encoders mapping  $\Sigma^{*}\to \mathbb{Z}^{k}$ . For  $m\in \mathbb{N},\circ \in \{+, - ,\times \}$ , let  $\circ m$  denote the function  $f(n) = n\circ m$ .
Definition 3 (General CM; Merrill, 2020). A  $k$ -counter CM is a tuple  $\langle \Sigma, Q, q_0, u, \delta \rangle$  with
1. A finite alphabet  $\Sigma$
2. A finite set of states  $Q$ , with initial state  $q_0$
3. A counter update function
$$
u: \Sigma \times Q \times \{0, 1 \} ^ {k} \rightarrow \{\times 0, - 1, + 0, + 1 \} ^ {k}
$$
4. A state transition function
$$
\delta : \Sigma \times Q \times \{0, 1 \} ^ {k} \rightarrow Q
$$
A CM processes input tokens  $\{x_{t}\}_{t = 1}^{n}$  sequentially. Denoting  $\langle q_t,\mathbf{c}_t\rangle \in Q\times \mathbb{Z}^k$  a CM's configuration at time  $t$ , define its next configuration:
$$
q _ {t + 1} = \delta \left(x _ {t}, q _ {t}, \overrightarrow {\mathbb {I}} _ {= 0} \left(\mathbf {c} _ {t}\right)\right) \tag {2}
$$
$$
\mathbf {c} _ {t + 1} = u \left(x _ {t}, q _ {t}, \overline {{\mathbb {I}}} _ {= 0} \left(\mathbf {c} _ {t}\right)\right) \left(\mathbf {c} _ {t}\right), \tag {3}
$$
where  $\vec{\mathbb{I}}_{=0}$  is a broadcasted "zero-check" operation, i.e.,  $\vec{\mathbb{I}}_{=0}(\mathbf{v})_i \triangleq \mathbb{1}_{=0}(v_i)$ . In (2) and (3), note that the machine only views the zerosness of each counter, and not its actual value. A general CM's encoding of a string  $x$  is the value of its counter vector  $\mathbf{c}_t$  after processing all of  $x$ .
Restricted CMs
1. A CM is  $\Sigma$ -restricted iff  $u$  and  $\delta$  depend only on the current input  $\sigma \in \Sigma$ .
2. A CM is  $(\Sigma \times Q)$ -restricted iff  $u$  and  $\delta$  depend only on the current input  $\sigma \in \Sigma$  and the current state  $q \in Q$ .
3. A CM is  $\Sigma^w$ -restricted iff it is  $(\Sigma \times Q)$ -restricted, and the states  $Q$  are windows over the last  $w$  input tokens, e.g.,  $Q = \Sigma^{\leq w}5$ .
These restrictions prevent the machine from being "counter-aware":  $u$  and  $\delta$  cannot condition on the counters' values. As we will see, restricted CMs have natural parallels in the realm of rational RNNs. In Subsection 3.2, we consider the relationship between counter awareness and rational recurrence.
2.4 Space Complexity
As in Merrill (2019), we also analyze encoders in terms of state space complexity, measured in bits.
Definition 4 (Bit complexity). An encoder  $M:\Sigma^{*}\to \mathbb{Q}^{k}$  has  $T(n)$  space iff
$$
\max _ {\theta} \left| \left\{s _ {M _ {\theta}} (x) \mid x \in \Sigma^ {\leq n} \right\} \right| = 2 ^ {T (n)},
$$
4WFAs are often defined over a generic semiring; we consider only the special case when it is the field of rational reals.
5The states  $q\in \Sigma^{<  w}$  represent the beginning of the sequence, before  $w$  input tokens have been seen.
where  $s_{M_{\theta}}(x)$  is a minimal representation of  $M$ 's internal configuration immediately after  $x$ .
We consider three asymptotic space complexity classes:  $\Theta(1)$ ,  $\Theta(\log n)$ , and  $\Theta(n)$ , corresponding to encoders that can reach a constant, polynomial, and exponential (in sequence length) number of configurations respectively. Intuitively, encoders that can dynamically count but cannot use more complex memory like stacks—such as all CMs—are in  $\Theta(\log n)$  space. Encoders that can uniquely encode every input sequence are in  $\Theta(n)$  space.
2.5 Saturated Networks
A saturated neural network is a discrete approximation of neural network considered by Merrill (2019), who calls it an "asymptotic network." Given a parameterized neural encoder  $M_{\theta}(x)$ , we construct the saturated network  $s - M_{\theta}(x)$  by taking
$$
\mathrm {s} - M _ {\theta} (x) = \lim  _ {N \rightarrow \infty} M _ {N \theta} (x) \tag {4}
$$
where  $N\theta$  denotes the parameters  $\theta$  multiplied by a scalar  $N$ . This transforms each "squashing" function (sigmoid, tanh, etc.) to its extreme values (0,  $\pm 1$ ). In line with prior work (Weiss et al., 2018; Merrill, 2019; Suzgun et al., 2019b), we consider saturated networks a reasonable approximation for analyzing practical expressive power. For clarity, we denote the saturated approximation of an architecture by prepending it with s, e.g., s-LSTM.
2.6 RNNs
A recurrent neural network (RNN) is a parameterized update function  $g_{\theta}:\mathbb{Q}^{k}\times \mathbb{Q}^{d_{x}}\to \mathbb{Q}^{k}$ , where  $\theta$  are the rational-valued parameters of the RNN and  $d_{x}$  is the dimension of the input vector.  $g_{\theta}$  takes as input a current state  $\mathbf{h}\in \mathbb{Q}^k$  and input vector  $\mathbf{x}\in \mathbb{Q}^{d_x}$ , and produces the next state. Defining the initial state as  $\mathbf{h}_0 = \mathbf{0}$ , an RNN can be applied to an input sequence  $x\in (\mathbb{Q}^{d_x})^*$  one vector at a time to create a sequence of states  $\{\mathbf{h}_t\}_{t\leq |x|}$ , each representing an encoding of the prefix of  $x$  up to that time step. RNNs can be used to encode sequences over a finite alphabet  $x\in \Sigma^{*}$  by first applying a mapping (embedding)  $e:\Sigma \rightarrow \mathbb{Q}^{d_x}$ .
Multi-layer RNNs "Deep" RNNs are RNNs that have been arranged in  $L$  stacked layers  $R_{1},\ldots ,R_{L}$  . In this setting, the series of output
states  $\mathbf{h}_1, \mathbf{h}_2, \dots, \mathbf{h}_{|x|}$  generated by each RNN on its input is fed as input to the layer above it, and only the first layer receives the original input sequence  $x \in \Sigma^*$  as input.
The recurrent update function  $g$  can take several forms. The original and most simple form is that of the Elman RNN. Since then, more elaborate forms using gating mechanisms have become popular, among them the LSTM, GRU, and QRNN.
Elman RNNs (Elman, 1990) Let  $\mathbf{x}_t$  be a vector embedding of  $x_{t}$ . For brevity, we suppress the bias terms in this (and the following) affine operations.
$$
\mathbf {h} _ {t} = \tanh  \left(\mathbf {W x} _ {t} + \mathbf {U h} _ {t - 1}\right). \tag {5}
$$
We refer to the saturated Elman RNN as the  $s$ -RNN. The s-RNN has  $\Theta(1)$  space (Merrill, 2019).
LSTMs (Hochreiter and Schmidhuber, 1997) An LSTM is a gated RNN with a state vector  $\mathbf{h}_t \in \mathbb{Q}^k$  and memory vector  $\mathbf{c}_t \in \mathbb{Q}^k$ .
$$
\mathbf {f} _ {t} = \sigma \left(\mathbf {W} ^ {f} \mathbf {x} _ {t} + \mathbf {U} ^ {f} \mathbf {h} _ {t - 1}\right) \tag {6}
$$
$$
\mathbf {i} _ {t} = \sigma \left(\mathbf {W} ^ {i} \mathbf {x} _ {t} + \mathbf {U} ^ {i} \mathbf {h} _ {t - 1}\right) \tag {7}
$$
$$
\mathbf {o} _ {t} = \sigma \left(\mathbf {W} ^ {o} \mathbf {x} _ {t} + \mathbf {U} ^ {o} \mathbf {h} _ {t - 1}\right) \tag {8}
$$
$$
\tilde {\mathbf {c}} _ {\mathbf {t}} = \tanh  \left(\mathbf {W} ^ {c} \mathbf {x} _ {t} + \mathbf {U} ^ {c} \mathbf {h} _ {t - 1}\right) \tag {9}
$$
$$
\mathbf {c} _ {t} = \mathbf {f} _ {t} \odot \mathbf {c} _ {t - 1} + \mathbf {i} _ {t} \odot \tilde {\mathbf {c}} _ {\mathbf {t}} \tag {10}
$$
$$
\mathbf {h} _ {t} = \mathbf {o} _ {t} \odot \tanh  \left(\mathbf {c} _ {t}\right). \tag {11}
$$
The LSTM can use its memory vector  $\mathbf{c}_t$  as a register of counters (Weiss et al., 2018). Merrill (2019) showed that the s-LSTM has  $\Theta (\log n)$  space.
GRUs (Cho et al., 2014) Another kind of gated RNN is the GRU.
$$
\mathbf {z} _ {t} = \sigma \left(\mathbf {W} ^ {z} \mathbf {x} _ {t} + \mathbf {U} ^ {z} \mathbf {h} _ {t - 1}\right) \tag {12}
$$
$$
\mathbf {r} _ {t} = \sigma \left(\mathbf {W} ^ {r} \mathbf {x} _ {t} + \mathbf {U} ^ {r} \mathbf {h} _ {t - 1}\right) \tag {13}
$$
$$
\mathbf {u} _ {t} = \tanh  \left(\mathbf {W} ^ {u} \mathbf {x} _ {t} + \mathbf {U} ^ {u} \left(\mathbf {r} _ {t} \odot \mathbf {h} _ {t - 1}\right)\right) \tag {14}
$$
$$
\mathbf {h} _ {t} = \mathbf {z} _ {t} \odot \mathbf {h} _ {t - 1} + (1 - \mathbf {z} _ {t}) \odot \mathbf {u} _ {t}. \tag {15}
$$
Weiss et al. (2018) found that, unlike the LSTM, the GRU cannot use its memory to count dynamically. Merrill (2019) showed the s-GRU has  $\Theta(1)$  space.
7 With respect to our presented definition of RNNs, the concatenation of  $\mathbf{h}_t$  and  $\mathbf{c}_t$  can be seen as the recurrently updated state. However in all discussions of LSTMs we treat only  $\mathbf{h}_t$  as the LSTM's 'state', in line with common practice.
$^6$ I.e., the minimal state representation needed to compute  $M_{\theta}$  correctly. This distinction is important for architectures like attention, for which some implementations may retain unusable information such as input embedding order.
Figure 2: Diagram of the relations between encoders. Neural networks are underlined. We group by asymptotic upper bound  $(O)$ , as opposed to tight  $(\Theta)$ .
QRNNs Bradbury et al. (2016) propose QRNNs as a computationally efficient hybrid of LSTMs and CNNs. Let  $*$  denote convolution over time, let  $\mathbf{W}^z, \mathbf{W}^f, \mathbf{W}^o \in \mathbb{Q}^{d_x \times w \times k}$  be convolutions with window length  $w$ , and let  $\mathbf{X} \in \mathbb{Q}^{n \times d_x}$  denote the matrix of  $n$  input vectors. AnIFO-QRNN (henceforth referred to as a QRNN) with window length  $w$  is defined by  $\mathbf{W}^z, \mathbf{W}^f$ , and  $\mathbf{W}^o$  as follows:
$$
\mathbf {Z} = \tanh  \left(\mathbf {W} ^ {z} * \mathbf {X}\right) \tag {16}
$$
$$
\mathbf {F} = \sigma \left(\mathbf {W} ^ {f} * \mathbf {X}\right) \tag {17}
$$
$$
\mathbf {O} = \sigma \left(\mathbf {W} ^ {o} * \mathbf {X}\right) \tag {18}
$$
$$
\mathbf {c} _ {t} = \mathbf {f} _ {t} \odot \mathbf {c} _ {t - 1} + \mathbf {i} _ {t} \odot \mathbf {z} _ {t} \tag {19}
$$
$$
\mathbf {h} _ {t} = \mathbf {o} _ {t} \odot \mathbf {c} _ {t} \tag {20}
$$
where  $\mathbf{z}_t, \mathbf{f}_t, \mathbf{o}_t$  are respectively rows of  $\mathbf{Z}, \mathbf{F}, \mathbf{O}$ . A QRNN  $Q$  can be seen as an LSTM in which all uses of the state vector  $\mathbf{h}_t$  have been replaced with a computation over the last  $w$  input tokens-in this way it is similar to a CNN.
The s-QRNN has  $\Theta (\log n)$  space, as the analysis of Merrill (2019) for the s-LSTM directly applies. Indeed, any s-QRNN is also a  $(\Sigma^w)$ -restricted CM extended with  $= \pm 1$  ("set to  $\pm 1$ " operations.
3 State Expressiveness
We now turn to presenting our results. In this section, we develop a hierarchy of single-layer RNNs based on their state expressiveness. A set-theoretic view of the hierarchy is shown in Figure 2.
Let  $\mathcal{R}$  be the set of rational series. The hierarchy relates  $\Theta (\log n)$  space to the following sets:
- RR As in Peng et al. (2018), we say that An encoder is rationally recurrent (RR) iff its state expressiveness is a subset of  $\mathcal{R}$ .
- RR-hard An encoder is  $RR$ -hard iff its state expressiveness contains  $\mathcal{R}$ . A Turing machine is RR-hard, as it can simulate any WFA.
- RR-complete Finally, an encoder is  $RR$ -complete iff its state expressiveness is equivalent to  $\mathcal{R}$ . A trivial example of an RR-complete encoder is a vector of  $k$  WFAs.
The different RNNs are divided between the intersections of these classes. In Subsection 3.1, we prove that the s-LSTM, already established to have  $\Theta (\log n)$  space, is not RR. In Subsection 3.2, we demonstrate that encoders with restricted counting ability (e.g., QRNNs) are RR, and in Subsection 3.3, we show the same for all encoders with finite state (CNNs, s-RNNs, and s-GRUs). In Subsection 3.4, we demonstrate that none of these RNNs are RR-hard. In Appendix F, we extend this analysis from RNNs to self attention.
3.1 Counting Beyond RR
We find that encoders like the s-LSTM—which, as discussed in Subsection 2.3, is "aware" of its current counter values—are not RR. To do this, we construct  $f_0: \{a, b\}^* \to \mathbb{N}$  that requires counter awareness to compute on strings of the form  $a^*b^*$ , making it not rational. We then construct an s-LSTM computing  $f_0$  over  $a^*b^*$ .
Let  $\#_{a - b}(x)$  denote the number of  $a$ s in string  $x$  minus the number of  $b$ s.
Definition 5 (Rectified counting).
$$
f _ {0}: x \mapsto \left\{ \begin{array}{l l} \# _ {a - b} (x) & \text {i f} \# _ {a - b} (x) > 0 \\ 0 & \text {o t h e r w i s e .} \end{array} \right.
$$
Lemma 1. For all  $f: \{a, b\}^* \to \mathbb{N}$ , if  $f(a^i b^j) = f_0(a^i b^j)$  for all  $i, j \in \mathbb{N}$ , then  $f \notin \mathcal{R}$ .
Proof. Consider the Hankel sub-block  $\mathbf{A}_n$  of  $H_{f}$  with prefixes  $P_{n} = \{a^{i}\}_{i\leq n}$  and suffixes  $S_{n} = \{b^{j}\}_{j\leq n}$ .  $\mathbf{A}_n$  is lower-triangular:
$$
\left( \begin{array}{c c c c} 0 & 0 & 0 & \dots \\ 1 & 0 & 0 & \dots \\ 2 & 1 & 0 & \dots \\ \vdots & \vdots & \vdots & \ddots \end{array} \right). \tag {21}
$$
Therefore  $\mathrm{rank}(\mathbf{A}_n) = n - 1$ . Thus, for all  $n$ , there is a sub-block of  $H_{f}$  with rank  $n - 1$ , and so  $\mathrm{rank}(H_f)$  is unbounded. It follows from Theorem 1 that there is no WFA computing  $f$ .
Theorem 2. The  $s$ -LSTM is not RR.
Figure 3: A 1-CM computing  $f_{0}$  for  $x \in \{a^{i}b^{j} \mid i, j \in \mathbb{N}\}$ . Let  $\sigma / \pm m$  denote a transition that consumes  $\sigma$  and updates the counter by  $\pm m$ . We write  $\sigma, = 0 / \pm m$  (or  $\neq$ ) for a transition that requires the counter is 0.
Proof. Assume the input has the form  $a^i b^j$  for some  $i, j$ . Consider the following LSTM:
$$
i _ {t} = \sigma \left(1 0 N h _ {t - 1} - 2 N \mathbb {1} _ {= b} \left(x _ {t}\right) + N\right) \tag {22}
$$
$$
\tilde {c} _ {t} = \tanh  \left(N \mathbb {1} _ {= a} (x _ {t}) - N \mathbb {1} _ {= b} (x _ {t})\right) \tag {23}
$$
$$
c _ {t} = c _ {t - 1} + i _ {t} \tilde {c} _ {t} \tag {24}
$$
$$
h _ {t} = \tanh  \left(c _ {t}\right). \tag {25}
$$