SlowGuess/ACL_2020_OCR huggingface dataset:
Let  $N\to \infty$  . Then  $i_t = 0$  iff  $x_{t} = b$  and  $h_{t - 1} = 0$  (i.e.  $c_{t - 1} = 0$  ). Meanwhile,  $\tilde{c}_t = 1$  iff  $x_{t} = a$  . The update term becomes
$$
i _ {t} \tilde {c} _ {t} = \left\{ \begin{array}{l l} 1 & \text {i f} x _ {t} = a \\ - 1 & \text {i f} x _ {t} = b \text {a n d} c _ {t - 1} > 0 \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {26}
$$
For a string  $a^i b^j$ , the update in (26) is equivalent to the CM in Figure 3. Thus, by Lemma 1, the s-LSTM (and the general CM) is not RR.  $\square$
3.2 Rational Counting
While the counter awareness of a general CM enables it to compute non-rational functions, CMs that cannot view their counters are RR.
Theorem 3. Any  $\Sigma$ -restricted CM is RR.
Proof. We show that any function that a  $\Sigma$ -restricted CM can compute can also be computed by a collection of WFAs. The CM update operations  $(-1, +0, +1$ , or  $\times 0)$  can all be reexpressed in terms of functions  $\mathbf{r}(x), \mathbf{u}(x): \Sigma^{*} \to \mathbb{Z}^{k}$  to get:
$$
\mathbf {c} _ {t} = \mathbf {r} \left(x _ {t}\right) \mathbf {c} _ {t - 1} + \mathbf {u} \left(x _ {t}\right) \tag {27}
$$
$$
\mathbf {c} _ {t} = \sum_ {i = 1} ^ {t} \left(\prod_ {j = i + 1} ^ {t} \mathbf {r} \left(x _ {j}\right)\right) \mathbf {u} \left(x _ {i}\right). \tag {28}
$$
A WFA computing  $[\mathbf{c}_t]_i$  is shown in Figure 4.
The WFA in Figure 4 also underlies unigram rational RNNs (Peng et al., 2018). Thus,  $\Sigma$ -restricted CMs are actually a special case of unigram WFAs. In Appendix A, we show the more general result:
Theorem 4. Any  $(\Sigma \times Q)$ -restricted CM is RR.
In many rational RNNs, the updates at different time steps are independent of each other outside of a window of  $w$  tokens. Theorem 4 tells us this independence is not an essential property of rational encoders. Rather, any CM where the update is conditioned by finite state (as opposed to being conditioned by a local window) is in fact RR.
Furthermore, since  $(\Sigma^w)$ -restricted CMs are a special case of  $(\Sigma \times Q)$ -restricted CMs, Theorem 4 can be directly applied to show that the s-QRNN is RR. See Appendix A for further discussion of this.
3.3 Finite-Space RR
Theorem 4 motivates us to also think about finite-space encoders: i.e., encoders with no counters" where the output at each prefix is fully determined by a finite amount of memory. The following lemma implies that any finite-space encoder is RR:
Lemma 2. Any function  $f: \Sigma^{*} \to \mathbb{Q}$  computable by a  $\Theta(1)$ -space encoder is a rational series.
Proof. Since  $f$  is computable in  $\Theta(1)$  space, there exists a DFA  $A_f$  whose accepting states are isomorphic to the range of  $f$ . We convert  $A_f$  to a WFA by labelling each accepting state by the value of  $f$  that it corresponds to. We set the starting weight of the initial state to 1, and 0 for every other state. We assign each transition weight 1.
Since the CNN, s-RNN, and s-GRU have finite state, we obtain the following result:
Theorem 5. The CNN,  $s$ -RNN, and  $s$ -GRU are RR.
While Schwartz et al. (2018) and Peng et al. (2018) showed the CNN to be RR over the max-plus semiring, Theorem 5 shows the same holds for  $\langle \mathbb{Q},\cdot , + \rangle$
3.4 RR Completeness
While "rational recurrence" is often used to indicate the simplicity of an RNN architecture, we find in this section that WFAs are surprisingly computationally powerful. Figure 5 shows a WFA mapping binary string to their numeric value, proving WFAs have  $\Theta(n)$  space. We now show that none of our RNNs are able to simulate an arbitrary WFA, even in the unsaturated form.
In which  $f_{t}$  and  $o_t$  are set to 1, such that  $c_{t} = c_{t - 1} + i_{t}\tilde{c}_{t}$
Figure 4: WFA simulating unit  $i$  of a  $\Sigma$ -restricted CM. Let  $\forall \sigma / w(\sigma)$  denote a set of transitions consuming each token  $\sigma$  with weight  $w(\sigma)$ . We use standard DFA notation to show initial weights  $\lambda(q_0) = 1, \lambda(q_1) = 0$  and accepting weights  $\rho(q_0) = 0, \rho(q_1) = 1$ .
Figure 5: A WFA mapping binary strings to their numeric value. This can be extended for any base  $>2$ . Cortes and Mohri (2000) present a similar construction. Notation is the same as Figure 4.
Theorem 6. Both the saturated and unsaturated RNN, GRU, QRNN, and LSTM are not RR-hard.
Proof. Consider the function  $f_{b}$  mapping binary strings to their value, e.g.  $101 \mapsto 5$ . The WFA in Figure 5 shows that this function is rational.
The value of  $f_{b}$  grows exponentially with the sequence length. On the other hand, the value of the RNN and GRU cell is bounded by 1, and QRNN and LSTM cells can only grow linearly in time. Therefore, these encoders cannot compute  $f_{b}$ .
In contrast, memory networks can have  $\Theta (n)$  space. Appendix G explores this for stack RNNs.
3.5 Towards Transformers
Appendix F presents preliminary results extending saturation analysis to self attention. We show saturated self attention is not RR and consider its space complexity. We hope further work will more completely characterize saturated self attention.
4 Language Expressiveness
Having explored the set of functions expressible internally by different saturated RNN encoders, we turn to the languages recognizable when using them with a decoder. We consider the following setup:
1. An s-RNN encodes  $x$  to a vector  $\mathbf{h}_t\in \mathbb{Q}^k$
2. A decoder function maps the last state  $\mathbf{h}_t$  to an accept/reject decision, respectively:  $\{1,0\}$ .
We say that a language  $L$  is decided by an encoder-decoder pair  $\mathbf{e}$ ,  $\mathbf{d}$  if  $\mathbf{d}(\mathbf{e}(x)) = 1$  for every sequence  $x \in L$  and otherwise  $\mathbf{d}(\mathbf{e}(x)) = 0$ . We explore which languages can be decided by different encoder-decoder pairings.
Some related results can be found in Cortes and Mohri (2000), who study the expressive power of WFAs in relation to CFGs under a slightly different definition of language recognition.
4.1 Linear Decoders
Let  $\mathbf{d}_1$  be the single-layer linear decoder
$$
\mathbf {d} _ {1} \left(\mathbf {h} _ {t}\right) \triangleq \mathbb {1} _ {> 0} \left(\mathbf {w} \cdot \mathbf {h} _ {t} + b\right) \in \{0, 1 \} \tag {29}
$$
parameterized by  $\mathbf{w}$  and  $b$ . For an encoder architecture  $E$ , we denote by  $D_{1}(E)$  the set of languages decidable by  $E$  with  $\mathbf{d}_1$ . We use  $D_{2}(E)$  analogously for a 2-layer decoder with  $\mathbb{1}_{>0}$  activations, where the first layer has arbitrary width.
4.2 A Decoder Adds Power
We refer to sets of strings using regular expressions, e.g.  $a^* = \{a^i \mid i \in \mathbb{N}\}$ . To illustrate the purpose of the decoder, consider the following language:
$$
L _ {\leq} = \{x \in \{a, b \} ^ {*} \mid \# _ {a - b} (x) \leq 0 \}. \tag {30}
$$
The Hankel sub-block of the indicator function for  $L_{\leq}$  over  $P = a^{*}$ ,  $S = b^{*}$  is lower triangular. Therefore, no RR encoder can compute it.
However, adding the  $D_{1}$  decoder allows us to compute this indicator function with an s-QRNN, which is RR. We set the s-QRNN layer to compute the simple series  $c_{t} = \#_{a - b}(x)$  (by increasing on  $a$  and decreasing on  $b$ ). The  $D_{1}$  layer then checks  $c_{t} \leq 0$ . So, while the indicator function for  $L_{\leq}$  is not itself rational, it can be easily recovered from a rational representation. Thus,  $L_{\leq} \in D_{1}(\mathrm{s - QRNN})$ .
4.3 Case Study:  $a^n b^n$
We compare the language expressiveness of several rational and non-rational RNNs on the following:
$$
a ^ {n} b ^ {n} \triangleq \left\{a ^ {n} b ^ {n} \mid n \in \mathbb {N} \right\} \tag {31}
$$
$$
a ^ {n} b ^ {n} \Sigma^ {*} \triangleq \left\{a ^ {n} b ^ {n} (a | b) ^ {*} \mid 0 <   n \right\}. \tag {32}
$$
$a^n b^n$  is more interesting than  $L_{\leq}$  because the  $D_{1}$  decoder cannot decide it simply by asking the encoder to track  $\#_{a - b}(x)$ , as that would require it to compute the non-linearly separable  $= 0$  function. Thus, it appears at first that deciding  $a^{n}b^{n}$  with  $D_{1}$
As well as CMs.
might require a non-rational RNN encoder. However, we show below that this is not the case.
Let  $\circ$  denote stacking two layers. We will go on to discuss the following results:
$$
a ^ {n} b ^ {n} \in D _ {1} (\mathrm {W F A}) \tag {33}
$$
$$
a ^ {n} b ^ {n} \in D _ {1} (\mathrm {s - L S T M}) \tag {34}
$$
$$
a ^ {n} b ^ {n} \notin D _ {1} (\mathrm {s} - \mathrm {Q R N N}) \tag {35}
$$
$$
a ^ {n} b ^ {n} \in D _ {1} (\mathrm {s} - \mathrm {Q R N N} \circ \mathrm {s} - \mathrm {Q R N N}) \tag {36}
$$
$$
a ^ {n} b ^ {n} \in D _ {2} (\mathrm {s} - \mathrm {Q R N N}) \tag {37}
$$
$$
a ^ {n} b ^ {n} \Sigma^ {*} \in D _ {1} (\mathrm {s - L S T M}) \tag {38}
$$
$$
a ^ {n} b ^ {n} \Sigma^ {*} \notin D (\mathrm {s} - \text {Q R N N}) \text {f o r a n y} D \tag {39}
$$
$$
a ^ {n} b ^ {n} \Sigma^ {*} \cup \{\epsilon \} \in D _ {1} (\mathrm {s} - \mathrm {Q R N N} \circ \mathrm {s} - \mathrm {Q R N N}) \tag {40}
$$
WFAs (Appendix B) In Theorem 8 we present a function  $f: \Sigma^{*} \to \mathbb{Q}$  satisfying  $f(x) > 0$  iff  $x \in a^{n}b^{n}$ , and show that  $H_{f}$  has finite rank. It follows that there exists a WFA that can decide  $a^{n}b^{n}$  with the  $D_{1}$  decoder. Counterintuitively,  $a^{n}b^{n}$  can be recognized using rational encoders.
QRNNs (Appendix C) Although  $a^n b^n \in D_1(\mathrm{WFA})$ , it does not follow that every rationally recurrent model can also decide  $a^n b^n$  with the help of  $D_1$ . Indeed, in Theorem 9, we prove that  $a^n b^n \notin D_1(\mathrm{s - QRNN})$ , whereas  $a^n b^n \in D_1(\mathrm{s - LSTM})$  (Theorem 13).
It is important to note that, with a more complex decoder, the QRNN could recognize  $a^n b^n$ . For example, the s-QRNN can encode  $c_{1} = \#_{a - b}(x)$  and set  $c_{2}$  to check whether  $x$  contains  $ba$ , from which a  $D_{2}$  decoder can recognize  $a^n b^n$  (Theorem 10).
This does not mean the hierarchy dissolves as the decoder is strengthened. We show that  $a^n b^n \Sigma^*$  — which seems like a trivial extension of  $a^n b^n$  — is not recognizable by the s-QRNN with any decoder.
This result may appear counterintuitive, but in fact highlights the s-QRNN's lack of counter awareness: it can only passively encode the information needed by the decoder to recognize  $a^{n}b^{n}$ . Failing to recognize that a valid prefix has been matched, it cannot act to preserve that information after additional input tokens are seen. We present a proof in Theorem 11. In contrast, in Theorem 14 we show that the s-LSTM can directly encode an indicator for  $a^{n}b^{n}\Sigma^{*}$  in its internal state.
Proof sketch:  $a^n b^n \Sigma^* \notin D(\text{s - QRNN})$ . A sequence  $s_1 \in a^n b^n \Sigma^*$  is shuffled to create  $s_2 \notin a^n b^n \Sigma^*$  with an identical multi-set of counter up
dates. $^{10}$  Counter updates would be order agnostic if not for reset operations, and resets mask all history, so extending  $s_1$  and  $s_2$  with a single suffix  $s$  containing all of their  $w$ -grams reaches the same final state. Then for any  $D$ ,  $D(\mathrm{s - QRNN})$  cannot separate them. We formalize this in Theorem 11.
We refer to this technique as the suffix attack, and note that it can be used to prove for multiple other languages  $L \in D_{2}(\mathrm{s - QRNN})$  that  $L \cdot \Sigma^{*}$  is not in  $D(\mathrm{s - QRNN})$  for any decoder  $D$ .
2-layer QRNNs Adding another layer overcomes the weakness of the 1-layer s-QRNN, at least for deciding  $a^n b^n$ . This follows from the fact that  $a^n b^n \in D_2(\mathrm{s - QRNN})$ : the second QRNN layer can be used as a linear layer.
Similarly, we show in Theorem 10 that a 2-layer s-QRNN can recognize  $a^n b^n \Sigma^* \cup \{\epsilon\}$ . This suggests that adding a second s-QRNN layer compensates for some of the weakness of the 1-layer s-QRNN, which, by the same argument for  $a^n b^n \Sigma^*$  cannot recognize  $a^n b^n \Sigma^* \cup \{\epsilon\}$  with any decoder.
4.4 Arbitrary Decoder
Finally, we study the theoretical case where the decoder is an arbitrary recursively enumerable (RE) function. We view this as a loose upper bound of stacking many layers after a rational encoder. What information is inherently lost by using a rational encoder? WFAs can uniquely encode each input, making them Turing-complete under this setup; however, this does not hold for rational s-RNNs.
RR-complete Assuming an RR-complete encoder, a WFA like Figure 5 can be used to encode each possible input sequence over  $\Sigma$  to a unique number. We then use the decoder as an oracle to decide any RE language. Thus, an RR-complete encoder with an RE decoder is Turing-complete.
Bounded space However, the  $\Theta (\log n)$  space bound of saturated rational RNNs like the s-QRNN means these models cannot fully encode the input. In other words, some information about the prefix  $x_{:t}$  must be lost in  $\mathbf{c}_t$ . Thus, rational s-RNNs are not Turing-complete with an RE decoder.
In Subsection 4.3, we showed that different saturated RNNs vary in their ability to recognize  $a^{n}b^{n}$  and  $a^{n}b^{n}\Sigma^{*}$ . We now test empirically whether
<sup>10</sup>Since QRNN counter updates depend only on the  $w$ -grams present in the sequence.
Figure 6: Accuracy recognizing  $L_{5}$  and  $a^{n}b^{n}\Sigma^{*}$ . "QRNN+" is a QRNN with a 2-layer decoder, and "2QRNN" is a 2-layer QRNN with a 1-layer decoder.
these predictions carry over to the learnable capacity of unsaturated RNNs.11 We compare the QRNN and LSTM when coupled with a linear decoder  $D_{1}$  We also train a 2-layer QRNN ("QRNN2") and a 1-layer QRNN with a  $D_{2}$  decoder ("QRNN+").
We train on strings of length 64, and evaluate generalization on longer strings. We also compare to a baseline that always predicts the majority class. The results are shown in Figure 6. We provide further experimental details in Appendix E.
Experiment 1 We use the following language, which has similar formal properties to  $a^n b^n$ , but with a more balanced label distribution:
$$
L _ {5} = \left\{x \in (a | b) ^ {*} \mid \left| \# _ {a - b} (x) \right| <   5 \right\}. \tag {41}
$$
In line with (34), the LSTM decides  $L_{5}$  perfectly for  $n \leq 64$ , and generalizes fairly well to longer strings. As predicted in (35), the QRNN cannot fully learn  $L_{5}$  even for  $n = 64$ . Finally, as predicted in (36) and (37), the 2-layer QRNN and the QRNN with  $D_{2}$  do learn  $L_{5}$ . However, we see that they do not generalize as well as the LSTM for longer strings. We hypothesize that these multi-
layer models require more epochs to reach the same generalization performance as the LSTM. $^{12}$
Experiment 2 We also consider  $a^n b^n \Sigma^*$ . As predicted in (38) and (40), the LSTM and 2-layer QRNN decide  $a^n b^n \Sigma^*$  flawlessly for  $n = 64$ . A 1-layer QRNN performs at the majority baseline for all  $n$  with both a 1 and 2-layer decoder. Both of these failures were predicted in (39). Thus, the only models that learned  $a^n b^n \Sigma^*$  were exactly those predicted by the saturated theory.
We develop a hierarchy of saturated RNN encoders, considering two angles: space complexity and rational recurrence. Based on the hierarchy, we formally distinguish the state expressiveness of the non-rational s-LSTM and its rational counterpart, the s-QRNN. We show further distinctions in state expressiveness based on encoder space complexity.
Moreover, the hierarchy translates to differences in language recognition capabilities. Strengthening the decoder alleviates some, but not all, of these differences. We present two languages, both recognizable by an LSTM. We show that one can be recognized by an s-QRNN only with the help of a decoder, and that the other cannot be recognized by an s-QRNN with the help of any decoder.
While this means existing rational RNNs are fundamentally limited compared to LSTMs, we find that it is not necessarily being rationally recurrent that limits them: in fact, we prove that a WFA can perfectly encode its input—something no saturated RNN can do. We conclude with an analysis that shows that an RNN architecture's strength must also take into account its space complexity. These results further our understanding of the inner working of NLP systems. We hope they will guide the development of more expressive rational RNNs.
We appreciate Amir Yehudayoff's help in finding the WFA used in Theorem 8, and the feedback of researchers at the Allen Institute for AI, our anonymous reviewers, and Tobias Jaroslaw. The project was supported in part by NSF grant IIS-1562364, Israel Science Foundation grant no.1319/16, and the European Research Council under the EU's Horizon 2020 research and innovation program, grant agreement No. 802774 (iEXTRACT).
<sup>12</sup>As shown by the baseline, generalization is challenging because positive labels become less likely as strings get longer.
<sup>11</sup>https://github.com/viking-sudo-rm/rr-experiments
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization.
Borja Balle, Xavier Carreras, Franco M. Luque, and Ariadna Quattoni. 2014. Spectral learning of weighted automata. Machine Learning, 96(1):33-63.
James Bradbury, Stephen Merity, Caiming Xiong, and Richard Socher. 2016. Quasi-recurrent neural networks.
J. W. Carlyle and A. Paz. 1971. Realizations by stochastic finite automata. J. Comput. Syst. Sci., 5(1):26-40.
Yining Chen, Sorcha Gilroy, Andreas Maletti, Jonathan May, and Kevin Knight. 2018. Recurrent neural networks as weighted language recognizers. In Proc. of NAACL, pages 2261-2271.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gul- cehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proc. of EMNLP, pages 1724-1734.
Corinna Cortes and Mehryar Mohri. 2000. Context-free recognition with weighted automata. Grammars, 3(2/3):133-150.
Jesse Dodge, Roy Schwartz, Hao Peng, and Noah A. Smith. 2019. RNN architecture learning with sparse regularization. In Proc. of EMNLP, pages 1179-1184.
Jeffrey L Elman. 1990. Finding structure in time. Cognitive Science, 14(2):179-211.
Patrick C Fischer. 1966. Turing machines with restricted memory access. Information and Control, 9(4):364-379.
Patrick C. Fischer, Albert R. Meyer, and Arnold L. Rosenberg. 1968. Counter machines and counter languages. Mathematical Systems Theory, 2(3):265-283.
Michel Fliess. 1974. Matrices de Hankel. J. Math. Pures Appl, 53(9):197-222.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. Proceedings of Workshop for NLP Open Source Software (NLP-OSS).
Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics, 8:156-171.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Computation, 9(8):1735-1780.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proc. of EMNLP, pages 1746-1751.
Yann Lecun and Yoshua Bengio. 1995. The Handbook of Brain Theory and Neural Networks, chapter "Convolutional Networks for Images, Speech, and Time Series". MIT Press.
William Merrill. 2019. Sequential neural networks as automata. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges, pages 1-13.
William Merrill. 2020. On the linguistic capacity of real-time counter automata.
Hao Peng, Roy Schwartz, Sam Thomson, and Noah A. Smith. 2018. Rational recurrences. In Proc. of EMNLP, pages 1203-1214.
Jacques Sakarovitch. 2009. Rational and recognisable power series. In Handbook of Weighted Automata, pages 105-174. Springer.
Roy Schwartz, Sam Thomson, and Noah A. Smith. 2018. Bridging CNNs, RNNs, and weighted finite-state machines. In Proc. of ACL, pages 295-305.
Hava T. Siegelmann and Eduardo D. Sontag. 1992. On the computational power of neural nets. In Proc. of COLT, pages 440-449.
Hava T. Siegelmann and Eduardo D. Sontag. 1994. Analog computation via neural networks. Theoretical Computer Science, 131(2):331-360.
Mirac Suzgun, Yonatan Belinkov, Stuart Shieber, and Sebastian Gehrmann. 2019a. LSTM networks can perform dynamic counting. In Proceedings of the Workshop on Deep Learning and Formal Languages: Building Bridges, pages 44-54.
Mirac Suzgun, Sebastian Gehrmann, Yonatan Belinkov, and Stuart M. Shieber. 2019b. Memory-augmented recurrent neural networks can learn generalized Dyck languages.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998-6008.
Gail Weiss, Yoav Goldberg, and Eran Yahav. 2018. On the practical computational power of finite precision RNNs for language recognition.
A Rational Counting
We extend the result in Theorem 3 as follows.
Theorem 7. Any  $(\Sigma \times Q)$ -restricted CM is rationally recurrent.
Proof. We present an algorithm to construct a WFA computing an arbitrary counter in a  $(\Sigma \times Q)$ -restricted CM. First, we create two independent copies of the transition graph for the restricted CM. We refer to one copy of the CM graph as the add graph, and the other as the multiply graph.
The initial state in the add graph receives a starting weight of 1, and every other state receives a starting weight of 0. Each state in the add graph receives an accepting weight of 0, and each state in the multiply graph receives an accepting weight of 1. In the add graph, each transition receives a weight of 1. In the multiply graph, each transition receives a weight of 0 if it represents  $\times 0$ , and 1 otherwise. Finally, for each non-multiplicative update  $\sigma / + m^{13}$  from  $q_{i}$  to  $q_{j}$  in the original CM, we add a WFA transition  $\sigma /m$  from  $q_{i}$  in the add graph to  $q_{j}$  in the multiply graph.
Each counter update creates one path ending in the multiply graph. The path score is set to 0 if that counter update is "erased" by a  $\times 0$  operation. Thus, the sum of all the path scores in the WFA equals the value of the counter.
This construction can be extended to accommodate  $= m$  counter updates from  $q_{i}$  to  $q_{j}$  by adding an additional transition from the initial state to  $q_{j}$  in the multiplication graph with weight  $m$ . This allows us to apply it directly to s-QRNNs, whose update operations include  $= 1$  and  $-1$ .
B WFAs
We show that while WFAs cannot directly encode an indicator for the language  $a^n b^n = \{a^n b^n\mid |n\in \mathbb{N}\}$ , they can encode a function that can be thresholded to recognize  $a^n b^n$ , i.e.:
Theorem 8. The language  $a^n b^n = \{a^n b^n \mid n \in \mathbb{N}\}$  over  $\Sigma = \{a, b\}$  is in  $D_1(\mathrm{WFA})$ .
We prove this by showing a function whose Hankel matrix has finite rank that, when combined with the identity transformation (i.e.,  $w = 1$ ,  $b = 0$ ) followed by thresholding, is an indicator for  $a^n b^n$ . Using the shorthand  $\sigma(x) = \#_{\sigma}(x)$ , the function
is:
$$
f (w) = \left\{ \begin{array}{l l} 0. 5 - 2 (a (x) - b (x)) ^ {2} & \text {i f} x \in a ^ {*} b ^ {*} \\ - 0. 5 & \text {o t h e r w i s e .} \end{array} \right. \tag {42}
$$
Immediately  $f$  satisfies  $\mathbb{1}_{>0}(f(x))\iff x\in a^{n}b^{n}$ . To prove that its Hankel matrix,  $H_{f}$ , has finite rank, we will create 3 infinite matrices of ranks 3, 3 and 1, which sum to  $H_{f}$ . The majority of the proof will focus on the rank of the rank 3 matrices, which have similar compositions.
We now show 3 series  $r, s, t$  and a set of series they can be combined to create. These series will be used to create the base vectors for the rank 3 matrices.
$$
a _ {i} = \frac {i (i + 1)}{2} \tag {43}
$$
$$
b _ {i} = i ^ {2} - 1 \tag {44}
$$
$$
r _ {i} = \operatorname {f i x} _ {0} (i, a _ {i - 2}) \tag {45}
$$
$$
s _ {i} = \operatorname {f i x} _ {1} (i, - b _ {i - 1}) \tag {46}
$$
$$
t _ {i} = \operatorname {f i x} _ {2} (i, a _ {i - 1}) \tag {47}
$$
where for every  $j\leq 2$