SlowGuess/ACL_2020_OCR huggingface dataset:
$$
\operatorname {f i x} _ {j} (i, x) = \left\{ \begin{array}{l l} x & \text {i f} i > 2 \\ 1 & \text {i f} i = j \\ 0 & \text {o t h e r w i s e .} \end{array} \right. \tag {48}
$$
Lemma 3. Let  $c_{i} = 1 - 2i^{2}$  and  $\{c^{(k)}\}_{k\in \mathbb{N}}$  be the set of series defined  $c_{i}^{(k)} = c_{|i - k|}$ . Then for every  $i, k\in \mathbb{N}$ ,
$$
c _ {i} ^ {(k)} = c _ {0} ^ {(k)} r _ {i} + c _ {1} ^ {(k)} s _ {i} + c _ {2} ^ {(k)} t _ {i}.
$$
Proof. For  $i \in \{0,1,2\}$ ,  $r_i, s_i$  and  $t_i$  collapse to a 'select' operation, giving the true statement  $c_i^{(k)} = c_i^{(k)} \cdot 1$ . We now consider the case  $i > 2$ . Substituting the series definitions in the right side of the equation gives
$$
c _ {k} a _ {i - 2} + c _ {| k - 1 |} (- b _ {i - 1}) + c _ {k - 2} a _ {i - 1} \tag {49}
$$
which can be expanded to
$$
(1 - 2 k ^ {2}) \quad \cdot \frac {i ^ {2} - 3 i + 2}{2} \quad +
$$
$$
\left(1 - 2 (k - 1) ^ {2}\right) \quad \cdot \left(1 - (i - 1) ^ {2}\right) \quad +
$$
$$
(1 - 2 (k - 2) ^ {2}) \quad \cdot \frac {(i - 1) i}{2}.
$$
$^{13}$ Note that  $m = -1$  for the  $-1$  counter update.
Reordering the first component and partially opening the other two gives
$$
\begin{array}{l} (- 2 k ^ {2} + 1) \frac {i ^ {2} - 3 i + 2}{2} + \\ (- 2 k ^ {2} + 4 k - 1) (2 i - i ^ {2}) + \\ (- k ^ {2} + 4 k - 3. 5) (i ^ {2} - i) \\ \end{array}
$$
and a further expansion gives
$$
\begin{array}{l} - k ^ {2} i ^ {2} + \quad 0. 5 i ^ {2} + 3 k ^ {2} i - 1. 5 i - 2 k ^ {2} + 1 + \\ 2 k ^ {2} i ^ {2} - 4 k i ^ {2} + i ^ {2} - 4 k ^ {2} i + 8 k i - 2 i + \\ - k ^ {2} i ^ {2} + 4 k i ^ {2} - 3. 5 i ^ {2} + k ^ {2} i - 4 k i + 3. 5 i \\ \end{array}
$$
$$
- 2 i ^ {2} + 4 k i - 2 k ^ {2} + 1 = 1 - 2 (k - i) ^ {2} = c _ {i} ^ {(k)}.
$$
We restate this as:
Corollary 1. For every  $k \in \mathbb{N}$ , the series  $c^{(k)}$  is a linear combination of the series  $r, s$  and  $t$ .
We can now show that  $f$  is computable by a WFA, proving Theorem 8. By Theorem 1, it is sufficient to show that  $H_{f}$  has finite rank.
Lemma 4.  $H_{f}$  has finite rank.
Proof. For every  $P, S \subseteq \{a, b\}^*$ , denote
$$
[ H _ {f} | _ {P, S} ] _ {u, v} = \left\{ \begin{array}{l l} [ H _ {f} ] _ {u, v} & \text {i f} u \in P \text {a n d} v \in S \\ 0 & \text {o t h e r w i s e} \end{array} \right.
$$
Using regular expressions to describe  $P, S$ , we create the 3 finite rank matrices which sum to  $H_{f}$ :
$$
A = \left. \left(H _ {f} + 0. 5\right) \right| _ {a ^ {*}, a ^ {*} b ^ {*}} \tag {50}
$$
$$
B = \left(H _ {f} + 0. 5\right) | _ {a ^ {*} b ^ {+}, b ^ {*}} \tag {51}
$$
$$
C = (- 0. 5) | _ {u, v}. \tag {52}
$$
Intuitively, these may be seen as a "split" of  $H_{f}$  into sections as in Figure 7, such that  $A$  and  $B$  together cover the sections of  $H_{f}$  on which  $u\cdot v$  does not contain the substring  $ba$  (and are equal on them to  $H_{f} + 0.5$ ), and  $C$  is simply the constant matrix  $-0.5$ . Immediately,  $H_{f} = A + B + C$ , and  $\mathrm{rank}(C) = 1$ .
We now consider  $A$ . Denote  $P_A = a^*, S_A = a^* b^*$ .  $A$  is non-zero only on indices  $u \in P_A, v \in S_A$ , and for these,  $u \cdot v \in a^* b^*$  and  $A_{u,v} = 0.5 + f(u \cdot v) = 1 - 2(a(u) + a(v) - b(v))^2$ . This gives that for every  $u \in P_A, v \in S_A$ ,
$$
A _ {u, v} = c _ {| a (u) - (b (v) - a (v)) |} = c _ {b (v) - a (v)} ^ {(a (u))}. \tag {53}
$$
Figure 7: Intuition of the supports of  $A, B$  and  $C$ .
For each  $\tau \in \{r,s,t\}$ , define  $\tilde{\tau} \in \mathbb{Q}^{\{a,b\}^*}$  as
$$
\tilde {\tau} _ {v} = \mathbb {1} _ {v \in a ^ {*} b ^ {*}} \cdot \tau_ {b (v) - a (v)}. \tag {54}
$$
We get from Corollary 1 that for every  $u \in a^*$ , the  $u$ th row of  $A$  is a linear combination of  $\tilde{r}, \tilde{s}$ , and  $\tilde{t}$ . The remaining rows of  $A$  are all  $\mathbf{0}$  and so also a linear combination of these, and so  $\mathrm{rank}(A) \leq 3$ .
Similarly, we find that the nonzero entries of  $B$  satisfy
$$
B _ {u, v} = c _ {| b (v) - (a (u) - b (u)) |} = c _ {a (u) - b (u)} ^ {(b (v))} \tag {55}
$$
and so, for  $\tau \in \{r,s,t\}$ , the columns of  $B$  are linear combinations of the columns  $\tau' \in \mathbb{Q}^{\{a,b\}^*}$  defined
$$
\tau_ {u} ^ {\prime} = \mathbb {1} _ {u \in a ^ {*} b ^ {+}} \cdot \tau_ {a (u) - b (u)}. \tag {56}
$$
Thus we conclude  $\mathrm{rank}(B) \leq 3$ .
Finally,  $H_{f} = A + B + C$ , and so by the subadditivity of rank in matrices,
$$
\operatorname {r a n k} \left(H _ {f}\right) \leq \sum_ {M = A, B, C} \operatorname {r a n k} (M) = 7. \tag {57}
$$
In addition, the rank of  $\tilde{H}_f\in \mathbb{Q}^{\{a,b\} ^{\leq 2},\{a,b\}^{\leq 2}}$  defined  $[\tilde{H}_f]_{u,v} = [H_f]_{u,v}$  is 7, and so we can conclude that the bound in the proof is tight, i.e.,  $\mathrm{rank}(H_f) = 7$  . From here  $\tilde{H}_f$  is a complete subblock of  $H_{f}$  and can be used to explicitly construct a WFA for  $f$  , using the spectral method described by Balle et al. (2014).
C s-QRNNs
Theorem 9. No  $s$ -QRNN with a linear threshold decoder can recognize  $a^n b^n = \{a^n b^n \mid n \in \mathbb{N}\}$ , i.e.,  $a^n b^n \notin D_1(s$ -QRNN).
Proof. An ifo s-QRNN can be expressed as a  $\Sigma^k$ -restricted CM with the additional update operations  $\{:= -1, := 1\}$ , where  $k$  is the window size of the QRNN. So it is sufficient to show that such a machine, when coupled with the decoder  $D_1$  (linear translation followed by thresholding), cannot recognize  $a^n b^n$ .
Let  $\mathcal{A}$  be some such CM, with window size  $k$  and  $h$  counters. Take  $n = k + 10$  and for every  $m\in \mathbb{N}$  denote  $w_{m} = a^{n}b^{m}$  and the counter values of  $\mathcal{A}$  after  $w_{m}$  as  $c^m\in \mathbb{Q}^h$ . Denote by  $u_{t}$  the vector of counter update operations made by this machine on input sequence  $w_{m}$  at time  $t\leq n + m$ . As  $\mathcal{A}$  is dependent only on the last  $k$  counters, necessarily all  $u_{k + i}$  are identical for every  $i\geq 1$ .
It follows that for all counters in the machine that go through an assignment (i.e., :=) operation in  $u_{k+1}$ , their values in  $c^{k+i}$  are identical for every  $i \geq 1$ , and for every other counter  $j$ ,  $c_j^{k+i} - c_j^k = i \cdot \delta$  for some  $\delta \in \mathbb{Z}$ . Formally: for every  $i \geq 1$  there are two sets  $I, J = [h] \setminus I$  and constant vectors  $\mathbf{u} \in \mathbb{N}^I, \mathbf{v} \in \mathbb{N}^J$  s.t.  $c^{k+i}|_I = \mathbf{u}$  and  $|c^{k+i} - c^k|_J = \mathbf{i} \cdot v$ .
We now consider the linear thresholder, defined by weights and bias  $\mathbf{w}, b$ . In order to recognise  $a^n b^n$ , the thresholder must satisfy:
$$
\mathbf {w} \cdot c ^ {k + 9} + b <   0 \tag {58}
$$
$$
\mathbf {w} \cdot c ^ {k + 1 0} + b \quad > 0 \tag {59}
$$
$$
\mathbf {w} \cdot c ^ {k + 1 1} + b \quad <   0 \tag {60}
$$
Opening these equations gives:
$$
\left. \mathbf {w} \right| _ {J} \left(\cdot c ^ {k} \mid_ {J} + 9 \mathbf {v} \mid_ {J}\right) \quad + \left. \mathbf {w} \right| _ {I} \cdot \mathbf {u} \quad <   0 \tag {61}
$$
$$
\left. \mathbf {w} \right| _ {J} \left(\cdot c ^ {k} \mid_ {J} + 1 0 \mathbf {v} \mid_ {J}\right) + \left. \mathbf {w} \right| _ {I} \cdot \mathbf {u} > 0 \tag {62}
$$
$$
\left. \mathbf {w} \right| _ {J} \left(\cdot c ^ {k} \mid_ {J} + 1 1 \mathbf {v} \mid_ {J}\right) + \left. \mathbf {w} \right| _ {I} \cdot \mathbf {u} <   0 \tag {63}
$$
but this gives  $9w|_{J} \cdot \mathbf{v}|_{J} < 10w|_{J} \cdot \mathbf{v}|_{J} > 11w|_{J} \cdot \mathbf{v}|_{J}$ , which is impossible.
However, this does not mean that the s-QRNN is entirely incapable of recognising  $a^n b^n$ . Increasing the decoder power allows it to recognise  $a^n b^n$  quite simply:
Theorem 10. For the two-layer decoder  $D_{2}$ ,  $a^{n}b^{n}\in D_{2}(\mathrm{s - QRNN})$ .
Proof. Let  $\#_{ba}(x)$  denote the number of  $ba$  2-grams in  $x$ . We use s-QRNN with window size
2 to maintain two counters:
$$
[ \mathbf {c} _ {t} ] _ {1} = \# _ {a - b} (x) \tag {64}
$$
$$
\left[ \mathbf {c} _ {t} \right] _ {2} = \# _ {b a} (x). \tag {65}
$$
$[\mathbf{c}_t]_2$  can be computed provided the QRNN window size is  $\geq 2$ . A two-layer decoder can then check
$$
0 \leq [ \mathbf {c} _ {t} ] _ {1} \leq 0 \wedge [ \mathbf {c} _ {t} ] _ {2} \leq 0. \tag {66}
$$
Theorem 11 (Suffix attack). No  $s$ -QRNN and decoder can recognize the language  $a^n b^n \Sigma^* = a^n b^n (a|b)^*$ ,  $n > 0$ , i.e.,  $a^n b^n \Sigma^* \notin L(s$ -QRNN) for any decoder  $L$ .
The proof will rely on the s-QRNN's inability to "freeze" a computed value, protecting it from manipulation by future input.
Proof. As in the proof for Theorem 9, it is sufficient to show that no  $\Sigma^k$ -restricted CM with the additional operations  $\{:= -1, := 1\}$  can recognize  $a^n b^n \Sigma^*$  for any decoder  $L$ .
Let  $\mathcal{A}$  be some such CM, with window size  $k$  and  $h$  counters. For every  $w\in \Sigma^n$  denote by  $c(w)\in \mathbb{Q}^h$  the counter values of  $\mathcal{A}$  after processing  $w$ . Denote by  $u_{t}$  the vector of counter update operations made by this machine on an input sequence  $w$  at time  $t\leq |w|$ . Recall that  $\mathcal{A}$  is  $\Sigma^k$  restricted, meaning that  $u_{i}$  depends exactly on the window of the last  $k$  tokens for every  $i$ .
We now denote  $j = k + 10$  and consider the sequences  $w_{1} = a^{j}b^{j}a^{j}b^{j}a^{j}b^{j}$ ,  $w_{2} = a^{j}b^{j - 1}a^{j}b^{j + 1}a^{j}b^{j}$ .  $w_{2}$  is obtained from  $w_{1}$  by removing the  $2j$ -th token of  $w_{1}$  and reinserting it at position  $4j$ .
As all of  $w_{1}$  is composed of blocks of  $\geq k$  identical tokens, the windows preceding all of the other tokens in  $w_{1}$  are unaffected by the removal of the  $2j$ -th token. Similarly, being added onto the end of a substring  $b^{k}$ , its insertion does not affect the windows of the tokens after it, nor is its own window different from before. This means that overall, the set of all operations  $u_{i}$  performed on the counters is identical in  $w_{1}$  and in  $w_{2}$ . The only difference is in their ordering.
$w_{1}$  and  $w_{2}$  begin with a shared prefix  $a^{k}$ , and so necessarily the counters are identical after processing it. We now consider the updates to the counters after these first  $k$  tokens, these are determined by the windows of  $k$  tokens preceding each update.
First, consider all the counters that undergo some assignment  $(\coloneqq)$  operation during these sequences, and denote by  $\{w\}$  the multiset of windows in  $w \in \Sigma^k$  for which they are reset.  $w_1$  and  $w_2$  only contain  $k$ -windows of types  $a^x b^{k - x}$  or  $b^{x}a^{k - x}$ , and so these must all re-appear in the shared suffix  $b^{j}a^{j}b^{j}$  of  $w_{1}$  and  $w_{2}$ , at which point they will be synchronised. It follows that these counters all finish with identical value in  $c(w_1)$  and  $c(w_2)$ .
All the other counters are only updated using addition of  $-1, 1$  and  $0$ , and so the order of the updates is inconsequential. It follows that they too are identical in  $c(w_1)$  and  $c(w_2)$ , and therefore necessarily that  $c(w_1) = c(w_2)$ .
From this we have  $w_{1}, w_{2}$  satisfying  $w_{1} \in a^{n}b^{n}\Sigma^{*}, w_{2} \notin a^{n}b^{n}\Sigma^{*}$  but also  $c(w_{1}) = c(w_{2})$ . Therefore, it is not possible to distinguish between  $w_{1}$  and  $w_{2}$  with the help of any decoder, despite the fact that  $w_{1} \in a^{n}b^{n}\Sigma^{*}$  and  $w_{2} \notin a^{n}b^{n}\Sigma^{*}$ . It follows that the CM and s-QRNN cannot recognize  $a^{n}b^{n}\Sigma^{*}$  with any decoder.
For the opposite extension  $\Sigma^{*}a^{n}b^{n}$ , in which the language is augmented by a prefix, we cannot use such a "suffix attack". In fact,  $\Sigma^{*}a^{n}b^{n}$  can be recognized by an s-QRNN with window length  $w\geq 2$  and a linear threshold decoder as follows: a counter counts  $\#_{a - b}(x)$  and is reset to 1 on appearances of  $ba$ , and the decoder compares it to 0.
Note that we define decoders as functions from the final state to the output. Thus, adding an additional QRNN layer does not count as a "decoder" (as it reads multiple states). In fact, we show that having two QRNN layers allows recognizing  $a^n b^n \Sigma^*$ .
Theorem 12. Let  $\epsilon$  be the empty string. Then,
$$
a ^ {n} b ^ {n} \Sigma^ {*} \cup \{\epsilon \} \in D _ {1} (\mathrm {s - Q R N N} \circ \mathrm {s - Q R N N}).
$$
Proof. We construct a two-layer s-QRNN from which  $a^n b^n \Sigma^*$  can be recognized. Let  $\$$  denote the left edge of the string. The first layer computes two quantities  $d_t$  and  $e_t$  as follows:
$$
d _ {t} = \# _ {b a} (x) \tag {67}
$$
$$
e _ {t} = \# _ {\mathbb {S} ^ {b}} (x). \tag {68}
$$
Note that  $e_t$  can be interpreted as a binary value checking whether the first token was  $b$ . The second layer computes  $c_t$  as a function of  $d_t, e_t$ , and  $x_t$  (which can be passed through the first layer). We will demonstrate a construction for  $c_t$  by creating
linearly separable functions for the gate terms  $f_{t}$  and  $z_{t}$  that update  $c_{t}$ .
$$
f _ {t} = \left\{ \begin{array}{l l} 1 & \text {i f} d _ {t} \leq 0 \\ 0 & \text {o t h e r w i s e} \end{array} \right. \tag {69}
$$
$$
z _ {t} = \left\{ \begin{array}{l l} 1 & \text {i f} x _ {t} = a \vee e _ {t} \\ - 1 & \text {o t h e r w i s e .} \end{array} \right. \tag {70}
$$
Now, the update function  $u_{t}$  to  $c_{t}$  can be expressed
$$
u _ {t} = f _ {t} z _ {t} = \left\{ \begin{array}{l l} + 0 & \text {i f} 0 <   d _ {t} \\ + 1 & \text {i f} d _ {t} \leq 0 \wedge \left(x _ {t} = a \vee e _ {t}\right) \\ - 1 & \text {o t h e r w i s e .} \end{array} \right. \tag {71}
$$
Finally, the decoder accepts iff  $c_{t} \leq 0$ . To justify this, we consider two cases: either  $x$  starts with  $b$  or  $a$ . If  $x$  starts with  $b$ , then  $e_{t} = 0$ , so we increment  $c_{t}$  by 1 and never decrement it. Since  $0 < c_{t}$  for any  $t$ , we will reject  $x$ . If  $x$  starts with  $a$ , then we accept iff there exists a sequence of  $bs$  following the prefix of  $as$  such that both sequences have the same length.
D s-LSTMs
In contrast to the s-QRNN, we show that the sLSTM paired with a simple linear and thresholding decoder can recognize both  $a^n b^n$  and  $a^n b^n \Sigma^*$ .
Theorem 13.
$$
a ^ {n} b ^ {n} \in D _ {1} (\mathrm {s - L S T M}).
$$
Proof. Assuming a string  $a^i b^i$ , we set two units of the LSTM state to compute the following functions using the CM in Figure 3:
$$
\left[ \mathbf {c} _ {t} \right] _ {1} = \operatorname {R e L U} (i - j) \tag {72}
$$
$$
[ \mathbf {c} _ {t} ] _ {2} = \operatorname {R e L U} (j - i). \tag {73}
$$
We also add a third unit  $[\mathbf{c}_t]_3$  that tracks whether the 2-gram  $ba$  has been encountered, which is equivalent to verifying that the string has the form  $a^i b^i$ . Allowing  $\mathbf{h}_t = \tanh (\mathbf{c}_t)$ , we set the linear threshold layer to check
$$
\left[ \mathbf {h} _ {t} \right] _ {1} + \left[ \mathbf {h} _ {t} \right] _ {2} + \left[ \mathbf {h} _ {t} \right] _ {3} \leq 0. \tag {74}
$$
Theorem 14.
$$
a ^ {n} b ^ {n} \Sigma^ {*} \in D _ {1} (\mathrm {s - L S T M}).
$$
Proof. We use the same construction as Theorem 13, augmenting it with
$$
[ \mathbf {c} _ {t} ] _ {4} \triangleq [ \mathbf {h} _ {t - 1} ] _ {1} + [ \mathbf {h} _ {t - 1} ] _ {2} + [ \mathbf {h} _ {t - 1} ] _ {3} \leq 0. \tag {75}
$$
We decide  $x$  according to the (still linearly separable) equation
$$
\left(0 <   \left[ \mathbf {h} _ {t} \right] _ {4}\right) \vee \left(\left[ \mathbf {h} _ {t} \right] _ {1} + \left[ \mathbf {h} _ {t} \right] _ {2} + \left[ \mathbf {h} _ {t} \right] _ {3} \leq 0\right). \tag {76}
$$
Models were trained on strings up to length 64, and, at each index  $t$ , were asked to classify whether or not the prefix up to  $t$  was a valid string in the language. Models were then tested on independent datasets of lengths 64, 128, 256, 512, 1024, and 2048. The training dataset contained 100000 strings, and the validation and test datasets contained 10000. We discuss task-specific schemes for sampling strings in the next paragraph. All models were trained for a maximum of 100 epochs, with early stopping after 10 epochs based on the validation cross entropy loss. We used default hyperparameters provided by the open-source AllenNLP framework (Gardner et al., 2018). The code is available at https://github.com/viking-sudo-rm/rr-experiments.
Sampling strings For the language  $L_{5}$ , each token was sampled uniformly at random from  $\Sigma = \{a, b\}$ . For  $a^{n}b^{n}\Sigma^{*}$ , half the strings were sampled in this way, and for the other half, we sampled  $n$  uniformly between 0 and 32, fixing the first  $2n$  characters of the string to  $a^{n}b^{n}$  and sampling the suffix uniformly at random.
Experimental cost Experiments were run for 20 GPU hours on Quadro RTX 8000.
F Self Attention
Architecture We place saturated self attention (Vaswani et al., 2017) into the state expressiveness hierarchy. We consider a single-head self attention encoder that is computed as follows:
1. At time  $t$ , compute queries  $\mathbf{q}_t$ , keys  $\mathbf{k}_t$ , and values  $\mathbf{v}_t$  from the input embedding  $\mathbf{x}_t$  using a linear transformation.
2. Compute attention head  $\mathbf{h}_t$  by attending over the keys and values up to time  $t$  ( $\mathbf{K}_{:t}$  and  $\mathbf{V}_{:t}$ ) with query  $\mathbf{q}_t$ .
3. Let  $\| \cdot \| _L$  denote a layer normalization operation (Ba et al., 2016).
$$
\mathbf {h} _ {t} ^ {\prime} = \operatorname {R e L U} \left(\mathbf {W} ^ {h} \cdot \| \mathbf {h} _ {t} \| _ {L}\right) \tag {77}
$$
$$
\mathbf {c} _ {t} = \left\| \mathbf {W} ^ {c} \mathbf {h} _ {t} ^ {\prime} \right\| _ {L}. \tag {78}
$$
This simplified architecture has only one attention head, and does not incorporate residual connections. It is also masked (i.e., at time  $t$ , can only see the prefix  $\mathbf{X}_{:t}$ ), which enables direct comparison with unidirectional RNNs. For simplicity, we do not add positional information to the input embeddings.
Theorem 15. Saturated masked self attention is not  $RR$ .
Proof. Let  $\#_{\sigma}(x)$  denote the number of occurrences of  $\sigma \in \Sigma$  in string  $x$ . We construct a self attention layer to compute the following function over  $\{a, b\}^*$ :
$$
f (x) = \left\{ \begin{array}{l l} 0 & \text {i f} \# _ {a} (x) = \# _ {b} (x) \\ 1 & \text {o t h e r w i s e .} \end{array} \right. \tag {79}
$$
Since the Hankel sub-block over  $P = a^{*},S = b^{*}$  has infinite rank,  $f\notin \mathcal{R}$
Fix  $\mathbf{v}_t = \mathbf{x}_t$ . As shown by Merrill (2019), saturated attention over a prefix of input vectors  $\mathbf{X}_{:t}$  reduces to sum of the subsequence for which key-query similarity is maximized, i.e., denoting  $I = \{i \in [t] \mid \mathbf{k}_i \cdot \mathbf{q}_t = m\}$  where  $m = \max \{\mathbf{k}_i \cdot \mathbf{q}_t | i \in [t]\}$ :
$$
\mathbf {h} _ {t} = \frac {1}{| I |} \sum_ {i \in I} \mathbf {x} _ {t _ {i}}. \tag {80}
$$
For all  $t$ , set the key and query  $k_{t}, q_{t} = 1$ . Thus, all the key-query similarities are 1, and we obtain:
$$
\begin{array}{l} \mathbf {h} _ {t} = \frac {1}{t} \sum_ {t ^ {\prime} = 1} ^ {t} \mathbf {x} _ {t ^ {\prime}} (81) \\ = \frac {1}{t} (\# _ {a} (x), \# _ {b} (x)) ^ {\top}. (82) \\ \end{array}
$$
Applying layer norm to this quantity preserves equality of the first and second elements. Thus, we set the layer in (77) to independently check  $0 < [\mathbf{h}_t^0 ]_1 - [\mathbf{h}_t^0 ]_2$  and  $[\mathbf{h}_t^0 ]_1 - [\mathbf{h}_t^0 ]_2 < 0$  using ReLU. The final layer  $c_{t}$  sums these two quantities, returning 0 if neither condition is met, and 1 otherwise.
Since saturated self attention can represent  $f\notin \mathcal{R}$  , it is not RR.
Space Complexity We show that self attention falls into the same space complexity class as the LSTM and QRNN. Our method here extends Merrill (2019)'s analysis of attention.
Theorem 16. Saturated single-layer self attention has  $\Theta (\log n)$  space.
Proof. The construction from Theorem 15 can reach a linear (in sequence length) number of different outputs, implying a linear number of different configurations, and so that the space complexity of saturated self attention is  $\Omega (\log n)$ . We now show the upper bound  $O(\log n)$ .
A sufficient representation for the internal state (configuration) of a self-attention layer is the unordered group of key-value pairs over the prefixes of the input sequence.
Since  $f_{k}:x_{t}\mapsto \mathbf{k}_{t}$  and  $f_{v}:x_{t}\mapsto \mathbf{v}_{t}$  have finite domain  $(\Sigma)$ , their images  $K = \mathrm{image}(f_k),V =$  image  $(f_v)$  are finite.14 Thus, there is also a finite number of possible key-value pairs  $\langle \mathbf{k}_t,\mathbf{v}_t\rangle \in$ $K\times V$  . Recall that the internal configuration can be specified by the number of occurrences of each possible key-value pair. Taking  $n$  as an upper bound for each of these counts, we bound the number of configurations of the layer as  $n^{|K\times V|}$  . Therefore the bit complexity is
$$
\log_ {2} \left(n ^ {| K \times V |}\right) = O (\log n). \tag {83}
$$
Note that this construction does not apply if the "vocabulary" we are attending over is not finite. Thus, using unbounded positional embeddings, stacking multiple self attention layers, or applying attention over other encodings with unbounded state might reach  $\Theta (n)$ .
While it eludes our current focus, we hope future work will extend the saturated analysis to self attention more completely. We direct the reader to Hahn (2020) for some additional related work.
G Memory Networks
All of the standard RNN architectures considered in Section 3 have  $O(\log n)$  space in their saturated form. In this section, we consider a stack RNN encoder similar to the one proposed by Suzgun et al. (2019b) and show how it, like a WFA, can encode binary representations from strings. Thus,
the stack RNN has  $\Theta (n)$  space. Additionally, we find that it is not RR. This places it in the upperright box of Figure 1.
Classically, a stack is a dynamic list of objects to which elements  $v \in V$  can be added and removed in a LIFO manner (using push and pop operations). The stack RNN proposed in Suzgun et al. (2019b) maintains a differentiable variant of such a stack, as follows:
Differentiable Stack In a differentiable stack, the update operation takes an element  $s_t$  to push and a distribution  $\pi_t$  over the update operations push, pop, and no-op, and returns the weighted average of the result of applying each to the current stack. The averaging is done elementwise along the stacks, beginning from the top entry. To facilitate this, differentiable stacks are padded with infinite 'null entries'. Their elements must also have a weighted average operation defined.
Definition 6 (Geometric  $k$ -stack RNN encoder). Initialize the stack  $\mathbf{S}$  to an infinite list of null entries, and denote by  $S_{t}$  the stack value at time  $t$ . Using 1-indexing for the stack and denoting  $[S_{t - 1}]_0 \triangleq \mathbf{s}_t$ , the geometric  $k$ -stack RNN recurrent update is:15
$$
\mathbf {s} _ {t} = \mathbf {f} _ {s} (x _ {t}, \mathbf {c} _ {t - 1})
$$
$$
\pi_ {t} = \mathbf {f} _ {\pi} \left(x _ {t}, \mathbf {c} _ {t - 1}\right)
$$
$$
\forall i \geq 1 [ \mathbf {S} _ {t} ] _ {i} = \sum_ {a = 1} ^ {3} [ \pi_ {t} ] _ {a} [ \mathbf {S} _ {t - 1} ] _ {i + a - 2}.
$$
In this work we will consider the case where the null entries are  $\mathbf{0}$  and the encoding  $\mathbf{c}_t$  is produced as a geometric-weighted sum of the stack contents,
$$
\mathbf {c} _ {t} = \sum_ {i = 1} ^ {\infty} \left(\frac {1}{2}\right) ^ {i - 1} [ \mathbf {S} _ {t} ] _ {i}.
$$
This encoding gives preference to the latest values in the stack, giving initial stack encoding  $\mathbf{c}_0 = \mathbf{0}$ .
Space Complexity The memory introduced by the stack data structure pushes the encoder into  $\Theta (n)$  space. We formalize this by showing that, like a WFA, the stack RNN can encode binary strings to their value.
Lemma 5. The saturated stack RNN can compute the converging binary encoding function, i.e.,  $101 \mapsto 1 \cdot 1 + 0.5 \cdot 0 + 0.25 \cdot 1 = 1.25$ .