SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Based on Appendix C.2 (Lemma C.8), we can use the first attention head to perform the MEAN operation that counts the percentage of equal signs in the preceding sentences (i.e.,  $n_i^= / i$ ). This can be achieved by setting  $Q = 0, K = 0, V = (e_{\mathrm{id}(\cdot = ')}, 0, 0)^{\top}$  (defined in Appendix C.2), so that
$$
\boldsymbol {q} _ {i} = \mathbf {0}, \quad \boldsymbol {k} _ {j} = \mathbf {0}, \quad v _ {j} = \boldsymbol {e} _ {\mathrm {i d} (‘ =) ^ {\prime}} \cdot \boldsymbol {e} _ {\mathrm {i d} (s _ {j})} = \mathbb {I} [ s _ {j} = ‘ = ^ {\prime} ], \quad \boldsymbol {S} _ {i} = [ i ].
$$
Similarly, we can use the second attention head to perform a COPY operation that copies the position index of the last equal sign (by Lemma C.7). This can be achieved by setting  $\pmb{Q} = (\mathbf{0}, 0, 1)^{\top}$ ,  $\pmb{K} = (\pmb{e}_{\mathrm{id}(\cdot = \cdot)}, 0, -1)^{\top}$ ,  $\pmb{V} = (\mathbf{0}, 1, 0)^{\top}$ ,  $r_j = j$  (defined in Appendix C.2), so that
$$
q _ {i} = 1, \quad k _ {j} = \mathbb {I} [ s _ {j} = ^ {\prime} = ^ {\prime} ] - 1, \quad v _ {j} = j, \quad \mathcal {S} _ {i} = \{j \leq i: s _ {j} = ^ {\prime} = ^ {\prime} \}.
$$
It is easy to check that the above construction outputs  $u_{i} = \max \{j:j\leq i,s_{j} = {}^{\prime} = {}^{\prime}\}$  when  $S_{i}\neq \emptyset$ . Note that  $u_{i}$  may not equal to  $p_i^{\equiv}$  when  $S_{i} = \emptyset$ .
Using the residual connection to perform concatenation, the output of the attention layer has the form  $(e_{\mathrm{id}(s_i)},i,1,n_i^{\equiv} / i,\max \{j:j\leq i,s_j = {}^{\prime} = {}\})$ . We can then use an MLP to multiply  $n_i^{\equiv} / i$  and  $i$  to obtain  $n_i^{\equiv}$  and use another MLP to compute  $i^2$  according to Lemma C.1; Simultaneously, we can compute the value  $p_i^{\equiv}$  using the following way:
$$
p _ {i} ^ {=} = \left\{ \begin{array}{l l} \max  \{j: j \leq i, s _ {j} = ^ {\prime} = ^ {\prime} \} & \text {i f} n _ {i} ^ {=} / i \geq 1 / n, \\ 0 & \text {i f} n _ {i} ^ {=} / i = 0, \end{array} \right.
$$
which is a conditional selection operation and can be implemented by an MLP (Lemma C.4). Also note that the gap in Lemma C.4 is  $\alpha = 1 / 2n$ , which can be implemented within log-precision. The final output of the first layer has the form  $x_{i}^{(1)} = (e_{\mathrm{id}(s_{i})}, i, i^{2}, n_{i}^{\overline{=}}, p_{i}^{\overline{=}}, 1)$ .
Layer 2. The second layer of the Transformer does some tricky preparation work for the next layer.
1. Compute the distance to the nearest and the last equal sign, denoted as  $d_i^{\equiv}$  and  $\hat{d}_i^{\equiv}$ , respectively. Formally,  $d_i^{\equiv} = i - \max \{j : j \leq i, s_j = ' = '\}$ ,  $\hat{d}_i^{\equiv} = i - \max \{j : j < i, s_j = ' = '\}$ . If the nearest/last equal sign does not exist, define  $d_i^{\equiv} = i$  or  $\hat{d}_i^{\equiv} = i$ . The relation between  $d_i^{\equiv}$ ,  $\hat{d}_i^{\equiv}$ , and  $p_i^{\equiv}$  can be expressed as  $d_i^{\equiv} = i - p_i^{\equiv}$ ,  $\hat{d}_i^{\equiv} = i - p_{i-1}^{\equiv}$ .
2. Count the number of equal signs in strictly previous tokens, denoted as  $\hat{n}_i^{\overline{=}}$ , i.e.,  $\hat{n}_i^{\overline{=}} = |\{j < i : s_j = ' = '\}|$ .
3. Compute  $(n_i^{\equiv})^2, (\hat{n}_i^{\equiv})^2, (d_i^{\equiv})^2$ , and  $(\hat{d}_i^{\equiv})^2$ .
The first and the second tasks can be done using the COPY operation by setting
$$
\boldsymbol {q} _ {i} = \boldsymbol {Q} \boldsymbol {x} _ {i} ^ {(1)} = ((i - 1) ^ {2}, i - 1, 1), \quad \boldsymbol {k} _ {j} = \boldsymbol {K} \boldsymbol {x} _ {j} ^ {(1)} = (- 1, 2 j, - j ^ {2}), \quad r _ {j} = 0, \quad \boldsymbol {v} _ {j} = (n _ {j}, j - p _ {j} ^ {=} + 1).
$$
Under the above construction, we have  $\pmb{q}_i \cdot \pmb{k}_j = -(i - j - 1)^2$ , and thus  $S_i = \{i - 1\}$ , namely, the output is  $(\hat{n}_i, \hat{d}_i^{\equiv})$ . We then use an MLP to calculate  $(d_i^{\equiv})^2$ ,  $(\hat{d}_i^{\equiv})^2$ ,  $(n_i^{\equiv})^2$ , and  $(\hat{n}_i^{\equiv})^2$  by using Lemma C.1. The output of the second layer is
$$
\pmb {x} _ {i} ^ {(2)} = (\pmb {e} _ {\mathrm {i d} (s _ {i})}, i, n _ {i} ^ {=} \hat {,} n _ {i} ^ {=} \hat {,} d _ {i} ^ {=} \hat {,} d _ {i} ^ {=} \hat {,} (n _ {i} ^ {=} \hat {,}) ^ {2}, (\hat {n} _ {i} ^ {=} \hat {,}) ^ {2}, (\hat {d} _ {i} ^ {=} \hat {,}) ^ {2}, 1).
$$
Layer 3. The third Transformer layer judges whether the calculation should be performed at the current position and computes the result when needed. Based on the CoT format given in Appendix B, we need to extract five previous tokens related to this position. Formally, we need five attention heads to perform the following tasks:
1. Copy the embedding  $e_{\mathrm{id}(s_j)}$  located at position  $j$  such that  $\hat{n}_j^= = n_i^= - 1$  and  $\hat{d}_j^= = d_i^= + t$  for  $t \in \{1, 2, 3, 4, 5\}$ , as shown in Figure 4.
2. Check if the copied expression can be evaluated at the current position according to the rule given in Appendix B. If it can be evaluated, compute the result and determine how much sentence length will be reduced after this calculation (see Appendix B for details on how the reduced sentence length depends on brackets); otherwise, keep the token  $e_{\mathrm{id}(s_j)}$  with  $\hat{n}_j^= = n_i^- - 1$  and  $\hat{d}_j^= = d_i^- + 1$ .
We can use the multi-head attention to perform the COPY operation five times in parallel. For each  $t$ , we construct the matrices  $Q, K, V$  of the COPY operation such that
$$
\begin{array}{l} \boldsymbol {q} _ {i} = \boldsymbol {Q} \boldsymbol {x} _ {i} ^ {(2)} = [ (n _ {i} ^ {=} 2 - 2 n _ {i} ^ {=} + 1, \quad 1, \quad n _ {i} ^ {=} - 1, \quad (d _ {i} ^ {=} 2 + 2 t d _ {i} ^ {=} + t ^ {2}, \quad 1, \quad d _ {i} ^ {=} - t ] ^ {\top}, \\ \boldsymbol {k} _ {j} = \boldsymbol {K} \boldsymbol {x} _ {j} ^ {(2)} = [ - 1, - (\hat {n} _ {j} ^ {\mp}) ^ {2}, 2 \hat {n} _ {j} ^ {\mp}, - 1, - (\hat {d} _ {j} ^ {\mp}) ^ {2}, 2 \hat {d} _ {j} ^ {\mp} ] ^ {\top}, \\ \boldsymbol {v} _ {j} = \boldsymbol {e} _ {\mathrm {i d} (s _ {j})}, \\ \end{array}
$$
Figure 4: Illustration of the proof of Theorem 3.3.
$$
\boldsymbol {K} \boldsymbol {x} _ {i} ^ {(2)} \cdot \boldsymbol {Q} \boldsymbol {x} _ {j} ^ {(2)} = - (n _ {i} ^ {=} - \hat {n} _ {j} ^ {=} - 1) ^ {2} - (d _ {i} ^ {=} - \hat {d} _ {j} ^ {=} + t) ^ {2}.
$$
Therefore,  $K\pmb{x}_i^{(2)} \cdot Q\pmb{x}_j^{(2)} = 0$  only when  $\hat{n}_j^= = n_i^- - 1$  and  $\hat{d}_j^= = d_i^- + t$ , and  $K\pmb{x}_i^{(2)} \cdot Q\pmb{x}_j^{(2)} \leq -1$  otherwise. It is easy to see that:
- Whenever  $n_i = 0$ , for any  $t$ , the number of indices  $j$  satisfying  $\pmb{q}_i \cdot \pmb{k}_j = 0$  is at most one (i.e., unique).
- Whenever  $n_i^= > 0$ , for any  $t$ , the index  $j$  satisfying  $\mathbf{q}_i \cdot \mathbf{k}_j = 0$  exists, unless there is a  $t' < t$  such that the copied token at  $t'$  is an equal sign ( $=$ ).
In other words, based on Lemma C.7, the above property guarantees that we can copy the desired tokens until reaching an equal sign, after which the copied tokens are invalid as illustrated in Figure 4(right). The output of the attention layer can be written as
$$
(\pmb {e} _ {\mathrm {i d} (s _ {i})}, \pmb {e} _ {j _ {1}}, \pmb {e} _ {j _ {2}}, \pmb {e} _ {j _ {3}}, \pmb {e} _ {j _ {4}}, \pmb {e} _ {j _ {5}}, i, n _ {i} ^ {=} \mathrm {,} (n _ {i} ^ {=} \mathrm {) ^ {2}} \mathrm {,} d _ {i} ^ {=} \mathrm {,} \hat {n} _ {i} ^ {=} \mathrm {,} (\hat {n} _ {i} ^ {=} \mathrm {) ^ {2}} \mathrm {,} \hat {d} _ {i} ^ {=} \mathrm {,} (\hat {d} _ {i} ^ {=} \mathrm {) ^ {2}} \mathrm {, 1}),
$$
where we slightly abuse notation and use  $e_{j_t}$  to denote the embedding we copied by the  $t$ -th attention heads.
We can then use an MLP to perform the second task. Note that whether the current position can be calculated or not depends on the following six tokens  $(e_{\mathrm{id}(s_i)}, e_{j_1}, e_{j_2}, e_{j_3}, e_{j_4}, e_{j_5})$ . Concretely, there are several cases:
-  $e_{\mathrm{id}(s_i)}$  corresponds to the embedding of a number or a right bracket. In this case, the current position should simply output  $e_{j_1}$  (which is an operator or a right bracket).
-  $e_{\mathrm{id}(s_i)}$  corresponds to the embedding of a left bracket, an operator, or the equal sign  $=$ . In this case,  $e_{j_1}$  corresponds to the embedding of a number or a left bracket. There are two subcases:
-  $e_{j_1}$  corresponds to the embedding of a number. In this case, whether the current position can be evaluated depends on  $(e_{\mathrm{id}(s_i)}, e_{j_1}, e_{j_2}, e_{j_3}, e_{j_4})$  according to Appendix B.
-  $e_{j_1}$  corresponds to the embedding of a left bracket. In this case, whether the current position can be evaluated simply depends on whether  $e_{j_5}$  corresponds to the embedding of a right bracket.
When all embeddings  $e_{j_t}$  are one-hot vectors, whether the expression at the current position can be calculated or not forms a look-up table. Therefore, it can be implemented by a two-layer MLP with hidden dimension  $O(p^6)$  according to Lemma C.5. Similarly, the computed result and how much sentence length will be reduced after this calculation can also be implemented as look-up tables. However, some of the embeddings  $e_{j_t}$  may not be one-hot vectors when reaching an equal sign (as discussed above). In this case, we can similarly implement extra look-up tables that take a fewer number of inputs, and the result of which lookup table will be used depends on the position of the equal sign. This corresponds to a multivariate conditional selection operation with multiple Boolean conditions  $\mathbb{I}[e_{j_t} \cdot e_{\mathrm{id}(\cdot =)} = 1]$  (for  $t \in \{1,2,3,4,5\}$ ), which can be similarly implemented by an MLP by extending Lemma C.4. Moreover, we note that the composition of look-up tables and the multivariate conditional selection operation can be merged in just one MLP by following the construction in Lemmas C.4 and C.5 (we omit the details for clarity).
The final output of the third layer is represented by
$$
\boldsymbol {x} _ {i} ^ {(3)} = \left(\boldsymbol {e} _ {j _ {1}}, n _ {i} ^ {=} \text {,} \left(n _ {i} ^ {=} \right. \right. ^ {2}, d _ {i} ^ {=} \text {,} \hat {n} _ {i} ^ {=} \text {,} \left(\hat {n} _ {i} ^ {=} \right. \text {,} ^ {2}, \hat {d} _ {i} ^ {=} \text {,} \left(\hat {d} _ {i} ^ {=} \right. \text {,} ^ {2}, f _ {i}, \boldsymbol {e} _ {i} ^ {\text {o u t c o m e}}, n _ {i} ^ {\text {r e d u c e}}).
$$
Here,  $f_{i}$  is a Boolean value recording whether the next output at position  $i$  is a computed value,  $e_{i}^{\mathrm{outcome}}$  is the one-hot embedding of the outcome when  $f_{i}$  is true, and  $n_{i}^{\mathrm{reduce}}$  records the reduced length after calculation when  $f_{i}$  is true. When  $f_{i}$  is false,  $e_{i}^{\mathrm{outcome}}$  and  $n_{i}^{\mathrm{reduce}}$  are undefined.
Layer 4. Note that in an arithmetic expression there can be multiple expressions that can be calculated (or handles defined in Appendix B), all of which are processed in the last layer. Therefore, the fourth layer of the Transformer should keep only the leftmost calculation and discard other calculations (according to Appendix B). Meanwhile, for subsequent positions  $i$  after the position that has been calculated, this layer finds the related token that should be copied for position  $i$  based on the reduced setence length. Formally, we need two attention heads to perform the following tasks:
1. Check whether there is an index  $j \leq i$  such that  $n_j^= = n_i^=$  and  $f_j$  is true. Denote the answer as  $\hat{f}_i = \sum_{j=i-d_i^=}^i f_j$ , where  $\hat{f}_i \geq 1$  means the answer is yes, and  $\hat{f}_i = 0$  otherwise.
2. If the answer is yes ( $\hat{f}_i \geq 1$ ), copy the value  $n_j^{\text{reduce}}$  at the leftmost position  $j$  satisfying  $f_j$  is true and  $n_j^{\overline{=}} = n_i^{\overline{=}}$ . Denote the result as  $\hat{n}_i^{\text{reduce}} := n_j^{\text{reduce}}$ . If  $\hat{f}_i = 0$ ,  $\hat{n}_i^{\text{reduce}}$  is undefined.
3. Filter the outcome: if  $f_{i}$  is true, then maintain  $e_{i}^{\mathrm{outcome}}$ , otherwise set  $e_{i}^{\mathrm{outcome}}$  to  $e_{j_1}$ .
Similar to the construction of the third layer, we can construct the matrices  $\mathbf{Q}$ ,  $\mathbf{K}$  and  $\mathbf{V}$  as follows. For the first task, we leverage the MEAN operation with
$$
\boldsymbol {q} _ {i} = \boldsymbol {Q} \boldsymbol {x} _ {i} ^ {(3)} = (1, (n _ {i} ^ {=} 2) ^ {2}, 2 n _ {i} ^ {=}  ), \quad \boldsymbol {k} _ {j} = \boldsymbol {K} \boldsymbol {x} _ {j} ^ {(3)} = (- (n _ {j} ^ {=} 2) ^ {2}, - 1, n _ {j} ^ {=}  ), \quad v _ {j} = f _ {j}.
$$
We have  $\pmb{q}_i \cdot \pmb{k}_j = -(n_i^= - n_j^=)^2$ . Therefore,  $\pmb{q}_i \cdot \pmb{k}_j = 0$  iff  $n_j^= = n_i^=$ , and  $\pmb{q}_i \cdot \pmb{k}_j \leq -1$  otherwise. This attention head thus outputs  $\frac{1}{d_i + 1} \sum_{j=i-d_i^=}^i f_j$ . For the second task, we leverage the COPY operation with
$$
\boldsymbol {q} _ {i} = \boldsymbol {Q} \boldsymbol {x} _ {i} ^ {(3)} = (1, (n _ {i} ^ {=} 2) ^ {2}, 2 n _ {i} ^ {=} 1, 1), \quad \boldsymbol {k} _ {j} = \boldsymbol {K} \boldsymbol {x} _ {j} ^ {(3)} = (- (n _ {j} ^ {=} 2) ^ {2}, - 1, n _ {j} ^ {=} f _ {j}, - 1), \quad v _ {j} = n _ {j} ^ {\text {r e d u c e}}.
$$
We have  $\pmb{q}_i \cdot \pmb{k}_j = -(n_i^- - n_j^-)^2 + f_j - 1$ . Therefore,  $\pmb{q}_i \cdot \pmb{k}_j = 0$  iff  $n_j^- = n_i^-$  and  $f_j = 1$ , and  $\pmb{q}_i \cdot \pmb{k}_j \leq -1$  otherwise. Moreover, we set  $r_j = -j$  in the COPY operation, by which the attention head copies  $n_j^{\mathrm{reduce}}$  where  $j = \min \{j : n_j^- = n_i^- , f_j = 1\}$ , as desired. The output of the attention layer has the form
$$
\left(e _ {i} ^ {\text {o u t c o m e}}, e _ {j _ {1}}, n _ {i} ^ {=} \text {,} (n _ {i} ^ {=} \text {)} ^ {2}, \hat {n} _ {i} ^ {=} \text {,} (\hat {n} _ {i} ^ {=} \text {)} ^ {2}, d _ {i} ^ {=} \text {,} \hat {d} _ {i} ^ {=} \text {,} (\hat {d} _ {i} ^ {=} \text {)} ^ {2}, f _ {i}, \frac {\hat {f} _ {i}}{d _ {i} ^ {=} + 1}, \hat {n} _ {i} ^ {\text {r e d u c e}}\right).
$$
We next use an MLP to perform the third task, which is a conditional selection operation and can be done according to Lemma C.4. We can simultaneously obtain  $\hat{f}_i$  by multiplying  $\frac{\hat{f}_i}{d_i^= + 1}$  with  $(d_i^= + 1)$ . We also compute  $(d_i^= + \hat{n}_i^{\text{reduce}})^2$ , which will be used in the next layer. The final output of the fourth layer is represented by
$$
\boldsymbol {x} _ {i} ^ {(4)} = \left(\tilde {\boldsymbol {e}} _ {i} ^ {\text {o u t c o m e}}, f _ {i}, n _ {i} ^ {=} \left(n _ {i} ^ {=} \right.\right) ^ {2}, \hat {n} _ {i} ^ {=} \left(\hat {n} _ {i} ^ {=} \right. \left(\hat {d} _ {i} ^ {=} \right., \hat {d} _ {i} ^ {=} \left(\hat {d} _ {i} ^ {=} \right. \left. \right. ^ {2}, \hat {f} _ {i}, \hat {n} _ {i} ^ {\text {r e d u c e}}, d _ {i} ^ {=} \left(d _ {i} ^ {=} + \hat {n} _ {i} ^ {\text {r e d u c e}}\right) ^ {2}),
$$
where  $\tilde{\pmb{e}}_i^{\mathrm{outcome}}$  is either  $\pmb {e}_i^{\mathrm{outcome}}$  or  $e_{j_1}$
Layer 5. The final layer of the Transformer uses one attention head to copy the corresponding token for generating the output when  $\hat{f}_i \geq 1$  and  $f_i$  is false. Similar to previous layers, we can copy the embedding  $e_{\mathrm{id}(s_j)}$  located at position  $j$  such that  $\hat{n}_{\overline{j}} = n_{\overline{i}} = -1$  and  $\hat{d}_{\overline{j}} = d_{\overline{i}} = +\hat{n}_{i}^{\mathrm{reduce}}$ . The output of the attention layer is  $(\tilde{e}_i^{\mathrm{outcome}}, e_{\mathrm{id}(s_j)}, f_i, \hat{f}_i)$ . We then use an MLP to obtain the output: if  $\hat{f}_i - f_i \geq 1$ , then output  $e_{\mathrm{id}(s_j)}$ ; otherwise output  $\tilde{e}_i^{\mathrm{outcome}}$ . This corresponds to a conditional selection operation and can be implemented by an MLP according to Lemma C.4. Finally, we pass the output through a softmax layer to generate the next token  $s_{i+1}$ .
Now it remains to conduct an error analysis and determine the scale of parameters. Note that we can tolerate  $O(1)$  error of the final layer output in the sense that the generated token  $s_{i+1}$  is still correct. Based on Lemmas C.1, C.4, C.5, C.7 and C.8, we can guarantee that when all parameters of the Transformer are bounded by  $O(\mathrm{poly}(n, 1/\epsilon))$ , all intermediate neurons will only induce an error below  $\epsilon$ . (Also note that Assumption C.6 in Lemmas C.7 and C.8 is satisfied when  $\epsilon$  is small enough.) Therefore, by picking a fixed small  $\epsilon = \Theta(1)$ , all parameter values in the Transformer are bounded by  $O(\mathrm{poly}(n))$ .
We now prove that evaluating arithmetic expressions without CoT is extremely difficult for bounded-depth autoregressive Transformers. We will make the widely-believed assumption that  $\mathsf{TC}^0\neq \mathsf{NC}^1$  (see Appendix A.2 for definitions of these complexity classes). We further need the notion of uniformity: informally, this condition says that there exists an efficient algorithm to construct the circuits. For a rigorous definition, we refer readers to Arora & Barak [2].
We first present some lemmas on the expressive power of the  $\mathsf{TC}^0$  circuits, based on which we will give a reduction proof of Theorem 3.1.
Lemma D.2. For any  $n$ $n$ -bits binary integers  $a_1, a_2, \dots, a_n$ , let  $f: \{0, 1\}^{n^2} \to \{0, 1\}^{2n}$  be a boolean function such that  $f(a_1, a_2, \dots, a_n) = \sum_{i \in [n]} a_i$ . Then,  $f$  can be implemented by the uniform TC<sup>0</sup> circuits.
This lemma demonstrates that computing the sum of  $n$ $n$ -bit integers (known as iterated addition or simply summation) is in uniform  $\mathsf{TC}^0$ . The detailed proof of this lemma can be found in the previous works [29, 15].
Lemma D.3. Consider any string  $s = s_1s_2 \cdots s_n$  of length  $n$  containing brackets  $('', '')$ , and other characters, and all brackets in  $s$  are paired. Let  $f$  be a boolean function taking  $s$  as input and output  $n$  pairs of integers defined as follows:
$$
\boldsymbol {f} _ {i} (\boldsymbol {s}) = \left\{ \begin{array}{l l} (- 1, j) & \text {i f s _ {i} i s a l e f t b r a c k e t a n d s _ {i} , s _ {j} a r e p a i r e d .} \\ (j, - 1) & \text {i f s _ {i} i s a r i g h t b r a c k e t a n d s _ {i} , s _ {j} a r e p a i r e d .} \\ (j, k) & \text {i f s _ {i} i s n o t a b r a c k e t , a n d s _ {j} , s _ {k} i s t h e n e a r e s t p a i r o f m a t c h i n g b r a c k e t s c o n t a i n i n g s _ {i}.} \end{array} \right.
$$
Then  $\pmb{f}$  can be implemented by the  $\mathsf{T}\mathsf{C}^0$  circuits.
Note that the input characters and the output integers are all encoded in binary. This lemma demonstrates that the task of bracket matching is in  $\mathsf{TC}^0$ .
Proof. Given the input string  $s$ , we first get the index  $i$  of each character  $s_i$ , which can be hard-coded into the circuits. Then for each character  $s_i$ , we calculate the following term:
$$
r _ {i} = \sum_ {j <   i} \mathbb {I} [ s _ {j} = ^ {\prime} (^ {\prime}) ] - \sum_ {j <   i} \mathbb {I} [ s _ {j} = ^ {\prime}) ^ {\prime} ] + \mathbb {I} [ s _ {i} = ^ {\prime} (^ {\prime}) ].
$$
Then, the output of  $f_{i}(s)$  can be written as the following form:
$$
f _ {i} (\boldsymbol {s}) = \left\{ \begin{array}{l l} (- 1, \min  \{j > i: s _ {j} = ^ {\prime}) ^ {\prime}, r _ {j} = r _ {i} \}) & \text {i f s _ {i} = ^ {\prime} (^ {\prime} ,} \\ (\max  \{j <   i: s _ {j} = ^ {\prime} (^ {\prime}, r _ {j} = r _ {i} \}, - 1) & \text {i f s _ {i} = ^ {\prime} ^ {\prime} ,} \\ (\max  \{j <   i: s _ {j} = ^ {\prime} (^ {\prime}, r _ {j} = r _ {i} \}, \min  \{j > i: s _ {j} = ^ {\prime}) ^ {\prime}, r _ {j} = r _ {i} \}) & \text {i f s _ {i} i s n o t a b r a c k e t .} \end{array} \right.
$$
It is simple to verify that the construction can be implemented by  $\mathsf{TC}^0$  circuits.
Theorem D.4. Assume  $\mathsf{TC}^0\neq \mathsf{uniform}\mathsf{NC}^1$ . For any prime number  $p$ , any integer  $L$ , and any polynomial  $Q$ , there exists a problem size  $n$  such that no log-precision autoregressive Transformer defined in Section 2 with depth  $L$  and hidden dimension  $d\leq Q(n)$  can solve the problem Arithmetic  $(n,p)$ .
Proof. Our proof is based on leveraging the  $\mathsf{NC}^1$ -completeness of a classic problem: Boolean Formula Evaluation. According to the Buss reduction [12], calculating whether a Boolean formula is true or false is complete for uniform  $\mathsf{NC}^1$ . Based on this theorem, it suffices to prove that the Boolean Formula Evaluation problem can be reduced to evaluating the arithmetic expression. This will yield the conclusion by using the result that bounded-depth log-precision Transformers with polynomial size are in  $\mathsf{TC}^0$  [42] as well as the assumption that  $\mathsf{TC}^0 \neq \mathsf{uniform}\mathsf{NC}^1$ .
Formally, let  $\Sigma = \{0,1,\land ,\lor ,\lnot ,(\cdot)\}$  be the alphabet. A Boolean formula is a string defined on alphabet  $\Sigma$  by the following recursive way:
4While the authors only proved that predicting a binary label using Transformer encoder can be implemented by  $\mathsf{TC}^0$  circuits, it is straightforward to extend the result to autoregressive Transformers predicting a label in a finite vocabulary.
- 0 and 1 are Boolean formulae;
- If  $\varphi$  is a Boolean formula, then  $(\neg \varphi)$  is a Boolean formula;
- If  $\varphi_{1},\varphi_{2}$  are two Boolean formulae, then both  $(\varphi_{1}\wedge \varphi_{2})$  and  $(\varphi_{1}\lor \varphi_{2})$  are Boolean formulae.
The Boolean Formula Evaluation problem aims to compute whether a Boolean formula is true (1) or false (0). We now show that we can translate this problem into the problem of evaluating arithmetic expressions. Given a Boolean formula  $s$ , the translation function  $f$  generates the corresponding arithmetic expression  $f(s)$  that has the same result as  $s$  under evaluation. The translation is recursively defined as follows:
$f(0) = 0$  and  $f(1) = 1$
- For any Boolean formula  $\varphi$ ,  $f(\neg \varphi) = 1 - \varphi$  and  $f((\varphi)) = (f(\varphi))$ ;
- For any Boolean formulae  $\varphi_1, \varphi_2$ ,  $f(\varphi_1 \wedge \varphi_2) = f(\varphi_1) \times f(\varphi_2)$ ;
- For any Boolean formulae  $\varphi_1, \varphi_2$ ,  $f(\varphi_1 \vee \varphi_2) = 1 - (1 - f(\varphi_1)) \times (1 - f(\varphi_2))$ .
It is easy to see that for any Boolean formula  $s$ , the length of  $f(s)$  is upper-bounded by  $O(|s|)$ . Moreover, the translation function can be simply implemented using the circuits within  $\mathsf{TC}^0$  complexity. To do so, we first replace the symbols  $\neg, \land$ , and  $\lor$  with  $1-$ ,  $\times$ , and  $\times$ , respectively. Furthermore, for each operator  $\lor$ , we must insert  $1 - (1-$  after the nearest left bracket containing the operator  $\lor$ , insert a right bracket before  $\times$ , insert  $(1-$  after  $\times$ , and insert a right bracket before the nearest right bracket containing the operator  $\lor$ . According to Lemmas D.2 and D.3, all of these operations can be implemented by the  $\mathsf{TC}^0$  circuits. Therefore, this translation function can be simply implemented by  $\mathsf{TC}^0$  circuits. Also, note that the above construction does not depend on the modulus  $p$ . Therefore, by reduction, we obtain that the problem of evaluating arithmetic expressions is  $\mathsf{NC}^1$ -hard.
E System of Linear Equations
In this section, we will prove that the autoregressive Transformer equipped with CoT can solve a system of linear equations, whereas the autoregressive Transformer without CoT cannot solve it.
E.1 Proof of Theorem 3.4
For ease of reading, we restate Theorem 3.3 below:
Theorem E.1. For any prime  $p$  and integer  $m > 0$ , there exists an autoregressive Transformer defined in Section 2 with hidden size  $d = O(\mathrm{poly}(p))$  (independent of  $m$ ), depth  $L = 4$ , and 5 heads in each layer that can generate the CoT solution defined in Appendix B for all inputs in Equation  $(m, p)$ . Moreover, all parameter values in the Transformer are bounded by  $O(\mathrm{poly}(m))$ .