SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Guhao Feng proposed to analyze the expressivity of Transformers using circuit complexity and proved all impossibility results in this paper, including Theorems 3.1, 3.2 and 4.8. He came up with the initial idea of Dynamic Programming. He and Bohang Zhang formalized the DP framework and proved all positive results in this paper, including Theorems 3.3, 3.4 and 4.7. He contributed to the paper writing of Appendices C to G.
Bohang Zhang supervised all undergraduate students. He raised the problem of linear equation. He and Guhao Feng formalized the DP framework and proved all positive results in this paper, including Theorems 3.3, 3.4 and 4.7. In the experimental part, he helped generate datasets with Yuntian Gu, conducted hyper-parameter tuning, and finalized all experiments in Section 5. He was responsible for writing the majority of this paper and checking/correcting all proofs.
Yuntian Gu was responsible for the experimental part. He wrote the entire code, including the model details, training pipeline, and evaluation. He created the datasets for all four tasks in Section 5 and conducted extensive experimental exploration during this project (with the help of Bohang Zhang).
Haotian Ye participated in regular discussions, raised ideas, and helped check the code in experiments.
Di He initiated the problem of studying the capability of Transformers in basic tasks like evaluating arithmetic expressions. Di He and Liwei Wang led and supervised the research, suggested ideas and experiments, and assisted in writing the paper.
[1] Ekin Akyurek, Dale Schuurmans, Jacob Andreas, Tengyu Ma, and Denny Zhou. What learning algorithm is in-context learning? investigations with linear models. In The Eleventh International Conference on Learning Representations, 2023.
[2] Sanjeev Arora and Boaz Barak. Computational complexity: a modern approach. Cambridge University Press, 2009.
[3] Yu Bai, Fan Chen, Huan Wang, Caiming Xiong, and Song Mei. Transformers as statisticians: Provable in-context learning with in-context algorithm selection. arXiv preprint arXiv:2306.04637, 2023.
[4] David A Barrington. Bounded-width polynomial-size branching programs recognize exactly those languages in nc. In Proceedings of the eighteenth annual ACM symposium on Theory of computing, pages 1-5, 1986.
[5] David A Mix Barrington and Denis Therien. Finite monoids and the fine structure of nc. Journal of the ACM (JACM), 35(4):941-952, 1988.
[6] Richard Bellman. The theory of dynamic programming. Bulletin of the American Mathematical Society, 60(6):503-515, 1954.
[7] Satwik Bhattachamishra, Kabir Ahuja, and Navin Goyal. On the ability and limitations of transformers to recognize formal languages. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7096-7116, 2020.
[8] Satwik Bhattachamishra, Arkil Patel, and Navin Goyal. On the computational power of transformers and its implications in sequence modeling. In Proceedings of the 24th Conference on Computational Natural Language Learning, pages 455-475, 2020.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in neural information processing systems, volume 33, pages 1877-1901, 2020.
[10] Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with GPT-4. arXiv preprint arXiv:2303.12712, 2023.
[61] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian richter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
[62] Gail Weiss, Yoav Goldberg, and Eran Yahav. Thinking like transformers. In International Conference on Machine Learning, pages 11080-11090. PMLR, 2021.
[63] Noam Wies, Yoav Levine, and Amnon Shashua. Sub-task decomposition enables learning in sequence to sequence tasks. In The Eleventh International Conference on Learning Representations, 2023.
[64] Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. An explanation of in-context learning as implicit bayesian inference. In International Conference on Learning Representations, 2022.
[65] Andrew C Yao. Circuits and local computation. In Proceedings of the twenty-first annual ACM symposium on Theory of computing, pages 186-196, 1989.
[66] Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. Self-attention networks can process bounded hierarchical languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3770–3785, 2021.
[67] Chulhee Yun, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Are transformers universal approximators of sequence-to-sequence functions? In International Conference on Learning Representations, 2020.
[68] Chulhee Yun, Yin-Wen Chang, Srinadh Bhojanapalli, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. O (n) connections are expressive enough: Universal approximability of sparse transformers. In Advances in Neural Information Processing Systems, volume 33, pages 13783-13794, 2020.
[69] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
[70] Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations, 2023.
The Appendix is organized as follows. Appendix A introduces additional mathematical background and useful notations. Appendix B presents formal definitions and CoT solutions of the arithmetic expression task and the linear equation task. Appendix C gives several technical lemmas, which will be frequently used in our subsequent proofs. The formal proofs for arithmetic expression, linear equation, and dynamic programming tasks are given in Appendices D, E and G, respectively. Discussions on other architectures such as encoder-decoder Transformers and RNNs are presented in Appendix F. Finally, we provide experimental details in Appendix H.
A Additional Background and Notation
A.1 Finite field
Intuitively, a field is a set  $\mathcal{F}$  on which addition, subtraction, multiplication, and division are defined and behave as the corresponding operations on rational and real numbers do. Formally, the two most basic binary operations in a field is the addition  $(+)$  and multiplication  $(\times)$ , which satisfy the following properties:
- Associativity: for any  $a, b, c \in \mathcal{F}$ ,  $(a + b) + c = a + (b + c)$  and  $(a \times b) \times c = a \times (b \times c)$ ;
- Commutativity: for any  $a, b \in \mathcal{F}, a + b = b + a$  and  $a \times b = b \times a$ ;
- Identity: there exist two different elements  $0, 1 \in \mathcal{F}$  such that  $a + 0 = a$  and  $a \times 1 = a$  for all  $a \in \mathcal{F}$ ;
- Additive inverses: for any  $a \in \mathcal{F}$ , there exists an element in  $\mathcal{F}$ , denoted as  $-a$ , such that  $a + (-a) = 0$ ;
- Multiplicative inverses: for any  $a \in \mathcal{F}$  and  $a \neq 0$ , there exists an element in  $\mathcal{F}$ , denoted as  $a^{-1}$ , such that  $a \times a^{-1} = 1$ ;
- Distributivity of multiplication over addition: for any  $a, b, c \in \mathcal{F}, a \times (b + c) = (a \times b) + (a \times c)$ .
Then, subtraction  $(-)$  is defined by  $a - b = a + (-b)$  for all  $a, b \in \mathcal{F}$ ; division  $(\div)$  is defined by  $a \div b = a \times b^{-1}$  for all  $a, b \in \mathcal{F}, b \neq 0$ .
Two most widely-used files are the rational number field  $\mathbb{Q}$  and the real number field  $\mathbb{R}$ , both of which satisfy the above properties. However, both fields contain an infinite number of elements. In this paper, we consider a class of fields called finite fields, which contain a finite number of elements. Given a prime number  $p$ , the finite field  $\mathbb{Z}_p$  is the field consisting of  $p$  elements, which can be denoted as  $0, 1, \dots, p - 1$ . In  $\mathbb{Z}_p$ , both addition and multiplication are defined by simply adding/multiplying two input integers and then taking the remainder modulo  $p$ . It can be easily checked that the two operations satisfy the six properties described above. Thus, subtraction and division can be defined accordingly. Remarkably, a key result in abstract algebra shows that all finite fields with  $p$  elements are isomorphic, which means that the above definitions of addition, subtraction, multiplication, and division are unique (up to isomorphism).
As an example, consider the finite field  $\mathbb{Z}_5$ . We have that  $2 + 3$  equals 0, since  $(2 + 3) \mod 5 = 0$ . Similarly,  $2 \times 3$  equals 1;  $2 - 3$  equals 4; and  $2 \div 3$  equals 4.
In Section 3, we utilize the field  $\mathbb{Z}_p$  to address the issue of infinite tokens. Both tasks of evaluating arithmetic expressions and solving linear equations (Section 3.1) are well-defined in this field.
A.2 Circuit complexity
In circuit complexity theory, there are several fundamental complexity classes that capture different levels of computation power. Below, we provide a brief overview of these classes; however, for a comprehensive introduction, we recommend readers refer to Arora & Barak [2].
The basic complexity classes we will discuss in this subsection are  $\mathsf{NC}^0$ ,  $\mathsf{AC}^0$ ,  $\mathsf{TC}^0$ ,  $\mathsf{NC}^1$ , and  $\mathsf{P}$ . These classes represent increasing levels of computation complexity. The relationships between these
classes can be summarized as follows:
$$
N C ^ {0} \subsetneq A C ^ {0} \subsetneq T C ^ {0} \subset N C ^ {1} \subset P
$$
Moreover, in the field of computational theory, it is widely conjectured that all subset relations in the hierarchy are proper subset relations. This means that each class is believed to capture a strictly larger set of computational problems than its predecessor in the hierarchy. However, proving some of these subset relations to be proper remains a critical open question in computational complexity theory. For example,  $\mathsf{NC}^1 = \mathsf{P}$  will imply  $\mathsf{P} = \mathsf{NP}$ , which is widely regarded as impossible but is still a celebrated open question in computer science.
To formally define these classes, we first introduce the concept of Boolean circuits. A Boolean circuit with  $n$  input bits is a directed acyclic graph (DAG), in which every node is either an input bit or an internal node representing one bit (also called a gate). The value of each internal node depends on its direct predecessors. Furthermore, several internal nodes are designated as output nodes, representing the output of the Boolean circuit. The in-degree of a node is called its fan-in number, and the input nodes have zero fan-in.
A Boolean circuit can only simulate a computation problem of a fixed number of input bits. When the input length varies, a series of distinct Boolean circuits will be required, each designed to process a specific length. In this case, circuit complexity studies how the circuit size (e.g., depth, fan-in number, width) increases with respect to the input length for a given computation problem. We now describe each complexity class as follows:
-  $\mathsf{NC}^0$  is the class of constant-depth, constant-fan-in, polynomial-sized circuits consisting of AND, OR, and NOT gates.  $\mathsf{NC}^0$  circuits is the weakest class in the above hierarchy with limited expressive power because they cannot express functions that depend on a growing number of inputs as the input size increases. For example, the basic logical-AND function with an arbitrary number of input bits is not in  $\mathsf{NC}^0$ . In [22], the authors considered a restricted version of the Transformer model with constant depth and a constant-degree sparse selection construction, which can be characterized by this complexity class.
-  $\mathsf{AC}^0$  is the class of constant-depth, unbounded-fan-in, polynomial-sized circuits consisting of AND, OR, and NOT gates, with NOT gates allowed only at the inputs. It is strictly more powerful than  $\mathsf{NC}^0$  mainly because the fan-in number can (polynomially) depend on the input length. However, there are still several fundamental Boolean functions that are not in this complexity class, such as the parity function or the majority function (see below).
-  $TC^0$  is an extension of  $AC^0$  that introduces an additional gate called MAJ (i.e., the majority). The MAJ gate takes an arbitrary number of input bits and evaluates to false when half or more of the input bits are false, and true otherwise. Previous work [42, 43] showed that the log-precision Transformer is in this class.
-  $\mathsf{NC}^1$  is a complexity class that consists of constant-fan-in, polynomial-sized circuits with a logarithmic depth of  $O(\log n)$ , where  $n$  is the input length. Similar to  $\mathsf{NC}^0$ , the basic logical gates are AND, OR, and NOT. Allowing the number of layers to depend on the input length significantly increases the expressiveness of the circuit. On the other hand, the logarithmic dependency still enables a descent parallelizability. Indeed,  $\mathsf{NC}^1$  is widely recognized as an important complexity class that captures efficiently parallelizable algorithms.
- P is the complexity class that contains problems that can be solved by a Turing machine in polynomial time. It contains a set of problems that do not have an efficient parallel algorithm. For example, the Context-Free-Grammar Membership Testing is in this class and is proved to be P-complete [33].
A.3 Log-precision
In this work, we focus on Transformers whose neuron values are restricted to be floating-point numbers of finite precision, and all computations operated on floating-point numbers will be finally truncated, similar to how a computer processes real numbers. In practice, the two most common formats to store real numbers are the fixed-point format and floating-point format (e.g., the IEEE-754 standard [31]). Likewise, there are several popular truncation approaches (also called rounding), such as round-to-the-nearest, round-to-zero, round-up, and round-down. Our results in this paper hold for both formats and all these truncation approaches.
Specifically, the log-precision assumption means that we can use  $O(\log(n))$  bits to represent a real number, where the length of the input sequence is bounded by  $n$ . For any floating-point format described above with  $O(\log(n))$  bits, an important property is that it can represent all real numbers of magnitude  $O(\mathrm{poly}(n))$  within  $O(\mathrm{poly}(1/n))$  truncation error. We next analyze how the truncation error will propagate and magnify in a log-precision Transformer from the input to the output layer. Note that since the functions represented by Transformers are continuous, the approximation error in a hidden neuron will smoothly influence the approximation error of subsequent neurons in deeper layers. This impact can be bounded by the Lipschitz constant of the Transformer, which depends on its basic layers. In particular, the softmax function (in attention) is 1-Lipschitz, the GeLU activation is 2-Lipschitz, and the Lipschitz constant of a linear layer depends on the scale of its weight parameters. Combining these together leads to the following result: given a bounded-depth log-precision Transformer of polynomial size, when all parameter values of the Transformer are further bounded by  $O(\mathrm{poly}(n))$ , all neuron values only yield an  $O(\mathrm{poly}(1/n))$  approximation error compared to the infinite-precision counterpart. Therefore, if a problem can be solved by a bounded-depth polynomial-size infinite-precision Transformer with polynomially-bounded parameters, it can also be solved by a log-precision Transformer of the same size. This finding is helpful for understanding Theorems 3.3, 3.4 and 4.7.
Finally, we point out that a key property of log-precision Transformer is that each neuron can only hold  $O(\log(n))$ -bit information and thus cannot store the full information of the entire input sequence. Therefore, the log-precision assumption captures the idea that the computation must be somehow distributed on each token, which well-resembles practical situations and the way Transformers work.
B Formal Definitions of CoT in Section 3
In this section, we will formally define the CoT derivation formats for the two math problems described in Section 3.
Arithmetic expression. In an arithmetic expression that contains operators, there exists at least one pair of neighboring numbers connected by an operator that can be calculated, which we refer to as a handle. More precisely, one can represent an arithmetic expression into a (binary) syntax tree where each number is a leaf node and each operator is an internal node that has two children. In this case, a pair of neighboring numbers is a handle if they share the same parent in the syntax tree. For instance, consider the arithmetic formula  $7 \times (6 + 5 + 4 \times 5)$ . Then,  $6 + 5$  and  $4 \times 5$  are two handles.
An important observation is that we can determine whether a pair of numbers  $a$  and  $b$  can form a handle with the operator op by examining the token before  $a$  and the token after  $b$ , where these tokens are either operators, brackets, or empty (i.e., approaching the beginning/ending of the sequence, including the equal sign '='). Specifically, given subsequence  $s_1$ $a$  op  $b$ $s_2$ , we have that  $a$  op  $b$  forms a handle iff one of the following conditions holds:
-  $\mathrm{op} \in \{+, -\}$  and  $s_1 \in \{(), \mathrm{empty}\}, s_2 \notin \{x, \div\}$ ;
-  $\mathrm{op} \in \{\times, \div\}$  and  $s_1 \notin \{\times, \div\}$ .
In the proposed chain of thought (CoT), an autoregressive Transformer calculates one handle at each step. If there are multiple handles, the leftmost handle is selected. The subsequence  $a$  op  $b$  is then replaced by the calculated result. For the case of  $s_1 = (\text{and} s_2 =)$ , there will be a pair of redundant brackets and thus the two tokens are removed. It is easy to see that the resulting sequence is still a valid arithmetic expression. By following this process, each CoT step reduces one operator and the formula is gradually simplified until there is only one number left, yielding the final answer.
System of linear equations. Assume that we have a system of  $m$  linear equations with variables  $x_{1}, x_{2}, \ldots, x_{m}$ . The  $i$ -th equation in the input sequence is grammatically written as  $a_{i1}x_{1} + a_{i2}x_{2} + \dots + a_{im}x_{m} = b_{i}$ , where  $a_{ij} \in \{0, \dots, p - 1\}$  and  $b_{i} \in \{0, \dots, p - 1\}$ . For simplicity, we do not omit the token  $a_{ij}$  or  $a_{ij}x_{j}$  in the input sequence when  $a_{ij} \in \{1, 0\}$ .
We can construct the following CoT to solve the equations by using the Gaussian elimination algorithm. At each step  $i$ , we select an equation  $k$  satisfying the following two conditions:
- The coefficient of  $x_{i}$  is nonzero.
- The coefficients of  $x_{1}, \dots, x_{i - 1}$  are all zero.
Such an equation must exist, otherwise the solution is not unique or does not exist. If there are multiple equations satisfying the above conditions, we choose the  $k$ -th equation with the smallest index  $k$ . We then swap it with equation  $i$ , so that the  $i$ -th equation now satisfies the above conditions.
We then eliminate the variable  $x_{i}$  in all other equations by leveraging equation  $i$ . Formally, denote the  $i$ -th equation at the  $i$ -th step as
$$
a _ {i i} ^ {(i)} x _ {i} + a _ {i, i + 1} ^ {(i)} x _ {i + 1} + \dots + a _ {i m} ^ {(i)} x _ {m} = b _ {i}, \tag {7}
$$
and denote the coefficient of  $x_{i}$  in the  $j$ -th equation ( $j \neq i$ ) as  $a_{ji}^{(i)}$ . We can multiply (7) by  $-(a_{ii}^{(i)})^{-1}a_{ji}^{(i)}$  and add the resulting equation to the  $j$ -th equation. This will eliminate the term  $x_{i}$  in the  $j$ -th equation. We further normalize equation  $i$  so that the coefficient  $a_{ii}^{(i)}$  becomes 1. Depending on whether  $j \leq i$  or  $j > i$ , the resulting equation in the CoT output will have the following grammatical form:
- If  $j \leq i$ , the  $j$ -th equation will be written as  $x_{j} + \tilde{a}_{j,i+1}x_{i+1} + \dots + \tilde{a}_{jm}x_{m} = \tilde{b}_{j}$ ;
- If  $j > i$ , the  $j$ -th equation will be written as  $\tilde{a}_{j,i+1}x_{i+1} + \dots + \tilde{a}_{jm}x_m = \tilde{b}_j$ .
Note that we remove all zero terms  $\tilde{a}_{jk}x_{k}$  for  $k\leq i,k\neq j$  in the CoT output and also remove the coefficient 1 in  $\tilde{a}_{kk}x_{k}$  for  $k\leq i$ , similar to how human write solutions (see Figure 1 for an illustration). However, to simplify our proof, we reserve the coefficient 0 or 1 (i.e., outputting  $0x_{k}$  or  $1x_{k}$ ) when  $k > i$  since it cannot be determined easily before computing the coefficient. The above process is repeated for  $m - 1$  steps, and after the final step we obtain the solution.
C.1 Technical lemmas for MLP
In this subsection, we will demonstrate the representation efficiency of two-layer MLPs in performing several basic operations, such as multiplication, linear transformation, conditional selection, and look-up table. These operations will serve as building blocks in performing complex tasks.
We first show that a two-layer MLP with GeLU activation can efficiently approximate the scalar multiplication, with all weights bounded by  $O(\mathrm{poly}(1 / \epsilon))$  where  $\epsilon$  is the approximation error.
Lemma C.1. Let  $f: \mathbb{R}^2 \to \mathbb{R}$  be a two-layer MLP with GeLU activation, and the hidden dimension is 4. Then, for any  $\epsilon > 0$  and  $M > 0$ , there exist MLP parameters with  $\ell_{\infty}$  norm upper bounded by  $O(\mathrm{poly}(M, 1/\epsilon))$  such that  $|f(a, b) - ab| \leq \epsilon$  holds for all  $a, b \in [-M, M]$ .
Proof. Denote the input vector to the MLP as  $(a,b)\in \mathbb{R}^2$ . After the first linear layer, it is easy to construct a weight matrix such that the hidden vector is  $\frac{1}{\lambda} (a + b, - a - b,a - b, - a + b)\in \mathbb{R}^4$ , where  $\lambda$  is an arbitrary scaling factor. Let  $\sigma$  be the GeLU activation. We can similarly construct a weight vector such that the final output of the MLP is
$$
f (a, b) = \frac {\sqrt {2 \pi} \lambda^ {2}}{8} \left(\sigma \left(\frac {a + b}{\lambda}\right) + \sigma \left(\frac {- a - b}{\lambda}\right) - \sigma \left(\frac {a - b}{\lambda}\right) - \sigma \left(\frac {- a + b}{\lambda}\right)\right).
$$
We will prove that the above MLP satisfies the theorem by picking an appropriate  $\lambda$ . By definition of GeLU activation,  $\sigma(x) = x\Phi(x)$  where  $\Phi(x)$  is the standard Gaussian cumulative distribution function. We thus have  $\sigma'(0) = 0.5$  and  $\sigma''(0) = \sqrt{2/\pi}$ . Applying Taylor's formula and assuming  $\lambda > 2M$ , we have
$$
\begin{array}{l} \left| \sigma \left(\frac {a + b}{\lambda}\right) + \sigma \left(\frac {- a - b}{\lambda}\right) - \sigma \left(\frac {a - b}{\lambda}\right) - \sigma \left(\frac {- a + b}{\lambda}\right) - \frac {8 a b}{\sqrt {2 \pi} \lambda^ {2}} \right| \\ \leq \left| \frac {1}{2} \sqrt {\frac {2}{\pi}} \left(\left(\frac {a + b}{\lambda}\right) ^ {2} + \left(\frac {- a - b}{\lambda}\right) ^ {2} - \left(\frac {a - b}{\lambda}\right) ^ {2} - \left(\frac {- a + b}{\lambda}\right) ^ {2}\right) - \frac {8 a b}{\sqrt {2 \pi} \lambda^ {2}} \right| \\ + \frac {4}{3 !} \frac {(2 M) ^ {3}}{\lambda^ {3}} \left| \max  _ {x \in [ - 1, 1 ]} \sigma^ {(3)} (x) \right| \\ = \frac {1 6 M ^ {3}}{3 \lambda^ {3}} \max _ {x \in [ - 1, 1 ]} \frac {1}{\sqrt {2 \pi}} (x ^ {3} - 4 x) \exp (- \frac {x ^ {2}}{2}) <   \frac {8 0 M ^ {3}}{3 \sqrt {2 \pi} \lambda^ {3}}. \\ \end{array}
$$