SlowGuess/NeurIPS_2023_OCR huggingface dataset:
William Merrill  
New York University  
willlm@nyu.edu
Ashish Sabharwal Allen Institute for AI ashishs@allenai.org
One way to interpret the reasoning power of transformer-based language models is to describe the types of logical rules they can resolve over some input text. Recently, Chiang et al. (2023) showed that finite-precision transformer classifiers can be equivalently expressed in a generalization of first-order logic. However, finite-precision transformers are a weak transformer variant because, as we show, a single head can only attend to a constant number of tokens and, in particular, cannot represent uniform attention. Since attending broadly is a core capability for transformers, we ask whether a minimally more expressive model that can attend universally can also be characterized in logic. To this end, we analyze transformers whose forward pass is computed in  $\log n$  precision on contexts of length  $n$ . We prove any log-precision transformer classifier can be equivalently expressed as a first-order logic sentence that, in addition to standard universal and existential quantifiers, may also contain majority-vote quantifiers. This is the tightest known upper bound and first logical characterization of log-precision transformers.
Any log-precision transformer can be re-expressed as a sentence in  $\mathsf{FO}(\mathsf{M})$  logic, e.g.:
$$
\begin{array}{c} M i.   \mathsf {a} (i) \wedge M j.   \mathsf {b} (j) \wedge \neg \exists k, \ell .   (\mathsf {a} (k) \wedge \mathsf {b} (\ell) \wedge \ell <   k) \\ (m   \mathsf {a} s f o l l o w e d b y m   \mathsf {b} s, i.e.,   \mathsf {a} ^ {m}   \mathsf {b} ^ {m}) \\ \text {a a a a b b b b} \checkmark \quad \text {a a a b b b b} X \quad \text {b a a a a b b b} X \end{array}
$$
Figure 1: A first-order logic with majority  $(\mathsf{FO}(\mathsf{M}))$  sentence for  $\mathsf{a}^m\mathsf{b}^m$ . In addition to standard  $\forall$  and  $\exists$  quantifiers over string indices,  $\mathsf{FO}(\mathsf{M})$  allows majority quantifiers (M) that take a majority-vote across indices.  $\mathsf{a}(i)$  indicates whether token  $i$  is a (and analogously for b). We prove  $\mathsf{FO}(\mathsf{M})$  can express any function computed by a log-precision transformer.
The incredible success of deep learning models, especially very large language and vision transformers with hundreds of billions of parameters (Brown et al., 2020; Thoppilan et al., 2022), has come at the cost of increasingly limited understanding of how these models actually work and when they might fail. This raises many concerns, such as around their safe deployment, fairness, and accountability. Does the inner working of a transformer defy description in a simpler symbolic system that we can better understand? Or can transformer computation be described using a familiar symbolic formalism? Understanding how to view the reasoning process of a transformer in terms of logic could potentially expand our ability to formally reason about their behavior over large domains of inputs.
Chiang et al. (2023) provide a partial answer to this question, showing that any finite-precision transformer classifier can be expressed as a sentence in a variant of first-order logic with counting
quantifiers and modular arithmetic over input position indices. Specifically, counting quantifiers take the form  $\exists^{=x}i:\phi(i)$  where  $x$  is a count variable and  $i$  is a position index. They show that there exists a single sentence in this logic that computes the output of the transformer for any input string of any length. This is a powerful result because it shows that a simple logical formalism is fully sufficient to describe all the complexity of a massive finite-precision transformer. It also provides an upper bound on finite-precision transformers: any function that cannot be defined in first-order counting logic with modular indexing cannot be expressed by the transformer.
However, Chiang et al.'s result is not fully general because it relies on the transformer precision being fixed with respect to the transformer's context length. More generally, as we will demonstrate in Section 3, finite-precision transformers are a fundamentally weak variant of transformers: crucially, cannot express uniform attention patterns, which are a core algorithmic primitive of transformers (Weiss et al., 2018). In fact, we show that they can only attend to a constant number of input positions, which may be seen as a rather limited generalization of hard attention. For example, Chiang et al. show that their logic for finite-precision transformers cannot recognize  $\mathsf{a}^m\mathsf{b}^m$ , whereas in practice, transformers can (Bhattamishra et al., 2020). This motivates studying a formal model of transformers where precision grows with context length (which we formalize as log-precision), making it possible to capture uniform attention as well as other broad attention patterns. This is useful both for recognizing  $\mathsf{a}^m\mathsf{b}^m$  and more generally for reasoning globally over the input.
We demonstrate that log-precision transformer classifiers can also be expressed as sentences in a simple logic: first-order logic with majority, or FO(M), over inputs strings (Barrington et al., 1990). In addition to standard existential and universal quantifiers, FO(M) has majority quantifiers that return true iff more than half the propositions they quantify are true. It also allows comparing input positions (e.g.,  $\ell < k$  in Figure 1) and accessing their individual bits. Our main result is as follows:
Theorem 1 (Informal version of Theorem 2). For any log-precision transformer  $\mathcal{T}$ , there exists an FO(M) sentence  $\phi$  that computes the same function as  $\mathcal{T}$ , i.e.,  $\phi(x) = \mathcal{T}(x)$  for any input string  $x$ .
Upper bound. Theorem 2 shows transformers with more than finite precision can also be expressed in a simple extension of first-order logic, going beyond Chiang et al. (2023)'s result. On the other hand,  $\mathsf{FO}(\mathsf{M})$  is a strict superset of Chiang et al.'s counting logic; it can simulate counting quantifiers (see Section 2.2) and allows non-modular position comparisons. Thus, handling a more general class of transformers powerful enough to express uniform attention slightly weakens the bound.
Still, our result constitutes (to our knowledge) the tightest upper bound on log-precision transformers and the first defined in terms of logic, building on a line of complexity-theoretic work analyzing the power of transformers (Hahn, 2020; Merrill et al., 2022; Liu et al., 2023; Merrill & Sabharwal, 2023). In particular,  $\mathsf{FO}(\mathsf{M})$  strengthens the upper bound of log-space-uniform  $\mathsf{T}\mathsf{C}^0$  by Merrill & Sabharwal (2023). The refined bound adds to the limitations of transformers identified by Merrill & Sabharwal (2023): for example, it establishes unconditionally that log-precision transformers cannot compute boolean matrix permanents, and shows that, in a certain formal sense, integer division and matching parentheses are among the formally hardest problems that transformers can solve (see Section 4).<sup>3</sup>
Mechanistic interpretability. Beyond providing an upper bound on the reasoning problems solvable by transformers, we believe Theorem 1 could guide the design of "transformer-complete" programming languages similar in spirit to RASP (Weiss et al., 2018). RASP is a declarative programming language designed to capture transformer computation, and Lindner et al. (2023) implement a compiler from RASP into transformers. Unlike RASP, FO(M) can provably express any transformer (Theorem 1), which we believe justifies using it (or an equivalent but more user-friendly variant) as a target language for programs extracted from transformers.
Similar to a decision tree, an  $\mathsf{FO}(\mathsf{M})$  sentence has the interpretable property that each sub-sentence corresponds to a constraint on input (see Figure 1). In contrast, the internal modules of a transformer or circuit do not satisfy this since they map between arbitrary latent spaces. We speculate this property
<sup>1</sup>Hard attention is provably substantially weaker than general attention (Hao et al., 2022; Merrill et al., 2022).
${}^{2}$  Technically,the empirical results of Bhattamishra et al. (2020) are for  ${\mathbf{a}}^{m}{\mathbf{b}}^{m}{\mathbf{c}}^{m}$  ,a harder variant of  ${\mathbf{a}}^{m}{\mathbf{b}}^{m}$  .
3To be clear, Theorem 1 is one-sided: every transformer can be expressed as an  $\mathsf{FO}(\mathsf{M})$  sentence, but not necessarily the other way. Moreover, we believe that many  $\mathsf{FO}(\mathsf{M})$  sentences cannot be expressed by transformers. An exact logical characterization of transformers remains an open problem.
could facilitate interpreting models by translating them to  $\mathsf{FO}(\mathsf{M})$ , though a careful exploration of the algorithmic and HCI aspects of this idea lies outside the current paper's theoretical scope.
Contributions. Our results shed new light on how to view the computation inside transformers in terms of logic. Specifically, our main contributions are to prove the following:
1. Fixed-precision transformers can only attend to a fixed number of tokens, and those with precision less than  $\log \log n$  cannot uniformly attend over length- $n$  contexts (Proposition 1).
2. Log-precision transformer classifiers can be expressed as sentences in  $\mathsf{FO}(\mathsf{M})$  (Theorem 2).
2 Preliminaries: Transformers and FO(M)
Let  $\Sigma$  be a finite alphabet. We denote by  $^*$  the Kleene star operator, i.e., for a set  $X$ ,  $X^{*} = \bigcup_{n = 0}^{\infty}X^{n}$ . We will view transformers and  $\mathsf{FO}(\mathsf{M})$  sentences both as functions from  $\Sigma^{*}\rightarrow \{0,1\}$ , and show that any function a transformer computes can also be computed by an  $\mathsf{FO}(\mathsf{M})$  sentence.
2.1 Transformers
We view the transformer precision  $p$  as a function of the context length  $n$ , writing  $p(n)$  where appropriate. Let  $\mathbb{D}_p$  be the datatype of  $p$ -precision floats, i.e., tuples  $\langle m, e \rangle$  where  $m, e$  are signed integers together taking  $p$  bits. Using  $|x|$  to mean the size of integer  $x$ , a float represents the value  $m \cdot 2^{e - |m| + 1}$ . Following Appendix A of Merrill & Sabharwal (2023), we define  $p$ -truncated addition  $(+, \sum)$ , multiplication  $(\cdot)$ , and division  $(/)$  over  $\mathbb{D}_p$ . We now define a transformer encoder binary classifier over  $\mathbb{D}_p$ , largely adopting Merrill & Sabharwal's notation.
Definition 1. A  $p$ -precision transformer  $\mathcal{T}$  with  $h$  heads,  $d$  layers, model dimension  $m$  (divisible by  $h$ ), and feedforward width  $w$  is specified by:
1. An embedding function  $\phi : \Sigma \times \mathbb{N} \to \mathbb{D}_p^m$  whose form is defined in Appendix C.1;
2. For each  $1 \leq \ell \leq d$  and  $1 \leq k \leq h$ , a head similarity function  $s_k^\ell: \mathbb{D}_p^m \times \mathbb{D}_p^m \to \mathbb{D}_p$  whose form is defined in Appendix C.2;
3. For each  $1 \leq \ell \leq d$  and  $1 \leq k \leq h$ , a head value function  $v_k^\ell : \mathbb{D}_p^m \to \mathbb{D}_p^{m/h}$  whose form is defined in Appendix C.2;
4. For each  $1 \leq \ell \leq d$ , an activation function  $f^{\ell}:(\mathbb{D}_p^{m / h})^h\times \mathbb{D}_p^m\to \mathbb{D}_p^m$  whose form is defined in Appendix C.3 and implicitly uses the feedforward dimension  $w$ ;
5. An output classifier head  $\kappa : \mathbb{D}_p^m \to \{0, 1\}$  whose form is defined in Appendix C.4.
Definition 2. We define the transformer computation and output as a function of an input  $x \in \Sigma^n$ .
1. Embeddings: For  $1 \leq i \leq n$ ,  $\mathbf{h}_i^0 = \phi(x_i, i)$ .<sup>6</sup>
2. Self Attention: For  $0 \leq \ell \leq d - 1$ , (multihead) self-attention block  $\ell + 1$  computes  $h$  attention heads:
$$
\mathbf {a} _ {i, k} ^ {\ell + 1} = \sum_ {j = 1} ^ {n} \frac {s _ {k} ^ {\ell + 1} \left(\mathbf {h} _ {i} ^ {\ell} , \mathbf {h} _ {j} ^ {\ell}\right)}{Z _ {i , k}} \cdot v _ {k} ^ {\ell + 1} \left(\mathbf {h} _ {j} ^ {\ell}\right), \quad \text {w h e r e} Z _ {i, k} = \sum_ {j = 1} ^ {n} s _ {k} ^ {\ell + 1} \left(\mathbf {h} _ {i} ^ {\ell}, \mathbf {h} _ {j} ^ {\ell}\right).
$$
3. Activation Block: For  $0 \leq \ell \leq d - 1$ , activation block  $\ell + 1$  aggregates the head outputs to produce  $\mathbf{h}^{\ell + 1}$ :
$$
\mathbf {h} _ {i} ^ {\ell + 1} = f ^ {\ell + 1} (\mathbf {a} _ {i, 1} ^ {\ell + 1}, \dots , \mathbf {a} _ {i, h} ^ {\ell + 1}, \mathbf {h} _ {i} ^ {\ell}).
$$
4. Classifier Head: The network prediction on  $x \in \Sigma^n$  is  $\kappa(\mathbf{h}_n^d)$ .
$^{4}\langle 101,010\rangle$  represents  $1.01_{2} \times 2^{10_{2}}$ . This is closer to the IEEE standard than the  $m \cdot 2^{e}$  semantics used in Merrill & Sabharwal (2023), letting us define the minimum representable float more realistically in Proposition 1.
<sup>5</sup>Increasing the classifier's output space arity (e.g., a transformer that predicts the next token) or switching to causal attention of a decoder-only model would not change our results. However, our proof no longer goes through if the decoder can generate tokens that get added to the input at the next step (cf. Pérez et al., 2019).
$^6\phi$ , like  $p$ , is actually a function of the context length  $n$ , and Appendix C.1 enforces that  $\phi$  is computable in  $\mathrm{O}(\log n)$  time, as standard choices of positional embeddings would satisfy.
We say  $\mathcal{T}(x) = \kappa (\mathbf{h}_{|x|}^{d})$  and  $L_{\mathcal{T}}$  is the language of  $x\in \Sigma^{*}$  such that  $\mathcal{T}(x) = 1$ . We refer to  $\phi, s_k^\ell, v_h^\ell, f^\ell$ , and  $\kappa$  as the core functions in  $\mathcal{T}$ , and to embeddings, self attention, activation, and the classifier head as the components of  $\mathcal{T}$ . We write  $\theta_{\mathcal{T}}$  for the concatenated vector of parameters for the functions  $\phi, s_k^\ell, v_h^\ell, f^\ell$ , and  $\kappa$ , for all  $1\leq \ell \leq d$  and  $1\leq k\leq h$ .
We define a log-precision transformer as one where  $p$  is at most  $O(\log n)$  and is a "simple" function, i.e., computable in  $O(\log n)$  time. In our model, the weights  $\theta_{\mathcal{T}}$  defining  $\mathcal{T}$  are fixed, but the precision  $p$  used to compute the forward pass can depend on  $n$  (see Footnote 13 for a generalization).
2.2 First-Order Logic with Majority
As we will show, transformers can be translated into sentences in  $\mathsf{FO}(\mathsf{M})$ . But what do such sentences look like? Informally,  $\mathsf{FO}(\mathsf{M})$  is first-order logic extended to also have majority (M) quantifiers. Following Barrington et al. (1990), our sense of  $\mathsf{FO}(\mathsf{M})$  takes strings in  $\Sigma^{*}$  as input and returns 0 or 1 to define a formal language. In this setting, quantifiers range over indices (positions) into the string. Predicates can be applied to the variables introduced by these quantifiers.
Definition 3 (FO(M) index). Indices in FO(M) are integers denoting positions in the input string:
1. The constant 1, representing the first token's position.
2. The constant  $n$ , representing the last token's position.
3. Strings (e.g.,  $i, j, k$ ) representing variables ranging over positions 1 to  $n$ .
4. Any index built by applying addition or subtraction to other indices.<sup>7</sup>
Definition 4 (FO(M) formula). Formulas in FO(M) are constructed as follows:
1. Let  $\Sigma$  be a finite alphabet. For each  $\sigma \in \Sigma$  and any index  $i$ ,  $\sigma(i)$ , e.g.,  $\mathsf{a}(i)$ , is a formula that is true if the  $i$ -th input token is  $\sigma$ .<sup>9</sup>
2. For any indices  $i, j$ , the formula  $\operatorname{bit}(i, j)$  returns the  $j$ -th bit of the binary expansion of  $i$ .<sup>10</sup>
3. For two indices  $i, j, i = j, i \leq j$ , and  $i \geq j$  are formulas with their conventional semantics.
4. For two formulas  $\phi, \psi, \phi \wedge \psi$  and  $\phi \lor \psi$  are formulas with their conventional semantics.
5. For any formula  $\phi$  (which may refer to  $i$ ), the following are valid formulas:
(a)  $\exists i.\phi$  means some value of  $i$  in  $[1,n]$  makes  $\phi$  true.
(b)  $\forall i.\phi$  means all values of  $i$  in  $[1,n]$  make  $\phi$  true.
(c)  $M_i$ .  $\phi$  means  $\geq n / 2$  values of  $i$  in  $[1, n]$  make  $\phi$  true.
We use parentheses where necessary to disambiguate the order of operations. General formulas may contain free (i.e., unbound) variables: e.g.,  $\forall i$ .  $i = j$ . A sentence is an  $\mathsf{FO}(\mathsf{M})$  formula  $\phi$  with no free variables. Sentences represent functions from  $\Sigma^{*}$  to  $\{0,1\}$  and thus define a formal language.[11]
Extensions. Beyond Definition 4,  $\mathsf{FO}(\mathsf{M})$  can express counting and threshold quantifiers in terms of majority quantifiers (Barrington et al., 1990). Given a formula  $\phi$ , a counting quantifier creates a new formula  $\exists^k i : \phi$  that is true iff  $\phi$  is true across exactly  $k$  values of  $i$ . Threshold quantifiers  $\exists^{\leq k}$  and  $\exists^{\geq k}$  work similarly but check if  $\phi$  is true for at least or at most  $k$  values of  $i$ . In addition, we show in Appendix A that  $\mathsf{FO}(\mathsf{M})$  can express conditional majority quantifiers, which create a formula  $\mathsf{Mi} : \phi[\psi]$  that is true iff  $\psi$  is true for at least half the values of  $i$  that make  $\phi$  true.
2.2.1 Examples
To illustrate the formalism, we provide example languages definable in  $\mathsf{FO}(\mathsf{M})$  with  $\Sigma = \{\mathsf{a},\mathsf{b}\}$ . First, we show two languages that do not require majority quantifiers to express:
Example 1 (Bigram matching). Strings containing the bigram  $\mathsf{ab}$ :  $\exists i[\mathsf{a}(i)\wedge \mathsf{b}(i + 1)]$ .
Example 2 (Skip-bigram matching). Strings containing the long-distance pattern  $\mathbf{a} \ldots \mathbf{b}$  (cf. "induction heads" of Elhage et al. 2021):  $\exists i [\mathsf{b}(i) \wedge \exists j [j \leq i \wedge \mathsf{a}(j)]]$ .
7 Barrington et al. (1990) did not introduce this as a primitive, but it can be simulated using the  $\leq$  predicate.
<sup>8</sup>We write parentheses to indicate the order of operations.
$^9$  Barrington et al. (1990) define  $Q_{b}(i)$  for  $b \in \{0, 1\}$ . We generalize this to an arbitrary vocabulary  $\Sigma$  by assuming each token is one-hot-encoded:  $\sigma(i) = Q_1(|\Sigma|i + s)$  where  $s$  is the index of  $\sigma$  in the vocabulary.
This predicate is included in the logic for technical reasons; see Barrington et al. (1990).
One can also take multiple sub-sentences within  $\phi$  to be labeled as ordered outputs, thus allowing  $\phi$  to be a function from  $\Sigma^{*}$  to  $\{0,1\}^k$  for some fixed constant  $k$ .
In contrast, Example 3 is a simple example that requires majority quantifiers (Furst et al., 1984):
Example 3 (Majority). Strings with more b's than a's:  $Mi[b(i)]$ .
Figure 1 showed how  $\mathsf{FO}(\mathsf{M})$  can be used to recognize patterns like  $\mathsf{a}^m\mathsf{b}^m$ . A similar idea can be used to model parentheses matching (Barrington et al., 1990):
Example 4 (1-Dyck). The well-balanced parentheses language (with a opening and b closing):
$$
\forall i. \left(\exists a, b. \left(\left(\exists^ {a} j: \mathbf {a} (j) \wedge j \leq i\right) \wedge \left(\exists^ {b} j: \mathsf {b} (j) \wedge j \leq i\right) \wedge b \leq a\right)\right) \wedge M i. \mathbf {a} (i) \wedge M j. \mathsf {b} (j).
$$
Example 5 (Integer Arithmetic). Iterated addition (i.e., summing  $n$ $n$ -bit numbers), iterated multiplication, and division (Hesse, 2001) can all be expressed in  $\mathsf{FO}(\mathsf{M})$ .
3 Finite Precision Transformers Cannot Attend Universally
Attention heads that spread attention weight uniformly across inputs have been observed in transformer LMs (Merrill et al., 2021) and make soft attention fundamentally more powerful than hard attention (Hao et al., 2022; Merrill et al., 2022). In particular, uniform attention is an important primitive that transformers can use to solve tasks involving counting (Bhattachamishra et al., 2020; Chiang et al., 2023), taking majority votes (Merrill et al., 2022), and matching parentheses or sorting (Weiss et al., 2021). A transformer with sufficient precision can easily implement uniform attention by setting the keys and queries across all positions to be constant. However, attention heads with finite precision cannot represent uniform attention over long sequences as a consequence of the following:
Proposition 1. Let  $\mathbf{a} \in \mathbb{R}^n$  s.t.  $\sum_{i=1}^{n} a_i = 1$  and  $\tilde{\mathbf{a}}$  its nearest  $p$ -precision float approximation.
1. Then the number of nonzero entries of  $\tilde{\mathbf{a}}$  is upper bounded by its precision: specifically,  $\tilde{\mathbf{a}}$  has at most  $2^{2^p}$  nonzero entries.
2. Moreover, if  $p < \log \log n$  and  $\mathbf{a}$  is uniform (i.e.,  $a_{i} = 1 / n$ ), then  $\tilde{\mathbf{a}} = \vec{0}$ .
Proof. The smallest positive value representable by a  $p$ -precision float is  $2^{-(p_m - 2 + 2^{pe - 1})}$  which is bounded below by  $2^{-2^p + 1}$ . Letting  $k = 2^{2^p}$ , it holds that  $2^{-2^p + 1} = 2 / k$ . So if  $\tilde{a}_i$  gets the minimum value, then  $a_i \geq 1 / k$ . Since  $\sum_{i} a_i = 1$ , there can be at most  $k$  indices satisfying this property. This implies there can be at most  $k$  nonzero entries in  $\tilde{\mathbf{a}}$ . If  $n > k$  and  $\mathbf{a}$  is uniform,  $1 / n$  is less than half of the minimum representable value of  $2 / k$ . Thus,  $\tilde{\mathbf{a}} = \vec{0}$ .
Proposition 1 says that fixed-precision transformers are artificially limited because they can only attend over bounded-length windows, making them similar to hard-attention transformers (Hao et al., 2022). Moreover, they cannot compute uniform attention over contexts of length  $n$  with less than  $\log \log n$  precision. This explains why Chiang et al. (2023) prove finite-precision transformers provably cannot recognize  $\mathsf{a}^m\mathsf{b}^m$ , while in practice transformers have been shown to learn even its harder variant  $\mathsf{a}^m\mathsf{b}^m\mathsf{c}^m$  even with long context lengths (Bhattamishra et al., 2020). In essence, their upper bound only applies in the asymptotic regime when  $n > 2^{2^p}$ .
In contrast, transformers in practice have enough precision both to compute uniform attention and recognize  $\mathsf{a}^m\mathsf{b}^m$  on practical context lengths. More concretely, the bfloat16 representation allows uniform attention over  $2^{6 + 2^7}\approx 10^{42}$  tokens and normal float16<sup>12</sup> allows  $2^{10 + 2^4}\approx 10^8$  tokens, both well above the typical context window of transformers. This motivates a formal model of transformers with enough precision to compute uniform attention and recognize languages such as  $\mathsf{a}^m\mathsf{b}^m$ .
4 Main Result: Expressing Log-Precision Transformers in FO(M)
By Proposition 1, precision must grow with the context length  $n$  ( $p > \log \log n$ ) for a transformer to compute uniform attention and other attention patterns with unbounded range, like practical transformers. In this paper, we analyze any transformer with up to  $O(\log n)$  precision. We show that any function computable by log-precision transformers can be expressed in  $\mathsf{FO}(\mathsf{M})$ :
<sup>12</sup>We account for the division of  $p$  into  $p_m$  and  $p_e$  rather than treating them together. Our minimum value differs slightly from numpy but is on the same order of magnitude. Moving to float8 lowers the length upper bound for uniform attention to  $2^{3 + 2^3} \approx 2048$ , which suggests float8 LMs will have limited length generalization.
Theorem 2. Let  $\mathcal{T}$  be a log-precision transformer with a parameter vector  $\theta_{\mathcal{T}}$  fixed for all context lengths  $n$ . Then, there exists an  $\mathsf{FO}(\mathsf{M})$  sentence  $\phi$  that computes the same function as  $\mathcal{T}$ , i.e.,  $\phi(x) = \mathcal{T}(x)$  for any input string  $x$ .