Title: A Logic for Expressing Log-Precision Transformers
Venue: NeurIPS 2023
Accepted: yes
Keywords: transformers

logic

reasoning

circuit complexity

mechanistic interpretability
Abstract: One way to interpret the reasoning power of transformer-based language models is to describe the types of logical rules they can resolve over some input text. Recently, Chiang et al. (2023) showed that finite-precision transformer classifiers can be equivalently expressed in a generalization of first-order logic. However, finite-precision transformers are a weak transformer variant because, as we show, a single head can only attend to a constant number of tokens and, in particular, cannot represent uniform attention. Since attending broadly is a core capability for transformers, we ask whether a minimally more expressive model that can attend universally can also be characterized in logic. To this end, we analyze transformers whose forward pass is computed in $\log n$ precision on contexts of length $n$. We prove any log-precision transformer classifier can be equivalently expressed as a first-order logic sentence that, in addition to standard universal and existential quantifiers, may also contain majority-vote quantifiers. This is the tightest known upper bound and first logical characterization of log-precision transformers.

A Logic for Expressing Log-Precision Transformers

Abstract

One way to interpret the reasoning power of transformer-based language models is to describe the types of logical rules they can resolve over some input text. Recently, Chiang et al. (2023) showed that finite-precision transformer classifiers can be equivalently expressed in a generalization of first-order logic. However, finite-precision transformers are a weak transformer variant because, as we show, a single head can only attend to a constant number of tokens and, in particular, cannot represent uniform attention. Since attending broadly is a core capability for transformers, we ask whether a minimally more expressive model that can attend universally can also be characterized in logic. To this end, we analyze transformers whose forward pass is computed in log n precision on contexts of length n . We prove any log-precision transformer classifier can be equivalently expressed as a first-order logic sentence that, in addition to standard universal and existential quantifiers, may also contain majority-vote quantifiers. This is the tightest known upper bound and first logical characterization of log-precision transformers.

Any log-precision transformer can be re-expressed as a sentence in FO ( M ) logic, e.g.:

Figure 1: A first-order logic with majority ( FO ( M ) ) sentence for a m b m . In addition to standard ∀ and ∃ quantifiers over string indices, FO ( M ) allows majority quantifiers ( M ) that take a majority-vote across indices. a ( i ) indicates whether token i is a (and analogously for b ). We prove FO ( M ) can express any function computed by a log-precision transformer.

1 Introduction

The incredible success of deep learning models, especially very large language and vision transformers with hundreds of billions of parameters (Brown et al., 2020; Thoppilan et al., 2022), has come at the cost of increasingly limited understanding of how these models actually work and when they might fail. This raises many concerns, such as around their safe deployment, fairness, and accountability. Does the inner working of a transformer defy description in a simpler symbolic system that we can better understand? Or can transformer computation be described using a familiar symbolic formalism? Understanding how to view the reasoning process of a transformer in terms of logic could potentially expand our ability to formally reason about their behavior over large domains of inputs.

Chiang et al. (2023) provide a partial answer to this question, showing that any finite-precision transformer classifier can be expressed as a sentence in a variant of first-order logic with counting

quantifiers and modular arithmetic over input position indices. Specifically, counting quantifiers take the form ∃ = x i : φ ( i ) where x is a count variable and i is a position index. They show that there exists a single sentence in this logic that computes the output of the transformer for any input string of any length. This is a powerful result because it shows that a simple logical formalism is fully sufficient to describe all the complexity of a massive finite-precision transformer. It also provides an upper bound on finite-precision transformers: any function that cannot be defined in first-order counting logic with modular indexing cannot be expressed by the transformer.

However, Chiang et al.'s result is not fully general because it relies on the transformer precision being fixed with respect to the transformer's context length. More generally, as we will demonstrate in Section 3, finite-precision transformers are a fundamentally weak variant of transformers: crucially, cannot express uniform attention patterns, which are a core algorithmic primitive of transformers (Weiss et al., 2018). In fact, we show that they can only attend to a constant number of input positions, which may be seen as a rather limited generalization of hard attention. 1 For example, Chiang et al. show that their logic for finite-precision transformers cannot recognize a m b m , whereas in practice, transformers can (Bhattamishra et al., 2020). 2 This motivates studying a formal model of transformers where precision grows with context length (which we formalize as log-precision ), making it possible to capture uniform attention as well as other broad attention patterns. This is useful both for recognizing a m b m and more generally for reasoning globally over the input.

We demonstrate that log-precision transformer classifiers can also be expressed as sentences in a simple logic: first-order logic with majority , or FO ( M ) , over inputs strings (Barrington et al., 1990). In addition to standard existential and universal quantifiers, FO ( M ) has majority quantifiers that return true iff more than half the propositions they quantify are true. It also allows comparing input positions (e.g., /lscript < k in Figure 1) and accessing their individual bits. Our main result is as follows:

Theorem 1 (Informal version of Theorem 2) . For any log-precision transformer T , there exists an FO ( M ) sentence φ that computes the same function as T , i.e., φ ( x ) = T ( x ) for any input string x .

Upper bound. Theorem 2 shows transformers with more than finite precision can also be expressed in a simple extension of first-order logic, going beyond Chiang et al. (2023)'s result. On the other hand, FO ( M ) is a strict superset of Chiang et al.'s counting logic; it can simulate counting quantifiers (see Section 2.2) and allows non-modular position comparisons. Thus, handling a more general class of transformers powerful enough to express uniform attention slightly weakens the bound.

Still, our result constitutes (to our knowledge) the tightest upper bound on log-precision transformers and the first defined in terms of logic, building on a line of complexity-theoretic work analyzing the power of transformers (Hahn, 2020; Merrill et al., 2022; Liu et al., 2023; Merrill & Sabharwal, 2023). In particular, FO ( M ) strengthens the upper bound of log-space-uniform TC 0 by Merrill & Sabharwal (2023). The refined bound adds to the limitations of transformers identified by Merrill & Sabharwal (2023): for example, it establishes unconditionally that log-precision transformers cannot compute boolean matrix permanents, and shows that, in a certain formal sense, integer division and matching parentheses are among the formally hardest problems that transformers can solve (see Section 4). 3

Mechanistic interpretability. Beyond providing an upper bound on the reasoning problems solvable by transformers, we believe Theorem 1 could guide the design of 'transformer-complete' programming languages similar in spirit to RASP (Weiss et al., 2018). RASP is a declarative programming language designed to capture transformer computation, and Lindner et al. (2023) implement a compiler from RASP into transformers. Unlike RASP, FO ( M ) can provably express any transformer (Theorem 1), which we believe justifies using it (or an equivalent but more user-friendly variant) as a target language for programs extracted from transformers.

Similar to a decision tree, an FO ( M ) sentence has the interpretable property that each sub-sentence corresponds to a constraint on input (see Figure 1). In contrast, the internal modules of a transformer or circuit do not satisfy this since they map between arbitrary latent spaces. We speculate this property

could facilitate interpreting models by translating them to FO ( M ) , though a careful exploration of the algorithmic and HCI aspects of this idea lies outside the current paper's theoretical scope.

Contributions. Our results shed new light on how to view the computation inside transformers in terms of logic. Specifically, our main contributions are to prove the following:

Fixed-precision transformers can only attend to a fixed number of tokens, and those with precision less than log log n cannot uniformly attend over lengthn contexts (Proposition 1).

Log-precision transformer classifiers can be expressed as sentences in FO ( M ) (Theorem 2).

2 Preliminaries: Transformers and FO ( M )

Let Σ be a finite alphabet. We denote by ∗ the Kleene star operator, i.e., for a set X , X ∗ = ⋃ ∞ n =0 X n . We will view transformers and FO ( M ) sentences both as functions from Σ ∗ → { 0 , 1 } , and show that any function a transformer computes can also be computed by an FO ( M ) sentence.

2.1 Transformers

We view the transformer precision p as a function of the context length n , writing p ( n ) where appropriate. Let D p be the datatype of p -precision floats, i.e., tuples 〈 m,e 〉 where m,e are signed integers together taking p bits. Using | x | to mean the size of integer x , a float represents the value m · 2 e -| m | +1 . 4 Following Appendix A of Merrill & Sabharwal (2023), we define p -truncated addition ( + , ∑ ), multiplication ( · ), and division ( / ) over D p . We now define a transformer encoder binary classifier over D p , largely adopting Merrill & Sabharwal's notation. 5

Definition 1. A p -precision transformer T with h heads, d layers, model dimension m (divisible by h ), and feedforward width w is specified by:

An embedding function φ : Σ × N → D m p whose form is defined in Appendix C.1; 6

For each 1 ≤ /lscript ≤ d and 1 ≤ k ≤ h , a head similarity function s /lscript k : D m p × D m p → D p whose form is defined in Appendix C.2;

For each 1 ≤ /lscript ≤ d and 1 ≤ k ≤ h , a head value function v /lscript k : D m p → D m/h p whose form is defined in Appendix C.2;

For each 1 ≤ /lscript ≤ d , an activation function f /lscript : ( D m/h p ) h × D m p → D m p whose form is defined in Appendix C.3 and implicitly uses the feedforward dimension w ;

An output classifier head κ : D m p → { 0 , 1 } whose form is defined in Appendix C.4.

Embeddings: For 1 ≤ i ≤ n , h 0 i = φ ( x i , i ) . 6

Definition 2. We define the transformer computation and output as a function of an input x ∈ Σ n .

Self Attention: For 0 ≤ /lscript ≤ d -1 , (multihead) self-attention block /lscript + 1 computes h attention heads:

Activation Block: For 0 ≤ /lscript ≤ d -1 , activation block /lscript +1 aggregates the head outputs to produce h /lscript +1 :

Classifier Head: The network prediction on x ∈ Σ n is κ ( h d n ) .

We say T ( x ) = κ ( h d | x | ) and L T is the language of x ∈ Σ ∗ such that T ( x ) = 1 . We refer to φ , s /lscript k , v /lscript h , f /lscript , and κ as the core functions in T , and to embeddings, self attention, activation, and the classifier head as the components of T . We write θ T for the concatenated vector of parameters for the functions φ , s /lscript k , v /lscript h , f /lscript , and κ , for all 1 ≤ /lscript ≤ d and 1 ≤ k ≤ h .

We define a log-precision transformer as one where p is at most O(log n ) and is a 'simple' function, i.e., computable in O(log n ) time. In our model, the weights θ T defining T are fixed, but the precision p used to compute the forward pass can depend on n (see Footnote 13 for a generalization).

2.2 First-Order Logic with Majority

As we will show, transformers can be translated into sentences in FO ( M ) . But what do such sentences look like? Informally, FO ( M ) is first-order logic extended to also have majority ( M ) quantifiers. Following Barrington et al. (1990), our sense of FO ( M ) takes strings in Σ ∗ as input and returns 0 or 1 to define a formal language. In this setting, quantifiers range over indices (positions) into the string. Predicates can be applied to the variables introduced by these quantifiers.

Definition 3 ( FO ( M ) index) . Indices in FO ( M ) are integers denoting positions in the input string:

The constant 1 , representing the first token's position.

The constant n , representing the last token's position.

Strings (e.g., i, j, k ) representing variables ranging over positions 1 to n .

Any index built by applying addition or subtraction to other indices. 7

Definition 4 ( FO ( M ) formula) . Formulas in FO ( M ) are constructed as follows: 8

Let Σ be a finite alphabet. For each σ ∈ Σ and any index i , σ ( i ) , e.g., a ( i ) , is a formula that is true if the i -th input token is σ . 9

For any indices i, j , the formula bit ( i, j ) returns the j -th bit of the binary expansion of i . 10

For two indices i, j , i = j , i ≤ j , and i ≥ j are formulas with their conventional semantics.

For any formula φ (which may refer to i ), the following are valid formulas:

For two formulas φ , ψ , φ ∧ ψ and φ ∨ ψ are formulas with their conventional semantics.

(a) ∃ i. φ means some value of i in [1 , n ] makes φ true.

(c) M i. φ means ≥ n/ 2 values of i in [1 , n ] make φ true.

(b) ∀ i. φ means all values of i in [1 , n ] make φ true.

We use parentheses where necessary to disambiguate the order of operations. General formulas may contain free (i.e., unbound) variables: e.g., ∀ i. i = j . A sentence is an FO ( M ) formula φ with no free variables. Sentences represent functions from from Σ ∗ to { 0 , 1 } and thus define a formal language. 11

Extensions. Beyond Definition 4, FO ( M ) can express counting and threshold quantifiers in terms of majority quantifiers (Barrington et al., 1990). Given a formula φ , a counting quantifier creates a new formula ∃ k i : φ that is true iff φ is true across exactly k values of i . Threshold quantifiers ∃ ≤ k and ∃ ≥ k work similarly but check if φ is true for at least or at most k values of i . In addition, we show in Appendix A that FO ( M ) can express conditional majority quantifiers, which create a formula M i : φ [ ψ ] that is true iff ψ is true for at least half the values of i that make φ true.

2.2.1 Examples

To illustrate the formalism, we provide example languages definable in FO ( M ) with Σ = { a , b } . First, we show two languages that do not require majority quantifiers to express:

Example 1 (Bigram matching) . Strings containing the bigram ab : ∃ i [ a ( i ) ∧ b ( i +1)] .

Example 2 (Skip-bigram matching) . Strings containing the long-distance pattern a . . . b (cf. 'induction heads' of Elhage et al. 2021): ∃ i [ b ( i ) ∧ ∃ j [ j ≤ i ∧ a ( j )]] .

In contrast, Example 3 is a simple example that requires majority quantifiers (Furst et al., 1984): Example 3 (Majority) . Strings with more b 's than a 's: M i [ b ( i )] .

Figure 1 showed how FO ( M ) can be used to recognize patterns like a m b m . A similar idea can be used to model parentheses matching (Barrington et al., 1990):

Example 4 ( 1 -Dyck) . The well-balanced parentheses language (with a opening and b closing):

Example 5 (Integer Arithmetic) . Iterated addition (i.e., summing n n -bit numbers), iterated multiplication, and division (Hesse, 2001) can all be expressed in FO ( M ) .

3 Finite Precision Transformers Cannot Attend Universally

Attention heads that spread attention weight uniformly across inputs have been observed in transformer LMs (Merrill et al., 2021) and make soft attention fundamentally more powerful than hard attention (Hao et al., 2022; Merrill et al., 2022). In particular, uniform attention is an important primitive that transformers can use to solve tasks involving counting (Bhattamishra et al., 2020; Chiang et al., 2023), taking majority votes (Merrill et al., 2022), and matching parentheses or sorting (Weiss et al., 2021). A transformer with sufficient precision can easily implement uniform attention by setting the keys and queries across all positions to be constant. However, attention heads with finite precision cannot represent uniform attention over long sequences as a consequence of the following:

Then the number of nonzero entries of ˜ a is upper bounded by its precision: specifically, ˜ a has at most 2 2 p nonzero entries.

Proposition 1. Let a ∈ R n s.t. ∑ n i =1 a i = 1 and ˜ a its nearest p -precision float approximation.

Moreover, if p < log log n and a is uniform (i.e., a i = 1 /n ), then ˜ a = /vector 0 .

Proof. The smallest positive value representable by a p -precision float is 2 -( p m -2+2 pe -1 ) which is bounded below by 2 -2 p +1 . Letting k = 2 2 p , it holds that 2 -2 p +1 = 2 /k . So if ˜ a i gets the minimum value, then a i ≥ 1 /k . Since ∑ i a i = 1 , there can be at most k indices satisfying this property. This implies there can be at most k nonzero entries in ˜ a . If n > k and a is uniform, 1 /n is less than half of the minimum representable value of 2 /k . Thus, ˜ a = /vector 0 .

Proposition 1 says that fixed-precision transformers are artificially limited because they can only attend over bounded-length windows, making them similar to hard-attention transformers (Hao et al., 2022). Morever, they cannot compute uniform attention over contexts of length n with less than log log n precision. This explains why Chiang et al. (2023) prove finite-precision transformers provably cannot recognize a m b m , while in practice transformers have been shown to learn even its harder variant a m b m c m even with long context lengths (Bhattamishra et al., 2020). In essence, their upper bound only applies in the asymptotic regime when n > 2 2 p .

In contrast, transformers in practice have enough precision both to compute uniform attention and recognize a m b m on practical context lengths. More concretely, the bfloat16 representation allows uniform attention over 2 6+2 7 ≈ 10 42 tokens and normal float16 12 allows 2 10+2 4 ≈ 10 8 tokens, both well above the typical context window of transformers. This motivates a formal model of transformers with enough precision to compute uniform attention and recognize languages such as a m b m .

4 Main Result: Expressing Log-Precision Transformers in FO ( M )

By Proposition 1, precision must grow with the context length n ( p > log log n ) for a transformer to compute uniform attention and other attention patterns with unbounded range, like practical transformers. In this paper, we analyze any transformer with up to O(log n ) precision. We show that any function computable by log-precision transformers can be expressed in FO ( M ) :

Theorem 2. Let T be a log-precision transformer with a parameter vector θ T fixed for all context lengths n . 13 Then, there exists an FO ( M ) sentence φ that computes the same function as T , i.e., φ ( x ) = T ( x ) for any input string x .

Theorem 2 is the tightest known upper bound for log-precision transformers and shows that it is still possible to characterize transformers in a simple variant of first-order logic even with log-precision and uniform attention. As alluded to earlier, Theorem 2 immediately implies that any problem complete for FO ( M ) (or a larger class) is also transformer-hard. Since integer division and Dyck language membership are known to be FO ( M ) -complete (Hesse, 2001; Aaronson et al., 2022), it follows, perhaps surprisingly, that the entire computation of any transformer on input x can be reduced to a single integer division or a finite number of Dyck-language queries:

Corollary 2.1. Let T be a transformer satisfying Theorem 2. For any input x , there exist first-order definable integers a, b, and i (dependent on T and x ) such that T ( x ) equals the i -th bit of /floorleft a/b /floorright . For any x , there also exist first-order definable strings w 1 , . . . , w m such that T ( x ) is first-order definable in terms of the membership of the w i 's in k -Dyck.

5 Preliminaries for Proving Theorem 2

5.1 Computation Graphs

A computation graph G over a datatype D ⊆ { 0 , 1 } ∗ and a countable set of primitive functions F ⊆ D ∗ × D is a directed acyclic graph where:

Each node is labelled by a node type : a function f ∈ F computed by this node.

Each edge represents a value D flowing as output from one node into another node. We consider the edges flowing into node j to have an order, i.e., be numbered.

F contains the special symbol input , which designates k nodes as input nodes. We refer to k as the arity and assume w.l.o.g. that nodes 0 , . . . , k -1 are inputs. 14

A single node is taken as the output node (w.l.o.g., the node with the largest index).

A computation graph G of arity k parameterizes a function D k → D in the standard way: the input nodes are assigned the input values, and the value of each node is computed (traversing the graph in a bottom-up topological order) as a function of the values of its children until the output node receives a value. The value of the output node is considered the output of the function. It is worth noting that computation graphs can only process inputs of bounded length. To process arbitrary-length inputs, we will need to generalize them to computation graph families (Section 5.2).

For a computation graph G , size ( G ) is the number of nodes, depth ( G ) is the length of the longest path from an input node to the output, and arity ( G,i ) is the number of inputs to node i .

Threshold circuits. A threshold circuit is a special case of a computation graph where D = { 0 , 1 } and F is the set of threshold functions of the form θ ≤ ∆ and θ ≥ ∆ over D ∗ , defined as follows: θ ≤ ∆ ( x ) = 1 if ∑ σ ∈ x σ ≤ ∆ and 0 otherwise; θ ≥ ∆ ( x ) is defined analogously. Typical AND, OR, and NOT gates are a special case of threshold gates, as is an IDENTITY gate. 15

We allow nodes with the k ′ ≥ 1 largest indices to all be designated as (ordered) output nodes. A threshold circuit with arity k and k ′ output nodes will thus be a function from { 0 , 1 } k to { 0 , 1 } k ′ . This will be convenient when simulating neural network components that output multiple bits.

We will find it useful to consider threshold circuits as a kind of compilation target for computation graphs: in other words, we will be concerned with simulating computation graphs defined over more complex functions and data types into threshold circuits.

5.2 Computation Graph Families

A computation graph family over D and F is a mapping from n ∈ N to a computation graph G n for processing inputs of size n . Thus, G defines a function from D ∗ → D , where G ( x ) = G | x | ( x ) .

Intuitively, computation graph families are useful because they generalize computation graphs to define functions over unbounded-length strings as inputs.

Size, depth, and arity. For computation graph families, the size, depth, and arity become functions of the input length n : size G ( n ) = size ( G n ) , depth G ( n ) = depth ( G n ) , arity G ( n, i ) = arity ( G n , i ) .

Uniformity. The infinite set G can be alternatively represented by two functions:

node G ( n, i ) , which returns the type of node i in G n if i ≤ size ( G n ) , and ∅ otherwise. For example, if node i computes the logical AND of its inputs, then node G ( n, i ) = ∧ .

edge G ( n, i, j ) , which returns the argument index of i into node j if G n contains an edge i → j and -1 otherwise. edge G ( n, i, j ) only needs to be defined over i, j < size ( G n ) . For example, if G n contains a node j with three incoming edges, the second of which comes from node i , then edge G ( n, i, j ) = 1 .

Apair of algorithms implementing these two functions uniquely specifies a computation graph family, as it enables building the computation graph G n for any n . Uniform computation graph families (generalizing uniform circuits; cf. Arora & Barak, 2009) are families where node G and edge G can be computed efficiently, i.e., under some constraints on space or time:

Definition 5 (Uniformity) . A computation graph family G is T ( n ) -uniform iff node G ( n, i ) and edge G ( n, i, j ) can be computed by a deterministic Turing machine in time T ( n ) . We focus on log-uniform computation graph families: i.e., where T ( n ) = O(log n ) . 16

Threshold circuit families. These are simply families of threshold circuits. We will be simulating computation graph families with threshold circuit families. Log-uniform TC 0 is the class of languages recognized by log-uniform constant-depth, poly-size threshold circuit families. See Merrill & Sabharwal (2023); Liu et al. (2023); Arora & Barak (2009) for more background on TC 0 and circuits.

6 Proof of Theorem 2

The idea is to simulate a transformer with a log-uniform TC 0 circuit family. Since log-uniform TC 0 = FO ( M ) , this would imply any transformer can be expressed in FO ( M ) . First, we note that transformers are log-uniform computation graphs:

Lemma 1 (Proof in Appendix B.1) . A transformer T is a log-uniform computation graph family where F contains embedding, self-attention, feedforward, and output components.

Further, each core module of the transformer can be simulated by a log-uniform TC 0 circuit family:

Lemma 2 (Proof in Appendix B.2) . Let T be a log-precision transformer with fixed parameters θ T . Then each component in F is computable in log-uniform TC 0 .

Intuitively, we can now simulate a transformer in log-uniform TC 0 by just simulating each of its components with a threshold circuit and routing their inputs and outputs appropriately. However, we will need two more technical conditions to verify that this construction is indeed log-uniform:

Lemma 3 (Proof in Appendix B.3) . Let T be a log-precision transformer with fixed parameters θ T . There exists a function bsize ( n ) that is a power of 2 and computable in O(log n ) time s.t. size F ( n ) ≤ bsize ( n ) for all F ∈ F .

Lemma 4 (Proof in Appendix B.4) . If F is a log-uniform TC 0 family and size F ( n ) ≤ bsize ( n ) , there exists a log-uniform TC 0 family F ′ s.t. F ( x ) = F ′ ( x ) for all x and size F ′ ( n ) = bsize ( n ) .

Combined, Lemmas 3 and 4 show that each F ∈ F is computable by a log-uniform TC 0 family with size bsize ( n ) that is a power of 2 and computable in time O(log n ) . We will show these conditions imply a transformer T can be simulated by a TC 0 family C (Theorem 3) and moreover that C is log-uniform (Corollary 3.2). By the equivalence of log-uniform TC 0 and FO ( M ) (Barrington et al., 1990), we then conclude that any log-precision transformer can be expressed in FO ( M ) .

/negationslash

Algorithm 1

node

C

(

n, i

)

Return the type of gate

i

in circuit

C

n

.

1:

F

node

G

(

n,

bnode

(

n, i

))

2:

if

F

=

∅

then

3:

return

node

F

(

n, i

-

bstart

(

n, i

′

))

4:

else return

∅

Algorithm 2

edge

C

(

n, i, j

)

If

C

n

contains an edge

i

→

j

, return the argument

number of that edge. Otherwise, return

-

1

.

1:

i

′

bnode

(

n, i

)

2:

j

′

bnode

(

n, j

)

3:

s

i

bstart

(

n, i

′

)

4:

s

j

bstart

(

n, j

′

)

5:

if

i

′

=

j

′

then

6:

F

node

G

(

n, i

′

)

7:

return

edge

F

(

n, i

-

s

i

, j

-

s

j

)

8:

else if

edge

G

(

n, i

′

, j

′

)

0

then

9:

b

i

i

-

(

s

i

+

bsize

(

n, i

′

)

-

p

(

n

))

10:

b

j

j

-

(

s

j

+

p

(

n

)

·

edge

G

(

n, i

′

, j

′

))

11:

if

b

i

=

b

j

< p

(

n

)

then return

j

-

s

j

12:

else return

-

1

13:

else return

1

-

6.1 Simulating Computation Graph Families with Circuit Families

We give algorithms that take a computation graph family and define a circuit family simulating it. Intuitively, the algorithms creates contiguous blocks of circuit gates simulating each node in the computation graph and route inputs and outputs between blocks appropriately.

Block mapping. This algorithm depends on a block mapping , which is an implementation of the following three functions:

The block node bnode ( n, i ) : the index of the node that gate i 's block is simulating.

The block start bstart ( n, i ′ ) : the smallest gate index in the block simulating node i ′ .

The block size bsize ( n, i ′ ) : the number of gates in the block simulating node i ′ .

Further, we enforce that a valid block mapping must satisfy that, for all i , with i ′ = bnode ( n, i ) ,

Let G be a computation graph whose primitive functions are computable by log-uniform threshold circuits. We can identify each primitive function with a log-uniform threshold circuit family F that computes it, where the first arity F ( n ) gates are IDENTITY gates reserved for taking input. For such a graph, node G can be taken to return a symbol identifying a circuit family F . In this case, our algorithm requires that, for all i ′ , the block size of i ′ must match the size of the circuit for the type of block i ′ , i.e., bsize ( n, i ′ ) = size node G ( n,i ′ ) ( n ) . These properties let us meaningfully identify a graph node i ′ with a block of nodes that will simulate it. This intuition enables us to develop Algorithms 1 and 2 for constructing a uniform threshold circuit family from a uniform computation graph family.

Theorem 3. Let G be a computation graph over a finite set of node types F , where each F ∈ F is specified by a log-uniform circuit family. Let bnode , bstart , and bsize be a valid block mapping in the sense above. Then Algorithms 1 and 2 define a circuit family C such that

depth C ( n ) ≤ depth G ( n ) · max F depth F ( n ) .

C and G compute the same D ∗ p → D p function (let the final p gates of each C i be its output).

size C ( n ) ≤ size G ( n ) · max F size F ( n ) .

Proof. Assume w.l.o.g. that the gates of C are topologically ordered. We show by induction over circuit gates j (with j ′ = bnode ( n, j ) ) that:

For all i ′ < j ′ , the last p nodes of block i ′ store the value of node i ′ .

For all i such that bstart ( n, j ′ ) ≤ i ≤ j , gate i of C (as a function of the input nodes of j ′ ) computes gate i -bstart ( n, j ′ ) of node G ( n, j ′ ) .

Base case. We have two circuits with no gates, so the premises are trivially satisfied.

Inductive case. Assume the premises hold up to j . We will show they hold for j + 1 . Let T = node G ( n, j ′ ) . By Premise 1, we know that the last p nodes of block i ′ store the output of node i ′ , for

i ′ < j ′ . By Algorithm 2, for each i ′ such that edge G ( n, i ′ , j ′ ) = a with 0 ≤ k < arity F ( n ) , gates kp through k ( p +1) -1 of block j ′ will copy the final p gates of block i ′ . Thus, the first k · arity F ( n ) gates of block j ′ store the inputs to node j ′ .

At this point, we use Premise 2 to conclude that the first j -bstart ( n, j ′ ) gates of block j ′ compute the same function as the first j -bstart ( n, j ′ ) gates of F with respect to this input. Thus, we just need to show that gate j +1 is also correct. Within Algorithm 2, we fall in case i ′ = j ′ , meaning that gate j +1 of block j ′ gates the same inputs as gate j +1 of F . By Algorithm 1, the type of gate j +1 in block j ′ is the type of gate j +1 of F . Thus, gate j +1 in block j ′ computes the same function of the input gates as gate j +1 in F . If j +1 = bsize ( n, j ′ ) , we conclude that the final p gates of block j ′ store the output of node j ′ .

Let XC 0 denote any family of constant-depth, poly-size circuits, including AC 0 and TC 0 . 17

Corollary 3.1. Let G be a constant-depth, poly-size computation graph family over a finite F . If every node type in F can be computed by XC 0 circuits, the function computed by G is in XC 0 .

Since a transformer has constant depth and polynomial size, Corollary 3.1 lets us easily recover prior results about hard-attention transformers (Hao et al., 2022; Hahn, 2020) and saturated attention transformers (Merrill et al., 2022) using a common framework. All one has to do is show that all individual node types in such transformers can be computed by AC 0 and TC 0 circuits, respectively.

Corollary 3.1 established that Algorithms 1 and 2 construct a circuit family that simulates G . With the right block mapping, C will be log-uniform as long as G and its node types are log-uniform.

Corollary 3.2. Let G be a log-uniform, constant-depth computation graph family over a finite F , where each F ∈ F is specified by a log-uniform TC 0 family with size F ( n ) = bsize ( n ) that is a power of 2 computable in O(log n ) time. Then G can be simulated by a log-uniform TC 0 family C that obeys the size and depth properties of Theorem 3.

Proof. Let C be the circuit family defined by Algorithms 1 and 2 given G and the following block mapping: bnode ( n, i ) = /floorleft i/ bsize ( n ) /floorright , bstart ( n, i ′ ) = i ′ · bsize ( n ) , bsize ( n, i ′ ) = bsize ( n ) . Since bsize ( n ) is a power of 2 , bnode and bstart are reducible to left and right shifting over O(log n ) -bit integers, which can be implemented in O(log n ) time. Thus, each block mapping function is computable in time O(log n ) . Since node G and edge G are just calling functions computable in time O(log n ) with constant overhead, we conclude that C , the circuit family they define, is log-uniform, and it is already known to simulate G with constant depth and polynomial size by Theorem 3.

7 Conclusion

We proved that any log-precision transformer classifier can be translated to an FO ( M ) sentence that computes the same function (on all inputs of any length). This result comes by first simulating a transformer with a highly uniform threshold circuit family, and then leveraging the established equivalence of log-uniform circuits and FO ( M ) . Transformers and other neural nets are often discussed in contrast with symbolic models based on logical formalisms (Garnelo & Shanahan, 2019)-an immediate implication of our result is that it is possible to express the inner workings of transformers also in a simple logic, challenging the premise of a rigid division between symbolic and neural models. Our results also provide the tightest known upper bound on log-precision transformers.

While it is striking that a full transformer can be translated to a sentence in a logic as simple as FO ( M ) , we believe the bound is not tight. In particular, we conjecture that it is possible to simulate any transformer with an FO ( M ) sentence of quantifier depth of at most 2, which could be proven by establishing a hierarchy theorem describing the FO ( M ) quantifier depth needed to simulate a TC 0 family of a certain size. It would also be an interesting extension to translate real transformers to FO ( M ) sentences. In this sense, we believe our results provide a theoretical foundation to guide mechanistic interpretability work (cf. Weiss et al., 2021; Lindner et al., 2023).

Our findings provide a novel view into transformer classifiers and their limits. It would be exciting for future research to extend our results to account for other common practical uses of transformers, such as for long-form generation, chain-of-thought reasoning, and in-context learning.

Acknowledgments

We thank Paul Beame, David Chiang, anonymous reviewers, and researchers at the Allen Institute for AI for feedback. WM was supported by an NSF graduate research fellowship and in part by NSF award 1922658.

References

Aaronson, S., Kuperberg, G., and Habryka, O. TC 0 : Constant depth threshold circuits, 2022. URL https://complexityzoo.net/Complexity_Zoo:T#tc0 .

Arora, S. and Barak, B. Computational Complexity: A Modern Approach . Cambridge University Press, 2009.

Barrington, D. A. M., Immerman, N., and Straubing, H. On uniformity within NC 1 . Journal of Computer and System Sciences , 41(3):274-306, 1990.

Bhattamishra, S., Ahuja, K., and Goyal, N. On the ability and limitations of transformers to recognize formal languages. In EMNLP , 2020.

Brent, R. P. and Zimmermann, P. Modern computer arithmetic , volume 18. Cambridge University Press, 2010.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert-Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In NeurIPS , 2020.

Chiang, D., Cholak, P., and Pillay, A. Tighter bounds on the expressivity of transformer encoders. ICML , 2023.

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S., and Olah, C. A mathematical framework for transformer circuits. Transformer Circuits Thread , 2021. https://transformer-circuits.pub/2021/framework/index.html.

Furst, M. L., Saxe, J. B., and Sipser, M. Parity, circuits, and the polynomial-time hierarchy. Mathematical systems theory , 17:13-27, 1984.

Garnelo, M. and Shanahan, M. Reconciling deep learning with symbolic artificial intelligence: representing objects and relations. Current Opinion in Behavioral Sciences , 29:17-23, 2019. ISSN 2352-1546.

Hahn, M. Theoretical limitations of self-attention in neural sequence models. TACL , 8:156-171, 2020.

Hao, Y., Angluin, D., and Frank, R. Formal language recognition by hard attention transformers: Perspectives from circuit complexity. TACL , 10:800-810, 2022.

Hesse, W. Division is in uniform TC 0 . In International Colloquium on Automata, Languages, and Programming , pp. 104-114. Springer, 2001.

Hunter, P., Bouyer, P., Markey, N., Ouaknine, J., and Worrell, J. Computing rational radical sums in uniform TC0. Foundations of Software Technology and Theoretical Computer Science , 2010.

Lindner, D., Kramár, J., Rahtz, M., McGrath, T., and Mikulik, V. Tracr: Compiled transformers as a laboratory for interpretability. arXiv , abs/2301.05062, 2023.

Liu, B., Ash, J. T., Goel, S., Krishnamurthy, A., and Zhang, C. Transformers learn shortcuts to automata. In ICLR , 2023.

Merrill, W. and Sabharwal, A. The parallelism tradeoff: Limitations of log-precision transformers. TACL , 11:531-545, 2023.

Merrill, W., Ramanujan, V., Goldberg, Y., Schwartz, R., and Smith, N. A. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. In EMNLP , 2021.

Merrill, W., Sabharwal, A., and Smith, N. A. Saturated transformers are constant-depth threshold circuits. TACL , 10, 2022.

Pérez, J., Marinkovi´ c, J., and Barceló, P. On the Turing completeness of modern neural network architectures. In ICLR , 2019.

Thoppilan, R., Freitas, D. D., Hall, J., Shazeer, N. M., Kulshreshtha, A., Cheng, H.-T., Jin, A., Bos, T., Baker, L., Du, Y., Li, Y ., Lee, H., Zheng, H., Ghafouri, A., Menegali, M., Huang, Y., Krikun, M., Lepikhin, D., Qin, J., Chen, D., Xu, Y., Chen, Z., Roberts, A., Bosma, M., Zhou, Y., Chang, C.-C., Krivokon, I. A., Rusch, W. J., Pickett, M., Meier-Hellstern, K. S., Morris, M. R., Doshi, T., Santos, R. D., Duke, T., Søraker, J. H., Zevenbergen, B., Prabhakaran, V., Díaz, M., Hutchinson, B., Olson, K., Molina, A., Hoffman-John, E., Lee, J., Aroyo, L., Rajakumar, R., Butryna, A., Lamm, M., Kuzmina, V. O., Fenton, J., Cohen, A., Bernstein, R., Kurzweil, R., Aguera-Arcas, B., Cui, C., Croak, M., Chi, E., and Le, Q. LaMDA: Language models for dialog applications. ArXiv , abs/2201.08239, 2022.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, I. Attention is all you need. In NeurIPS , 2017.

Weiss, G., Goldberg, Y., and Yahav, E. On the practical computational power of finite precision RNNs for language recognition. In ACL , 2018.

Weiss, G., Goldberg, Y., and Yahav, E. Thinking like transformers. ICML , 2021.

Xiong, R., Yang, Y., He, D., Zheng, K., Zheng, S., Xing, C., Zhang, H., Lan, Y., Wang, L., and Liu, T.-Y. On layer normalization in the transformer architecture. In ICML , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this submission, the authors generalize an expressivity result of transformers from Chiang et al. (2023), who showed that finite-precision transformers can be equivalently expressed in a counting generalization of first-order logic. In the paper at hand, the authors generalized this result that log-precision transformers, i.e., that scale with the input size, are expressible in First-order logic with majority votes, FO(M). This formalism is needed to express uniform attention patterns, to, for example, express counting languages such as a^m b^m. The authors provide a well-written introduction to the topic and rigorous proof.\n\nSTRENGTHS:\n- A novel bound for the expressive power of transformers, which generalizes previous results; the problem is interesting to the NeurIPS community. Transformers are ubiquitous.\n- The paper is well written; the transformer formalization and FO(M) is well-explained with the NeurIPS community in mind. Examples are provided (section 2).\n- Intuition paragraphs are provided after the main results and parts of the proof (second half of the paper), making it ok to follow.\n\nWEAKNESSES:\n- Although not necessary for this type of theoretical work, there would have been room to conduct some interesting experimental results to visualize and empirically analyze the authors claims\n- The most important lemma (I feel), Lemma 2, is moved to the appendix and scattered across multiple sections there with little intuition given. The reviewer found this the most interesting part and more important for understanding the proof than section 6.1, as the simulation is straight forward.\n\nQUESTIONS:\n- I wonder how the claim that uniform attention is necessary relates to the findings of “ALiBi” of Press et al. (ICLR, 2022)? \n- What are the author’s opinions on the next steps to tighten the bound further? \n- Would you consider moving Lemma 2 to the main part of the paper? What are your opinions on that?\n\nLIMITATIONS:\nThe bound is not tight. The missing tightness of the provided bound would have deserved its own section or a more involved explanation. It is only partially touched on at the end of the conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides two contributions which extend previous work showing that finite-precision transformers may be expressed in FOL. \n* Firstly, the authors prove that finite-precision transformers can only uniformly attend to bounded-length windows over their input sequence, with the constraints on previous theoretical work (those of Chiang et al. on finite-precision models) being too tight to model the capabilites of real models (which are capable of expressing sentences of the form $a^mb^mc^m$). This motivates them to consider log-precision transformers which are capable of attending to unbounded ranges at $O(\\log n)$ precision.\n* Secondly, the authors prove that the computation performed by a log precision transformer can be expressed in FO(M) logic, a variant of first order logic presupposing the typical quantifiers (universal, existential) as well as a majority quantifier (if a variable is present $\\geq \\frac{1}{2}$ of the entries in a set). They prove this by showing that log-precision transformers belong to computation graph families which are computable in log-uniform threshold circuits, which in-turn (leveraging existing results) are expressable in FO(M).\n\nAlongside the proofs for the second claim, they also provide an algorithm which demonstrates how (in principle) one can express computational graph families with circuit families, by breaking up each node in the computational graph into contiguous blocks of circuit gates; also considering the case of log-uniform computation graphs.\n\nSTRENGTHS:\nThe paper appears technically sound, though proofs outside of the main text (those in the appendices) were not thoroughly checked (the first claim, regarding limited expressivity of finite-precision transformers, is small and was checked). \n\nThe approach taken to demonstrate the FO(M) expressibility draws on existing results from various domains and uses these to extend previous findings on logical expressivity of transformers, thus it is both quite novel and valuable. Whilst these findings remain far from useful for practical mechanistic interpretability, they serve as a useful foundation for work hoping to explore any such applications.\n\nWEAKNESSES:\nNo major weaknesses were identified. \n\nTwo minor weaknesses are:\n1. The limited applicability which is only partially addressed in the section on \"Mechanistic Interpretability\". In particular, the size of a FO(M) sentence required to express a practical transformer would presumably be significant enough that its interpretability would be questionable at best. The authors do conjecture that quantifier depth \"of at most 2\" may suffice to express transformers, but this has yet to be shown. This criticism is applicable to other technical attempts to produce discrete / tree-like representations of transformers, and so is not a significant drawback, besides saying that any attempts to motivate such research from a practical perspective seem highly optimistic at present.  \n2. The paper is quite dense in parts, which is understandable. However, two potentially useful figures could be envisioned: 1) a high-level overview of the proof (log-precision transformers -> computational graph -> TC -> FO(M)) 2) A figure illustrating the contiguous circuit construction algorithm from $6.1\n\n## Nitpicks\n* 55: \"powerful enough *to* express\"\n* Example 5: Seems incomplete?\n* 252: \"the second of which\" should probably read \"one of which\" as it is otherwise somewhat confusing why \"the second\" is being specified\n* 310: \"without loss of generality\" -> \"w.l.o.g\" to be consistent with previous use\n* 355: \"Challenging the premise of a rigid division between symbolic and neural models\". The mathematical sense in which this is being challenged is quite loose, and so it is not clear that this meaningfully \"challenges\" the division in the way it is typically framed (in a sense, matrices are symbolic, especially if a model is quantized, but this does not make the model *symbolic* in the way a logician would find useful)\n\nQUESTIONS:\nNone\n\nLIMITATIONS:\nTheoretical limits of the proofs and claims are precise and well-stated. Claims around practical applicability of these results may be slightly overstated.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper presents a theoretical analysis on the expressiveness of transformer-based models. In particular, the authors have managed to prove that the any log-precision transformer is equivalently expressible as normal first-order logic plus majority-vote quantifiers, FO(M). This yields the tightest known upper bound of log-precision transformers.

STRENGTHS:
The result is of significant theoretical interest and can potentially bridge well-studied first-order logic and widely used transformer architecture (e.g., via mechanistic interpretability work). The paper is nicely structured and with well crafted examples. To the best of my knowledge, I don't see major gaps in their proof derivations.

WEAKNESSES:
I don't see any significant weakness in this paper.

QUESTIONS:
Minor:
- Figure 1 on page 1 is admittedly a great motivation example but we don't know the exact meaning of an and b until page 4. It is perhaps a good idea to shed more words about Figure 1 earlier.
- line 80, 'atend'

LIMITATIONS:
The authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the computational power of transformers is studied subject to floating point precision and related to that of first-order logic. In particular, the authors show that (1) fixed-precision is not sufficient to compute uniform attention for arbitrary context lengths and (2) log-precision can be simulated by first-order logic with majority. The latter is shown by establishing that log-precision transformers can be simulated by a family of log-uniform circuits which is known to be equivalent to first-order logic with majority.\n\nSTRENGTHS:\n- Some of the existing theoretical results on the computational power of transformers seem to only partially support observations in practice. For example, it has been shown that the language $a^nb^n$ provably can not be recognized by a fixed-precision transformer while experiments indicate the opposite. The paper contributes to better understanding this gap by deriving theoretical results for log-precision transformers that seem to resemble observations in practice although practical implementations are in fact fixed-precision.\n- While the expressive power of log-precision transformers has been characterized by first-order logics in previous work, the paper established the tightest bound so far.\n- The extensive use of transformers and language models in formal domains such as programming languages and theorem proving makes the connection between transformers and formal logics particularly interesting.\n\nWEAKNESSES:\n- The role of positional encoding is not discussed in the analysis. Yet, fixed-precision and log-precision has direct implications on the ability to represent a positional encoding.\n- Merrill & Sabharwal (2023) show that a log-precision transformer can be simulated by a uniform $TC^0$ circuit family which at first seems fairly similar to the result in this paper. Merrill & Sabharwal bound space and prove the result for logspace-uniform $TC^0$ whereas in this paper the authors bound time and prove the result for log-uniform $TC^0$. I think this is not obvious and making it more explicit would help to understand the contribution of this paper.\n- No definition of circuit complexity is given in the paper.\n\nQUESTIONS:\n- Is there a straightforward argument that generalizes the results for decoder architectures or are the results in fact limited to encoder architectures?\n- In the conclusion the threshold circuit family is referred to as highly uniform. What is the meaning of highly in this context?\n\nLIMITATIONS:\n- The definition of the transformer resembles the encoder part of the full transformer architecture. If the results do not generalize to the transformer decoder this should be addressed as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}