Title: Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages
Venue: NeurIPS 2024
Accepted: yes
Keywords: Transformers

Formal Language Theory

Logic

Automata

Expressivity
Abstract: The expressive power of transformers over inputs of unbounded size can be studied through their ability to recognize classes of formal languages. In this paper, we establish exact characterizations of transformers with hard attention (in which all attention is focused on exactly one position) and attention masking (in which each position only attends to positions on one side). With strict masking (each position cannot attend to itself) and without position embeddings, these transformers are expressively equivalent to linear temporal logic (LTL), which defines exactly the star-free languages. A key technique is the use of Boolean RASP as a convenient intermediate language between transformers and LTL. We then take numerous results known for LTL and apply them to transformers, showing how position embeddings, strict masking, and depth all increase expressive power.

Masked Hard-Attention Transformers Recognize Exactly the Star-Free Languages

Abstract

The expressive power of transformers over inputs of unbounded size can be studied through their ability to recognize classes of formal languages. In this paper, we establish exact characterizations of transformers with hard attention (in which all attention is focused on exactly one position) and attention masking (in which each position only attends to positions on one side). With strict masking (each position cannot attend to itself) and without position embeddings, these transformers are expressively equivalent to linear temporal logic (LTL), which defines exactly the star-free languages. A key technique is the use of Boolean RASP as a convenient intermediate language between transformers and LTL. We then take numerous results known for LTL and apply them to transformers, showing how position embeddings, strict masking, and depth all increase expressive power.

1 Introduction

Significant progress has been made in the last few years on characterizing the expressivity of transformers (Vaswani et al., 2017) in terms of well-understood classes of formal languages (Strobl et al., 2024b). Results have been obtained for a wide range of variants of transformers, and nearly all take the form of either upper bounds (transformers recognize only languages in class 𝐶 ) or lower bounds (transformers recognize all languages in class 𝐶 ). In this paper, we establish exact characterizations of transformers with hard attention (in which all attention is focused on exactly one position) and attention masking (in which each position 𝑖 only attends to positions on one side of 𝑖 ).

With strict masking (in which each position cannot attend to itself) and without position embeddings, these transformers recognize exactly the class of star-free regular languages. The left side of Figure 1 summarizes our results relating masked hard-attention transformers and linear temporal logic ( LTL ), which defines exactly the star-free regular languages.

A key technique in these proofs is the use of B-RASP , which, like RASP (Weiss et al., 2021), is a small programming language that compiles into transformers. B-RASP is restricted to Boolean values and compiles to masked hard-attention transformers. Additionally, a masked hard-attention transformer can be decompiled back to a B-RASP program. We use B-RASP as an intermediate language between transformers and LTL .

The equivalence of masked hard-attention transformers with LTL (and other equivalent characterizations, like counter-free automata and first-order logic) enables us to take numerous results known for LTL and apply them to transformers, as shown on the right side of Figure 1:

Strict future-masked rightmost-hard attention is sufficient; adding past-masked, non-masked, and/or leftmost-hard attention does not increase expressivity (Section 5.1).

Strict masking is important (Section 5.2); without it, masked hard-attention transformers are less expressive, recognizing only the stutter-invariant star-free languages.

masked hard-attention transformers

strict future-

masking

strict future-

and past-

masking

B-RASP

LTL

nonstrict

masking

depth

2

depth

1

depth

0

rational

sinusoidal PEs

image PEs

§5.2

§3.4

§4.2

§5.1

§5.4

§5.3

Adding position embeddings increases the class of recognized languages to other wellstudied classes (Section 5.3); for example:

-With rational sinusoidal position embeddings, masked hard-attention transformers recognize exactly the regular languages in AC 0 .

-With arbitrary finite-image position embeddings, they are equivalent to LTL [ Mon ] (linear temporal logic with arbitrary monadic predicates).

Adding more layers always increases expressive power (Section 5.4).

2 Background

2.1 Preliminaries

Let Σ be a finite alphabet, and let 𝑤 = 𝑤 1 · · · 𝑤 𝑛 be an input string of length 𝑛 , where each 𝑤 𝑖 ∈ Σ . Throughout, we assume that 𝑤 is not empty. We write Σ + for the set of all non-empty strings over Σ . (We disallow empty strings because several formalisms used here require a designated position where an accept/reject decision appears. Adding a BOS or EOS token not in Σ for this purpose would make it possible to handle the empty string.) We write [ 𝑛 ] for the set { 1 , . . . , 𝑛 } .

The star-free regular languages are the closure of ∅ , { 𝜖 } , and { 𝜎 } for each 𝜎 ∈ Σ , under the operations of union, concatenation, and complementation. For example:

Σ ∗ is star-free because Σ ∗ = ∅ c .

( ab ) ∗ is star-free because ( ab ) ∗ = ( b Σ ∗ ∪ Σ ∗ a ∪ Σ ∗ aa Σ ∗ ∪ Σ ∗ bb Σ ∗ ) c .

( aa ) ∗ is regular but not star-free.

This class of languages has several other characterizations, including counter-free automata (Appendix B.5), first-order logic with order (McNaughton and Papert, 1971), and linear temporal logic (Kamp, 1968), which is what we will focus on in this paper.

2.2 Transformer variants

The original transformer (Vaswani et al., 2017), designed for machine translation, had both an encoder and a decoder. In practice, both encoder-only models like BERT (Devlin et al., 2019) and decoderonly models like GPT (Brown et al., 2020) are common. Like much previous work on transformer expressivity (e.g. Hahn, 2020), we study an encoder-only setup, where the input is a string and the output is a binary classification; but our results could easily be adapted to a decoder-only setting where the input is a prefix and the output is the next symbol.

The transformers studied here use unique hard attention (or simply hard attention ), in which an attention head focuses all attention on the position with the highest score, with ties broken to the left or right. Although this is different from the soft attention in actual transformers, theoretical studies unavoidably involve models of the real objects of study, and we are using unique-hard attention as a stepping-stone towards understanding real transformers. However, unique-hard attention may be more appropriate than it appears:

Real transformers are often observed to focus attention on a very small number of positions (Merrill et al., 2021). On Dyck languages, they have been found to learn effectively uniquehard attention in their second layer (Ebrahimi et al., 2020, Figure 1).

There exist soft-attention transformers that compute parity (Chiang and Cholak, 2022), but in practice, transformers cannot learn parity (Bhattamishra et al., 2020). Unique-hard attention transformers also cannot compute parity (Hahn, 2020), so they are in some sense more realistic.

Hard attention has occasionally been used in practice in previous research on interpretability (Kinley, 2020) and efficiency (Gupta et al., 2021; Xu et al., 2021).

In this paper, we use future masking , in which every position may only attend to positions to its left. This kind of masking is common in decoder-only models and has been studied in encoder-only models as well (Bhattamishra et al., 2020). We also consider past masking (Yao et al., 2021).

2.3 Previous work

P´ erez et al. (2021) show that average-hard attention transformer encoder-decoders, where the decoder runs for a polynomial number of steps before accepting or rejecting a string, recognize all of P (that is, all languages decidable by a deterministic Turing machine in polynomial time). Merrill and Sabharwal (2024) prove another version of this result, and further observe that all such transformers are in P . This result is the only other exact characterization of any transformer variant that we are aware of.

Hao et al. (2022) show that (non-masked) hard-attention transformer encoders with arbitrary position embeddings have an upper bound of AC 0 (that is, languages defined by circuit families with polynomial size, unbounded fan-in, and bounded depth), and Barcel´ o et al. (2024) show that they have a lower bound of LTL [ Mon ] , which is linear temporal logic with all possible monadic numerical predicates. They leave open the question of whether these transformers are equivalent to LTL [ Mon ] -a question which, with suitable adjustments, we answer here in the affirmative.

3 Boolean RASP

RASP (Weiss et al., 2021) is a programming language intended to help programmers 'think like transformers.' It has the same basic operations as transformers, but it is easier to compose these operations in RASP than to write transformers by hand. Variants of RASP have been used fruitfully to study transformers' length-generalization capabilities (Zhou et al., 2024) and expressive power (Strobl et al., 2024a; Yang and Chiang, 2024). In this section, we define a version of RASP restricted to Boolean values, which we call Boolean RASP or B-RASP . As we will see, it can be compiled into masked hard-attention transformers, and masked hard-attention transformers can be decompiled back into B-RASP . We use it as an intermediate language between transformers and LTL , and find it more convenient to work with than either of them.

3.1 Definition

The input to a B-RASP program is a string 𝑤 = 𝑤 1 · · · 𝑤 𝑛 ∈ Σ + . There is one type of data, a Boolean vector , which is a vector of Boolean values indexed by 𝑖 ∈ [ 𝑛 ] . The initial Boolean vectors are 𝑄 𝜎 for each 𝜎 ∈ Σ , where 𝑄 𝜎 ( 𝑖 ) = 1 iff 𝑤 𝑖 = 𝜎 .

A B-RASP program is a sequence of operations that compute new Boolean vectors. Although they may have descriptive names, and names may be reused, here, to streamline definitions and proofs, we assume that all the Boolean vectors are numbered consecutively. That is, 𝑃 1 , . . . , 𝑃 | Σ | are the initial Boolean vectors 𝑄 𝜎 for 𝜎 ∈ Σ , and the Boolean vectors computed by the program are numbered

where:

ℓ

ℓ

r

r

input

ℓ

ℓ

r

r

ℓ

ℓ

r

ℓ

r

r

𝑄

ℓ

1

1

0

0

1

1

0

1

0

0

𝑄

r

0

0

1

1

0

0

1

0

1

1

𝑃

ℓ

0

1

1

0

0

1

1

0

1

0

𝑆

r

0

1

1

0

0

1

0

1

1

0

𝐼

0

1

1

0

0

1

1

1

1

0

𝐵

ℓ

0

1

1

1

0

1

1

1

1

1

𝐴

r

1

1

1

0

1

1

1

1

1

0

𝐶

1

1

1

1

1

1

1

1

1

1

𝑌

1

1

1

1

1

1

1

1

1

1

input

ℓ

r

r

ℓ

ℓ

ℓ

r

r

r

ℓ

𝑄

ℓ

1

0

0

1

1

1

0

0

0

1

𝑄

r

0

1

1

0

0

0

1

1

1

0

𝑃

ℓ

0

1

0

0

1

1

1

0

0

0

𝑆

r

1

1

0

0

0

1

1

1

0

0

𝐼

1

1

0

0

0

1

1

0

0

0

𝐵

ℓ

0

0

0

0

1

1

1

1

0

0

𝐴

r

1

1

0

0

1

1

1

1

0

0

𝐶

1

1

0

0

1

1

1

1

0

0

𝑌

0

0

0

0

0

0

0

0

0

0

starting from 𝑃 | Σ | + 1 without repetition. After the first 𝑡 vectors, vector 𝑃 𝑡 + 1 is computed using one of the following operations.

Position-wise operations. 𝑃 𝑡 + 1 ( 𝑖 ) can be be computed by 𝑃 𝑡 + 1 ( 𝑖 ) : = 𝑅 ( 𝑖 ) , where 𝑅 ( 𝑖 ) is a Boolean combination of zero or more of { 𝑃 1 ( 𝑖 ) , . . . , 𝑃 𝑡 ( 𝑖 )} .

Attention operations. 𝑃 𝑡 + 1 ( 𝑖 ) can be computed by either of

𝑀 ( 𝑖, 𝑗 ) , the mask predicate , is one of 𝑀 ( 𝑖, 𝑗 ) = 1 (no masking), 𝑀 ( 𝑖, 𝑗 ) = ( 𝑗 < 𝑖 ) (strict future masking), or 𝑀 ( 𝑖, 𝑗 ) = ( 𝑗 > 𝑖 ) (strict past masking).

𝑆 ( 𝑖, 𝑗 ) , the score predicate , is a Boolean combination of zero or more atomic formulas from { 𝑃 1 ( 𝑖 ) , . . . , 𝑃 𝑡 ( 𝑖 )} ∪ { 𝑃 1 ( 𝑗 ) , . . . , 𝑃 𝑡 ( 𝑗 )} .

𝑉 ( 𝑖, 𝑗 ) , the value predicate , has the same form as the score predicate.

𝐷 ( 𝑖 ) , the default value predicate , is a Boolean combination of zero or more atomic formulas from { 𝑃 1 ( 𝑖 ) , . . . , 𝑃 𝑡 ( 𝑖 )} .

For each 𝑖 ∈ [ 𝑛 ] , let 𝑗 𝑖 be the minimum (if the operator is ◀ ) or maximum (if ▶ ) value of 𝑗 ∈ [ 𝑛 ] such that 𝑀 ( 𝑖, 𝑗 ) = 1 and 𝑆 ( 𝑖, 𝑗 ) = 1 . If 𝑗 𝑖 exists, then 𝑃 𝑡 + 1 ( 𝑖 ) = 𝑉 ( 𝑖, 𝑗 𝑖 ) . If 𝑗 𝑖 does not exist, then 𝑃 𝑡 + 1 ( 𝑖 ) = 𝐷 ( 𝑖 ) .

If 𝑃 is a Boolean vector computed by program P , we write 𝑤 | = 𝑃 ( 𝑖 ) just in case 𝑃 ( 𝑖 ) = 1 when P is run on input string 𝑤 . To make a B-RASP program P recognize a language, one Boolean vector 𝑌 is designated the output vector, and position 𝑛 is designated the output position. Then, the input string 𝑤 is accepted iff 𝑤 | = 𝑌 ( 𝑛 ) . To make a B-RASP program compute a length-preserving sequence-to-sequence function from Σ + to Γ + , we designate a collection of output Boolean vectors 𝑌 𝛾 indexed by the symbols 𝛾 ∈ Γ , and consider the output at position 𝑖 to be 𝛾 iff 𝑌 𝛾 ( 𝑖 ) is true.

3.2 Example: Dyck-1 of depth 2

As an example, we consider the Dyck language with 1 pair of parentheses, limited to depth 2, or 𝐿 1 , 2 for short. It is recognized by the DFA in Figure 2a, where ℓ and r are left and right brackets. We show how to define this language in B-RASP , with a construction very similar to that of Yao et al. (2021).

Consider the input string ℓℓ rr ℓℓ r ℓ rr , which should be accepted. The basic idea is to identify brackets that are immediately matched ( ℓℓ rr ℓℓ r ℓ rr ), then look at the remaining brackets ( ℓℓ rr ℓℓ r ℓ rr ) to make sure they are matched. We describe the B-RASP program for this problem below; the resulting Boolean vectors are shown in Figure 2b.

We first construct Boolean vectors 𝑃 ℓ ( 𝑖 ) and 𝑆 r ( 𝑖 ) that indicate whether the predecessor (respectively, successor) symbol of 𝑖 is ℓ (respectively, r ). This is done with attention operations:

Vector 𝑃 ℓ ( 𝑖 ) makes position 𝑖 attend to the position immediately to its left, and its value predicate 𝑄 ℓ ( 𝑗 ) tests whether that position has an ℓ . Vector 𝑆 r is similar.

The Boolean vector 𝐼 ( 𝑖 ) indicates whether position 𝑖 is in a consecutive pair ℓ r , that is, whether it is immediately matched :

The Boolean vectors 𝐵 ℓ ( 𝑖 ) and 𝐴 r ( 𝑖 ) test if the symbol before (respectively, after) 𝑖 that is not immediately matched is ℓ (respectively, r ). Then 𝐶 checks each position 𝑖 to see if it is immediately matched, or it has ℓ and the following not-immediately-matched symbol is r , or it has r and the preceding not-immediately-matched symbol is ℓ :

Finally, the output Boolean vector 𝑌 tests if 𝐶 ( 𝑖 ) is true everywhere:

Boolean vectors for deciding non-membership of ℓ rr ℓℓℓ rrr ℓ in 𝐿 1 , 2 are shown in Figure 2c. It is straightforward to generalize this technique to recognize Dyck𝑘 of depth 𝐷 in B-RASP . 1 For another example program for an associative recall task, please see Appendix A. A B-RASP simulator that allows one to write and run additional examples can be found at https://b-rasp.github.io/ .

3.3 Normal forms

In B-RASP , the value predicate 𝑉 ( 𝑖, 𝑗 ) depends on both 𝑖 (the query position) and 𝑗 (the key/value position), but in actual transformers, it depends on 𝑗 only. The dependence on 𝑖 is sometimes convenient, but it does not change expressivity (see Appendix B.1).

The score predicate 𝑆 ( 𝑖, 𝑗 ) depends on both 𝑖 and 𝑗 in both B-RASP and actual transformers. Perhaps surprisingly, in B-RASP , it too can be made to depend only on 𝑗 without reducing expressivity, but as a tradeoff the program may become exponentially larger in size (see Appendix B.2).

3.4 Equivalence with linear temporal logic

We prove that B-RASP recognizes exactly the star-free languages, by proving that B-RASP is equivalent to linear temporal logic. Appendix B.5 gives another proof of the star-free-toB-RASP direction via counter-free automata.

In linear temporal logic or LTL (Kamp, 1968), every formula implicitly depends on a single 'time' (or position). The atomic formulas are 𝑄 𝜎 for every 𝜎 ∈ Σ , and we have the usual connectives ∧ , ∨ , and ¬ , as well as operators since and until . 2 For any input string 𝑤 = 𝑤 1 · · · 𝑤 𝑛 and position 𝑖 ∈ [ 𝑛 ] ,

we define 𝑤,𝑖 | = 𝜙 as follows:

To use a formula 𝜙 of LTL to define a language over Σ , for an input string 𝑤 ∈ Σ + of length 𝑛 we designate the last position as the output position, so that 𝑤 ∈ L( 𝜙 ) if and only if 𝑤, 𝑛 | = 𝜙 .

For example, let Σ = { a , b , # } and consider the following formulas:

The formula 𝜙 1 defines the language Σ ∗ # , which contains all and only strings with a # in the last position. The formula 𝜙 2 defines the language Σ ∗ # b ∗ # , and 𝜙 3 defines the language Σ ∗ # a ∗ # b ∗ # . Finally, 𝜙 4 defines the language # a ∗ # b ∗ # , because ¬( 0 since 1 ) is only true at the first position.

Theorem 1. For any formula of LTL that defines a language 𝐿 ⊆ Σ + , there is a B-RASP program that recognizes 𝐿 .

Proof.

See Appendix B.3. This is shown via direct construction. □

Theorem 2. For any B-RASP program that recognizes a language 𝐿 ⊆ Σ + , there is a formula of LTL that defines 𝐿 .

Proof. See Appendix B.4. We use the unary normal forms (Section 3.3) to facilitate this proof. □

4 Masked Hard-Attention Transformers

4.1 Definition

A masked hard-attention transformer layer with width 𝑑 > 0 is a length-preserving function

The self-attention layer att is specified by

A score function, which is a bilinear function 𝑓 𝑆 : R 𝑑 × R 𝑑 → R .

A mask, which is 𝑀 ( 𝑖, 𝑗 ) = 1 (no masking), 𝑀 ( 𝑖, 𝑗 ) = ( 𝑗 < 𝑖 ) (strict future masking), or 𝑀 ( 𝑖, 𝑗 ) = ( 𝑖 < 𝑗 ) (strict past masking).

A tie-breaking function 𝐶 to select one element of a finite non-empty set 𝐼 ⊂ N + , which is either 𝐶 ( 𝐼 ) = min 𝐼 (choose leftmost position) or 𝐶 ( 𝐼 ) = max 𝐼 (choose rightmost position).

A value function, which is a linear transformation 𝑓 𝑉 : R 𝑑 → R 𝑑 .

The layer works as follows, for each 𝑖 ∈ [ 𝑛 ] . Let

best-scoring unmasked positions

If 𝑈 𝑖 ≠ ∅ , let 𝑗 𝑖 = 𝐶 ( 𝐵 𝑖 ) and output 𝑐 𝑖 = 𝑓 𝑉 ( 𝑥 𝑗 𝑖 ) ; but if 𝑈 𝑖 = ∅ , output 𝑐 𝑖 = 0 .

The function ffn is a feed-forward neural network with 2 layers and ReLU activations in between.

Then a masked hard-attention transformer is a length-preserving function

where emb : Σ + → ( R 𝑑 ) + is a position-wise function (a word embedding), and each layer ℓ is a masked hard-attention transformer layer.

We write [T ( 𝑤 )] 𝑖 for the final activation value at position 𝑖 ∈ [ 𝑛 ] when T is run on input 𝑤 . To use T as a language recognizer, we add an output layer, which linearly projects [T ( 𝑤 )] 𝑛 to a scalar. If the result is nonnegative, we accept 𝑤 ; otherwise, we reject. The exact criterion does not matter much, as the transformers we construct only output + 1 2 or -1 2 , and could easily be changed to another convention. The language recognized by T (with the output layer) is the set of strings it accepts.

Our definition above differs from the standard definition (Vaswani et al., 2017) in a few ways besides unique-hard attention, which was discussed above in Section 2.2. Ours lacks layer normalization and position embeddings, but we add them in Sections 4.3 and 5.3, respectively. We only use single-head attention; multi-head attention can be simulated by summing the outputs of multiple single-head attentions, or it can be added to the definition, as in Appendix D.3.1. Our attention masking is strict, but we consider non-strict masking in Section 5.2.

4.2 Equivalence with B-RASP

Theorem 3. For any B-RASP program that recognizes a language 𝐿 ⊆ Σ + , there is a masked hard-attention transformer (with output layer) that recognizes 𝐿 .

Proof. See Appendix C.1. Attention layers simulate attention operations, and FFNs simulate positionwise operations. □

Theorem 4. For any masked hard-attention transformer (with output layer) that recognizes a language 𝐿 ⊆ Σ + , there is a B-RASP program that recognizes 𝐿 .

Proof. See Appendix C.2. To convert a masked hard-attention transformer to B-RASP , we first show that all of the intermediate values computed by the transformer are drawn from a finite set and therefore can be represented using 𝑂 ( 1 ) bits. 3 □

4.3 Layer normalization

Standard transformers (Vaswani et al., 2017) include layer normalization (Ba et al., 2016), but our definition above does not. Since layer normalization is a position-wise function, the proof of Lemma 24 is unaffected. But the construction of Lemma 21 does need to be modified to circumvent layer normalization (cf. Chiang et al., 2023, Proposition 22). Previously, we used 1 to represent true and 0 to represent false; now, we use a pair of activations to represent a truth value, ( 1 , 0 ) for true and ( 0 , 1 ) for false. This ensures that every vector has mean and variance independent of the input 𝑤 , so we can set the parameters of each layer normalization so that it has no effect. (In the proof of Theorem 3, we use a flag to indicate whether there are any unmasked positions or not. This flag already uses the encoding described above, and does not need to be modified.)

5 Further Results

In this final section, we leverage results from temporal logic and the equivalences established above to obtain numerous new results for masked hard-attention transformers (and B-RASP ).

5.1 Asymmetric attention

Our definitions of both B-RASP and masked hard-attention transformers include both leftmost-hard and rightmost-hard attention, and both future and past masking. But we can use the fact that, in LTL , if the output is read out only at the last position, it suffices to have only since and not until (Gabbay et al., 1980) to obtain the following result.

Theorem 5. Both B-RASP and transformers with only future-masked rightmost-hard attention recognize exactly the star-free languages.

Proof. Any star-free language can be defined in LTL using only since (Gabbay et al., 1980), and restricting Theorem 1 to translate from LTL with only since into B-RASP will only use future-masked ▶ . Therefore, B-RASP with only future-masked ▶ can define any star-free language. Similarly, the translation (Theorem 3) from B-RASP with only future-masked ▶ to masked hard-attention transformers only uses future-masked rightmost-hard attention. Therefore, transformers with only future-masked rightmost-hard attention can define any star-free language. □

Note that this applies only in a setting where we accept or reject strings by looking at the output at the last position. It does not apply to other settings, like transduction (Strobl et al., 2024a).

5.2 Non-strict masking

Our definitions of both B-RASP and masked hard-attention transformers use strict masking, in which a position cannot attend to itself. Standard transformers, however, use non-strict masking. We can modify the definitions to use non-strict masking, that is, 𝑖 ≤ 𝑗 or 𝑗 ≤ 𝑖 .

Non-strictness is known to reduce expressivity in LTL (Peled and Wilke, 1997), so it reduces expressivity in B-RASP and masked hard-attention transformers as well. Intuitively, non-strict masked operations are unable to distinguish between consecutive positions that have the same symbol. More formally, a language over Σ is called stutter-invariant 4 iff for all 𝑢, 𝑣 ∈ Σ ∗ and 𝜎 ∈ Σ , 𝑢𝜎𝑣 ∈ 𝐿 iff 𝑢𝜎𝜎𝑣 ∈ 𝐿 . An example of a language that is stutter-invariant star-free is ( a + b + ) ∗ (where 𝜎 + means 'one or more occurrences of 𝜎 '); a language that is star-free but not stutter-invariant is ( ab ) ∗ .

Theorem 6. Both B-RASP and masked hard-attention transformers with only non-strict masking recognize exactly the stutter-invariant star-free languages.

Proof. Peled and Wilke (1997) prove that LTL with non-strict since ′ and until ′ recognizes exactly the stutter-invariant star-free languages. The proofs of Theorems 1 and 2 may be adapted to use non-strict temporal operators and non-strict masking. Thus, non-strict B-RASP and non-strict LTL are equivalent. Similarly, using 𝑗 ≤ 𝑖 or 𝑗 ≥ 𝑖 as 𝑀 ( 𝑖, 𝑗 ) in the proofs of Theorems 3 and 4, we can show that non-strict masked hard-attention transformers are equivalent to non-strict B-RASP . □

In Section 3.2, we showed how to define 𝐿 1 , 2 , Dyck-1 of depth 2. Bhattamishra et al. (2020, §7.1) find experimentally that 𝐿 1 , 2 is not learnable by transformers, and they argue that it is not even expressible by transformers (with soft attention, non-strict masking, and no position embeddings). The reason is that while reading the prefix of ℓ 's at the start of the string, the soft-attention layer computes the same value vector at every position and cannot count the number of occurrences of ℓ . However, with the addition of a BOS symbol, soft attention can measure what fraction of symbols are ℓ , overcoming this limitation as observed empirically by Ebrahimi et al. (2020). The similarities between how strict masking in the hard attention setting and the addition of BOS in soft attention both enable positions to be distinguished are notable for future investigation.

5.3 Position embeddings

Our definition of a transformer does not, so far, include position embeddings; all information about ordering comes from attention masking. A position embedding is a family of functions Θ = ( 𝜃 𝑛 ) 𝑛 ≥ 0 where 𝜃 𝑛 ( 𝑖 ) is a scalar or vector representation of position 𝑖 in a string of length 𝑛 . Then the input layer emb becomes the sum of a word embedding and a position embedding.

We say that Θ has finite image if -𝑛 ≥ 0 Im 𝜃 𝑛 is finite. In general, our results extend to transformers with any position embedding that has finite image. The class of languages recognized may grow, and we give a recipe for characterizing the new class of languages.

We can add numerical predicates to LTL and initial Boolean vectors to B-RASP as follows. Let Π = ( 𝜋 𝑛 ) 𝑛 ≥ 0 be a family of functions 𝜋 𝑛 : [ 𝑛 ] → { 0 , 1 } . Then there is an additional predicate symbol Π such that for any string 𝑤 with length 𝑛 ,

For example, if Mid 𝑛 ( 𝑖 ) is true iff 𝑛 is odd and 𝑖 = ⌈ 𝑛 / 2 ⌉ , then we can define the language { # a 𝑚 # b 𝑚 # | 𝑚 ≥ 0 } in LTL [ Mid ] as:

A similar definition could be written in B-RASP [ Mid ] .

Theorem 7. Let Θ = ( 𝜃 𝑛 ) 𝑛 ≥ 0 be a position embedding with finite image. There exists a collection of predicates P Θ such that the following classes of languages are the same:

languages recognized by masked hard-attention transformers with position embedding Θ

languages defined by B-RASP [P Θ ]

languages defined by LTL [P Θ ] .

Proof. See Appendix D.1.

We discuss two important special cases below.

Sinusoidal position embeddings The original transformer (Vaswani et al., 2017) used position embeddings with coordinates of the form sin ( 2 𝜋 𝑓 𝑖 ) or cos ( 2 𝜋 𝑓 𝑖 ) . If the 𝑓 's are rational (though in the original definition they were not), then the position embeddings form a finite set, so Lemma 22 still holds. For any even 𝑑 , let us define a rational sinusoidal positional embedding with 𝑑 dimensions to be a position embedding Θ = ( 𝜃 𝑛 ) 𝑛 ≥ 0 where

Corollary 8. Masked hard-attention transformers with rational sinusoidal position embeddings recognize exactly the regular languages in AC 0 (that is, regular languages definable by a family of Boolean circuits with polynomial size and constant depth).

Proof. This uses the fact that the regular languages in AC 0 are exactly the languages definable in first-order logic with modular predicates (Barrington et al., 1992). See Appendix D.2 for details. □

An example of a language that belongs to this class but is not star-free is ( aa ) ∗ . The classic example of a language that is regular but not in AC 0 is PARITY = { 𝑤 ∈ { a , b } ∗ | 𝑤 has an odd number of b 's } (Furst et al., 1984).

Arbitrary position embeddings Finally, we may consider arbitrary position embeddings, subject to the condition of finite image. The corresponding collection of predicates is the set of all possible monadic predicates, which we call Mon. 5

Corollary 9. Masked hard-attention transformers that have position embeddings with finite image recognize exactly the languages definable in LTL [ Mon ] .

Barcel´ o et al. (2024) show that any language definable in LTL [ Mon ] can be recognized by a hardattention transformer without attention masking and with some position embedding (with infinite image), but left the other direction as an open question. Here, by making use of attention masking and restricting position embeddings to those with finite image, we have obtained an exact characterization.

The addition of attention masking appears to be important. With finite image position embeddings but without attention masking, there must be two positions 𝑖 and 𝑗 with the same position embedding (by the pigeonhole principle), so an unmasked attention transformer would not be able to distinguish one string with a and b at positions 𝑖 and 𝑗 and another string with a and b at positions 𝑗 and 𝑖 . So no masked hard-attention transformer with finite image position embeddings and unmasked attention can recognize the language # a ∗ # b ∗ # , but we showed already how to define this language even in LTL .

□

5.4 Depth hierarchy

Finally, we establish that (unlike with feed-forward networks), we can always increase the expressive power of masked hard-attention transformers by adding more self-attention layers. We consider masked hard-attention transformers with only future masking (as is typical in practice) and with only rightmost-hard attention. Other masking and tie-breaking schemes are treated in Appendix D.3. We also add multi-head attention (as is typical in practice).

First, we define depth for all models in this paper. The layer depth of a masked hard-attention transformer is the number of attention layers. The temporal depth of an LTL formula is as follows:

The attention depth of a B-RASP expression is defined as follows:

We then extend this definition to B-RASP operations. If 𝑃 ( 𝑖 ) : = 𝜙 ( 𝑖 ) (a position-wise operation),

Finally, the attention depth of a program is the maximum of the attention depths of all of its operations.

Let MUHAT ( ▶ 𝐹 ) 𝑘 (respectively, B-RASP ( ▶ 𝐹 ) 𝑘 ) be the languages recognizable by multi-head transformers of depth 𝑘 (respectively, B-RASP programs of depth 𝑘 ) using only future-masked rightmost-hard attention. Let LTL ( since ) 𝑘 be the languages definable by LTL formulas of depth 𝑘 without until .

Theorem 10. For every 𝑘 ≥ 0 , there is a language 𝐿 𝑘 + 1 such that no multi-head masked hardattention transformer of depth 𝑘 recognizes 𝐿 𝑘 , but a transformer of depth ( 𝑘 + 1 ) does recognize 𝐿 𝑘 + 1 .

Proof. The constructions in the proofs of Theorems 1 and 2 preserve depth, so B-RASP ( ▶ 𝐹 ) 𝑘 = LTL ( since ) 𝑘 . Moreover, by Theorem 4 (shallower version in Appendix C.2), and by Theorem 27 (a depth-preserving version of Theorem 3 found in Appendix D.3), MUHAT ( ▶ 𝐹 ) 𝑘 = B-RASP ( ▶ 𝐹 ) 𝑘 . Finally, Etessami and Wilke (2000) prove that LTL ( since ) 𝑘 ⊊ LTL ( since ) 𝑘 + 1 . Namely, the classes are separated by 𝐿 𝑘 + 1 = STAIR 𝑘 + 1 , which is the language over Σ = { a , b , c } of strings which, after deleting c 's, contain a 𝑘 + 1 as a substring. This gives the following picture:

Therefore, MUHAT ( ▶ 𝐹 ) 𝑘 ⊊ MUHAT ( ▶ 𝐹 ) 𝑘 + 1 .

6 Limitations

This work focuses exclusively on masked hard-attention transformers. We discussed the rationale for hard attention in Section 2.2. These results do not apply to softmax-attention transformers, although they demonstrate what kinds of results one might hope to obtain for softmax-attention transformers. Nor do they apply to transformers with unmasked attention.

Finally, our restriction of position embeddings to have finite image, and in particular our restriction of sinusoidal position embeddings to have angles that are rational multiples of 𝜋 , does not exactly match the standard definition.

□

References

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. In NIPS Deep Learning Symposium .

Pablo Barcel´ o, Alexander Kozachinskiy, Anthony Widjaja Lin, and Vladimir Podolskii. 2024. Logical languages accepted by transformer encoders with hard attention. In Proceedings of the 12th International Conference on Learning Representations (ICLR) .

David A. Mix Barrington, Kevin Compton, Howard Straubing, and Denis Th´ erien. 1992. Regular languages in NC 1 . Journal of Computer and System Sciences , 44(3):478-499.

David A. Mix Barrington, Neil Immerman, Clemens Lautemann, Nicole Schweikardt, and Denis Th´ erien. 2005. First-order expressibility of languages with neutral letters or: The Crane Beach conjecture. Journal of Computer and System Sciences , 70(2):101-127.

Satwik Bhattamishra, Kabir Ahuja, and Navin Goyal. 2020. On the ability and limitations of Transformers to recognize formal languages. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 7096-7116.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33 (NeurIPS) , pages 1877-1901.

David Chiang and Peter Cholak. 2022. Overcoming a theoretical limitation of self-attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL) , pages 7654-7664.

David Chiang, Peter Cholak, and Anand Pillay. 2023. Tighter bounds on the expressivity of transformer encoders. In Proceedings of the 40th International Conference on Machine Learning (ICML) , pages 5544-5562.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional Transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT) , pages 4171-4186.

Javid Ebrahimi, Dhruv Gelda, and Wei Zhang. 2020. How can self-attention networks recognize Dyck-n languages? In Findings of the Association for Computational Linguistics: EMNLP 2020 , pages 4301-4306.

Kousha Etessami and Thomas Wilke. 2000. An until hierarchy and other applications of an Ehrenfeucht-Fra¨ ıss´ e game for temporal logic. Information and Computation , 160(1-2):88-108.

Dan Friedman, Alexander Wettig, and Danqi Chen. 2023. Learning Transformer programs. In Advances in Neural Information Processing Systems 36 (NeurIPS) .

Merrick Furst, James B. Saxe, and Michael Sipser. 1984. Parity, circuits, and the polynomial-time hierarchy. Mathematical Systems Theory , 17:13-27.

Dov Gabbay, Amir Pnueli, Saharon Shelah, and Jonathan Stavi. 1980. On the temporal analysis of fairness. In Proceedings of the 7th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages (POPL) , pages 163-173.

Ankit Gupta, Guy Dar, Shaya Goodman, David Ciprut, and Jonathan Berant. 2021. Memory-efficient transformers via top-k attention. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing , pages 39-52.

Michael Hahn. 2020. Theoretical limitations of self-attention in neural sequence models. Transactions of the Association for Computational Linguistics , 8:156-171.

Yiding Hao, Dana Angluin, and Robert Frank. 2022. Formal language recognition by hard attention Transformers: Perspectives from circuit complexity. Transactions of the Association for Computational Linguistics , 10:800-810.

Johan Anthony Willem Kamp. 1968. Tense Logic and the Theory of Linear Order . Ph.D. thesis, University of California, Los Angeles.

Jambay Kinley. 2020. Two-Stream Transformer Architecture With Discrete Attention for Better Interpretrability and Separation of Model Concerns . Bachelor's thesis, Harvard College.

Oded Maler. 2010. On the Krohn-Rhodes cascaded decomposition theorem. In Time for Verification: Essays in Memory of Amir Pnueli , pages 260-278. Springer.

Robert McNaughton and Seymour Papert. 1971. Counter-Free Automata . Number 65 in M.I.T. Press Research Monographs. The M.I.T. Press.

William Merrill, Vivek Ramanujan, Yoav Goldberg, Roy Schwartz, and Noah A. Smith. 2021. Effects of parameter norm growth during transformer training: Inductive bias from gradient descent. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1766-1781.

William Merrill and Ashish Sabharwal. 2024. The expressive power of transformers with chain of thought. In Proceedings of the 12th International Conference on Learning Representations (ICLR) .

Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. 2022. In-context learning and induction heads. arXiv:2209.11895.

Doron Peled and Thomas Wilke. 1997. Stutter-invariant temporal properties are expressible without the next-time operator. Information Processing Letters , 63(5):243-246.

Binghui Peng. 2023. Personal communication.

Jorge P´ erez, Pablo Barcel´ o, and Javier Marinkovic. 2021. Attention is Turing-complete. Journal of Machine Learning Research , 22:75:1-75:35.

P. Sch¨ utzenberger. 1965. On finite monoids having only trivial subgroups. Information and Control , 8(2):190-194.

Lena Strobl, Dana Angluin, David Chiang, Jonathan Rawski, and Ashish Sabharwal. 2024a. Transformers as transducers. arXiv:2404.02040.

Lena Strobl, William Merrill, Gail Weiss, David Chiang, and Dana Angluin. 2024b. What formal languages can transformers express? A survey. Transactions of the Association for Computational Linguistics , 12:543-561.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30 (NIPS) .

Gail Weiss, Yoav Goldberg, and Eran Yahav. 2021. Thinking like Transformers. In Proceedings of the 38th International Conference on Machine Learning (ICML) , pages 11080-11090.

Hongfei Xu, Qiuhui Liu, Josef van Genabith, and Deyi Xiong. 2021. Learning hard retrieval decoder attention for Transformers. In Findings of the Association for Computational Linguistics: EMNLP , pages 779-785.

Andy Yang and David Chiang. 2024. Counting like transformers: Compiling temporal counting logic into softmax transformers. In Proceedings of the Conference on Language Modeling (CoLM) .

Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. 2021. Self-attention networks can process bounded hierarchical languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL-IJCNLP) , pages 3770-3785.

Hattie Zhou, Arwen Bradley, Etai Littwin, Noam Razin, Omid Saremi, Josh Susskind, Samy Bengio, and Preetum Nakkiran. 2024. What algorithms can Transformers learn? A study in length generalization. In Proceedings of the 12th International Conference on Learning Representations (ICLR) .

A Additional B-RASP Example: Associative Recall

We consider a variation of the simple associative recall task studied by Friedman et al. (2023) and many others in connection with induction heads and in-context learning (Olsson et al., 2022). Inputs are strings over the alphabet { a , b , c , 1 , 2 , 3 } , where letters and numbers alternate, starting with a letter and ending with a number, for example 𝑤 = a3b2b1a2c1a1c3 . The output alphabet adds the symbol ? . The desired output sequence copies the letters, and for a number at position 𝑖 , if 𝜎 is the letter at position 𝑖 -1 , then the most recent previous occurrence of 𝜎 is found, say at position 𝑗 , and the number at position 𝑗 + 1 is output. If there is no previous occurrence of 𝜎 , then ? is output instead. For the given example input 𝑤 , the output should be 𝑦 = a ? b ? b2a3c ? a2c1 .

The Boolean vector 𝑃 a determines whether there is an a in the preceding position. Similarly, the Boolean vectors 𝑃 b and 𝑃 c indicate whether there is a b or c , respectively, in the preceding position:

The program has an output Boolean vector 𝑌 𝜎 for each symbol 𝜎 ∈ Σ indicating whether the output symbol is 𝜎 . For 𝜎 ∈ { 1 , 2 , 3 } , if position 𝑖 is preceded by a letter, the output Boolean vector 𝑌 𝜎 attends to the most recent position 𝑗 preceded by the same letter (if any), and returns the value of 𝑄 𝜎 ( 𝑗 ) for that 𝑗 :

For 𝜎 ∈ { a , b , c } , the output Boolean vector 𝑌 𝜎 is just 𝑄 𝜎 :

Finally, the output Boolean vector 𝑌 ? is true if the position is preceded by a letter but no number was assigned:

The Boolean vectors in the computation for the example string 𝑤 are shown below.

B Proofs for Section 3 (Boolean RASP)

B.1 Unary value predicate

Proposition 11. Every B-RASP program is equivalent to one in which all value predicates 𝑉 ( 𝑖, 𝑗 ) depend only on 𝑗 .

Proof. This can be seen by the fact that the simulation of since / until (Appendix B.3) does not use a value predicate that depends on 𝑖 , but we can show this more directly by induction on the structure of 𝑉 ( 𝑖, 𝑗 ) . We only show how to handle ▶ ; the case of ◀ is very similar.

Consider an attention operation with the form

The base cases are 𝑉 ( 𝑖, 𝑗 ) = 𝐵 ( 𝑗 ) , for some Boolean vector 𝐵 , which already has the desired form, and 𝑉 ( 𝑖, 𝑗 ) = 𝐵 ( 𝑖 ) , in which case 𝑃 is equivalent to

If 𝑉 ( 𝑖, 𝑗 ) = 𝑉 1 ( 𝑖, 𝑗 ) ∧ 𝑉 2 ( 𝑖, 𝑗 ) , then 𝑃 ( 𝑖 ) is equivalent to

Similarly for disjunction and negation.

B.2 Unary score predicate

Lemma 12. Every B-RASP program is equivalent to one in which all score predicates 𝑆 ( 𝑖, 𝑗 ) depend only on 𝑗 .

Proof. Again, we only show the case of ▶ , as the case of ◀ is very similar. Consider a B-RASP attention operation 𝑃 ,

where 𝑓 is some Boolean function, and the 𝐴 𝑡 and 𝐵 𝑡 are B-RASP operations. Let A = { 𝐴 1 , . . . , 𝐴 𝑇 𝐴 } , and for each 𝜒 ⊆ A , define an assignment of truth values to the 𝐴 𝑡 ,

and use it to define a unary score 𝑆 𝜒 ( 𝑗 ) which uses the truth assignment of 𝜒 plugged into the 𝐴 𝑡 :

Now, 𝑃 ( 𝑖 ) is equivalent to 𝑃 ′ ( 𝑖 ) , where:

«

‹

To see why, observe that for any 𝑖 , there is exactly one truth assignment 𝜒 𝑖 that satisfies GLYPH<211> 𝑡 ∈[ 𝑇 𝐴 ] GLYPH<0> 𝐴 𝑡 ( 𝑖 ) ↔ 𝐴 𝜒 𝑡 GLYPH<1> . This 𝜒 𝑖 also makes 𝑆 𝜒 𝑖 ( 𝑗 ) equivalent to 𝑆 ( 𝑖, 𝑗 ) , and 𝑃 𝜒 𝑖 ( 𝑖 ) equivalent to 𝑃 ( 𝑖 ) .

+ □

Note that each attention operation translates into as many as 2 𝑇 𝐴 𝑇 𝐵 operations.

Observe that

□

B.3 Proof of Theorem 1 (LTL to B-RASP)

Theorem 1 follows from the following lemma.

Lemma 13. For any formula 𝜙 of LTL , there is a B-RASP program with a Boolean vector 𝑃 𝜙 such that, for any input 𝑤 of length 𝑛 and all 𝑖 ∈ [ 𝑛 ] , we have 𝑤,𝑖 | = 𝜙 iff 𝑤 | = 𝑃 𝜙 ( 𝑖 ) .

Proof. By induction on the structure of the formula 𝜙 . We assume that a B-RASP program always contains the initial Boolean vectors 𝑄 𝜎 for 𝜎 ∈ Σ .

If 𝜙 = 𝑄 𝜎 : Add operation

If 𝜙 = 𝜙 1 ∨ 𝜙 2 : Similar, but add

If 𝜙 = 𝜙 1 since 𝜙 2 : Similar, but add

If 𝜙 = 𝜙 1 until 𝜙 2 : Similar, but add

B.4 Proof of Theorem 2 (B-RASP to LTL)

Theorem 2 follows from the following lemma.

Lemma 14. For any Boolean vector 𝑃 of a B-RASP program P , there is a formula 𝜙 𝑃 of LTL such that for any input 𝑤 of length 𝑛 and all 𝑖 ∈ [ 𝑛 ] , we have 𝑤 | = 𝑃 ( 𝑖 ) iff 𝑤,𝑖 | = 𝜙 𝑃 .

Proof. First, by Lemma 12 and Proposition 11 we can rewrite P to an equivalent program such that every attention operation only uses unary scores and unary values.

Each initial Boolean vector 𝑄 𝜎 ( 𝑖 ) translates to the atomic formula 𝑄 𝜎 .

For each operation 𝑃 𝑡 ( 𝑖 ) (for 𝑡 > | Σ | ), if it is a position-wise operation, that is,

where 𝑓 is a Boolean function, then by the inductive hypothesis, there are LTL formulas 𝜙 𝑡 for each 𝑃 𝑡 ( 𝑖 ) . Then we convert 𝑃 𝑡 ( 𝑖 ) into 𝜙 𝑡 = 𝑓 ( 𝜙 1 , . . . , 𝜙 𝑡 -1 ) .

If 𝑃 𝑡 ( 𝑖 ) is an attention operation, define

also known as eventually

If 𝑃 𝑡 ( 𝑖 ) uses ▶ and future masking, that is,

If 𝜙 = ¬ 𝜙 1 : By the induction hypothesis, convert 𝜙 1 to B-RASP operations, including one that computes 𝑃 𝜙 1 . Then add the operation

If 𝜙 = 𝜙 1 ∧ 𝜙 2 : By the induction hypothesis, convert 𝜙 1 to B-RASP operations, including one that computes 𝑃 𝜙 1 , then convert 𝜙 2 to B-RASP operations, including one that computes 𝑃 𝜙 2 . Then add operation

□

then by the inductive hypothesis, there are LTL formulas 𝜙 𝑆 , 𝜙 𝑉 and 𝜙 𝐷 for the corresponding B-RASP operations. Then we can convert 𝑃 𝑡 ( 𝑖 ) into the LTL formula

If 𝑃 𝑡 ( 𝑖 ) uses ▶ and past masking, that is,

then

And if 𝑃 𝑡 ( 𝑖 ) uses ▶ with no masking, that is,

then

The cases for ◀ are symmetric.

B.5 Counter-free automata to B-RASP

In this section, we give an alternative proof of Theorem 1 using the fact that the star-free languages are exactly those recognized by counter-free automata .

A deterministic finite automaton (DFA) is a tuple 𝐴 = ( Σ , 𝑄, 𝛿 ) , where Σ is the input alphabet, 𝑄 is the finite set of states, and 𝛿 : 𝑄 × Σ → 𝑄 is the transition function. A counter-free automaton is a DFA in which no string induces a permutation on any subset of 𝑄 other than the identity. Sch¨ utzenberger (1965) proved that the star-free languages are exactly those recognized by counter-free automata.

Theorem 15. For any counter-free DFA that recognizes a language 𝐿 ⊆ Σ + , there is a B-RASP program that recognizes 𝐿 .

A counter-free automaton can be decomposed using Krohn-Rhodes theory into a cascade of identityreset automata , each of which can be simulated in B-RASP .

Maler (2010) gives the following automata-theoretic version of the Krohn-Rhodes decomposition theorem, which we explain below.

Theorem 16 (Maler, 2010, Theorem 3) . For every [deterministic finite] automaton 𝐴 there exists a cascade decomposition

such that the following are true.

Each 𝐵 𝑖 is a permutation-reset automaton.

There is a homomorphism 𝜙 from 𝐶 to 𝐴 .

Any permutation group in some 𝐵 𝑖 is homomorphic to a subgroup of the transformation semigroup of 𝐴 .

The pair ( 𝐶, 𝜙 ) is called a cascade decomposition of 𝐴 .

If 𝐵 1 = ( Σ , 𝑄 1 , 𝛿 1 ) and 𝐵 2 = ( 𝑄 1 × Σ , 𝑄 2 , 𝛿 2 ) are DFAs, their cascade product 𝐶 = 𝐵 1 ◦ 𝐵 2 is the automaton ( Σ , 𝑄 1 × 𝑄 2 , 𝛿 ) such that 𝛿 ( 𝑞 1 𝑞 2 , 𝜎 ) = ( 𝛿 1 ( 𝑞 1 , 𝜎 ) , 𝛿 2 ( 𝑞 2 , ( 𝑞 1 , 𝜎 ))) . (To reduce clutter, we write tuples of states without commas.) The cascade product allows the automaton 𝐵 2 to see the current state of 𝐵 1 in deciding what transition to take. We define iterated cascade product inductively by 𝐵 1 ◦ 𝐵 2 ◦ · · · ◦ 𝐵 𝑘 = ( 𝐵 1 ◦ 𝐵 2 ◦ · · · 𝐵 𝑘 -1 ) ◦ 𝐵 𝑘 . This allows each 𝐵 𝑖 to see the current state of all 𝐵 𝑗 with 𝑗 ≤ 𝑖 in deciding what transition to take. (For readers more familiar with finite transducers (Mealy machines), 𝐵 1 and 𝐵 2 could be thought of as transducers whose transitions are all of the form

□

0

1

2

3

R

L

R

L

R

L

L

R

(a) Automaton 𝐴 3 : move right ( R ), move left ( L ).

𝐴

𝐵

𝐶

𝐷

𝐸

𝐹

R

L

R

L

𝐵,

R

𝐴,

L

𝐴,

L

𝐵,

R

𝐵𝐷,

R

𝐴𝐶,

L

𝐴𝐶,

L

𝐵𝐷,

R

(b) Cascade of identity-reset automata for 𝐴 3 . Omitted transitions are self-loops; for example, inputs 𝐴, R and 𝐵, L are self-loops on 𝐶 and 𝐷 .

𝐴𝐶𝐸

(

0

)

𝐵𝐶𝐸

(

1

)

𝐴𝐷𝐸

(

1

)

𝐵𝐷𝐸

(

2

)

𝐴𝐶𝐹

(

1

)

𝐵𝐶𝐹

(

2

)

𝐴𝐷𝐹

(

2

)

𝐵𝐷𝐹

(

3

)

R

L

L

R

L

R

L

R

R

L

L

R

L

R

L

R

(c) The global automaton of the cascade product in part (b). For state 𝑞 , the number in parentheses is the state 𝜙 ( 𝑞 ) in 𝐴 3 .

In Property (1), a permutation-reset automaton is a DFA 𝐴 = ( Σ , 𝑄, 𝛿 ) such that for every 𝜎 ∈ Σ , the mapping 𝑞 ↦→ 𝛿 ( 𝑞, 𝜎 ) is either a permutation (for all 𝑟 , there is a 𝑞 such that 𝛿 ( 𝑞, 𝜎 ) = 𝑟 ) or constant (there is a 𝑞 𝜎 such that 𝛿 ( 𝑞, 𝜎 ) = 𝑞 𝜎 for all 𝑞 ).

Property (2) says that there is a mapping 𝜙 from the states of 𝐶 to the states of 𝐴 such that for any state 𝑞 of 𝐶 , we have 𝜙 ( 𝛿 𝐶 ( 𝑞, 𝜎 )) = 𝛿 𝐴 ( 𝜙 ( 𝑞 ) , 𝜎 ) .

Property (3) implies that if the automaton 𝐴 is counter-free, then all of the automata 𝐵 𝑖 in the cascade decomposition are identity-reset automata. An identity-reset automaton is a DFA 𝐴 = ( Σ , 𝑄, 𝛿 ) such that for every 𝜎 ∈ Σ , the mapping 𝑞 ↦→ 𝛿 ( 𝑞, 𝜎 ) is either the identity ( 𝛿 ( 𝑞, 𝜎 ) = 𝑞 for all 𝑞 ) or constant (there is a 𝑞 𝜎 such that 𝛿 ( 𝑞, 𝜎 ) = 𝑞 𝜎 for all 𝑞 ).

For example, consider the automaton 𝐴 3 shown in Figure 3a. A decomposition of this automaton into a cascade product of three identity-reset automata is shown in Figure 3b. The global automaton derived from the decomposition is shown in Figure 3c with the homomorphism 𝜙 to states of 𝐴 3 .

Let 𝐴 = ( Σ , 𝑄, 𝛿 ) be a DFA and 𝑠 ∈ 𝑄 . For a string 𝑤 1 · · · 𝑤 𝑛 ∈ Σ ∗ , the sequence of states traversed by 𝐴 from state 𝑠 on this input is 𝑞 0 , . . . , 𝑞 𝑛 , where 𝑞 0 = 𝑠 and for each 𝑘 , 𝑞 𝑘 + 1 = 𝛿 ( 𝑞 𝑘 , 𝑤 𝑘 ) . A B-RASP program 𝑝 simulates 𝐴 started in state 𝑠 iff for every input word 𝑤 ∈ Σ ∗ , the output Boolean vectors of 𝑝 on input 𝑤 encode the sequence of states traversed by 𝐴 from state 𝑠 on input 𝑤 . The state at position 𝑖 is the state before the symbol at position 𝑖 is read.

Lemma 17. Let 𝐵 = ( Σ , 𝑄, 𝛿 ) be any identity-reset automaton, and let 𝑠 ∈ 𝑄 be a start state. There exists a B-RASP program P 𝐵 that simulates 𝐵 started in state 𝑠 .

Proof. For each state 𝑟 ∈ 𝑄 , let 𝑅 𝑟 ⊆ Σ be the state of symbols that reset to 𝑟 (that is, 𝛿 ( 𝑞, 𝜎 ) = 𝑟 for all 𝑞 ∈ 𝑄 ). Let 𝑅 = -𝑟 ∈ 𝑄 𝑅 𝑟 . To determine if 𝐵 is in state 𝑞 ≠ 𝑠 before reading 𝑤 𝑖 , it is sufficient to attend to the closest position 𝑗 < 𝑖 that contains a symbol from 𝑅 , if any. If 𝑗 exists and 𝑤 𝑗 ∈ 𝑅 𝑞 , then 𝐵 is in state 𝑞 at position 𝑖 . Otherwise, it is not.

The case of state 𝑠 is slightly different. In the case that there is no position 𝑗 < 𝑖 that contains a symbol from 𝑅 , then 𝐵 never left the initial state 𝑠 , so it is still in state 𝑠 at position 𝑖 .

In B-RASP , we can define a Boolean vector 𝐵 𝑞 ( 𝑖 ) , which is true iff 𝐵 is in state 𝑞 at position 𝑖 :

Lemma 18. Let 𝐵 1 = ( Σ , 𝑄 1 , 𝛿 1 ) be a DFA that can be simulated from state 𝑠 1 by a B-RASP program P 𝐵 1 . Let 𝐵 2 = ( 𝑄 1 × Σ , 𝑄 2 , 𝛿 2 ) be an identity-reset automaton and let 𝐶 = 𝐵 1 ◦ 𝐵 2 . Then there is a B-RASP program P 𝐶 that simulates 𝐶 started in state ( 𝑠 1 , 𝑠 2 ) for an arbitrary 𝑠 2 ∈ 𝑄 2 .

Proof. Let 𝐵 1 ,𝑞 ( 𝑖 ) be predicates that test whether 𝐵 1 is in state 𝑞 at position 𝑖 started in state 𝑠 1 , and let 𝐵 2 ,𝑞 ( 𝑖 ) be predicates that test whether 𝐵 2 is in state 𝑞 at position 𝑖 started in state 𝑠 2 (by Lemma 17). Define

and for every line in the definition of the 𝐵 2 ,𝑞 , replace every occurrence of 𝑄 ( 𝑞,𝜎 ) with 𝑄 ′ ( 𝑞,𝜎 ) . This yields the definition of new predicates 𝐵 ′ 2 ,𝑞 . Then we can define predicates 𝐶 ( 𝑞,𝑟 ) ( 𝑖 ) that test whether 𝐶 = 𝐵 1 ◦ 𝐵 2 is in state ( 𝑞, 𝑟 ) at position 𝑖 :

By induction on 𝑘 we have the following.

Lemma 19. Let 𝐶 = 𝐵 1 ◦ 𝐵 2 ◦ · · · ◦ 𝐵 𝑘 be a cascade product such that each 𝐵 𝑖 is an identity-reset automaton, and 𝑠 𝑖 is a state of 𝐵 𝑖 . Then there is a B-RASP program P 𝐶 that can simulate 𝐶 from state ( 𝑠 1 , 𝑠 2 , . . . , 𝑠 𝑘 ) .

If we add instructions to the program in Lemma 19 that compute the homomorphism 𝜙 (from property (2) of Theorem 16) from the states of the cascade product 𝐶 to the automaton 𝐴 :

then we get a B-RASP program P 𝐴 that simulates 𝐴 started in state 𝑠 .

Finally, we add to this program position-wise operations 𝑌 𝑞 ( 𝑖 ) that decide whether 𝐴 started in state 𝑠 ends up in state 𝑞 after reading the symbol at position 𝑖 :

If 𝑓 is the final state of 𝐴 , then let 𝑌 𝑓 be the output vector of the program. Since 𝑌 𝑓 ( 𝑛 ) = 1 if and only if 𝐴 accepts 𝑤 , this completes the proof of Theorem 15.

C Proofs for Section 4 (Masked Hard-Attention Transformers)

C.1 Proof of Theorem 3 (B-RASP to masked hard-attention transformers)

We will make use of the following lemma repeatedly:

Lemma 20. Any function 𝑓 : { 0 , 1 } 𝑑 →{ 0 , 1 } 𝑑 can be computed by a two-layer FFN with ReLU activation functions.

Proof. Any Boolean function can be written in full disjunctive normal form (DNF), which is a disjunction of conjunctions, and each conjunction is a conjunction of one positive or negative literal for each of the arguments, so that at most one conjunction is 1 for any assignment.

Each component of 𝑓 can be put into full DNF and computed by a two-layer FFN with ReLU activation functions. The first layer computes all the negations and conjunctions, using the fact that for any Boolean values 𝑏 1 , 𝑏 2 , . . . , 𝑏 𝑚 we have

The second layer computes the disjunction simply by adding the values of the conjunctions. □

Let P be a B-RASP program with Boolean vectors 𝑃 1 , . . . , 𝑃 𝑇 . We say that a masked hard-attention transformer T with width 𝑑 ≥ 𝑇 simulates P iff for every input 𝑤 ∈ Σ + with length 𝑛 , we have, for all 𝑖 ∈ [ 𝑛 ] and 𝑡 ∈ [ 𝑇 ] ,

Theorem 3 follows from the following lemma:

Lemma 21. Let P be a B-RASP program. There exists a masked hard-attention transformer TP that simulates P .

Proof. We prove that the first 𝑡 operations of P can be simulated by a masked hard-attention transformer, by induction on 𝑡 .

The base case is 𝑡 = | Σ | . For 𝑐 ∈ 1 , . . . , | Σ | , let 𝜎 𝑐 be the 𝑐 -th symbol in Σ . Let emb ( 𝜎 𝑐 ) be the one-hot vector with [ emb ( 𝜎 𝑐 )] 𝑐 = 1 .

For the inductive step, assume that the Boolean vectors 𝑃 1 , . . . , 𝑃 𝑡 can be simulated by a masked hard-attention transformer. We want to show that 𝑃 𝑡 + 1 can be simulated as well.

If 𝑃 𝑡 + 1 ( 𝑖 ) is a Boolean combination of { 𝑃 1 ( 𝑖 ) , . . . , 𝑃 𝑡 ( 𝑖 )} , it can be computed by a two-layer FFN by Lemma 20.

The most important case is if 𝑃 𝑡 + 1 ( 𝑖 ) is an attention operation, either of

We only show the ▶ case; ◀ is similar.

We need to slightly modify the value predicate:

This does not change the behavior of 𝑃 𝑡 + 1 (because 𝑆 ( 𝑖, 𝑗 ) is always true when evaluating the value predicate), but it will preserve the correct behavior in a transformer, where attention attends to the leftmost maximum score. By Proposition 11, the attention operation can be rewritten so that that the value predicate depends only on 𝑗 . So, without loss of generality, assume that there is a Boolean function 𝑔 such that

The score predicate 𝑆 ( 𝑖, 𝑗 ) can be written in full DNF in terms of the 𝑃 ℓ ( 𝑖 ) and 𝑃 ℓ ( 𝑗 ) . We separate each conjunction of 𝑆 ( 𝑖, 𝑗 ) into a conjunction of literals depending on 𝑖 and a conjunction of literals depending on 𝑗 . Thus, there is a collection of Boolean functions 𝛼 ℓ and 𝛽 ℓ such that

We construct two layers, as follows. For brevity, we write

and for any 𝑚 we write 0 𝑚 for the 𝑚 -dimensional zero vector.

Assume that the input to the first layer is





The first self-attention has value vectors set to 0 , so that the residual connection computes the identity function ( 𝑐 ( 1 ) 𝑖 = 𝑥 ( 1 ) 𝑖 ), and the first position-wise FFN can be constructed, by Lemma 20, so that





In the second layer, the self-attention has mask 𝑀 . The score function is

where 0 𝑚 × 𝑚 and I 𝑚 × 𝑚 are the 𝑚 × 𝑚 zero and identity matrices.

The value function 𝑓 ( 2 ) 𝑉 is such that for any 𝑗 ∈ [ 𝑛 ] ,





So the output of the second self-attention, after the residual connection, is as follows.

If 𝑖 > 1 :

If

𝑖

=

1

:

The second feed-forward network 𝑓 ( 2 ) 𝑃 checks the default flag. If it is GLYPH<2> 1 0 GLYPH<3> , it copies the attended-to value 𝑔 ( 𝑃 1 ( 𝑗 𝑖 ) , . . . 𝑃 𝑡 ( 𝑗 𝑖 )) to the ( 𝑡 + 1 ) -st coordinate. If it is GLYPH<2> 0 1 GLYPH<3> , it computes 𝐷 ( 𝑖 ) (using Lemma 20) in the ( 𝑡 + 1 ) -st coordinate. Thus, the output after the residual connection is as follows.

If 𝑖 > 1 :

If

𝑖

=

1

:

In either case, the ( 𝑡 + 1 ) -st coordinate is now 1 if 𝑤 | = 𝑃 𝑡 + 1 ( 𝑖 ) and 0

otherwise. □

The last step is to construct the output layer, which simply projects the final-layer activation vectors down to the coordinate that simulates 𝑌 and subtracts 1 2 . This completes the proof of Theorem 3.

C.2 Proof of Theorem 4 (masked hard-attention transformers to B-RASP)

The key to the translation from masked hard-attention transformers to B-RASP is the following lemma:

Lemma 22. Let T be a masked hard-attention transformer. There is a finite set F ⊆ R such that for all input strings 𝑤 , all the attention scores and activations computed by T on input 𝑤 belong to F .

Proof. We prove that regardless of the input, layer 𝑘 has at most (| Σ | + 1 ) 2 𝑘 -1 different possible output activation vectors, by induction on 𝑘 . The base case is just the embedding function. Since there are no position embeddings, the embedding at position 𝑖 is determined entirely by 𝑤 𝑖 , so there are at most | Σ | ≤ (| Σ | + 1 ) 2 𝑘 -1 possible activation vectors.

Assume that T has ( 𝑘 + 1 ) layers and that layer 𝑘 has at most (| Σ | + 1 ) 2 𝑘 -1 possible activation vectors. The self-attention's output at position 𝑖 depends only on two vectors: (1) layer 𝑘 's output at position 𝑖 (because of the residual connection) and (2) either layer 𝑘 's output at position 𝑗 𝑖 (the position that 𝑖 attends to) or 0 (if there are no unmasked positions). Thus the number of possible activation vectors that the self-attention can output is at most

And the number of possible activation vectors that the position-wise FFN can output is also at most (| Σ | + 1 ) 2 𝑘 + 1 -1 .

As for the attention scores, at layer ( 𝑘 + 1 ) , every attention score depends on two activation vectors from layer 𝑘 , so there are at most GLYPH<0> (| Σ | + 1 ) 2 𝑘 -1 GLYPH<1> 2 ≤ (| Σ | + 1 ) 2 𝑘 -1 possible scores.

Then F is the union over all layers of the possible attention scores and components of the possible activation vectors. □

So any attention score or component of an activation vector can be represented using 𝐵 = ⌈ log 2 | F |⌉ bits. Define a mapping ⟨·⟩ : F →{ 0 , . . . , 2 𝐵 -1 } such that 𝑢 < 𝑣 iff ⟨ 𝑢 ⟩ < ⟨ 𝑣 ⟩ , and write ⟨ 𝑣 ⟩ 𝑏 for the bit of ⟨ 𝑣 ⟩ with place value 2 𝑏 .

Lemma 23. For any function 𝑓 : F → F , there are Boolean formulas 𝜙 𝑏 ( 𝑥 1 , . . . , 𝑥 𝐵 ) for 𝑏 ∈ [ 𝐵 ] such that for any 𝑥 ∈ F , 𝜙 𝑏 (⟨ 𝑥 ⟩ 1 , . . . , ⟨ 𝑥 ⟩ 𝐵 ) holds iff ⟨ 𝑓 ( 𝑥 )⟩ 𝑏 = 1 .

Proof. One way to define 𝜙 𝑏 is:

Depending on the mapping ⟨·⟩ , more efficient definitions may be possible.

□

Hopefully, it is clear how to generalize this lemma to functions F 𝑑 × F 𝑑 →{ 0 , 1 } or F 𝑑 → F 𝑑 .

Next, we prove Theorem 4. Let T be a masked hard-attention transformer with width 𝑑 . Let 𝐵 be the number of bits needed to store T 's activation vector components and attention scores, by Lemma 22. A B-RASP program P simulates T if in P there are Boolean vectors 𝑌 𝑐,𝑏 for 𝑐 ∈ [ 𝑑 ] and 0 ≤ 𝑏 < 𝐵 such that for any input 𝑤 ∈ Σ + of length 𝑛 , for all 𝑖 ∈ [ 𝑛 ] , 𝑐 ∈ [ 𝑑 ] , and 0 ≤ 𝑏 < 𝐵 , we have 𝑤 | = 𝑌 𝑐,𝑏 ( 𝑖 ) iff ⟨[T ( 𝑤 )] 𝑖,𝑐 ⟩ 𝑏 = 1 .

Lemma 24. For any masked hard-attention transformer T , there is a B-RASP program P 𝑇 that simulates T .

Proof. We proceed by induction on the depth of T . The base case is the input embedding function emb , which is simulated by Boolean vectors for 𝑐 ∈ [ 𝑑 ] and 0 ≤ 𝑏 < 𝐵 :

Assume that the first 𝑘 layers of T are simulated by a program P . We extend P to simulate layer ( 𝑘 + 1 ) as follows.

If the self-attention uses rightmost-hard attention with mask 𝑀 ( 𝑖, 𝑗 ) , assume (by Lemma 23) that the score function 𝑓 𝑆 ( 𝑖, 𝑗 ) has been converted to Boolean expressions 𝑆 ′ 𝑏 ( 𝑖, 𝑗 ) for the 𝑏 -th bit of the score for positions 𝑖 and 𝑗 , and the value function 𝑓 𝑉 ( 𝑗 ) has been converted to Boolean expressions 𝑉 ′ 𝑐,𝑏 ( 𝑗 ) for the 𝑏 -bit of the 𝑐 -th coordinate of the value.

We give two translations. The first version has depth 1 , which is important in Section 5.4. The second version is deeper in general, but much smaller.

Shallower version:

Because F is finite, by Lemma 23 we can define, for all 𝑣 ∈ F , predicates

Then for each 𝑣 ∈ F , add operations for Max 𝑣 ( 𝑖 ) , which check that the score 𝑣 is the maximum, and Rightmost 𝑣,𝑐,𝑏 ( 𝑖 ) , which retrieve the value at the rightmost position with score 𝑣 :

Then we can add operations for Att 𝑐,𝑏 ( 𝑖 ) , which hold just in case the 𝑏 -th bit of the 𝑐 -th coordinate of the attention output is 1 , by taking a disjunction over the finitely many possible scores:

Smaller version:

We need to define a predicate Argmax ( 𝑖, 𝑗 ) that tests whether 𝑗 maximizes 𝑆 ( 𝑖, 𝑗 ) . To do this, we define a sequence of Boolean vectors that test whether 𝑗 maximizes bits 𝑏, . . . , 𝐵 -1 of 𝑆 ( 𝑖, 𝑗 ) :

Finally, we add operations that simulate attention:

To simulate leftmost-hard attention, simply change ▶ to ◀ .

For the position-wise feed-forward network, use Lemma 23.

□

The last step in the program is to use position-wise operations to simulate T 's output layer, yielding an output Boolean vector 𝑌 . This completes the proof of Theorem 4.

D Proofs for Section 5 (Further Results)

D.1 Proof of Theorem 7 (position embeddings can be simulated by predicates)

Because Θ has finite image, Lemma 22 still holds for any masked hard-attention transformer with position embedding Θ . Let P Θ be the collection of predicates that test whether the 𝑏 -th bit of the 𝑐 -th coordinate of 𝜃 𝑛 ( 𝑖 ) is set. The proof of equivalence of masked hard-attention transformers with B-RASP extends easily to equivalence of masked hard-attention transformers with position embedding Θ and B-RASP [P Θ ] . When converting a transformer to a B-RASP [P Θ ] program, we represent each coordinate of Θ with 𝐵 predicates from P Θ . When converting a B-RASP [P Θ ] program to a transformer, we represent each predicate in P Θ with its own coordinate, whose value is in { 0 , 1 } .

Since Theorems 1 and 2 hold for any collection of unary predicate symbols, B-RASP [P Θ ] is equivalent to LTL [P Θ ] .

D.2 Proof of Corollary 8 (masked hard-attention transformers with sinusoidal position embeddings recognize the regular languages in AC 0 )

Let MOD be the collection of predicates MOD 𝑟 𝑚 ( 𝑖 ) for all 0 ≤ 𝑟 < 𝑚 , which hold just in case 𝑖 ≡ 𝑟 ( mod 𝑚 ) .

Let Θ be a sinusoidal positional embedding. Since the 𝑓 𝑐 are rational, Θ has finite image. By Theorem 7, transformers with positional embedding Θ are equivalent to LTL [P Θ ] .

It's easy to see that every predicate in P Θ can be expressed in terms of MOD; for the converse, observe that we can use a 2-layer ReLU network to compute MOD 𝑟 𝑚 (Chiang et al., 2023, Lemma 20):

Thus transformers with sinusoidal positional embeddings are equivalent to LTL [ MOD ] , which is equivalent to FO [ <, MOD ] (Kamp, 1968), which defines exactly the class of regular languages in AC 0 (Barrington et al., 1992).

D.3 Details for Section 5.4 (depth hierarchy)

D.3.1 Multi-head attention

To prove Theorem 10 and related results, we need to make Theorem 3 more efficient in terms of the depth of the constructed transformer. To do this, we'll need to make use of multi-head attention. This allows multiple self-attentions at the same depth to be run in parallel. In a multi-head masked hard-attention transformer transformer layer, the equation for the self-attention (Equation (1)) is replaced by

where each att .ℎ is a self-attention layer.

It is straightforward to extend Theorem 4 to multi-head masked hard-attention transformers, simulating a multi-head masked hard-attention transformer of depth 𝑘 with a B-RASP program of depth 𝑘 . Each head at depth 𝑘 can be simulated by a B-RASP attention operation of attention depth 𝑘 , and their sum can be simulated by a position-wise operation (Lemma 23).

D.3.2 Parallel composition

The parallelization is accomplished by the following construction.

Lemma 25. A transformer T 1 of depth 𝑘 1 with 𝐻 1 heads and a transformer T 2 of depth 𝑘 2 with 𝐻 2 heads can be parallel-composed into a transformer T 1 ⊕ T 2 of depth max ( 𝑘 1 , 𝑘 2 ) with 𝐻 1 + 𝐻 2 heads such that

Proof. First, add layers that compute the identity function to the shallower transformer so that both have depth max ( 𝑘 1 , 𝑘 2 ) .

Next, concatenate their word embedding vectors

At each level, we compose the self-attentions using multiple heads to simulate them in parallel. For each multi-head self-attention layer att 1 and att 2 at the same depth in each transformer, we use multiple heads to simulate both att 1 and att 2 in parallel. Let att 1 .ℎ. 𝑓 𝑆 be the score function of

the ℎ -th head of att 1 , and similarly for att 1 .ℎ.𝑀 , att 1 .ℎ.𝐶 , and att 1 .ℎ. 𝑓 𝑉 , and similarly for att 2 . Let 𝑑 = 𝑑 1 + 𝑑 2 and 𝐻 = 𝐻 1 + 𝐻 2 . Construct a new self-attention layer att 1 ⊕ att 2 with



For the feed-forward networks ffn 1 and ffn 2 , create a new network ffn 1 ⊕ ffn 2 with

It is straightforward to verify the correctness of this construction.

□

D.3.3 B-RASP to masked hard-attention transformers, preserving depth

We give a more efficient version of Theorem 3, which uses parallel composition to optimize the depth of the constructed transformer.

Lemma 26. Let T be a transformer (without output layer) with width 𝑑 and depth 𝑘 , and whose activations are in { 0 , 1 } . For any function 𝑔 : { 0 , 1 } 𝑑 →{ 0 , 1 } 𝑑 , there is a transformer ( 𝑔 ◦ T) with depth 𝑘 such that, for all 𝑤 and 𝑖 , [( 𝑔 ◦ T)( 𝑤 )] 𝑖 = 𝑔 ([T ( 𝑤 )] 𝑖 ) .

Proof. If 𝑘 = 0 : T consists of just an embedding function emb : Σ →{ 0 , 1 } 𝑑 . Then 𝑔 ◦ T = 𝑔 ◦ emb is also an embedding function and therefore a depth0 transformer.

If 𝑘 > 0 : Let 𝑓 : { 0 , 1 } 𝑑 → { 0 , 1 } 𝑑 be the top FFN of T . Then 𝑔 ◦ 𝑓 is also a function { 0 , 1 } 𝑑 → { 0 , 1 } 𝑑 and can therefore be computed by a single FFN, by Lemma 20. □

Theorem 27. For any B-RASP program P of depth 𝑘 that recognizes a language 𝐿 ⊆ Σ + , there is a multi-head masked hard-attention transformer with depth 𝑘 that recognizes 𝐿 .

Proof. Let P be any B-RASP program. For any operation 𝑃 𝑡 ( 𝑖 ) of P , we say that a transformer T 𝑡 with width 𝑑 simulates 𝑃 𝑡 ( 𝑖 ) if there is a 𝑐 ∈ [ 𝑑 ] such that, for all 𝑤 ∈ Σ + ,

We prove the following statement by induction on 𝑡 : For any operation 𝑃 𝑡 ( 𝑖 ) of P with depth 𝑘 , there is a multi-head masked hard-attention transformer with depth 𝑘 that simulates 𝑃 𝑡 ( 𝑖 ) .

The base cases are 𝑡 ≤ | Σ | , where every operation can be simulated by a transformer with depth 0 using one-hot word embeddings, just as in the proof of Theorem 3.

If 𝑡 > | Σ | , assume that each previous operation 𝑃 𝑡 ′ ( 𝑖 ) with depth 𝑘 ′ can be simulated by a transformer with depth 𝑘 ′ . We will construct a transformer of depth 𝑘 that simulates 𝑃 𝑡 ( 𝑖 ) .

If 𝑃 𝑡 ( 𝑖 ) is an attention operation, then its 𝑆 , 𝑉 , and 𝐷 predicates have depth at most ( 𝑘 -1 ) and therefore depend only on operations which can be simulated by transformers of depth ( 𝑘 -1 ) , by the inductive hypothesis. Parallel-compose all of these into a single transformer

(using Lemma 25) to obtain a transformer ˜ 𝑇 𝑡 of depth ( 𝑘 -1 ) that simulates all the operations that 𝑃 𝑡 ( 𝑖 ) depends on.

Then, extend ˜ 𝑇 𝑡 to compute the attention operation as in the proof of Theorem 3. Although that construction uses two transformer layers, the first layer's self-attention just computes the identity function, and its FFN can be fused with the last FFN in ˜ 𝑇 𝑡 by Lemma 26. So there exists a transformer 𝑇 𝑡 of depth 𝑘 that simulates 𝑃 𝑡 ( 𝑖 ) .

If 𝑃 𝑡 ( 𝑖 ) is a position-wise operation of depth 𝑘 , it may depend on earlier operations that also have depth 𝑘 . By the inductive hypothesis, these can be simulated by transformers of depth 𝑘 . Parallel-compose them to get a transformer ˜ 𝑇 𝑡 of depth 𝑘 , by Lemma 25. Then, the computation of 𝑃 𝑡 ( 𝑖 ) itself can be fused with the last feed-forward network in ˜ 𝑇 𝑡 by Lemma 26. This again results in a transformer 𝑇 𝑡 of depth 𝑘 that simulates 𝑃 𝑡 ( 𝑖 ) .

Thus, if 𝑃 𝑇 is the output vector of P and has depth 𝑘 , then there is a transformer that simulates 𝑃 𝑇 and has depth 𝑘 . Add an output layer that transforms the output at position 𝑛 to + 1 2 if 𝑤 | = 𝑃 𝑇 ( 𝑛 ) and -1 2 otherwise. Then, 𝑇 recognizes the same language that P recognizes. □

D.3.4 Other attention variants

Earlier, we used transformers and B-RASP with only future-masked rightmost-hard-attention, and LTL with only since . Theorem 10 showed that

The separating language was STAIR 𝑘 + 1 , which is the language over Σ = { a , b , c } of strings which, after deleting c 's, contain a 𝑘 + 1 as a substring. We can recognize STAIR 𝑘 with a formula 𝜑 𝑘 = 1 since 𝛾 𝑘 , defined as follows:

Note the slight deviation from Etessami and Wilke (2000), because their presentation of LTL used the next and eventually operators as well as non-strict until ′ .

Next, we allow both future-masked rightmost and past-masked leftmost attention. We will notate these with MUHAT ( ▶ 𝐹, ◀ 𝑃 ) 𝑘 and B-RASP ( ▶ 𝐹, ◀ 𝑃 ) 𝑘 , respectively. In LTL , we allow access to both temporal operators; let LTL 𝑘 be the languages definable by formulas of depth 𝑘 .

Proposition 28. Restricted to only rightmost future-masked and leftmost past-masked attention, multi-head masked hard-attention transformers with depth ( 𝑘 + 1 ) are strictly more expressive than multi-head masked hard-attention transformers with depth 𝑘 .

Proof. The constructions in Theorems 1 and 2 preserve depth for rightmost future-masked and leftmost past-masked attention, so B-RASP ( ▶ 𝐹, ◀ 𝑃 ) 𝑘 = LTL 𝑘 . Moreover, the constructions in Theorems 3 and 4 are identical regardless of attention type, so MUHAT ( ▶ 𝐹, ◀ 𝑃 ) 𝑘 = B-RASP ( ▶ 𝐹, ◀ 𝑃 ) 𝑘 . Finally, Etessami and Wilke (2000) show that with access to both temporal operators it is still the case that LTL 𝑘 -1 ⊊ LTL 𝑘 (except the separating language is STAIR 2 𝑘 instead of STAIR 𝑘 ). Thus, we have

and in particular, MUHAT ( ▶ 𝐹, ◀ 𝑃 ) 𝑘 ⊊ MUHAT ( ▶ 𝐹, ◀ 𝑃 ) 𝑘 + 1 .

Finally, we allow all six types of attention. These will be notated with MUHAT 𝑘 and B-RASP 𝑘 .

□

Proposition 29. With access to future-, past-, and no masking and both leftmost-hard and rightmosthard attention, multi-head masked hard-attention transformers of depth ( 2 𝑘 + 1 ) are strictly more expressive than multi-head masked hard-attention transformers of depth 𝑘 .

Proof. As above, MUHAT 𝑘 = B-RASP 𝑘 . However, in the proof of Theorem 2, the simulations of leftmost future-masked and rightmost past-masked attention require two levels of nesting of since and until , so it only shows that B-RASP 𝑘 ⊆ LTL 2 𝑘 . As in the previous proof, LTL 2 𝑘 ⊊ LTL 2 𝑘 + 1 (Etessami and Wilke, 2000). Finally, by Theorem 1, we again have LTL 2 𝑘 + 1 ⊆ B-RASP 2 𝑘 + 1 . Using all these observations, we conclude that:

Thus MUHAT 𝑘 ⊊ MUHAT 2 𝑘 + 1 .

□

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and introduction both summarize the paper.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Sections 2.2 and 6 address the main limitation of the paper, which is that the kind of transformers studied (unique-hard attention) differ from actual transformers.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate 'Limitations' section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Theorems 1 to 7 and 10 and Corollary 8 all have proofs in the appendix and brief proof ideas in the main text, when appropriate. Corollary 9 is very straightforward and in our opinion does not need an explicit proof.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [NA]

Justification: This paper does not include any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The only code is a public github page which contains a B-RASP simulator implemented in HTML and Javascript. Using it, it is straightforward to recreate and run the example programs in the paper.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [NA]

Justification: This paper does not include any experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [NA]

Justification: This paper does not include any experimental results.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer 'Yes' if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [NA]

Justification: This paper does not include any experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We have reviewed the NeurIPS Code of Ethics and believe that the research in this paper conforms to it.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper is purely theoretical, and we do not foresee any direct societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper is purely theoretical and poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification: This paper does not use existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The only new asset is a webpage hosted in a publicly accessible github repository.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: This paper does not involve crowdsourcing or research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents results about several forms of masked unique hard-attention transformers by classifying the class of languages recognized by these models using formal languages. Roughly put, a masked unique hard-attention transformer is an encoder that uses unique hardmax as its attention mechanism. In cases of ambiguity, either the leftmost or the rightmost position is attended to. Additionally, the attention heads are masked, meaning that a position only attends to (strictly) preceding ones (future masking) or (strictly) succeeding ones (past masking).\n\nIn summary, the presented results are:\n- Future-masked rightmost-hard attention transformers without positional embedding recognize exactly the star-free languages (Theorem 5).\n- Non-strict masked hard-attention transformers without positional embedding recognize exactly the stutter-invariant star-free languages (Theorem 6).\n\nTo establish these results, the paper introduces an auxiliary language called B-RASP to analyze masked hard-attention architectures. The paper shows that B-RASP exactly captures the considered transformer models (Theorems 3 and 4) and that it is as expressive as strict or non-strict Linear Temporal Logic (LTL) (Theorems 1 and 2). Then, the general proof idea of Theorems 5 and 6 is to translate results from strict or non-strict LTL via B-RASP to the respective transformer model. Additionally, the authors also establish that masked-hard attention transformer recognize the same languages as LTL with monadic predicates.\n\nSTRENGTHS:\nThe results, mainly Theorem 5 and 6, in this paper follow an intriguing line of research trying to understand the expressive capabilities of transformer models from a formal languages perspective and, therefore, this research is well placed.  \n\nIn general, the technical proofs of the appendix are convincing and sufficiently argued and, thus, allow to verify them.\n\nWEAKNESSES:\nThe contribution of this work is intriguing, though its significance may not be immediately apparent. The primary contributions are Theorems 5 and 6, and the transformer architectures considered are quite specific, possibly tailored for these particular results. There is some uncertainty about their broader interest. As the authors note, masked attention is almost exclusively used in the decoder parts of transformers. However, in this work, it is extensively used in encoder-only transformers.\n\nThe presentation of the paper is mixed. The first 5 pages are primarily devoted to preliminaries and the introduction of B-RASP, which, while interesting, serves mainly as a tool. This allocation of space would be justified if B-RASP were particularly complex, but it is relatively straightforward. Consequently, almost all theorems in the main paper lack proof sketches or intuitive explanations. Although the appendix adequately supports the theorems’ statements, the main paper itself does not. Additionally, the paper features a very brief introduction and lacks sections like ‘outlook’, ‘open questions’, ‘limitations’, or an equivalent closing and summarizing section. This omission makes it challenging for readers to place the results in the broader context of research.\n\nThe clarity of the paper is also mixed. First, the paper uses a form of LTL with a strict until operator and no next operator. Most readers are likely more familiar with non-strict until and the inclusion of the next operator. While this is a minor issue, a single sentence clarifying this would assist non-expert readers. There are more significant issues, which I address in my questions below. For example, in the brief sketch of Theorem 4, it is stated that the transformer “… can be represented using $\\mathcal{O}(1)$ bits.” It is unclear why this result focuses on representation sizes. Another example is Theorem 5, which states that future-only masking transformers recognize exactly the star-free languages. The proof relies on a translation from B-RASP to the transformer, and in this translation, it appears they use a self-attention mechanism requiring mixed masking. This may be a case of unclear explanation, but the current version of the paper makes it difficult to understand. If not clarified, this result could be incorrect.\n\nEdit: My concern about the correctness were cleared in the rebuttal. I changed my rating accordingly.\n\nQUESTIONS:\n- Can you explain in more detail why masked attention is of interest in the encoder parts of transformers?\n- In the proof of Theorem 3, Lemma 20 establishes a translation from B-RASP to masked hard-attention transformers. It appears that you use non-strict masking in the first layer to achieve self-attention. Am I correct? If so, how can you prove Theorem 5 based on this, since it concerns future-only masked hard-attention transformers?\n- In the proof sketch of Theorem 4, can you elaborate on why you focus on precision?\n- Can you explain in more detail the idea presented in line 264 regarding how a specific symbol BOS helps?\n- Can you explain in more detail how attention masking helps in recognizing all LTL[Mon] languages, as discussed in the paragraph starting on line 308?\n\nLIMITATIONS:\nThe authors clearly describe their considered transformer model and specify where it differs from those commonly used in practice. However, these statements are somewhat scattered throughout the paper. A dedicated subsection summarizing all relevant limitations would be very helpful.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents new theoretical results related to the expressive power of Transformers. The authors focus on Transformers with \"hard\" attention (a simplifying assumption) and strict future masking (i.e. attention can only attend to positions to the left). The paper develops an equivalence between such Transformers and B-RASP, and between B-RASP and Linear Temporal Logic (LTL), where B-RASP is a binary-valued version of RASP, a programming language for Transformers proposed by prior work. Thereby, the authors establish an equivalence between the proposed class of Transformers and LTL.\n\nSTRENGTHS:\n* The paper adds to our understanding of the theoretical expressiveness of Transformers.\n* The use of B-RASP as an intermediate representation to establish the equivalence between Transformers and LTL was an interesting approach, and perhaps could inspire future work towards establishing the expressiveness of various Transformer variants.\n\nWEAKNESSES:\n* While the authors justify their choice of focusing on \"hard attention\", this puts some limits on the applicability of their results to real Transformers. However, this is also a common assumption made by prior theoretical work related to the expressiveness of Transformers.\n* Prior work has already established that hard attention Transformers can express LTL.\n* The main results that extend beyond prior work to establish the *equivalence* with LTL focuses on architectures with strict future masking and without positional encodings. Both choices are a very uncommon configuration for Transformers.\n\nQUESTIONS:\nQuestions:\n\n* Is there an example of a commonly used positional encoding scheme with \"infinite image\"? (section 4.3)\n* It seems commonly used schemes for relative position encodings (e.g. https://arxiv.org/abs/1803.02155, https://arxiv.org/abs/1910.10683) could presumably enable expressing attention operations with strict future masking, without requiring this to be an implicit part of the architecture. Is this true? (section 4.3)\n\nSuggestions:\n\n* Nit - perhaps briefly defining terms such as complexity classes AC0 and P on their first mention could help make paper more accessible. \n* The main results focus on architectures with strict future masking and without positional encodings. Both choices are a very uncommon configuration for Transformers. However, per the question above, strict future masking is not necessary if using common positional encoding schemes. This is discussed to some degree in section 5.3, but this result could have potentially been mentioned earlier to justify the otherwise somewhat odd focus on strict future masking without positional encodings.\n* Section 5.2 mentions that \"Soft-attention can measure what fraction of symbols are l\". Potentially relevant: the RASP paper also discusses an algorithm for this, termed `selector_width`, and how this can be implemented with a start symbol or without (relying on positional encodings).\n* Section 5.3 mentions that positional information only comes from attention masking. Perhaps relevant: https://arxiv.org/abs/2305.19466 shows that absolute and relative positions can be recovered from only future masking (although I believe their proof relies on soft attention).\n\nLIMITATIONS:\nSome limitations are mentioned throughout the paper (e.g. section 2.2), but it might be helpful to have an explicit limitations section.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the expressive power of transformer encoders in terms of their ability to recognize regular languages. The main result in the paper establishes that if such encoders are equipped with hard attention, future masking is permitted, and positional encodings are disallowed, then the languages accepted by this model are precisely the ones that can be defined in linear temporal logic (LTL), which in turn coincides with the star-free languages. Several other results are presented in the paper, but in my view, these are corollaries of the main result and known properties of the relationship between LTL and certain classes of regular languages.\n\nSTRENGTHS:\nThe ability of Transformers to recognize languages, under different assumptions (e.g., attention mechanism, presence of decoders, type of positional encoding), is by now an active area of research. But this is one of the few results in the area that presents a precise characterization of an important Transformer model. While the proofs are more or less straightforward, I find the results beautiful and relevant, so I support the acceptance of this paper.\n\nWEAKNESSES:\nI find the detour through B-RASP unnecessary. First of all, proofs are not presented in the main body, and hence for someone without access to the appendix, there is no support for the claim that this detour is key. What is it that makes it \"key\"? Second, I think that a direct translation from LTL to Transformers, and back, is possible. The upper bound only depends on the finiteness of the vectors considered by the Transformers. In turn, for the lower bound, you can use similar techniques to those in Barceló et al, i.e., induction on the structure of formulas (this time by using positional masking as opposed to positional encodings).\n\nQUESTIONS:\nPlease explain why the detour through B-RASP is key to your proof and why you have decided to leave this connection in the body of the paper instead of keeping it in the appendix as part of the main proof. What kind of conceptual benefit do you feel that this connection brings to the paper?\n\nLIMITATIONS:\nLimitations are correctly addressed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors connect masked hard-attention transformers (with single-head attention) with LTL via a Boolean version of RASP, namely B-RASP. Due to established results on LTL, they show that this type of transformer recognizes exactly the star-free languages.\n\nWhile unique hard attention transformers have been studied before, the authors also use strict future (and past) masking which essentially means that each position can't attend to itself but only to positions to the left (or right), respectively. RASP is a programming language by Weiss et al that was invented to have a language very close to how transformers work. This paper uses a Boolean version of it and shows equivalence with hard-attention transformer encoders with strict future masking. Linear Temporal Logic (LTL) is a well researched logic which recently has been linked to unique hard attention transformer encoders and shown equivalent in this paper. Some known results lead to further connections to and characterisations of transformers, including some extensions such as the use of position embeddings. The authors also show that increasing the number of attention layers always increases expressive power in multi-head masked hard-attention transformers.\n\nSTRENGTHS:\nThe paper is very well-written and I enjoyed the examples for B-RASP which made it very easy to understand both the programming language and the equivalence proofs.\nThe first 4 sections lead through the results nicely. The last section is a bit more over the place but gives the impression that the authors looked into multiple directions and found numerous possible extensions that follow with little adjustments from their main result.\nThe discussion of previous work in between the results gives a good overview about similar lines of work and was very fitting.\n\nWEAKNESSES:\nFirst of all, I was missing a discussion about implications of the results and possible future work.\nUnique hard attention transformer encoders have been known to be in AC⁰ and have been linked to LTL[Mon] before. In my eyes, the main contribution of this paper is therefore the result that this characterisation is exact and the results on strict vs. non-strict masking. While these are useful realizations, they might not make the biggest impact on practical applications. Even more so, because hard attentions is hardly used in practice.\n\nThere is a typo in line 31: exp(r)essivity\n\nQUESTIONS:\nNone.\n\nLIMITATIONS:\nThe authors have thoroughly discussed the limitations of their findings one by one.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}