Efficient Turing Machine Simulation with Transformers
Authors: Qian Li
Yuyi Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Transformers expressiveness
Turing completeness
sparse attention
nearly optimal simulation
reasoning efficiency
Primary area: learning theory
TL;DR: Constant bit-size Transformers with log-space attention can simulate multi-tape TMs with an optimal context window length and nearly optimal CoT length.

Abstract: Constant bit-size Transformers are known to be Turing complete, but existing constructions require $\Omega(s(n))$ chain-of-thought (CoT) steps per simulated Turing machine (TM) step, leading to impractical reasoning lengths. In this paper, we significantly reduce this efficiency gap by proving that any $(t(n),s(n))$-bounded multi-tape TM can be simulated by a constant bit-size Transformer with an optimal $O(s(n))$-long context window and only $O(s(n)^c)$ CoT steps per TM step, where $c>0$ can be made arbitrarily small by letting the Transformers' head-layer product sufficiently large. In addition, our construction shows that sparse attention with fixed geometric offsets suffices for efficient universal computation. Our proof leverages multi-queue TMs as a bridge. The main technical novelty is a more efficient simulation of multi-tape TMs by synchronous multi-queue TMs, improving both time and space complexity under stricter model assumptions.

Published as a conference paper at ICLR 2026
EFFICIENT TURING MACHINE TRANSFORMERS
SIMULATION WITH
Qian Li
Shenzhen International Center For Industrial And Applied Mathematics Shenzhen Research Institute of Big Data
Shenzhen, China
Email: liqian. ict www.mmmmmma.......m com
Yuyi Wang
CRRC Zhuzhou Institute & Tengen Intelligence Institute
Zhuzhou, China
Email: 0uyiwang9200mmm com
ABSTRACT
Constant bit-size Transformers are known be Turing complete, but existing constructions require (()(n)) chain-of-thought (CoT) steps per simulated Tur- ing machine (TM) step, leading to impractical reasoning lengths. In this paper, we significantly reduce this efficiency gap by proving that any (f(z), ((n)))))sn))ooooo -bounddddd multi-tape TM can be simulated by constant bit-size Transformer with an op- timal ((s(n))-long context window and only (n))))) CoT steps per TM step. where > 0 can be made arbitrarily small by letting the Transformers' head-layer product sufficiently large. In addition, our construction shows that sparse atten- tion with fixed geometric offsets suffices for efficient universal computation. Our proof leverages multi-queue TMs bridge. The main technical novelty is a more efficient simulation multi-tape TMs by synchronous multi-queue TMs. improving both time and space complexity under stricter model assumptions.
INTRODUCTION
Transformer- based large language models (LLMs) equipped with cfain-of-thought (CoT) reasoning (Gemini al., 2023; Anthropic, 2024; Guo al., 2025; OpenAI, 2025) have shown remarkable performance in various challenging reasoning tasks, including solving mathematical problems (Lu- ong & Lockhart, 2025) and generating computer programs (Rein et al., 2023; Alberto Manzi, 2025). Beyond the empirical successes, there has been growing interest in understanding the mechanisms underlying the reasoning abilitics of Transformers. recent line of work (Pérez et al., 2021; Bhat- tamishra al., 2020; Merrill & Sabharwal, 2024; Li et al., 2024; Qiu et al., 2024; Yang et al., 2025a; Bavandpour et al., 2025; Li & Wang, 2025) has established the Turing completeness of Transform- ers. In particular, it turns out (Li & Wang, 2025) that even constant bit-size Transformers, where the number of parameters and precision are both constant independent of the input length, are Turing complete, provided sufficiently long context windows and chain-of-thought (CoT) reasoning steps are allowed. These results demonstrate that Transformers' reasoning ability is universal in principle.
However, critical gaps remain in our theoretical understanding: the efficiency of such simulations is left open. Table summarizes the efficiency of existing Turing completeness constructions. In particular, the constant bit-size construction of Li & Wang (2025) achieves an optimal context win- dow of length ((s(n)), which reflects the amount of memory required for reasoning, but incurs cost of (x) CoT steps to simulate one Turing machine (TM) step. Here s(n) denotes the space bound of the simulated computation. For many problems, this f(nt)faccor slowdown is prohibitively large, causing the total CoT length to far exceed the number of reasoning steps observed prac- tice. Therefore, of practical significance to investigate whether the slowdown can be avoided without compromising optimality in other aspects.
pointed out by Li & Wang (2025), it is not only of theoretical interest but also

---

Published as a conference paper at ICLR 2026
Table 1: Comparing Transformers' complexity to simulate ((()), ((om)))-uuuudddd Turing machines. 'Dim'' denotes embedding dimension, 'Window' means effective window size, and 'Total CoT denotes the total CoT length. This paper focuses on the nontrivial case where t(n), 2 n.
Source
Precision
Dim.
Window
Total CoT
Pérez et al. (2021)
(log t(n))
0(1)
t(n)
O(t(n)
Bhattamishra et al. (2020)
unbounded
0(1)
+ t(n)
O(t(n)
Merrill & Sabharwal (2024)
Olog ((n))
0(1)
+ t(m)
O(t(n)
Li et al. (2024)*
((1)
O f(m))
(t((n) log f(n))
O(t(n) log t(n))
Qiu et al. (2024)
O (log t(n))
0(1)
((t(n) logt(n))
((t(n) log t(n))
Yang et al. (2025a)t
O (log t(n))
0(1)
s(n)
O(t(n)
Bavandpour al. (2025)
(log f(m))
0(1)
+ t(n)
O(t(n))
Li & Wang (2025)
O(1)
0(1)
s(n)
O(t(n) s(n))
This work
O(1)
0(1)
s(n)
((((n) 8 (n))
Any multi-tape TM running in time U(n) be simulated by a Boolean circuit of size ((( log (m))
( (rrra & Barak, 2009).
The construction incorporates reduction mechanism recursively erase intermediate CoT steps.
By a standard technique, a I-time s-space multi-tape TM can converted into Post machine running in time Ol s) and space (s)).
The exponent >> 0 be made arbitrarily small by letting the precision and emhedding size sufficiently
large constants.
1.1 OUR CONTRIBUTION
We make significant progress on the open efficiency problem by reducing the per-step slowdown from s(n) to snn),, where the exponent e >> can be made arbitrarily small. Formally, letting the head-layer product denote the product of the number of attention heads and the number of layers, our main theorem is stated as follows:
Any (t(n), s(n)) time-space bounded k-tape Turing Machine can be simulated by constant bit-size Transformer with head-layer product K and context window length s(n) + 1, that takes (( 6k/K) (n))k/ CoT steps for each simulated TM step, leading to a total CoT length of O(((n). s(n) 6k/
Theorem 1.
Moreover, our construction exhibits sparse geometric-offse attention property: every query at- tends to only few tokens located at fixed relative positions, with offsets chosen in geometric progression'. Interestingly, the geometric-offee behavior observed: empirical analyses report that attention is typically sparse and spatially clustered, with many heads focusing on local neighborhoods and a minority capturing structured long-range connections, e.g. (Xiao al., 2024; Ribar al., 2024; Jiang et al., 2024; OpenAI, 2025). As sequence, each token can be produced in O(1) time, and simulating Turing machines incurs only a per-step slowdown of (s())), in contrast to the t(n) overhead full attention. This suggests that the wide concern that the quadratic time complexity of full attention constitutes fundamental throughput bottleneck for long-contes Transformers (e.g., (Khan al., 2022: Sarikaya, 2025))) is not necessarily a principled expressiveness limitation of Transformer architectures.
attention pattern is in the same spirit as attention
Our results highlight geometric-offse attention as a promising architectural direction and principled design choice for efficient reasoning: a query attends to only the tokens 1, 2, 4, ......27 2 ... steps earlier 2. Such exponentially spaced connections avoids the quadratic overhead of full attention, while still being sufficient for efficient universal computation as shown by our theory. Notably, sim- ilar exponentially spaced or logarithmically sparse patterns have already been explored in practice, such as in LogSparse Transformers (Li et al., 2019) and PowerAttention (Chen et al., 2025). Our
SSpecifically, in Theorem 1, the offsets are [s(n) 1/"1,, [s(n) 1/.../ ( with K' = K/ K/(6)). II the regime of practical interest, the head-layer product К is typically large (e.g. K on the order of -10 in modern LLMs). Moreover, is natural to focus on & 2, since 2-tape TM can efficiently simulate multi-tape TMs with only a logarithmic slowdown. So, if we take K = 6 x 10 and l = 2, then the common ratio would be [s(n)ak/K
6k/ K
= 2 for any s(n) 2500
2

---

Published as a conference paper at ICLR 2026
results thus provide theoretical evidence that these sparse designs can retain the computational universality while improving efficiency.
Our proof strategy is inspired by the approach of Li & Wang (2025), which established bridge between TMs and Transformers via one-queue Turing machines (a.k.a. Post machines). We gen- eralize this idea by introducing multi-queue Turing machines the bridge. Concretely, Step 2 of our proof can be seen as a generalization of that of Li & Wang (2025) to multi-queue machines; the main technical novelty of this work lies in Step 1.
Step 1: From multi-tape TMs to multi-queue TMs. Previous works have studied the relation- ship between multi-tape and multi-queue machines; see (Petersen, 2013) for an overview of existing results. In particular, Hühne (1993) showed that any (t(n), -bou)ddddndedd k-tape TM be sim- ulated by k queuue TM that runs in (O(t(n)(+1/R), ((t(m)))))1
time and space. However, their multi-queue model is more permissive: queues are allowed to remain idle in step (i.c., neither pop nor push symbols). By contrast, we consider the more restrictive synchronous model, where ev- ery queue pops exactly one symbol and also appends exactly one symbol at each step. Despite this restriction, we show that any (t(n), ((nz))-bounde k-tape TM can be simulated by synchronous
(6kkk)-queue TM that runs in (O(t(n) s(n) (s(())) time and space (Theorem 2). Our improvement over (Hühne, 1993) is threefold: (i) extending the result to the more restrictive syn- chronous model, (ii) reducing the space complexity from t(n)1+1/k' to s(n), and (iii) reducing the time slowdown from t(z)1/k' to s(n)2/K'
Step 2: From qunlti-qeueeeeeeeee TMs to Transformers. Once Step 1 is established, we adapt the simu- lation technique of Li & Wang (2025) to synchronous multi-queue TMs. Specifically, we prove that K K-queue TM can be simulated by constant bit-size Trans-
any synchronous (t(n), ((n))--oundee former with head-layer product K, context window (s())), and CoT length ((((n) (Theorem 3). Now, our main theorem, namely Theorem follows immediately from these two steps.
1.2 OTHER RELATED WORKS
Formal-language expressivity. A substantial body of work (Hahn, 2020; Yao al., 2021; Hao et al., 2022; Feng et al., 2023; Chiang et al., 2023; Merrill & Sabharwal, 2023; Yang et al., 2024; Barcelo et al., 2024; Merrill & Sabharwal, 2025; Chen et al., 2024a) studies Transformers through the lens of formal-langus architectural and ppsitional-encodin assumptions, c.g., absolute vs. relative PE, with vs. without COT, restricted attention, precision assumptions, and depth. The survey by Strobl al. (2024) pro- vides a comprehensive overview of upper and lower bounds across variants, clarifying how design choices affect expressivity.
recognition, asking which language classes are captured under different
Reasoning efficiency. Current reasoning models tends to overthink, generating excessively long CoT's that causes significant computational redundancy (Chen et al., 2024h; Sui et al., 2025). More- over, has been shown that longer CoTs do not necessarily improve accuracy and can even hurt (Wu et al., 2025; Yang et al., 2025b; Jin et al., 2024). To mitigate overthinking, proposed methods include RL with length penalties (Luo al., 2025; Aggarwal Welleck, 2025), SFT on variable- length traces (Ma et al., 2025; Xia et al., 2025), and latent reasoning that avoids generating explicit CoTs (Hao et al., 2024; Su al., 2025). Beyond shortening CoTs, another line of work seeks to maintain strong reasoning with smaller models by applying compression techniques (e.g., distilla- tion, quantization, and pruning) and by directly training smaller models with RI. (Li et al., 2023; 2025; Zhu et al., 2024). For comprehensive surveys of efficient reasoning methods, see (Feng al., 2025) and (Sui et al., 2025).
2 PRELIMINARIES
Throughout this paper, adopt the following standard notation. The sets of real and natural num- bers are denoted by RR and N respectively, and [n] := 1, ..2..... n for n E N. Vectors and sequences written in bold lowercase (e.g., 23), and matrices in bold uppercase (e.g., A). For vector or
3

---

Published as a conference paper at ICLR 2026
sequence I; let 2; denote its i-th element, and let 3y denote (zy, 141, xx) for I i. For matrix A, let A,,y denote the j-th element in the i-th row. We write O, for the n-dimensional zero vector.
2.1 MULTII-TAE TURING MACHINES
A k-tape Turing machine (TM) has tapes, one of which initially contains the input and is called the imput tape. Each tape is infinite to the right and bounded on the left, and equipped with tape head. It can be defined as a tuple (Σ, (2 where
Σ is a finite tape alphabet, including O, 1, and a blank symbol
Q is a finite set of states, including start state Istart and halting state Thalt-
6 x E Q x - (fLeft, Stay, Right))* k is transition function.
Initially, the input tape contains finite {0, 1)-string x the imput, and the other k 1 tapes all empty. The computation repeats the following until the TM enters ghatt: if the machine is in state E Q and reading symbols (oo. ... .k-1) from the tapes, and if 8(g, To. --1)) (1', 20, of Z1,"' " "011, 2k11 where 2; E (Left, Stay, Right), then the TM changes its state to g', writes to the working tapes, and moves the heads according 2.
Without loss of generality, and for technical convenience, assume that at the start of the compu- tation the Turing machine first performs a sean over the input tape.
The space complexity of k-tape Turing machine is the function s(n) that, for each input length n, gives the maximum number of distinct cells visited on the tapes during the computation on any input of length
2.2 MULTI-QUEUE TURING MACHINES
In contrast to multi-tape TMs that operate on tapes, multi-queue TMs operate on queues. Previous works multi-queue TMs (e.g. (Hühne, 1993)) studied the model in which each queue remain idle in a step (i.c., neither pop nor push symbols), or equivalently (up to a constant factor slowdown), only queue can pop append element at each step. In this paper, for technical reasons, consider a more restrictive variant, called the synchronous multi-queue TM, where each queue pops exactly one element and also append exactly one element at each step. Note that synchronous multi-queue TM can be simulated by the traditional model with only constant factor slowdown, whereas the reverse direction is unlikely hold.
Formally, a k-queue synchronous TM has k queues, one of which initially contains the input and is called the input queue. can be defined as a tuple (2, Q, 8) where
Y is a finite tape alphabet, including 1. and a blank symbol
Q is a finite set of states, including start state gatart and a halting state qhalt-
6: Q x n + Q x * is transition function.
Initially, the input queue contains the input string, and the other k - 1 queues contains only blank symbols. The computation repeats the following until the TM enters Thatt: if the machine is in some state q E Q and reading symbols (To. ... ok-1) popped from the queues, and if ( .o. . 1)1) then the TM changes state q' and appends o' to
ö(g, "o..oo.....kk..... o the queues.
The space complexity of a q-queee (synchronous) TM is the function s(n) that, for each input length n, gives the maximum total length of the & queues during the computation on any input of length n.
2.3 TRANSFORMER ARCHITECTURE
Let be the vocabulary. A decoder-only Transformer TFa is parameterized function Ul>1 Di V composed of embedding layer, multiple decoder layers, and output layer.
4

---

Published as a conference paper at ICLR 2026
Token embedding layer For each previous token v V (with I i), map it to d- dimensional embedding vector ky := emb(u;). We do not add absolute positional embeddings to hg. Instead, introduce relative positional information only inside the self-attention scores, fol- lowing the formulation of Shaw et al. (2018). Here, we call d the model dimension or embedding dimension.
Decoder Layers The h is then processed by L. decoder layers. Each decoder layer dec, consists of self-attention sub-layer followed feed-forward network sub-layer, with residual connections and layer normalization applied around each sub-layer. For simplicity and following (Li et al., 2024), we omit layer normalization; see (Li al., 2024) for how it can be incorporated without affecting our results. In addition, following common practice', we use hardmax as a proxy for softmax.
Self-attention sublayer: For each head 0,
I. 1, compute the attention score as* $ hardmax (hi K + pos(i - 1), h Q ... (h! K + pos(i - i). h Q(),
where pos(i -) E Rd/" The head output is then
a.
Σ K.is u (ht), where (n)) h V j=1
Concatenating all heads yields of = ((a{,)) ; .... )²) E Rd. The residual update is
h f+0.5 == W af + b' + h/
Here Q K% V E Rdxd/", W E Bdxd and b' Id are learnable parameters.
Feed-forward sub-layer: Apply fully-connected ReLU neural network FF*:
h ++
FF* FF' (h.5 h +0055
Output layer The final representations hf is mapped to the vocabulary by:
U+1 out(h!") arg max Wrutt h + bout), where Wrout RRVV xd and hour E RP
A Transformer is said to have a context window of length if the query can attend only to the most recent s tokens. Its bit-size is defined as p x |0), where p denotes the precision, i.e., all learnable parameters are stored with bits scalar, and (0) the number of learnable parameters. The head- layer product defined as the product of the number of heads H and the number of layers L.
STEP I: EFFICIENT SIMULATION OF MULTI-TAPE TMS BY MULTI-QUEUE TMS
This section proves the following theorem.
Theorem 2. Any (t(n),,(r) time-space bounded k-tape Turing Machine M can be simulated by synchronous 6kk' lkkeee Turing Machine M' which is (O(Un) s(n)2/K) O, s(n))) time-space bounded.
The basic idea is to simulate each tape using two stacks, and then simulate single stack with k' levels of queues whose sizes grow geometrically. Concretely, level consists of a content queue of size 1/ (realized as two half quenes of size !/!//' each), and an auxiliary buffer queue of size 2 2[s//K' The content queue maintains the invariant that: block of real symbols followed by a block of dummy symbols. The top of the simulated stack always sits at the boundary of real and dummy cells in level 1, while deeper stack elements are stored in higher levels.
As we will see, in our construction, the input to hardmax is always one-hot vector, and so is the output. As consequence, Transformer construction in Theorem 3 works with all variants of hardmax
'Our conclusions (specifically Theorem and Theorem 1) continue to hold under other relative positional encoding formations, e.g. using (h. Kf, Af Q+ + pos(i - jj instead of (hς K + pos(i - j). he Q.)) In fact, our construction in Theorem 3 uses relative positional encoding to realize attention to tokens located fixed relative positions.
5

---

Published as a conference paper at ICLR 2026
0ie
Gi (buffork: Used only in Stack Push
Read
add 1
Lv
TT1
11111
Queue
by :....
b
bn b2
bg
Gia (builer)
1
Lv
> d37 1 Og 11
g/
Tape
b
b3
1
Ge (huffer)
shifting cell right eyclically each step
Lv k'
11
11
Figure 1: Left: A queue can be viewed as tape with the head shifting one cell to the right cyclically each step. Right: Illustration of the queues simulate stack.
Stack Push. To push new symbol onto the stack, we insert it into the first dummy cell of level 1. If level I becomes full, half of its content is moved to level 2; if level 2 is full, half of its content is moved to level 3, and so on.
Stack Pop. To pop from the stack, remove the last real symbol from level 1. If level becomes empty, refill the left half queue by pulling elements from level 2, again recursively needed.
The crucial property is that higher-level queues much larger but accessed far less frequently, so expensive transfers rare. A basic push pop costs D(s1/k). A transfer between levels and costs (ss/k') but occurs only once every ((s(t-1)/kk) simulated stack steps, giving an amortized overhead of ((s1/k) per stack step. Thus simulating t(n) steps taks ((((n).s) s/1/k) time while using overall queue space O(s). Since each tape equals two stacks and each stack requires 3k' queues, simulating l tapes needs 6kk!' queues in total.
Proof. Since each tape can be simulated by two stacks, suffices describe how simulate single stack PD of the TM M using 3k' queues of M':
(K,, VI Qi,B), (L',,, C' Q.B) (O'. Ok RR Qw.B).
Here, the subscripts "L", "R", and "B" stand for "Left Half", "Right Half", and "Buffer", whose precise roles will be clarified below. We assume that PD contains a distinguished bottom element that is never popped. During the simulation, the length of Qi and Q¹, is fixed at ([51/k)) 1 and IQk, Bl fixed at 2( (/T(/) So the total queue length is
k'
Σ
4x
would be helpful to imagine queue Qh,, a tape Ty of the same length, with the tape head shifting exactly one cell the right cyclically each step of M'.
Let E denote the alphabet of PD. Then we define E' == {a, ā, E the alphabet of each T.,- Here, : and " mark the leftmost and rightmost cells of T respectively, and denotes dummy placeholder. A tape TV, is said to be full if contains only real symbols (i.e., non-dummy symbols), and empty if it contains only dummies.
as
Te = Ty T. be the concatenation of T' and TV Intuitively, TV plays the role of the content queue of level i the proof intuition: it stores the actual stack symbols, and its content is arranged left to right) during the simulation. We call Tę balanced if it contains equal numbers of non-dummies and dummies. Equivalently, this means Ty is full and Ty is empty.
block of real symbols followed by block of dummies (from

---

Published as a conference paper at ICLR 2026
Algorithm Simulation of pushing into PD
1: Rotate the tape heads until reach the first dummy element of Tj;
2: replace 1 with in the current cell of ;;;
3: if the current cell of Ty has an accent then
4: run PUSH(2);
5: end if
6: procedure PUSH())
7: Rotate the tape heads until reach the first dummy element of T7:
8: while the current cell of T- R is non-dummy do
9:
write this non-dummy into the current cell of Ty; write into the current cell of T 1
11:
rotate the tape heads one cell to the right eyclically. end while
12:
13: T" is full then
14:
run PUSH(i+1);
15:
end if
16:
end procedure
Algorithm 2 Simulation of poping element from PD
1: Rotate the tape heads until reach the last non-dummy element of T7, denoted as a; the current cell of T/; 3: if the current cell of T has an accent then
2: replace a with
4: run procedure POP(2);
5: end if
6: return a;
7: procedure POP(i)
8:
if TI is not empty then
9:
Rotate the tape heads until reach the leftmost element of TV: while the current cell of T is non-dummy do the current cell of T is non-dummy then write this non-dummy of T. into the current cell of T7 end
11:
12:
13:
write this non-dummy of T into the current cell of T- rotate the tape heads one cell to the right cyclically;
15:
end while end if
17:
if TI empty then
19:
run POP(i+1);
end
21:
end procedure
During the simulation, we will maintain the following property: The top stack elements of PD are stored in 7í and deeper elements are stored in Ty T Ty such that the concatenation TV o... PD.
By the synchronous condition, each step of M'. every queue Qh,- pops exactly one symbol and appends exactly symbol. Accordingly, the head of each tape TV shifts exactly one cell to the right cyclically.
At the beginning of the simulation, PD contains only the distinguished bottom element; all TV, are empty except Ti stores the bottom element in its lefumost cell; all heads start at the leftmost tape cell. Now, show how to simulate the stack push and pop operations.

---

Published as a conference paper at ICLR 2026
Stack Push To push symbol of T is reached and replaces with a.
stack PD, M'' rotates the tape heads until the first dummy cell
If If becomes full, then the first /2 non-dummies of are pushed into the first [T]] /2 dummy cells of T22 (i.c., PUSH(2) in Algorithm 1). Specifically, in PUSH(2), the entire content of Ty, is copied into these cells of Ty. Ti, cleaned, and then the remaining contents of TÝ is shifted left by // cells (or equivalently, just swap the roles of Ty, and T. This preserves the arrangement that dummies to the left and dummies to the right. If If is full, then execute PUSH(3), and so on.
Fact Right after PUSH(i), all Ti. ,
.T are balanced.
Stack Pop To pop the top element of PD. M' rotates the head of Ti to the last non-dummy element", outputs it, and replaces it with L.
If TÝ becomes empty, then the last /2il/2 non-dummy elements of T2 are transferred into the first /T///2 cells of ( (i.e.. POP(2) in Algorithm 2). Specifically, in POP(2), the last [TY) 22 non-dumm, elements of 732 are copied into Ti,, after which those cells of 7 are cleaned. The main challenge to implement POP(2) is that we cannot directly identify the last /2 non-dummies overcome this difficulty,
Ty To employ the buffer queue 2,8 to temporarily store a block of elements scan T. from left right, and move each
of T3 during the scan. The intuition is as follows: encountered symbol to Ti whenever a cell of T' is about to be overwritten, its previous content is diverted to T2 In this way, we maintain the invariant that T Ti, o T3 always equals the original content of Tz. In particular, when we reach the first dummy of Ty we have T2,8' o Ti equals the original content of Tyy. Formally,
First, rotate the head of Tý the leftmost cell; (one can see that the TH. and T2 heads also locates at the leftmost cell this time, since i divides [T and | = (T2].)
As the head of TV goes from left right until reaches the first 1.
the currect cell of Ti,, store a non-dummy, then move this element T 2. The current element of Ty is copied the currect cell of Ti, and then cleaned.
Finally, the content in T 2, is copied to T% and then cleaned.
If TV becomes empty, the procedure recurs to POP(3) and so on.
Fact 2. Right after POP(i), all of "T are balanced.
What remains is analyze the correctness and efficiency of this simulation.
Claim 1. During the simulation, we always have the following properties.
(a) The numbers of dummies and non-dummies of T are both multiples of (T/ for any i
2. ,kk.
(b) When the head of TV is the first dummy cell, the head of TŞ- I. the leftmost cell. Similarly, when the head of Ty is on the last non-dummy cell, the head of Ti-1, is the rightmost cell.
(c) At the beginning of the stack push simulation, each T contains at least (Ti dummies. Con- sequently, combining with (a) and (b), implies that PUSH(i) doable whenever it called.
(d) If is empty, then all higher levels "". T
are empty as well.
'A non-d con-dummy cell is the last one iff carries the rightmost mark its next cell dummy. avoid explictly pre-reading the next cell, we apply a one-step delayed append trick: initially, pop the leftmost queue symbol and store it in the state controller Q. without appending. Thereafter, at each step (i) pop the next leftmost symbol, and (ii) append symbol determined by the two most recently popped symbols. In this way, the appended symbol depend on the leftmost two queue symhols, without explicit pre-reading.
8

---

Published as a conference paper at ICLR 2026
(e) The concatenation of non-dummy contents of ".... Ti (in this order) equals exactly the tent of PD (we start with the bottom element).
Consequently, .i.... .Tk faithfully simulate the stack PD. Precisely, whenever PD pops the top element, M' can obtain as well.
Proof. (a). Intially, Tς have 0 non-dummies and 2(Ts)/81)' 2T - dummies. The numbers changes only if PUSH(i) POP()) is executed. Through PUSH(),, ITi-1.z/ dummies changed non-dummies, or T becomes balanced. Through POP(i). 1T, non-dummies changed dummies, or TV becomes balanced.
(b). Immediate from (a).
(c). At the start of stack push, T/ cannot be full, since otherwise PUSH(i 1) would be executed in the previous execution of PUSH(i) and would balance TV. Combining with (a) yields (c).
(d). It suffices to show that T empty then T/41 is empty. Suppose Tç is empty at time, then either
TV has been empty since the beginning of the simulation, where T Ty have been empty as well; or
TV becomes empty after POP(i). Right after this POP(i). T should be empty as well since otherwise POP(i + 1) would be executed which makes T. balanced.
(e). From the description of (i)( one can see that right after POP(i), the concatenation of non- dummy contents of T and T (in this order) equals the non-dummy content of T before POP()). Now, (c) is immediate.
In the following, we analyze the computational time cost. The simulation of one step (without calling PUSH or POP) costs O( = O/s1/k), ). execution of PUSH( or POP(i) (without calling PUSH(1 + 1) or POP(i + 1)) costs ( = O(s²/k), Since after PUSH(1) or POP(i), (), all levels that the number of stack operations between successive calls
Ty. T are balanced, one
to (i) or PiP(i) at least as 1//21//2 = ((((i-1))) 1)/K). So the total simulation time bounded by
W Σ ((s(i-1//))) O(t s/"")
O(Z)))
i=1 Finally, the total space used by the multi-queue machine is
2k Σ 4x [s(n)] < 8k ((((n) 1)1/K' 8k 2(s(n) + +1 ((k((n))
Remark 1. Our proof builds on the high-level idea of Theorem 3.2 in (Hühne, 1993), namely the use of queues of geometrically increasing size. However, their model allows each queue to remain idle in a step (i.e., neither pop nor push symbols), whereas we impose the stricter synchronous condition that every queue must pop and append exactly one symbol at every step. This requirement significantly complicates the simulation: when transferring data between adjucent levels, the two participating heads must arrive at the correct cells simultaneously. issue, simulation departs from that of Hühne (1993) in a crucial way, including the of auxiliary queues as buffers and the splitting of each content queue into two halves to regulate head positions.
resolve the head-alignment
4 STEP II: EFFICIENT SIMULATION OF MULTI-QUEUE TMs BY TRANSFORMERS
This section proves the following theorem, which together with Theorem 2 immediately implies Theorem Theorem is a generalization of Theorem in (Li & Wang, 2025), extending the single-queue construction the multi-queue setting.

---

Published as a conference paper at ICLR 2026
Theorem 3. Let TM be synchronous K K-queue Turing machine that, on input {0, 1]", uses most Smax(1) (n) maximum queue length and runs for at most t(n) steps. There exists a constant bit- size Transformer with (i) head-layer product K and (ii) context window of length O m (n)) that, on input x, takes (((()) CoTT steps and then outputs TM(x).
The complete proof is given in Appendix B; below we sketch the main idea. Let TM be such a synchronous K -uque Turing machine. Because each of the K queues pops exactly one element and also appends exactly one element at each step, their sizes are fixed during the execution of TM. Let so(n), 81 (n), ... 88-1( (n) denote the queue sizes. Let maax (n) 2 max, (n).
We now construct constant bit-size Transformer TF with context window of length Smax (n) to faithfully simulate TM step by step. The intuition is as follows:
We view each queue as a tape that is (i) infinite to the right and bounded on the left, and (ii) equipped with two tape heads, namely the front head and the rear head. The queue content corresponds to the tape segment between the two heads. Initially, the rear head of the r-th tape positioned at cell n. while the front head is at n s, (n) + 1 At each step, the front head first reads the current symbol, both heads move exactly one cell to the right, and then the rear head writes a symbol.
We represent the K tapes by stacking them cellwise: the i-th token in the Transformer's context records the i-th cells of all the K tapes, with the cells pointed by the rear heads corresponding to the newest token. Since the context window length is (n)) max, 8, (n). it can store the entire content of the queues. In addition, we set the vocabulary of the Transformer to be V Σ x Q, that a token can also track the TM state information.
The transition function of TM takes as input the lefumost elements of the queues together with the current state, and outputs the next state and symbols. To simulate this, we partition the K queues into L groups of size /I := K L. In each decoder layer l 0, ..... L 1, each of the H attention heads retrieves the leftmost symbol of one queue in the f-th group from previous tokens. In addition, the current state can be retrieved from the current token. Subsequently, ford-forward network is applied to implement the transition function 6.
Remark 2. Following Li & Wang (2025), we employ unlearnable relative positional encoding that evolves with n as defined in Equation (2). simulate the given TM on longer inputs, we only need to adjust the relative PE according to Equation (2); all learnable parameters, including their precision, remain unchanged. Thus, for a given TM, there is a single learnable parameter vector that is reused for all input lengths.
Our construction is of constant bit-size, since the notion of bit-size refers to the number and precision of learnable parameters, and the positional encoding is treated as part of the model architecture rather than as learnable parameters.
5 CONCLUSION AND FUTURE DIRECTIONS
This paper proves that constant bit-size Transformers can simulate multi-tape TMs with an optimal (ss(n) context window and only (s)))))) CoT steps per simulated step, where e > 0 can be made arbitrarily small. Beyond the theoretical advance, our construction suggests geometric-offset attention as promising architectural direction to enhance reasoning efficiency: each CoT token can be generated in (11) time, demonstrating that dense quadratic attention not essential for efficient universality. The key technical contribution is more efficient simulation of multi-tape TMs via multi-queus TMs. We propose three future directions: (i) Can the per-step overhead be further reduced to ((1) (ii) Our construction uses nonstandard relative positional encoding that assumes the space bound is known in advance; how can positional encodings be designed to adapt dynamically when the space bound is unknown? (iii) Our results focuses only the expressiveness. It remains an open question whether standard training protocols can actually discover such behavior; and empirical experiments are still to be done.
'eere for convenience, allow the front head point to negative indices; whenever this occurs, the corresponding cell is assumed to contain the blank symhol 1.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Qian Li's work was supported by Hetao Shenzhen-Hong Kong Science and Technology Innovation and Guangdong Basic and Applied Cooperation Zone Project (N..HH/OSW-KCCCB--0220000 Yuyi Wang's s work was supported by Basic Research Foundation (Grant the Hunan Provincial Natural Science Foundation (Grant No. 20.2002115.22).
Reproducibility Statement. This paper is theoretical. We make all assumptions, model defini- tions, and notation explicit in Section and Appendix A, present a complete statement of our main results in Thcorem 1, and provide full proofs of every lemma and theorem in Sections 3, 4, and Appendix B.
LLM usage. LLMs is used to aid and polish writing.
REFERENCES
Pranjal Aggarwal and Sean Welleck. LI: Controlling how long reasoning model thinks with reinforcement learning. arXiv preprint (0t::5533044997 2025.
and Sofya Klyuchereva Alberto Manzi. Evaluating & ranking gpt-5 reasoning ability, 2025.
Anthropic. The claude 3 model family: Opus, sonnet, haiku, 2024.
Sanjeev Arora and Boaz Barak. Computational complexity: modern approach. Cambridge Uni- versity Press, 2009. 2
Pablo Barcelo, Alexander Kozachinskiy, Anthony Widjaja Lin, and Vladimir Podolskii. Logical languages accepted by transformer encoders with hard attention. In The Twelfth International Conference on Learning Representations, 2024. 3
Alireza Amiri Bavandpour, Xinting Huang. Mark Rofin, and Michael Hahn. Lower bounds for chain-of th-uuughtt reasoning in aard-attention transformers. In Forty-second International Confer- ence on Machine Learning, 2025. 1,
Satwik Bhattamishra, Arkil Patel, and Navin Goyal. On the computational power of transformers and its implications in sequence modeling. In Proceedings of the 24th Conference on Computa- tional Natural Language Learning, pp. 455-475, 2020. 1.2
Lida Chen, Dong Xu, Chenxin An, Xintao Wang, Yikai Zhang, Jiangjie Chen, Zujie Liang, Feng Wei, Jiaqing Liang, Yanghua Xiao, et al. Powerattention: Exponentially scaling of receptive fields for effective sparse attention. arXiv preprint arXiv:2503.0355 2025.
Lijie Chen, Binghui Peng, and Hongxun Wu. Theoretical limitations of multi-layer transformer. CoRR, abs/2412.02975, 2024a.
Xingyu Chen, Jiahao Xu, Tian Liang, Zhiwei He, Jianhui Pang, Dian Yu, Linfeng Song, Qiuzhi Liu, Mengfei Zhou, Zhuosheng Zhang, et al. Do not think that much for 2+ 3=? on the overthinking of ol-e llms. arXiv preprint arXiv:2412.21187, 2024b.
David Chiang, Peter Cholak, and Anand Pillay. Tighter bounds on the expressivity of transformer encoders. In International Conference Machine Learning, pp. 5544 5562. PMLR, 2023. 3
Guhao Feng. Bohang Zhang, Yuntian Gu, Haotian Ye, Di He, and Liwei Wang. Towards revealing the mystery behind chain of thought: theoretical perspective. Advances in Neural Information Processing Systems, 36:70757-70798. 2023. 3
Sicheng Feng, Gongfan Fang, Xinyin Ma, and Xinchao Wang. Efficient reasoning models: survey.
arXiv preprint 2504 2504 2025. 3
Team Gemini, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. /1805 2023.

---

Published as a conference paper at ICLR 2026
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
Michael Hahn. Theoretical limitations of self-attention in neural sequence models. Trans. Assoc
Comput. Linguistics, 8:156-171, 2020. 3
Shibo Hao, Sainbayar Sukhbaatar, DiJia Su, Xian Li, Zhiting Hu, Jason Weston, and Yuandong Tian. Training large language models to reason in continuous latent space. arXiv preprint arXiv:2412.06769 2024. 3
Yiding Hao, Dana Angluin, and Robert Frank. Formal language recognition by hard attention trans- formers: Perspectives from circuit complexity. Transactions of the Association for Computational Linguistics, 10:800-810, 2022.
Martin Hühne. On the power of several queues. Theor. Comput. Sci., 113(1):75-11, 1993. 3, 4,
Huiqiang Jiang, Yucheng Li, Chengruidong Zhang, Qianhui Wu, Xufang Luo, Surin Ahn, Zhenhua Han, Amir H Abdi, Dongsheng Li, Chin- Lin, et al. Minference 1.0: Accelerating pre-filling for cong-context Ilms via dynamic sparse attention. Advances in Neural Information Processing Systems, 37:52481-52511 2024. 2
Mingyu Jin, Qinkai Yu, Dong Shu, Haiyan Zhao, Wenyue Hua, Yanda Meng. Yongfeng Zhang, and Mengman Du The impact of reasoning step length on large language models. arXiv preprint arXiv:2401.0m 04925, 2024. 3
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):
1-41, 2022. 2
Chenglin Li, Qianglong Chen, Liangyue Li, Caiyu Wang. Yicheng Li, Zulong Chen, and Yin Zhang. Mixed distillation helps smaller language model better reasoning. arXiv preprint arXiv:2312.10730, 2023. 3
Qian Li and Yuyi Wang. Constant bit-size transformers are turing complete. arXiv preprint arXiv:2506. 12027, 2025. 1, 2, 3, 9, 10, 14
Shiyang Li, Xiaoyong Jin, Yao Xuan, Xiyou Zhou, Wenhu Chen, Yu-Xia Yu-Xiang Wang, and Xifeng Yan. Enhancing the locality and breaking the memory bottleneck of transformer on time series forecasting, Advances in neural information processing systems, 32, 2019. 2
Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ra- masubramanian, and Radha Poovendran. Small models struggle to learn from strong reasoners. arXiv preprint 2:X::2502.122444 2025. 3
Zhiyuan Li, Hong Liu, Denny Zhou, and Tengyu Ma. Chain of thought empowers transformers to solve inherently serial problems. In The Twelfth International Conference on Learning Represen- tations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024. 1, 14
Haotian Luo, Li Shen, Haiying He, Yibo Wang, Shiwei Liu, Wei Li, Naiqiang Tan, Xiaochun Cao, and Dacheng Tao. Ol-pruner: sength-harmonizigg fine-tuning for ol-like reasoning pruning. arXiv preprint arXiv:2501.12570 2025. 3
Thang Luong and Edward Lockhart. Advanced version of gemini with deep think officially achieves gold-medal standard at the International Mathematical Olympiad. Google DeepMind Blog, July 2025.
Xinyin Ma, Guangnian Wan, Runpeng Yu, Gongfan Fang, and Xinchao Wang. Cot-valve: Length- compressible chain-of-thought tuning. arXiv preprint arXiv:2502.09601, 2025.
William Merrill and Ashish Sabharwal. A logic for expressing prepriici transformers. Advances in neural information processing systems, 36:52453-522633 2023.
12

---

Published as a conference paper at ICLR 2026
William Merrill and Ashish Sabharwal. The expressive power of transformers with chain of thought.
In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Aus- tria, May 7-11, 2024, 2024. 1,2, 14
William Merrill and Ashish Sabharwal. A little depth goes long way: The expressive power of log-depth transformers. CoRR, 1tss22533.0361,, 2025.
OpenAl. Introducing gpt-oss. OpenAl Blog, 2025. 2
OpenAl. Gpt-5 new params and tools, 2025.
Jorge Pérez, Pablo Barceló, and Javier Marinkovie, Attention is turing-complete. Mach. Learn. Res., 22:75:1-75:35, 2021. URL https:// httt::/://..l..oogr/paa/s/////// 2,
hhmll. 1,
Holger Petersen. Some remarks on lower bounds for queue machines (preliminary report). arXiv preprint arXiv:1310.6398, 2013. 3
Ruizhong Qiu, Zhe Xu, Wenxuan Bao, and Hanghang Tong. Ask, and it shall be given: Turing completeness of prompting. CoRR, abs/2441100199) 2024. 2,
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Di- rani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a bench- mark. arXiv preprint arXiv:2311. 12022, 2023.
Luka Ribar, Ivan Chelombiev, Luke Hudlass- Gallle,, Charlie Blake, Carlo Luschi, and Douglas Orr. Sparq attention: Bandwidth-efficient IIm inference. In Forty-first International Conference Machine Learning, 2024.
Ruhi Sarikaya. Path to artificial general intelligence: Past, present, and future. Annual Reviews in Control, 60:101021) 2025. 2
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp-
464-468, 2018.
Lena Strobl, William Merrill, Gail Weiss, David Chiang, and Dana Angluin. What formal lan- guages transformers express? survey. Transactions of the Association for Computational Linguistics, 2::53-551,, 2024.
DiJia Su, Hanlin Zhu, Yingchen Xu. Jiantao Jiao, Yuandong Tian, and Qinqing Zheng. Token assorted: Mixing latent and text tokens for improved language model reasoning. arXiv preprint 22Xi:25502.0325.. 2025. 3
Yang Sui, Yu-Neng Chuang, Guanchu Wang. Jiamu Zhang, Tianyi Zhang, Jiayi Yuan, Hongyi Liu, Andrew Wen, Shaochen Zhong, Hanjie Chen, al. Stop overthinking: A survey efficient reasoning for large language models. arXiv preprint arXiv:2503. 16419, 2025. 3
Yuyang Wu, Yifei Wang. Ziyu Ye, Tianqi Du, Stefanic Jegelka, and Yisen Wang. When more is less: Understanding chain-of-thougle length in llms. arXiv preprint arXiv:2502.07766 07266 2025. 3
Heming Xia, Chak Tou Leong, Wenjie Wang, Yonggi Li, and Wenjie Li. Tokenskip: Controllable tfain-of-thought compression in llms. arXiv preprint 2502:22002.1066 2025.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. In The Twelfth International Conference Learning Rep- resentations, 2024. 2
Andy Yang. David Chiang, and Dana Angluin. Masked hard-attention transformers recognize ex- actly the star-free languages. Advances in Neural Information Processing Systems, 37:10202- 10235, 2024.7
Chenxiao Yang, Nathan Srebro, David McAllester, and Zhiyuan Li. Pencil: Long thoughts with short memory. arXiv preprint arXiv:2503. 14337, 2025a. 1,2
13

---

Published as a conference paper at ICLR 2026
Wenkai Yang, Shuming Ma, Yankai Lin, and Furu Wei. Towards thinking optimal scaling of test- time compute for IIm reasoning. arXiv preprint arXiv:2502.188000, 2025b.
Shunyu Yao, Binghui Peng, Christos Papadimitriou, and Karthik Narasimhan. Self-attention net- works can process bounded hierarchical languages. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3770-3785, 2021.
Xunyu Zhu, Jian Li, Can Ma, and Weiping Wang. Improving mathematical reasoning capabilities of small language models via feedback-driven distillation. arXiv preprint wrXiv2241..14666 2024.
A MODIFICATIONS ON TRANSFORMERS IN TURING COMPLETENESS PROOFS
Existing proofs of seriingcomplltenesss for Transformers typically do not employ the vanilla ar- chitecture, but instead introduce specific modifications. For clarity, we summarize these changes below.
Pérez al. (2021) consider Transformer with an absolute positional encoding given by the triplet (i, 1/i, 1/2²) at position i. In their construction, the attention score is defined as the negative absolute value of the dot product, and the attention mechanism uses average-hard attention. Moreover, the feed-forward layers employ sigmoid activations in place of ReLUs.
Merrill & Sabharwal (2024) dispense with positional encodings altogether and instead adopt Strict Causal Masking, where attention at position i can access all tokens up to position - 1 but not the current token i. Their construction also uses average-hard attention and introduces Projected Pre-Norm: an arbitrary linear projection is allowed before layer normalization.
In the proof of Li et (2024), the simulated circuit directly encoded into the absolute positional encoding, rather than into the parameters.
Qiu al. (2024) consider Transformers with nonstandard absolute positional encodings. In their construction, the query, key, and value maps in the attention sublayer are implemented as ReLU networks rather than linear transformations.
Finally, both Li & Wang (2025) and this work employ nonstandard relative positional encodings.
B PROOF OF THEOREM 3
Proof. Let TM = (E, be synchronous K-queue Turing machine running ((n) time and S s(n) space. Without loss of generality, let the tape alphabet be Σ {0, 1, 1) and the set of states be Q = {0, 13rh for some c E N. We further assume that fstort never appears in any transition 8(00. ""oo,"",,Ki,, --)1,)))
i.c., once TM leaves qstart the beginning, it never returns. Because each of the K queues pops exactly one element and also appends exactly one element at each step, their sizes fixed during the execution of TM. Let so(n), (n).... s (n) denote the queue sizes, with 8max (n) max 8r (n). For technical convenience, without loss of generality, we assume 81 (1)),, 82()), ... "......,,"" are always even numbers.
We now construct constant bit-size Transformer TF with context window of length s(n) to faithfully simulate TM step by step. Let V = S x Q x P = (0, 1.13 x {0, 13 x {0, 1}. where P iisa bit indicating the parity of the location. The intuition is as follows:
1. Token embedding layer Map each token v = (a = (ou, -k-1), q,P) to a vector emb(a,, p) = (embo(ao.4.2) (.), (anbb//oo_p........xx(rrrr--..
E ()+ K as follows:
(1.1,000.gg00,000p.p0.. if ar = 0; (1.00.1...400000.p.))) if = 1;
emb,(ar.g) =
(1)
(1.00.0111.00000.2200) ifa, = L.
14

---

Published as a conference paper at ICLR 2026
loapecary sutagelle akailan
1 poleena embr (110
All he patiee
Oue cusodag Colese
Eoondegal guaeti (ieeiso
embii
(o0 , ,0. g
amb Meaa
( beads)
(00-1(0))+ 0g (-50(1)+3 1x1
embtuur
Exaera added Di proppne (db 1020.
squese 1M1
Figure 2: Illustration of our construction in the proof of Theorem 3.
Relative positional encoding
Inside the attention score, add relative information follows?:
1a/H- 2 (2Er.-s.(m))) Cyy
if A >
(A) RR/H
(2)
where e, Rd/H stands for the standard basis vector with 1 at the r-th coordinate.
3. Decoder layer The Transformer has L decoder layers, and each decoder layer has H attention
= K. For the k-th attention head the f-th layer, where & = 0,... ... -I- and 0= 0.... L 1, we set the matrix K Q V E Rdxd/H Rdx((c+9)xL) where d = (c 9) as follows: let r k L
heads, with
Matrix KG all-zeros matrix;
Matrix Q it has only one non-zero entry Q Qc+t)))+1,r =
Matrix Vg:: it has four non-zero entries: V(c+0))44.((4990)ttt = V(c+9)r = ((++9))+1((044))4+== ((009))+4448((00))))) =1
We set W RRd the identity matrix and set the bias vector to Og-
The feed-forward network FF maps h' to Oa, except that FF!-1 is designed to simulate the transi- tion function d. Specifically, we let
FFL-1(h) + h e(6(01.-- xx..)) where q = h5:5+4 and y/ hots
(ct0))+448 h(t9))+ot9)
or (
otherwise;
E R Vl denotes the standard basis vector with a 1 at the coordinate indexed by Here, ((a.9.)) (a, P)..
4. Output layer We set Wout to be the identity matrix and yout the all-zeros vector. Then the output layer outputs 1i+ = arg m
Analysis What remains is prove that our construction TF faithfully simulates TM. Let (a (o',9)), (a² (0².))).... denote the execution log of TM running on x {0, 1)", where o² (o). ... T k denote the elements added the queues in the i-th step and q the state when adding o¹. For convenience, we assume initially the input queue (r = 0) contains æ, and the other K - queues contain only blank symbols 1; that is, for n. (o' .'') (I, 1k K- qstart). Then, one can see that
(o"t1.g ( = 5
(0 (g)000())+
j-ak (x))+1())+1 .g¹). K-1
Without loss of generality, assume d/H 2 H I equivalently e + 9 2 H by making sufficiently
large.
15

---

Published as a conference paper at ICLR 2026
where i (n)(n) + 1 S O, then set
of o0- = Rig if 0;
[1 if 1,
1.
Let U1, U2,.. denote the token sequence in the context of TF when it takes 1-1 1K1 Istart, 1),-., Un = (Iz: K-1 Istart, n mod 2) as input. By induction, we will show that for each 2 1, we have v; = (ar q', mod 2). and then finish the proof. The base case and By (o" o y' mod 2) for any 1' S..
is is trivial. Now, assume i
Claim 2. For any 0 < l I L L, h and he differ only in the entries at indices (c+ 9) . + (e + 9) r + 7 and (c+ 9) r c 9.
Proof. By the description of TF, we have
h FF(5 + h4
f.5
h
15-05
1 + (..
where
t-1 f-1 ai, kk
h Ve
1-1 f-1 V Skiit h
By definition of VL t-1 is zero at all indices except (c + 9) rr 5 (c+ 9) r r 7 and
i,k
(c+ 9) - 9. By induction on e, we get the claim.
Claim 3. 1c4994404 = (c+0)r+040 if and only if - (n) + +1 0. Furthermore, if s, (n) +
1 1> O, then h 4-005
Proof. Let (f, be the unique pair satisfying that r = k. L + l. One can easily check that
h +l
1-5.5
= ( + (d1AA1CH11441))))))))))
For the k-th attention head in the f-th layer, by Claim 2, we have K hl Oaji. Q h
Q% h er.
$f. hardmaxx ((pos(i-1).kk) - h? Q0), (pos(i 1), h Q())
(0, .............................. 0,1) E R',
if S s, (n) - 1,
(0 (0:-..(m) (m)) 1, 1,O,,, (m)))) R.
if (n S s(n) - 1,
( (0stn)-.,
1.0 1,
R*
if 2 s(n).
and
d
Σ
Spid (h/. vz) Σ shns (h) (A).vi)
ak
1=1
Σ shas (0 hagp:eaccaaa....ooooaeaaaeooooe pog (ce
he ( (e (0(49))44444 (0
((e-))(---)). 0(+48))--2-1) otherwise
if < 1,
Recall the assumption (n), (n), ... (n) are always even numbers, now the claim is clear by the induction hypothesis.
Besides, in the last forward network layer FF4-1 we have
h 1-0.5
1:00+ = h
1,5:+4 = 9t and h f.-0.5
= (n)s,
The vector h according to Equation (3). Then one can see that the output layer outputs (ait1. () as desired.
0.5.5 mapped to FF(h! -..5 + h 0.5 which is e (000) ( ,m .K 1 (
16


Reviews:
[{"review_id": "NNfd47tAkG", "reviewer": "Reviewer_iCy5", "summary": "This paper contains a recent line of work providing Turing-completeness proofs for Transformers. This paper specifically provides a construction of constant bit-size (that is, constant number of parameters at constant precision) transformers with bounded (moving) context window and custom relative positional encoding vectors that simulates Turing machine computations with a smaller slowdown than in the closest prior work (Li&Wang 2025). That prior construction used \\Omega(s(n)) CoT steps to simulate a single step of a Turing machine operating in space s(n). In contrast, the current paper cuts this down to O(s(n)^c) for arbitrarily small positive c, where larger transformers can achieve smaller c.", "questions": "Missing citations:\n- Another relevant recent paper is [1], which provides another Turing completeness construction, and also proves optimality of CoT lengths.\n\n[1] Amiri et al, Lower Bounds for Chain-of-Thought Reasoning in Hard-Attention Transformers, ICML 2025", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Contributes to emerging theoretical understanding of the Turing completeness of Transformers.\n- Improves over prior work by reducing the CoT overhead in a constant bit-size setting.", "weaknesses": "- The design of the model is nonstandard. In particular, adding relative positional encoding vectors in line 195 appears confusing: is the idea that the encoding vector pos(i-j) depends on both the current position j and a later position i from which an attention head looks back at position j? This seems to make both parallel training impossible and autoregressive decoding extremely inefficient, as the whole transformer activations would have to be recomputed throughout the entire context for every next-token generation?\n\n- The construction assumes positional encodings depending on the space bound s(n), and thus does not inherently length-generalize over all n. Is it really fair then to say that the model has fixed bit-size?\n\nMissing references:\n- line 077: the claim about practical attention patterns appears speculative, especially the link to the geometric progression attention. Is there a reference for the claim?\n- line 080: can the authors provide a citation for this \"common argument\"?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "atiR8u5Bhl", "reviewer": "Reviewer_VMLy", "summary": "Summary: this paper presents a refined theoretical analysis of the ability of CoT transformers to simulate Turing machines, which has been studied in prior work. The paper shows that a multitape TM with time t(n) and space s(n) can be simulated by a constant bit-size transformer with context length O(s) and runtime O(t * s^c), for arbitrarily small c. Thus, constant bit-size transformers can quite effectively simulate transformers. The proof technique is interesting in its own right, going through a conversion of multitape Turing machines to multiqueue Turing machines.", "questions": "See \"Weaknesses\"", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. I find the main result about the space requirements needed to simulate a Turing machine with constant-bit-size CoT transformers to be valuable.\n2. The intermediate result converting multitape Turing machines to multiqueue Turing machines is interesting in its own right and technically innovative.\n3. The high-level technical plan and proofs are clear and rigorous", "weaknesses": "### Make Dependence of Space/Context Window on k' Explicit\n\nIn theorem 2, how does the space O(s) depend on the queue factor k'? The way you are reducing time overhead is increasing k', so it would be nice to understand how space scales with this.\n\nIt would also be good to understand how this shows up in the main result about transformers: you say that we can make the time overhead arbitrarily small, but how does this increase the context window we need?\n\n### Theorem 3 Suggestions\n\nOverall, the theorem looks solid, but I have some minor suggestions for improvement:\n- Clarify non-standard positional encodings in theorem statement\n- Clarify use of hardmax in theorem statement; also it would be helpful to explicitly reference unique hard (UHAT) vs. averaging hard (AHAT) variants of hardmax. presumably your construction should work with either\n- How uniform is this construction? (see below)\n\nI understand constant bit-size (a la Li and Wang) to mean that the number of params and precision independent of context length. Does this imply that the transformer constructed in Theorem 3 is fully uniform, i.e., the parameters can't change at all with n? It would be helpful to explicitly mention the level of parameter uniformity that your construction attains (potentially in Table 1). E.g., the Merrill & Sabharwal's construction re-uses the same parameters for any n, so it is fully uniform, but this is not true for all constructions (e.g., Li et al., where position embeddings can evolve in a complicated way with n).\n\n### Quadratic vs. Linear-Time Attention Discussion Deserves More Nuance\n\n> This shows that the common argument “quadratic-time attention= ⇒fundamental throughput bottleneck” against Transformer-based AGI may not be a principled limitation but a byproduct of dense attention.\n\nThis sets up a bit of a strawman and makes a vague/strong claim against it. Can you clarify the argument in scare quotes (and attribute to someone) and clarify your counterargument. E.g., you're saying that expressive power does not require quadratic-time attention?\n\nMoreover, while quadratic time attention is sometimes invoked as a drawback of attention, the related issue that matters more in practice (e.g., for GPU memory limitations) is the linear memory incurred by attention on long sequences to store the KV cache. This limitation is not overcome by your construction (and, in fact, is more fundamental), since for any function requiring s(n) = Omega(n), you will need to store linear memory to compute it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Qe2d3clHS7", "reviewer": "Reviewer_Carr", "summary": "This is a theoretical work whose main contribution is the proof that any (t(n), s(n)) time-space bounded many-tape Turing Machine can be simulated by a constant bit-size Transformer with context window of optimal length and using O(s(n)^c), with any c>0, chain-of-thought (CoT) steps for a singe simulation step. This improves on previous results regarding efficient TM simulations by Transformers. To show this result, the authors first present a time-efficient simulation of k-tape Turing Machines using so-called “synchronous” queue TMs and then describe a simulation of multi-queue TMs by Transformers. The proposed simulation of TMs using synchronous queues seems to be the main technical achievement of this paper.", "questions": "According to the proof, it seems that in the statement of Theorem 1 the total length of the CoT of the simulation should be $O(t(n) \\cdot s(n)^{1/k'} \\cdot s(n)^{6k/K})$, not $O(t(n) \\cdot s(n)^{6k/K})$, where $k'$ is used in Theorem 2.\n\nAccording to your description L. 182-186 a TM cannot empty the stack.\n\nL. 310: what do you mean by \"rotates the tape heads\"?\n\nUse uniformly throughout the document: stack or queue.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "This work improves on previously presented results on efficient TM simulations by constant bit-size Transformers. In particular, it is shown how to reduce the number of CoT steps per simulated step from O(s(n)) to O(s(n)^c), for any constant c>0. The analysis is non-trivial and the result is in line with what is actively being researched by many authors studying Transformers through the lens of computational complexity.", "weaknesses": "Although the paper improves on known results, the work presents incremental progress in the field. In particular, as shown in Table 1, the submitted paper improves on the results of Li and Wang (2025) essentially only with respect to the number of CoT per single TM step. Importantly, the simulation results (presented in Table 1) do not take into account the total CoT length of the simulation. So, by simulating a (t(n), s(n)) time-space bounded multi-tape TMs with a single-tape that increases time but leaves space unchanged, we can conclude that the result given in the line “Li & Wang (2025)” holds true for multi-tape TM.\n\nThe main technical achievement of this paper seems to be the simulation of TMs by TMs using synchronous queues. But the idea behind this construction is not entirely new: as the authors note, the proof relies on the general idea of ​​Theorem 3.2 in (Hühne, 1993). The modification proposed in the submitted paper is somewhat non-obvious, but in my opinion it is not strong enough for ICLR.\n\nIt would be nice to have *precise* descriptions for the key notions used in the paper as, e.g., precision, CoT, Window, embedding dimension (Dim.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "beh0h1zGb2", "reviewer": "Reviewer_vfEe", "summary": "This paper presents a theoretical advance in understanding the computational efficiency of Transformers.\nIt proves that any $(t(n), s(n))$-bounded multi-tape Turing Machine can be simulated by a constant bit-size Transformer with an optimal $O(s(n))$ context window and only $O(s(n)^c)$ chain-of-thought (CoT) steps per simulated step, where $c > 0$ can be made arbitrarily small.\nThe work also introduces geometric-offset sparse attention, linking theoretical universality with practical efficiency.\nThe key technical innovation lies in efficiently simulating multi-tape TMs via synchronous multi-queue TMs, followed by a Transformer-based realization.", "questions": "Could the per-step overhead be further reduced to $O(1)$, and what are the theoretical obstacles?\n\nHow might geometric-offset attention be implemented or approximated in modern Transformers with dynamic context lengths?\n\nCould a simplified Transformer experiment empirically demonstrate the predicted efficiency?", "limitations": "", "rating": 10, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Improves efficiency bounds for Turing-complete Transformer simulations from $O(s(n))$ to $O(s(n)^c)$ for any small $c > 0$.\n\nThe synchronous multi-queue TM construction is original and cleverly designed.\n\nThe geometric-offset attention model aligns with sparse attention mechanisms used in practice (e.g., LogSparse, PowerAttention).", "weaknesses": "A small experimental validation or simulation would make the theoretical results more tangible.\n\nSeveral sections (especially Step I and Appendix B) are heavy in notation and could benefit from clearer examples or figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)