Title: Constant Bit-size Transformers Are Turing Complete
Venue: NeurIPS 2025
Accepted: yes
Keywords: Transformer

Turing complete

Post machine

context window length

space complexity
Abstract: We prove that any Turing machine running on inputs of arbitrary length can be simulated by a constant bit-size transformer, as long as the context window is sufficiently long. This improves previous works, which require scaling up either the model's precision or the number of parameters on longer inputs. Furthermore, we prove that the complexity class SPACE$[s(n)]$ exactly characterizes the expressive power of a constant bit-size transformer with a context window of length $s(n)$. Our approach relies on simulating Post machines, a Turing-complete computational model. Post machines can be modeled as automata equipped with a queue, exhibiting computational behaviors naturally aligned with those of transformers. The behavioral similarity between transformers and Post machines may offer new insights into the mechanisms underlying the reasoning abilities of transformers.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is aimed at understanding the computational power of transformers with respect to the classical model of Turing Machines. Recent work has shown that transformers are Turing complete. This paper refines this body of work by showing that transformers of fixed size (in terms of size of embedding, number of layers, number of parameters) are sufficient. The equivalence proof does not require arbitrary precision arithmetic. It establishes precise relationship between the space bound of TM computation and the length of context window.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe main result (Theorem 1) is precise and concise. The authors make a compelling case that this is a tight result, in particular, allowing the context window length to grow with the input size is unavoidable.\n\nThe proof structure is very clear and intuitive. It uses a classical result that shows that TMs can be simulated by queue machines. They explain how a transformer can then simulate a queue machine: the context window stores the queue, adding elements corresponds to emitting next token. The main technical challenge is using the attention mechanism to retrieve the current state at the oldest symbol and simulate a transition.\n\nWeakness:\nThe problem has been studied in a sequence of earlier papers. This is a nice and clean result, but doesn't seem surprising. Also not clear if this is relevant to practice. the claimed connection to understanding  \"reasoning ability\" of transformers is not justified.\n\nQUESTIONS:\nThe introduction highlights \"fixed numerical precision\" as a challenge w.r.t. earlier work. This is not discussed during the proof of the main result. It's likely obvious given the very discrete encoding, but could be explained (or more pertinently, why was this an issue in some of the earlier work, and what insight avoids it).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAllowing CoT steps enables transformers with the computational power to perform complex reasoning tasks. A body of theoretical work has shed light on this phenomenon through proof that transformers can simulate Turing machines, under various architectural assumptions. This work investigates the setting where the precision and context window of the transformer must be bounded, and provides a novel proof of Turing-completeness using Post machines. They show that transformers with constant bit-size and context window s(n) are equivalent to SPACE[s(n)]. The computational behavior of Post machines aligns very well with CoT.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n- Very clearly written, constructions are shown in detail and I could follow them without much difficulty. \n- The identification of Post machines as a natural analogue of CoT computation is very nice and I agree sheds more light on the behavior of transformers with CoT than the proofs with Turing machines. \n- The work highlights the sometimes subtle differences between different algorithms that transformers may use in CoT\n\n\nWeaknesses\n- I’m not sure that the other proofs requiring an O(log n) precision is a very severe weakness that needs addressing – in practice the precision is often large compared to the sequence length (e.g. float16 can handle inputs on the order of 10^8 tokens without any problems (https://arxiv.org/pdf/2210.02671)). \n- If I understand correctly, the construction crucially upon the relative positional encoding pos(i,j) which sweeps some things under the rug. While other works use non-constant bit precision in order to compute the token to attend to, the pos(i,j) simply provides location of the Post machine’s head, thus streamlining the process. \n- For non-theory readers it would be informative to add a brief section on limitations or practical insights (given that you have 1.5 pages of space left over). For instance, what problems are solved in (say) an O(1) sized context window and how it compares to what problems transformers with CoT can solve in practice.\n\nQUESTIONS:\nQuestions\n- Is it possible to achieve the same effect of using the pos(i,j) positional encoding using some kind of an attention construction? While no longer constant bit-size, this would still be interesting in contrast with (https://arxiv.org/abs/2503.14337) by achieving space-efficiency without their reduction process. \n- Other proofs also use non-standard positional encodings (like pos(i)=i^2 or 1/i), but in this case the positional encoding depending on s(n) seems even more nonstandard. Could the authors make these and other assumptions more clear? \n- Is there more to say about the “natural alignment” between Post machines and transformers? Any future work or practical implications? There is a healthy amount of room left in the paper. \n\nSuggestions\n- It could be quite informative for the authors to create a table of various CoT Turing-completeness proofs to compare their result with others in the literature. For instance, the table could have columns for “precision” and “positional encoding” to clearly demonstrate the distinction with previous work. \n- Technically, the definition of hardmax computing 1/t means that O(log n) bit precision is possible in the transformers. Your construction gets around this by enforcing that only one item in the context window has non-zero score. A note could be made about this, or you could use some form of argmax (like right-most hardmax https://arxiv.org/abs/2310.13897) \n- It seems that [Liu et al. 2024] should be [Li et al. 2024]? \n- There seems to be a typo on line 263 with the angle brackets\n\nLIMITATIONS:\nI think the authors could be more upfront about the positional encodings used and other assumptions they make, but otherwise things are pretty clear.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is part of an active and growing line of theoretical research on the expressive power of transformer architectures. The central question is: how expressive are transformers as language deciders compared to traditional resource-bounded models like time- and space-bounded Turing machines (TMs)? This is a relatively new area with many open directions. A foundational result by Pérez et al. showed that certain transformer models are Turing complete when allowed to use Chain-of-Thought (CoT) reasoning, enabling them to simulate arbitrary computations. Since then, several works have explored what transformer parameter settings are necessary and sufficient to capture various classical models (e.g., space, time, Boolean circuit depth). A key parameter in this context is precision—the number of bits used to represent real numbers. Prior work established that logarithmic precision suffices for general computation.\n\nThis paper advances the field by showing that even constant-bit precision—a fixed number of bits—suffices to simulate Turing machines efficiently. Specifically, the authors define a transformer-based class ${\\rm WINDOW}(s(n))$, where the ith token depends only on the preceding $s(n)$ tokens, and prove that ${\\rm WINDOW}(s(n)) = {\\rm SPACE}(s(n))$ for all $s(n) ≥ n$. Thus, constant-precision transformers with an attention window of size $O(s(n))$ recognize exactly the languages in ${\\rm SPACE}(s(n))$.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper makes a significant theoretical contribution by precisely characterizing the computational power of constant precision transformers in terms of classical complexity classes. It is indeed good to know that constant-bit precision suffices and this result is  relevant to both theory and practice.\n\nWeaknesses: The literature review could be more thorough. In particular, the authors should look at the recent survey “What Formal Languages Can Transformers Express? A Survey” by Strobl et al. (https://arxiv.org/abs/2311.00208), which provides a comprehensive overview of formal language expressivity results for transformers. Comparing and contrasting your result with  relevant results cited in this survey would help contextualize the current contribution within the broader research landscape. For example, in results such as those by Merrill and Subharwal involving ${\\rm CoT}(T(n))$, what is the implicit or explicit window size? Is it effectively $T(n)$? Clarifying how this relates to the ${\\rm WINDOW}(s(n))$ definition would strengthen the positioning of the result.\n\nQUESTIONS:\nIt would be interesting for the authors to explore and discuss the implications of their results when $s(n)$ is sublinear (e.g., $O(\\log n))$. Does ${\\rm WINDOW(O(log n))$ capture LOGSPACE? Are constant window transformers interesting?\n\nLIMITATIONS:\nThere is no explicit Limitation section. But I am not concerned about that for this theoretical work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a theoretical result on the computational power of Transformers. The authors prove that a Transformer with a constant bit-size (i.e., fixed number of parameters and fixed numerical precision) is Turing complete, capable of simulating any Turing machine on inputs of arbitrary length, provided its context window is sufficiently long. Theoretical statements and proofs are provided.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The paper is well-motivated and clearly written.\n- The paper provides more than a simple \"yes/no\" answer to Turing completeness. The equivalence creates a direct and formal link between architectural parameters (e.g., context length) and computational budgets (memory/space).\n\n## Weaknesses\n- **Expressivity vs. Learnability**: The proof is a constructive one, meaning it shows that a Transformer with specific, carefully engineered parameters exists. It does not address whether this complex, algorithmic behavior could be learned using standard gradient-based training methods. This is a significant gap between the theoretical capability and practical reality.\n\n- **Computational Cost of Simulation**: The simulation requires a number of chain-of-thought (CoT) steps that scales with O(t(n) * s(n)), where t(n) is the time complexity of the original Turing Machine. For many problems, this is prohibitively large and far exceeds the number of reasoning steps seen in practice, which may limit the direct applicability of the result.\n\n- **Idealized Model Assumptions**: The construction relies on an idealized Transformer model that uses hardmax attention and omits layer normalization. While the authors note these are common simplifications in theoretical work, it creates a disconnect from the standard architectures used in practice.\n\n- **Lack of Empirical Validation**: The work is purely theoretical. While the claims are powerful, the paper would be strengthened by even simple experiments that explore or demonstrate aspects of the proposed construction, which could provide insights into its feasibility or the challenges in realizing it.\n\nQUESTIONS:\nSee the Weaknesses part above.\n\nLIMITATIONS:\nThe authors do not discuss limitations of this work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTransformers with Chain of Thoughts (CoTs) were initially proven by [Perez et al. 2021] to be Turing-complete. Essentially, this means that they are allowed to feed the output back into the input several times (that is, \"think out loud\") before outputting the final result of the computation. In subsequent years, researchers have revisited the result from [Perez et al. 2021] by simplifying the proof, or removing some unpractical assumptions in the model. For example, in the following paper Bhattamishra et al. simplified some of the assumptions in [Perez et al.] including replacing positional encodings with positional masking:\n\nSatwik Bhattamishra, Arkil Patel, Navin Goyal: On the Computational Power of Transformers and Its Implications in Sequence Modeling. CoNLL 2020: 455-475\n\nIn this paper and in [Perez et al. 2021], finite but unbounded precision is assumed. More precisely, this means that computation is done over rational numbers, which amounts to potentially infinitely many decimal places (it is possible to actually restrict finitely many decimal places, but the number of required decimal places will grow in the size of the input). This result has been further refined in [Merrill and Sabharwal 2023] with an even simpler usage of positional masking and additionally complexity analysis.\n\nA different set of assumptions for Turing-completeness was obtained by [Liu et al. 2024], where the authors use an unbounded \"embedding dimension\" but constant precision. Roughly speaking, this means that different parameter instantiations (possibly of different dimensions) exist for different input lengths. [Actually, the paper itself doesn't talk about another proof of Turing completeness of transformers, but as the current submission also acknowledges it follows from [Liu et al. 2024]]. \n\nThe current paper improves on and simplifies the result by [Liu et al. 2024] in that the dependence on the input length exists only in the relative positional encoding; the other parameters stay the same. More precisely, the bit size of a transformer is essentially the number of bits (with only floating points allowed, not rational numbers) required to represent the parameters of the transformer. Crucially, a relative positional encoding is allowed as a parameter. With this assumption, the authors establish a \"universal CoT transformer\" with a constant bitsize. Actually, this translates to non-fixed (absolute) positional encoding that depends on the input length. Additionally, the current paper also assumes a sliding window of a non-fixed size on the concatenation of the input and the intermediate CoT results, which is reasonable. \n\nAdditionally, the paper also proves a characterization of the complexity class SPACE[s(n)] of problems solvable in s(n) space, by allowing a sliding window of size s(n). The proof goes by encoding a universal Post Machine (a queue machine) by means of a CoT transformer. Note that the number of bits required to store values in the intermediate computation might also grow in the length of the input (for example, owing to the use of average hard attention mechanism, which divides by a non-fixed number).\n\nSTRENGTHS AND WEAKNESSES:\nTo the best of my knowledge, this is the first Turing-completeness proof that assumes some sort of constant bit size in the entire parameter set. The only catch is that one has to use relative positional encoding, which translates to a parametric absolute positional encoding. That said, unlike [Li et al. 2024], this is the *only* parametric component; the others are fixed. To the best of my knowledge, this is the strongest Turing-completeness result under the \"constant precision\" regime.\n\nThe proof is also relatively simple and quite nicely presented. Essentially, the queue content in the post machine corresponds to the sliding window content in the CoT transformers. The explicit construction also makes it clear what varies in the size of the input. My only quibble here is that the authors have 1.5 extra pages, which can be used to provide more intuition of the proof (as well as some definitions). \n\nIn addition, I found that some definitions are excessively overcomplicated. For example, multi-head attention is defined in the paper, whereas only single-head attention is used in the proof. \n\nOverall, I think the strengths overweigh the weaknesses in the paper and I am leaning towards acceptance.\n\nMinor:\n- l. 145: \"in some state\"\n- l. 180: $s^l_{k,i,j}$ is not defined. Do you mean the jth entry of $s^l_{k,i}$?\n- l. 247: at first sight, it seems that you need to know i and j in pos(i-j), instead of just the difference i-j. Justify that this is not the case.\n\nQUESTIONS:\n- Can you say what happens in the case of a fixed \"absolute\" positional encoding?\n- Is it possible to replace your use of positional encoding with a casual positional masking?\n- Can you explain key technical differences in the proofs in your paper compared to [Perez et al. 2021] and [Li et al. 2024]?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}