Title: Pause Tokens Strictly Increase the Expressivity of Constant-Depth Transformers
Venue: NeurIPS 2025
Accepted: yes
Keywords: complexity theory

transformers

learning theory

circuit complexity
Abstract: Pause tokens, simple filler symbols such as "...", consistently improve Transformer performance on both language and mathematical tasks, yet their theoretical effect remains unexplained. We provide the first formal separation result, proving that adding pause tokens to constant-depth, logarithmic-width Transformers strictly increases their computational expressivity. With bounded-precision activations, Transformers without pause tokens compute only a strict subset of $\mathsf{AC}^0$ functions, while adding a polynomial number of pause tokens enables expressing the complete class. For logarithmic-precision Transformers, we show that adding pause tokens achieves expressivity equivalent to $\mathsf{TC}^0$, matching known upper bounds. Empirically, we demonstrate that two‑layer causally masked Transformers can learn parity when supplied with pause tokens, a function that they appear unable to learn without them. Our results provide a rigorous theoretical explanation for prior empirical findings, clarify how pause tokens interact with width, depth, and numeric precision, and position them as a distinct mechanism, complementary to chain-of-thought prompting, for enhancing Transformer reasoning.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper formally characterizes how pause tokens increase the expressivity of transformers. In the fixed precision case, the paper shows that transformers with pause tokens are equivalent to the circuit class $\\mathsf{AC}^0$, and strictly more expressive than transformers without pause tokens. In the log precision case, transformers with pause tokens are equivalent to $\\mathsf{TC}^0$, but the paper does not provide a separation result for transformers without pause tokens. Empirically, the paper shows that (log precision) transformers with pause tokens can solve the parity task on long inputs, where tranformers without pause tokens fail.\n\nSTRENGTHS AND WEAKNESSES:\n*Strengths*\n\n- This paper provides theoretical insight into the empirical finding that pause tokens can improve performance on some tasks. This can help explain some empirical results that have found pause tokens to be helpful in some cases but not others, and (as the authors discuss) possibly have implications for understanding chain-of-thought and the limitations of quantized models.\n- I found the writing to be very clear, and I appreciated the discussion section and the detailed discussion of areas for future work.\n- Experiments show that pause tokens allow transformers to solve the parity task on long inputs, where models without pause tokens fail. There is a helpful ablation showing that the benefit of pause tokens is not simply due to overcoming the causal mask.\n\n*Weaknesses*\n\n- The paper does not provide a theoretical separation result between $\\mathsf{TF}[L,L,0]$ and $\\mathsf{TF}[L,L,P]$ (i.e., showing that log-precision transformers with pause tokens are strictly more expressive than without pause tokens). This limitation is discussed in detail and seems reasonable to me.\n- The experiment (section 5) is not directly related to the theoretical results. To my understanding, a log precision transformer without pause tokens is expressive enough to compute parity (using the two layer circuit construction). The experiment shows that these models struggle to learn it in practice, while pause tokens make learning easier in practice. This result is interesting, but not directly related to the expressivity results in the paper. The authors discuss these limitations.\n- It would have been interesting to see experiments with fixed precision transformers, but I understand that it is impractical to train models with precision $< O(\\log n)$, which the authors discuss.\n\nOverall, I think this paper makes a clear and valuable contribution to our theoretical understanding of transformers. I think all of the weaknesses are reasonable, and acknowledged in the paper.\n\nQUESTIONS:\n- Would it be possible to give an example of a problem in $\\text{AC}^0$ that can be solved with padding tokens but not without padding tokens?\n- In the experiment section, is the non-causal model (without pause tokens) also trained with hints? Would it be possible to provide an analogous figure to the figure at the bottom of page 7?\n\n\nMinor comments:\n\n- What is $\\mathcal{A}$ (in line 150)?\n- It could be helpful to briefly define \"advice\" (in line 261), for readers who aren't familiar with the reference.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors theoretically studied the expressivity of constant-depth log-embedding-dimension transformers with pause tokens, which are filler symbols appended to the end of the input sequences that provide more space for computation. \n\nFor constant precision transformers, the authors prove that transformer with polynomial pause tokens is computationally equivalent to AC\\^0. Moreover, for the non-uniform class of transformers, transformers without pause token are strictly less expressive than its counterpart with polynomial pause tokens; for the uniform case and for a fixed length $l$, an $l$-layer transformer without pause token is strictly less expressive than its $l$-layer counterpart with quasi-polynomial pause tokens.\n\nFor logarithmic precision transformers, the authors show it is equivalent to TC\\^0 when allowing polynomial number of pause tokens, although the separation for logarithmic precision transformers with and without pause tokens still remains open.\n\nFinally, the authors conduct experiments on learning parity function, and show it is easier to learn with pauses tokens and appropriate hints during training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1.\tThis is the first theoretical work studying the benefits of pause tokens. Since pause token is empirically showed useful, it is important to (theoretically) understand its benefit. \n2.\tThe paper provides strict separation for constant precision transformers with and without pause tokens. For log-precision transformers, the paper shows it’s equivalent to $TC\\^0$.\n3.\tThe paper is well written and easy to follow.\n\n**Weaknesses:**\n\n1.\tFor logarithmic precision transformer, the paper does not show a separation between “no pause token” and “polynomial pause token”. (although it is acceptable to me since the lower bound of $TC\\^0$ is still open and might not be easy to improve)\n\n2.\tIt would be interesting to investigate more in section 5. Currently, the authors only compare the test accuracy among three different settings. It would be interesting to know their respective underlying mechanisms and see why those different mechanisms lead to performance difference (does not necessarily need to be theoretical analysis since analyzing training dynamics theoretically could be much more complicated and might be beyond the scope of the current paper). This could provide a more solid evidence of the benefits of using pause token for certain TC\\^0 tasks.\n\n3.\tIt might also be helpful to add some discussions of the comparison of pause tokens and CoTs. In particular, it could provide a clearer picture of the theoretical gain of purely increasing the computation space (or “width”) vs allowing step-by-step intermediate computation.\n\nQUESTIONS:\n1.\tIn Figure 2, for the causal and non-causal settings, there seem to be thresholds for each setting where the model has perfecta test accuracy below the threshold and collapse to random guess when exceeding the threshold. Also, the value of the threshold is not very small (~100 or 200). Do authors have any intuition about the value of the threshold given a fixed configuration of the transformer class?\n\n2.\tThere could be two possible explanations of the results of Section 5. One is the performance gap is due to expressivity with and without pause tokens. The other is they might have same expressivity but the one with pause tokens requires much fewer samples to learn. (Similar to what authors mentioned at the end of the main text as “expressivity vs learnability). Do authors have any (indirect) proof of which explanation might be correct?\n\nLIMITATIONS:\nSee weaknesses and questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the expressivity of transformer when the input is extended by a sequence of blanks (\"pause tokens\"). Since positional encoding is applied to these additional tokens, the resulting transformers are more expressive. To be more precise, the authors consider Boolean functions in AC0 and TC0, well known classes defined via sequences of circuits. I give a rough description of how these classes are defined. A Boolean function is treated as a sequence of functions f_1, f_2,... (where f_n takes inputs of length n). We say it is in the class if there is a sequence of circuts, satisfying some conditions, that can be generated in uniform, space-efficient way, such that n-th circuit computes the n-th function. \nIn particular, the authors show that transformers with constant precision and poly number of pause tokens are equivalent to AC0 and allowing logarithmic precision gives equivalence to TC0. So for instance for a sequence of functions from AC0 or TC0 there exists a sequence of transformers, satisfying the corresponding conditions, such that n-th transformer computes the n-th function and there is a uniform, space-efficient way to generate these transformers.\nWhat's important is that the addition of pause tokens increases expressivity in the sense described above. This happens because the authors assume positional encodings that are dependent on the input length and then for a given input length, they can take a corresponding circuit and hardcode it inside these positional encodings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Expressivity of transformers is an important theoretical topic of practical relevance. Papers in this area have been presented at NeurIPS before.\n- The presented results are novel and technically sound. The proofs seem correct.\n\n\nWeaknesses:\n- This particular contributions is probably of very limited practical relevance. While there is some sort of uniformity in this setting, it is not the kind of uniformity that is relevant from the practical point of view, i.e., having a single transformer which is capable of computing the function on inputs of arbitrary length (perhaps even with length-dependent positional encodings). Indeed, the authors give some experimental results for parity, but these are not very exciting, since generally the problem with training transformers for parity is more about length generalization and not training on fixed length inputs. That being said, it is plausible that this work will have practical impact by motivating the other authors to extend this setting to a more practical scenario.\n\nQUESTIONS:\nQ1: Can you say something about the situation where we have a single transformer for all input lengths and add pause tokens? A partial answer to this question can be found in your proofs but perhaps you have some additional insights.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper provides a theoretical investigation of the empirically observed phenomenon that adding \"pause tokens\"  to the input of the Transformer can improve its performance. The authors analyze the expressive power of constant-depth Transformers through the lens of circuit complexity. The paper proves that constant-precision, logarithmic-width Transformers with a polynomial number of pause tokens are computationally equivalent to the circuit class AC0. With logarithmic precision, they are equivalent to TC0. The authors demonstrate that a 2-layer causally masked Transformer, which struggles to learn the parity function, can successfully learn it when provided with pause tokens and appropriate training hints.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- This work is the first one to study the expressive power of Transformers with \"Pause Tokens\".\n- This paper is well-written and structured. It clearly motivates the problem and states its contributions.\n\n## Weaknesses\n**Gap Between Theory and Practice**: The theoretical results rely on adding a polynomial number of pause tokens to simulate arbitrary circuits. However, the empirical works that motivate this paper (e.g., Goyal et al., 2024) see benefits with a much smaller, often constant, number of tokens. The paper does not fully bridge the gap between its theoretical requirements and these practical observations.\n\n**Limited Empirical Scope**: The experiments are confined to the parity problem. While this is a canonical benchmark in complexity theory, it is a synthetic task. It remains an open question how these findings translate to the more complex natural language and mathematical reasoning tasks where pause tokens have been shown to help.\n\n**Incomplete Separation for Log-Precision**: The paper does not prove a strict separation between logarithmic-precision Transformers with and without pause tokens (TF[L,L,0]⊊TF[L,L,P]), as this would require solving open problems in circuit complexity. While the authors are transparent about this limitation, it means the results for the log-precision case are less conclusive than for the constant-precision case.\n\nQUESTIONS:\nSee the Weaknesses part above.\n\nLIMITATIONS:\nThe authors discuss the limitations of this work and point out future directions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAttention mechanisms and transformer architectures have achieved significant advancements in natural language processing and computer vision, forming the foundation of many recent large-scale models. Incorporating intermediate reasoning steps and chain-of-thought prompting has led to improved performance across tasks such as mathematical reasoning and question answering. Recent studies have found that inserting intermediate, seemingly meaningless tokens—pause tokens—can also improve model performance, though this effect remains unclear within the COT framework. This paper provides a theoretical explanation for the benefit of pause tokens in enhancing transformer expressivity, grounded in circuit complexity. Specifically, it proves that 1) constant-precision transformers with a polynomial number of pause tokens match the expressivity of AC0 (constant-depth circuits with and/or/not gates), and 2) logarithmic-precision transformers with pause tokens correspond to TC0 (AC0 with threshold gates). These results support the conclusion that 3) pause tokens increase transformer expressivity. The theoretical claims are empirically validated on a parity benchmark, building on prior findings that parity is hard for transformers without pause tokens. Results show that pause tokens enhance performance on this task and enable causally masked transformers to learn globally dependent functions.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n-- While transformers have achieved remarkable success across various domains, their theoretical foundations remain limited. This paper offers a theoretical analysis that clarifies the non-obvious reasons behind the performance gains observed when pause tokens are introduced. \n\n-- Explaining the benefit of adding pause tokens to improve transformers’ expressivity through the lens of circuit complexity is both novel and non-trivial.\n\n-- The results presented in Figure 1 and their analysis show the advantages of pause tokens on enhancing the transformer performance when the sequence length increases. This is particularly interesting because the transformers with pause tokens stay absolutely stable while other versions drop substantially. This shows the advantages of capturing global information once the pause tokens are introduced. \n\nWeaknesses\n\n-- The paper may be challenging to understand for a general audience due to its highly technical and theoretical nature. This could be improved by incorporating concrete examples, visualizations, and other intuitive elements to make the content more accessible and engaging for non-expert readers.\n\n-- Although the abstract and introduction highlight the advantages of using pause tokens in QA and mathematical reasoning tasks, the experimental evaluation is limited to the parity benchmark. Expanding the evaluation to include additional datasets in QA and mathematical reasoning would strengthen the connection between the theoretical insights and real-world applications.\n\nQUESTIONS:\nTransformers have been used in many domains such as NLP and Computer vision. Could you please provide some tasks in various domains that can benefit from pause tokens while being tied to the proposed theories in this paper?\n\n\nHow does increasing the number of layers affect the number of pause tokens required and accuracy?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}