Title: Knee-Deep in C-RASP: A Transformer Depth Hierarchy
Venue: NeurIPS 2025
Accepted: yes
Keywords: Transformers

Depth

Layers

Formal Language Theory

Temporal Logic

Expressivity
Abstract: It has been observed that transformers with greater depth (that is, more layers) have more capabilities, but can we establish formally which capabilities are gained? We answer this question with a theoretical proof followed by an empirical study. First, we consider transformers that round to fixed precision except inside attention. We show that this subclass of transformers is expressively equivalent to the programming language $\textsf{C}$-$\textsf{RASP}$ and this equivalence preserves depth. Second, we prove that deeper $\textsf{C}$-$\textsf{RASP}$ programs are more expressive than shallower $\textsf{C}$-$\textsf{RASP}$ programs, implying that deeper transformers are more expressive than shallower transformers (within the subclass mentioned above). The same is also proven for transformers with positional encodings (like RoPE and ALiBi). These results are established by studying a temporal logic with counting operators equivalent to $\textsf{C}$-$\textsf{RASP}$. Finally, we provide empirical evidence that our theory predicts the depth required for transformers without positional encodings to length-generalize on a family of sequential dependency tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the role of depth in transformer architectures by drawing a connection to a logical framework known as temporal logic with counting (TL[↼#]). The authors focus on fixed-precision transformers: all computations are performed with limited numerical precision, except within the attention mechanism. The core theoretical contribution is to show that these fixed-precision transformers are equivalent in expressivity to the programming language C-RASP, which is equivalent to the logic TL[↼#]. The equivalence is depth-preserving: the number of layers in the transformer corresponds to the nesting depth of the counting operators in the logic. Using this correspondence, the authors establish a depth hierarchy: deeper TL[↼#] formulas can define more complex sequence patterns than shallower ones. This implies that deeper transformers are more expressive than shallower ones. To support the theory, the paper presents empirical results showing that transformers require greater depth to solve formal language tasks as the complexity of the sequence dependencies increases. These results align with the theoretical depth requirements predicted by TL[↼#].\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n* The authors provide a mathematically grounded depth hierarchy theorem, building on logics with counting and extending previous results in a novel and robust way.\n* The equivalence between fixed-precision transformers and TL\\[↼#] is not only elegant but also preserves depth, which is central to the paper’s claims.\n* A careful experimental setup shows that the theory’s predictions about depth requirements hold up in practice, at least on synthetic language tasks.\n\n**Weaknesses:**\n\n* The results apply specifically to fixed-precision transformers without positional encodings. While this is reasonable in a theoretical sense, it limits immediate applicability to real-world transformer architectures.\n* The empirical validation is limited to formal language tasks. It remains unclear how these results translate to real-world NLP tasks.\n* The no-position-encoding assumption diverges from most practical models, which might limit the audience’s perception of the impact.\n\nQUESTIONS:\n1. Are there insights into whether width can compensate for depth (especially on these synthetic tasks), even if not in a fully expressive way?\n2. The authors note performance degradation at higher k values (e.g., $k \\geq 10$). Is this due to optimization difficulties, data sparsity, or theoretical limitations?\n\nLIMITATIONS:\n* The theoretical framework assumes fixed-precision computation and omits positional encodings. This diverges from common transformer use cases, which limits immediate applicability to current architectures.\n* The experimental validation is limited to synthetic tasks. While effective for proving the point, these tasks do not capture the complexity or variability of natural language.\n* While the authors prove depth separation for expressivity, they do not provide results regarding learnability or training stability at increasing depths.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses a simplified form of a transformer to study the relation between depth of the transformer (number of stacked transformer blocks) and depth in a formal language modeled by TL (temporal Logic) which is equivalent to C-RASP. The paper establishes a tight connection between depth of a transformer model and a programming language: C-RASP. Using the established connection, authors claim to characterize a family of transformers that can length-generalize.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: The paper presents a solid theoretical work, and proves a non-trivial result.\n\nWeaknesses: \n\nThe paper accumulates a large number of definitions and also heavily relies on objects defined in a few previous background papers (e.g. Chiang et al 2024) which are necessary for readers to follow the paper's arguments. It is therefore a hard to follow paper. Authors could make an effort in summarizing the background content required for the reader to read through the paper and only present the necessary. \n\nIt is also not clear which of the assumptions are made to make proof of theoretical results easier and which are made to improve the model / results. \n\nEmpirical experiments have some shortcomings. Authors seem to have run some more experiments during the rebuttal. I am uncomfortable with those results since they seem to have let the model train for longer / have slightly different hyperparameters.\n\nQUESTIONS:\n1. In empirical results, authors claim that the drop after 10 layers does not contradict their findings. I thought their findings were tight so I am surprised why this happens. Can you elaborate?\n\n2. Can you elaborate on how findings on TL generalize to other classes of logic?\n\n3. It is unclear to me if the choice of no positional embedding is to ease theoretical demonstrations or if it is something that also helps improve results. Having no positional embeddings can clearly be problematic in many cases. It would help to have a more clear explanation on this.\n\nLIMITATIONS:\nThe paper's language highly relies on notations and definitions defined in a narrow set of papers. It may be hard to follow for a general audience. Empirical results are limited and there is a gap there which is uncovered (see Question 1 above)", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nBuilding on prior work in C-RASP and temporal logic (TL[#]), the authors show that (a subclass of) transformers are equivalent to C-RASP and TL[#]-left. This subclass includes models where computations use fixed-precision rounding everywhere except within the attention mechanism. Based on this equivalence, the paper establishes a \"depth hierarchy,\" demonstrating that there are specific tasks deeper transformers can solve which shallower ones cannot. Finally, the authors provide empirical evidence showing their theoretical predictions on model expressivity align well with practice. They test this on a next-token prediction variant of the $L_k$ language recognition task.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\nThe paper is well-written, and the presentation is solid. The theoretical demonstration of a clear depth hierarchy under non-prohibitive constraints is a very interesting contribution to the broader community. I also appreciate that the authors provided empirical results to support their theoretical findings. This strongly demonstrates the practical applicability of their results and shows that the underlying theoretical assumptions were practical.\n\n**Areas for Improvement:**\nThe paper's primary weakness is its accessibility. While the results are interesting for a general audience, the work may be difficult to follow for those not already familiar with temporal logic or C-RASP. I understand that these concepts are built on previous work, but including more intuitive or illustrative examples of them would be very helpful. Adding a brief introductory section on these topics in an appendix could significantly improve the paper's clarity and open it up to a larger audience.\n\nQUESTIONS:\n1.  The work of Yang and Chiang (2024) seems to present similar results regarding the equivalence of C-RASP, TL[#]-left, and certain transformers. Could you elaborate on how your work differs from theirs? In particular, their work was not included in the comparison table in Section 1, and the equivalence of transformers and TL[#]-left, which appears to be their contribution, is presented as a contribution of the current work on line 29. Clarifying the distinction would be very helpful.\n\n2.  Based on the equivalence results you provide, could you elaborate on the depth required to learn tasks like parity and modular addition?\n\n3. Regarding the experiments, accuracy is shown to sharply decline for $k \\geq 10$, even for models with sufficient depth. You attribute this to challenges with learnability, not expressibility. Could you provide more insight into the specific learning challenges for these more difficult cases and speculate on what might be required to solve them?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to investigate the expressive power of a specific transformer, the future-masked fixed-precision transformer. The authors first establish an equivalence between this transformer and the programming language C-RASP, then prove a depth hierarchy for the language, demonstrating that deeper C-RASP models possess greater expressive power. Overall, the paper provides a clear understanding of how the expressive power of transformers varies with depth. The accompanying experiments are well aligned with the theoretical analysis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. This work establishes a connection between transformers of varying depth and a specially designed language with corresponding depth, making it possible to analyze and understand the expressive power of these models.\n\n2. The paper rigorously analyzes the expressive power of transformers with varying depths, accomply with solid theoretical proofs, provding insights for mechanisms interpretion and architecture design for practical transformers.\n\n3. The experimental results are consistent with the theoretical findings.\n\nWeaknesses\n\nThe analysis appears to overlook an important aspect of practical transformers: the residual connections between layers. Additionally, since positional encoding is not considered in this work, it may be challenging for the model to distinguish tokens at different positions, potentially limiting its expressive power.\n\nQUESTIONS:\nIn addition to the concern raised in the Weaknesses section, I have a question regarding the experiments. In some cases, the accuracy for the depth-(k-1) model remains quite high, while for depth-(k-2) and lower, the accuracy drops significantly. For example, in Figure 5, the \"Accuracy on [201,250]\" shows this. Why does the depth-(k-1) model perform so differently compared to models with lower depth? Could you provide some explanation or intuition for this observation?\n\nLIMITATIONS:\nAs noted in the Weaknesses and Questions sections, the omission of residual connections and positional encoding may impact the generality of the results. I am not very familiar with this research area, while the statement seems valid with sufficient experiemtns, I lean toward accepting this paper, if there are any misunderstandings in my comments, please let me know.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}