Title: Compositional Reasoning with Transformers, RNNs, and Chain of Thought
Venue: NeurIPS 2025
Accepted: yes
Keywords: learning theory

expressiveness

transformers
Abstract: It is understood that different neural network architectures are suited to different tasks, but is there always a single best architecture for a given task? We compare the expressive power of transformers, RNNs, and transformers with chain of thought tokens on a simple and natural class of tasks we term Compositional Reasoning Questions (CRQ). This family captures multi-step problems with tree-like compositional structure, such as evaluating Boolean formulas. We prove that under standard hardness assumptions, *none* of these three architectures is capable of solving CRQs unless some hyperparameter (depth, embedding dimension, and number of chain of thought tokens, respectively) grows with the size of the input. We then provide constructions for solving CRQs with each architecture. For transformers, our construction uses depth that is logarithmic in the problem size. For RNNs, logarithmic embedding dimension is necessary and sufficient, so long as the inputs are provided in a certain order. For transformers with chain of thought, our construction uses $n$ CoT tokens. These results show that, while CRQs are inherently hard, there are several different ways for language models to overcome this hardness. Even for a single class of problems, each architecture has strengths and weaknesses, and none is strictly better than the others.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper investigates the expressive power of transformers, recurrent neural networks (RNNs) and transformers with chain of thought (CoT) for Compositional Reasoning Questions --- which they formally define as tree structured hierarchical problems. They investigate the trade-offs between the number of nodes and depth of a CRQ, and the architecture requirements (number of parameters, runtime and parallel-runtime) for Transformers, RNNs and CoT.  They provide the following results:\n\n1. The provide a sufficiency criterion on a Transformer's depth, embedding dimension for solving a CRQ (Theorem 4.1)\n\n2. Any constant‑depth transformer, no matter how wide, fails on some CRQs unless (TC^{0} \\neq NC^{1}).\n\n3. Role of ordering of the nodes of a CRQ on the size of RNNs hidden dimensions required to solve the task.\n\n4. They show that under correct ordering of nodes, shallow RNNs can solve CRQs\n\n5. They show that a 2-layer transformer (with CRQ size dependent embedding sizes and precision) with CoT can solve CRQs.\n\nSTRENGTHS AND WEAKNESSES:\nS1[Originality]. A key strength of the paper is the formalization CRQs as a well-defined compositional task (although, I am slightly ambivalent on this, also check W3).\n\nS2[Clarity and Quality]. Quite intuitive and natural results that are presented in a well-structured format. The paper clearly outlines the necessary and sufficient conditions for Transformers and RNNs to express CRQs. I think fair amount of intuition is provided for the results. I did not check the proofs in the appendix. \n\nW3 [Significance]  Although I appreciate the CRQ formalization and the presented results. I am not sure that the paper's results actually motivate any guidance for practitioners, or explain any observed behavior in transformer design. Furthermore, the problem does not have a very natural motivation and the proof ideas (at least as discussed in the paper) do not seem to be of major impact on other areas.\n\nQUESTIONS:\nQ1. Do you think that your results provide any meaningful guidance to practitioners for specific applications? If yes, then such applications should be highlighted.\n\nQ2. I am not a circuit complexity expert, but why is the result of Thm 4.3 conditional on TC$^{0} \\neq NC^{1}$, when you seem to have a clear counter example (line 239)?\n\nQ3. Are CRQs generalization/restriction of some other problems investigated in theoretical CS?\n\nLIMITATIONS:\nThere is no discussion of limitations. And I think some discussion of the sort should be there, as at the moment I do not see why the results are relevant to any potential application or how are they connected to other natural problems investigated in Theoretical CS.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper inspects RNNs, Deep Transformers, and shallow Transformers with CoT tokens in the context of Compositional Reasoning Questions (CRQ).\n\nCRQs are defined as a tree representation of a complex question with each sub question being a child node. Each node as a vectorial representation.\n\nThe authors show that transformers with depth $L$ can solve all CRQs of depth up to $L$; RNNs with O(log n) hidden dimension and constant depth can solve all CRQs of size $n$; and transformers with O(n) CoT tokens can solve all CRQs of size $n$.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\nThis paper provides theoretical analysis to better understand the representation capacity of Transformers, RNNs, and Transformers with CoT tokens.\n\nThis is an important research topic to better understand and interpret Transformer architectures.\n\n\n## Weaknesses\n\n1. The paper provides theorems, proofs, and intuitions, but no experiments to validate the work. Experiments with Transformers and RNNs of a given size must be run on problems of small to large sizes to validate all the theory presented in this work.\n\n2. CRQs are defined in the paper but no dataset is presented as examples. Presenting and releasing a dataset that follows the CRQ principles would greatly improve the quality of this work. As an example of an already existing reasoning Q&A dataset that follows a graph structure, the authors can have a look at CLUTRR by Sinha et. al, 2019 which evaluates RNNs and Transformers like this paper.\n\n3. The distinction between Transformers and Transformers with CoT tokens is not clearly motivated. These are the exact same model architecture. The only difference is the number of input / output tokens. Given that the paper is motivated by exploring the strengths and weaknesses of neural network **architectures** for a class of problems; I would encourage further discussion why this split is necessary.\n\nQUESTIONS:\nIn Table 1, the caption says that “_Each architecture minimizes one kind of resource at the expense of the others_”, however, in the RNN row, the minimal resource is the number of parameters (with O(log n)), similar to the CoT row;  and not the runtime ( with O(n . log n)) like what is highlighted in green in the table.\nThe caption is somewhat confusing, it could maybe be rephrased to something like “_each ressource has an architecture that will minimize its usage_”?\n\nLIMITATIONS:\nNo societal impact as this is theoretical work. The main limitation of this work is the lack of experiments and experimental analysis.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}

comments: SUMMARY:
The paper provides theoretical analysis of transformers and RNNs in solving a subset of reasoning tasks which the authors named as Compositional Reasoning Question. The analysis shows some interesting insights but the proof is only for the ideal case and without experimental support.

STRENGTHS AND WEAKNESSES:
Strength:
1. The paper is in general written clearly with easy to follow usage of math notations.
2. The paper shares some insights on the theoretical grounding of transformer and RNN's ability to solve reasoning tasks that can be structured as binary trees. The paper illustrates why CoT can allow shallower transformer to generalize for the reasoning task.
Weakness:
1. Some statements are not substantiated. e.g. Line 168 "(though the lower bounds generally hold against models that
include them)". The authors also didn't explain why they pick transformers with single attention head that uses hardmax rather than softmax (presumably to simplify the proof?). 
2. The proof seems to be only for the ideal case. For example, the authors "use a different input format that encodes the tree structure directly into the positional encodings," but this in reality when the reasoning problem is natural language based and fuzzy, how well can the proof be extended to those cases?
3. While this is more of a theoretical paper, I think some experimental evidence such as results on some reasoning datasets will strength the paper more.

QUESTIONS:
None

LIMITATIONS:
None
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a formal problem class called Compositional Reasoning Questions (CRQs) to analyze the expressive power of different neural architectures. CRQs are problems with a tree-like structure, such as Boolean formula evaluation, that require multi-step, compositional reasoning. The authors theoretically compare the capabilities of transformers, RNNs, and transformers with chain of thought (CoT) on this task. They prove a series of complementary results showing that no single architecture is strictly best. Instead, there are fundamental trade-offs: deep transformers can solve CRQs in parallel but require depth that scales with the CRQ's tree depth; RNNs can solve them with constant depth but require a specific input ordering and a hidden state size that scales logarithmically; and constant-depth transformers with CoT can also solve them, but require a number of thought tokens that is linear in the problem size.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n\n* The primary strength of this paper is its novel and insightful comparison of major neural architectures on a single, well-defined class of problems.\n* The introduction of Compositional Reasoning Questions (CRQs) provides a simple yet powerful framework for studying an essential capability of language models: multi-step, hierarchical reasoning. By proving that this problem is NC¹-hard, the paper grounds the analysis in established complexity theory, providing a solid foundation for its claims.\n* The theoretical analysis is rigorous and comprehensive. I appreciate that the authors provide both positive results (constructions showing how a model can solve the task) and negative results (conditional lower bounds showing what a model cannot do) for each architecture. This dual approach gives a much more complete picture of each model's capabilities and limitations. The summary of these trade-offs in Table 1 is particularly effective.\n* The paper is exceptionally well-written. The motivation is clear, the CRQ problem is defined precisely, and the main results are stated upfront and then elaborated on in dedicated sections. The comparison with related work, especially Feng et al. [2024], is sharp and clearly articulates this paper's distinct contributions.\n* While the proof techniques themselves (e.g., reductions from circuit complexity, communication complexity arguments) build on prior work, their application to the novel CRQ framework to draw a three-way comparison between transformers, RNNs, and CoT is highly original. The resulting insights into the architectural trade-offs feel new and important.\n\nWeakness:\n* The main weakness lies in the gap between expressiveness and practical learnability. The constructions for solving CRQs rely on highly engineered mechanisms, such as specific positional encodings that perfectly capture the tree structure or a presorted input sequence for the RNN. It is an open question whether these solutions can be learned via standard gradient-based optimization, a limitation the authors rightfully point out in their future work section.\n* The analysis uses a hardmax attention mechanism and assumes finite precision, which are standard in theoretical analyses but differ from the softmax attention used in practice. A brief discussion on how these results might translate to the standard setting would have been beneficial.\n\nOverall, the strengths far outweigh the weaknesses. This is a strong theoretical paper that makes a significant contribution to our understanding of the fundamental capabilities of different language model architectures.\n\nQUESTIONS:\n* The constructions presented are elegant proofs of expressiveness, but they rely on very specific, pre-designed components like the tree-aware positional encodings for the transformer and the memory-rank sorting for the RNN. Do you have any intuition on whether these mechanisms, or functionally equivalent ones, could be learned by these architectures from data, or do you view these results as primarily characterizing the limits of what a pre-programmed network can do?\n* Your analysis uses a hardmax attention function. How do you expect your positive results (the constructions) to change if using a standard softmax? Could a low-temperature softmax approximate the hardmax behavior, perhaps at the cost of requiring higher precision, or would it fundamentally break the constructions?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}