Title: Atom of Thoughts for Markov LLM Test-Time Scaling
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM Reasoning

Reasoning Framework
Abstract: Large Language Models (LLMs) achieve superior performance through training-time scaling, and test-time scaling further enhances their capabilities by conducting effective reasoning during inference. 
However, as the scale of reasoning increases, existing test-time scaling methods suffer from accumulated historical information, which not only wastes computational resources but also interferes with effective reasoning. 
To address this issue, we observe that complex reasoning can be achieved by solving a series of independent and self-contained subquestions. These subquestions are essentially \textit{atomic questions}, exhibiting the memoryless property similar to Markov processes.
Based on this observation, we propose Atom of Thoughts (\our), where each state transition consists of decomposing the current question into a dependency-based directed acyclic graph and contracting its subquestions, forming a simplified question that maintains answer equivalence with the original problem. This answer preservation enables the iterative \textit{decomposition-contraction} process to naturally form a meaningful Markov reasoning process.
Furthermore, these atomic states can be seamlessly integrated into existing test-time scaling methods, enabling \our to serve as a plug-in enhancement for improving reasoning capabilities.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Atom of Thoughts (AoT), a reasoning framework that leverages Markovian transitions to minimize historical dependencies during inference. There are two key components of the AoT framework: (1) two-phase transition where the system first decomposes the problem before solving it; (2) DAG-guided transition: when transforming the temporary DAG structure into the next Markov state, the method prunes nodes without incoming edges and reformulates it into an answer-equivalent independent question. Experimental results show that AoT consistently outperforms baselines as computational budgets increase, which provides a potential path of inference-time scaling.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The proposed method provides a novel approach for leveraging the structure information for reasoning task. While language models simplify many problems into next token prediction, it is interesting to see adding back the structure could help language models to reason and it also has the good property of modular integration as discussed in the paper.\n2. The baseline comparison is comprehensive. Figure 3 is also informative as it shows the potential of AoT for inference-time scaling. Besides the aggregated performance, I am wondering where there exists patterns of what we could get out of scaling with AoT. For example, after giving the framework more compute, what types of problems get resolved and why other methods fail to resolve them as the compute scales.\n\nWeaknesses:\n1. It is unclear how the two-phase transition is exactly executed. According to my understanding, it is using language model to generate the DAG. As this step is complicated, I am wondering what the accuracy is for this step.\n2. One assumption of AoT is that for each question $Q_0$, we can construct a sequence of subquestions that have equivalent answers but monotonically reduces in complexity. As the datasets experimented in this paper all have a short gold answer, I am wondering whether the method could be generalized to other reasoning scenarios, like writing complicated code, or writing mathematical proofs.\n\nQUESTIONS:\nBesides questions raised in \"Weaknesses\", I have the following suggestion:\n\nWhile Figure 1 (b) is helpful for understanding the proposed method, it would be great if the paper could include a step-by-step example in the Appendix to help readers go through the process of DAG construction and reformation. As a reader, I found the process is conceptually intuitive but still somewhat complicated to apply in LLM reasoning domains.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the \"Atomic of Thought\" (AoT), a new way for large language models (LLMs) to reason that intends to improve testing scalability by reducing the need for historical context.  AoT models the reasoning process as a Markov chain, which is different from other methods like Chain-of-Thought and Tree-of-Thought. Each \"state\" in the chain is a self-contained, equivalent restatement of the original problem, with reduced complexity. Through a large number of experiments on mathematical, coding, and question-answering benchmarks, AoT consistently demonstrates superior performance compared to existing methods. Moreover, this paper has a unique finding, the \"emergent\" property of atomic reasoning, where complex problems are progressively decomposed into indivisible, self-contained units.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Modeling reasoning as a Markov process of answer-equivalent states is an effective approach. This solves the important issue of context explosion and redundant computation in test-time scaling, which is an important issue for complex LLM inference. \n  -The suggested AoT framework gives a reasonable and useful answer. A probabilistic formulation and a real two-phase (Decomposition-Contraction) implementation define the methodology. The extensive experimental evaluation well supports the AoT's effectiveness, includes a wide range of tasks, several state-of-the-art LLMs, and a full set of strong baselines. The submission's claims are also backed up by the inclusion of detailed ablation studies and scaling analyses, such as integration with ToT/FoT.\n- This submission is generally well-organized. \n\n**Weaknesses**\n\n- This submission makes strong claims, such as that \"answer-equivalence\" will be kept, but it doesn't explain how this will be checked or enforced.\n- This submission says that \"atomic reasoning\" is a new property that was found when the process was scaled up. However, calling it the discovery of a \"latent granularity\" might be too much. Another way to look at it is that the system is reaching a logical endpoint, beyond which breaking it down further is not useful or possible for the given problem structure. \n- AoT seems to work best when the prompts for the decompose and contract functions are well thought out. Giving the prompts is good for making sure the results can be repeated, but it also raises problems about whether the framework is sensitive to how the prompts are worded. Further discussion is needed on how well AoT can handle changes to these master prompts to improve the work. It would also be interesting to see if it could be added to the prompt tuning framework, just like [Dspy](https://dspy.ai/).\n\nQUESTIONS:\n1. The author notes that the chain's memoryless property makes it vulnerable to poor transitions. What is the failure rate of these transitions, and what are the predominant modes of failure?\n2. AoT introduces new computational costs, including calls for decomposition, contraction, and quality-aware termination using an LLM-as-a-judge. Could you provide more details on the computational cost?\n3. Could you discuss what degree the AoT framework demonstrates sensitivity to the specific prompts utilized for deconstruction, contraction, and evaluation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a prompting framework, which interatively instructs an LLM to (1) reason over a question, (2) decompose its reasoning path into sub-thoughts and offers their dependency structure, and (3) generate the simpler question by reflecting on the generated structure.\n\nSTRENGTHS AND WEAKNESSES:\n### **Strengths**\n\nThis paper discusses a paradigm of problem solving in LLM, which is to progressively simplify a complex problem as a whole step by step. The core concept makes a lot sense to me, and seems to be rarely addressed in exsiting literature. The core concepts, practical experiences, and some technical approaches presented in this paper, may offer some insights to the research community engaged in tasks that involving similar problem-solving frameworks.\n\n### **Weaknesses**\n\n#### **W1: Significant Presentation Issues**\n\nThe overall organization and some detailed presentations of this paper makes it difficult to follow. Just to list a few points:\n\n-   Figures\n    - Too small font size for all figures.\n    - Confusing presentation for Figure 1 and 2. I failed to grasp the x-axis meaning in Figure 1 and the logical flow in Figure 2 until comprehending the core methodology.\n-   Method\n    - The intro section fails to outline the general framework of the proposed method, instead listing unexplained concepts.\n    - The basic and overall instruction detailed in Appendix C should be given in Section 3.1. I fully understand and respect the author(s)' efforts to keep the mathematical rigor, but I think giving certain intuition is not harmful.\n-    Organization\n     - Section 3.2 is confusing. It contains both addtional tricks and analysis, some of which don't fall under the title \"Emerged Atomic Reasoning\".\n     - So is Section 4.4.\n\n#### **W2: Unsupported Claims**\n\nThis paper gives unsupported claims. For example:\n\n> These structured explorations reveal a statistically supported phenomenon: deeper reasoning states tend to converge into irreducible forms, maintaining a stable and relatively low reasoning token count, from which the original problem’s answer can be directly inferred with high execution stability.\n\nThe \"structured explorations\" and statistics are missing.\n\n> Due to AOT ’s design and termination mechanism, longer chain lengths increase the performance ceiling without linearly increasing costs.\n\nI don't think the existing results can support \"increase the performance ceiling\".\n\n> In practice, this can also be query-specific based on the current problem; that is, we can add an additional LLM invocation to calculate the depth of the DAG as an estimate of a theoretically optimal maximum number of transitions.\n\nThe practical experiment is missing. And it's debatable whether the LLM estimation can be viewed as \"theoretically optimal\".\n\n#### **W3: Missing Important Analytical Details**\n\n- One of the paper's main contribution is the design to instruct LLMs converting a reasoning path into a DAG. However, the quality of DAG generated is not discuessed, making the result not interpretable and less reliable.\n- the choice of transition hyperparameters as 3 is not supported. According to Appendix E, it seems that a larger number may be a better choice.\n- The proposed method uses LLM-as-a-Judge as a important trick, but the statistics of it is missing. For example, how often does the judge interupt the process?\n\nQUESTIONS:\nplease refer to **Weakness** section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces \"Atom of Thoughts\" (AOT), a new reasoning framework for Large Language Models that uses Markovian processes to reduce computational overhead during test-time scaling. Unlike existing methods like Chain-of-Thought that accumulate extensive historical information, AOT decomposes complex reasoning into self-contained \"atomic\" units that don't rely on previous context, making inference more efficient. The approach can be integrated with various existing reasoning methods and demonstrates consistent performance improvements as computational budgets increase.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Different from previous decompositions, this paper defines a memoryless state in a Markov process, where each state only relies on the one step before. The decomposition requires that each sub-question preserve answer equivalence with the original question. AoT uses a two-phase transition to formulate such a memoryless Markov process, which first creates a DAG and then transfers the DAG to \nanswer an equivalent independent question. To ensure the equivalence, it uses LLM-as-a-judge to evaluate the state transition. \n* The paper provides comprehensive experiments to demonstrate the effectiveness of AoT in different tasks and settings. Experimental results on various tasks (math, code, and QA) show that AoT can improve the model performance in different domains. It also includes the integration of AoT to other reasoning frameworks and the investigation of the scaling law from different aspects. \n\n\nWeaknesses: \n* The motivation is not very clear to me. The construction of memoryless state Qi is based on the DAG. If there is a correct DAG decomposition of the original problem Q0, why not solve the problem directly following the dependencies in this DAG? Since each node in this DAG is an individual thought or sub-question, it should be easier to solve.  However, AoT chooses to convert the DAG to a new answer-equivalence question and solve it, which may lead to unnecessary cost by recreating a similar graph structure again (as in ToT and GoT).\n* The performance improvement over the baselines is not significant on MATH, GSM8k, MBPP, and AIME in Table 1. \n* The reliability of the answer-equivalence of the state transition depends on LLM-as-a-Judge. In other words, the LLM-based judge is used to check the correctness of the new conditions (independent sub-problems or thoughts). Therefore, the baselines such as FoT and ToT should also use the same LLM-as-a-Judge as the reward function. \n\nOthers:  \n* In Table 1,  AFlow's performance based on DeepSeek-V3 on GSM8k is the highest and should be bolded. \n* The font of legends in Figures 3 and 4 is too small.\n\nQUESTIONS:\nDoes the cost in Figure 4 include the LLM-as-judge and the construction and transition of AoT?\n\nLIMITATIONS:\nAuthors have discussed their limitations and social impact in the paper", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}