Title: When Can Model-Free Reinforcement Learning be Enough for Thinking?
Venue: NeurIPS 2025
Accepted: yes
Keywords: reinforcement learning

thinking

model-free

planning
Abstract: Recent work on large language models has demonstrated the use of model-free reinforcement learning (RL) to train reasoning-like capabilities. The emergence of "thinking" through model-free RL is interesting as thinking actions neither produce reward nor change the external world state to one where the agent is more likely to get reward. This paper seeks to build a domain-independent understanding of when model-free RL will lead to such "thinking" as a strategy for reward maximization. To build this understanding, we first introduce a theoretical model which we call a thought Markov decision process (MDP). Thought MDPs minimally extend the classical MDP model to include an abstract notion of thought state and thought action. Using the thought MDP model, we prove the importance of policy initialization in determining whether or not thinking emerges and show formally that thought actions are equivalent to the agent choosing to perform a step of policy improvement before continuing to act. We then show that open-source LLMs satisfy the conditions that our theory predicts are necessary for model-free RL to produce thinking-like behavior. Finally, we hypothesize sufficient conditions that would enable thinking to be learned outside of language generation and introduce a toy domain where a combination of multi-task pre-training and designated thought actions enable more data-efficient RL compared to non-thinking agents.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper developed a formal framework studying the role of thoughts in reward maximization. The paper proposes a model of \"thought MDP\", which augments the conventional MDP for RL modeling with thought states and actions. With this formulation, the paper shows that thought-augmented MDP improves policy and is more effective in maximizing rewards. The paper also shows that policy initialization is critical to the emergence of thoughts, and how the thought MDP helps language and non-language models to maximize rewards.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThis paper provides a rigorous formal mechanism that elucidates the role of thoughts in reward optimization of RL, which offers a solid theoretical contribution to the understanding of how COT + RL improves agent model's performance.\n\nWeakness:\nAlthough the paper's title is \"Is reward enough for thinking\", this paper actually touches several sub-topics (how thinking helps reward optimization, condition for thought emergence, language/non-language thought MDP etc.). This divergence makes the central argument somewhat difficult to follow. A clearer high-level summary and more structured line of reasoning would strengthen the paper's coherence.\n \nThe paper could also have compared existing theoretical analysis on the role of COT and latent thinking steps.\n\nQUESTIONS:\n1. While the paper provided analysis of how thinking improves policy, it remains unclear what conditions of the thinking steps need to satisfy so that policy can be improved and reward optimized. An agent's performance cannot be improved with arbitrary thought states and actions, it would be great to provide some theoretical characterization of the quality of thoughts.\n\n2. In some sense the \"thought MDP\" can be regarded as introducing a latent variable to the MDP process. It would be great if the paper could provide some comparisons between the proposed framework differ and existing analysis on RL with latent variables, e.g.\n\n\"Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model\", NeurIPS 2020\n\"Latent Thought Models with Variational Bayes Inference-Time Computation\", arXiv 2025\n\"Latent Variable Representation for Reinforcement Learning\", ICLR 2023\n\"A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration\", arXiv, 2024\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author extends standard MDP by adding a thought action space, such that at each step the agent may either act in an environment ($A$) or take a zero-reward \"think\" step ($C$) that deterministically updates an internal thought state ($\\tau$) while environment state (s) remains unchanged. Under these three assumptions:\n- thought transitions are deterministic,\n- all external rewards are non-negative\n- for any state there exists a policy that can eventually reach a non-zero reward with finite number of steps,\n\nthe authors prove:\n- a fully-optimal policy never thinks,\n- during learning a thought step is selected exactly when its discounted value exceeds the value of acting immediately\n\nThus, thinking emerges only when it raises expected returns enough to offset the discount. Experiment using 4-digit addition shows thought actions indeed improve expected return in LLMs as perplexity significantly reduced.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The formalization is clean and correctly derived. The paper gives a unified theory to explain conditions in which a bunch of test-time compute scaling strategies arise during RL training.\n- The proofs clearly separate optimal behavior from learning time, suboptimal behavior and show how reflection can shorten the effective horizon in sparse-ward tasks. This is particular useful to explain recent effort in trying to make model think more efficiently by applying discount during training.\n\nWeaknesses:\n- The LLM-based experiments is in a trivial domain (4-digit addition), and it's hard to see how it generalizes to more complicated tasks.\n- The paper does not provide a training recipe or concrete algorithm for training LLMs despite showing the value estimates as perplexity.\n\nQUESTIONS:\n- Could authors provide a training recipe along with an experiment showing thinking emergence during training? Using the multi-digit addition with a small LLM should suffice. \n- Many existing RL for LLM is implemented using token-level rewards where a single trajectory reward gets broadcasted to every position of the token. Does this work assume discount $\\gamma$ apply at the whole generation level as well?\n- What happens when there is no discount factor applied to thinking? $\\gamma=1$\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the emergence of thinking in Large Language Models (LLMs). Recent works have discovered that model-free reinforcement learning with special tokens can encourage LLMs to think (e.g., generating long chain-of-thought) and improve reasoning performance. To study this phenomenon, this paper models the reasoning process of LLMs as a Thought Markov Decision Process. Thus, reasoning is to learn the transition between the thought states. Furthermore, it develops a theoretical framework to show that initialization is important for the emergence of thinking. It further discusses the emergence of thinking in other domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n1. This paper studies how thinking emerges from LLMs, which is receiving increasing attention. Empirical findings reveal that model-free reinforcement learning can improve the reasoning ability of LLMs by teaching them to think. Although such a paradigm has achieved superior performance in mathematical reasoning and coding [1], the mechanism behind this emergence of thinking is unknown. Recent work [1] also questioned whether RL can improve the reasoning ability of LLMs. To the best of my knowledge, this paper is one of the earliest ones to study this problem from a theoretical perspective.\n\n2. To study the emergence of thinking, this paper established a Thought MDP and theoretically analyzed the importance of policy initialization to thinking. This finding aligns well with recent empirical results, as shown in [2].\n\nWeakness:\n\nMy major concerns are about the clarity and empirical evaluation. For example, what are the thought states and actions in the reasoning process? It is better to have some illustrative examples. Moreover, the evaluations in Section 5 are only conducted on small scale models (~1B). It would be better to evaluate on large-scale models.\n\n\n[1]. Deepseek-r1: Incentivizing reasoning capability in LLMs via reinforcement learning.\n\n[2]. Does reinforcement learning really incentivize reasoning capacity in LLMs beyond the base model?\n\nQUESTIONS:\nI am \\textbf{not} a theoretical researcher. Please address the weaknesses part above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper the authors explored why \"thinking\" (defined as agent actions that does not produce reward and does not modify the environment state) might emerge as part of model-free RL learning. The proposed a framework named Thinking MDP to ground their research. They provided theoretical results on 1) the benefit of thinking as a form of test-time policy improvement and 2) how policy initialization influences the whether thinking emerges. \n\nThe authors further provided toy example to demonstrate that 1) thinking could improve LLM's policy at test time, and therefore would be expected to emerge as a policy during RL, though the final optimal policy would not require thinking. And 2) some empirical evidence that multi-task pretraining is a core criteria for models where thinking could be useful (as a form of helping the policy navigate to learned sub-policies).\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- Well written paper.\n- Provided strong theoretical results on how thinking can be a viewed as a form of test-time policy improvement given suboptimal policy / value functions.\n- Provided interesting demonstration on their theoretical results.\n\nWeakness:\n- The paper stopped short of showing that \"some LLM can benefit from thinking\" necessarily implies the \"some LLM will get thinking ability through model-free RL\". The authors argue that if an optimal policy can be learned then thinking is necessary, but some additional research in realistic settings (RL on pretrained LLM) that thinking emerges and then goes away.\n- The paper did not provide much grounding on their claim that multi-task pretraining is key criteria for thinking. Do the authors mean that thinking is essentially a way to generalize from learned sub-tasks during multi-task pretraining? Some more exploration on this could be meaningful.\n- The paper hinted that thinking shares certain similarity with MCTS. This could be explored more in settings such as teaching a LLM to play a game such as Chess. \n- Some minor grammar mistakes.\n\nQUESTIONS:\n1. In Section 6, why does the thinking action still occupies 15% of final policy if Proposition 4 suggests an optimal policy doesn't need thinking?\n\n2. Suggestion: provide some empirical evidence on the emergence of thinking in LLMs as RL iteration goes on, similar to Figure 1 but in a real-life LLM setting. \n\n3. Provide more ablations on the effects of multi-task pretraining, for example if the pretraining tasks are not strictly \"subtasks\" for the RL task, what would happen.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work studies the question: \"when is it possible, from an RL theoretic point of view, to encourage 'thinking' steps by just pure RL?\" They formally define a construct called a \"Thinking MDP\" which is an MDP which incorporates \"thinking states\" and \"thinking actions\". The reward is discounted for each thinking or acting step taken, in accordance with standard MDPs, and is assumed to be non-negative. This formulation leads to several mathematical propositions: an optimal policy for a thought MDP never takes any thinking steps, but policy iteration algorithms may sometimes empower the model to take thinking steps if the expected reward is better under the current policy; and thinking MDPs may also have a smaller effective horizon (i.e., are \"less complex\") than their ordinary counterparts, under some cases. The authors attempt to validate their theory using some controlled experiments in LLMs and outside of LLM context using purely synthetic settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The Thought MDP formulation is interesting, and (potentially one of) the first attempts to connect thinking/reasoning in LLMs to RL theory.\n- The mathematical consequences of the formulation seem solid and interesting, albeit very connected with the specific assumptions made, which may not be realistic.\n- The synthetic experiments in non-LLM RL environments demonstrate the possible generalization of reasoning RL methodology outside of LLMs.\nWeaknesses:\n- The thought MDP formulation is not fully connected to LLMs. In reasoning LLM training, my understanding is that each action or thought is a full \"turn\", not just a single token, and this affects the reward formulation. Often, there is no discount factor, and in any case the discount factor often does not apply to thinking steps. In this sense, the reward formulation of the thinking MDP in this paper seems like it does not apply to the practice of LLMs.\n- The synthetic experiment with LLMs is also similarly heuristic, and in particular, does not actually apply RL to the model, so it is hard to show how the results in this experiment corroborate the theoretical results.\n- The theoretical assumptions, which seem not very applicable to practice, seem to underpin all the theoretical results, so these may not apply either or be slightly misleading when applied to practice.\n\nQUESTIONS:\n- How resilient are the theoretical results to the specific assumptions made? Is it possible to adapt the results to other assumptions?\n- How can we formalize the state/actions in terms of the LLM token generation process?\n- Can a modified RL procedure which uses the assumptions made in the paper actually do RL training of a fine tuned model? (I.e., instead of making theory fit the practice, you can instead introduce a new practical method which fits the theory and show that it does reasonably well.)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}