Reducing Belief Deviation in Reinforcement Learning for Active Reasoning of LLM Agents
Authors: Deyu Zou
Yongqiang Chen
Jianxiang Wang
Garry YANG
Mufei Li
Qing Da
James Cheng
Pan Li
Yu Gong
Venue: ICLR 2026 Oral (2026)
Keywords: Large language models
LLM reasoning
Agentic multi-turn reasoning
Primary area: foundation or frontier models, including LLMs

Abstract: Active reasoning requires large language model (LLM) agents to interact with external sources and strategically gather information to solve problems in multiple turns. Central to this process is belief tracking: maintaining an accurate representation of the underlying state and uncertainty in understanding and solving the problem. However, due to limited reasoning capabilities, LLM-based agents often suffer belief deviation: their internal beliefs drift from the true problem state, leading to loss of state awareness and uninformative or repetitive actions. Once this happens, errors compound in the trajectories used for reinforcement learning (RL), leading to misattributed credits and limited exploration. To address this issue, we propose to track belief deviation and develop $\mathbf{T^3}$, a simple yet principled method that detects excessive deviation and truncates training trajectories to suppress uninformative tail effects. Hence, $\mathbf{T^3}$ preserves credits for informative prefixes and systematically improves policy optimization. Across 5 challenging tasks, $\mathbf{T^3}$ consistently
enhances training stability and yields performance gains of up to 30 points while cutting token cost by up to 34%. These results highlight belief control as a key principle for building robust LLM agents capable of active reasoning.

Published as a conference paper at ICLR 2026
REDUCING BELIEF DEVIATION IN CEINFORCEMENT LEARNING FOR ACTIVE REASONING OF LLM AGENTS
Deyu Zou', Yongqiang Chen', Jianxiang Wang", Haochen Yang', Mufei Li', Qing Da James Cheng'; Pan Li Yu Gong²
'Tee Chinese University of Hong Kong, BayteDance, 'Georgia Institute of Technology (dyzou24 4, jcheng) cuhk. edu hk, panliggatech. edu, yuxiaofes @bytedance.com
ABSTRACT
Active reasoning requires large language model (LLM) agents to interact with exter- nal sources and strategically gather information to solve problems in multiple turns. Central to this process is belief tracking: maintaining an accurate representation of the underlying state and uncertainty in understanding and solving the problem. However, due limited reasoning capabilities, LLM-based agents often suffer belief deviation: their internal beliefs drift from the true problem state, leading to loss of state awareness and uninformative or repetitive actions. Once this happens, emrors compound in the trajectories used for reinforcement learning (RL., leading to misattributed credits and limited exploration. To address this issue, we propose track belief deviation and develop T simple yet principled method that detects excessive deviation and truncates training trajectories to suppress uninformative tail effects. Hence, T preserves credits for informative prefixes and systemati- cally improves policy optimization. Across challenging tasks, T consistently enhances training stability and yields performance gains of up to 30 points while cutting token cost by up to 34%. These results highlight belief control as a key principle for building robust L.M agents capable of active reasoning."
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable reasoning capabilities across diverse domains (Huang& & Chang. 2022; Plaat al., 2024; Li et al., 2025b), further advanced by reinforce- ment learning (RL) with outcome rewards (Wang et al., 2024; Srivastava & Aggarwal, 2025; Xu al., 2025; Guo et al., 2025: OpenAI, 2025: Team al., 2025). Recently, along with the increasing agentic applications of LLMS (Zhang et al., 2025a; Plaat et al., 2025), the community seeks to extend the success of RL long-horizon and multi-turn reasoning (Wu al., 2025; Laban al., 2025; Li et al., 2025a). A key capability of LLM agents in multi-turn reasoning is active reasoning where the agent must strategically raise questions and actively acquire missing information to complete the task through multi-turn interactions with the external environment (Zhou et al., 2025; Badola et al., 2025).
However, LLM-based agents often struggle in multi-ture turn and active reasoning settings: as interactions unfold, they generate redundant, irrelevant, or uninformative actions (Yuan et al., 2025; Fu et al., 2025; Zhang et al., 2025b), and may even collapse into unproductive loops (Zhou et al., 2025). Moreover, RL training alone does not fully resolve these issues. Empirically, the learned policies can still yield globally suboptimal outcomes (Wang et al., 2025) or exhibit poor robustness to unseen tasks (Zhang et al., 2025b). Hence, it raises intriguing research question:
Why do LLM agents get trapped active reasoning, and how can we mitigate it?
To answer the question, we start by modeling active reasoning as a Partially Observable Markov Decision Process (POMDP). Classical POMDP formulations assume perfect belief estimate (e.g..
"These authors contributed equally to this work.
*Corespooning Authors. "Our implementation is available httpp://githhb..co wtwp:tttww...oommmmmmm..eee/

---

Published as a conference paper at ICLR 2026
A naideed as ancike taa was rocored fira muster was saily sf, Squngds, exsety
Area wasted andfer copettdd mundrr war puily Sentgels the meceted rear lolklt reactly like lin, How could this be?
Og 0:
TM the ron bave in steviial tei who was ovectted. allowing las to wilsow Inc coccuiio??
Og
Dad th luve iicetiill bean wia wis eseezhal Elkowing hin to wilnes the execatiol
Yes.
Yos
b
ag
O
Earty Truncation
Dhc the man muspuaaee eválece and befiancties to frame bis akonical twis
Eid coocutiin dnvolve dee bis warch
6
Ogy
R2
Rollout Generazion
Again! Yes...
Credit Assignment
Yes!
BTR Entry
Vanilla Method
Ours: Truncating Belief-Tropped Trajectories
Figure I: Overall framework of T where (br, as, 0r) denote the agent's internal belief, its chosen action, and the resulting feedback at turn L, respectively. By truncating belief-trapped trajectories, we prevent the agent from entering the belief-trap region (BTR) where credit assignment is contaminated in RL training, allowing learning signals to concentrate genuinely informative actions. As result, policy optimization becomes more stable and effective under complex active reasoning.
via Bayesian filtering) conditioned on past observations (Kaclbling ct al., 1998). In contrast, when instantiated with LLM agents, belief tracking must be approximated by the model itself, which is inherently imperfect due to the limited reasoning capabilities of LLMSs. Under mild assumptions, we show that such imperfect belief updates can lead the rollout trajectories into a Belief-Tray Trap Region (BTR, Def. 1), where actions cease to be informative, errors accumulate, and the reasoning progress stagnates (Thm. 1). Moreover, we show that standard policy optimization will be systematically misled by such belief-trap dynamics: once trapped, the uninformative tail of the trajectory can contaminate the credit assigned crucial early-stage actions, and even invert their estimated gradients (Thm. 2). therehy hindering effective exploration and leading to suboptimal policies.
To mitigate belief-trap dynamics, we propose T (Truncating Belief-Trappe Trajectories), a simple yet principled method that halts trajectories upon detecting entry into the BTR. Since the exact onset of the BTR is intractable in LLM agents, we introduce the T condition (Def. 2), a theory-grounded criterion that characterizes entry into the BTR. In practice, this condition is instantiated via observable proxy signals within the reasoning trace. Empirically, we find that even simple signals, e.g., redundant queries, proved to be effective indicators of belief trapping. By truncating the uninformative tail, T3 preserves the credit assigned to informative prefixes, resulting in lower-variance and less-biased gradient estimates (Cor. 1). Owing to its simplicity, T can be seamlessly integrated into standard policy optimization frameworks (e.g., PPO, GPRO, and GSPO) without altering the underlying algorithm, providing a practical drop-in solution to the credit assignment problem.
We evaluate T on 4 datasets and tasks from recent challenging active reasoning benchmarks, including AR-Bench (Zhou al., 2025) and Multi-Turn Puzzles (Badola et al., 2025). Across all settings, T consistently improves training stability. token efficiency, and final performance, achieving gains of up to 30 points while cutting rollout tokens by up to 34%. further shows robust benefits across LLMM sizes, architectures, and even under out-of-distributio scenarios. These results demonstrate that controlling belief traps not only systematically improves policy optimization but also provides a principled path toward building reliable active reasoning agents.
2 REINFORCEMENT LEARNING FOR ACTIVE REASONING
2.1 THEORETICAL FORMULATIONS
Due to space limits, in this section, we will state the necessary setup to derive our theoretical results and leave the details to Appendix B. To strengthen the connection between theoretical analysis and the practical behavior of LLM- agents, we conduct empirical studies that directly examine the key theoretical components and summarize the findings in Appendix C (an overview in Fig. 2).
2

---

Published as a conference paper at ICLR 2026
(a) Qwen-2.5-7B
(b) Qwen-2.5-333 2.5-22
(c) CD
(d) PE
Figure 2: Overview of empirical verification for key theoretical components (details in Appendix C). (a)(b) Empirical lower-bound fitting for Asmp. 1. We visualize the fitted lower bound (red line)
Eo = mo V0 co. over the region V 2 Un (vertical dashed line) for the PE task across Qwen-2.5-7B and 32B models. Both models exhibit a clear positive lower-bound slope. (c) Empirical validation of advantage drift (Thm. 2). We report token-wise mean GAE values on failed rollouts for the CD and PE tasks (Qwen-2.5-7B) comparing without vs. with T In both tasks, early-token advantages display negative drift, and this drift is attenuated under T consistent with Cor. 1.
We model active reasoning as a Partially Observable Markov Decision Process (POMDP) (Kaelbling et al., 1998) (8, A, O, T, O, R, 7), where S is the space of latent states, A the action space, O the observation space, T the transition dynamics, O the observation model, R the reward function, and the discount factor. At each step, the agent selects action (question) a, E AA based on a belief state by E (S), i.e., a distribution over latent states summarizing the interaction history. Note that the true latent state 8* E S is unobservable the agent (s* is introduced solely for theoretical analysis); for analytical clarity, we assume s" is fixed within an episode. The environment returns an observation or E via (- s*, at), and the agent updates its belief bi+1 accordingly.
Belief' Updates. We consider two agentic reasoners operating under the same interaction protocol but differing their belief-update mechanisms: an oracle reasoner and an imperfect LLM reasoner. The oracle maintains an oracle belief bū and updates it via the Bayesian operator B*:
bit1 (b) = (b;,,, 0t, oz) O(o, 8, (g)
Po(0; aa)
where (o(0;) aa)) := Ewes O(or | 8', as (6( is the Bayes normalizer. In contrast, the LLM agent maintains I.IM belief be (its internal estimate of the latent state) and updates it through potentially imperfect rule Bg, where 0 denotes the LLM parameters.
Task Progress. We analyze how the agent's belief updates influence task progress during interaction. To quantify progress, we introduce a truth-anchored potential function (b) := log b(s which measures the negative log- log-belief mass assigned the true latent state s*. We have (bb E [0, oc). with v (b) 0 iff b(s*) = 1 (task completion), and smaller values indicate higher confidence in s". For brevity, we write , == L (bū) and V,, == (b,) when analyzing their dynamics. We then define the belief-update discrepancy as the expected gap in v after one update between the I.I.M update rule Bg and the Bayesian operator Bt:
(b:):) == Eu, Eo, [ (Bo at, o.) (B (Br (t, (.)))
Accurately modeling belief states in active reasoning requires the agent to maintain precise estimate of the underlying problem state and the remaining uncertainty, which is inherently challenging for LLMs. To formalize imperfect belief modeling, introduce the following assumption.
Assumption 1 (Update-Erre) Error Growth). There exist constants me > O, co 2 O, and a threshold Uo 2 0 such that for all beliefs b satisfying T 2 Uo. co(b) 2 mg (bb) Co-
Intuitively, Assumption states that belief-update errors are amplified as the deviation increases. In high-uncertainty regimes, the update error grows at least linearly with V. Then, we have
Theorem (Informal). Assume (i) non non-degeee observations, (ii) an -ipsschhi policy w.r.t. beliefs, and (iū) Asmp. 1. Define U := max Uo, (Vi + +B+ BBco)
with B := 2(- log 4 1/n), and let is ;= inf(t V, 2 U}. If < oc, then for all t 2 ts, the expected potential ceases to decrease: 1E[4 6.] > Wr Moreover, additionally assuming Uo 0 and 4 2 μ > for all
t ts. holds that ts I 1+ logi+ maU-- mg for 8 == map (co+ B) > 0 and A : 11 W
A formal statement and proof are given in Appendix B.3. Intuitively, Thm. indicates that, once ts is reached, the belief trajectory enters an absorbing region in which expected task progress becomes non-positive. We refer such regions Belief Trap Regions and define them formally below.
3

---

Published as a conference paper at ICLR 2026
Definition 1 (Belief Trap Region, BTR). A set Re S A(S) is called a belief trap region for an agent parameterized by 0 if it is absorbing and induces non- nonttivi progress: for any belief b E Rg and all subsequent times t once entered. [[ b, b 2 (b)..
Misguided credit assignment. Within BTRs, the potential sequence V. becomes non-decreasing in expectation, i.e., E[ by] 2 v, Consequently, once trajectory enters the BTR, subsequent steps contribute little task progress and reinforce the stalled dynamics. This degrades sample efficiency, as extended uninformative interactions provide limited learning signal. More critically, entry into the BTR distorts credit assignment: the uninformative tail of a trajectory contaminates the credit attributed to earlier exploratory actions, and may even invert their estimated advantages. This mechanism discourages exploration and leads to suboptimal policies.
We formalize this by analyzing the generalized advantage estimator (GAE) (Schulman et al., 2015), A. ET- t-1 (イλ)* where 7 E (0, 1) is the discount factor, x [0, 1] the GAE parameter, and the TD-error is defined as 6, rr 1 VG, with r, the intermediate reward and V. the value function at step t. We consider the outcome- based RL setting, in which only the terminal step yields a non-zero reward. The following theorem characterizes how entry into the BTR can drive the expected advantage of early actions negative, thereby inverting the gradient direction.
Theorem 2 (Informal). Under the same setup as Thm. 1, assume (i) the value function in policy
optimization satisfies Ve_ = g(br(s*) () for an increasing, differentiable with inf g' (x) 2 kv > O, and (ii) belief drop in BTRs: there exists pb > such that E [bx++1 (o=) (x(s*) Ful S - Pa for k tg. Then, for any t < tg. the expected advantage is bounded: E E[AA]] I Y (t) - po (t)) where Spre (t) = ド== -/s-t-((7N)) (T2)² and (tallt) = - Therefore, sufficient condition for [[[Ad] < 0 is: pv > /t (t). In particular, when >d 1 (often used in practice for Long-horiza agentic RL), this reduces to kov ps > L,L, where A = ts - t and L = T - 1 tg are the prefix and tail lengths, respectively.
formal statement is given Appendix B.4. Thm. quantifies the credit assignment failure: a sufficiently long uninformative tail (large L.) induces a negative drift that can dominate the positive contribution from the informative prefix, causing its overall gradient to point in the wrong direction and penalizing earlier exploratory actions. This analysis directly motivates T truncating rollout upon entering the BTR preserves the credit assigned to informative prefix actions and eliminates the adverse effect of the uninformative tail.
Corollary (Value of Truncation). Lei ATT denote the advantage estimator truncated at t... Under the assumptions of Thm. 2, early truncation yields a less biased gradient estimate: E[AP" >
B[A.] + Po (t.
Corollary indicates that truncating the trajectory at infft:: V 2 U} removes the uninforma- tive tail and yields a less biased policy optimization update. However, this idcalized truncation rule not directly implementable in practice for two-fold reasons. 1) Belief modeling complexity; the belief state b is defined over the latent state space S, which is often high-dimensional, structured, and intricate. In LLM agents, belief not explicitly represented: instead, it is only implicitly encoded in intermediate reasoning traces or internal activation status, and thus precisely recovering their underly- ing belief states is infeasible in practice. 2) Unobservable thresholds: Although Thm. provides sufficient conditions for entry into the BTR, the critical threshold U and its related parameters (e.g.. mg. co. B) are agent-specific and cannot be directly measured.
2.2 FROM THEORY TO PRACTICE: PROXY SIGNALS
Operational criterion - T3 condition. We now translate the theoretical characterization of belief trapping into an operational criterion. Although the exact BTR entry time is unobservable, its defining feature, i.e., stalling of epistemic progress, can be approximated through observable surrogates. This motivates a general truncation principle based on detecting sustained stalls of progress:
Definition 2 (T3 Condition). Let Hi denote the hypothesis space at step t. The T condition for trajectory truncation at step t is defined as follows: there exists a minimum progress threshold Amin 0 such that for all steps in the window it it- t,),, d(H,, Hr+1) S Amiu: where k is the window size and d(...) .) is a refinement measure capturing the degree to which the hypothesis set contracts between two consecutive steps.
4

---

Published as a conference paper at ICLR 2026
T truncates the trajectory step f when the condition is satisfied. In goal-directed active reasoning tasks, the space of latent states could correspond to the set of candidate solutions, where we could interpret H, as the subset of states that remain plausible given the interaction history up to step t. Its concrete instantiation may vary across tasks and can be finite or infinite (ef. Sec. 3.1). In particular, for tasks with finite and enumerable hypothesis space HH. if one models the agent's belief as uniform over Hz (assuming s* E Hz), then the identity V ((bb) log [Hel follows, which provides an exact observable surrogate for the potential dynamics in this setting.
Relation to the BTR formalism. Conceptually, the T principle is structurally aligned with the BTR formalism: BTRs are characterized by stalled progress in the potential function, i.e., EE v 2 In goal-directed reasoning tasks, such stagnation typically manifests as persistent lack of contraction in the hypothesis spaces. Def. 2 formalizes this point by introducing: /) measure d(H,, H4+1) to quantify incremental contraction of the hypothesis representation; 2) threshold Amin to capture the notion of a minimally informative update; and 3) a window of length that enforces temporal persistence, reflecting that BTRs arise from sustained stalls rather than a single noisy fluctuation.
To further quantify this alignment, the following proposition establishes a guarantee under a standard biased noisy model, linking T ingredients to an upper bound on false-truncation probability.
Proposition 1. Let the true single-step potential progress be gt == ((bb)) - (bb+1) and define the observable refinement signal d, == d(H,, H.+1). Assume that (i) outside the BTR, single-step potential progress admits a uniform positive margin: 9t 2 p > 0, and (ii) the proxy follows biased Gaussian-nois model: d;- 9t. + 3r + Et, where [3el < Ma. &t ~ NN0,00 o²) are independent across t.
If A min < p - Ma, then a sufficient condition for the T rule to keep the false-truncation probubility on any k-step non-BTR segment below 8 E (0,1) is k -Ma 22 2 20² log(1/5)
proof is given in Appendix B.9, This proposition shows that, even under both systematic bias and stochastic noise in the proxy, the T rule remains statistically robust. In particular, the choice of H. and metric d(-,-) determines the bias bound Mg. Reducing this bias, increasing kt or decreasing Amin reduces the probability of false truncation at an exponential rate. We additionally present an analysis on the effect of dilse-truncation in Appendix C.3.
Practical instantiation and toward general-purpon detectors. In practice, since the structure of hypothesis spaces and notions of progress differ across tasks, constructing H4 and d(-..) naturally leverages task-level structure to define observable proxies that track epistemic progress. We show how to instantiate it for practical tasks in Sec. 3. Moreover, guided by T we can further reduce the reliance on task-specific structures by designing general-purpose truncation detectors. We conduct preliminary explorations and find that these surrogates can be incorporated into T while still yielding improvements across multiple tasks. Details and discussion are provided in Appendix E.
Key advantages. This principle functions as a meta-wrapper: it provides structured guidance for designing effective proxy signals grounded in progress-based criteria that capture the essence of belief-trap dynamics, rather than relying on complex heuristics or heavy engineering. Importantly, the resulting truncation rules integrate seamlessly into standard policy optimization frameworks (e.g.. PPO, GRPO, GSPO) without altering their algorithms, making T a practical drop-in solution for mitigating credit assignment distortion in active reasoning.
3 EXPERIMENTS
3.1 TASK-SPECIFIC INSTANTIATIONS OF THE T CRITERION
We evaluate 3 on five interactive reasoning tasks from AR-Bench (Zhou et al., 2025) and Multi-Tum Puzzles (Badola et al., 2025). The T criterion (Def. 2) provides a task-agnostic principle. In practice, its components (H, d, etc.) are instantiated using observable proxies tailored to each task, Note that we do adaptations to some of these datasets for RL training. See more details in Appendix F.I.
GuessNumbers (GN). The agent aims to identify a hidden number through iterative guesses, re- ceiving structured feedback that indicates the number of digits in the correct position or misplaced. The hypothesis space Hi consists of all candidate numbers consistent with the interaction history {agi. 051). ). We naturally define the refinement metric as d(H,, H++1) := |7,, |Hr+1l, which directly measures reduction in the candidate set. Early truncation: a trajectory is cut at step t if the agent's guess a, lies outside H4-1, corresponding to the case k- = 1 where we treat d(H4,11 H.) O.
5

---

Published as a conference paper at ICLR 2026
Table 1: Main results across active reasoning tasks (all metrics are scaled by 100). indicates absolute improvement (in points) over the vanilla RL baseline. We report the average rank across all metrics.
CD
SP
GN
PE
MR
Avg.
EM
Fl-word
Fl-char
EM
Binary Sim
EM
Rank
Direct Inference
03-mini
92.67
20.64
39.35
95.28
44.67
83,33
4.67
Gemini-2..-PPru
92.23
24.12
49.28
90.84
16.67
83,00
5.67
Qwen-2.5-7B-Ing
12.50
19.46
41.62
20.94
23.67
27.67
8.17
Reinforcement Learning
PPO
61.67
28.77
74,56
91.62
42.00
24,33
6,50
PPO w/ T
77.83 16.2
36.85 1.811
81.50 6.69
93.98 f
49.00 7.0
38,00 13.6
4.50
GRPO
79.33
36.46
83.73
61.26
51.67
12.00
5.50
GRPO T
81.33 1.2.
39.45 1.0
84.58 0.8
91.36 30.1
52.33 1.0.7
32.67 1.20.
3.17
GSPO
77.67
36.63
82.17
96.07
59.00
14.67
4.33
GSPO w/ T
81.00 1.3
36.96 p.3.
82.08 1.0.1
99.74 1.7.
62.00 1:300
55.67 4.4.0
2.50
Such guesses violate the logical constraints accumulated from previous observations and reflect failure to correctly track the candidate set.
SituationPuzzles (SP). The agent resolves a paradoxical puzzle by posing yes/no questions judge model. Here Hi denotes the set of plausible explanations consistent with the dialogue history. Since H4 can be complex or unbounded, we approximate stalled refinement using judge feedback: step is considered uninformative if the judge responds with "unknown," which serves as proxy for d(H,, 1++1) < Ami. Early truncation: this occurs for trajectory, signaling entrapment in an unproductive line of questioning. We employ an L.M--based judge proxy in the main experiments and additionally evaluate a judge-free proxy in Sec. 3.3.3.
consecutive steps, we truncate the
CircuitDecoding (CD). The agent identifies hidden boolean circuits from large candidate pool. At each step, the agent queries a candidate circuit with a binary input and eliminates inconsistent candidates based the feedback. The hypothesis space HA consists of all surviving candidates consistent with the interaction history, and we define the refinement metric analogously to GN: d(H,, H.+1) = |H-1| 17,-11- Early truncation: we monitor [He and truncate if it fails to contract, Le., d(H,, H++1) ) S 0, for k = 3 turns, indicating that queries no longer reduce uncertainty.
(MR). In PE, the agent infers hidden vector o² about user preference on movies by iteratively raising pairwise comparisons over the given reference movies. In MR, the agent recommends unseen movies the user based the inferred preference vector, requiring generalization beyond the training distribution. Here Hi corresponds to the subspace of plausible preference vectors consistent with past feedback. Since this space is continuous and not explicitly enumerable, we approximate its epistemic refinement progress via the LLM's explicit estimate et. Concretely, we prompt the agent to report current estimate e, in fixed format at each turn. Early truncation: approximate the refinement signal d(H-, H++1) by the change in similarity between the agent's estimate and the ground -ruth preference, i.e., Sim(U7+1, v²) Sim(e,, )²".. If similarity decreases for k = 2 consecutive steps, the trajectory is truncated, reflecting persistent divergence in the inferred preference representation. As the proxy de- pends access the ground-trui preference v" during training, also explore alternative proxies that do not require ground-truth information and demonstrate the promise of T in Appendix D.3.
PreferenceEstimattio (PE) / MovieRecommendatiee
3.2 EXPERIMENTAL SETUP
Baselines. To evaluate the effectiveness of T compare it against the following baselines: 1) Direct Inference without Training, where we evaluate representative proprietary reasoning L.MMs,
including 03-mini and Gemini-2.5-Pro: 2) PPO (Schulman et al., 2017); 3) GRPO (Shao et al., 2024);
and 4) GSPO (Zheng et al., 2025). See more details of the adopted RL algorithms in Appendix F.2.
Implementation Details. The main experiments of RL training are conducted on Qwen2.5-7B-
Instruct (Yang et al., 2024). Analyses on other architecture scales and types can be seen in Sec. 3.3.4. For the GN, CD, PE, and MR tasks, the interactive feedback is rule- base-based; for the SP dataset, Qwen2.5-14B 1-0-255--4B-Insttrur model simulates the "user" and provides the interactive feedback. See more implementation details in Appendix F.3.

---

Published as a conference paper at ICLR 2026
PPitae wase
coz
0991 u1..
35
(a) CD (PPO)
(b) SP (GRPO)
(c) GN (GSPO)
(d) PE (PPO)
Figure 3: Training dynamics of rewards w.P.I. training steps.
e011 cears
perthe are
olgttee wid
4080
130
Trairing
122
"raniig 1o 1o
(a) CD (PPO)
(b) SP (GRPO)
(c) GN (GSPO)
(d) PE (PPO)
Figure 4: Training dynamics of response length w.E.f. training steps.
Evaluation Metrics. For the GN, CD, and MR tasks, we report Exact Match (EM), which measures whether the final prediction made by the LLM exactly matches the hidden number, ground-truth circuit, or the correct movie recommendation. For the SP task, we use the FI score (both word-level and character-level) assess the similarity between the ground-truth explanation and the solution produced by the LLM. For PE, we report Binary Similarity, which compares the LLM-estimater vector against the ground-truth preference vector using cosine similarity. Specifically, we threshold the cosine score at 0.88: values above the threshold are labeled as 1, and values below as O. In Appendix D.1, we also explore the sensitivity with other thresholds.
3.3
EXPERIMENTAL RESULTS AND ANALYSES
In this part, we first present overall performance, followed by analyses of T on uut-of-distribution generalization, ablation studies of truncation conditions, and the impact of LLM architectures.
3.3. OVERALL PERFORMANCE
The main experimental results are summarized in Table Across tasks, all RL-trained agents, both with and without T substantially outperform the zero-shot baseline, confirming the necessity of RL in incentivizing active-reasoning capabilities. Compared to vamilla RL methods, incorporating T consistently improves final performance across datasets and algorithmss, with non-marginal gains observed in 14 out of 18 reported metrics. On CD, PPO+T boosts EM by 16.2 points and GRPO+T yields further gains. On SP, GRPO+T achieves the best F1-word and Fl-char scores. On GN, T leads to substantial improvements. raising GRPO by 30. points and enabling GSPO reach a near-perfect 99. 79.74 EM. In PE and MR, T3 also brings steady gains, with GSPO+T3 improving movic recommendation accuracy by 41.0 points. Overall, these results indicate that 3 provides consistent benefits across diverse active reasoning tasks.
Overall Performance.
Comparing to frontier reasoning models. We can also find that advanced reasoning LLMs perform strongly on tasks where the hypothesis space H is finite and enumerable (e.g., GN and CD), However, their performance degrades on tasks with large, infinite, or continuous hypothesis spaces (e.g., .. SP and PE), where they lag behind RL-trained Qwen-7B models equipped with T These observations suggest that large-scale RL with outcome reward training alone may be insufficient for effective active reasoning over unbounded hypothesis spaces, and that mechanisms explicitly addressing credit assignment, e.g., T could provide complementary benefits.
Beyond final performance, T improves training dynamics. As shown in Fig. 3, vanilla RL methods exhibit higher variance and instability, with rewards prone to collapsing after partial convergence. By contrast, incorporating T leads to more stable training trajectories, with largely monotonic or near-monotonic reward improvement and much fewer abrupt drops, and enables better optimization behavior. These results suggest the dual benefit of 8 stabilizing optimization while encouraging more informative exploration.
Better Training Stability and Optimization Behavior.
7

---

Published as a conference paper at ICLR 2026
Higher Token Efficiency. Although the reward curves wrr. training steps (Fig. 3) suggest slightly slower reward growth in the early stage when incorporating T early truncation reduces the average number of tokens per rollout (cf., Fig. 4). As result, when measured against token consumption, T achieves higher training efficiency. For example, under PPO on CD, to reach a reward level of 0.65, our method consumes 66.4% of the total tokens compared to vanilla on average, under GSPO on GN, to reach 0.96, it requires 76.3% of the tokens. More importantly, while vanilla methods often stagnate and fail improve further, incorporating T continues to enhance rewards, achieving up to 0.8 on CD and 0.99 on GN.
3.3.2 OUT-OF-DISTRIRIUE
ANALYSIS
To better understand whether the agents learn the generalizable policies for active reasoning, we further evaluate T under distribution shifts in two representative tasks: CireuitDecoding (CD) and Prefer- ence Estimation (PE). In CD, we vary two key factors relative to training: the number of hidden circuits (training uses 2, we test up to 4) and the candidate pool size (train- ing uses 10, we test up to 30). In PE, we vary the number of reference movies (train- ing uses 10, we test 5-30) and the sampling distribution of their scores (training uses uniform, we test skewed side distributions).
Table 2: Evaluations of T on dutto--distiobution
(OOD) scenarios of PE (Qwen-2.5-7B-Ing and CD
(Qwen-2.5-14B-Int)) tasks under the PPO algorithm.
PE (PPO)
CD (PPO)
Vanilla
w/ T
Vanilla
w/ T
Reference Size (S)
Candidate Size (S)
S 5 40.0
44.3 1:43
S = 10
67.8
86.3 1
S = 10
42.0
49.0 1070
= 155
61.7
74.7 13.0
S= 15
39.3
47.0 177
= 20
48.2
55.8 1.2
S 20
41.0
53.7 p
25
35.2
46.0
1088
S= 30
42.3
46.3 1:40
30
31.5
35.7 142
Reference Sampling
Hidden Circuit Size (C)
min-max
45.7
56.0 1033
67.8
86.3
uniform
42.0
49.0
1:70
C
60.3
75.3
15.0
The results are given in Table 2. Across all OOD settings, T consistently improves over vanilla PPO. In CD, although accuracy drops as the task becomes harder with larger candidate pools or more hidden circuits, the gains from T remain substantial, reaching t 10.8 points with 25 candidates and 15.0 points with 3 circuits. In PE, performance varies non-monotonically with the reference size, where moderate contexts (e.g.. S 20) achieve the best results ( 12.7 points). We conjecture that too few references increase the ambiguity of preference estimation, while too many may introduce noise and redundancy, which may in turn exacerbate belief-trap dynamics. See Appendix D.2 for an empirical evidence. Similarly, for reference sampling, T yields improvements across all conditions, with the largest margin observed under max-skewed sampling. Overall, these results show that T consistently enhances OOD robustness across diverse settings, even in more challenging regimes where the distribution deviates substantially from the training.
max
50.7
61.3
10.7
C 42.7
49.3
3.3.3 ABLATION STUDY ON TRUNCATION CONDITIONS
The effectiveness of T depends on the design of the proxy sig- nal for truncating the BTR tail. We therefore conduct ablation studies to examine the robustness of different truncation conditions and their associated trade-offs. First, we vary the window size k to evaluate the effect of tempo- ral persistence in detecting stalls. Furthermore, we consider alter- native truncation strategies. For
Table 3: Ablation Study of Truncation Conditions on the SP, CD, and PE tasks. Beyond the window size k as seen in Def. 2, we consider alternative truncation methods, described in a and B.
SP (GRPO)
CD (PPO)
PE (PPO)
Method
Fl-word
Method
EM
Method
Binary Sim
Vanilla
36.46
Vanilla
61.67
Vanilla
42,00
& = 3
38.62 1216
k 2
69.17 1.550
I 2
49,00 7.00
& ==3
39.45 1299
k 3
77.83
162
I 4
44.33 1:233
è 9
36.96 10.50
k 4
79.33
176
I 7
42.00 1.000
0.9
39.44
1298
g = 0.1
69.00
7,333
3= 0.2
43.33 1:33
0.93
38.81
12.35
9 0.2
57.50
14.17
3 0.5
44.67
1.67
0.96
37.93 1.47
8 0.5
13.17
1.8.5
3 0.8
39.00 3.300
the SP task, we evaluate Ques- tion Semantic Similarity (Sim-a); a trajectory is truncated the cosine similarity between the embedding of the current query and any previous query exceeds threshold a, where we leverage the E5-1arge-ve model (Wang et al., 2022) to calculate embeddings. This proxy detects redundant or circular questioning, and we evaluate random truncation (Rand-3) 3) baseline, where each step is truncated independently with probability B. We test 3 0.21, 0.5) for CD and (0.2, 0.5, 0.8) for PE.
{0.9, 0.93, 0.96]. For the CD and PE tasks, we include
8

---

Published as a conference paper at ICLR 2026
1.50
Rato of Earty Tuncation
Rato et Extr Punaaion
0.8
0.6
0.4
kod
0.30
6.55
0.4
0.2
0.40
slm-0.93
sim-0. 0. 96
0.0
20
20
60
80
100
0.70
350
Step
(a) SP (GRPO)
(b) CD (PPO)
(c) PE (PPO)
Figure 5: Training dynamies of the ratio of early truncation w.r.f. training steps under different
truncation conditions for the SP (a), CD (b), and PE (c) tasks.
The results are reported in Table 3. For SP, increasing i improves performance up around k- after which the gains diminish. The similarity- proxy also improves over vanilla GRPO, suggesting that T is robust to different proxy formulations long as they can detect the BTR entry reasonably. For CD, varying k shows stable improvements, with k = 3, 4 yielding the largest gains over vanilla PPO. We further observe that even random truncation can produce a mild improvement when the ratio / is appropriately chosen. This suggests the significance of the BTR issue: mitigating long uninformative tails, even via simple truncation heuristics, partially improves optimization quality, For PE, k 2 achieves the best performance, while the gains diminish the condition becomes looser. Overall, these results indicate that the proxy condition should be calibrated at a moderate level. If is too loose (e.g.. k- = 9 for SP), truncation has limited effect, causing traiei-g errors to accumulate. it is too strict (e.g.. 3 = 0.2, 0.5 for CD), it may terminate trajectories prematurely, suppressing early-stage exploratory actions and reducing the effective learning signal.
Training Dynamics of Early Truncation, Furthermore, we examine the temporal evolution of the early-truncation frequency during training, as shown in Fig. 5. For clarity, the truncation ratio at training step is defined as ratio, = # rollouts truneated step This quantity tracks how frequently # lotal rollouts at step e
trajectories satisfy the truncation condition throughout optimization. Combining these dynamics with the final performance (Table 3) yields a clear pattern. For tasks where the hypothesis space H is unbounded (SP and PE), stronger performance is associated with relatively high and stable truncation ratios from early training steps. For example, in SP, the query- query-similarity proxy with = 0.9 quickly reaches near 1.0 and achieves the best F1; in PE, k = 2 likewise achieves both a higher truncation ratio and the highest performance. These observations suggest that, in unbounded hypothesis spaces, promptly removing uninformative tails could contribute to improved training performance. Notably, in PE, random truncation with 8 = 0.5, 0.8 yields truncation ratios comparable to k = 2 but leads to inferior final performance. This underscores the importance of proper truncation condition design: it should meaningfully approximate the BTR entry rather than cut indiscriminately.
By contrast, for tasks with finite and enumerable spaces (CD), a low-to-moderate truncation ratio would be preferable: settings such k = 3, 4 maintain low truncation frequencies throughout training and yield the largest EM gains; more aggressive truncation (c.g., k = 1,2) increases the truncation ratio and is associated with reduced performance, consistent with premature termination of potentially informative trajectories. In summary, these dynamics suggest that, given a properly designed truncation condition, the appropriate truncation intensity depends on the structural properties of the hypothesis space and the task: relatively aggressive truncation could be beneficial in unbounded settings, while moderate truncation would be preferable in finite settings.
3.3.4 IMPACT OF LLM ARCHITECTURE.
We further evaluate T across different LLM scales and architectures, including Qwen-2.5 (3B, 7B, and 14B) and multiple variants of ILLMMA33. 1-8B. As shown in Fig. 6a and 6b, across Qwen-2.5 3B, 7B, and 14B, we observe that the 3B model shows only limited improvements, whereas the 7B and 14B variants achieve clear gains under RL. Morcover, larger models tend to benefit more substantially from T compared to the 3B variant. One possible explanation, consistent with our formulation in abilities may correspond to a larger update-error growth (i.e.,
Sec. 2, is that weaker belief-trackin larger mg. of., Asmp. 1), making smaller models more prone to quickly falling into BTRs, where even truncation cannot provide sufficient informative training signals.
similar pattern holds across architecture types. As shown in Fig. 6c, we compare the effectiveness of T across I.LMMA33. 1-8B--Instuee Dwenn22.5-78-instruc and DeepSeek-RR R 1--Distill-LLL LMMA-BB...

---

actions. For example, Published as a conference paper at ICLR 2026
40
60
10
10
40
40
10
20
20
38
78
148
148
Lama
Owen
Bestlee-Uis
(a) PE (PPO)
(b) CD (PPO)
(c) CD (PPO)
Figure 6: Effectiveness of T on different sizes (a, b) and types (c) of LLM architectures. The
"Performance Gain" denotes the improvement of T compared the vanilla RL method.
We observe that HLAMA-8B-Instrrr improves only marginally under T while its DeepSeek- distilled variant and Qwen-7B benefit more substantially. This echoes recent reports that Qwen exhibits stronger reasoning behaviors than LLaMA (Gandhi et al., 2025). Such differences may extend to belief-tracking abilities under partial observability. Notably, the distilled LLaMA variant with T -quuipeedd RL achieves the best overall performance, exhibiting the largest performance gains. We conjecture that distillation may improve the belief-tracking related capabilities, thereby enhancing the utility of T in preserving credit assignment. In our formulation, both scale- and urchitecture-depeet
differences may be interpreted through variations in belief-tracking abilities and the associated me, which governs how easily trajectories get trapped in the BTR.
4 RELATED WORK
Active Reasoning requires LLMs to interact with external sources and actively acquire missing information to solve complex tasks. Prior work has improved I.LMs' ability to handle ambiguity and incompleteness through making clarification and information-sekkinn
Proactive CoT (Deng et al., 2023) prompts LLMs to identify ambiguous problems and generate clarification questions, while UoT (Hu al., 2024) quantifies the contribution of each question in reducing uncertainty. However, challenges remain when transitioning from LLMs' single-turn success multi-turn active reasoning (Kwan et al., 2024; Liang et al., 2024; Badola al., 2025), even with several advanced strategies such as tree-based searching or post-training approaches, as highlighted in existing works (Zhou al., 2025). In contrast, we leverage RL incentivize active reasoning capabilities, and propose 3 to address key issues when applying RL in this setting.
Credit Assignment and Multi-turn RL. Credit assignment is crucial long-horizon or multi-turn RL. Existing methods have extensively explored rule-based approaches (Yu et al., 2024; Dou et al., 2024; Zhang et al., 2025b) to shape intermediate rewards. Several recent works also proposed to measure the progress of stepwise actions toward overall task completion as intermediate rewards. Specifically, CURIO (Wan al., 2025) constructs potential function over an ideal belief state to assign intermediate rewards, assuming that the latent state space is finite and enumerable. Sotopia- RL (Yu et al., 2025) relies on reward labeling with proprictary LLMs. SPA-RL (Wang et al., 2025) trains reward models for intermediate rewards by enforcing a summation constraint with respect to the final outcome reward. In our studied active reasoning scenario, belief deviation under partial observability makes it difficult for outcome-bor based rewards to properly assign credit key reasoning steps. Our proposed 3 mitigates this by halting the trajectory before the reasoning process becomes trapped in excessive belief deviation and the error accumulation overwhelms credit assignment.
CONCLUSION
In this work, identify belief deviation and entry into the belief-trap region as critical failure mode underlying instability and sub-optimality in RL for LLM--bssed active reasoning. To mitigate its harmful accumulation, we proposed T an carly-truncation mechanism that halts belief-trapped trajectories. Empirical results on five active-reasoning tasks show that T consistently improves both training stability and final performance across multiple RL algorithms. Overall, our findings highlight belief deviation as a central bottleneck and show that controlling it provides principled pathway toward building robust and generalizable active reasoning agents.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank the reviewers for their constructive comments and suggestions. Deyu Zou, Yongqiang Chen, Haochen Yang, and James Cheng were supported by a CRF (No. C2005-24Y) from the RGC of Hong Kong. This work was collaboration between Husky Data Lab at CUHK and ByteDance, supported by ByteDance University Collaboration Project Grant.
REFERENCES
Kartikeya Badola, Jonathan Simon, Arian Hosseini, Sara Marie Mc Carthy, Tsendsuren Munkhdalai, Abhimanyu Goyal, Tomás Kočiský, Shyam Upadhyay, Bahare Fatemi, and Mehran Kazemi. Multi- turn puzzles: Evaluating interactive reasoning and strategic dialogue in llms. arXiv preprint arXiv:2508. 10142, 2025.
Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, and Tat-Seng Chua. Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration. arXiv preprint 2idi::2300..33222 2023.
Shihan Dou, Yan Liu, Haoxiang Jia, Limao Xiong, Enyu Zhou, Wei Shen, Junjie Shan, Caishuang Huang, Xiao Wang, Xiaoran Fan, et al. Stepcoder: Improve code generation with reinforcement learning from compiler feedback. arXiv preprini arXiv:2402.0139) 2024.
Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma Gongque, Weihao Zeng, Wei Wang, Jingang Wang, Xunliang Cai, and Weiran Xu. Agentrefine: Enhancing agent generalization through refinement tuning. arXiv preprint arXiv:2501.01702, 2025.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D) Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503. 000007 2025.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501. 2025.
Zhiyuan Hu, Chumin Liu, Xidong Feng, Yilun Zhao, See-Kiong Ng. Anh Tuan Luu, Junxian He, Pang Wei W Koh, and Bryan Hooi. Uncertainty of thoughts: Uncertainty-aware planning enhances information seeking in Ilms. Advances in Neural Information Processing Systems, 37:24181-242155. 2024.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. arXiv preprint arXiv:2212. /2..00333 2022.
Leslie Pack Kaelbling, Michael Littman, and Anthony R Cassandra, Planning and acting in partially observable stochastic domains. Artificial intelligence, 101((-2):99-13) 1998.
Wai-Chung Kwan, Xingshan Zeng, Yuxin Jiang, Yufei Wang, Liangyou Li, Lifeng Shang, Xin Jiang, Qun Liu, and Kam-Fai Wong. Mt-eval: A multi-turn capabilities evaluation benchmark for large language models. arXiv preprint arXiv:2401. 16745, 2024.
Philippe Laban, Hiroaki Hayashi, Yingbo Zhou, and Jennifer Neville. Lims get lost in multi-turn conversation. arXiv preprint arXiv:2505.06122 2025.
Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding, Yidi Miao, Ramayya Krishnan, and Rema Padman. Beyond single-turn: survey on multi-turn interactions with large language models. arXiv preprint 2rXiv:2504.04717. 2025a.
Zhong-Zhi Li, Duzhen Zhang, Ming-Liang Zhang, Jiaxin Zhang, Zengyan Liu, Yuxuan Yao, Haotian Xu, Junhao Zheng, Pei-Jic Wang, Xiuyi Chen, et al. From system to system 2: A survey of reasoning large language models. arXiv preprint arXiv:2502.1744 2025b.
Zhenwen Liang, Dian Yu, Wenhao Yu, Wenlin Yao, Zhihan Zhang, Xiangliang Zhang, and Dong Yu. Mathchat: Benchmarking mathematical reasoning and instruction following in multi-turn interactions. arXiv preprint 2405::2405. /9444, 2024.

---

Published as a conference paper at ICLR 2026
Wenquan Lu, Yuechuan Yang, Kyle Lee, Yanshu Li, and Enqi Liu. Latent chain-of-thought! decoding the depth-recurrent transformer. arXiv preprint arXIv:2507.02199, 2025.
OpenAL. Openai o3-mini. httpps://peenn...com//////////// 2025.
January
Aske Plaat, Annie Wong, Suzan Verberne, Joost Broekens, Niki van Stein, and Thomas Bäck. Reasoning with large language models, survey. CoRR, 2024.
Aske Plaat, Max van Duijn, Niki van Stein, Mike Preuss, Peter van der Putten, and Kees Joost Batenburg. Agentic large language models, survey. arXiv preprint arXiv:2503.2303: 2025.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 150.0:350..02438, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17Xv::7077.6347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rIhf framework. In Proceedings of the Twentieth European Conference Computer Systems, pp. 1279-1297, 2025.
Saksham Sahai Srivastava and Vaneet Aggarwal. A technical survey of reinforcement learning techniques for large language models. arXiv preprint arXiv:2507.04136. 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12599 2025.
Yanming Wan, Jiaxing Wu, Marwa Abdulhai, Lior Shani, and Natasha Jaques. Enhancing personal- ized multi-turn dialogue with curiosity reward. arXiv preprint arXhv:2504.03206 2025.
Hanlin Wang, Chak Tou Leong, Jiashuo Wang, Jian Wang, and Wenjie Li. Spa-rl: Reinforcing Ilm agents via stepwise progress attribution. arXiv preprint 25XXi::505.20732.. 2025.
Liang Wang, Nan Yang, Xiaolong Huang. Binxing Jiao, Linjun Yang. Daxin Jiang, Rangan Majumder, and Furu Wei. Text embeddings by weakly-s contrastive pre-training. arXiv preprint arXiv:2212.03553: 2022.
Shuhe Wang, Shengyu Zhang, Jie Zhang, Runyi Hu, Xiaoya Li, Tianwei Zhang, Jiwei Li, Fei Wu, Guoyin Wang, and Eduard Hovy. Reinforcement learning enhanced llms: A survey. arXiv preprint arXiv:2412. 10400, 2024.
Shirley Wu, Michel Galley, Baolin Peng, Hao Cheng, Gavin Li, Yao Dou, Weixin Cai, James Zou, Jure Leskovec, and Jianfeng Gao. Collabllm: From passive responders to active collaborators. arXiv preprint arXhv:2502.006400 2025.
Fengli Xu, Qianyue Hao, Zefang Zong Jingwei Wang. Yunke Zhang, Jingyi Wang. Xiaochong Lan, Jiahui Gong, Tianjian Ouyang, Fanjin Meng, et al. Towards large reasoning models: A survey of reinforced reasoning with large language models. arXiv preprint 2xXi:25501.06686 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprini arXiv:2412.15115 2024.
Haofei Yu, Zhengyang Qi, Yining Zhao, Kolby Nottingham, Keyang Xuan, Bodhisattwa Prasad Majumder. Hao Zhu, Paul Pu Liang, and Jiaxuan You. Sotopia-rl: Reward design for social intelligence. arXiv preprint 2508::2508.03305 2025.
12

---

Published as a conference paper at ICLR 2026
Yuanqing Yu, Zhefan Wang, Weizhi Ma, Zhicheng Guo, Jingtao Zhan, Shuai Wang, Chuhan Wu, Zhiqiang Guo, and Min Zhang. Steptool: A step-gre grained reinforcement learning framework for tool learning in llms. 2024.
Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. Agent-r: Training language model agents to reflect via iterative self-training. arXiv preprint AxXi::2501.11225 2025.
Guibin Zhang, Hejia Geng, Xiaohang Yu, Zhenfei Yin, Zaibin Zhang, Zelin Tan, Heng Zhou, Zhongzhi Li, Xiangyuan Xue, Yijiang Li, al. The landscape agentic reinforcement learning for Ilms: A survey, arXiv preprini arXiv:2509.0254: 2025a.
Zijing Zhang, Ziyang Chen, Mingxiao Li, Zhaopeng Tu, and Xiaolong Li. Rlymr: Reinforcement learning with verifiable meta-reasoning rewards for robust long-horizon agents. arXiv preprint 7rXiv:2507.22844, 2025b.
Chujie Zheng, Shixuan Liu, Mingze Li, Xiong-Hui Chen, Bowen Yu, Chang Gao, Kai Dang, Yuqiong Liu, Rui Men, An Yang, et al. Group sequence policy optimization. arXiv preprint arXiv:2507.1807/ 18071, 2025.
Zhanke Zhou, Xiao Feng, Zhaocheng Zhu, Jiangchao Yao, Sanmi Koyejo, and Bo Han. From passive to active reasoning: Can large language models ask the right questions under incomplete information? arXiv preprint arXiv:2506.0822 2025.
Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. How alignment and jailbreak work: Explain Ilm safety through intermediate hidden states. arXiv preprint arXiv:2406. 2024.
13

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Reinforcement Learning for Active Reasoning
2.1
Theoretical Formulations
2.2
From Theory to Practice: Proxy Signals
3
Experiments
3.1
Task-Specific Instantiations of the T Criterion
3.2
Experimental Setup
3.3
Experimental Results and Analyses
3.3.1
Overall Performance
3.3.2
Out-of-Distributti Analysis
8
3.3.3
Ablation Study on Truncation Conditions
3.3.4
Impact of LLM Architecture
4 Related Work
10
5
Conclusion
10
Notation Summary
15
More Details on the Theory
16
B.1 Detailed Theoretical Setup
16
B.2
Dynamics of Belief Trapping L.MM Agents Active Reasoning
17
B.3
Detailed Statement of Theorem
18
B.4
Detailed Statement Theorem
18
B.5
Important Lemmas
19
B.6 Proof of Proposition B.1
19
B.7 Proof of Proposition B.2
20
Proof of Theorem B.I
B.9 Proof of Proposition
21
B.10 Proof of Lemma B.1
22
B.11 Proof of Lemma B.2
23
Empirical Verification of the Theory
24
C.I Empirical Verification of Assumption
24
C.2 Verification of Theorem 2 and Corollary
25
C.3 Complementary Analysis of
Truncation and Its Impact
26
D Complementary Empirical Analysis
27
14

---

ASS) Published as a conference paper at ICLR 2026
D.1 Rationale for Selecting Binary Similarity Threshold in PE
27
D.2 Effect reference-sor size on redundancy-induse stalling
27
D.3 T3 PE-like tasks without access the ground truth
27
D.4 Exploration of adaptive truncation rule
28
E Potential Future Work
29
E.1 More general-purpose purpose proxy design
29
Setup Details
29
F.1
Dataset Details and Prompt Templates
29
F2
Baseline Details
F3
Supplementary Implementation Details
32
LLM USAGE DISCLOSURE
In our work, we mainly use GPT-5 for writing enhancements, primarily to improve grammar and text clarity.
REPRODUCIBILITY STATEMENT
We describe our dataset details Appendix F.I. For additional training details, see Sec. 3.2 and Appendix F.3. For prompt templates, see Figures 10 to 15.
A NOTATION SUMMARY
Symbol
Meaning
Domain / Notes
Spaces, states, dynamics
O
Latent state, action, observation spaces
Sets
s*
Episode-wise fixed true latent state Transition function
s* S
T(s' 8,a)
Degenerate our setting Assump. B.2; O 2 1)
( s)
Observation model
R, Y
Reward; discount factor
E [0
Beliefs and policies
Probability simplex over S
Set
b9 b
Oracle belief; agent belief at time t
b;, b, A(S) Posterior under O
B ab,,a,,))
Oracle Bayesian belief update
O)
Agent belief update
Parametrized by Distribution on A
b(-
Belief-conditioned policy
Distances and potentials
d(b, b))
distance on belief's
d(b, B) Σ, (6(s) 6'(s) E [0,2] Probability measures; Assump. B.3
TV(P,Q)
Total variation distance
(bb
Truth anchored potential
(bbb Def. B.
log b(s") E [0, oo):
(continued on next page)
15

---

Published as a conference paper at ICLR 2026
Symbol
Meaning
Domain / Notes
(6,)); (6,))
Scalars
Progress quantities
a(b,a)
One-step oracle informativeness
Def. B.2
Pb(b)
Agent expected one-step progress
Def. B.3
cb(b)
Agent-Baye Bayes update error
Def. B.4; Assump. B.4
Belief Trap Region (BTR)
R
Belief trap region
Def. B.5; Prop. B.1 and B.2
is
First time of reaching the BTR sufli- cient condition
RL / GAE quantities
V
(bb))
Value function
Vi = g(br(s))): Thm. B.1
6
TD-error
6, fi + 7V441 V/
A
GAE parameter
À [0,
A,
GAE estimator
A, = EnA))o64
Model and technical constants
1)
Observation non-degeneracy bound
1) (0, 1]: Assump. B.2
L
Policy sensitivity constant
Assump. B.3
ma, co: Uo
Update-error growth parameters
cb(6) > V)) coi Assump. B.4
B
Technical constant
B 2 log l] T, + 1/n): Prop. B.
U
Sufficient BTR threshold
Prop. B.1
A
Initial gap
A (bb (bbi): Prop. B.2
Oracle lower bound before Is
Prop. B.2
8
Technical constant
5 moje (co + B); Prop. B.2
Auxiliary weights
(tre(t)
Geometric prefix weight
Σ 13 ts-t- (71)² Thm. B.1
=0
$9 (t)
Geometric tail weight
Lj=ts- (TA); Thm. B.1
B
MORE DETAILS ON THE THEORY
B. DETAILED THEORETICAL SETUP
Problem Formulation We consider the active reasoning where an LLM agent interacts with an external environment to acquire missing information and infer the solution via a sequence of actions and observations (Zhou al., 2025). This can be modeled as a Partially Observable Markov Decision Process (POMDP), defined by the tuple AS.A O, T, O, R, r),, where S is the space of unobservable latent states, AA the action space, O the observation space, the transition dynamies, O the observation model, R the reward function, and the discount factor. At each step, the agent selects an action (question) o, E based on belief state b ASS), i.e., distribution over latent states summarizing the interaction history. Note that the true latent state s* E S is unobservable to the agent (s* is introduced solely for theoretical analysis); for analytical clarity, we assume s* is fixed within episode. The environment returns observation o O via O ss" (s",a)) at), and the agent updates its belief to b accordingly.
We consider two agentic reasoners operating under the same interaction protocol but differing in their belief-update mechanisms: an oracle reasoner and an imperfect LLM reasoner. An ideal oracle reasoner would maintain oracle belief distribution by E A(S). Specifically, the oracle belief b is
16

---

Published as a conference paper at ICLR 2026
recursively updated via Bayes' rule B* upon taking action and observing o:
bi (s) == B" (b) a, O 8, (1))
(3)
Pb(o a)
where Pb(o (((o|a)= Eves O a0)b (s') is the Bayes-normalizer.
In contrast, an LLM agent does not perform exact Bayesian filtering. Instead, it maintains an LLM belief be, which represents its internal understanding of the latent state and what information remains missing. Given the retiin-obseervatin pair (a, o), the LLM belief evolves by b (s) == Bb a, o), where 0 denotes LLM model parameters.
We compare the LLM agent's trajectory (ht; 0z, 0x)e21 with that of the oracle reasoner (bf aj of (21-
Specifically, the oracle samples action a; x(- b,) and receives observations from the environment generated via own update rule Ba, sampling actions from x(- by) and receiving observations via O-- s at). Note that s* denotes the truth latent state (fixed within episode) used for analysis; both LLM and oracle agents do not observe s". To quantify the discrepancy between beliefs, we use the & disisttnce: d(b, b') = Exes (b)) V/ (s)) s and denote de := d(be, bt).
O- s", aq). updating its belief via 1 (Eq. 3). The LLM agent follows its
B.2 DYNAMICS BELIEF TRAPPING OF LLM AGENTS ACTIVE REASONING
We begin by modeling task progress of active reasoning. Specifically, we introduce a truth-anchored potential function V A(S) + 20 that captures how concentrated the belief is the true s*.
Definition B.1 (Truth-anchored potential). For belief b E ASS) and ground-truth state st, define
(b) == log b(s*).
It holds that (bb) E 10, oo), with (b) = 0 iff b(s*) = 1 (task completion). Lower values of (b)
indicate higher confidence the true state.
Based on this, we assume that the oracle's belief (67) (b)) is well-behaved and guaranteed to eventually converge to the truth.
Assumption B.1 (Oracle Potential Convergence). Along the oracle trajectory (b, aj )221, the potential V (bi) bounded and convergent to zero. Specifically, there exists a deterministic nonincreasing sequence (u4)221 with V and > such that V S u4 for all 2 1.
To analyze the agent's behavior, we define several key quantities. Through the following definitions, we measure the expected information gain of an action under the ideal Bayesian update (Def. B.2), and the actual one-step progress when updating belief via the agent L.LM (Def. B.3). We further quantify the discrepancy between the LLM agent's update and the Bayesian update (Def. B.4).
Definition B.2 Shee-Step Informativeness). For belief b and action a, define
I(b.a) = (b) - (B-o((-)) ( (B* (b, a, o))
This captures the expected improvement of V-progress when taking action from belief b. Definition B.3 (One-step LLM-agent Progress). The LLM agent's expected & V-progress given the current belief b:
Pb( = (b) (B Bo V (Ba(B,, a, o))]
Definition B.4 (LLM- (LLM-Bayes update error). For a belief b. define the conditional update error
(b) = Ba>x((b) Eo-o (Bu(b,, a, (BB((b,,,)))
We now state several technical assumptions required for our analysis.
Assumption B.2. There exists E (0, such that O(o s, s. 2 for all reachable (o, s, a.). Assumption B.3 (Policy Sensitivity). There exist L 2 0 such that for any beliefs b. b'.
TV(a(- b), T(- (')) < L,, d(b.b').
where (P, = SuPAÇA distributions.
denotes the total variation distance between probability
17

---

Published as a conference paper at ICLR 2026
Assumption B.4 (Update-Enee Error Growth). There exist constants mg 0, Co 2 0, and threshold Uo 2 O such that for all b with (b) 2 Up.
co(b) 2 me (b) co-
That is, in hiig--uncertainty regimes, the LLM agent's ss update error grows at least linearly with w.
Accurately modeling belief states active reasoning requires the agent to maintain a precise estimate of the underlying problem state and the remaining uncertainty, which is inherently challenging for LLMs. Assumption B.4 formalizes the imperfect belief modeling of LLM agents, which states that belief-update eπrors amplified as the deviation increases. In high-uncertainty regimes, the update error grows at least linearly with V. We next formalize the regime in which such misspecification dominates the oracle's 's informativeness:
Definition B.5 (Belief Trap Region, BTR), set R4 S A(S) is called a belief trap region for an agent parameterized by 0 if it is absorbing and induces non-positive progress: for any belief b Rs and all subsequent times once entered, Pb( S 0, and equivalently, E( b, = 5) 2 (b).
Within BTRs, the potential sequence V, becomes non-decreasing in expectation, i.e., E[ t+ b.) > vr Consequently, once trajectory enters the BTR, subsequent steps contribute little task progress and reinforce the stalled dynamics.
B.3 DETAILED STATEMENT OF THEOREM
Next, we investigate the characteristics of the BTR as follows:
Proposition B.1 (Sufficient Condition of entering BTR). Under Assumptions B.2-B.4, define the constant B ; (-L log + 1/1), the threshold U == max (oax(Uo, (V, + + co)o /mm ). and let ts == inf (t: v U} the first time the Ve --potential reaches the threshold U. Then the following holds: ig co, then for all 1 Lg, Po(be)
and equivalently, E (bb+1) | b.] > ((bb)..
This result formalizes the absorbing nature of the belief-tray trap region: once the potential Vi exceeds the threshold U, the trajectory is locked into regime where exploration is ineffective and the task progress no longer proceeds. Now we delve into the properties of the BTR entry time:
Proposition B.2 (Upper bound of BTR entry time). Strengthen Asmp. B.4 global, i.e., Un 0.
Assune there exists μ > 0 such that ; Z for all < tg. Assume == mape - (Co + B) > Then the (expected) hitting time into BTR, denoted by tETR inf b, E Ro obeys the upper bound
mg U + 5
LNTR S is < 1 + log 1-mmp mg A +
Here A = 1 V/. The proofs for Proposition B. and Proposition B.2 are given in Appendix B.6 and Appendix B.7, respectively. This yields an explicit upper bound on the time to enter the trap: without corrective mechanisms, belief errors accumulate, so entering BTR becomes inevitable and can occur quickly once belief updates deteriorate.
B.4 DETAILED STATEMENT OF THEOREM 2
Theorem B.1 (BTR Induces Advantage Inversion). Under the following assumptions: (i) the value function in policy optimization satisfies V₂ = g(b((s))) for an increasing. differentiable g with inf, g' (x) 2 kv >> O, and (ii) belief drop in BTRs: after entering the BTR, the belief exhibits uniform negative drift, i.e., E(bx-1(s*)) (8")
[F] S for 2 ts. Then, for any 1 < Lg, the
expected advantage is bounded:
E[ÁN| S 7 (Spre - Kv Po kill (1))
-(s-tt (72) and (t) Ej (1 Therefore, a sufficient condition for where Spr(() =Ei=0'
B[A.] < is:
KvPs > (p) (t)
(5)
In particular, when 7. - (often used in practice for horizon agentic RL), the condition simplifies to Ky Ps. > A/ L. where A = ts t and L = T 1 - ts are the prefix and tail lengths, respectively.
18

---

Published as a conference paper at ICLR 2026
The proof for Theorem B.1 is given in Appendix B.8. This theorem quantifies the credit assignment failure: a sufficiently long uninformative tail (large L) induces a negative drift that can dominate the positive contribution from the informative prefix, causing its overall gradient to point in the wrong direction and penalizing earlier exploratory actions.
B.5 IMPORTANT LEMMAS
Before proving the propositions, we start by providing two important lemmas, and their proofs in Appendix 10 and B.11.
Lemma B.1 (Belief-Lipschttz Continuity of Informativeness). Under Assumption B.2, for any fixed action a E A and any beliefs b, E (S)), we have
[[bb a) I(b'.a) a) S |6 - 1/|
(6)
Consequently, for any action distribution q.
Ba- - Ba-g S - 16 b'111.
(7)
Lemma B.2 Crolicy-Lipschiie Continuity of Informativeness). Under Assumption B.2, for any fixed beliefb A(S) and any action distributions q. q' on A, have
Ia I(bb a)- Ea- I(b, a)l A -''1/V
where A = log 7 and |1q 1|rv supAÇA (A) (A) denotes the total variation norm.
B.6 PROOF OF PROPOSITION B.
Proof. From Definitions B.2, B.3, and B.4, we have:
(b( = (.)))( (b), a:) (b,).
(8)
Let 0r ~ T(- b.) and oz ~ (- bt). Leveraging the results in Lemma B.1 and B.2, we bound the difference in expected informativeness:
Eu;; [I(b; at)l Eu, [Tbb, a:)l
(9)
< Emg [I(D) at)) - En, [I(b; ae]] + En, [Ibb,, at) Ea, (bee, ac)
(10)
A (-(( b(), T(- (())) + La b(b;,bb)
(11)
S (AL, + Lo) de.
(12)
From Assumption B.1, we have:
Eag [Ibb; at}) = (b() - E. [bi++i)) - Vo-
(13)
Combining with Eq. 12 yields:
En, [I(b, 0t)] S vo + (A + Ln)ck.
(14)
Since d, S 2, we obtain:
Eg, [[be, at)l $ wo 2(AL, + Lb) K.
(15)
Now, from Assumption 1, (b..) 2 Uo. then:
co(b:) mn V (b.) co.
(16)
Substituting into Eq. gives:
Po(b:) < K - (ma cc).
(17)
Thus, if (bb))) > (K + co) ma and (bb) 2 Un (i.e., ((b.) 2 U), then Po(be) 0, meaning:
E(U(be+1) bx) 2 V(b.).
(18)
Since ca(-) is lower-bounded by function that is nondecreasing in V (Assumption 1), this argument applics inductively for all 2 to, confirming the supermartingale property and the stalling behavior.
19

---

Published as a conference paper at ICLR 2026
B.7 PROOF OF PROPOSITION B.2
Proof. For simplicity, let V, := (b,, and V :- (1(2)). From the definitions of agent progress
Pa, (b) and update error co (b), we have the one-step expectation:
E FF] = Vt - Baiaz( b.) (Zb,, at)] + co (b.)..
(19)
For the oracle, it holds that:
2 Fi] = Emg ( (1bb [I(b],
(20)
Subtracting these two equations yields the fundamental drift identity for the gap 4, = V 1.:
B|A(+1- - A, | FF|] = (Eu; [I(b], at)] - Es, (Ibb at)l) + (bt).
(21)
From what have been shown in Eq. 12, we have,
Ea; (b) a7]] Eg, [b,, ai] & (ALx + Lo)ds S (AL + Lb) =: B.
(22)
Substituting into 21 gives:
E|A++1 A, Fi] 2 - B + co(b)).
(23)
The strengthened Assumption implies:
(b(b)) 2 mo - co = mo + w/) Co.
(24)
Substituting into 23 yields:
E|A.11 - D. Fi) 2 m0 + (mg V (co + B))
(25)
Rearranging terms:
B[A++1 Fi) 2 + m,) + (moV; -coo + +B))
(26)
By the law of total expectation, we have,
E E
[i.] > E + mm) m. + (mo (co + B))
(2
E[A++] 1+1 2 (1 + ma [[] + mo (co + B).
(28)
Iterating this inequality gives:
B|AT| 2 (1 -a +
(1 + -0)))=))-+--- mm v (co + B)]
(29)
As assumed in the proposition, there exists μ > 0 such that for all & 2 1, V 2 i almost surely. This implies E[V 2 μ. Then:
E [ - (co + B)] 2 map - (co + B) =: 6.
(30)
Substituting into Eq. 29:
T-1
E E| 2 (1 (m+ T-
A + 8
(1)m)) ) T-1-k
(31)
k=1
(1 + ma)
+ +1+ma) T-1
(32)
mg
We now show that [r] exceeds U finite time. Recall:
[ = [[Ar]+ 2 E[Ar].
(33)
A sufficient condition is therefore:
(1 + mo A + 8 (1 mo)
mo) mo U.
(34)
mg
Since 5 > 0 and 1 + ma 1, the left-hand side grows exponentially with T. Thus, for any U > 0,
there exists a finite such that Eq. 34 holds. Specifically, we have:
mnU + 5
(1 + mo) m > 11 8
(35)
Taking logarithms yields the explicit bound:
mnU 8 mm +6) ]
T T> 1+
log
(36)
log(1 + m2)
This completes the proof.
20

---

Published as a conference paper at ICLR 2026
B.8 PROOF OF THEOREM B.1
Proof. We decompose the advantage estimator: A. = Pre(()) + Tail(t), where
fs-t-1
T-t-1
Pre(t)=
Σ
plots
Tail(t) = Σ
g and g = 7X.
==
j=fs-t
For any l < Ls. the TD-error 65: >V++1 Vi (since 1% = 0). Because V& E [0, 1),
Elós Fi]
>B[V Fu) V4 5 07
Taking full expectation and summing over the prefix yields:
E[Pre(t)] - pSp (t).
(37)
We split the tail into the main part and the terminal step:
T- 2-2
Ttil(t) =
Σ
o² dets +o --- +-=- 87-11
J j=ta
Tail (t)
For the terminal step, 5T-1 Rr Vr- B(ör-1 Fr-il = 0, and thus --g1-t-18 81-11 0.
Now, fix & E (ts...., - 2}. We analyze E E'dx FR]:
E F&] = 7E[V++1 Vv FFl + (7 - 1)
(38)
S > V/2
FF] (since V 2 and S 0).
(39)
By the calibration assumption, V 4+ Vv - g(bx+ (8 g (((bx((')))) )). Since is differentiable with gf 2 KV 0, and since [[bx++ (8" ) (a(s)) Fx] Pp by assumption, we have:
E|[++1 Vx Fx] E'g' (Ex) ( (bx+1 (0001((1")))))) bb(ss))) Fi
(40)
I Ebx++ - ba(s)) Fi] (since g' (Ex) 2 kv)
(41)
Kvpbb
(42)
Therefore, E[8x Fi] S -YKY Pb. Taking full expectation and summing over the tail gives:
Tail (1) S - S (t)-
(43)
Combining Eq. 37 and Eq. 43 proves the main bound Eq. 4. The inversion condition Eq. 5 follows directly by requiring the right-hand side of Eq. 4 to be negative.
From what have been proved above, have:
E [[A = [t)re(t)] + B[Tail(tt] s E[ĀT" - Pr (t).
Rearranging terms yields: E[AM"] B[A.] + po (t).
B.9 PROOF OF PROPOSITION 1
k) that lies entirely outside the BTR, so that gs > p for all s E + 1,..., k}. By definition of the biased beussian-noise model,
Proof. Fix any k-step segment (L + I,...,
.....,,- L
have d, = 9s + 8, + E., where [3,| < M, ~ N' o²) independently across s. On a step s outside the BTR, local false truncation event occurs when the proxy falls below the threshold Amin (cf., Def. 2)
despite gs 2
E, = (d, Amin } {g. + + + < Aminj.
Using gs 2 p and |8,1 < M, obtain ge 3, 2 p M. Hence
Pr(E.) Pr(g. 3, Amin) & Pr(p M Amin) Pr (Ex Amin (p M))
21

---

Published as a conference paper at ICLR 2026
Define the margin p- M Amin- By the assumption Amin p M, we have 0 and therefore,
Pr(E.) I Pr(E, <a).
Since &, ~ (N0, d the standard concentration inequality gives, for any >> 0, we have
Pr(f: - aa) I exp
a²
20
Applying this with a = p- M Amin
yields
Pr(E.) v exp( (p- M Amin)
(44)
2
Recall that the T rule with window size triggers at the end of k-step segment only if all steps in the window classified "non- nonn-inorrmttiv". For non-BTR segment (I.1...., t k), activating T therefore corresponds to the intersection of the k single-step events E Ex4x:
I+-k
E++1
n
E.
s=t+1
By independence of the noises (E.) across and because each &, is determined by &s, we have
Pr(Eet1,...ttt)) .+++) II Pr(E.).
s=f+1
Applying the single-step bound (Eq. 44) uniformly yields
Pr(E(41....14e)) I exp k(p - Aminn i
To ensure that the false-truncation probability on any k-step non-BTR segment is at most d E (0, 1), suffices to require
exp( k(p- M Amin
S
20
which is equivalent to
1 (p M Amiu)² 2 20 log (1g))
B.10 PROOF OF LEMMA B.1
Proof. We begin by showing the closed form of one-step informativeness I(b, a). Combing Defini- tions B.1, B.2 and Eq. 3, we have,
I(b. a) (b) En-o(,lat
(b. (V,(B(b,a,)))
(45)
O 8* a;,a(5))))
log b(+*)
Bono a)
- log
(46)
Pb(o a)
Eono(Horaa) log Olo s*)a)
(47)
Po(o a)
For fixed a, Let P(o) = 0 a), and Qo(o) | a)== ((s) s, s, Then have:
P(o) Qu(o)
I(b, a) Rawp log
Er(log P(o) Er log Qo)))
(48)
constant in
By the non-degeneracy assumption (Assumption B.2), O(o | s, 2 n for all reachable o, Consequently, for any belief and any observation o,
Qo(o) = Σ b(s)0(o s,a) >
b(s) 1)
(49)
#ES
SCS
22

---

Published as a conference paper at ICLR 2026
Thus, Qo(o) 77 and Qo) 77 hold for all
For any 2, y 2 17 > O, we have the clementary bound
100 yl
x y
llog x log vl
di
(50)
min (in (x,y)
7)
Applying this with Qo(o) and Qu (o) yields:
log (o) log Qu v 1Q((o) - Car(o)
for all o.
(51)
Taking expectation under P and properties of expectation, we get:
[Ep[log Qo(o)] Ep [log Qw (o)] < Ep [| log (x - log Qu())]
(52)
[Q((o) - Qu
< Ex
(53)
"
1 < 1Qu Qu
(54)
"
Since I(b, a) const - Epllog Qx(o)), it follows that
[I(b,a) I(b', a)l _ QQQ Qmlli-
(55)
")
We have
Qx(0) - Qw(())
(b(()) (o)))(o 8,0) S Σ
(b(s) - 8(() 8,
(56)
1ES
8ES
Summing over o gives:
|Qs Quil =
Σ
1Qu(0) - Qu(0)) - ΣE Σ b(s) - (((s) s,a)
(57)
DEO
nEO FE
= Σ
lb(s) bs(s) Σ 0 s,a)
(58)
SES
DEO
= 11b - b'
(59)
Combining this with Eq. 55 yields the pointwise bound:
[[bb a) - = b'll1.
|6
(60)
For any action distribution by the lincarity of expectation:
IEu,,g(b, a) - I(b, a)l S Eang [I(b,a) a) - I(b', a)] S Eany
- 11b - 61111
= -1bb 6'111.
(61)
B.11 PROOF OF LEMMA B.2
Proof. For fixed b, define f(a) == I(b, a). We first show that f is bounded. By non-degeneracy. O(o s,a) 2 for all o, 8, Consequently, for any
Po(o a) Σ ((s)0(o s. 2n and ((0|st,a) 2 n.
By Eq. 47, we have
0 = -E log O s'.a)
S (B = - log
po(o a)
Hence |f v logg n, where || bo denotes the supremum norm || == sup (f())
The result now follows from standard property of the total variation norm: for any bounded function f.
Eurgy (a) Eurgg' f(a)) < al |1g g'lluv (-logn) -//lvv
23

---

Published as a conference paper at ICLR 2026
C EMPIRICAL VERIFICATION OF THE THEORY
C.I EMPIRICAL. VERIFICATION OF ASSUMPTION
A direct empirical validation of Assumption 1 is inherently challenging, neither the oracle Bayesian update B" nor the LLM agent's internal belief state by is directly observable. To address this, we design controlled study the PE task that enables practical and theory- approximations of all relevant quantities Uo, ma, Co
(i) Approximating the potential V. Each interaction round in PE provides the model's 's explicit estimate of the latent user user-preference vector, denoted by we (here we use w to denote the preference vector, same in main text). We define
d(wr) = l| - w w=
and use d(ww:) to serve as an operational proxy of the potential, i.e., v, : d (wx)). This proxy preserves the essential properties of the theoretical potential: it non-negative and equals zero if and only if the task is solved. Note that w* is not available to the agent.
(ii) A Approximating the oracle Bayesian update B* Although the true Bayesian posterior is inaccessible, we construct a principled surrogate update rule B following standard linear- linear-Gaussinn Gaussian update. Specifically, given the model's query == (A, B) where A, B denote the movie pair to compare and the observed feedback oz, we define
o
w = B(we, ae, 0z) wt + K, mi (o. m, we),
K;
m + 02
Here, my E Rd is the aovie-attribute difference vector for the pair of movies selected by the LLM's query, i.e., m, aAt attr (B). The binary observation or E (-1,+1) corresponds to the user's ' response and is given by o, = sign (m, w*). The terms of and o² denote prior and observation noise variances; following standard practice, set both to 1.0. In contrast, the L.LM agent updates its estimate via
W++1 == Bg(w,, ae, 0z),
which reflects the internal belief dynamies induced by its parameters A.
(iii) Constructing observable samples of the update-error term. Using the above approximations, instantiate the update-error quantity via
(b, == d(w(+1) - d(w(+1) ~ cb.b.).
We totally collect over 150k samples of (v,,, (b,)) using rollouts from the Qwen-2.5 series models, which provide sufficiently rich empirical basis for inspecting the assumption.
(iv) Estimating ma. Un. co via lower-envelope fitting. Since Assumption concerns only lower bound relationship, we estimate the empirical lower envelope using a principled two-step procedure:
(a) Lower-envelope extraction via binning. According to Asmp. 1, belief deviation of the LLM agent will be further amplified once it progresses into a high- region. Hence we empirically select a proper value of Up such that large belief deviations are observed. We then partition the range [Uo v maxx into B equal-width bins [1/--1, 1/b,, where Vmax represents maximum observed v in data. For each bin b, we compute:
Tb = EE V. E bin b]. Yb = Quantileo. (ca(be) V,, E bin b),
where yb captures the empirical 1oth-percentile lower envelope within the bin.
(b) Linear estimation on the active region. Restricting the active region V U. we fit linear model to the extracted lower-envelope points:
Yb Rs mh Ib dg-
The resulting (o) provide empirical estimates of the coefficients in Assumption
We visualize the whole procedure and the fitted linear model in Fig. The above procedure yields an interpretable empirical characterization of the lower-bound growth pattern required by Assumption 1.
24

---

Published as a conference paper at ICLR 2026
Bitrec lEAR amoeloze lidd lowar bond U
Bireed lover Fiteed ower bound
3-
175 230
(a) Qwen-2.5-7DB 1wen-2.5-7B-Instrueet
(b) -nstructt
Figure 7: Empirical visualization Assumption on the PE task. The dashed vertical line marks the empirically determined threshold Ug. Blue points show all samples, while orange points represent the binned lower envelope, obtained by partitioning the range of (v, 2 Uo into equal-width bins and taking the 10th percentile of êG within each bin. The red line a linear fit these lower-envelope points. For (a), we empirically select Up 10, and obtain the linear fit: čg 0.0969 v 3.0478. For (b), similarly, we select Up and obtain 0.4655 x V 1.5158.
Tuecaton
-1.0
-0.70
Tiven Advortage imear)
Theem Mdweeagg Imsean
0.775
-122
-0.85
0.50
1.44
-0.95
-1.00
300
200
300
400
500
100
200
300
400
500
Toloon Indcx
Totlon Indox
(a) PE
(b) CD
-1.0
Taken Avvrrtage (Peee))
TNlm Adveraage
lend)
1.3
(ead)
Bime (An Tumnotor
Tame rece Tramaation)
Tarn nnl T5
-1.5
Tama redo Transation
500
Tolan Indlex
Tikken Indder
(c) Tail Length (PE)
(d) T Truncation Strength (CD)
Figure 8: Empirical verification of Theorem 2 and Corollary 1. (a-b) Without truncation, carly-token advantages exhibit clear negative drift, while T consistently elevates them across PE and CD tasks. (c) Longer uninformative tails (higher maximum interaction turns, from 6 to 15) cause stronger suppression of early advantages. (d) Stronger T3 truncation (smaller k) yields cleaner, less-biased early advantages.
C.2 VERIFICATION OF THEOREM 2 AND COROLLARY 1
To empirically validate the credit assignment pathology formalized in Theorem and the mitigating effect of T stated in Corollary 1, we designed a controlled experiment to isolate the impact of the uninformative trajectory tail on the advantage estimates of preceding exploratory actions.
Experimental Setup. Given fixed policy optimized via standard PPO paradigm, generated two sets of rollouts: one using the standard method (wo Truncation) and one using the T truncation rule (w/ Truncation). To precisely measure the contamination effect of the uninformative tail without the
25

---

Published as a conference paper at ICLR 2026
1.25
Theen Advartage imesn)
100
Teaccation
Them Advanagg (mmeen)
1.00
0.75
0.50
0.0
-1.0
-0.25
-0.50
Tokeen Index
Thken Index
(a) PE
(b) CD
Figure 9: Empirical verification of the effect of false positive on (a) PE and (b) CD tasks. More aggressive false-positive truncation (larger a:) systematically reduces the advantages of early ploratory actions, reflecting the removal of positive future return contributions.
confounding factor of successful outcomes, we filtered and exclusively analyzed rollouts that resulted in a failure (i.e., a final reward of 0). We then computed the Generalized Advantage Estimation (GAE) for each token in the first 500 tokens of these failed trajectories. Finally, calculated the mean advantage at each token index across all rollouts within each condition.
Main Results. The results across the CD and MR datasets presented in Fig. 8a and 8b. In the w/o Truncation condition, the mean advantage of early tokens is suppressed, while applying the T truncation rule (w/ Truncation) consistently elevates the mean advantage of the early tokens. This suggests that the uninformative tail inside the BTR introduces a negative drift that systematically corrupts the advantage estimates of the preceding exploratory actions, and shows that the T early- truncation mechanism effectively alleviates this issue, preserving the integrity of the gradient signal during policy optimization.
Effect of Tail Length and Truncation Strength: We further vary the effective tail length and truncation strength. As shown in Fig. 8c, longer uninformative tails in the w/o Truncation setup led to more severe suppression of early-token advantages. Fig. 8d exhibits that stronger (more aggressive) truncation in the w/ Truncation setup resulted in higher and less corrupted advantage estimates for the preserved trajectory prefix. This is consistent with the theoretical outcome of this work.
C.3 COMPLEMENTARY ANALYSIS OF
TRUNCATION AND ITS IMPACT
Since T leverages observable surrogates of the BTR to construct the truncation condition, the frequency of false positives is empirically limited. However, premature (false-positive) truncation can, in principle, remove useful exploratory steps and harms optimization. We provide both an analytical discussion and diagnostic experiment.
Analytical perspective. Under standard GAE decomposition, the advantage of an early token aggregates future TD-errors: A, I (*A) u-t & Theorem characterizes the "uninformative tail" regime which the expected TD-errors ò, are negative; failing truncate such tails induces downward drift on A,. A premature truncation corresponds to the opposite scenario: the trajectory has yet entered the belief-trap region, and truncating at this point may discard future steps whose TD-errors 6g could have been positive. Consequently. Ag may be reduced due to the loss of these potentially informative and cowarr--onntibutingg steps.
Diagnostic experiment. To make this effect concrete, we conducted controlled diagnostic experi- ment. We fixed a trained vanilla-PPO policy and generated a set of full rollouts. To focus our study on the effect of false positives, we filtered the rollouts to those with final reward of 1, ensuring that the retained trajectories contain genuinely informative future signals and do not enter the BTR. On these trajectories, trajectory is forcibly truncated at turn (the maximum allowed turn is 10). With probability 1 1-6 a, the trajectory proceeds normally completion. This creates clean setting in which any degradation can be attributed solely to premature truncation. For each carly-stage token position 1:500, we computed the mean GAE advantage across rollouts for different values.
simulated false- false-positive truncation as follows: With probability
the
26

---

Published as a conference paper at ICLR 2026
Results. We present the results for the CD and PE datasets in Fig. 9. As expected, more aggres- sive false-positive truncation systematically reduces the advantages early exploratory actions, confirming that premature (false-ponl collse-positvee truncation negatively impacts credit assignment.
D COMPLEMENTARY EMPIRICAL ANALYSIS
In this section, we present complementary experimental results to provide further insights.
D.1 RATIONALE FOR SELECTING BINARY SIMILARITY THRESHOLD IN PE
In the PE task, the reward is derived from the cosine similarity between the model-predicee preference vector and the ground-truth preference. We convert the similarity into binary reward by activating only when the similarity exceeds a prescribed threshold. To understand the effect of this threshold, we evaluate several settings (0.85, 0.88, 0.90, 0.95] using Dwwen22.-77B-Instrile trained with PPO.
Table summarizes the results. Lower thresholds (e.g.. below 0.80) cause the reward to activate (i.e., a reward) almost continuously, which diminishes the discriminative value of high-quality predictions. Conversely, very high thresholds (e.g., above 0.95) make activations extremely rare, preventing PPO from learning effectively. Mid-range thresholds between 0.85 and 0.90 consistently yield stable training dynamics and strong downstream performance. We use 0.88, which lies within this empirically robust region, in the main experiments of the PE task.
Table 5: Effect of the binary-similarity threshold on PE performance (BinarySim). All results use Qwen-2.5-7B Instrrcct trained with PPO.
Threshold
0.85
0.88
0.90
0.95
PPO (vanilla)
55.33
42.00
33.67
4.33
PPO + T
63.00
49.00
37.67
3.67
D.2 EFFECT OF REFERENCE-SET SIZE ON REDUNDANCY-INDUCHH STALLING
Empirical Verification. To further examine the role of redundancy in inducing belief-trap regions (BTR) in the PE task as mentioned in Sec. 3.3.2, we investigate how the frequency of truncation varies with the size of the reference set S. evaluate truncation ratios across different reference-set sizes S E (10, 15, 20, 25, 30} for the Qwen-2.5-Instrruet model family, Table 6 reports the results.
Table Truncation ratio (%) under different reference-set sizes S for Qwen-2.55Insttee models. Larger S corresponds to more potentially redundant comparisons.
S
10
15
20
25
30
3B
41.67
39.67
46.67
44.33
50.00
7B
50.67
53.67
54.00
56.67
56.67
14B
23.33
30.33
27.00
33.00
33.33
32B
38.00
39.67
39.33
50.33
46.33
Across all model scales, the truncation ratio exhibits general upward trend as increases from 10 to 30. This pattern suggests that larger reference sets may introduce additional noisy or redundant pairwise comparisons, which in turn make epistemic progress harder to achieve and increase the likelihood of entering redundancy-induced BTR.
D.3 T3 PE-LIKE TASKS WITHOUT ACCESS TO THE GROUND TRUTH
The proxy rule for the PE/MR task described in Sec. 3.1 relies the ground trunt preference vector u* However, the truncation mechanism does not require access to the ground-truth Instead, we employ fully helief-driven truncation rule that relies solely on the agent's internal preference estimates. Let
27

---

Published as a conference paper at ICLR 2026
de denote the model's predicted preference vector at round We define epistemic-stallngg signal via a k-step moving average of update magnitudes:
(
stall,
I
10 1 0yll2 j
Σ
<E
(62)
where & the Bidding-window length and : a truncation threshold. The threshold obtained from the empirical distribution of the k-step moving-aversy updates A! computed from offline rollouts. Specifically, E is set chosen quantile (e.g., 60%, 75,, 85%) of this distribution, ensuring that the criterion is entirely ground-truth-free. A trajectory is truncated once Eq. 62 is triggered, i.e, the agent's s belief updates become small for consecutive steps, indicating epistemic stalling.
Table 7 summarizes the results on the PE dataset. Despite the absence of oracle information, the belief-based truncation retains strong performance, closely matching or surpassing the oracle-based T reported in the main paper.
Table 7: Performance T on the PE task without access to v*. Thresholds e correspond to quantiles of offline A statistics. BinarySim accuracy is reported for -wen-2..--BBIImm vanilla a and 73-gt: represent vanilla-PPO and 8 in the main text (with access to the ground-truth v*), respectively.
trained with PPO.
Quantile
60%
75%
85%
vanilla
T3-gt
0.18
0.28
0.36
BinarySim
44.33
50.67
49.00
42.00
49.00
D.4 EXPLORATION ADAPTIVE T3 TRUNCATION RULE
Adaptive T via online threshold selection. Motivated by extending T beyond fixed, offline- chosen thresholds, we further investigate adaptive variant in which the truncation threshold evolves alongside the policy. For the PE task, the belief- based stalling criterion is employed the same as Appendix D.3 and Eq. 62 with k 4. To obtain adaptively, every 6 training steps collect batch of fully untruncated rollouts under the current policy and compute the empirical distribution of
the s-step moving average update magnitudes A fixed quantile a of this distribution:
The threshold is then updated according to
Quantile (14(*)) (k)
Jonline)) (K)
This mechanism yields dynamically adjusted truncation threshold that tracks the scale of the model's ongoing belief updates.
Table 8 reports the performance across quantiles a. The results exhibit non-monotonic dependence on a. Notably, at a 0.6, the adaptive variant achieves substantial improvement, outperforming both the PPO baseline and the oracle-based 3 result reported in the main text. These results highlight the potential for extending the a principle to adaptive thresholding, and we leave more in-depth exploration to future work.
Table 8: Adaptive T on the PE dataset. The threshold is updated online from the a-quantile the current A (k) distribution.
0x
20%
40%
60%
80%
90%
vanilla
T3-gt:
BinarySim
43.67
44.33
60.33
43.67
39.67
42.00
49.00
28

---

Published as a conference paper at ICLR 2026
E POTENTIAL FUTURE WORK
E. MORE GENERAL- PROXY DESIGN
Task-agnostic surrogate signals for epistemic stalling. In main experiments, since the structure of hypothesis spaces and notions of progress differ across tasks, instantiating T naturally leverages task-level structure define observable proxies that track epistemic progress. However, guided by the T principle, we can further reduce the reliance on task-specific knowledge via utilizing general-purpose truncation detectors. We explore two broad, task-agnostic families of surrogate signals follows.
(i) Semantic redundancy signals. In multi-turn LLM-agent settings, epistemic stalling frequently manifests semantic redundancy, where the model repeatedly issues circular queries revisits previously resolved informational subgoals, as shown in prior studies (Zhou et al., 2025; Yuan et al., 2025). Such redundancy is often detectable via embedding similarity, clustering, etc.
Building on this intuition, we have several successful explorations in this direction: i) In the SP task, the truncation based on question-semantic similarity (cf., Sec. 3.3.3) yields consistent performance gains. ûi Moreover, for tasks with continuous latent spaces, such as the PE task, tracking the conver- gence of the model's internal preference vector estimate provides an effective proxy for redundancy: truncation is triggered when the estimate ceases to change meaningfully (cf., Appendix D.3 and D.4). This convergence reflects an epistemic "stall" analogous to query redundancy in dialog scenarios such as the SP. Our experiments show the effectiveness of these redundancy- -baeed surrogates.
(ii) Internal state signals. Recent empirical analyses suggest that hidden representations of Trans- former and LLM models could encode intermediate judgment or reasoning states (Lu et al., 2025; Zhou et al., 2024). Although the precise hidden-state signatures corresponding to epistemic stalling re- main an open question, characterizing such patterns (e.g., consecutive high similarity of hidden states) a promising direction for future work. Such signals may be especially valuable in open-domain tasks where a structured hypothesis space is not readily defined.
F SETUP DETAILS
F.1 DATASET DETAILS AND PROMPT TEMPLATES
In this section, we present more details for the datasets and tasks evaluated in this work. See dataset statistics in Table 9.
PiuzltionPuzzles (SP). This task introduces a challenging active reasoning task where the LLM player must uncover a coherent narrative from an initially puzzling scenario. Each puzzle begins with a brief, paradoxical statement. The solver interacts iteratively with a judge by asking binary yes-no questions, gathering feedback from the judge constrain the solution space. The goal is to formulate complete and plausible explanation that resolves the apparent contradiction. We directly use this dataset from the AR-Bench (Zhou al., 2025). In our experiments, we utilize 1BB
model to provide the interactive feedback.
The prompt template for the SituationPuzzles dataset can be seen in Fig. 11. For SituationPuzzles. put a specific puzzle to solve into [puzzle} } of the prompt. The prompt template for the judge LLM shown in Fig. 13. The judge will receive (surface) and [bottom) to understand the whole puzzle, and give yes-no feedback according to the player LLM'' question.
GuessNumbers (GN). Adapted from the original dataset proposed by AR-Bench (Zhou et al., 2025) which the player must crack a 4-digit secret (digits unique in 0-9), newly constructed GN(a, 6) a series of reasoning tasks that involve the LLM agent's interactive deduction with external sources: the target is a-digit number, where each digit sampled from set of unique symbols without repetition. This yields P(b, a) = (b - a)! possible targets.
At each step, the LLM agent makes guess and receives structured feedback in the form of xAyB, where x denotes the number of digits that are both correct value and position (denoted as "A"), and y denotes the number of digits that are correct in value but placed in the wrong position (denoted as
29

---

Published as a conference paper at ICLR 2026
"B"). The agent is expected to actively perform reasoning based accumulated observations and interact with an external source to efficiently reduce uncertainty and locate the correct answer.
To control for randomness in the first move, which plays minor role in evaluating the LLM agent's ability to understand and update based on observations, we fix the first guess to a deterministic number that differs from the answer. This means we need (a, b, go. Zo; yo) specify question for the LLM player, where 9o denotes the initial guess, and (zo. yo) denotes the corresponding initial feedback of the form xgAyoB.
We group data items by their tuple (a, b, zo, yo), since items sharing the same (a, b, To, 10) corre- spond to tasks with similar uncertainty reduction dynamics and reasoning logic patterns. Specifically, our constructed dataset covers all data items of the following sub-groups: (3, 4, 0, 3). (3, 4, 2, 0), (3, 4, 1,2), (3, 5, 1, 2). (3, rations are carefully selected to ensure diversity in task complexity: varying (a, b) controls the size of the hypothesis space, while varying (zo, Uo) shapes the initial reasoning landscape by introducing distinct patterns of partial evidence. Finally, we perform randomized train-test split over the obtained set for training and evaluation.
3). (3, 5, 1, 0). (3, 5, 2, 0). (4, 4, 0, 4). and 5, 0). These configu-
The prompt template for the Guess dataset can be seen in Fig. 12. For GuessNumbers, we need to first specify {num_d gits) } and (ummunniquss corresponding to (a, b) mentioned above, and then specify the initial guess in {initial.quesa/ }. and the resulting initial feedback in (initial feedback same pos} and {initial fee.back..eeccccc....k. pos)..
CircuitDecoding (CD). Adapted from Badola et al. (2025), in this dataset, each instance presents collection of unknown Boolean circuits, each taking a fixed number of binary inputs and producing binary output. There set of logical structures, and the player must identify which candidates correspond to the hidden circuits. To achieve this, the solver engages in multi-turn interaction protocol: each turn, the player must query one circuit with a binary input configuration of their choice, and receives the corresponding output. These queries serve as informative probes, allowing the player to iteratively eliminate inconsistent candidates and refine their hypotheses. The task requires strategic planning to maximize information gain under limited query budgets, and finally the solver must output the candidate indices of all underlying circuits. In our experiments, we adopt the prompt template shown in Fig. 10, where the LLM solver aims to figure out {num ci circuita (s) hidden ground-trum -ruth circuits from {nom candi dates } candidates specified as: {cand (tandidate_]e wad.aaallsssssrrrr..iss
set of ground truth circuits which are drawn from finite candidate
PreferenceEsstimatton (PE). Adapted from Badola et al. (2025), this dataset targets the problem of interactive preference elicitation, where the agent must infer latent user preference vector governing utility over movies. Specifically, each movie is associated with a list of attribute scores ($1, ... sn)) where n is the total dimensions of attributes. In this task, the user evaluates a movie as weighted sum of its attribute scores Σ W 8s, with the weights ( (w".... w;)) forming the hidden preference vector to be discovered. At the beginning of an interaction episode, the agent is presented with a set of reference movies annotated by their attribute values. At each round, the agent outputs both its current vector guess and a pairwise comparison query between two reference movies. The user provides feedback ("Yes", "No", or "Equal") according to the weighted sum scores of the two mentioned movies. Through multiple turns, the agent iteratively updates its estimate of the preference vector by reasoning over past user feedback.
The prompt template for the PreferenceEstimation dataset is illustrated in Fig. 14. The LLM player is given (len_seer) reference movies for raising pairwise questions, to iteratively refine its guess on the (len (len.attrinl aattributes bisiibuutdddddsiiioooee hidden user preference vector.
Movieeecoommendtiio (MR). Building upon the preference estimation setup, this dataset further evaluates the generalization ability of an agent's inferred user model. After completing several rounds of interaction mentioned in the PE task, the agent is tasked with recommending from set of unseen movies. Each unseen movie is described by the same attribute dimensions, but the agent has not encountered them during training or interaction. In the final turn, the agent applies its preference vector guess to score each candidate unseen movie, and is required to select the movie that the user is most likely to prefer as its recommendation. This task thus demands transferring preference inference to out-o--distribution recommendation, and evaluates reasoning consistency, robustness, and generalization in interactive recommender systems.
30

---

Published as a conference paper at ICLR 2026
The prompt template for this task is shown in Fig. 15. The agent is expected to leverage its estimated preference vector to make personalized recommendation from [unseen.movie.list)]
Table 9: Dataset Statistics in this work.
Train
Test
SituationPuzzles (SP)
400
100
GuessNumbers (GN)
1526
382
CircuitDecoding (CD)
1000
300
PreferenceEstimatiin (PE)
700
300
MovieRecommendation (MR)
700
300
F.2
BASELINE DETAILS
Here we introduce RL algorithms used our experiments. Formally, given an actor model no. the like-
lihood of response to a query x under the policy Ha is modeled as To(y)a) ar) = I| To x Vee).
Given query-response pair (x, y), verifier generates its reward r(z, y) [0.
Proximal Policy Optimization (PPO) (Schulman et al., 2017) employs the following objective for policy optimization:
lgl mün (wr(0) clip
21)
JPro (0) ED yo (1 (12)
+6) (63)
where the importance ratio of the token yr is defined as w. w.(() ne (3rs wece) the advantage A, of Yt is typically computed via Generalized Advantage Estimation (GAE) (Schulman al., 2015) with chrferal-difference errors, and E is the clipping range of importance ratios.
molyr (o.uec)
Group Relative Policy Optimization (GRPO) (Shao al., 2024) proposes computing the relative advantage of each response within group of responses of the same query using the following objective (omitting the KL regularization term):
(O) = Er, (M
min (O), clip (w,,, &)) Aies)
G
(64)
where f ~ Thue (-1z) and is the group size. The importance ratio wr, (0) and advantage A,, of token Fi,e are defined as:
To(Un,12, Ui (0) Te. (yr. )1,
A = r(æ, vi) mean (((a,,)))
(65)
std (fr(z, w))
respectively, where all the tokens in share the same advantage.
Group Sequence Policy Optimization (GSPO) (Zheng et al., 2025) extends GRPO by defining the importance ratio at the sequence level with length normalization, with sequence-level clipping, rewarding, and optimization. The objective is:
JaSPo (0) = RR
min (s:(0) clip(s:(0), -,
(66)
G
where
s(()) = (B(()(z))
1/1mcl
To(Vi.e| Vr,co)
= exp
log
To. (y)l2x)
Ter (Vr, Vi.ce)
31

---

Published as a conference paper at ICLR 2026
Input Prompts for the CireuitDecoding dataset
Welcome to the Circuit Deduction Challenge!
4A The Setup:
There (nom at reult t.s) circuits, (cl rous labels))
Each circuit accepes [nun_inpute) binary inputs (0 or and produces a single binary ourput or 1).
Each circuit drawn from a fixed candidate list of {aum, candidater possible logical structures, each associated with index: [candi date. list_ste]
#* Your Goal:
Identify which circuits from the candidate list correspood exactly to circuits fcircuit.labeld
Play:
You can internct with me for several turns determine the true underlying circuits:
1. At each turn, query one circuit with any binery input of your choice.
2. Use the specified format for your query. For example, query circuit A with inputs x0m1, ximo, x2 ask:
<t rtterct>> A (1, D, 1) </1 s/inten intaract.
3. You make
query each
will the binary output for that circuit the given inpat.
4. Ask strategic queries that maximize information gain. Your goal to minimize the number of turns by leveraging the feedback at step narrow down candidate possibilities.
#* Final Submission:
Once you are confident, submit your final answer by providing the indices of the identified circuits from the candidate list inside <azswer> answer>. For example, if corresponds candidate 13 corresponds your answer must <onswer>113, 6</ r2s
Please start with your first query.
Figure 10: Prompt Template for CircuitDecoding.
F.3 SUPPLEMENTARY IMPLEMENTATION DETAILS
Here we provide additional implementation details. The maximum number of interaction turns is set at 10 for GuessNumbers, 15 for SituationPuzzles, 10 for CircuitDecoding, 10 for PrefcrenceEssiimation and for MovieRecommendadte evaluation metrics: for GuessNumbers, the reward is Exact Match (binary {0, 1}, given only at the final step); for SituationPuzzles, the reward the F/-word / character score (continuous in [0, 1], computed against the ground-truth answer); for CircuitDecoding and MovieRecommendation, the reward is also Exact Match; and for Preferencesstimation. the reward Binary Similarity between the predicted and ground-trutt preference vectors. All rewards are provided only at the terminal step of each trajectory, consistent with the outcome-bar based RL setting.
For RL training, we define task ass-sspeciiic rewards aligned with their
Training for GuessNumbers and SituationPuzzles is conducted on single node equipped with 8 H100 GPUs, while CireuitDecoding and PrefreneEssimattoo/ooiieeeeee /ww.//iiiiieeomeenttiinoooo are trained on single node with 8 B200 GPUs, based on the implementations of Verl (Sheng et al., 2025). All training tasks are conducted for 200 steps with the actor model optimized using a learning rate of 1.0x 10-6. For distributed training, we adopt Fully Sharded Data Parallelism (FSDP), using BF precision throughout both training and evaluation. For efficient LLM rollouts, we adopt vLLM with tensor parallel size of 1. The rollout sampling uses temperature of 1.0 for Situation IituationPuzzles and 0.6 for GuessNumbers, and a top-p value 0.95 for all datasets.
For the PPO baseline, and y = 1. The KL divergence regularization coefficient 3 and clip ratio e are set to 0.001 and 0.2. For GRPO training, we sample responses per prompt, and the rollout parameters, KL divergence coefficient, and the clip ratio are consistent with the PPO setting. For the GSPO algorithm, we do the KI. divergence constraint, and the clip ratio fow and hhig set to 0.0003 and 0.0004, respectively, while others keep consistent with GRPO training.
Generalized Advantage Estimation (GAE) with parameters À 1
'https://docs. v1 wm.. www..ai/en/maae.eeeeeeeeeeo
32

---

Published as a conference paper at ICLR 2026
Input Prompts for the SituationPuzzles dataset
Let's situation puzzle gume. I'll give puzzle. You interact with several turss during the question phase reach the final answer. For cach tur, you will:
Review all provicus questions foodback.
Ask me yes-or-no question inside <interact> and </ </interact>.
answer your latest question "Yes", "No". "Unknown".
Repeat the process until you coulident the answer.
you believe you have confidently determined the correct solution, present your answer inside <answer> and </anower>.
here's the puzzle: Puzzle: (puzzle)
Figure 11: Prompt Template for SituationPuzzles.
Input Prompts for the GuessNumbers dataset
Let's play a number guessing game. The rules are as follows: have a secret (numadi git J-digit number mind, composed digits from to (num.uni ques], }. with repeated digils. You will take guessing the number, using leedback after each guess tn progressielly narrow down the possibilities For each turn, you will:
Review all provious guesses and feedback.
through your reasoning process inside <thinks </think>, reasoning
how your beliel about the the and in the correct positions, and how
secret mumber evolves based on the accumalated evidence.
Make strategic guess inside <i <inters <interact> and </ s/interact>, based on your current belief.
Receive feedback of latest guess describing:
many digits
many digits are present in the answer hut in the different positions.
If you believe you confideutly found the conrect number, present your Repeat process until you confident in the answer inside < <answez> and </ c/asswe>>.
is your turn:
<Lhink>No knowledge, with random guess that covers diverse digits gather information </think> <interact>> (initial Lanitialgueess>> ) /ntterace>>
The feedbback of your latest guess: finitial feedback same pos) digits are present in the answer and in correct positions, but different positions.
[init hel..eeebaacc.co Cf.pos digits present
Now it is your turn:
Figure 12: Prompt Template for Gumberr
33

---

Published as a conference paper at ICLR 2026
Input Prompts for the Judge LLM in the SituationPuzzles dataset
You the referee of game where players shown <aurface> you the <Bottom>. You need understand the entire story hased on both the <Surface> and <Bottom>. Players will ask questions based on the <Surfaces and judge whether their guesses correct. strictly adhere answering only three specified responses: Yes, No, Unknown, without explanation.
#* Judging Rules
the given If the player's question contradicts the given story. Please ooly answer "No" without any explanation.
player's question
toottom:: Please only answer "Yes" without any explanation.
<Surface> and <Bottcm>, and camnot be deduced through reusoning: Please only answer "Unknown" without any explanation.
the answer to the player's question cannot found
the player directly ask for answer, pleuse only answer "This is not question, please progose your next question." the player does propose question question that for solve puzzle, please only answer "This question, please propose your next question."
#* Important Notes
Fully understand the cause. process,
of the entire story, make logical inferences.
2. If conclusion cannot be drawn from the provided story or through reasonable inference, answer "Unknown". Strictly adhere answering
Unkmown. provide additional explanations. 4. Carefully check whether the player ask for the answer, if player do so, please only answer "This not question, please propose your next question."
only
specified responses:
Examples
#nd Example I: The Hiccuping Man
<Bur Cace>
A walks into bar and asks the bartender for glass of water. The bartender suddenly pulls out gun and points him. The man smiles and says, "Thank you!" then calmly leaves. What happened?
The man had hiccups and wanted glass of water to cure them. The bartender realized this and chose to scare him with gun. The sudden shock, so sincerely the barlender before leaving.
hiccups disaapeeaedd
Possible questions and corresponding answers:
Does the man have chronic illness? Unknown A: No
Was the man scared
Q: Did the bartender want to kill the man? No
Did bartender intend
man? Yes
Q: Did the man sincerely thank the burtender? A: Yes
Question Content
#*t <Su=face>
[surface]
<Rott.om>
[botton)
Now, please judge the following player question:
(question)
Answer with only one of the three specified responses: Yes, No, or Unknown, without any explanation.
Figure 13: Prompt Template for the Judge LLM in SituationPuzzles.
34

---

Published as a conference paper at ICLR 2026
Input Prompts for the PreferenceEssiimtiio task
You movie recommendation Your to infer the hidden preference (wl,....w (wl.....[[[oonnttrrrrbr through interaction.
Setup:
You are given an seenn movies with [len abt butee) attrihutes (indexed I... attri butee):
[seen_movi pooovveeaamne ee_ample) }
satisfaction wa wwwen attributes "atte attributes] each in [0, 1]. The always
answers consistently.
## Interaction Rules (per round);
Redlect past feedback and reason about how changes your
preference
TTink about which attributes gained or lost importance.
Adjust your estimate strategically.
2. Output both your updated guess and new pairwise
the exact format:
<imteroct>>
w.w.....
Question: Would you prefer opt ion.1 over opt lon.2? </ //interact>
Guess must be commu separated numbers in [0.1].
ooptioonn on_1 opt oon_2 must movie only.
The user replics with one of: "Yos" (prefer opt Lon.l), "No" (prefer Lon.2), "Equal".
#A Final Stage:
Once you are confident
user preference several turns, output your preference vector
<answer>> wwwl w2, * (len.atle Ur libutes] ) </awwrr>>
Please Start with your first <interect> block.
Figure 14: Prompt Template for PreferenceEstimattio.
Input Prompts for the MovieRccommmendation task
Final Tum: Now you have reached the last turn. Instead asking a new question, use your most recent preference guess to score the following unseen movies and recommend the best
Junseen.novie.ll _iie)
Here is example how to proceed: Preference voctor (guess): 0.2.0.7,0.5
Example Unseen movies:
Movie_A [0.66....0.8
Movie B: [1.2.0.3.0.4]
C: [0.5.0.8.0.9]
Scoring:
Movie A -
0,770. -...0.9
Muvie.A
<an sarswr>Movie x</answer>
Your goal:
your latest preference the given unseen After scoring, retur answer within canswer> and </ </anawer>, The answer must he exactly one of the unseen movie names.
Figure 15: Prompt Template for MovicRecoommendation.
35


Reviews:
[{"review_id": "qRBlDfEAHd", "reviewer": "Reviewer_nLrS", "summary": "The paper targets multi-turn active reasoning with LLM agents, where belief tracking often drifts, which produces redundant actions that poison RL credit assignment. It proposes T3 - early truncation of rollouts once belief-trap are detected via simple proxies (e.g., non-shrinking hypothesis sets, repeated “unknown” feedback, off-manifold guesses, or declining preference similarity). T3 is claimed to be a drop-in rule for PPO/GRPO/GSPO that preserves credit for informative prefixes and removes noisy tails. Across five tasks, it improves stability and OOD robustness, increasing performance gains with fewer rollout tokens.", "questions": "- is the ratio of early truncation ever defined/formalized in the text?\n- what are variables $b, a, o$ in fig 1?\n- How would section 2 choose ($k, \\alpha, \\beta$)? For instance, can we map $d(H_\\tau, H_{\\tau + 1})$ to an empirical estimator of $\\Delta \\Psi$ and pick $k$ so that the false-truncate probability is < δ (with a concentration bound)?\n\n- Why does GRPO+T3 still collapse later in SP? Is this due to proxy false-truncation starving exploration, or value-function drift? Any mitigation (pro\n\n- for binary similarity threshold 0.88 (PE): why 0.88?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "the $T^3$ framework provides a useful lens to connect information gain with progress in active reasoning (though its usefulness is a bit questionable, see more below), and the proxy-based T3 is simple to implement and, empirically, shows consistent gains across tasks/algorithms", "weaknesses": "- while the BTR lens motivates truncation, Def. 2’s T3 condition does not give actionable guidance on setting $ k, \\alpha, \\beta $ beyond \"detect stalling\". The task-specific proxies in Sec. 3.1 are essentially heuristic and could be understood without Section 2. In other words, the T3 condition does not concretely guide how to pick the metric $d(\\cdot,\\cdot)$, the window size $k$, or the threshold $\\Delta_{\\min}$, nor does it yield principled settings for the concrete proxies (\"unknown\" counts in SP, hypothesis-set shrinkage in CD/GN, similarity trends in PE/MR).There is no quantitative link between $ \\Delta \\Psi$ and $ d (H_\\tau, H_{\\tau +1} ) $; can the authors articulate decision rules (e.g., choose $k$ to control an estimated upper bound on false truncations via concentration of an estimator of $\\Delta \\Psi$)?\n\n- as to the two theorems established\n  - assumption 1 appears a bit strong: it posits systematic error growth with uncertainty across all actions/observations, yet the experiments do not validate (or estimate) $m_\\theta, c_0, U_0$.\n   - the extra constants/conditions ($\\eta$, $L_\\pi$-Lipschitz policy, “non-degenerate observations”) appear task/model-dependent and unverified; the bound $U$ seems not computable from observable quantities.\n  - theorem 2 motivates truncation but does not establish when actual policy gradients are meaningfully biased on the reported tasks.\n  - the sufficient condition shows the possibility of a sign flip, but does not quantify when it occurs for real critics/discounts. \n\n- as to training dynamics/stability: figures show improved stability in several cases, but GRPO+T3 still collapses later in SP. The rationale for pairing PPO (CD/PE), GRPO (SP), GSPO (GN) is not explained; differences in optimizer/entropy/clip may confound stability claims. If T3 is “algorithm-agnostic,” a matched comparison across all tasks/algorithms would strengthen its stabilization role.\n\n- as to OOD analysis consistency, CD OOD uses Qwen-14B (Table 2) while the main CD uses 7B; can the authors justify the switch? The narrative “too many references induce redundancy $\\Rightarrow$ more BTR” in PE is plausible but unsubstantiated; no proxy trajectory statistics (e.g., stall frequency, hypothesis-shrinkage rates) are shown to support the causal link.\n\n- authors need to be explicit about what default $\\alpha$ and $\\beta$, $k$ values are used for all 5 reasoning tasks. \n\n- regarding ratio of early-truncation vs. performance, the trends are confusing: authors report SP with $\\alpha=0.9$, where the truncation ratio converges to 1 and performance increases, whereas PE with $\\beta=0.8$ also $\\rightarrow 1$ but performance decreases. Similarly, for CD, higher ratios ($k=1,2$) appear to hurt exploration, unlike SP. Is there actually a consistent correlation between performance and the truncation ratio? Readers need the rationale and a principled conclusion, not case-by-case explanations. A controlled sweep reporting correlation between ratio and performance, conditioned on proxy type, would clarify when \"more truncation\" helps vs. harms.\n\n- in addition, regarding the summary claim in lines 418-420, can authors elaborate on this alignment: what theoretical quantities map to the chosen proxy thresholds and settings, and under what observable conditions (e.g., provable relation between $|H_t|$-shrinkage and $\\Psi$-decrease in CD/GN)?\n\n- some claims are casual: \"This suggests that distillation can effectively boost the belief-tracking capability under finite state spaces\"; that size/type differences “can be attributed to $m_\\theta$\" (lines 450-452) is speculative with current evidence. The first statement is based on one distilled LLaMA baseline on one task. It should be labelled as a hypothesis; broader study across tasks/models is needed.\n\n- $m_\\theta$ is introduced in Assumption 1 as a slope in the belief-update error lower bound, but later used to explain size/type effects qualitatively. can authors elaborate on it? How can we connect $m_\\theta$ to measurable quantities (e.g., proxy stall rates) and estimate it empirically?\n\n- minor presentation issues\n\n  - authors need to list default ($k, \\alpha, \\beta$) per task and how you tuned them\n\n  - define the “ratio of early truncation” in text (denominator, per-episode vs. per-step?)\n\n  - in Fig. 3a,d the y-axis should read “Response length”\n\n  - Table 1 should be placed near Sec. 3.3.1 analysis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "plufwDcsF2", "reviewer": "Reviewer_K1QM", "summary": "The paper formalizes belief deviation in multi‑turn, active reasoning and defines a Belief‑Trap Region (BTR) where epistemic progress stalls. It proves that long, uninformative “tails” can flip the sign of early‑step advantages (advantage inversion), then proposes T³, an early‑truncation rule triggered by simple proxy tests of stalled progress in the reasoning trace. T³ is a drop‑in wrapper for PPO/GRPO/GSPO and shows consistent gains across five interactive tasks, smoother learning, and shorter responses.", "questions": "No‑oracle proxies: How would you instantiate T³ on PE/MR‑like tasks without access to the true preference vector (e.g., entropy/disagreement/self‑consistency signals)? \nDirect test of Theorem 2: Can you measure early‑token advantages with/without truncation and show the predicted sign flips as tail length grows? \n\nAdaptive T³: Any online procedure to target a truncation‑ratio band (or validation return) and auto‑tune window/thresholds? Sensitivity to the SP judge identity/temperature? \n\nActionable feedback\n\nAdd a one‑page algorithm box for T³ (inputs, trigger, where truncation enters GAE/PPO).\nProvide equal‑length ablations to disentangle truncation vs. better credit assignment; include γ/λ/KL sensitivity.\nAdd a small calibration check: value vs. correctness reliability plots. Report token & wall‑clock to fixed reward thresholds.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Crisp failure‑mode lens. Clear POMDP framing with a truth‑anchored potential Ψ and a precise BTR definition; theory links tail length to advantage inversion (Theorem 2) \n\nSimple, general mechanism. T³ uses observable stalling proxies (e.g., non‑shrinking hypothesis sets, repeated Unknown judgments) so it can integrate with standard RL without changing optimizers \n\nThe proposed method shows broad, consistent empirical gains. Improvements on CircuitDetection (CD), SituationPuzzles (SP), GuessNumbers (GN), PreferenceEstimation (PE), MovieRecommendation (MR) with 14/18 metric wins; OOD robustness; works across model sizes/types.\n\nStability & efficiency: Learning curves are smoother and responses shorter, implying fewer wasted tokens.", "weaknesses": "Theory–practice gap. Assumptions (e.g., value calibration to truth probability; linear update‑error growth) are strong and not empirically validated against measured beliefs/values. \n\nPrivileged proxies. PE/MR use oracle preference similarity to trigger truncation, limiting deployability without ground truth. \n“Shorter is better” confound. Random truncation sometimes helps (Table 3), so more direct evidence of advantage inversion would strengthen the causal story. \n\nTuning guidance. Heuristics for window size/thresholds are task‑specific; no adaptive rule is provided. \nComparisons & scope. Limited real‑world tasks and reliance on an LLM judge in SP; baselines could be more apples‑to‑apples vs. open, strong models trained under similar budgets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GpqueP9K2G", "reviewer": "Reviewer_DC2y", "summary": "The paper presents a method to provide early stop during the RL if the hypothesis space did not shrink by a meaningful amount. The paper proposes $T^3$ (Truncating Belief-Trapped Trajectories), a mechanism that uses task-specific proxy signals to detect and halt trajectories entering the \"Belief Trap Region (BTR)\". Experiments across five diverse reasoning tasks demonstrate that $T^3$ consistently improves varied RL algorithms (PPO, GRPO, GSPO) in both final performance and token efficiency.", "questions": "1. In Figure 3, the first and fourth figures have different y-axis label than the second and third figures, why?\n2. If we want to extend this to more real world cases, how would $T^3$ fit in these use cases?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper provides a strong theoretical grounding for why RL fails in long-horizon active reasoning, formally characterizing the BTR and proving how it inverts expected advantage in GAE, thereby corrupting gradient estimates.\n2. $T^3$ is a straightforward, drop-in enhancement for standard RL algorithms (PPO, GRPO, GSPO) that yields significant gains with small change.\n3. The authors include thorough analyses, including out-of-distribution generalization tests (where $T^3$ maintains robustness) and ablations of different proxy truncation conditions, confirming the importance of detecting BTR entry rather than indiscriminate truncation.", "weaknesses": "1. Dependency on Task-Specific Proxies: While the theoretical $T^3$ condition is general, its practical implementation relies on manually designing task-specific proxies (e.g., repetitive queries for Situation Puzzles, failure to reduce hypothesis space for Guess Numbers)999999999. Finding these proxies for novel, less-structured tasks might be challenging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ttp1Px6bmk", "reviewer": "Reviewer_4y4o", "summary": "This paper proposes a novel method, T3 (Truncating Belief-Trapped Trajectories), to mitigate belief deviation in LLM-based agents during multi-turn active reasoning under reinforcement learning (RL). The core idea is to detect when an agent enters a “Belief Trap Region” (BTR) where further reasoning becomes uninformative and truncate the trajectory early to preserve credit assignment for the earlier informative steps. Theoretical analysis establishes the adverse effect of BTR on gradient estimation, and experiments on five tasks (from AR-Bench and Multi-Turn Puzzles) demonstrate significant gains in stability, efficiency, and performance across multiple RL methods (PPO, GRPO, GSPO).", "questions": "How does T3 behave under dense reward or shaped intermediate feedback?\nHave you considered extending T3 beyond outcome-only reward settings?\n\nCan the proxy condition for BTR be learned, rather than hand-crafted?\nFor instance, via classification over hidden states or CoT traces?\n\nWhat is the impact of false-positive truncation in early training stages?\nDoes it increase gradient variance or harm exploration?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Novel Theoretical Insight: Identifies belief deviation and BTR as key bottlenecks in multi-turn RL for reasoning.\n\nSimple & Effective Method: T3 integrates seamlessly into existing RL algorithms with minimal changes.\n\nRigorous Theory: Theorems formally show how uninformative trajectory tails can invert gradients and harm exploration.\n\nImpressive Results: Up to 30% performance gain, and 34% reduction in token usage across various benchmarks.\n\nRobustness: T3 generalizes well to OOD settings, LLM scales (3B–14B), and architectures (Qwen, LLaMA, DeepSeek).", "weaknesses": "(W1) Truncation proxy design remains task-specific and heuristic :While the theoretical formulation of T3 is grounded in the notion of epistemic stalling (i.e., halted belief progress), the practical implementation relies on task-specific heuristic proxies such as:\nrepeated “unknown” feedbacks in Situation Puzzles,\ninvalid guesses outside the hypothesis set in Guess Numbers,\nsimilarity drops in Movie Recommendation.\nThese conditions work well for the selected benchmarks, but lack generality. In tasks with ill-defined or open-ended hypothesis spaces, such proxies may fail to accurately detect BTR (Belief Trap Region) entry, leading to either missed truncation or premature stopping.\nmy suggestion: The authors could explore more general-purpose or learnable truncation detectors, such as classifiers over internal LLM hidden states or CoT consistency metrics, to broaden T3’s applicability across domains\n (W2) Limited to sparse outcome-based reward setting:\nT3 is explicitly designed for environments where only the final step yields a non-zero reward. While this aligns with many current RLHF-style settings, it raises concerns regarding:tasks with dense or shaped intermediate rewards, environments where step-wise feedback is available or desirable..\nIn such scenarios, truncating trajectories could remove valid informative signals from later steps, inadvertently harming credit propagation rather than helping it. T3’s compatibility with these more general reward formulations remains unexplored. Suggestion: Discuss how T3 would behave in dense-reward or hybrid reward settings, or propose adaptations for preserving mid-trajectory feedback signals.\n(W4) Lack of analysis on false positive truncation and its impact.While the authors provide some ablations on hyperparameters (e.g., window size k), they do not explicitly model the cost of mis-triggered truncation or propose mitigation strategies.\nWhat if the truncation proxy fires prematurely (false positive)?\nCould this discard useful exploratory steps that would have led to reward?\nDoes it increase variance in gradient estimation during early-stage training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)