RLVMR: Reinforcement Learning with Verifiable Meta-Reasoning Rewards for Robust Long-Horizon Agents
Authors: Zijing Zhang
Ziyang Chen
Mingxiao Li
Zhaopeng Tu
Xiaolong Li
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
RL
Agent
Primary area: foundation or frontier models, including LLMs

Abstract: The development of autonomous agents for complex, long-horizon tasks is a central goal in AI. However, dominant training paradigms face a critical limitation: reinforcement learning (RL) methods that optimize solely for final task success often reinforce flawed or inefficient reasoning paths, a problem we term inefficient exploration. This leads to agents that are brittle and fail to generalize, as they learn to find solutions without learning how to reason coherently. To address this, we introduce RLVMR, a novel frame-work that integrates dense, process-level supervision into end-to-end RL by rewarding verifiable, meta-reasoning behaviors. RLVMR equips an agent to explicitly tag its cognitive steps—such as planning, exploration, and reflection—and provides program-matic, rule-based rewards for actions that contribute to effective problem-solving. These process-centric rewards are combined with the final outcome signal and optimized using a critic-free policy gradient method. On the challenging ALFWorld and ScienceWorld benchmarks, RLVMR achieves new state-of-the-art results, with our 7B model reaching an 83.6% success rate on the most difficult unseen task split. Our analysis confirms these gains stem from improved reasoning quality, including significant reductions in redundant actions and enhanced error recovery, leading to more robust, efficient, and interpretable agents.

Published as a conference paper at ICLR 2026
RLVMR: REINFORCEMENT LEARNING WITH VERI- FIABLE META-REASONING REWARDS FOR ROBUST
LONG-HORIZE AGENTS
Zijing Zhang" 1.2.
.,,,+
Ziyang Chen
Mingxiao Li'
Zhaopeng Tulst
Xiaolong Li'
Tenceet Multimodal Department, Digital Human Center.
Peking University
ABSTRACT
The development of autonomous agents for complex, long-horizon tasks is central goal in AI. However, dominant training paradigms face a critical limitation: reinforcement learning (RL) methods that optimize solely for final task success often reinforce flawed or inefficient reasoning paths, a problem we term ineflicient exploration. This leads to agents that are brittle and fail to generalize, as they learn to find solutions without learning how to reason coherently. To address this, we introduce RLVMR, novel framework that integrates dense, process-leve supervision into end-to-end RL by rewarding verifiable, meta-reasoning behaviors. RLVMR equips an agent to explicitly tag its cognitive steps- such as planning, exploration, and reflection- and provides programmatic, rule-based rewards for actions that contribute to effective problem-sole solving. These process-centric rewards combined with the final outcome signal and optimized using critic-free policy gradient method. On the challenging ALFWorld and Science World benchmarks, RLVMR achieves new state-of-the-at results, with our 7B model reaching an 83.6% success rate on the most difficult unseen task split. Our analysis confirms these gains stem from improved reasoning quality, including significant reductions in redundant actions and enhanced error recovery, leading to more robust, efficient, and interpretable agents.
INTRODUCTION
The quest to build autonomous agents capable of solving complex, long-horizon tasks has gained significant momentum with the rise of Large Language Models (LLMs) (Wang al.) [002]] Zeng let al. 20244] Bai et al. 2024). However, dominant training paradigms face a fundamental trade-off. On one hand, Supervised Fine-Tuning (SFT) on expert trajectories can teach agents efficient behaviors, but these policies are often brittle and fail to generalize to novel situations (Chu et al. 2025). On the other hand, RL from environmental feedback encourages exploration and can lead to better generalization, but it typically optimizes for single, sparse reward signal: final task success.
This reliance on outcome-only rewards raises a critical, yet underexplored question: Are agents learning to reason coherently, or are they just finding brittle shortcuts to success? Our work investigates a pervasive issue we term inefficient exploration, where agents are rewarded for successful outcomes even when their path success is built on flawed, or redundant reasoning. This leads to agents that exhibit high rates of repetitive actions and struggle to adapt to unseen tasks, because their underlying problem-solving process is unsound. Standard RL inadvertently reinforces any successful trajectory. failing to distinguish between robust and flawed reasoning processes. This deficiency undermines agent reliability and generalization, especially as tasks grow complexity.
We argue that to build truly robust and generalizable agents, we must move beyond rewarding only the final outcome and begin to supervise the reasoning process itself. Inspired by metacognitive theory Martinez, 2006), which posits that effective problem-solving depends on "thinking about
"Tii work was done during an internship at Tencent. Correspondence Zhaopeng Tu < @yptuu@tencent.com: and Ziyang Chenewwillycchanetteeentccom

---

Published as a conference paper at ICLR 2026
Behavior
Output
Rank
Behavior with Mleta-Keasoning Rewards
Outpot
Rank
oPe
resgentse
obs
response
Env
obs
action
obs
T2
action
V
obs
respotte
obs
respuris
X
N
oba
action
obs
action
rollout
rollout
obe
response
obs
response
action
obs
T4"
action
(a) Standard RLVR (GRPO)
(b) RLVMR (Ours)
Figure I: Comparison of LLM agent RL training paradigms: (a) Standard RL with outcome-only rewards (e.g., GRPO) inadvertently reinforces trajectories with inefficient or illogical intermediate reasoning steps. (b) Our RLVMR approach provides dense, verifiable rewards for beneficial meta- reasoning behaviors (e.g., T1-T4), directly shaping more robust and coherent reasoning process.
thinking". we propose to directly reward beneficial cognitive behaviors. Our key insight is that high-level skills like planning. monitoring progress, exploring alternatives, and reflecting on errors can be operationalized as distinct, verifiable steps within an agent's 's reasoning process.
To this end, we introduce Reinforcement Learning with Verifiable Meta-Reasoning Rewards (RLVMR), a novel framework that integrates dense, process- supervision into end-to-end RL. As illustrated in Figure I RLVMR contrasts with standard RL by rewarding not only the final outcome but also the intermediate reasoning steps. Our framework defines a set of core meta- reasoning behaviors planning, exploration, and reffectiionmmonitorinng to articulate its cognitive state through special tags. During online interaction, we use lightweight, programmatic rules grant verifiable rewards for these behaviors. For example, an 'exploration' tag is rewarded when the agent discovers a new state, while a 'reflection' tag is rewarded when it leads to the correction of prior mistake. These process-centric rewards are combined with the global outcome reward and optimized using policy gradient method. After brief "cold-start" supervised fine-tuning (SFT) phase on only 200 trajectories learn the tag syntax, the agent is trained entirely through environmental interaction.
and enables the agent
We demonstrate the effectiveness of RLVMR on two challenging long-horizon benchmarks, ALF- World and ScilnceWorld. Our experiments show that RLVMR achieves new state-of wffe-offthhe-as art results across all settings. Notably, on the hardest unseen task split (L2), our 7B model achieves an 83.69 success rate, and surpasses the performance of the much larger models. In-depth analysis reveals that these gains are driven by tangible improvement in reasoning quality: RLVMR- exhibit significant reductions in repetitive and invalid actions. This confirms that by rewarding the process of good reasoning, we create agents that are not only more successful but also more robust, efficient, and generalizable.
agents
In summary, our contributions are as follows:
1. We identify and analyze a critical inefficient exploration issue in outcome-only end-to-end RL for long-horizon LLM agents, where spurious state-action correlations override genuine reasoning, leading to redundant reasoning steps and illogical action loops.
2. We introduce a novel framework, RLVMR, that provides dense, verifiable rewards for meta meta-reasoning behaviors like planning, exploration, and reflection, enabling more robust and efficient problem-solving.
3. We achieve SOTA performance on ALFForlld and ScienceWorld, with in-depth analysis confirming reductions in redundant actions and improved generalization to unseen tasks.
2 INEFFICIENT EXPLORATION IN LONG-HORIZON AGENTS
This section investigates the phenomenon of "inefficient exploration" in agents designed for long- horizon tasks. We analyze its detrimental effects on performance, which manifest as brittle efficiency on previously seen tasks and poor generalization unseen ones.
2

---

Published as a conference paper at ICLR 2026
2.1 EXPERIMENTAL SETUP
Benchmarks To evaluate Foundational capabilities and generalization, we conduct experiments on the widely-used and challenging ALFWorld benchmark (Shridhar et 2020), which comprises embodied household tasks. To systematically measure generalization, we define three evaluation splits based on the original benchmark:
Lo (seen-LO); seen task variants and seen task categories;
LI (unseen-L.I): unseen held-out task variants but seen task categories;
L2 (unseen-L2): unseen held-out task variants and unseen task categories.
LO and L1 follow the official benchmark splits. For L2, we further partition ALFWorld by task category, holding out entire categories from training for exclusive use in evaluation.
Training Paradigms We experiment with Qwen2.55-1.B--Instree and Qwen2.5-7B Beenn..5-B-Instteee models using the ReAct (ao al. 2023) framework, which alternates between reasoning and acting steps. We evaluate two dominant training paradigms:
SFT (Yang et al.
2023 lang et al. 2023 Xi et al. 2024): A widely adopted paradigm that applies
supervised fine-tuning high-quality expert trajectories.
GRPO (Feng al., 2025a Wang et al., 2025b): An end-to-end RL method that optimizes the policy by comparing the final rewards of multiple trajectories sampled from the same initial state.
Evaluation Metries
We assess performance using the following metrics:
Success Rate (%, 1): The percentage of tasks successfully completed by the agent each evaluation split.
Invalid Action Rate (%, 1): The proportion of generated actions that are invalid in the current state, reflecting basic comprehension and error frequency.
Repetitive Action Rate (%, 1): The percentage of steps where the agent executes meaningless repeated action, defined in prior work (Yuan al. 2025 Fu et al.) 2025 Feng et al., 2025b). This metric quantifies inefficient exploration, indicating that the agent's policy may be overfitting familiar action sequences rather than being guided by robust reasoning.
2.2 THE INEFFICIENT EXPLORATION PROBLEM
While aggregate statistics show that methods like GRPO improve agent success rates, closer look at individual trajectories reveals a critical flaw: the inefficient exploration problem. Even when an agent successfully completes a task, its path to a solution is often littered with redundant or illogical steps. This behavior, illustrated qualitatively in Appendix A indicates a gap between achieving correct outcome and demonstrating robust reasoning. Our large-scale empirical results (Figure 2 quantify the pervasiveness of this issue and expose a fundamental trade-off in current training paradigms.
SFT creates efficient but brittle policies that fail to generalize. Supervised Fine-Tuning (SFT) models achieve high success rates and efficiency on tasks they have seen during training. For instance, the 7B SFT model's success rate in in-distribution tasks (LO) jumps from 23.1% 1% (ReAct baseline) to 63.3%, with a low invalid action rate of 6.2%. However, this performance is brittle. On the most challenging out-of-distribution split (L22,, the model's success rate plummets to 37.5%, and its repetitive action rate nearly doubles. This reveals that when faced with novel situations, the agent falls into non- producciee loops, demonstrating that SFT teaches mimicry without instilling a generalizable reasoning process.
GRPO improves generalization but fosters inefficient, flawed reasoning. contrast, reinforce- ment learning with outcome-only rewards (GRPO) achieves substantially better generalization, with the 7B model attaining success rates of 77.3% on L1 and 52.3% on L2. This success, however, comes at the cost of severe inefficiency, validating our hypothesis. The agent's performance

---

Published as a conference paper at ICLR 2026
100
Vanilla
SFT
GIRP
100
Vanilla
SFT
GRPO
60
Vanilla SFT
12.3
Ropetáive Actions (%(
GRPO
Success Rato
60
60
43.0
19.7
215
20
20
164
20
11.2
10.2
20
(a) Success Rate (1.5B)
(b) Invalid Actions (1.5B)
(c) Repetitive Actions (1.5B)
100
100
60
Vanilla
SET
GRRO
Vaniila
SFT
GRPO
Success Rate (%)
80
77.3
Actions (%6
450
633
Repetitive AopetiiveAAiiis
40
37.5
Invalad
20
21.5
045
23.1
135
148
147
133
55
LI
L2
(d) Success Rate (7B)
(c) Invlaid Actions (7B)
(f) Repetitive Actions (7B)
Figure 2: Performance ALFWorld. While SFT excels seen tasks (LO) but fails generalize, GRPO achieves better generalization at the cost of significant inefficiency. This highlights a funda- mental trade-off between brittle efficiency and inefficient generalization.
undermined by high invalid and repetitive action rates across difficulty levels; on the hardest L2 tasks, the 7B model's 's repetitive action rate is staggering 31.2%. By optimizing solely for task success, GRPO reinforces any path to positive outcome, even those built on illogical steps and inefficient exploration.
Scaling model size does not fix the underlying reasoning deficiencies. While scaling from 1.5B to 7B model improves overall success rates, it does not resolve this fundamental issue. Notably, while the 7B GRPO model is more successful on L2 tasks than its 1.5B counterpart (52.3% vs. 29. 29.7%), it also exhibits a higher repetitive action rate (31.2% vs. 27.1%). This suggests a larger model's enhanced capacity can be misdirected more effectively exploit flawed strategies rather than to reason more coherently. This finding underscores that the limitation is rooted in the training objective itself, not merely model capacity, and that simply increasing model size is not panacea.
Current paradigms force a trade-off between brittle efficiency and inefficient generalization. Our analysis reveals core dilemma: SFT produces efficient but brittle policies that fail to generalize, while GRPO achieves generalization the cost of reinforcing inefficient and logically flawed reasoning. Neither paradigm effectively teaches the agent how reason well. This establishes clear need for a new framework that moves beyond sparse, outcome-only signals to provide direct, process-level supervision. By rewarding coherent and efficient reasoning steps, to not only find solutions but to do so robustly and intelligently - the precise goal of our work.
guide agents
METHODOLOGY: RLVMR
Our methodology equips LLM agents with an explicit meta-reasoning framework to mitigate inef- ficient exploration in complex tasks. As shown in Figure the agent is trained in two phases: an initial SFT stage to bootstrap the agent's meta-reasoning capabilities, followed by reinforcement learning phase that uses a custom policy optimization algorithm to refine these skills based on task outcomes and process-centrie rewards.
Cold Start: Initial Meta-Reasonine Acquisition via SFT To equip the base LLM with the foundational ability to generate structured meta-reasoning. begin with supervised fine-tuning phase. This step is crucial, as reasoning patterns learned during subsequent reinforcement learning heavily influenced by the base model's capabilities. The SFT data is constructed follows:
4

---

Published as a conference paper at ICLR 2026
Rollout Trajectories
Start
X1
Expert Trajeckor
oty City Of dit
Ganemie Mallewesoring Typea
x
x
A
a$
Y
Group Relative Advantage Computing
Annotated Trajectory oy. Mi... ffy
reasoning Type Grouping
reasoning Type Grouping
reasoning Type Grouping
Trajectory Outcome Reward Grouping
SFT
A
79
Figure 3: A schematic diagram of the RLVMR framework, which consists of two training phases: cold start and reinforcement learning. Our method provides rule-verifiable feedback signals based on the final outcome and the relative advantages of different types of meta-reasoning behaviors.
1. We collect a dataset of successful task trajectories containing only observation-action pairs.
2. We employ a more powerful teacher model (e.g., GPT-4) to annotate these trajectories with our meta-reasoning tags, inferring the most likely cognitive step preceding each action. This process creates synthetic, reasoning-riee rich expert demonstrations.
3. The target LLM is fine-tuned on these annotated trajectories, learning to imitate the expert's meta-reasoning and action generation patterss.
3.1
FRAMEWORK
We begin by formalizing the agent-envvrrommet interaction Markov Decision Process. We then introduce a novel meta meta-nncssnnng framework that extends existing agent architectures by operational- izing principles from cognitive science.
Task Formulation as a Markov Decision Process We formalize the interaction between an agent and its environment long-horizon tasks as a Markov Decision Process (MDP). An MDP is defined by tuple (S, A,O, F, R), where S is the set of environment states, A is the action space, O is the observation space, F S x A + S is the state transition function, and R x A + IR is the reward function. In our setting, which is tailored for LLM agents, the state, action, and observation spaces (S, A, O) are all represented as natural language sequences over a finite token vocabulary.
At each timestep t,, the agent's 's policy me generates thought process thy and action a, based on the current state sy; (the, ae) ~ TT(. se). The agent's 's interaction with the environment produces a trajectory {(o,, a,), (02, the, a2)...., (On, thn a,,) }. In many long-horizon tasks, reward signals are sparse, typically provided only as a final outcome reward R(T) at the end of an episode. This sparsity poses significant challenges for credit assignment. The agent's objective is to learn an optimal policy ma that maximizes the expected cumulative reward:
max Erons [[(T)]
(1)
Operationalizing Meta -eaa-nneson in LLM Agents Our approach is grounded in metacognitive theory (Martinez, 2006 Lai 2011). which emphasizes "thinking about thinking". Metacognition comprises two key components: metacognitive knowledge (an agent's 's self-awareness of its own reasoning strategies) and metacognitive regulation (the active control of these processes, including planning, monitoring, and adaptive revision). This theoretical lens suggests that for LLM agents to solve complex tasks, they require not just domain knowledge but also the capacity for dynamic planning, self-mon melf-moniorring, and creative exploration.
5

---

Published as a conference paper at ICLR 2026
To operationalize these principles, extend the ReAct framework. While ReAct interleaves reasoning and actions (c.g., "Think: Act: ..."), treats reasoning as a monolithic process. We refine this by introducing structured set of meta-reasoning tags to explicitly represent distinct cognitive functions. This decouples reasoning from actions and enables fine-grained analysis and supervision. Specifically; we define four meta-reasoning tags, each enclosed in XMI.-ttye tags (e.g.. <pl splanning>>, while all actions are contained within the caction: tag.
Planning (<planning>); Decomposes the task into high-level steps to formulate overall strategy. Used at the start of a task or when replanning is needed.
Exploration (<explore>): Generates hypotheses or options to navigate uncertainty or bottlenecks, encouraging creative problem-solving.
Reflection (sreflection)) Reviews history to analyze errors and formulate corrective actions. Typically triggered after unsuccessful attempts.
Monitoring (<monitor>): Tracks task progress against the overall plan, ensuring actions remain aligned with subgoals. Applied during routine execution.
3.2 META-REESSNINGGAWAAR REWARD SHAPING
During reinforcement learning, guide the agent with a composite reward signal that combines task completion with the quality of the reasoning process. This signal comprises a sparse outcome reward and dense. process-basse based meta-reasoning reward.
Outcome Reward (R(T)): A binary signal awarded at the end of a trajectory: R(T) = Ts for task success and 0 otherwise, where T, is positive constant.
Meta- RettaReea Reward (r:M)) A dense reward assigned at each step t to incentivize locally beneficial behaviors.
Planning Reward (Fplanning): Awarded for a <planning> step if the trajectory succeeds.
Exploration Reward (Texplore): Awarded if the current action targets a new object or location, discouraging redundancy.
Reflection Reward (Freflection): Awarded if <reflection> step is followed by corrective action after sequence of failures.
Format Reward (rfoommat): A penalty, -Aiormar. is applied if the model's output at step t does not conform the expected <tag>... </tag><act
</accioo>>>> structure.
The total step-level reward the sum of the process-based rewards: = r MR + format
3.3 GROUP RELATIVE POLICY OPTIMIZATION WITH META-REET DETTA-REAOONING (GRPO-MR)
To effectively leverage composite reward signal, introduce Reta-Reesonnig Group Policy Op- timization (GRPO-MR). GRPO-MR computes a step-level advantage by combining global trajectory performance with local, context-aware reasoning quality.
Trajectory-level Relative Advantage: For a batch of K trajectories collected from the same environment, first calculate normalized trajectory-lever advantage to capture overall performance:
R(TE) MR
trajj
OR
where μR and oR are the mean and standard deviation of outcome rewards across the batch.
Meta-reasoning Level Relative Advantage: The core of GRPO-MR is the computation of a context- aware advantage. We group all steps within batch that share the same meta-reasoning tag (e.g.. all <explore> steps) and normalize their rewards within that group:
MR L,tag
MR MR t.tag
Peag
(3)
Frag

---

Published as a conference paper at ICLR 2026
Table 1: Performance comparison on the benchmarks. We report the success (%) on seen (LO: seen task variants and categories) and unseen (L1: unseen task variants but seen task categories; L2: unseen task variants and categories) task variations. We also report the average cumulative reward (score) on the World benchmark.
ALFWorld
ScienceWorld
Model
Method
I.0
1.2
I,2
succ.
succ.
SUCC.
succ.
SCOTC
SUcC.
Score
Succ.
GPT-4o
ReAct
57.3
66.0
68.8
45.4
54.3
49.2
57.0
41.0
52.0
DeepSeek- V3
ReAct
60.2
65.9
53.9
27.3
39.1
35.2
43.0
26.5
37.1
DeepSeck-RI RI
ReAct
68.8
70.2
67.3
22.2
32.0
31.4
39.5
29.1
37.9
AgeniGym
SFT+RC
76.6
63.3
46.9
56.3
33.6
45.2
ReAct
11.3
13.7
10.2
1.2
9.0
0.8
7.8
0.8
7.4
SFT
43.0
38.7
17.6
20.3
30.9
18.0
27.8
12.5
20.9
ETO
64.1
66.4
25.8
39.1
47.3
22.7
29.8
15.6
23.4
Qwen2.5-1.5B
GLIDER
66.0
68.8
35.2
40.2
50.2
25.8
32.0
19.5
25.1
GRPO
76.6
71.1
29.7
21.1
31.7
13.7
22.5
10,9
21.2
GiGPO
86.7
83.2
48.0
25.8
35.6
15.2
22.8
4.7
11.2
RLVMR
89.1
87.9
56.3
46.9
60.3
34.4
45.2
26.5
33.9
ReAct
23.1
28.5
27.0
7.8
17.4
11.3
19.6
6.3
16.5
SFT
63.3
57.0
37.5
36.7
43.5
32.0
41.6
23.4
32.2
ETO
70.3
74.2
51.6
62.5
71.2
40.6
50.4
28.1
35.0
Qwen2.5-7B
GLIDER
75.4
74.6
53.1
62.9
68.8
41.4
52.8
25.8
32.5
GRPO
79.3
77.3
52.3
49.1
61.8
30.1
43.1
26.6
34.3
GiGPO
89.5
90.2
67.2
53.4
69.2
35.2
50.7
25.8
33.2
RLVMR
91.4
91.8
83.6
67.2
77.8
43.0
59.4
32.2
49.1
ReAct
19.5
22.3
17.6
8.6
18.8
11.7
19.9
11.7
20.3
SFT
62.5
60.9
39.1
39.8
47.6
30.1
39.8
22.3
32.6
ETO
69.5
67.5
47.3
57.0
64.3
36.8
45.2
29.3
35.4
Llama3.1-8B
GLIDER
72.7
73.4
50.8
64.4
71.2
38.7
53.8
28.5
35.6
GRPO
73.0
70.7
45.3
45.6
55.2
28.8
40.1
25.8
33.7
GiGPO
86.0
87.1
68.8
60.2
73.5
39.1
55.2
30.1
42.3
RLVMR
92.2
91.0
83.2
71.1
80.3
49.2
63.7
38.7
51.2
where prag and Otag are mean and standard deviation meta-reasoning rewards for all steps with that specific tag. The final step-level advantage A, is weighted combination of these two signals:
A traj a).
MR
where [0, 1] is a hyperparameter balancing the influence of the global outcome and local reasoning quality. Finally, we optimize the policy using clipped surrogate objective with KL divergence regularization:
Linal = E, imin (0),,, clip(r.))) c, +. c)).+) - AKT DKI. [Too|
(5)
where (0) the importance sampling ratio, e the clipping hyperparameter, and ÅKI. controls the KL penalty against reference policy Teef-
4 EXPERIMENT
4.1
MAIN RESULTS
In this section, we present the core experimental results to evaluate the effectiveness our proposed RLVMR. In addition ALFWorld, also conduct experiments ScienceWorls (Wang et al. which focuses on text-based scientific experimentation.
2022..
We compare approach with two major categories of advanced RL training methods in addition to SFT: (1) Offline RL, including (i) ETO (Song et al.) (2024). which iteratively refines actions using step-level feedback along trajectories; (ii) GLIDE (Hu et al. 2025b), which decomposes complex
7

---

Published as a conference paper at ICLR 2026
Invalid Actions (%4)
GRIO
Repetiive Actions (%)
GRRP
REYMR
21.6
REVMR
27.1
14.0
18.4
29.2
11.8
8.0
10.7
57
1.0
1.2
1.0
L1
1,2
(a) Invalid Actions
(b) Repetitive Actions
30
40
SET
SET
Iovaial Actions (%)
GRPO
GRPO
31.2
Repetitive Actions
30
20
24.5
14.8
13.5
20
21.5
14.4
10
11.
6.2
7.1
10
7.8
10
1.2
L.0
L1
L2
(c) Invlaid Actions (7B)
(d) Repetitive Actions (7B)
Figure 4: Exploration efficiency of RLVMR compared to SFT and GRPO baselines on ALFWorld.
tasks into coherent sub-tasks to improve transferability. (2) Online End-to-end RL, including (iii) Multi-turn GRPO (Wang al. 2025b), which adapts the original GRPO (Shao et al. 2024) for online multi-turn RL tasks; (iv) GiGPO (Feng et al., 2025b), which introduces a two-level structure for finer-grained credit assignment. For broader comparison, we also report the performance of
GPT-4o, PeepSeek-33/R11. and AgentGym (Xi et al. 2024). Detailed information is provided in Appendix
RLVMR achieves new SOTA performance across all benchmarks and model sizes. As listed in T our RLYMR framework consistently sets a new standard for performance, outperforming all baseline methods both ALFWorld and ScienceWorld With the Qwen-7B model, RLVMR achieves success rates of 91.4% on seen ALFWorld tasks and 67.2% on seen Science tasks, surpassing the next-best method, GiGPO. This consistent superiority highlights the broad applicability and effectiveness of integrating verifiable meta-reasoning rewards into the RL training loop, leading more capable and successful agents.
Rewarding meta-reas significantly enhances generalization to unseen tasks. A primary contribution of this work is addressing the inefficient exploration issue to improve generalization. Our results validate this claim, showing that RLVMR excels in novel scenarios, especially the most challenging Unseen-L2 split, which involves entirely new task categories. On ALFWorld's L2 split, our 7B model reaches an impressive 83.6% success rate, substantial 16.4 percentage point improvement over the strongest baseline (GiGPO), Similarly, on Science World's 's L2 split, RLVMR outperforms all other methods. This demonstrates that by learning how than just memorizing solutions-our agent develops more robust and transferable problem-solving skills, leading superior performance on unfamiliar challenges.
rafe
4.2 ANALYSIS
Our analysis reveals that RLVMR's verifiable meta-reasoning rewards lead to superior exploration and training efficiency, enabling the agent to find more direct solutions with greater stability than strong baselines. Unless otherwise stated, we report results based on Qwen2.5-1.558 on ALFWorld.
Exploration Efficiency We analyze agent exploration efficiency by measuring invalid and repetitive actions (Figure|4) 4). RLVMR's verifiable meta-reasoning rewards cultivate more efficient problem- solving strategies, significantly reducing flawed or redundant steps. On seen tasks, our 1.5B model slashes the invalid action rate from 18.15 1% (GRPO) to 11.1% and the repetitive action from 18.4%
8

---

Published as a conference paper at ICLR 2026
to 6. %. This efficiency gain is robustly maintained novel challenges; while GRPO's repetitive action rate worsens on the hardest unseen tasks (from 21.4% to 27.1%), RLVMR'S s rate remains controlled at 5.7%. This demonstrates that RLVMR learns generalizable problem- principles rather than overfitting to familiar paths.
ALFWORLD Success Rate
ALFWORLD Episode Length
Episode Length (mean)
0.8
Rate
0.6
Success
15
0.2
10
0.0
100
Step
Step
GIGPO
GRAD
Figure 5: Success rate and step count curves of different approaches on ALFWorld during RL training.
Training Efficiency We evaluate training efficiency via learning stability (convergence) and policy quality (action sequence length) in Figure Agents trained with RLVMR learn more direct solutions and converge faster and more stably than baselines. In contrast, baselines like GRPO are unstable and produce longer solution paths. This stems from its process-level rewards, which provide clearer and more robust learning signal that prevents inefficient and unproductive loops.
RELATED WORK
LLM Reinforcement Learning RL is widely used to align LLMs with human preferences (RLHF, DPO) OOuyang et al.) 2022; Rafailov et al.) (023). Beyond alignment, RL has been applied to improve reasoning and emotional intelligence (Hu et al. 2025a Muennighoff et al. 2025 Wang et al. 2025a). Group-hased methods such as GRPO, Dr.GR and DAPO estimate advantages from multiple samples the same prompt, removing the critic and improving efficiency over actor-critic approaches like PPO (Feng al. 2025a Liu et al. 2025 Yu et al.) 2025 Schulman al. 2017). These methods achieve strong results on mathematical reasoning, search, and tool use (Yu et al.. 2025 Hu et al. 2025a). However, applying RL to multi-turn, long-horize tasks remains difficult due to sparse, delayed rewards a challenge we address [Wang et al. 2025b).
LLM Agents
LLMs increasingly act as agents for code generation, web interaction, embodied
control, and affective tasks (Huang et al.) 2023 Zhang et al. 2024 Bai et al.
2024 Agashe et al.
2024
Abuelsaad al. 2024 Zeng et al. [024]] Qiao al. [024] Fu al. 2025
Zhang et al. 2025).
Early systems relied on prompting and external tools (e.g., ReAct) (Yao et al. 2023 Shinn et al.
2023), but smaller models often lack strong reasoning; SFT can improve decisions (Zhang & Zhang
Xi et al. 2024. Qin et al. (024). Other work studies single-step or offline RL (Yu et al. 2024 Zanette 2024), while recent efforts train agents end to end with online RL, learning directly from interaction and reducing reliance on complex data preparation step- level reward models (Wang et al., 2025b; Feng et al. 2025b). Despite progress, fine-grained credit assignment and generalization remain challenging (Wang et al. 2025b). We employ reward shaping grounded in verifiable meta-cognitive behaviors to promote effective reasoning and robustness.
2024
Xiong et al. <024//hou
CONCLUSION
We tackled the challenge of inefficient exploration in long-horizon agents by introducing RLVMR, a new framework that guides agents using process-level supervision. Instead of relying solely on sparse success- based rewards, RLVMR provides dense, verifiable feedback for key reasoning behaviors like planning, exploration, and reflection. Our approach combines lightweight initialization phase with end-to-end training to develop more effective and adaptable agents. Experiments on ALFWorld and Science World show that RLVMR achieves state-of-the-art performance, with better generalization

---

Published as a conference paper at ICLR 2026
new tasks and noticeable improvements in reasoning quality- fewer redundant actions and better recovery from mistakes. These results highlight the value of directly supervising reasoning steps. Future research could extend RLVMR multi-modal environments, explore adaptive reward mecha- nisms that dynamically adjust to task complexity, and apply the framework to real-world domains such robotics and software engineering.
REFERENCES
Tamer Abuelsaad, Deepak Akkil, Prasenjit Dey, Ashish Jagmohan, Aditya Vempaty, and Ravi Kokku. Agent-e: From autonomous web navigation to foundational design principles in agentic systems. arXiv preprint arXiv:2407.13052 2024.
Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, and Xin Eric Wang. Agent s: An open agentic framework that uses computers like human. arXiv preprint arXiv:2410.88164 2024.
Hao Bai, Yifei Zhou, Jiayi Pan, Mert Cemri, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training in wildh device-control agents with autonomous reinforcement learning. Advances in Neural Information Processing Systems, 37:12461-12495, 2024.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. arXiv preprini urXiv:2501.1716: 2025.
Lang Feng, Zhenghai Xue, Tingcong Liu, and Bo An. Group-in-group policy optimization for Ilm agent training. arXiv preprint 25X.::25055.077 2025a.
Lang Feng. Zhenghai Xue, Tingcong Liu, and Bo An. Group-in- -ngguu policy optimization for Ilm agent training. arXiv preprint arXiv:2505. 10978, 2025b.
Dayuan Fu, Keqing He, Yejie Wang, Wentao Hong, Zhuoma Gongque, Weihao Zeng, Wei Wang, Jingang Wang, Xunliang Cai, and Weiran Xu. Agentrefine: Enhancing agent generalization through refinement tuning. arXiv preprint 2011::250..01702, 2025.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Open-reasoner-zerr:: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint wxx::i2003.222000 2025a.
Shum.
Zican Hu, Wei Liu, Xiaoye Qu, Xiangyu Yue, Chunlin Chen, Zhi Wang, and Yu Cheng. Divide and conquer: Grounding LLMs as efficient decision- making agents via offline hierarchical rein- forcement learning. In Forty-second International Conference on Machine Learning, 2025b. URL httpss://opeet iew net. httt:/formmm1ddppdi ji3ktF
Dong Huang, Jie M Zhang, Michael Luck, Qingwen Bu, Yuhao Qing, and Heming Cui. Agent- coder: Multi-agent- based code generation with iterative testing and optimisation. arXiv preprint arXiv:2312.13010, 2023.
Emily R Lai. Metacognition: A literature review. 2011.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. arXiv preprint arXiv:2503.20783, 2025.
Michael E Martinez. What is metacognition? Phi delta kappan, 87(9):96-669 2006.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang. Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint NXNv:2501111399 2025.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
10

---

Published as a conference paper at ICLR 2026
Shuofei Qiao, Runnan Fang, Ningyu Zhang, Yuqi Zhu, Xiang Chen, Shumin Deng, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. Agent planning with world knowledge model. Advances in Neural Information Processing Systems, 37:114843-114871, 2024.
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, al. Toolllm: Facilitating large language models to master 16000+ real- world apis. In ICLR, 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-53744 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang. Y. K. Li. Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of abs/
mathematical reasoning in open language models, 2024. URL
2402. 03300
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-8653 8652, 2023.
Mohit Shridhar, Xingdi Yuan, Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning. arXiv preprint ar arXiv:2010.03766 2020.
Côté, Yonatan Bisk, Adam Trischler, and Matthew
Yifan Song. Da Yin, Xiang Yue, Jie Huang, Sujian Li, and Bill Yuchen Lin. Trial and error: Exploration-base trajectory optimization for Ilm agents, arXiv preprint 2xiv:2403.02502, 2024.
Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang. Boxi Cao, and Le Sun. Toolal- paca: Generalized tool learning for language models with 3000 simulated cases. arXiv preprint arXiv:2306.05301, 2023.
Peisong Wang. Ruotian Ma, Bang Zhang, Xingyu Chen, Zhiwei He, Kang Luo, Qingsong Lv, Qingxuan Jiang, Zheng Xie, Shanyi Wang. Yuan Li, Fanghua Ye, Jian Li, Yifan Yang, Zhaopeng Tu, and Xiaolong Li. Rlver: Reinforcement learning with verifiable emotion rewards for empathetic agents, 2025a. URL /aatts::/aariiv.org///ea/ 2507. 03112,
Ruoyao Wang, Peter Jansen, Marc-Alexandre Côté, and Prithviraj Ammanabrolu. Scienceworld: Is your agent smarter than a 5th grader? arXiv preprint arXiv:2203.07540. 2022.
Zihan Wang. Kangrui Wang, Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Xing Jin, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, et al. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning, arXiv preprint arXiv:2504.20073, 2025b.
Zhiheng Xi, Yiwen Ding, Wenxiang Chen, Boyang Hong, Honglin Guo, Junzhe Wang, Dingwen Yang, Chenyang Liao, Xin Guo, Wei He, et al. Agentgym: Evolving large language model-based agents across diverse environments. arXiv preprini arXiv:2406.0415 04111, 2024.
Weimin Xiong, Yifan Song, Xiutian Zhao, Wenhao Wu, Xun Wang, Ke Wang. Cheng Li, Wei Peng, and Sujian Li. Watch every step! 1lm agent learning via iterative step-level process refinement, In EMNLP, 2024.
Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. Gpt4tools: Teaching large language model to use tools via self-instruction. Advances in Neural Information Processing Systems, 36:71995-720077, 2023.
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR), 2023.
11

---

Published as a conference paper at ICLR 2026
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint wrXt:22503.14476 2025.
Yuanqing Yu, Zhefan Wang, Weizhi Ma, Zhicheng Guo, Jingtao Zhan, Shuai Wang. Chuhan Wu, Zhiqiang Guo, and Min Zhang. Steptool: A step-grained reinforcement learning framework for tool learning in llms. 2024.
Siyu Yuan, Zehui Chen, Zhiheng Xi, Junjie Ye, Zhengyin Du, and Jiecao Chen. Agent-r: Training language model agents to reflect via iterative self-training. arXiv preprint arXiv:2501.11425, 2025.
Aohan Zeng, Mingdao Liu, Rui Lu, Bowen Wang, Xiao Liu, Yuxiao Dong, and Jie Tang. Agenttuning Enabling generalized agent abilities for llms. In Findings of the Association for Computational Linguistics ACL 2024, pp. 3053-3077, 2024.
Bang Zhang, Ruotian Ma, Qingxuan Jiang, Peisong Wang, Jiaqi Chen, Zheng Xie, Xingyu Chen, Yue Wang, Fanghua Ye, Jian Li, Yifan Yang, Zhaopeng Tu, and Xiaolong Li. Sentient agent as a judge: Evaluating higher-order social cognition in large language models, 2025. URL
btttss:/aaxae
org/
02847
Kechi Zhang, Jia Li, Ge Li, Xianjie Shi, and Zhi Jin. Codeagent: Enhancing code generation with tool-integrated agent systems for real-world repo-level coding challenges. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), ACL 2024, Bangkok, Thailand, August 1 11-16, 2024, pp. 13643-13658 Association for Computational Linguistics, 2024, doi: 10. 10.18653/11/2024. ACI-LONG.737, 737. URL
Zhuosheng Zhang and Aston Zhang. You only look at screens: Multimodal chain-of-action agents. In Findings of the Association for Computational Linguistics ACL 2024, pp- 3132-31111333 11129,149, 2024.
Yifei Zhou and Andrea Zanette. Archer: training language model agents via hierarchical multi-tur rl. In Proceedings of the 41st International Conference Machine Learning, pp. 7222099, 2024.
12

---

Published as a conference paper at ICLR 2026
A CASE STUDY OF INEFFICIENT EXPLORATION
The following is trajectory segment of a vamilla GRPO-trained agent performing a novel task put two keychains in the safe in ALFWorld (corresponding our L2 split). By step 7, the agent has already arrived at dresser 1. However, we observe that in the subsequent steps, the agent falls into sequence of inefficient decisions: its next intention is to find the second keychain, yet it persistently tries to go to dresser for several steps, disregarding the fact that it already there. This indicates that its policy mainly reflects the action distribution present in the training data, rather than allowing the reasoning process itself to truly regulate decision-making. Although the agent can form relatively effective action strategies for completing tasks, its capacity for critically evaluating its own behavior and understanding the underlying task requirements remains limited. This suggests that the agent has not truly acquired the reasoning patterns necessary for robust task-solving.
A trajectory segment of an agent trained with GRPO
Step 7
You arrive at dresser keychain 2, keychain 1, laptop 1, statue 1, and tissuebox 2.
Observation:
dresser 1, you see box I, a creditcard 1,
Reasoning: The current situation that am at dresser 1, which has multiple items including keychain 1, keychain
creditcard etc. To complete the task of putting keychains in the safe, need to select one of these keychains from the dresser. Action: take keychain from dresser
Step 8
Observation: You pick the keychain from the dresser 1. Reasoning: The current situation is that I am at dresser 1 with the keychain in my possession. To put two keychains in the safe, need to find another keychain and then move both keychains to the safe. Action: go to dresser
Step 9
Observation: Nothing happens.
Reasoning: The player has been in the room for several steps but has not yet interacted with the safe or taken any keychains. The safe is closed initially, which might need interaction to open it.
Action: go to dresser
Step 10
Observation: Nothing happens.
Reasoning: The previous steps were not successful in opening the safe or taking the keychains. Let's try another approach.
Action: go to dresser
Step 11
Observation: Nothing happens. Reasoning: The user has taken all the steps necessary to obtain both keychains but has yet actually put them into the safe. The last known location of the keychains was on the dresser before the attempt to open the safe failed (in Observation 8). To complete the task, the user needs to move one or both keychains to the safe before attempting to open it again. Action: move keychain safe
Observation: Nothing happens.
This qualitative example highlights how an ostensibly "successful" policy can still waste a large number of steps redundant illogical moves.
13

---

Published as a conference paper at ICLR 2026
B SETUP DETAILS
B. DATASET DETAILS
ALFWorld is a household task environment that requires agents to explore rooms and employ common-sense reasoning accomplish tasks, such "put the pencil the desk". The environment provides feedback on whether the agent successfully completes the task within a given number of steps. ALFWorld describes the environment in purely textual form and supplies a reward signal indicating only the final task completion status.
Science World is text-based virtual environment designed as a comprehensive testbed for evaluating and enhancing scientific reasoning abilities in Al systems. It features tasks spanning 10 scientific domains and 30 subcategories, simulating broad range of experiments found in elementary science curricula, including state changes of matter, measurement, electricity, life sciences, plant growth, chemical reactions, genetics, among others. Each task involves multiple subgoals, and the final reward is computed based on the completion of these subgoals. However, to better reflect real-world scenarios, we only use the final reward and disregard intermediate rewards. Notably, some tasks in Science require agents make conclusive judgments based experimental outcomes common sense; a task considered successful only if the agent provides the correct final answer.
Both ALFWorld and ScienceWorld offer "seen" and "unseen" variants for evaluating generalization capabilities. To further assess the agents' robustness and generalization, we define three difficulty levels (LO, L1, L.2,, with 1.2 comprising entirely held-out task types. Specifically, for ALFWorld, designate Cool & Place and Pick Two & Place as held-out tasks; for ScienceWorld the final task type of each topic is reserved for unseen evaluation.
In the ALFWorld environment, since only the final task success signal is provided, we evaluate model performance using the average success rate (succ.). In contrast, the ScienceWorke environment offers more fine fine-grained step rewards, enabling the agent to obtain immediate rewards based on the importance of the steps completed, even without achieving the final goal. Therefore, in addition to the average success rate (succ.), we also report the average cumulative reward (score).
B.2 IMPLEMENTATION DETAILS
We conducted experiments on both the Qwen2.5-1 1-555---11BBnnnttrrue and Qwen2.5-7B Insttruct models. During the cold start phase, set the batch size per GPU to 16, used learning rate of 1 x 10-5, and trained for 5 epochs. For the RL phase, we adopted the veRL framework with necessary modifications. The batch size per GPU was also set to 16. At each training step, sampled from 16 distinet environments, with each environment rolling out 8 trajectories.
The weights for outcome advantage and meta- meta-reas advantage were both set to 0.5 by default. To penalize outputs that did not adhere to the required format, we applied a reward penalty of -0.1, where output freflection) and one action tag (e.g., (action)). ) The KL regularization coefficient was set to 0.01. For all environments, the maximum number of steps per episode was fixed 30. In the cold-start phase, we performed supervised fine-tuning on 200 trajectories for 5 epochs. In the RL training stage, our method was run for 100 epochs, whereas RL-based baselines were trained for 150 epochs.
considered valid only if it included at least one meta-reasoning tag (e.g.,
AgentGym is trained on Llama-2 2-Chat-7B 7B, first with behavior cloning on the AgentTraj (Xi et al. 2024) dataset from multiple environments, and then further improved via exploration and self- evolution on a broader instruction set.
C DETAILED EXPERIMENT RESULTS
We further report the success rates of different methods various tasks in ALFWorld. Table 2 provides the results using the 1-....5-1.5B model as the base model, while Table 3 presents the results using the Qwen2.5-7B model. Additionally, we also evaluated another model from a different family, Llama3. 1ama3.11-B, and the results are shown in Table As shown the tables. RLVMR generally outperforms other methods across all tasks, and particularly exhibits strong performance in more
14

---

Published as a conference paper at ICLR 2026
complex tasks. This demonstrates that RLVMR, by rewarding high-quality reasoning behaviors, significantly enhances the robustness and adaptability of agents in multi-step interactions.
Model
Method
Pick
Look
Clean
Heat
Cool
Pick2
All
ReAct
23.1
18.3
10.8
8.7
3.5
0.0
13.7
+SFT
43.2
42.0
35.9
33.2
29.4
29.7
38.7
+ETO
73.6
46.3
66.2
68.3
62.8
55.6
66.4
Qwen2.5-1.5B
+GLIDER
78.8
58.2
63.6
73.7
61.6
66.1
68.8
+GRPO
80.3
55.6
88.1
76.2
62.0
72.1
71.1
+GiGPO
92.8
66.5
90.7
90,9
80.2
73.8
83.2
+RLVMR
95.2
78.8
91.2
90.2
83.9
77.6
87.9
Table 2: Success rates on ALFWorld using Qwen2.5-1.5B model.
Model
Method
Pick
Look
Clean
Heat
Cool
Pick2
All
ReAct
43.1
33.2
18.7
16.4
20.2
12.8
28.5
+SFT
70.8
63.0
61.1
46.3
49.7
33.2
57.0
+ETO
88.2
70.5
82.3
83.6
71.0
51.2
74.2
Qwen2.5-7B
+GLIDER
89.6
72.1
83.9
81.6
69.5
53.0
74.6
+GRPO
90.2
76.7
86.0
80.1
68.3
56.4
77.3
+GiGPO
91.7
85.9
93.3
90.3
89.0
83.6
90.2
+RLVMR
95.3
88.2
90.1
92.4
89.8
86.7
91.8
Table 3: Success rates on ALFWorld using Qwen2.5-7B model.
Model
Method
Pick
Look
Clean
Heat
Cool
Pick2
All
ReAct
40.3
30.1
17.8
13.9
19.5
9.3
22.3
+SFT
70.8
69.0
58.6
47.7
58.9
40.4
60.9
+ETO
83.3
64.5
76.9
73.0
66.4
46.2
67.5
Llama3.1-8B
+GLIDER
87.7
71.2
78.0
79.5
68.2
49.7
73.4
+GRPO
87.0
75.9
82.8
74.0
67.2
55.0
70.7
+GiGPO
90.3
87.5
90.1
85.2
83.6
82.5
87.1
+RLVMR
93.5
90.0
86.5
91.5
86.5
83.5
91.0
Table 4: Success rates on ALFWorld using Llama3. model.
D PSEUDOCODE OF RLVMR
We present the pseudocode for the RLVMR training procedure in Algorithm 1 and the pseudocode for computing the relative advantage of composite groups in Algorithm 2 Additionally, we provide the pseudocode for computing meta-reasoning rewards in Algorithm
E TRAINING CURVES ON SCIENCEWORLD
We also report the success rate curves and average step counts of different RI. training methods on ScienceWorld shown in Figure 6
Counterintuitively, when training GRPO or GiGPO on ScienceWorld, the average action steps do not decrease success rates improve; in some cases, the number even rises. This may be because Science World tasks require the agent not only to plan, explore, and reflect, but also to connect scientific theories to concrete actions, which smaller models may not perform sufficiently well. Early in training, agents often terminate trajectories early with incorrect answers before sufficient experimentation. As training progresses, these unproductive trajectories reduced, leading to
15

---

Published as a conference paper at ICLR 2026
Algorithm 1 RLVMR:
Reinforcement Learning with Verifiable Meta Meta-Reasoning Rewards
Require: Policy ma, Environment &, Reward function R. Hyperparameter Ameta
Ensure: Optimized policy parameters
1: for iteration t 1, ........ do
2: Initialize trajectory set D =
3: for episode 1, ................................ .N do
4: Get initial state from environment (
5: Initialize trajectory 7(0) = 0
6:
t 0
7:
while episode not terminated do
8:
Sample action ot (-)))
Execute action and observe 9 r1 E(s,",,))))) .
9:
10:
// Extract reasoning type tag
tag. -- ExtractReasoningTage // (planning). (explore). (reflection), (monitor)
12:
(0) < (0) U {(st)) at .r (()() tag; (
14:
t set+1
15:
end while
16:
// Compute Outcome Reward
R(0) Rauicome
(()
18:
end for
19:
Compute Meta Reasoning Rewards for all trajectories N do
20:
for episode i 1,2,.... ...,,
21:
for each step / in (1) do [6]
m ComputeMetaRewarddee (1) tag, ((), R
22:
[(] [(] (i)
23:
Attach meta-reasoning reward step
24:
end for
D DU {(7(0, R(lccome)
25:
26:
end for
Compute group relative advantage + {A
28:
29:
// Group Relative Policy Optimization
for update step k: ..... K do
31:
Compute policy gradient: Va.((0) = E,-D Vo log - -A] Update policy parameters: 8 (()
33:
end for
34:
end for
increase in average action steps as agents perform more comprehensive experiments. This reveals limitation RL: while can align LLM behavior with the environment, its effectiveness constrained by the foundation model's capabilities. Our method mitigates this by applying cold start phase, allowing the foundation model to acquire essential environmental knowledge. As result, action steps ScienceWorf are more stable and exhibit reliable convergence.
F PROMPTS
Below are the prompts we used in the ALFWorld and ScienceWorld environments.
Prompt Template for ALFWorld Enviroment
You are an expert agent operating in the ALFRED Embodied Environment. Your task to:
(task_descr- ieescriptii i on)
Prior to this step, you have already taken
step(s).
16

---

Published as a conference paper at ICLR 2026
Algorithm 2 Step-Level Group Relative Advantage Computation
Require: Trajectory data D {(r(), (2()) p J Weight Amota
Ensure: Advantage estimates {
J==1
1: // ===== Outcome Advantage Computation
2: Group by enviroment index: Goutooe
{gg} where = {i env wiw = j}
3: for each group Рд Goutcome do
R
4: Compute group mean: Mjj
iigg
oulcome
5: Compute group std: jj 19
Lieg; ( 11))
6: for i E 9j do
7:
A ootcooe
R(
4
8: end for
9: end for
10: // =/==== Meta Reasoning Advantage Computation ===-=
11: Group by (enviroment index, reasoning tag): Imera = {ggjkl
12: where 9j.k {i: env idx = A tag = =k}
for each group 9j.k E 9ineta do Uj,k 19..
Exeor, (f)
14: Compute group
'meta
15:
Compute group std: jj V 15..
Eregr, (r 15..))
16: for i E 9j.k do
17:
A A
(e]
Ma.k s.
18:
end for
19: end for
//
Final Advantage Combination
21: for i = 1,2,..., N do
22: A A Outcome + Ameta A (t) Aeta
23: end for
return {10)) M=1
SCIWORLD Success Rate
SCIWORLD Episode Length
0.6
Episode Length (mean)
Success Rate
0.3
10
0.1
60
80
100
120
140
20
40
60
100
Step
Step
RLVMR
GIGPO
GRPO
Figure 6: Success rate and step count curves of different approaches on Science during RL training.
Below are the most recent (history.length observations and the corresponding actions
you took: faction_histors You are now at step (current.step) and your current observation is:
(current.observationn
Your admissible actions of the current situation are: (admi cidmissibledd salbiissible.cttonns..
Your previous overall plan is: (planning). Please strictly adhere your plan.
17

---

Published as a conference paper at ICLR 2026
Algorithm 3 Meta-Reasoning Reward Computation
Require: Action de, State St, Reasoning tag tag,, Trajectory т, Outcome reward R(r)
Require: Reward hyperparameters (Mplan, Texplore; Treftea discount factor
Ensure: Meta-reasoning reward Timetatt
1: m +-
2: valid, f IsAction (ae)
3: if' valid, False then
4: return 0
Invalid actions receive no reward
5: end if
6: if tag, = (planning) then
7: if (((T) > 0 then
Planning rewarded only successful trajectories
8:
+ NumP NummlaaniingAfterrfee T)
9:
Ticta,1 < Tlaan
10: else
11:
Timetut 0
12: end
end if
isRepeated False 14:
if tag, (explore) then
16:
for = 0 to 10-100 do
Extract transition (st, 0e', Ae', 8e+1)
18:
if (8t, (z,, 8t+1) (8y, 0e, 88+1) then
isRepeated <- True break
20:
21: end
22:
end for
23:
if isRepeated = False then
24:
T'metat < Texplore
Novel transition
25: else
Tmeta.1 f- 0 end if
27:
end if
(reflection) then 29: if tag,
if then
31:
Extract previous transition (st-1,0(-1 valid, -1 +- Valiii(a
32:
33:
if valid; False and (st, ae) ($t-1, ac-1) then
34:
m + Preflect
> Effective reflection
35:
else
36:
immea,t +- O
37:
end if
38:
else
39:
Timet.:
40:
end if
end if
return l'meta,t
Now it's your turn to take action, following these steps:
1. First, reason using ONLY ONE tag pair and express your reasoning in one concise, brief sentence:
<planning> Plan replan the entire task by breaking it down into high- steps. Focus on outlining the full sequence required to complete the overall task, just the immediate next action. Use this the beginning of complex tasks or whenever the previous plan is incorrect or insufficient. It is necessary list all the points separately. eg, step xxx, step 2: xxx, step xxx, etc.
18

---

Published as a conference paper at ICLR 2026
<explore> When results unexpected information is lacking, rent observations to think outside the box and list as many possible locations, items, or actions as possible. Use this approach when facing obstacles that require creative and innovative thinking.
Preflectioon Analyze the reasons for emrors in task execution and correct them by exploring alternative approaches. 'No known action matches that input." indicates the action is invalid. This is typically used when several consecutive actions yield no substantial progress.
<monitor> Continuously track the current progress and history of reasoning and execution throughout the task. Recall the current subgoal and consider the next concrete action, ensuring agent alignment with the overall plan. Typically used when task outcomes are as expected and no other mode reasoning is required.
2. After your reasoning, you MUST select and present admissible action for the current step within <action> ... < //aciion> tags.
Specify the next action the agent should take to progress toward the task goal, following these guidelines:
(a) Object and Receptacle References: Use specific identifiers:
[ob] id] for objects (e.g., apple 1).
[recep id] for receptacles (e.g., countertop 1).
(b) Action Validity: Follow the exact format below. Any deviation renders the action invalid:
Valid actions:
go to [recep id], take [obj id] put [obj id] in/on [recep
from [recep id], id]. open/close [recep id), use [cbj id],
heat/cool/ [obj id] with [recep id].
Prompt Template for ScienceWorld Environment
You are an expert agent operating in the ScrldceWorld environment, which is text-based virtual environment centered around accomplishing tasks from the elementary science cur- riculum.
Your current task is: (task_description))
Prior to this step, you have already taken {step count} step(s).
Below the most recent (history.length) observations and the corresponding actions you took: (action.histon You are now at step (current.step) and your current observation is:
(ourrent.observatio)
Here are the actions you may take:
{
["action": "open OBJ", "description": "open container"}
("action": "close OBJ", "description": container")
"close
("action": "activate OBJ", "description": "activate a device")
["action": "deactivate OBJ", "description":
"decctivate a device"]
("action": "connect OBJ to CBJ", "description":
"coneect electrical components")
("action": "disconnect OBJ", "description":
"disconnect electrical components")
19

---

Published as a conference paper at ICLR 2026
("action": "use OBJ [on QBJ]", "description": "use a device/item")
["action": "look around", "description": "describe the current room")
("action": "look at OBJ", "description": "describe an object in detail")
I"action": "look in OBJ", "description': "describe a container's contents")
("action": "read OBJ", "description": "read note or book")
("action": "move OBJ to OBJ", "description": "move an object to container")
l"action": "pick up OBJ", "description": "move an object to the inventory")
{ l"action": "put down OBJ", "description": "drop an inventory item")
{ I"action": "pour OBJ into ORJ", "description": "pour liquid into a container")
{ :"action": "dunk OBJ into OBJ", "description": "dunk a container into a liquid")
("action": "mix OBJ", "description": "chemically mix a container")
("action":
"go to LOC", "description": "move to new
location")
["action": "eat OBJ", "description": "eat food")
:"action":
"flush OBJ", "description": "flush
toilet"}
"focus OBJ", "description": "signal intent on a task object")
("action":
("action": "wait", "description": "take action for 10 iterations")
("action": "waiti" "description": "take no action for 1 iteration")
("action": "task", "description": "describe current task")
("action"; "inventory", "description": "list your inventory")
Your previous overall plan is: (planning). Please strictly adhere to your plan.
Now it's your turn to take an action, following these steps:
1. First, reason using ONLY ONE tag pair and express your reasoning in one concise, brief sentence:
<pl Jolanningr
Plan replan the entire task by breaking down into high-level steps. Focus on outlining the full sequence required to complete the overall task, not just the immediate next action.
Use this at the beginning of complex tasks or whenever the previous plan is incorrect or insufficient.
It is necessary to list all the points separately, eg, step 1: xxx, step 2: XXX, step
xxx, etc.
20

---

Published as a conference paper at ICLR 2026
<explore>
When results are unexpected or information is lacking, use current observations think outside the box and list many possible locations, items, actions as possible.
Use this approach when facing obstacles that require creative and innovative thinking.
<refl seffeeciion
Analyze the reasons for errors in task execution and correct them by exploring alternative approaches. "No known action matches that input.' indicates the action is invalid.
This is typically used when several consecutive actions yield substantial progress.
<monitor>
Continuously track the current progress and history of reasoning and execution throughout the task. Recall the current subgoal and consider the next concrete action, ensuring agent alignment with the overall plan. Typically used when task outcomes are expected and other mode of reasoning is required.
2. After your reasoning, you MUST select and present an appropriate action for the current step within <action> ... </action> tags.
G CASE STUDY OF RLVMR
Case Study Below is trajectory segment of agent trained with RLVMR on ALFWorld, per- forming a novel task: find two soapbars and put them the cabinet. We observe that, guided by mega-cognitive rewards, the agent demonstrates more comprchensive and meticulous reasoning process, exhibiting significantly more adaptive reasoning patterns. At step 13, the agent is already carrying soapbar and is actively searching for the second. Lacking explicit information about the soapbars' location, the agent engages structured and efficient exploration by systematically hypothesizing possible locations via exploratory actions. Concurrently, keeps track of its current sub-goal, thereby avoiding the task confusion commonly seen in SFT-trained agents.
Notably, in step 14, the agent experiences brief hallucination, leading However, at step 15, rather than falling into an eπrr loop or blindly continuing, the agent immediately triggers reflective behavior, proposing 'double-check my inventory" to collect more information and guard against the possibility of having already acquired both soapbars. This demonstrates more rigorous and robust reasoning and decision- decision-making process. Experimental results show that such meta-cognitively driven behavioral patterns enhance agent performance and generalization previously unseen tasks compared baselines.
ineffective action.
A trajectory segment of an agent trained with RLVMR
Step 13
Observation: You arrive at sinkbasin 2.
On the sinkbasin 2, you see nothing.
Reasoning: <explore>>innce there are no soapbars at sinkbasin 2, I should broaden my search to other potential locations. The next logical area to check is the toilet, as soapbars could potentially be found there. </explo// Action: go to toilet 1
Step 14
Observation: You arrive at toilet On the toilet you a candle and spraybottle 3. Reasoning: <monitor>>y current sub-gou -o-aggaal is to obtain the second soapbar. Since there are no soapbars visible here, I will attempt to examine the environment further, just in case the second soapbar is present but not immediately listed in the observation.</monitoo </monitor>> Action: examine soapbar 2
21

---

Published as a conference paper at ICLR 2026
Step 15
Observation: Nothing happens.
Reasoning: Creflection>Reflectttin on my recent actions, realize failed to place the second soapbar in the cabinet earlier. To proceed efficiently, I need to double-check my inventory and ensure I'm actually carrying both soapbars before trying again. </reflecioon Action: inventory
Step 16
Observation: You are carrying: soapbar 1.
Reasoning: <monitor>AI this point, my sub-goal is still to locate the second soapbar. Since am currently only carrying soapbar 1, need to continue searching for the other o/e.</monito>>
Action: go to countertop
Step 17
Observation: You arrive at countertop On the countertop you see soapbottle 1, spraybottle 1, and a toiletpaper 1.
<explore>>No soapbar found on the countertop either, so should expand my search to nearby areas. Checking places like the toilet paper hanger may help ensure I'm not missing any possible locations for the second Action: go to toiletpaperhanger 1
Reasoning:
Step 18
Observation: On the toiletpaperhanger 1, you see a toiletpaper 3.
Reasoning: <explore>Since still haven't 'tt found new soapbar, should continue exploring other potential areas, such as the hand towel holders, to make sure don't overlook any possible spots where soapbar could be located.</explore Action: go to handtowelholder 1
H PARAMETER SENSITIVITY ANALYSIS
H.1 ADVANTAGE COEFFICIENT &
(a) Success Rate
(b) Average Trajectory Length
1.0
Average Trajectory Length
0.6
20
13
0.2
20
40
60
80
100
20
60
80
100
Training Step
Training Step
Figure 7: Sensitivity analysis of advantage coefficient a ALFWorld using Qwen2.5-1..B-Insimee
Our composite advantage function (defined in Algorithm 2 combines complementary signals: the outcome advantage Agpisodo (which guides the agent toward task success) and the meta-reasoning advantage Aug (which promotes effective step-level reasoning behaviors). The hyperparameter controls their relative weighting in the final advantage computation:
A(i) Aepisodee (i) a) Aua (i)
22

---

Published as a conference paper at ICLR 2026
To evaluate the robustness of our method this hyperparameter choice, performed sensitivity analysis on ALFWorld using Qwen2.5-1.55 --ssttrututtt with a E {0.0, 0.05, 0.2, 0.4, 0.6, 0.8, 1.0). The results shown in Figure
When a is very small (0.0 or 0.05), performance drops sharply because the local meta- reasoning reward overwhelms the global outcome signal; in fact, 0.0 removes the success feedback entirely, preventing effective learning.
When a is close 1.0, the model under- weights meta-reasoning behaviors, leading to noticeably degraded success rates and longer episode lengths, which harms the quality of reasoning within successful trajectories as well as the model's generalization ability.
Empirically, performance is most stable when is moderate. In this regime, outcome and meta- reasoning advantages provide complementary guidance: the model improves task success while also learning hugher-quality intermediate reasoning. We also observed that, as long as a is not near the extremes, its effect on training speed and final performance remains small and within the natural variance of RL training. Based these findings, and without evidence favoring more skewed weighting, we choose a 0.5 as balanced and robust setting.
H.2 IMPACT OF DISCOUNT FACTOR
To examine whether discounting factor further improve long-horizon learning, we introduce standard discounted return of the form
G.
rrinks
where 7 E (0, 1] controls the degree of temporal discount- ing. Setting 7 = recovers the undiscounted case used in our main experiments. We conduct controlled comparisons on ALFWorld using Qwen2.5-14 1.ve2.5--1.5B-Insff InStuntt evaluating both GRPO and RLVMR under 7 = and 7 < 1.
(a) GRPO
1.0
Success Rate
0.5
y=1.0
The results in Figure 8 reveal the following trends.
y=
0.0
For vanilla GRPO, adding a discount factor leads to slightly faster improvement at the beginning of training, which aligns with the intuition that discounting reduces the influence of late-stage noise in sparse reward settings. However, the improvement remains modest and does not fundamentally enhance overall performance or alleviate inefficient exploration key limitation we target this work.
50
100
150
200
Training Step
(b) RLVMR
1.0
Success Rate
0.5
y=1.0
For RLVMR, the effect of discounting is negligible. The two curves almost overlap, indicating that the explicit meta-reasoning rewards already provide dense, temporally structured guidance (e.g., discouraging repeated actions). As result, additional temporal discounting offers little benefit on top of the meta-reasoning signals. These find- ings further support the robustness of RLVMR'S 's reward design.
0=9
0.0
20
40
80
100
Training Step
Figure 8: Comparison methods with and without discount factor ALF- World using Qwen2.5-1.5B-Innsme
I ABLATION STUDY
I.1 ABLATION ON KEY COMPONENTS
We conduct ablation studies on the Unseen-L22 split using Swren..5-1.5B-Intrrue to analyze the impact of our framework's 's key components: the trajectory-level outcome advantage signal (AT). the meta-nncgn advantage signal (AMC), and the cold-start process (CS). The results in Table confirm that each component is critical for achieving optimal performance.
23

---

Published as a conference paper at ICLR 2026
Table 5: Ablation results on ALFWorld and ScienceWorld (success rates (%) on L2 variant).
Variant
ALFWorld
ScienceWorld
RLVMR (Full)
56.3
26.5
w/o AT (Outcome Reward)
12.5
7.8
w/o A A (Meta-Reat Reattanng Reward)
45.3
20.3
w/o CS (Cold-Start)
40.6
18.8
Verifiable meta-reasoning rewards are essential for tackling complex, unseen tasks. Removing the meta-reasoning advantage signal (AMC) causes significant performance drop, with the success rate on ALFWorld falling by 11.0 percentage points (from 56.3% to 45.3%) and on Science World by 6.2 points. This variant is equivalent model. The sharp decline validates our central hypothesis: directly rewarding beneficial reasoning processes crucial for developing robust problem-solving skills. This component directly addresses the "inefficient exploration issue" by providing dense, process-level signals that guide the agent toward more efficient and logical behaviors, benefit that outcome-only rewards (A")) cannot provide alone.
standard GRPO agent fine-tuned from the cold-star
Oased rewards remain indispensable for guiding the agent toward final task success. Eliminating the trajectory-level outcome advantage (A*) results in catastrophic performance collapse, with the success rate plummeting to just 12.5% on ALFWorld and 7.8% on World. This demonstrates that while meta-reasoning rewards effectively shape the process, the global signal of task success is vital for orienting the agent toward the ultimate goal. The meta-rcasoning rewards locally effective- for instance, rewarding non-repetitive exploration- but without the final outcome signal, the agent cannot learn which explorations ultimately lead to a successful trajectory. This confirms that the synergy between process-level and outcome-level rewards is key strength of the RLVMR framework.
A lightweight cold-start phase is critical for bootstrapping the agent's reasoning capabilities. Training the agent without the supervised fine-tuning cold-start (CS) phase leads to a substantial performance decrease on both ALFWorld (down 15.7 points) and Science World (down 7.7 points). The cold-start phase, which uses only 200 trajectories. is not intended to solve the tasks but to equip the model with the basic ability to generate syntactically correct meta-reasoning tags and follow instructions. For smaller models (e.g.. 1.5B), this initial grounding is vital; without it, the agent often fails to produce parsable outputs during RL, leading to training instability and policy collapse. This finding underscores the efficiency of our approach: a brief, low-data cold-start is sufficient to unlock the model's capacity for complex reasoning, which is then honed by the RL phase.
I.2 ABLATION ON SETTA-REASONING TYPES
To further understand the role of each meta-reasoning component, conduct fine- mne-grained ablation study by removing one type of meta meta-rcasoning tag and its corresponding meta-reasoning reward at time, while keeping all other settings unchanged. Experiments are performed on the ALFWorld L2 split using the Qwen2.5-11.5 1.5-2 -Isttructt model.
We evaluate models using the same metries as the main paper: (i) success rate, the percentage of episodes in which the agent completes the task; (ii) average trajectory length, the mean number of steps taken per episode, capturing overall efficiency and the agent's ability find direct solutions; (iii) repetitive action rale, the percentage of actions that repeat previous action without changing the environment state, quantifying inefficient exploration or loops; and (iv) invalid action rate, the proportion of actions that are not executable in the current environment state, reflecting basic comprehension and error frequency.
summarizes the results. Removing reflection significantly increases both repetitive and invalid actions, indicating that the agent struggles to recover from sequences of ineffective steps without an explicit mechanism for self-correction. Removing explore produces substantially longer trajectories, as the agent tends to fall into inefficient search patterns without leveraging contextual cues from history to guide exploratory decisions. Eliminating either planning or monitor also leads to clear
24

---

Published as a conference paper at ICLR 2026
performance degradation. We find that correct etrly-stage planning provides global structure for the task, reducing disorganized execution, while monitoring helps track subgoals and maintain adherence to the planned sequence.
Overall, these ablations demonstrate that each meta-reasoning type contributes meaningfully to robust long-horizon behavior. The meta-reasoning patterns operationalized from metacognitive theory therefore essential for improving both effectiveness and reliability the agent.
Variant
SR( ()) f
avg_steps
repeat( %)
invalid %
full RLVMR
56.3
15.4
5.7
12.5
w/o planning
47.5
15.9
8.6
12.8
w/o explore
55.8
17.2
12.6
16.1
w/o reflection
46.2
16.5
14.5
20.2
monitor
16.0
15.8
Table Ablation of meta-reasoning tag types on ALFWorld 1.2 with Qwven..5-1.5B-Instuee
1.3 IMPACT OF ANNOTATION MODEL CHOICE
Success Rate on ALFWorld
To further examine the dependence of RLVMR on the teacher model used in the cold-start SFT phase, we addi- tionally compare annotations generated by a strong closed- source model (GPT-4o) and much smaller open-source model (Llama-3.1-70B) Specifically, we replace the cold- start annotations with those produced by different teacher models to train the Qwen2.5-1. 1..52---..B--nntrruc model, fol- lowed by the subsequent RL process. As shown in Fig- ure [ the downstream RL performance under the two annotation sources almost identical across the training set and the most challenging L2 evaluation split for the Qwen2.5-1 1-..5-11.BB. 1--11.5BIIsst Instructt agents. This indicates that the an- notation task required in cold-start is relatively simple, and that RLVMR s final reasoning behaviors emerge primar- ily from the RL stage rather than from teacher-specifie annotation quality.
Success Rate
0.6
0.4
Annotate (Trii))
0.2
Libma3.1 708 Annotate (Vai L2)
GFT-4b Annotate (Ve1-
20
40
60
80
100
Training Step
Figure 9: Success-rate ALFWorld-L2 for Qwen2.5-1 5--..5--Innsttt under differ- ent cold-start annotation sources.
J THE USE OF LLMS
In this work, large language models were partially employed to assist with spelling and grammar checking, as well as minor text polishing. Specifically, used the following prompt:
You expert in Al. Please check the provided text for any spelling or gram- matical errors, and point out inappropriate expressions: (text segment]
No unverifiable content was produced by the LLMs, and all technical ideas, results, and conclusions presented in this paper originate from the authors.
25


Reviews:
[{"review_id": "UwDHJlfOcT", "reviewer": "Reviewer_qZCd", "summary": "This paper invesitgates the sparse reward problem in RLVR on LLMs for multi-turn long-horizon tasks. They identify an \"inefficient exploration\" problem which leads the agent to frequently output invalid or redundant actions, as the LLM is optimized to solely maximize the final succuss rate. They propose a reward shaping solution by adding more reward terms to reward meta-cognition behaviors like planning, exploration, reflection, and monitoring. Experimental results show that their method significantly outperform baseline RLVR methods on two benchmark ALFWorld and ScienceWorld, and nearly matches the performace of some strong close-sourced models like GPT-4o.", "questions": "1. Do you apply a discounting factor to your sparse reward function? As intuitively, I think using discounting factor is an easy approach to reduce repetive and invalid actions such that the task can be solved in a shorter time to gain higher reward. \n2. Have you applied the same cold start phase to the baseline methods? Just to make sure that different methods are compared in a fair way. If yes, do you apply SFT to the baseline methods with just the observation-action pairs? Or also with meta-reasoning tags?\n3. How scalable is your method to other long-horizon tasks? E.g., can the meta-reasoning rewards defined in your paper be useful also for other benchmarks, or are they specifically tuned for the two benchmarks used in your paper? And if I've got it right, you need some way to label the meta-reasoning reward for each step right? Do you label with some hand-designed rules or with another LLM? Can these labeling method generalize to other benchmarks we are interested in?\n4. Can you give some explanations on why the 7B and 8B variants tuned with your method significantly outperforms GPT-4o on ALFWorld and L0 of ScienceWorld, but gradually underperforms GPT-4o in L1 and L2 of ScienceWorld?\n\nI'm happy to raise my score if the authors can help clarify on these points and address my concerns.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly motivated with a detailed investigation on the ALFWorld benchmark. \n2. The paper is clearly presented, and the method is clearly explained. \n3. The paper shows significantly improvement upon baseline methods on the two benchmarks they use.", "weaknesses": "See my questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MHRDnQbqlk", "reviewer": "Reviewer_6tBG", "summary": "This work proposes RLVMR to augments end-to-end RL for LLM agents with rule-verifiable process rewards tied to four meta-reasoning tags (planning, exploration, reflection, monitoring). A brief SFT phase teaches the tag format; online training then optimizes a clipped policy objective using a blend of trajectory-level and tag-grouped step advantages. On ALFWorld and ScienceWorld, RLVMR reports SOTA success and large drops in invalid/repetitive actions, attributing gains to improved reasoning quality rather than shortcut paths.", "questions": "* Could you report a brief ablation of $\\alpha$ (Eq. 4) and the format-penalty weight?\n\n* Could you precisely define the process-level rewards (for planning, exploration, reflection, monitoring) in both ALFWorld and ScienceWorld?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Clearly targets inefficient exploration and quantifies it with invalid action rate and repetitive action rate, tying process quality to task success.\n* Simple, practical method: explicit meta-reasoning tags, verifiable meta-reasoning rewards, and a tag-grouped relative advantage blended with a trajectory-level relative advantage in a clipped objective.\n* Consistent gains across base models and benchmarks, especially for the harder split; also shows shorter, more stable solution paths.\n* Goes beyond final accuracy with behavior-quality metrics and training stability, reducing degenerate loops.\n* Ablation studies indicate each component matters (outcome advantage, meta-reasoning advantage, cold-start SFT, format penalty).", "weaknesses": "* The claim that this work is “the first study offering a definitive explanation and comprehensive analysis of the inefficient exploration issue” overstates its novelty.\nThe idea that outcome-only RL reinforces flawed reasoning paths has already been recognised in prior works on process reward models and step-level or action-type-conditioned rewards. These earlier studies also analyse how intermediate reasoning quality affects exploration efficiency and generalisation.\n\n* Despite claiming “verifiable” process rewards, the paper does not provide the exact rule logic needed to compute them.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P7t1Kfh2hw", "reviewer": "Reviewer_NHAZ", "summary": "This paper proposes RLVMR, a new reinforcement learning framework that incorporates verifiable meta-reasoning rewards to improve long-horizon reasoning in large language model (LLM) agents. Traditional RL methods in this domain (like GRPO) optimize for sparse, final outcome rewards, which often reinforce inefficient or illogical reasoning paths.  RLVMR addresses this by introducing dense, process-level supervision via rule-based rewards for meta-reasoning behaviors such as planning, exploration, reflection, and monitoring.\n\nThe method combines a brief supervised “cold-start” phase, where a teacher model (e.g., GPT-4) annotates reasoning tags, with a critic-free policy gradient optimization phase (GRPO-MR). Each reasoning tag receives local verifiable rewards that shape the reasoning process in addition to global task rewards. Experiments on ALFWorld and ScienceWorld benchmarks show that RLVMR achieves state-of-the-art (SOTA) performance", "questions": "First of all, I would like to thank the authors for their work. I agree that reinforcement learning (RL) post-training for large language models (LLMs) often encourages inefficient or illogical reasoning paths, and I appreciate that this paper tackles such an important research question.\n\nHere are a few concerns. With the introduction of a custom reward design, how do the authors ensure that the model does not engage in reward hacking? Since the model is trained to maximize cumulative rewards, this could potentially exacerbate the issue the paper aims to address.\n\nAdditionally, how do the authors ensure that the teacher model generates correct or reliable tags? And finally, how do different types of tags (e.g., planning vs. reflection) contribute individually to the overall performance?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- clear motivation and well presented\n- Improved efficiency and generalization", "weaknesses": "- lack of theoretical analysis of the composite reward which can leads to reward hacking\n \n- Dependence on teacher annotation, since one powerful teacher LLM is used without guarantee\n\n- Limited ablation on tag definitions, missing ablation on the contribution of individual meta-reasoning tags", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "r3UtSoIuTX", "reviewer": "Reviewer_hzw2", "summary": "This paper introduces RLVMR, a RL framework designed to improve the robustness and generalization of long-horizon agents. The authors identify a key problem they term inefficient exploration, where standard outcome-only RL reinforces flawed or redundant reasoning paths that happen to lead to success. To address this, RLVMR integrates dense, process-level rewards based on verifiable meta-reasoning behaviors. The agent is trained to explicitly output tags for cognitive steps. These tags are then rewarded programmatically based on rules that encourage efficient and logical problem-solving (e.g., rewarding exploration that discovers new states). These process-centric rewards are combined with the final task success reward and optimized using a group-based policy gradient method. The authors demonstrate SotA performance on the challenging ALFWorld and ScienceWorld benchmarks, showing significant improvements in success rates, generalization to unseen tasks, and reductions in inefficient actions.", "questions": "1.  Could you provide more concrete details on the programmatic implementation of the meta-reasoning rewards? Specifically for the `<reflection>` reward, what is the exact rule used to determine a \"sequence of failures\" and a subsequent \"corrective action\"?\n2.  The advantage signal is a linear interpolation $A_t = \\alpha A^{\\text{traj}} + (1-\\alpha) A^{\\text{MR}}$. How was $\\alpha=0.5$ chosen? Have you performed a sensitivity analysis on this hyperparameter, and how do the results change with different values of $\\alpha$?\n3.  The reward for `<planning>` is granted only if the entire trajectory succeeds. This seems to defeat the purpose of a dense, process-level reward, as the feedback for the initial plan is still sparse and delayed. Have you considered alternative, more immediate rewards for planning, such as evaluating plan quality independently of the final outcome?\n4.  The cold-start SFT phase relies on annotations from a superior model (GPT-4). Did you analyze whether the agent simply learns to mimic the meta-reasoning style of GPT-4, and could this potentially limit the agent's own emergent problem-solving strategies during the RL phase?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "*   **Originality & Significance:** The paper introduces a novel approach to supervising the reasoning process of LLM agents, moving beyond sparse outcome-based rewards. The idea of rewarding verifiable meta-reasoning steps is a significant and practical contribution to building more robust agents that not only solve tasks but do so efficiently and logically.\n*   **Quality:** The experimental evaluation is of high quality. It is comprehensive, including multiple strong baselines, different model sizes, and rigorous generalization testing. The results are compelling and demonstrate improvement over prior work.\n*   **Clarity:** The paper is written with clarity. The motivation, methodology, and results are presented in a clear and intuitive manner, making the work accessible.\n*   **Impact:** The work directly addresses a fundamental challenge in agent training and provides a practical, scalable solution that achieves SotA results.", "weaknesses": "*   **Heuristic Reward and Advantage Design:** The programmatic rules for assigning meta-rewards and the method for combining advantage signals are heuristic. For example, the planning reward is still tied to the final task success, making it sparse. The paper would be strengthened by a discussion of alternative designs or a more formal justification for the current choices. A sensitivity analysis on the weighting parameter $\\alpha$ is also missing.\n*   **Vagueness in Reward Implementation:** The paper could be more specific about the implementation of the programmatic reward rules. For instance, how is a \"corrective action after a sequence of failures\" (for the reflection reward) precisely defined and detected? Providing more detail would improve reproducibility.\n*   **Dependence on \"Cold-Start\" SFT:** The method relies on an initial SFT phase using a powerful teacher model (GPT-4) to learn the tag syntax. The ablation study shows this step is critical. While the authors frame it as \"lightweight,\" it still constitutes a dependency that could introduce teacher model biases and complicates the training pipeline compared to a pure RL approach.\n*    **Source Code:** While the pseudocode for RLVMR is provided in Appendix D, sharing the actual implementation would greatly enhance reproducibility.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)