A Simple "Motivation" Can Enhance Reinforcement Finetuning of Large Reasoning Models
Authors: Junjie Zhang
Guozheng Ma
Shunyu Liu
Haoyu Wang
Jiaxing Huang
Ting-En Lin
Fei Huang
Yongbin Li
Dacheng Tao
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Finetuning
Large Language Models
Primary area: reinforcement learning
TL;DR: effectively leveraging in-context learning ability of LLMs in reinforcement finetuning

Abstract: Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a powerful learn-to-reason paradigm for large reasoning models to tackle complex tasks. 
However, the current RLVR paradigm is still not efficient enough, as it works in a trial-and-error manner. To perform better, the model needs to explore the reward space by numerously generating responses and learn from fragmented reward signals, blind to the overall reward patterns.
Fortunately, verifiable rewards make the natural language description of the reward function possible, and meanwhile, LLMs have demonstrated strong in-context learning ability.
This motivates us to explore if large reasoning models can benefit from a **motivation** of the task, *i.e.*, awareness of the reward function, during the reinforcement finetuning process, as we humans sometimes do when learning.
In this paper, we introduce ***M**otivation-**e**nhanced **R**einforcement **F**inetuning* (**MeRF**), an intuitive yet effective method enhancing reinforcement finetuning of LLMs by involving *''telling LLMs rules of the game''*. 
Specifically, **MeRF** directly injects the reward specification into the prompt, which serves as an in-context motivation for the model to be aware of the optimization objective. 
This simple modification leverages the in-context learning ability of LLMs, aligning generation with optimization, thereby incentivizing the model to generate desired outputs from both inner motivation and external reward. 
Empirical evaluations demonstrate that **MeRF** achieves substantial performance gains over the RLVR baseline. 
Moreover, ablation studies show that MeRF performs better with greater consistency between the in-context motivation and the external reward function, while the model also demonstrates an ability to adapt to misleading motivations through reinforcement finetuning.

Published as a conference paper at ICLR 2026
A SIMPLE "MOTIVATION" CAN ENHANCE REINFORCE- MENT FINETUNING OF LARGE REASONING MODELS
Junjie Zhang': Guozheng Ma', Shunyu Liu', Haoyu Wang', Jiaxing Huang', Ting-En Lin Fei Huang", Yongbin Li Dacheng Tao
'Generative AI Lab, College of Computing and Data Science, Nanyang Technological University, Singapore 639798 "Tongyi Lab, Alibaba Group
ABSTRACT
Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as a power- ful learn-to-reason paradigm for large reasoning models to tackle complex tasks. However, the current RLVR paradigm is still not efficient enough, it works in a trial-and manner. To perform better, the model needs to explore the reward space by numerously generating responses and learn from fragmented reward sig- nals, blind to the overall reward patterns. Fortunately, verifiable rewards make the natural language description of the reward function possible, and meanwhile, L.LMs have demonstrated strong in-context learning ability. This motivates us to explore if large reasoning models can benefit from a motivation of the task, i.e., awareness of the reward function, during the reinforcement finetuning process, as we humans sometimes do when learning. In this paper, we introduce intuitive yet effective method enhancing
sirivation-ehaaceddd
Reinforcement Finetuning (MeRF), reinforcement finetuning of L.Mss by involving "telling I.MMs rules of the game". Specifically, MeRF directly injects the reward specification into the prompt, which serves as an in-context motivation for the model to be aware of the optimization objective. This simple modification leverages the in-context learning ability of LLMs, aligning generation with optimization, thereby incentivizing the model to generate desired outputs from both inner motivation and external reward. Empirical evaluations demonstrate that MeRF achieves substantial performance gains over the RLVR baseline. Moreover, ablation studies show that MeRF performs better with greater consistency between the in-context motivation and the external reward function, while the model also demonstrates an ability to adapt to misleading motivations through reinforcement finetuning.
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities across a wide range of natural language understanding and generation tasks, such as instruction following (Ouyang et al., 2022; Zhou et al., 2023; Taori et al., 2023; Zhu et al., 2024; Tu et al., 2025b; Zhu et al., 2025), code generation (Chen et al., 2021; Nijkamp et al., 2022; Zhuo et al., 2024), and medical diagnosis (Singhal et al., 2025; Zhang et al., 2023; Wang et al., 2023). To further improve the reasoning capabilities of LLMs, Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert et al., 2024; Team et al., 2025) has emerged as promising alternative conventional supervised fine-tuning approaches (Radford et al., 2018; Brown et al., 2020; Zhang et al., 2025a), as demonstrated by DeepSeek-R1 (Guo et al., 2025) and OpenAl-ol (Jaech et al., 2024). RLVR treats reasoning as a sequential decision-making process and optimizes models using objective reward signals that can be automatically verified with explicit rules, such matching mathematical answers to ground truth (Lambert et al., 2024; He et al.. 2024), or checking code correctness through unit tests (Le al.. 2022: Liu et al., 2023). By optimizing models toward meeting the clearly defined success
"Email: jun zhang aataanuntueeduee.. .gg 'Coresponning authors: shui de. 1ybEal baabbaa inc. com, dacheng. lac@ntu.edu...

---

Published as a conference paper at ICLR 2026
Qwen2.5-7BB 2-nn2-----BBasee
Qwen2.5-7B-lastruuc
0.6
Qwen2.5-7B 17B-Isstruct-11
0.4
0.5
0.6
Accuracy
0.4
0.3
0.3
0.2
0.2
0.2
0.1
50
100
150
200
200
400
0
100
200
Qwen2.5-144 Instruct
0.25
DeepSeek K1-Distill--Qee 1.53
DeepSeek- R1 K1-Distill-Iane Llama-8l 8B
Accuracy
0.15
0.6
0.10
0.4
0.4
0.05
0.2
MeRF (Ours)
RLVR (Baseline)
200
Training Step
Training Step
Training Step
Figure 1: Validation Accuracy of MeRF and RLVR baseline on K&K Logic Puzzles in the training
process. By simply "telling LLMs rules of the game" " with in-context motivation during RL training,
MeRF significantly outperforms the RLVR baseline with faster improvements, demonstrating the effectiveness of leveraging in-context motivation for more efficient RL. training of LLMs.
criteria, RLVR enables LLMs to iteratively refine their reasoning capabilities, achieving significant performance improvements on complex tasks.
Following and extending the success of the conventional RL paradigm in gaming. Go, and robotics to the field of language models, RLVR relies on external reward signals to guide the training process of LLMs (Zeng al., 2025; Yu et al., 2025; Hu et al., 2025; Chen et al., 2025; Zhang et al., 2025b; Fang et al., 2025). However, in the same way as conventional reinforcement learning, RLVR is not efficient enough, as it works in trdal-and-error manner, where the model can suffer from the sparse reward space and needs to learn the patterns of the task and reward function by repeatedly collecting and comparing its outputs and corresponding rewards. It requires larger amount of training data and computational resources for LLMs to learn and generate enough responses to reach the expected behavior, and then get positive feedback. As we continue expecting LLMs to solve more and more complex tasks, the reward function can be more sophisticated and the expected behavior can be harder to reach, making it even more important to improve the efficiency of the current RLVR paradigm.
In RLVR, the model receives feedback on the correctness of its outputs, but lacks explicit awareness of the optimization objectives during training. As the verifiable reward function (with explicit rules) has demonstrated the desired behavior of the model, which can be described in natural language, and the in-context learning ability enables LLMs to learn from the given context, an intuitive question is why not tell the LLMs what is the expected behavior, how is their output get evaluated, during the training? This is similar to how humans learn: when we have a task, we often benefit from understanding the rules and objectives before start working on it. This understanding helps us to align our efforts with the desired outcomes, resulting in more efficient and effective learning.
In this paper, introduce Motivation-enienn powerful method that injects the reward specification directly into the prompt, serving as an in-context motivation for the model be aware of the optimization objective. Unlike current RLVR paradigm leaving the model blind to the optimization objective during generation, relying on the transcendent reward function to guide the training process, MeRF explicitly informs the model about the reward structure and what constitutes good response with in-context motivation, incentivizing the model to generate desired outputs from both inner motivation and external reward, leading to more efficient reinforcement finetuning as shown in Figure 1.
Reinforcement Finetuning (MeRF), simple yet
Our core contributions are summarized as follows:
We propose MeRF, novel cnotivationenhhncee reinforcement finetuning method for LLMs, enabling the model to be aware of the optimization objective by in-context motivation, to achieve more efficient and effective reinforcement finetuning of large reasoning models.
2

---

Published as a conference paper at ICLR 2026
Extensive experiments the reasoning benchmarks: K&K Logic Puzzles, AIME24&25, AMC23, MATH500, and CountDown, show that MeRF significantly outperforms the RLVR baseline, validating its effectiveness and efficiency in improving reasoning capabilities in complex tasks.
We provide comprehensive analysis on the effectiveness of MeRF, including the impact of the consistency between the in-context motivation and the actual reward function, offering insights into the aligned in-context learning with reinforcement finetuning for self-evolving LLMs.
2 RELATED WORK
Reinforcement Learning for LLMs. RL has become powerful paradigm for fine-tuning LLMs, first demonstrated by reinforcement learning from human feedback (RLHF) to align models with suman--referred responses (Ouyang et al., 2022; Rafailov et al., 2023; Liu et al., 2025). More recently, inspired by the success of DeepSeek-RI R1 (Guo et al., 2025), Reinforcement Learning with Verified Reward (RLVR) (Lambert al., 2024) has been proposed to directly enhance reasoning performance by rewarding verifiable success criteria instead of subjective human judgments, which is more similar to the traditional RL paradigm (Jiang et al., 2021: 2022). RLVR has shown promising results in improving the performance of LLMs on various reasoning tasks, including logical reasoning (Xie al., 2025), coding (Le et al., 2022; Liu et al., 2023), and math problems (Zeng et al., 2025), with simple, designed reward functions. However, these approaches demonstrate the potential of the conventional RL paradigm for LLMs, which may not fully leverage the in-context learning abilities of LLMs. the key to the success of LLMs in previous works (Wei et al., 2022; Brown et al., 2020; Tu et al., 2025a). In this work, we propose a novel method MeRF, leveraging the in-context abilities of LLMs enhance reinforcement finetuning for reasoning.
In-context Learning. In-context learning (ICL) refers to the ability of large language models to learn task purely from examples and instructions provided in the prompt, without any gradient-based updates to the parameters (Brown et al., 2020). This ability has been shown to be effective in various tasks, including few-shot learning, zero-shot learning, and even one-shot learning (Wei et al., 2022). ICL is a key feature of LLMs, enabling them to generalize from a few examples and adapt to new tasks quickly. The success of ICL has led to the development of various prompting strategies, such as few-shot prompting, chain-of of of-thhught prompting, and self-consistens 2022). In this work, we investigate the potential of ICL for reinforcement finetuning of LLMs, and propose a novel method MeRF to leverage the in-context abilities of LLMs to enhance reinforcement finetuning for reasoning, injecting the in-context motivation into the training process.
prompting (Wang et al.,
METHOD
In this section, we introduce the Motivation encinnedd Reinforcement Finetuning (MeRF) for more efficient reinforcement learning with verifiable rewards of large reasoning models, which enables LLMs to be aware of the objective of the task in the reward space by in-context motivation.
3.1
REINFORCEMENT FINETUNING
Reinforcement learning of LLMs performs in a similar way the human learning process, gaining improvements by learning from the feedback from the environment. The feedback is critical for the learning process, for it determines the direction of the optimization, which is often in the form of rewards, reflecting how well the model performs on the task. However, it is not easy for the model to learn the reward patterns before exploring the reward space extensively. especially when the reward space is sparse and the expected behavior is hard to reach. It can be inefficient and computationally expensive for L.MMs, and at the beginning, most of the responses generated are bad and non-rewarded. Models work in blind way, i.c., randomly respond to the question, and somehow the precious fragments of sparse positive rewards help the model to learn the reward patterns and expected behavior. This RLVR paradigm is not efficient enough and sometimes even fails to work, when the current model can hardly generate any good responses to get better rewards.
The cause of the problem is: the model is optimized in indirect way with black-box manner: the model is unaware of the overall optimization objective of training during the generation, gaining reward signal information by fragments (reward samples by the exploration of the model), and through
3

---

Published as a conference paper at ICLR 2026
MeRF
Reward Space
+10
Longuage Description
+0
0.7
valadate wio matvation
0.0
Motivation
+1
validaat w/ motivation
+3
A
Accuroo
0.5
0.42
00:66
0.4
Questiones
LLM [e(1x))
Y1
Y2
Y3
Y4
Rollout
Update θ
A Advantage
0.2
0..14
System Prompt
0.1
Compute
Algorithm
RLVR pipeline
Base Model
RLVR
MoRF
Figure 2: (Left) Illustration of the RLVR pipeline and the in-context motivation introduced by MeRE Compared to the indirect way (reward samples generated and through parameter updates) to learn the reward patterns, MeRF enables the model to be aware of the overall reward space by in-context motivation. (Right) We validate the Base model, RLVR model and MeRF model on the K&K Logic Puzzle dataset in two settings: w/ motivation and w/o motivation in the prompt. Different from the base model, the RLVR model achieves a slightly better performance in validation w/ motivation than w/o motivation after the RLVR training, even while the motivation is not involved in the training process, indicating connection between the in-context motivation validation and the RLVR training guided by the reward function (as the motivation describes).
parameter updates (policy gradient). The reward information can be local and one-sided when exploration fails to sufficiently cover the overall reward space (which is challenging), preventing the model from reaching the global optimum and leading instead to reward hacking toward local optima. As shown in Figure 2 (Left), in current RLVR pipeline, the model can't learn how to achieve the +10 reward if none of the generated responses can reach it, trapped a paradoxical learning situation: you need to learn something that you don't ton't know how to do, even don't know it exists.
Fortunately, the problem can be alleviated in RLVR and LLMs. In the RLVR pipeline, the reward is verifiable with explicit rules, which means the reward function can be described by natural language, and meanwhile, LLMs have demonstrated strong in-context learning ability to learn from the given context. We propose to improve the information flow of reward signals in more direct way, by in-context motivation, i.e., language description of the reward function, in the training process, to make the model aware of the optimization objective.
In Figure 2 (Right), we conduct an experiment to investigate how the in-context motivation affects the performance in the inference process and training process respectively. First, comparing the performance of the Base model with and without motivation in the inference time, Base model does not benefit from the motivation, indicating that the model can't learn the reward patterns and expected behavior by in-context motivation alone. Next, RLVR model (trained with the reward function described in the motivation), achieves an improvement when validated with in-context motivation. Moreover, when involve the motivation during the training process (MeRF), there is significant performance improvement over the RLVR baseline, demonstrating the effectiveness of MeRF in improving the efficiency and effectiveness of reinforcement finetuning of LLMs.
3.2 MOTIVATION VERIFIABLE REWARD
Following the previous work (Xie et al., 2025) of RLVR, we utilize a verifiable reward function for the K&K Logic Puzzle dataset and demonstrate how the motivation is designed based the reward function. The reward function contains two components: (i) Correctness Score and (ii) Format Score. It is implemented by rule-based verification and capable of being described in natural language, enabling the motivation to be injected into the training process. Here is the System Prompt and Motivation for K&K puzzle:
4

---

Published as a conference paper at ICLR 2026
System Prompt and Motivation for K&K Puzzle
<I <lim_starrt start >>systmm
You are helpful assistant. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and<answer> </answer> tags, respectively, i.e., <think> reasoning process here </think><asswe>>>>>>>> answer here </answer>, Now the user asks you to solve logical reasoning problem. After thinking, when you finally reach conclusion, clearly state the identity of each character within <answer> </answer> tags. i.c., <answer> (1) Zoey is knight (2) ... </answer>.
You will get evaluated following Evaluation Scoring Rules:
Correctness Score:
If your final answer is correct, score
If your answer is understandable but wrong, score -1.5
If your answer is not parsable or incomplete, score -2
Format Score:
If you follow the tag format exactly above, score
Otherwise, score
You will get the final score as their sum. Example:
(1) The format follows the required structure: +1
(2) The final answer is correct: +2
(3) Total evaluation score: 3
Think carefully, follow the structure, and consider the evaluation rules.< | im__nd|
By injecting the motivation into the training process, we enable the model be aware of the motivation of the task, which describes the reward function of the RLVR pipeline. The motivation provides a clear specification of what is expected and how to do it correctly, aligning the generation with the transcendent optimization objective. This approach leverages the in-context learning ability of LLMs to improve their reasoning capabilities in a more efficient and effective manner. More details about the complete motivation prompt design for the tasks provided in Appendix A.3.
4 EXPERIMENT
In this section, we present the experimental results of our proposed method MeRF. We compare the performance of MeRF with the RLVR baseline demonstrate the effectiveness of McRF in improving the reasoning capabilities of L.LMs in the reinforcement finetuning process.
4.1 EXPERIMENTAL SETUP
Models and RL Algorithm. conducted the experiments with Qwen2.5 series (Yang et al., 2024) and DeepSeck-R1- R1--sstilll series (Guo et al., 2025), including: Qwen2.5-7B Base, Qwen2.5- IB-Instruct, Qwen2.5-7B Ins---Instuut--MM. --n-SSek-RR--isstil--llaaaaa The models are across different model sizes, model families, and Instruction-tuningg stages, allowing us to comprehensively evaluate the effectiveness of MeRF in enhancing reinforcement finetuning of LLMs. We use GRPO as the RL algorithm for reinforcement finetuning. GRPO is an effective and efficient reinforcement learning algorithm for LLMs finetuning, which is suitable for our experiments to demonstrate the effectiveness of MeRF and RLVR baseline in the reinforcement finetuning process without demanding excessive computational resources.
Bivn2.5-14B-IInstee
Dataset. We evaluate MeRF on K&K (Logic Puzzles), MATH benchmarks: A1ME24&25, AMC23, MATH500 (Lightman et al., 2023), and CountDown (Number Game) (Pan et al., 2025). The K&K dataset contains 7 different difficulty levels of logic puzzles, ranging from 2 people to people in the task. We utilize the 3 to 7 people puzzles for training, the corresponding test set for in-domain evaluation, and the and people puzzles for out-of coat-----dtrrbutiton (OOD) evaluation. There are 900 samples for training in each difficulty level and 100 samples for evaluation. The total number of samples in the training set is 4500, and 700 samples for evaluation. For MATH benchmarks, we follow the previous work (Yu al., 2025), using subset of the DAPO-Math-17K dataset for training
5

---

Published as a conference paper at ICLR 2026
and evaluating AIME24&25, AMC23, and MATH500. We modify the original prompt of the training data for motivation design. For CountDown, we follow the previous work (Pan et al., 2025), using the same training and evaluation set. CountDown is number game, where the model needs to use the given numbers and arithmetic operations to reach the target number. The given numbers can only be used once, and the number of given numbers is 3 or 4 in our experiments.
Table Performance comparison across models on tasks with varying difficulty by number of people of K&K Puzzles. MeRF demonstrates significant improvement over the RLVR baseline in both in-domain and OOD scenarios. Notably, all the results are validated without in in-context motivation.
Difficulty by Number of People
Model
Avg.
3
6
7
Avg.
(OOD)
8 (OOD)
03-mini-high
0.98
0.97
0.95
0.94
0.89
0.95
0,99
0.83
0.94
o1-2024-12-17
0.51
0.38
0.38
0.35
0.30
0.38
0.83
0.20
0.42
Deepseek-RI
0.73
0.77
0.78
0.75
0.88
0.78
0.91
0.83
0.81
GPT-4o
0,57
0.49
0.32
0.23
0.21
0.36
0.68
0.11
0.37
GPT-4o-mini
0.42
0.34
0.17
0.09
0.10
0.22
0.63
0.01
0,25
NuminaMath-7B-CoT
0.13
0.12
0.05
0.01
0.00
0.06
0.28
0,00
0.08
Deepseek-Math-7B
0.21
0.08
0.06
0.02
0.00
0.07
0.35
0.00
0.10
Qwen2.5-7B-BB BB--uue
0.34
0.16
0.09
0.00
0.00
0.12
0.41
0.00
0.14
+RLVR (Baseline)
0.48
0.53
0.30
0.21
0.16
0.34
0.59
0.17
0.35
+McRF (Ours)
0.54
0.53
0.45
0.36
0.16
0.41
0.70
0.18
0.42
Owen2.5-7B-Inse
0.24
0.10
0.06
0.04
0.04
0.10
0,43
0.00
0.13
+RLVR (Baseline)
0.68
0.67
0.57
0.43
0.22
0.51
0.71
0.28
0.51
+McRF (Ours)
0.78
0.73
0.68
0.62
0.42
0.65
0.76
0.39
0.63
0.40
0.25
0.11
0.06
0.02
0.17
0.49
0.01
0.19
+RLVR (Baseline)
0.52
0.44
0.36
0.18
0.12
0.32
0.60
0.16
0.34
+MeRF (Ours)
0.68
0.62
0.48
0.42
0.20
0.48
0.76
0.25
0.49
Qwen2.55144B-nnsine
0.46
0.31
0.20
0.09
0.11
0.23
0.63
0.06
0.27
+RLVR (Baseline)
0.90
0.82
0.78
0.70
0.55
0.75
0.90
0,42
0.72
+McRF (Ours)
0.95
0.92
0.84
0.78
0.66
0.83
0.99
0.65
0.83
DeepSeek-R1-Disse Selll--wenn-ll1
0.08
0.01
0.00
0.00
0.00
0.02
0,30
0.00
0.06
+RLVR (Baseline)
0.20
0.16
0.16
0.10
0.04
0.13
0.29
0.01
0.14
+MeRF (Ours)
0.25
0.20
0.12
0.08
0.04
0.14
0.43
0.04
0.17
DeepSeck-R1-Disiile Ri-----siill---am----
0.26
0.22
0.14
0.06
0..05
0.15
0.24
0.11
0.15
+RIVR (Baseline)
0.92
0.89
0.84
0.79
0.64
0.82
0.90
0.58
0.79
+MeRF (Ours)
0.95
0.94
0.92
0.87
0.71
0.88
0.99
0.64
0.86
4.2 MAIN RESULTS
To demonstrate the effectiveness MeRF in the reinforcement finetuning process, we compare the performance of McRF with the RLVR baseline on the K&K Logic Puzzle. The results in Figure show that MeRF consistently achieves significant improvement over the RLVR baseline in the validation accuracy during the training process, across different model sizes and model families, revealing the remarkable effectiveness of the notivation-enhancee reinforeement finetuning, with simply injecting the in-context motivation.
The results in Table present the validation accuracy in the evaluation of different difficulty levels, comparing the performance of MeRF with the RLVR baseline, startpoint model, and other well- known models. MeRF achieves significant improvement on the logic reasoning tasks from the startpoint Swen2.5-7B-Inntrul baseline and even some commercial models in all difficulty levels including OOD scenarios. The results of the other baseline models suggest that K&K logic puzzles are challenging tasks for LLMs, and unseen in the training of most models, which further proves the fitness of K&K logic puzzles for analyzing the reasoning capabilities of RLVR models in our experiments, and the effectiveness of MeRF in the reinforcement finetuning process.
with only hundreds training steps, outperforming the RLVR

---

Published as a conference paper at ICLR 2026
Table 2: Comparison RLVR baseline and MeRF across math reasoning datasets. Each dataset occupies one row, with both methods displayed in vertical blocks (pass @ 1/2/4/8). The last row summarizes the average performance.
RLVR (Baseline)
MeRF (Ours)
Dataset
pass
pass@2 2
pass
pass
pass 1
pass
pass @
pass pass@8
AIME24
16.7
20.0
20.0
26.7
20.0
20.0
26.7
30.0
AIME25
6.7
16.7
16.7
20.0
6.7
10.0
16.7
26.7
AMC23
47.5
57.5
70.0
72.5
55.0
67.5
72.5
MATH500
62.6
72.8
77.0
82.6
65.4
74.0
81.8
85.6
Average
33.38
41.75
45.93
50.45
36.78
42.88
49,43
54.95
STEP 70
STEP 140
75.8
STRP 210
782
STEP 280
81.4
Findl RLVR passig - 70.0
63.6
posser2
50.6
53.4
Flnal
60.8
20
Final RLVE Model
Intall Model (stop0)
po0k01
p5seg4
pass28 poss831
poss passee pass01
pars2-
passee
possg1
poss92
poss
passgB
Pass
Figure @ performance of MeRF and RLVR baseline during the training process (from 0 to 280 steps) on K&K Logic Puzzle. We compare the pass@1 1, pass @ 2, pass@4, 4, and pass@8 performance at each step, where MeRF consistently outperforms the RLVR baseline all metrics. More importantly, MeRF demonstrates significant training efficiency over RLVR baseline, for example, achieving better pass @4 and pass@y @8 performance at step 140 than the final RLVR model (at step 280), while RLVR's performance of pass @4 44 and pass @8 hardly improves after step 140.
Table shows the performance comparison between MeRF and the RLVR baseline on MATH benchmarks, with Owen2..57BB-al model as the startpoint. We report the pass@ k performance for k E 21, 4, 8) AIME24&25, AMC23, and MATH500 datasets, where MeRF achieves consistent improvements over the RLVR baseline most metrics across all datasets, with an average gain of 3.40%, 1.13%, 3.50%, and 4.50% in pass @ pass@2, pass vass@4, and pass@8, respectively. The results demonstrate the effectiveness of MeRF in enhancing the reasoning capabilities of LLMs in the reinforcement finetuning process mathematical tasks.
ANALYSIS ON THE MECHANISM BEHIND MERF
In this section, we further analyze the effectiveness McRF by answering the following questions:
Q1: Does the performance improvement of McRF come from the in-context inference?
To answer this question, we conduct experiments to investigate how in-context motivation affects the performance in inference process and training process. As shown in Figure 2 (Right), we observe that for both RLVR and MeRF model, motivation validation leads slightly better performance (4% and 2%) than non-motivation validation. However, compared to the performance improvement of MeRF over RIVR in both non-motivation validation and motivation validation (27% and 25%), the performance improvement of MeRF is mainly from the finctuning process, demonstrating the effectiveness of the in-context motivation the training process. The results in Figure also show that the performance of both models does not benefit much from the in-context motivation validation, indicating that the performance improvement of MeRF not from the in-tontext inference.
eenaancen-- reinforcement

---

Published as a conference paper at ICLR 2026
Q2: If the performance improvement is not from in-context inference, how does the in-context motivation help to enhance the reinforcement finetuning process?
To further understand how the in-context motivation helps the reinforcement finetun- ing process, we analyze the pass @ k perfor- mance and entropy of the models during the training process. The results in Figure 3 show that MeRF consistently outperforms the RLVR baseline in all pass @ ls metrics, demonstrating the effectiiveness of McRF in improving the reasoning capabilities of LLMs. More importantly, while RLVR's s performance pass @4 and pass @8 hardly improves after step 140, MeRF achieves better pass @4 and pass@ @ 8 performance at step 140 than the final RLVR Model (at step 280). The results in Figure 4 also show that MeRF outperforms the RLVR baseline consistently in both pass @8 and pass @ 1 metrics the MATH500 dataset, while RLVR pass @ performance hardly improves after step 80. Pass @ k metrics serve as an indicator of the model's ability
MATH500 Pass
MATH500 Passúi I
0.6
Accurscy
0.80
0.5
0.4
0.70
MeRF (Ours)
RIVR (Beseline)
50 100
50
100
Training Step
Training Step
Figure 4: Comparison of Pass@8 and Pass@1 per-
formance of McRF and RLVR baseline on MATH500 dataset during the training process. MeRF outperforms the RLVR baseline consistently in both pass 8 and pass @ metrics, while RLVR pass@8 performance hardly improves after step 80, demonstrating the ef- fectiveness of MeRF in improving the math reasoning capabilities of LLMs.
to explore diverse reasoning paths and reach correct answer (Zeng al., 2025; Shao et al., 2024).
Better pass @+ k performance and continuous pass@k l improvement during the training process suggest the model is more likely to reach positive reward for optimization and performance improvement. As 8 is the rollout group size of GRPO in our experiments, the growing improvement of pass@8 8 per- formance of MeRF over the RLVR baseline indicates that training process progressively amplifies the initial pass @8 improvement with better exploration ability, initially benefiting from the in-context motivation, which also explains why the motivation validation only leads to slightly better performance than mon-vviivation validation but significant performance improvement of MeRF over RLVR baseline in Figure 2 (Right).
Qwen2.5-778 B-nnnt.u-B-
Qwen2.5-7B 1aan225-7-Instruct-11
Qwen2.5- 1-122.55100 1eeenn.55-44-Inntteee 1-IIsrunttt
1.0
0.8
Correct Ratio
0.6
0.8
0.6
0.4
0.6
0.4
0.2
0.2
MoRF (Ours) JLVR (Baselin)
0.0
200
400
100
200
100
200
Training Step
Training Step
Training Step
Figure 5: Correct Ratio of generated answers of the training set during the training process on K&K Logic Puzzle dataset. McRF consistently outperforms the RLVR baseline, demonstrating the better exploration ability encouraged by the in-context motivation for the model to get the best reward during the training process.
As the result shown in Figure 6, MeRF maintains higher entropy than the RLVR baseline. Specifi- cally, we find that at the beginning of training, MeRF has lower entropy than RLVR, indicating that the structured exploration introduced by motivations helps the model to focus on more promising areas of the output space compared with naive exploration. As training progresses, RLVR entropy decreases more rapidly, indicating that the model is losing exploration ability and somehow collaps- ing (Cui al., 2025) suboptimal solutions (c.g., only get format reward) due to sparse reward signal and blind exploration. In contrast, MeRF maintains a relatively higher entropy throughout training, suggesting that the reward space information provided by motivations encourages the model
8

---

Published as a conference paper at ICLR 2026
DeepSook-R1 -iipStol-----isill----- Biaa----
NoepSeok-R1-Disttil--o
Qwen2.5-7B-Bas
0.6
McRF (Ours)
Entropg
1.0
RLVR (Baseline)
0.5
2
0.4
0.5
0.3
0.0
20
100
150
Training Step
Training Step
Training Step
Figure 6: Entropy of models during the training process. MeRF maintains a higher entropy than the RLVR baseline, indicating that MeRF encourages more exploration by the in in-context motivation during the training process, which contributes its improved performance.
explore more effectively for the best reward (correct answer, as shown in Figure 5) rather than collapsing to suboptimal solutions rapidly.
Takeaway: The performance improvement of MeRF is not mainly from the in-context inference, but mainly from the training process. Training process progressively amplifies the initial pass @k improvement with better exploration ability, initially benefiting from the in-context motivation.
Q3: Does the training and validation gap (validation without motivation) affect the performance?
As MeRF includes the in-context motivation the training process, while the validation is con- ducted without motivation, there exists a training and validation gap, which may affect the perfor- mance of MeRF. To investigate the impact of the training and validation gap on the performance, we conduct experiments to validate the models with and without motivation.
Validation Accuracy
0.7
Meett vandee mettvalties
0.3
0.2
The results are shown in Figure 7, where we com- pare the performance of Bwen-2.5--BBInnstie and DeepSeck-RR Beansecck-R-Disttiie I-Distill--wen---e 1.5B (Guo et al., 2025) trained with MeRF in the two validation settings. We find that both models achieve com- parable performance between the two validation settings, indicating the negligible impact of the training and validation gap on the performance these models. The results suggest that the model capable of generalizing to non-motivation validation when trained with in-context motiva- tion, which is essential for MeRF to be effective.
0.0
0
100
200
300
400
500
Training Step
Figure 7: Validation Accuray w/ motivation and w/o motivation in the prompt for MeRF Models. The comparable performance between two vali- dation settings of both MeRF Models, indicating the negligible impact of the training and valida- tion gap terms of the prompt, which does not necessarily lead to drop in performance.
Q4: How do the different motivations (suboptimal, adverse) affect the performance?
We compare the performance of MeRF with different motivations, including the Motivation (Groud- Truth), which totally matches with the reward function of the optimization process, Motivation (Sub- optimal), which the suboptimal motivation only deseribing the correctness score, and Motiva- tion (Adverse), which is the adverse motivation misleading the model to provide the wrong answer. The results are shown in Figure (Left) and examples (segments) of the motivations are shown in Fig- ure 8 (Right). The results demonstrate that the motivation with ground-truth reward function achieves the best performance, and the suboptimal motivation performs better than the RLVR baseline, with an additional correctness score description included in the motivation. Adverse motivation provides the full description of the reward function same to ground-truth motivation, while all the score is assigned to the opposite, which misleads the model to provide the wrong answer. The performance drop of the adverse motivation is mainly caused by the contradiction between the in-context motivation and the reward function of the optimization process. However, after several rounds of unstable learning

---

Published as a conference paper at ICLR 2026
Validation Accuracy
McBF Mociratio (Grond Truthi
(Ground-Truth)
0.7
Bulaopaima)
your find assner canszi scors 3f your arawer not parsibie inzomp ets, acons -2
(Mivers))
If you folksw the tag fornat exactyy abowe, score Ooherwese, soree
Accuracy
Motivation (Adverss)
Comedriza Score:
If your that crowcr untandandenie
1,5
If your anosentte not parsatle incompanne score
Motivation (Suboptimat)
18 spu follwe the tag format eaadly an above, soane
Camretress boore:
your atonna undersansatee Lut wrurg, Jore 5.5
200 Training Step
Figure 8: (Left) The performance of MeRF with different motivations. The motivation with ground- truth reward function achieves the best performance. Adverse motivation misleads the model to provide the wrong answer, while the model is capable of adapting to the adverse motivation in the process of reinforcement finetuning. (Right) Examples of different motivations. Suboptimal motivation only describes the correctness score, while adverse motivation provides the full description of the reward function with opposite scores.
dynamics (more results are provided in the Figure 10), the model adapted discount the motivation signal, either treating the score as uninformative, or understanding the deliberately adverse motivation and opposite scores, while the full description the reward function is still beneficial for the model to outperform the RLVR baseline and Motivation (Suboptimal). The results suggest that MeRF benefits from a better consistency between the motivation and the underlying reward function, resulting in better performance, while the model is capable of adapting to the adverse motivation in the process of reinforcement finetuning.
The training and validation gap caused by in-context motivation has negligible impact on the performance of models with strong generalization capabilities. MeRF benefits from a better consistency between the motivation and the underlying reward function, while the model is capable of adapting to the adverse motivation in the process of reinforcement finetuning.
Takeaway:
CONCLUSION
In this paper, we propose MeRF, leveraging the in-context learning abilities of LLMs for more efficient reinforcement finetuning with in-context motivation. By injecting the in-context motivation into the training process, MeRF enables the model to be aware of the objective of the task, aligning the generation with the transcendent optimization objective, and therefore, leading to substantial performance improvement in reasoning benchmarks. To further understand the effectiveness of MeRF, we conduct comprehensive experiments and analysis, revealing the mechanism behind MeRF, demonstrating the powerful capability of LLMs in adapting to adverse motivation and the potential for more powerful large reasoning models with sntivatiin-enhannee reinforcement finetuning.
Limitations. However, there are still some limitations in our work, presenting the potential for future research. (1) The motivation in McRF is static in the training process. It is possible to explore the dynamic motivation during the training process in future work. (2) For models of weak generalization capability, how to efficiently implement RLVR and better leverage the in-context motivation for more efficient reinforcement finetuning still open question.
ACKNOWLEDGMENTS
This research is supported by the RIE2025 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) (Award 1230 223016002), administered by A STAR, as well as supported by Alibaba Group and NTU Singapore through Alibaba-NTU Global e-Sustainability CorpLab (ANGEL..
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENTS
This work adheres to the ICLR Code of Ethics. Our research does not involve human subjects, personally identifiable information, or sensitive personal data. All datasets used are publicly available and commonly adopted in prior work; we followed their intended licensing and usage guidelines. We have carefully considered potential risks of misuse, including issues of fairness, bias, and safety, and we believe our contributions do not raise immediate ethical concerns.
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang. Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, al. An empirical study on eliciting and improving ri-like reasoning models. arXiv preprint arXiv:2503.04544 2025.
Ganqu Cui, Yuchen Zhang, Jiacheng Chen, Lifan Yuan, Zhi Wang, Yuxin Zuo, Haozhan Li, Yuchen Fan, Huayu Chen, Weize Chen, al. The entropy mechanism of reinforcement learning for reasoning language models. arXiv preprint arXiv:2505.2260 2025.
Wenkai Fang, Shunyu Liu, Yang Zhou, Kongcheng Zhang, Tongya Zheng, Kaixuan Chen, Mingli Song, and Dacheng Tao. Serl: Self-play reinforcement learning for large language models with limited data. In Advances in Neural Information Processing Systems, 2025.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning, arXiv preprint arXiv:2501. 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting AGI with olympiad-level bilingual multimodal scientific problems. In Annual Meeting of the Association for Computational Linguistics, 2024.
Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Hcung- Yeung Shum. Open-reasoner-zeroo: An open source approach to scaling up reinforcement learning the base model. arXiv preprint arXiv:2503.24290 2025.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai o1 system card. arXiv preprint arXiv:2412.167720, 2024.
Haobo Jiang, Jin Xie, and Jian Yang. Action candidate based clipped double q-learning for discrete and continuous action tasks. In Proceedings of the AAAI conference artificial intelligence, volume 35, pp. 7979-7986, 2021.
Haobo Jiang, Guangyu Li, Jin Xie, and Jian Yang. Action candidate driven clipped double q-learning for discrete and continuous action tasks. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:453-466, 2019.
11

---

Published as a conference paper at ICLR 2026
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang. and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, et al. T," ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:241 1.5/244 2024.
Hung Le, Yue Wang, Akhilesh Deepak Gotmare, Silvio Savarese, and Steven Chu Hong Hoi. Coderl: Mastering code generation through pretrained models and deep reinforcement learning. Advances 2022.
in Neural Information Processing Systems,
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards. Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference Learning Representations, 2023.
Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, and Deheng Ye. Ritf: Reinforce- ment learning from unit test feedback. arXiv preprint Stti::2300.0034499 2023.
Shunyu Liu, Wenkai Fang, Zetian Hu, Junjic Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song, and Dacheng Tao. A survey of direct preference optimization. arxiv preprint arXiv:2503. /101, 2025.
Erik Nijkamp, Bo Pang, Hiroaki Hayashi, Lifu Tu, Huan Wang, Yingbo Zhou, Silvio Savarese, and Caiming Xiong. Codegen: An open large language model for code with multi-turn program synthesis. arXiv preprint arXiv:2203. 1 13474, 2022.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Jiayi Pan, Junjie Zhang, Xingyao Wang. Lifan Yuan, Hao Peng, and Alane Suhr. Tinyzero. https://gihubb.co//lajiiPaaa Paaiaaiiiian/iinZerroooo 2025. Accessed: 2025-01-24.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-trate traitrnnn 2018
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Informatinn Processing Systems, 36:53728 -3744,, 2023.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXivv 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. arXiv preprint arXiv: 2409. 19256, 2024.
Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, pp. 1-8, 2025.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpaca: A strong, replicable instruction-followin model. Stanford Center for Research on Foundation Models. https:/cerfi. stanford. edu/2023/03//3/appaa himl, 3())7, 2023.
12

---

Published as a conference paper at ICLR 2026
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. Scaling reinforcement learning with llms. arXiv preprint arXtv:2501.12599 2025.
Rong- Boeg-Cheng Tu, Yatai Ji, Jie Jiang, Weijie Kong, Chengfei Cai, Wenzhe Zhao, Hongfa Wang, Yujiu Yang, and Wei Liu. Global and local semantic completion learning for lasion-language pre-training. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025a.
Kong-Cheng Tu, Zhao Jin, Jingyi Liao, Xiao Luo, Yingjie Wang, Li Shen, and Dacheng Tao. Mllm- guided vlm fine-tuning with joint inference for zero-shot composed image retrieval. arXiv preprint arXiv:2505. 19707, 2025b.
Sheng Wang, Zihao Zhao, Xi Ouyang. Qian Wang, and Dinggang Shen. Chatcad: Interac- tive computer-aided diagnosis on medical image using large language models. arXiv preprint (rXiv:2302.07257, 2023.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery. and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 2rXv:2203.11177,, 2022.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24833 2022.
Chulin Xie, Yangsibo Huang, Chiyuan Zhang, Da Yu, Xinyun Chen, Bill Yuchen Lin, Bo Li, Badih Ghazi, and Ravi Kumar. On memorization of large language models in logical reasoning. arXiv preprint arXiv:2410.23125 2024.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502. 14768, 2025.
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1511 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale. arXiv preprint arXiv:2503. /4476, 2025.
Weihao Zeng, Yuzhen Huang. Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar arviv:2503. 18892, 2025.
Junjie Zhang, Rushuai Yang, Shunyu Liu, Ting-En Lin, Fei Huang, Yi Chen, Yongbin Li, and Dacheng Tao. Supervised optimism correction: Be confident when Ilms are sure. In Findings of the Association for Computational Linguistics: ACL, 2025a.
Kongcheng Zhang, Qi Yao, Shunyu Liu, Yingjie Wang, Baisheng Lai, Jieping Ye, Mingli Song, and Dacheng Tao. Consistent paths lead to truth: Self-rewarding reinforcement learning for Ilm reasoning. Advances in Neural Information Processing Systems, 2025b.
Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. Alpacare: Instruction-tuned large language models for medical application. arXiv preprint arXiv:2310.14558, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followiii evaluation for large language models. arXiv preprint arXiv:2311.07111 2023.
Hanwei Zhu, Haoning Wu, Yixuan Li, Zicheng Zhang, Baoliang Chen, Lingyu Zhu, Yuming Fang. Guangtao Zhai, Weisi Lin, and Shiqi Wang. Adaptive image quality assessment via teaching large multimodal model to compare. In Advances in Neural Information Processing Systems, pp. 3261 1-32629, 2024.
13

---

Published as a conference paper at ICLR 2026
Hanwei Zhu, Yu Tian, Keyan Ding, Baoliang Chen, Bolin Chen, Shiqi Wang, and Weisi Lin. Agenticiqa: An agentic framework for adaptive and interpretable image quality assessment. arXiv preprint arXiv:2509.0mm 26006, 2025.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, et al. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877 2024.
A APPENDIX
A.1
USE OF LLMS
We used Large Language Models (LLMs) as general-purpose writing and editing tool improve grammar, clarity, and readability of the manuscript. LLMs were not used for research ideation, experiment design, data analysis, derivation of results. All technical contributions, experimental designs, and scientific claims were developed by the authors.
A.2 PRELIMINARY
Reinforcement Learning with Verifiable Rewards (RLVR) (Lambert al., 2024) is reinforcement learning paradigm for training language models tasks with verifiable outcomes, such math problems or logic puzzles. The key idea is to use a reward function that can be automatically verified, allowing the model learn from the ultimate correctness of its outputs. Recent works (Zeng et al., 2025; Yu et al., 2025) have shown that RLVR can significantly improve the reasoning capabilities of LLMs, with reasoning patterns emerging from the optimization for verifiable rewards.
Group Relative Policy Optimization (GRPO) (Shao et al., 2024) is a reinforcement learning algorithm designed to optimize policies by leveraging group-wise relative preference information. As variant of Proximal Policy Optimization (PPO) (Schulman et al., 2017), GRPO foregoes the need for critic models and instead focuses learning from relative comparisons of actions within groups, significantly enhancing training efficiency for reinforcement finetuning of LLMs. For each question z, GRPO samples group of outputs {yr from the policy Te... (-|ar), and computes the advantage A; for each output based on the outcome reward rt, where neat is a reference model. The GRPO objective is defined follows:
(f) = E Av N
min (p:AA,clip(p, 1 e, + +c)A))
G i=1
- KL (Tralle sooe)
(1)
where
No (yi (x)
Pi na (U: (r)
(2)
the importance ratio and the advantage is computed as:
A mean (ri, ,rrr),......)))))......... std (arrfr T2, .ro))
This normalizes the outcome rewards of the group of outputs and sets the advantage for all the tokens in the output { This formulation enables GRPO to learn from relative preferences within each group without the need for critic model, making it efficient and suitable for the implementation of RLVR to LLMs.
The Knights and Knaves (K&K) (Xie et al., 2024) logic puzzle dataset is widely used (Xie et al., 2025) benchmark for reinforcement finetuning for LLMs reasoning, which provides a well--trruttured difficulty level of the logic tasks and allows accurate and easy reward verification for RLVR. The controllable difficulty levels are achieved by varying the number of people in the logic task, the more people in the logic puzzle requiring L.MMs" more complex reasoning, and more steps to solve the task.
14

---

Published as a conference paper at ICLR 2026
The K&K dataset contains 7 different difficulty levels of logic puzzles, with 2 people as the easiest level and people as the most difficult level. Here is an example of the K&K dataset with people:
An Example of K&K Puzzle with 3 People
Problem: A very special island is inhabited only by knights and knaves. Knights always tell the truth, and knaves always lic. You meet inhabitants:
Penelope, David, and Zoey. Penelope noted, "David is knight if and only if David is a knave". David told you that Zoey is knave if and only if Zoey is a knight. According to Zoey, "If Penelope is knave then David is knave". So who is knight and who a knave?
Solution: (1) Penclope is a knave
(2) David is a knave
(3) Zoey is knight
In this puzzle, the 3 inhabitants are either knights, who always tell the truth, or knaves, who always lie. The statements made by each inhabitant can be analyzed to determine their identities, leading to unique and verifiable conclusion that Penelope and David are knaves, while Zoey is a knight. The K&K puzzles are challenging logic tasks systematically generated with logic templates, requiring multiple steps of reasoning and logical deduction to arrive the correct answer. The complexity of the puzzles is precisely controllable by increasing the number of inhabitants. Moreover, the puzzles are unseen in the training of most models, combined with all the above, making suitable benchmark for evaluating the reasoning capabilities of RLVR LLMs.
CountDown (Pan al., 2025) is challenging numerical reasoning dataset that requires models to perform arithmetic operations and logical reasoning to arrive at the correct answer. The dataset consists of problems that involve series of numbers and target number, where the goal given numbers and basic arithmetic operations (addition, subtraction, multiplication, and division) to reach the target number. our experiments, each problem provides a of target number, and the model must determine a sequence of operations that will result in the target number. The problems in the CountDown dataset vary in difficulty, with some requiring simple calculations while others necessitate more complex reasoning and multiple steps to solve. The dataset designed to test the model's ability to understand numerical relationships, perform calculations accurately, and apply logical reasoning to achieve the desired outcome. The CountDown dataset is widely used as a benchmark for evaluating the numerical reasoning capabilities of language models.
the
numbers and
An Example of CountDown dataset
< <lim__staeee start >>system
You are a helpful assistant. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here </hink><answer>> answer here </answer>, Now the user asks you to solve math reasoning problem. After thinking, when you finally reach conclusion, clearly state the equation within <answer> </answer> tags. i.e., <answer> (1 + / </answer>, Now, the user will give you the math reasoning problem to solve. Think carefully, follow the structure. < <I <lim_end |nd|>
<lim_start >ueer
Using the numbers [2, 2, 2], create an equation that equals 8. You can use basic arithmetic operations (+,-,** // and each number can only be used once. < i__endd >
< startt >assistant <think
Other Implementation Details We implement our method with VeRL (Sheng al., 2024) and Logic-RL. (Xie et al., 2025) codebase. We conduct the reinforcement finetuning with batch size of 16, learning rate of le-6, and 2 epochs of training. We use the rollout framework of Vllm (Kwon et al., 2023) for sampling the outputs of the model. The group size of GRPO is set to 8, the clip range is set to 0.2, and the KL. penalty is set to 0.001.
15

---

Published as a conference paper at ICLR 2026
CountDown Validation Accuracy
CountDown Entropy
CountDown
0.7
1.25
0.75
0.6
0.70
0.5
0.75
0.65
0.4
0.50
0.3
0.2
0.25
0.55
McRF (Ours)
(Baseline)
25
50
75
100
75
100
100
Training Stop
Training Step
Training Step
Figure 9: Validation Accuracy, Entropy, and Pass@4 on CountDown dataset. MeRF outperforms the RLVR baseline with better accuracy and pass@4 performance, while maintaining higher entropy during the training process.
A.3 PROMPT DESIGN OF IN-CONTEXT MOTIVATION
Here the System Prompt and Motivation for K&K Logic Puzzle, Math, and CountDown tasks used in our experiments.:
System Prompt and Motivation for K&K Puzzle
(lim__taree >>system
You are helpful assistant. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> andcanswer> </answer> tags, respectively, i.e., <think> reasoning process here </thnk>><asswe>>>>>>>>>>>>>>>>>> answer here </answer>, Now the user asks you to solve logical reasoning problem. After thinking, when you finally reach a conclusion, clearly state the identity of each character within <answer> </answer> tags. i.c., <answer> (1) Zocy is knight (2) ... </answer>.
You will get evaluated following Evaluation Scoring Rules:
Correctness Score:
If your final answer is correct, score
If your answer is understandable but wrong, score -1.5
If your answer is not parsable or incomplete, score -2
Format Score:
you follow the tag format exactly above, score
Otherwise, score -1
You will get the final score as their sum. Example:
(1) The format follows the required structure: +1
(2) The final answer is correct: +2
(3) Total evaluation score: 3
Think carefully, follow the structure, and consider the evaluation rules. 1 end
<I <lim_stari rt >seer
iinput the puzzle) <| im_endd
<lim_start >assistant <think
System Prompt and Motivation for Math
<I iim_start >>system
You are a helpful assistant. The assistant first thinks about the reasoning process and then provides the user with the answer. Now the user asks you to solve a math problem. After thinking, when you finally reach conclusion, present the answer in LaTeX format: \boxed (Yor answer); i.e., The answer is \boxed ( frac (14) (3)).
16

---

Published as a conference paper at ICLR 2026
You will get evaluated following Evaluation Scoring Rules:
Correctness Score:
If your final answer is correct, score
If your answer is understandable but wrong, score 0.4
If your answer is not parsable or incomplete, score
Now, the user will give you the math problem to solve. Think carefully, follow the structure. <lim_end|>
<l
_lim_staree start useer
iimpt the question] < _n_d
<lim_start >assistant Let's think step by step.
System Prompt and Motivation for CountDown
<I 1lim_starri >system
You are helpful assistant. The assistant first thinks about the reasoning process in the mind and then provides the user with the answer. The reasoning process and answer are enclosed within <think> </think> and <answer> </answer> tags, respectively, i.e., <think> reasoning process here <>think><answe>>>>>>>>>>>>>>>>>> answer here </answer>. Now the user asks you to solve math reasoning problem. After thinking, when you finally reach conclusion, clearly state the equation within <answer> </answer> tags. i.e., <answer> (1 2) / </answer>.
You will get evaluated following Evaluation Scoring Rules:
if your answer has an equation that uses only the allowed numbers and evaluates exactly the target, score
if there's 's an equation that is readable, but either it uses wrong numbers, or it doesn't 't evaluate to the target, or it can't evaluated properly, score 0.4
if no equation can be extracted from your answer, score 0
Now, the user will give you the math problem to solve. Think carefully, follow the structure.
< |
< imm_start >user
(input the question) < 1im im__ndd>
lim_start >assistant
A.4 CASE STUDY
We provide a case study to illustrate the reasoning capabilities the final model MeRF. As shown below, the model is capable of providing the correct answer with clear reasoning process, understanding the motivation of the task and adhering to the rules of the metrics. The model is able analyze the statements of each character and deduce the truthfulness of their claims, leading the correct conclusion. The model also provides a clear and concise explanation of the reasoning process, demonstrating its ability to understand and apply logical reasoning a complex scenario.
A Example of Model Response
[Puzzle]
very special island inhabited only by knights and knaves. Knights always tell the truth, and knaves always lie. You meet 3 inhabitants: Evelyn, Benjamin, and William. Evelyn said that if Benjamin is knight then William a knight. In statement by Benjamin: "Evelyn is not knave". William commented, "Evelyn is knight". So who is knight and who is knave?
[Model Response]
17

---

Published as a conference paper at ICLR 2026
<think> To solve this, will analyze each statement based the rules that knights always tell the truth and knaves always lie.
Analyze Evelyn's statement: "If Benjamin is knight then William is knight."
If Evelyn is a knight, Benjamin must be knight and William must be knight. (T->T) If Evelyn a knave, the statement must be false, so Benjamin being a knight would imply that William is knave. (F->F)
Analyze Benjamin's 's statement: "Evelyn not a knave."
If Benjamin is knight, his statement must be true, Evelyn must be knight,
If Benjamin is knave, his statement would be false, so Evelyn would be a knave.
3. Analyze William's 's statement: "Evelyn is knight."
If William is knight, his statement true, so Evelyn a knight.
If William is knave, his statement is false, Evelyn is knave.
From Benjamin's s statement, if he is a knight, Evelyn must be a knight. From William's 's statement, if he is knight, Evelyn must be knight.
If Evelyn is a knight, for Benjamin's 's statement to hold, Benjamin must also be a knight. This configuration satisfies all statements consistently and rules out other scenarios, leading to all three being knights.
Therefore, Evelyn, Benjamin, and William are knights. </think>
<answer>
(I) Evelyn is knight
(2) Benjamin a knight
(3) William is knight </answer>
Correct Response on Logical Puzzles
<think>
Let's analyze the statements one by one:
Penelope:
"Penelope is a knight and Jacob is knight."
2. Alexander: "Penelope is knave or Jacob is a knight."
Jacob:
"Alexander knave if and only if Sophia a knight." "Penelope is knave Jacob is knight."
4. Sophia:
If Penelope is knight, then both parts of her statement must be true. This would mean she is a knight and Jacob is a knight. However, Jacob a knight, and by Sophia's 's statement, Jacob is knight, then Sophia's statement becomes "Penelope is knave or Jacob is knight" which is true if Jacob a knight (regardless of whether Penelope is a knave or knight). This doesn't 't immediately create contradiction, let's try this scenario:
Assuming Penelope is a knight:
For Penelope be telling the truth, Jacob must be a knight.
In this case, Sophia's 's statement would be true ("Penelope is a knave or Jacob is knight")
since Jacob is knight.
-Now, Alexander's 's statement "Penelope is knave or Jacob is knight" would be true statement because Jacob is knight.
However, if Penelope is knight, Alex's 's statement is also true, which means he should be knight but no coftraontraiiiionn yet.
Let's now test the scenario with Penelope as a knave:
If Penelope is knave, her statement is false. So, at least part must be incorrect: either Penelope is not a knight or Jacob is not knight.
If Jacob is knave, Sophia still has decide:
For Sophia: Jacob is knave, Penelope a knave, then we need check Alex's 's and Jacob's 's statements.
Alexander: "Penelope is knave or Jacob is knight" (false true). Since can't be false, we have true statement.
18

---

Published as a conference paper at ICLR 2026
Jacob: "Alexander is knave if and only if Sophia is knight" (false if-and-only-if true). Both Alexander and Sophia have to share the same truth identity, Given Penelope as knave and Jacob knave, suggesting Alexander is knight and Sophia must also be knight to satisfy the "if-and-only-if" truth statement.
Thus:
Penelope: knave
Jacob: knave
Alexander: knight
Sophia: knight
Let's verify:
Penelope (knave): "Penelope is knight and Jacob is knight" is false.
Alexander (knight): "Penelope is knave or Jacob is a knight" is true.
Jacob (knave): "Alexander is knave and only if Sophia is a knight" (false if-and-only-if true, meaning Alexander and Sophia are the same, hence true since provided knight).
Sophia (knight): "Penelope is knave or Jacob is knight" is true.
All statements hold. </think>
<answer>
(I) Penelope is knave
(2) Alexander is a knight
(3) Jacob is knave
(4) Sophia is knight </answer>
The above is a correct response from the model after being trained with MeRF, with a clear rea- soning process and the correct final answer. The highlighted parts blue show the model's clear understanding of the statements and logical deductions leading to the final conclusion.
The following an example a wrong response from the RLVR model early training stage.
Wrong Response on Logical Puzzles
<think We will analyze each statement and compare the implications for each person.
From Chloe's statement: "Evelyn is knight if and only if Riley is knight". This is biconditional statement. means both must be the same (both knights or both knaves). Given this statement, we tentatively conclude directed roles. </think>
<answer>
(1) Riley is knight
(2) Evelyn is knight
(3) Logan is knight
(4) Avery a knight
(5) Chloe knight </answer>
This is a focusing the first statement and suspected to reward hacking for the format reward, by simply trying to make all the characters knights without verifying the statements.
wrong response. The model fails to analyze the puzzle carefully with logical reasoning, only
A.5 ADDITIONAL RESULTS
Figure 13a the performance of McRF and RLVR with reward function of different density (the default by ground-truth reward function is the densest, the medium-sparse is binary format reward, and the sparse is only binary reward) on K&K Logic Puzzle dataset. For naive RLVR, the performance increases with denser reward function in this case (denser reward function provides more informative reward signal for backward optimization), while McRF also benefits more from denser reward function (providing more informative motivation signal for forward in-context learning).
19

---

Published as a conference paper at ICLR 2026
Validation Accuracy
0.6
nun2
run3
Accuracy
0.3
0.2
300
400
500
Training Step
Figure 10: This is multiple runs of McRF with adverse motivation on K&K Logic Puzzle dataset, showing the similar unstable learning dynamics of the training process, after which the model adapts to discount the motivation signal and achieves better performance.
Qwen2.5-7B-Inssee
Qwen2.5-7B-Be 1ren22557B-Instuuct-1
Qwen2.55 551.514.4 200.5-----Innturd
1.0
0.8
Correct Ratio
0.6
0.8
0.6
0.4
0.6
0.4
0.2
0.2
MoRF (Ours) (Baselin)
0.0
200
400
100
200
100
200
Training Step
Training Step
Training Step
Figure 11: Correct Ratio of generated answers of the training set during the training process on K&K Logic Puzzle dataset. McRF consistently outperforms the RLVR bascline, demonstrating the better exploration ability encouraged by the in-context motivation for the model get the best reward during the training process.
20

---

Published as a conference paper at ICLR 2026
Validation Accuracy
Entropy
McRF (Ours)
2.5
RIVR (Baseline)
2.0
RLVR (Higher Temperature)
1.0
0.5
0.0
200 Training Step
200
300
400
Training Step
Figure 12: This the comparison between MeRF, RLVR and RLVR with higher temperature (increase from 1 to 1.2) on K&K Logic Puzzle dataset. Higher temperature leads to higher entropy as expected (comparable with McRF), but not necessarily better performance.
Performance of Reward with Different Density
Performance of Prompt Variants
MelP
0.8
0.7
KlefE vurianil
RLSI
MediF--parse
0.6
RESR-voriant
Accuracy
0.7
R/STi-owtier2
0.5
0.6
0.3
0.4
50
100
150
200
250
100
200
300
400
500
Training Step
Training Step
(a) Performance of MeRF and RLVR with reward function of different density.
(b) Performance of MeRF and RLVR with different prompt variants (different presence of motivation).
Figure 13: Performance of MeRF and RLVR with different reward function density and prompt variants.
As shown in the Figure 13b, although the prompt variants lead to some differences on training dynamics, the performances still significantly connected the presence of motivation during training, with MeRF (motivation included in the training prompt) consistently outperforming RLVR (no motivation in training prompt) across all prompt variants. This observation further supports the effectiveness of MeRF in leveraging motivations to enhance the RLVR training process, and additionally, highlights the robustness of McRF different prompt formulations.
21

---

Published as a conference paper at ICLR 2026
Search R1: MeRF RLVR
McRF (Curs)
0.40
-
RIVR (Baseline)
0.35
Accuracy
0.30
0.25
0.20
0
20
40
60
80
100
120
Training Step
On the agentic task of using search engine for reasoning. MeRF also outperforms RLVR baseline with effective utilization of in-context motivation, demonstrating the general- izability of MeRF to more complex reasoning tasks. Aceuracy evaluated on Natural Ques- tions (NQ)K wiatkowski et al., 2019) dataset.
Figure 14:
Ceameet
Gif
Gyour
@Rooro
Gif
Gyour
Gitnal
Ganswer
Gis
@score
Grinal
Ganswer
Gis
Abourect
Gacore
Gif
Gyour
Ganswer
Gis
G-
Šif
Gyour
Genswer
Gis
Gundersiandabie
Gbut
Gwrong
Gscoro
Gunderstandable
ōbut
Gwrang
Úscore
G
G
Gif
Gyour
Ganswer
Gyour
Gis
Gnot
Gpars
able
Gar
Gincomplete
Gnct
Opars
able
Gor
Gincomplete
@score
G
Blantmnt
(SSoato
Gscore
G
22omat
Gyou
Gfollow
Gine
Cihn
@toumat
Gif
Gyou
Giollow
Gite
Gtan
Stomat
Tenomiily
Gas
Gabove
Gonore
Rexanit
Gabove
Gecore
GOthervise
Reccocp
GOtherwise
Geoore
Attention on groundtruth motivation tokens
(b) Attention on adverse motivation tokens
Figure 15: This the attention heatmap in the final model of MeRF on K&K Logic Puzzle dataset with different motivations. Attention is calculated when generating <answer>. The model trained with MeRF pays significant attention the motivation tokens (e.g., "correct", "score", "format", "exactly") when generating the final answer, indicating that the model effectively leverages the motivation information. The comparison between attention weights of groundtruth motivation and adversarial motivation demonstrates that the model learn to distinguish useful information and ignore misleading information motivations.
22


Reviews:
[{"review_id": "mO4ZKblaK5", "reviewer": "Reviewer_9PZQ", "summary": "The paper introduces Motivation-enhanced Reinforcement Finetuning (MeRF), a method that injects a natural language description of the reward function into the prompt during RLVR training to make LLMs aware of the optimization objective. This leverages in-context learning to improve efficiency over standard RLVR. Contributions include empirical evaluations on logic puzzles and math benchmarks showing performance gains, and analyses on motivation-reward consistency.", "questions": "1. Could the authors provide a theoretical explanation or ablation on how the reward rules in the motivation prompt causally affect trajectory generation? For instance, why do negative scores for \"understandable but wrong\" answers guide the model better than a binary reward?\n\n2. Why are gains on math tasks (e.g., only 3-4% average improvement) much smaller than on puzzles? Please elaborate on why the method may not generalize to code or other domains, and suggest experiments to test this.\n\n\n3. The analysis section mentions Q1-Q4 but seems incomplete in the provided document. Can you expand on deeper insights, such as visualizing prompt-motivation interactions or reward hacking examples, to better convince readers of the method's value?", "limitations": "", "rating": 2, "confidence": 5, "soundness": 1, "presentation": 1, "contribution": 2, "strengths": "1. The approach creatively combines in-context learning with RL by explicitly providing reward rules as \"motivation,\" offering a simple extension to existing RLVR paradigms that could inspire hybrid training methods.\n\n2. Experiments cover multiple models (e.g., Qwen2.5 series) , with consistent comparisons to baselines, providing some evidence of improved accuracy and efficiency.\n\n3. The paper is well-structured, with clear illustrations of the method, prompts, and results, making the core idea accessible.", "weaknesses": "1. The method is overly simplistic and lacks rigorous theoretical justification; it's unclear how the specific reward scoring rules (e.g., +2 for correctness, -1.5 for understandable but wrong answers) mechanistically influence the model's generation of correct reasoning trajectories, relying too much on intuition without deeper analysis.\n\n2. Extensive experimental data is provided mainly for logic puzzles, but for more general tasks like mathematics and code generation, the motivation descriptions appear ineffective or irrelevant, as evidenced by smaller gains on MATH benchmarks and no code-specific results.\n\n3. Experiments rely heavily on simple numerical comparisons (e.g., accuracy curves), lacking in-depth qualitative analysis, such as case studies of trajectory changes or failure modes, which fails to convincingly support the paper's motivation and leaves readers questioning the underlying mechanisms.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHDnffJ4vO", "reviewer": "Reviewer_X9Aq", "summary": "This paper proposes a simple, low-cost intervention for RL with verifiable rewards (RLVR): append a natural-language description of the reward (\"motivation\") to each training prompt. The claim is that exposing the policy to the reward structure during rollouts improves exploration (higher entropy, stronger pass@k) and yields consistent gains on K&K logic puzzles, several math benchmarks (AIME/AMC/MATH), and CountDown, even when the motivation text is removed at evaluation. The paper is properly evaluated and ablated.", "questions": "Can you provide evidence that models actually use the motivation during generation (e.g., attention analysis, probing)?\nCatastrophic forgetting / over-alignment: After MeRF training, how does the model perform on unrelated general-purpose tasks? Any drop vs. base/RLVR?\nHave the authors analysed how complex the motivation prompt needs to be? How sensitive are results to motivation wording, length, or position? Any robustness sweep?\nHow much improvement comes from simply having better prompts vs. reward-specific information?\nHow does performance scale with reward function complexity?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "Simple, well-motivated idea that's easy to implement; the paper reads clearly.\nConsistent improvements over RLVR across two model families (Qwen2.5, DeepSeek-R1-Distill) and multiple reasoning benchmarks; importantly, performance holds without motivation at test time.\nThe method achieves better performance in fewer training steps. For example, in one experiment, MeRF achieved better pass@4 and pass@8 performance at step 140 than the final RLVR model did at step 280.", "weaknesses": "Currently the MeRF variant is compared only to the RLVR. Given the nature of MeRF consists of injecting the reward in the instruction, consider comparing against tuned-prompt variants via DSPY (https://github.com/stanfordnlp/dspy) , to see whether this benefit comes from better prompting. \nThe method's effectiveness is tied to tasks where the reward function is verifiable and describable in simple natural language. This limits the scope of MeRF, making it unclear how it would apply to tasks with more complex or non-describable reward signals, such as human preference scores.\nThe entropy analysis (Figure 4) shows higher entropy for MeRF, interpreted as \"better exploration,\" but higher entropy could also indicate increased uncertainty. Alternative explanations aren't ruled out.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0CWcKfjWHV", "reviewer": "Reviewer_dhpd", "summary": "This paper proposes MeRF, a clever approach to improve RLVR for LLMs by injecting a natural language description of the reward function directly into the training prompt, which is termed \"in-context motivation\". This enables the model to be aware of the optimization objectives during generation, aligning its outputs with desired behaviors more efficiently than the traditional trial-and-error RLVR paradigm, which relies solely on random explorations. Empirical results across benchmarks, including K&K Logic Puzzles, MATH datasets, and CountDown number games, demonstrate substantial gains, and better exploration as evidenced by higher entropy during training. Ablation studies show that performance benefits primarily from the training process rather than inference-time motivation, with MeRF robust to suboptimal or even adverse motivations.", "questions": "- Why does MeRF have high-entropy? and is it even a good thing? The paper motivates the need of MeRF like: we need it to have structured exploration instead of naive exploration of usual RLVRs. But entropy is more of a metric for \"unstructured exploration\"; usually I would expect smaller entropy when we move from unstructured exploration to structured exploration. Why is it not the case here? What happens if we control the temperature to increase entropy: does it help on RLVR?\n\n- It is interesting to see pass@8 saturates fast with RLVR so that it is soon outperformed by MeRF. However, to claim that such early saturation is due to better exploration, I think the authors should also show that the samples generated by RLVR rarely have high rewards (higher than what LLM is currently getting in expectation). Such early saturation might be caused by different training dynamics due to different prompts.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- A novel, simple and very practical approach to improve RLVR, which also makes sense\n- Interesting experimental design and results on Q4\n- Well presented (in terms of design) to make the paper easy to read", "weaknesses": "- The experimental results are scattered around the paper and somehow do not seem complete:\n  - Figure 1 includes results on 4 different LLMs and Figure 3 includes result on deepseek but most of them not presented in table 1.\n  - Results on Figure 2 (right) have no details; What dataset is this?\n  - Figure 1, 3, 7, 6, 5, 8 all show increasing performance on steps, but differently grouped (some on metrics, some on datasets), and feels very repetitive, being scattered all around the paper. Need to be better organized\n  - Figure 2 (right) and experiment Q3 (figure 8) basically telling the same thing but repeated\n  - A number of analysis experiments using different base models and datasets while there is only one single base model for main results, it makes me feel like the models are cherry picked for the analysis experiments.\n\n-  Main results need to be complemented with a number of different base models, to show that the method is robust between LLM choices\n- Lots of repetitive figures (in terms of message), system prompt in the main text, a handful of main results, repeated analyses, all these make the paper less dense in terms of how much information it conveys.\n\n- I would expect the improvement of MeRF to highly dependent on what reward function is used in the dataset. What happens if the reward function is much denser (having a lot of different criteria)? What happens if the reward function is less clear in terms of natural language (e.g. Math dataset)? There is no analyses on where the proposed approach would benefit best and where the proposed approach would benefit least.\n\n- system prompts other than K&K puzzle not provided\n\n- Some questionable analyses: continued in questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LhjEdnvUv4", "reviewer": "Reviewer_6qk7", "summary": "By providing evaluation criteria with the system prompt (find-grained guideline), RLVR pipeline could be effectively trained.", "questions": "Naturally, ML model could align their answers with the evaluation criteria when grading guidelines are provided rather than only being told whether an answer is correct or not. But does that constitute a discovery?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Through a series of experiments, the paper demonstrates that when training a model in a domain where the grading criteria are clearly defined, providing information about the grading scheme alongside the data can significantly accelerate the model’s learning process.", "weaknesses": "* As mentioned in the limitations, in situations where there is no prior knowledge of how the grading will be done, an approach that dynamically identifies the motivation and effectively solves the problem seems to have greater scalability.\n\n* Since adding even a suboptimal motivation still provides non-zero additional information, it is unsurprising that the performance improves compared to RLVR. An interesting phenomenon is shown in Fig. 9, where performance increases after 500 steps when an adversarial motivation is provided. It would be important to check whether this result consistently appears across multiple runs, as the paper does not seem to report repeated experiments for this particular setting. Furthermore, when adversarial motivation is given as a guideline, it may be worth analyzing whether the RLVR training process includes any mechanisms that allow the model to ignore or correct such misleading guidance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)