From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation
Authors: Yuxin Jiang
Yufei Wang
Qiyuan Zhang
Xingshan Zeng
Liangyou Li
Jierun Chen
Chaofan Tao
Haoli Bai
Lifeng Shang
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
verifiable reference-based rewards
open-ended generation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Reinforcement learning with verifiable rewards (RLVR) succeeds in reasoning tasks (e.g., math and code) by checking the final verifiable answer (i.e., a verifiable dot signal). However, extending this paradigm to open-ended generation is challenging because there is no unambiguous ground truth. Relying on single-dot supervision often leads to inefficiency and reward hacking. To address these issues, we propose reinforcement learning with verifiable reference-based rewards (RLVRR). Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e, reward chain). Specifically, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts (e.g., keywords), and style, which evaluates adherence to stylistic properties through LLM-based verification. In this way, RLVRR combines the exploratory strength of RL with the efficiency and reliability of supervised fine-tuning (SFT). Extensive experiments on more than 10 benchmarks with Qwen and Llama models confirm the advantages of our approach. RLVRR (1) substantially outperforms SFT trained with ten times more data and advanced reward models, (2) unifies the training of structured reasoning and open-ended generation, and (3) generalizes more effectively while preserving output diversity. These results establish RLVRR as a principled and efficient path toward verifiable reinforcement learning for general-purpose LLM alignment.

Published as a conference paper at ICLR 2026
FROM VERIFIABLE DOT TO REWARD CHAIN: HAR- NESSING VERIFIABLE REFERENCE-BASED REWARDS FOR REINFORCEMENT LEARNING OF OPEN-ENDED GENERATION
Yuxin Jiang', Yufei Wang', Qiyuan Zhang", Xingshan Zeng', Liangyou Li', Jierun Chen'. Chaofan Tao', Haoli Bai', Lifeng Shang HHaawii Technologies Co.,Ltd, CCiyy University Hong Kong (jiang. yuxin22 yufeil
1 com, (aleammg22-mmmmmmmmmmmm cityu. edw.tyyueeuuu...
ABSTRACT
Reinforcement learning with verifiable rewards (RLVR) succeeds in reasoning tasks (e.g., math and code) by checking final verifiable answer (i.e., a ver- ifiable dof signal). However, extending this paradigm to open-ended genera- tion is challenging because there is no unambiguous ground truth. Relying on single-dot supervision often leads to inefficiency and reward hacking. To address these issues, we propose reinforcement learning with verifiable reference-bane rewards (RLVRR). Instead of checking the final answer, RLVRR extracts dered linguistic signal from high-quality references (i.e, reward chain). Specifi- cally, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts (e.g., keywords), and style, which evaluates adher- ence to stylistic properties through LLM-based verification. In this way, RLVRR combines the exploratory strength of RL with the efficiency and reliability of su- pervised fine-tuning (SFT). Extensive experiments on more than 10 benchmarks with Qwen and Llama models confirm the advantages of our approach. RLVRR (I) substantially outperforms SFT trained with ten times more data and advanced reward models, (2) unifies the training of structured reasoning and open-ended generation, and (3) generalizes more effectively while preserving output diversity. These results establish RLVRR a principled and efficient path toward verifiable reinforcement learning for general-purpon LLM alignment. We release our code and data at https: //gi thub. com/YJJ angem/ RLVRR.
INTRODUCTION
Reinforcement learning with verifiable rewards (RLVR) (Shao et al., 2024; Yu et al., 2025a: Team, 2025) has emerged promising paradigm for enhancing large language models (LLMs) in rea- soning tasks such as mathematics and code generation. At its core, RLVR sidesteps the complicated Chain-of-Thoughh (CoT) supervision and only checks the correctness of the final reasoning result (i.e., the verifiable dor) within the reasoning solution. The presence of unambiguous ground truth makes such verifiable dot reliable signal, guiding exploration toward correct CoTs while pre- venting drift into spurious reasoning paths.
While RLVR simple yet effective for reasoning tasks (e.g., math and code generation), fails in open-ended generation tasks, where no unambiguous ground truth exists and reliable verification cannot reduced to a single dot. In many cases, high-quality responses in open-ended generation should satisfy list of content requirements simultaneously: for instance, a safe-response policy answer should explain the risk, refuse the harmful request, cite the relevant rule, and offer safer alternative. In practice, researchers often resort to reinforcement learning from human feedback (RLHF) (Christiamo et al., 2017; Bai et al., 2022; Ouyang et al., 2022) using preference-bastee models (Liu et al., 2024; 2025a) generative reward models (Jia et al., 2025; Gunjal et al., 2025). Despite their widespread adoption, reward models suffer from two major limitations: (1) they are prone to reward hacking, often overfitting superficial artifacts and spurious correlations (Chen al.,
reward

---

Published as a conference paper at ICLR 2026
2024); (2) they require large-scale pairwise annotations, making training costly and brittle during RL optimization. This motivates a critical research question: how can we extend RL optimization to open-ended generation by moving beyond single-dot supervision?
To this end, we introduce RLVRR (reinforcement learning with verifiable reference-basee rewards), framework that extends RLVR to open-ended generation. Instead of relying single veri- fiable dot, RLVRR extracts an ordered sequence of verifiable linguistic signals from high-quality references, transforming the dot supervision into reward chain, akin how mathematical reason- ing derives rules from ground truth. A reference is high-quality exemplar for the same prompt, which can be drawn from synthetic WebR) (Teknium, 2023; Xu et al., 2025; Jiang et al., 2025) at scale and low cost. Mechanistically, RLVRR mirrors the single-dot principle: the reward chain anchors exploration to a standardized, ver- ifiable checklist derived from the reference. To make supervision both reliable and efficient, RLVRR decomposes rewards into two complementary dimensions: content and style. The content reward uses reference-derived key points (e.g., key entities or keywords) score rollout by whether those deterministic core concepts are present, which remains flexibility in phrasing and expression; The style reward runs small set of LLM ----eenerttdd, verifiable Python checks the rollout to firm adherence to sreerence-specfffe stylistic properties (e.g., length, format). By integrating these complementary signals, RLVRR retains RL's exploratory dynamics but injects SFT-like token-level guidance, yielding lightweight reward, stable learning, and better generalization.
cntrruttionfollowwiit
corpora (e.g.. OpenHermes, Magpie,
Comprehensive experiments across over 10 benchmarks show that: (1) RLVRR substantially out- performs SFT with 10x more data, advanced reward models, and confidence-base rewards; (2) RLVRR can be effectively integrated into RLVR, unifying the training of structured reasoning and open-ended generation; (3) RLVRR eliminates loading reward models during RL training, incurring merely 0.71 % computational overhead compared to random rewards. Moreover, our in-depth anal- yses reveal why RLVRR generalizes more effectively and confirm that it preserves output diversity despite relying on rule-based verifiers, underscoring its practical potential.
2 RELATED WORK
Reinforcement learning with verifiable rewards. RLVR has demonstrated strong capabilities on reasoning tasks such as math and code (Shao et al., 2024; Yu et al., 2025a; Team, 2025). By leverag- ing deterministic verifiers like Math-Verify (Kydlfček, 2024) and SandboxFusion (Bytedance-Seed. Foundation- Coriionn--oo---Teee Team et al., 2025). RLVR enables direct correctness evaluation. Building on this paradigm, recent work has extended RLVR to broader reasoning domains. For instance, (Su et al., 2025; Ma et al., 2025) train specialized LLMs as verifier models to assess whether generated re- sponses equivalent reference answers. VeriFree (Zhou et al., 2025) and RLPR (Yu et al., 2025b) bypass answer verification by leveraging policy likelihood for reference answer as a reward signal. However, these methods merely conduct experiments on datasets comprising short-form answers (nearly 10 words), overlooking the challenges of open-ended generation.
Reinforcement learning for open-ended generation. A pivotal advancement in applying rein- forcement learning (RL) open-ended generation is RLHF, which leverages human preference data to train reward model that guides policy optimization. While effective, RLHF introduces several drawbacks including high training costs and susceptibility to reward hacking (Gao et al., 2023). These challenges have spurred the development of offline methods such as Direct Preference Optimization (DPO) (Rafailov et al., 2023), which optimizes policies directly from preference data without an explicit reward model. More recently, (Chang et al., 2025) directly uses BLEU (Pap- ineni et al., 2002) between the reference and the rollout as a reward signal for open-ended tasks. Despite its simplicity, n-gram precision metrics such as BLEU fail to capture key content aligned with human preferences, resulting in misaligned and noisy reward signals during training.
3 METHODOLOGY
We propose reinforcement learning with verifiable reference-batee based rewards (RLVRR), framework designed to provide reliable, low-cost rewards for open-ended generation by leveraging Reward Chain extracted from reference responses. RLVRR decomposes the reward signal into content and
2

---

Published as a conference paper at ICLR 2026
Verifiable Keywords for Content
Question: X
Verifiable Code
idently key points onswering the question
Create list The mod and and randl the givan technology (ootshaa
Gonjormity shecking
Reference: z
Point 2
Weight
Code
esplain advantages of blockchain technology
explain
244 Pros of Blockchatn Technology
diaddaattagss blockcham technology.
Decemtaliatioon Boockhain
0.25
eneelatteeemcceee si! book 'ill'
Sperates on deoenttalled
+(x,2) 2)
Transpornccy Transactions on
network
cralustioaaann nswer S0G Mewrchi//MM---70000 answerl
Morkchain are recorded publicly
Keywords "Decmmtalization
Keywords
limutability Once data
"Sealability
brockhhinn
"Transparenc""
"Immutabliit"
"Enengy Consumption"
"Reguaatory Uncertaimty
(nhannedd Security. Blockchath
atte red;;
0.20
esahistrosa end boot return 150 lengemmeresppiiif
"Erisanced Securityy "Efic Speed"
"Complecity"
ises crypoo propooompphiccc techniques
Concems
(fficiency and Spoed Biockchain Can steemnine
Sel Cons Blockchain Technology
Content Reward
Style Reward
Matched Keywords Reference: K2
Question:
Keywords K
Policy:
Code
Codee
Gode
k.))
Rollout:
rewsrd
max (en (K,),
reward Codefivaly) (0.1)
Welphted
Tc
Figure 1: Overview of our proposed RLVRR framework. (1) Upper (data construction): given
Question x and Reference z, we use an off-the-shelf LLM to generate verifiable components in terms of content and style for open-ended generation. (2) Lower (RL training): these verifiable components are leveraged to calculate the rule-based reward of the Rollout
style dimensions, each computed through rule-based verification rather than subjective model-based scoring, as illustrated in Figure 1. We first introduce the problem formalization of RLVRR in $3.1, followed by illustrating the details of content and style rewards $3.2 and $3.3.
3.1 PROBLEM FORMULATION
Let x denote an open-ended instruction sampled from the training data D. Our goal is train policy Te that generates response y maximizing the RL objective:
(0) EE,D,, y-m (- [r) y)] 3DKL [No [x) Tye x)],
(1)
where TO is the reward function, DxL the KL divergence, and Tref denotes the reference model. Unlike conventional RLHF methods that rely on learned reward models instantiate ræ. RLVRR derives rewards directly from verifiable linguistic signals based on a reference answer 2:
(s y) = F (re(x, y, r,, y,2)),
(2)
where ra and rs quantify content fidelity and stylistic conformity, respectively, and denotes the aggregation function (simple averaging in our experiments). Since both re and тa computed via reference-greee reffreenc-grounded rule-based reward, RLVRR greatly mitigates the reward hacking and the inefficiency of reward models, enabling robust and scalable RL training.
3.2 CONTENT REWARD OF RLVRR
Verifiable keywords for content. Numerous studies have shown that active learning, engaging with core concepts and rephrasing information, more effective than passive memorization (Miller, 1956; Newport, 1990). Building on this insight, we propose novel approach to verifiable reward design: our method extracts critical keywords (or phrases) from reference responses and optimizes the policy to maximize their inclusion during reinforcement learning, Rather than directly selecting keywords that loosely capture the semantics of the reference, we propose a novel two-level hier- archical extraction method: (1) an L.MM first identifies set of essential key points {p Am.MM m= that
3

---

Published as a conference paper at ICLR 2026
the Al assistant must address when answering the question (See prompt in Figure 4); (2) for each key point p", the LLM extracts a set of keywords Tmm (each fewer than three words) from the ref- erence answer that encode the core facts, concepts, or entities required to assess the correctness and relevance of the response (See prompt in Figure 5). This strategy enables broader and more sys- tematic keyword coverage while decomposing the content reward into fine-grained, verifiable units. As shown in Table 3, this separation significantly improves performance by 0.9 points. On average, the extracted keywords constitute approximately 15% of the reference response, striking a balance between coverage and conciseness.
Content reward calculation. To assess content fidelity during RI. training, we propose a reward function based on keyword alignment between generated rollout y and reference text(s) z. For each key point pr (where m E MM). we extract the matched keyword sequences K from y and K from using regular expression matching. Crucially, KT and KT preserve both the frequency and sequential order of matched keywords, ensuring fine- aane-graindd alignment evaluation. For each key point pm" compute the semantic coherence between and using the longest common subsequence (LCS) metric (Wagner& & Fischer, 1974). LCS is chosen because it inherently captures keyword ordering and repetition, making it well-suited for evaluating the structural and semantic fidelity of generated text. The alignment score for pd is given by the normalized LCS length, while the overall content reward (I y/ 2) is defined as the mean alignment score across all key points:
len (LCS (KT, KT
re(0, v.
Σ
(3)
M
max (len (KT), len (Kr))
To improve robustness and accommodate multiple references { { extend Eq. (3) by selecting the highest alignment score per key point across all references. This ensures tolerance to variations in reference phrasing while maintaining rigorous content fidelity assessment:
1 M
M max
len (LCS (K KT)))
ro(o,, v. (2)
(4)
max (Ien
(K m len (Km)
In RLVRR, we set / = and show in Table that multiple references consistently improve policy performance, suggesting diversified references enhance robustness.
3.3 STYLE REWARD OF RLVRR
Verifiable code for style. Unlike reasoning tasks, stylistic quality significantly influences model performance in open-ended generation tasks. To quantify stylistic alignment, we employ an LLM to generate a set of verifiable Python functions (-odeEval,,.))1 rollout y adheres to stylistic properties of a reference 2. These properties include answer length, markdown formatting, and other measurable features (See prompt in Figure 6). Additionally, the relative importance-an approach validated empirically in our ablation study (Table 3). While our current implementation focuses on verifiable stylistic elements, semantic aspects such as tone are implicitly captured through the content reward.
each assessing whether the
LLM assigns weight wn, to each CodeEvvll, (-), reflecting
During reinforcement learning, we compute the style reward r, 2) by evaluating y against each CodeEvvll, and aggregating the results weighted sum:
Style reward calculation.
Ts Wn Code
(5)
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Models and training data. We conduct experiments using the Qwen2.5 (Qwen Team, 2024) and Llama3.1 (Dubey et al., 2024) model series to ensure fair comparisons with prior work and en- able comprehensive evaluation. For training data, adopt the dataset released by (Jiang et al.,
4

---

Published as a conference paper at ICLR 2026
2025), comprising 100K open-ended Instruction-responn pairs curated from diverse high-qualig instruction-tuning datasets. All responses are regenerated by GPT-44-mini to maintain consistency in response quality. During the data construction of RLVRR, also leverage GPT-44-mini as the off-the-shelf LLM to generate verifiable components. Besides, we cross-validate the quality of the verifiable components using the reference, filtering out cases where both content and style wards of the reference fall below 0.7. Finally, we randomly sample 1OK data for RL training, where GRPO (Shao al., 2024) is applied as the optimization algorithm ensure that all other settings are consistent with our approach.
Evaluation benchmarks. We assess our models using five of the most popular open-ended instruction-folloving benchmarks: AlpacaEval 2 (Li et al., 2023), Arena-Hard (Li et al., 2024), MT-Bench (Zheng et al., 2023), IFEval (Zhou et al., 2023), and FollowBench (Jiang et al., 2024). For AlpacaEval 2, we report the length-controlled win rate (LC), which ensures robustness against verbosity, For Arena-Hard, we report the win rate (WR) against the baseline model. For MT-Bench, provide the average score, using GPT-4.1-min as the evaluation judge. For IFEval and Fol- lowBench, we report the prompt-level strict accuracy and the hard satisfaction rate, respectively. Besides, we evaluate the impact of diverse methods on tasks across multiple domains: (1) Knowl- edge: MMLU (Hendrycks et al., 2021a); (2) Reasoning: ARC (Clark et al., 2018); (3) Math: MATH (Hendrycks et al., 2021b); (4) Code: HumanEval (Chen et al., 2021). More evaluation details are listed in Appendix B.
Baselines. We compare RLVRR with seven established and contemporaneous methods. catego- rized into SFT, reward strategies, and DPO. (1) SFT: Standard supervised fine-tuning (Wei et al., 2022; Mishra et al., 2022) on (i) 10K data which shares identical prompts with RI., or (ii) 100K data. (2) Random: We examine whether random rewards ~ Uniform(0, (0, 1) can benefit open-ended generation. (3) BLEU: (Chang et al., 2025) directly uses BLEU (Papineni et al., 2002) between the reference and the rollout as reward signal for RL-based alignment. (4) RM: We use Skywork- Neward-V2-Llamme 3-B (Liu et al., 2025a) trained on well-curated preference data as the reward model to score output in GRPO. (5) GRM: Following Rubries as Rewards (Gunjal et al., 2025), we use GPT-4o-mini as the generative reward model to judge whether the rollout satisfies checklist- style rubrics. (6) RLPR (Yu al., 2025b): RLPR is a verifier-free framework that uses the LLM's own token probability scores of reference answers as the reward signal. (7) DPO (Rafailov et al., 2023): We generate the preference dataset following (Meng et al., 2024). For each question æ, we first generate 5 responses using the INSTRUCT model and then use GPT-4o-mini to select the best one as win and the worst one as lose. All implementation details are illustrated in Appendix A.
4.2
MAIN RESULTS
Table summarizes the performance of various methods across five open-ended benchmarks and four additional tasks, revealing several key findings. (I) Superiority over SFT: Remarkably, RLVRR outperforms SFT by a significant margin on open-ended tasks, even when SFT is trained with 10x more data. (2) Advantages over alternative reward strategies: RLVRR consistently surpasses other reward strategies, including random reward, BLEU, reward model (RM), genera- tive reward model (GRM), and RLPR. Notably, it improves over the RM-based approach-which requires loading an auxiliary reward model during training- +2.3 and +2.7 points on Qwen2.5- 3B- BB-Base and Instruct, respectively. (3) Improved over DPO: RLVRR exhibits stronger performance than DPO. a widely adopted alignment method, further validating effectiveness. (4) Robustness across scales and initializations: The benefits of RL.RR persist across varying model sizes and training starting points, demonstrating its general applicability. (5) Generalization to diverse tasks: Beyond open-ended generation, RLVRR achieves state-of-the-art results on knowledge-intensinn reasoning, mathematical, and coding tasks, highlighting its superior generalization capability.
4.3 INTEGRATION WITH MATHEMATICAL REASONING
To examine the compatibility of our method in jointly optimizing for both closed-form reasoning and open-ended generation within RLVR, we focus on the mathematical domain as a representative setting. The reasoning template is shown in Appendix A.3. Following SimpleRL-Zoo (Zeng al.,
'This model ranks first on RewardBench (Lambert et al., 2025) of September 24th, 2025.
5

---

Published as a conference paper at ICLR 2026
Table 1: Evaluation results across five open-ended benchmarks and four other tasks. The results of Llama3.1, which indicate consistent findings, are shown in Appendix C.
Method
#Dat
Alpaca Eval
Aremn Hard
MT Bench
Fval
Fallow Banch
Avg.
MILLU
ARC
MATH
Human Foal
Avg.
Qwen2.5-3B Models
6.5
6.4
12.4
66.8
65.7
SET
10K
22.0
27.3
1.5
31.8
45.0
26.7
66.1
83.7
-9.6
05.9
68.8
SET
100K
25.1
32.9
7.5
35.9
51.3
30.5
60.4
81.4
58.7
65.9
66.6
GRPO (Random)
3.6
GRPO (BLEU)
1OK
14.4
26.6
6.9
29.2
41.8
23.8
67.2
82.0
59,9
62.8
68.0
GRPO (RM)
10K
22.4
33.7
7.3
32.8
47.6
26.8
67.1
84.2
59.6
65.1
69.0
GRPO (GRM)
30.5
7.4
43.3
63.9
67.2
GXPO (KLVRR)
10K
23.7
35.3
7.6
37.7
51.2
31.1
67.9
85.7
60.6
66.3
70.0
Instract
17.0
19.3
7.8
54.9
47.5
29.3
67.3
84.8
63.2
71.3
71.6
31.1
GRPO (RM)
22.3
34.1
7.6
55.3
19.3
33.7
67.5
63.2
70.7
71.7
+ GRPO (RLVRR)
10K
24.3
36.5
7.9
61.3
$1.9
36.4
67.88
85.4
63.6
71.9
72.2
Qwen2.5-77 Models
Base
2.1
8.9
7.3
24.7
14.9
11.6
74.2
69.8
76.0
74,9
+ SFT
10K
30.0
53.2
8,3
42.3
47.5
36.3
75.2
89.8
67.5
77.4
77.5
SET
32.3
52.0
8.3
56.3
38.5
76.7
75.7
(Random)
7.8
7.4
15.0
GIPO (BLEU)
10K
19.9
44.1
7.8
39.3
46.8
31.6
74.6
83.5
08.0
77.1
75.8
GRPO (RM)
JOK
32.8
53.5
8.2
433.2
49.0
37.3
74.8
88.3
68.8
76.5
77.1
GRPO (GRRM)
8.1
51.3
25.7
GRPO (REVRR)
10K
33.6
$4.9
8.3
47.8
54.6
39.8
75.7
89.6
70.1
77.5
78.2
35.6
37.1
8.7
53.8
82.4
GRPO (RM)
JOK
37.6
53.6
8.4
69.1
33.9
44.5
75.1
89.5
80.2
82.8
$1.9
+
GRPO (RLVRR)
10K
41.4
55.8
8.8
70.3
55.7
46.4
75.6
90.3
80.6
84.1
82.6
Table 2: Performance comparison of math tasks based on Qwen2.5-3B-BB.
MATH
Mincrva Matli
GanKan
Olympiad
Open-ended
Avg.
Avg.
Base
61.8
61.2
21.0
46.1
25,0
38.5
46.1
9.6
GRRD
(RLVRR)
GRPO open ended (RLVRR)
64.0
25.4
56.3
26.8
43.0
49.8
31.1
GRPO 5k mush (RLVR) opea ended (RLVER)
86.11
67.8
29.4
57.3
25.0
42.7
51.9
3017
GRPO 5k moth (RLVR) 3k open ended (RM)
$4.6
67.0
24.7
36.5
27.3
42.3
50.4
28.2
2025), we stratify the MATH dataset (Lewkowycz et al., 2022) into five difficulty levels and ran- domly sample 10K examples from levels 2-5 as the base for math-foeused RL training. To explore integration, we construct a mixed training set by combining 5k math-focused samples (using rule- based reward) with 5k open-ended instances (using RLVRR- reward). We evaluate the result- ing models on six standard mathematical reasoning benchmarks, including GSMSK (Cobbe et al., 2021), MATH 500 (Hendrycks al., 2021b), Minerva Math (Lewkowycz et al., 2022), GaoKao 2023 En (Liao et al., 2024), Olympiad Bench (He et al., 2024), and College Math (Tang et al., 2024). We report the performance of CoT reasoning with greedy decoding.
As shown in Table 2, RLVR trained solely on mathematical data significantly boosts performance on math benchmarks generalizes poorly to open-ended tasks (Avg. 22.6). In contrast, RLVRR trained only on open-ended data achieves strong performance in open-ended tasks and also im- proves mathematical reasoning (Avg. 49.8), indicating positive transfer. Unified training on mixed data provides the best balance, reaching 51.9 on math benchmarks and 30.7 open-ended tasks. Remarkably, this setting even surpasses the INSTRUCT model trained on millions samples, despite using only 10K RL training instances. Furthermore, RLVRR demonstrates better compatibility with reasoning tasks compared to RM. These results demonstrate that our method seamlessly integrates with RLVR, R, unifying the training of structured reasoning and open-ended generation.

---

Published as a conference paper at ICLR 2026
Table Ablation study based on Qwen2.5-3B-Bas
Method
2
Arena-Hard
MT-Bench
IF-Eval
FollowBench
Avg.
GRPO (RLVRR)
23.7
35,3
7.6
37.7
51.2
31.1
Effect of content reward and style reward
-woo content reward
9.6
10.2
6.7
28.6
35.5
18.1
-woo multiple references
23.6
35.1
7.6
36.
50.9
30.7
-repl. LCS direct matching
10.3
29.0
34.7
-wlo style reward
19.6
28.5
6.6
36.6
50.1
28.3
-woo weight in style
22.2
7.6
36.2
48.6
29.9
Effect of keyword extraction
-Woo two-level extraction
22.6
35.3
34,3
51.3
30.2
extract keywords randomly
19.0
43.7
15% keywords TF-IDF
19.6
31.3
7,4
33.3
45.5
27.4
extract 30% keywords by TE-DE
19.4
30.5
7.3
32.7
45.9
27.2
5 ANALYSIS
5.1
ABLATION STUDY
To systematically evaluate method components and offer a comprehensive understanding of RLVRR, conduct ablation studies based Qwen2.5-3B-Bas in Table 3.
Effect of content reward. Our ablation study reveals that removing the content reward results in severe performance degradation, with the average score dropping by 13.0 points compared to the full method. This underscores the critical role of content alignment in response generation. Interestingly, when using only a single reference (instead of multiple references) for content reward computation, performance remains robust, declining marginally from 31.1 to 30.7. demonstrating the method's resilience to reference variability. Finally, we attempt to replace LCS with naïve "direct matching" approach, which calculates the percentage of keywords appearing in the rollout as the content reward. This approach leads to catastrophic failure (1) disregards keyword ordering and (2) incentivizes reward hacking (Skalse al., 2022), where the model generates excessively verbose outputs to artificially inflate keyword coverage.
Effeet of style reward. The absence of style reward reduces performance by 2.8 points, confirm- ing that learning presentation, structure, and formatting from references is essential for high-quality responses. Moreover, when style reward components aggregated without LLM- LLM- generated impor- tance weights, performance drops by 1.2 points, validating that LLM--erived weighting effectively captures stylistic nuances.
Effect of keywords extraction. We analyze the impact of keyword extraction strategies align- ment performance. First, we ablate the two-level hierarchical extraction process in favor of single- step approach where keywords directly extracted from the full response. This leads 0.9-point drop in average score, confirming that hierarchical extraction improves keyword precision and cov- erage. Next, we compare LLM-bbaeed extraction with rule-based alternatives: (I) random selection after stopword filtering and (2) TF-IDF-based selection (Sparck Jones, 1972), both extracting 15% of words for fairness. As shown in Table LLM extraction outperforms both variants by 3.7-4.1 points, demonstrating its superiority in identifying semantically critical keywords. Notably, increas- ing the TF-IDF keyword ratio to 30% further degrades performance, suggesting that quality matters more than quantity a sparse set of high-value keywords suffices for effective learning.
Effect of reference LLMs. Table 4 examines the impact of diverse reference LLMs, where the LLM is used for (1) reference generation and (2) verifiable component generation of RLVRR. Re- markably, substituting the GPT-4o-mini model with less powerful yet open-source alternative, such as Llama3-77 70BB- P0B-Instruce (Dubey et al., 2024), yields consistent results RLVRR continues to out- perform SFT when SFT is trained with 10x more data. This demonstrates the robustness

---

Published as a conference paper at ICLR 2026
Table Impact of various reference LLMs based on Qwen2.5-3B-Ban
Table 5: Results of SFT via self-data distilla- tion based on Qwen2.5-3B-Ban Base..
Method
LLM Ref
#Data
Open-ended
Open-ended Avg.
SFT
GPT-4o
10K
26.7
Base
9.6
30.5
GRPO (RIVRR)
GPT-4o mini
1OK
31.1
SFT
100K
30.5
SFT
Liam3- 70B- Inst
1OK
21.8
SFF SFT
10K
25.0
SET
Lama33 70B- Inst
100K
28.3
RLVRR-distilled
29.2
GRPO (RLVRR)
Llama3 70B Inst
1OK
28.9
BLEU
0.6
54.26
SFT
RLVAR
30-
0.4
20
24.03
19:77
20.08
Revera
0.3
0.2
0.1
Random
0.0
5-
ILEU
Train
Dev
20
40
Stop
60
80
100
Cosine Similarity
750
0.85
0.78
700
0.6
550
0.2
400
20
69
80
100
Train
Dev
Figure 2: Average BLEU scores and semantic similarities between references and generated responses for SFT and RLVRR.
Figure 3: Curves of reward and response length during RL training with different methods.
of our approach across varying levels of LLM sophistication and highlights its potential to reduce reliance on proprietary commercial models without compromising downstream performance.
5.2 LEARNING WHAT MATTERS: WHY RLVRR GENERALIZES BETTER THAN SFT
In this section, we investigate why RLVRR, which reinforces quality signals (keywords or phrases), outperforms SFT, which models the entire reference sequence liken-by-token. To compare their generalization behaviors, we conduct a controlled study on 1,000 randomly sampled prompts, each from the training and development sets. For each prompt, we generate responses using two models trained separately with SFT and RLVRR, and evaluate their quality with BLEU and cosine se- mantic similarity against references. Semantic similarity computed using embeddings from all- mpnet-base-v2 (Reimers & Gurevych, 2019). Figure 2 show that while SFT achieves higher BLEU on the training set, this advantage vanishes and even reverses on the development set, indicating strong memorization but poor generalization (Chu al., 2025). The limitation stems from SFT's imitation learning objective, which minimizes token-level prediction enror under teacher forcing:
LSFT = - t- log To Jx, zcr). This training paradigm enforces exact mimiery but suffers from exposure bias (Zhang et al., 2019; Schmidt, 2019), as the model never recovers from its own mis- takes. RLVRR, in contrast, rewards the preservation of key semantic elements while allowing flexi- ble phrasing, leading to stable performance across both training and development sets. Specifically, RLVRR maintains consistent BLEU and higher semantic similarity (0.84 0.85 training; 0.78 vs. 0.76 on development, compared to SFT). These results suggest that RLVRR better captures the semantic essence of references and generalizes more effectively unseen inputs.
8

---

Published as a conference paper at ICLR 2026
Table Average runtime per step for different reward strategies in RL training, based on Qwen2.5-3B-Ban
Table Average best@5 performance and Self- BLEU cross five open-ended benchmarks, based on Qwen2.5-3B-Ban
Method
Time (s)
Random (%)
Method
Open-ended Avg.
Self-BLEU (U)
Best@ (1)
Random
121.56
0.00%
Base
11.1
27.1
BLEU
122.38
0.67%
SFT
29.8
24.2
RM
131.62
8.28%
GRPO (BLEU)
25.2
26.6
GRM
128.92
6.05%
GRPO
RLPR
129.38
6,43%
GRPO (RLPR)
29,6
24.5
GRPO (RLVRR)
33.2
24.0
RLVRR 122.42
0.71%
Instract
31.0
23.7
5.3 SELF-DATA DISTILLED RLVRR OUTPERFORMS STANDARD SET
Recent work such DeepSeek-RI (eeepSeek-AI, 2025) demonstrates that fine-tuning on trajecto- ries sampled from the same model post-RL training, an approach we refer to as self-data distillation. can yield better performance than standard SFT on reasoning tasks. In this section, extend this idea to open open-ended generation and examine whether a similar benefit holds. As shown in Table 5, self-data distillation using RLVRR significantly improves performance over standard SFT (2.5-point gain in average performance) when both are trained on the same 10K dataset. While it does not match the performance of SFT trained on the full 100K data, it notably narrows the gap using only 10% of the data. These results highlight the superior quality of supervision signals produced by RLVRR. Moreover, since the distilled data remains close in distribution to the base model's outputs, the resulting student model benefits from both strong alignment and distributional consistency.
5.4 TRAINING DYNAMICS ANALYSIS COST ANALYSIS
Training dynamics analysis. Figure 3 visualizes the curves of reward and response length during RL training with different methods (Random, BLEU, RM, and RLVRR). We observe that RLVRR achieves a more stable and substantial increase in reward compared to the other methods, highlight- ing its effectiveness in providing consistent and high-quality learning signals. This trend further validated by the content/style reward curves in Figure 8. Notably, RLVRR's response length surges initially, reflecting exploratory behavior for informative outputs, then declines and stabilizes as the model learns conciseness. This demonstrates RLVRR's robustness against reward hacking, as it avoids exploiting length for reward gains. In contrast, RM persistently favors longer responses, likely due over-reliance superficial heuristies rather than true quality.
Cost analysis. We present a detailed cost breakdown of RIVRR in Appendix D, covering both the data construction and RI. training phases. Key findings include: (1) the total cost of API calls during data construction is $21.36, which is highly economical given the scale of the task; (2) in the RL training phase, RLVRR introduces only a 0.71% % computational overhead compared to the Random Reward baseline (refer to Table 6). These results underscore RLVRR's practicality for real-world deployment, with minimal financial and computational burdens.
5.5 RLVRR DOES NOT COMPROMISE DIVERSITY
potential concern with RLVRR's reference- based verifiable reward is that could restrict output diversity. To examine this, we set the decoding temperature to 1.0 and sampled five responses per method across five open-ended benchmarks, reporting average best @ 5 and Self-BLEU in Table 7. The relative performance improvements of RLVRR over baselines remain consistent with Table 1. Notably, RLVRR attains a SeLf-BLEU of 24.0, comparable to RM (23.9) and the INSTRUCT model (23.7). These findings indicate that RLVRR does not sacrifice diversity despite its reliance verifiable references, and in fact, enhances the model's 's ability to generate diverse responses relative to other reward strategies.

---

Published as a conference paper at ICLR 2026
6 CONCLUSION
In this paper, propose RLVRR, novel framework that extends verifiable reward learning beyond reasoning tasks to open-ended generation. By constructing rule-based verifiers derived from high- quality references across content and style dimensions, RLVRR retains RL's exploratory dynamics but injects SFT-like token-level guidance, thus providing reliable and low-cost training signals. Our results establish RLVRR as an efficient and scalable path toward verifiable reinforcement learning for general purpose LLMs.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the transparency and reproducibility of our research. To support this commitment, we will publicly release our annotated dataset and all source code, facilitating future extensions and community research. Comprehensive details of our methodology are provided throughout this paper: the prompts used for data construction are illustrated in Appendix A.2; the evaluation details are shown in Appendix B. Furthermore, the experimental implementations can be found in Appendix A. We believe that releasing these assets will lower the barrier for replication, enable fair comparisons, and foster further exploration in this line of research.
REFERENCES
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training helpful and harmless assistant with reinforcement learning from human feedback. arXiv, 2022. URL https: / htlaaxii..rr/abb////2//a 05862.
:, Yao Cheng, Jianfeng Chen, Jie Chen, Li Chen, Liyu Chen, Wentao Chen, Zhengyu Chen, Shijic Geng, Aoyan Li. Bo Li, Bowen Li. Linyi Li, Boyi Liu, Jiaheng Liu, Kaibo Liu, Liu, Shukai Liu, Siyao Liu, Tianyi Liu, Tingkai Liu, Yongfei Liu, Rui Long, Jing Mai, Guanghan Ning. Z. Y. Peng, Kai Shen, Jiahao Su, Jing Su, Tao Sun, Yifan Sun, Yunzhe Tao, Guoyin Wang, Siwei Wang. Xuwu Wang. Yite Wang. Zihan Wang, Jinxiang Xia, Liang Xiang, Xia Xiao, Yongsheng Xiao, Chenguang Xi, Shulin Xin, Jingjing Xu, Shikun Xu, Hongxia Yang, Jack Yang, Yingxiang Yang, Jianbo Yuan, Jun Zhang, Yufeng Zhang, Yuyu Zhang, Shen Zheng, He Zhu, and Ming Zhu. Fullstack bench: Evaluating llms as full stack coders, 2025. URL https: : / /aaaaaivvv...o org. has//bb/22450 221.2225525...
Yapei Chang, Yekyung Kim, Michael Krumdick, Amir Zadeh, Chuan Li, Chris Tanner, and Mohit lyyer. Bleuberi: Bleu is a surprisingly effective reward for instruction following, 2025. URL hittps://arxia oroooog///aooo abs /200 11080.
Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: disentangled reward mitigates hacking in RLHF In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL httpps://ppenrevier.. Forum?ld=zcIv8000
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards. Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Mathhis Plappert, Fo- tios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss. William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob Mc- Grew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained code. 2021.
10

---

Published as a conference paper at ICLR 2026
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, pp. 4299-4307, 2017.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. SFT memorizes, RL generalizes: A comparative study of founda- tion model post-training. Forty-second International Conference on Machine Learning. 2025. URL httph:tt////s.. /ttpsss//pppeeeeeeeeeeeeeee net forum? id= 3yabbMjj..
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint 1rX1v:2110.14168. 2021.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. http:////soooooooooo/////oooooop 2023.
DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning,
2025. URL https: org/ aass/aa///// wttpa////...a.. 129448
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 10835-10866. PMLR, 23-29 Jul 2023. URL httpt:tt//// ttttps://proceediigg
mlr wtt...pressvv22222 ga// 3h. html.
Anisha Gunjal, Anthony Wang, Flaine Lau, Vaskar Nath, Bing Liu, and Sean Hendryx. Rubrics as rewards: Reinforcement learning beyond verifiable domains, 2025. URL https: erg/abs/2500 17746.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual mul- timodal scientific problems, 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. URL http::/ http::::////eeeweee/ee net/ forum did-d 3
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurIPS. 2021b.
Jian Hu, Xibin WU, Zilin Zhu, Xianyu, Weixun Wang, Dehao Zhang, and Yu Cao. Openrlhf: An easy-to-use, scalable and high-performance rlhf framework. arXiv preprint 1rX.v.2405.11143, 2024.
Ruipeng Jia, Yunyi Yang. Yongbo Gai, Kai Luo, Shihao Huang, Jianhe Lin, Xiaoxi Jiang, and Guanjun Jiang. Writing-zero: Bridge the gap between non-verifiable tasks and verifiable rewards, 2025. URL https: /tatips://rxii...../ wwwpwiwwwor.///////e//eoo
00103.

---

Published as a conference paper at ICLR 2026
Yuxin Jiang, Yufei Wang, Xingshan Zeng, Wanjun Zhong, Liangyou Li, Fei Mi, Lifeng Shang, Xin Jiang, Qun Liu, and Wei Wang. Follow Bench: A multi-level ame-grained constraints fol- lowing benchmark for large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 4667-46666 7667-4688, Bangkok, Thailand, August 2024, As- sociation for Computational Linguistics. doi: 10. 11118663/1/2224.a00 a/2.0/2.....111og22555555 URL https: //aclanthology org/ / 2024. acl- 1aol-long.-. 257.
Yuxin Jiang, Yufei Wang, Chuhan Wu, Xinyi Dai, Yan Xu, Weinan Gan, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, and Wei Wang. Instruction-tuning data synthesis from scratch via web reconstruction. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.). Findings of the Association for Computational Linguistics, ACL 2025, Vienna, Austria, July 27 August 1, 2025, pp. 6603-6618. Association for Computational Linguistics, 2025. URL https: hta/cclannttt hology 2225 . findings acl. 343/.
Hynek Kydlíček. Math- Verifyy Math Verification Library, 2024. URL ht / /gi thub. com/ wwwpinnffa//mmaa--haeee this file.
If you use this software, please cite it using the metadata from
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling. In Luis Chiruzzo, Alan Rit- ter, and Lu Wang (eds.). Findings of the Association for Computational Linguistics: NAACL 2025, pp. 1755-1797, Albuquerque, New Mexico, April 2025. Association for Computational Linguisties. ISBN 979-8-89 891879991-77777 doi: 10. 10.18653/ 1/202//1122222222 www.s.iinninnnnnanaa naac1.96. https::: / //aclanthology crt. findings nww.aaaaa........m........aa...a .96/
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quan- titative reasoning problems with language models. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http:: wwtppspppppppcpppppappeppccceeee www.fffltsspeeeeeeeeaeaa html.
Tianle Li. Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, 2024.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacaeval: An automatic evaluator of instructionnfollooinng models, 2023.
Minpeng Liao, Chengxi Li, Wei Luo, Wu Jing, and Kai Fan. MARIO: MAth reasoning with code interpreter output reproducible pipeline. In Lun-Wei Ku, Andre Martins, and Vivek Sriku- mar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 905-924, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/ v1/2024. www.fiinnnnna...55555 URL https: http///aaaaannataaaea 2owpthoooo..or//22444444444 202/2024. fiieeefiinningaaccccc 53/.
Chris Yuhao Liu, Liang Zeng, Jiacai Liu, Rui Yan, Jujie He, Chaojie Wang, Shuicheng Yan, Yang Liu, and Yahui Zhou. Skywork- Bag of tricks for reward modeling in llms. arXiv preprint arXiv:2410. 188.511,,, 2024.
Chris Yuhao Liu, Liang Zeng, Yuzhen Xiao, Jujie He, Jiacai Liu, Chaojie Wang, Rui Yan, Wei Shen, Fuxiang Zhang, Jiacheng Xu. Yang Liu, and Yahui Zhou. -kwadrk-- Scaling preference data curation via human-ai synergy, 2025a. URL
/
2507. 01352.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective. In Conference on Language Modeling (COLM), 2025b.
12

---

Published as a conference paper at ICLR 2026
Xueguang Ma. Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. General--easoner: Advancing Ilm reasoning across all domains. urXiv:2505114622, 2025. URL WtLLttsss//aaii org/abs/250t 2505 14652.
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with reference-free reward. Advances Neural Information Processing Systems (NeurIPS), 2024.
George A Miller. The magical number seven, plus or minus two: Some limits on our capacity for processing information. Psychological review, 63(2):11, 1956.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. Cross-task general- ization via natural language crowdsoureing instructions. Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Anmual Meeting of the Association for Computational Linguisties (Volume Long Papers), pp- 3470-3487, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 120.0/3312222222221122121 https: http::://aaaaathhlooooo
URL
wtgp//222/00 aono2.accllooe aci-long 244.
Elissa L Newport. Maturational constraints on language learning. Cognitive science, 14(1):11-28, 1990.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In Advances in neural information processing systems, pp- 27730-27744, 2022. URL httpt:tpp///m httptt::tt/ppeeeeeeeee /wwp//vieeeeeeeeeeooeeeeee..e.we net forummmummammmmmmoomm Aee.Am/AAE-TKAAAEEA
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp- 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguis- URL https: //acc//aclllc aallalathoooyyy. org/ 1040/.
tics. doi: 10.3115/1003383.00775
Qwen Team. Qwen2.5: A party of foundation models, September 2024. URL https: /qwenlm. github.ic/blon httbb/////bl///oooooo http///llg//we..... . 5/.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In Advances in Neural Information Processing Systems, 2023.
Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 11 2019. URL https: /tatts::/arxxi..ott /aw.iv..vv../////. abs/1908. 10084.
Florian Schmidt. Generalization in generation: A closer look at exposure bias. In Alexandra Birch, Andrew Finch, Hiroaki Hayashi, Ioannis Konstas, Thang Luong, Graham Neubig, Yusuke Oda, and Katsuhito Sudoh (eds.), Proceedings of the 3rd Workshop on Neural Generation and Translation, pp. 157-167, Hong Kong, November 2019. Association for Computational Linguis- tics. doi: 10.18653/v 1/D19-5616. URL https: http:///aaaaaaahaaaeeeee www.anhwwwooooooooooooeooooooeee wtt.//g--gg1///////
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathe- matical reasoning open language models, 2024. URL httts: org/ abs/ 03300.
Joar Skalse, Nikolaus Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and charac- terizing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471, 2022.
Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 28(1): 1972.
Yi Su, Dian Yu, Linfeng Song. Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding rl with verifiable rewards across diverse domains, 2025. URL https::///.. //ass://arxiee org/ htc//bb//2555 23829.
13

---

Published as a conference paper at ICLR 2026
Zhengyang Tang, Xingxing Zhang, Benyou Wang, and Furu Wei. Mathscale: Scaling instruc- tion tuning for mathematical reasoning. In Forty-fire first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, OpenReview.net, 2024. URL https: //openreviee formm? d=K 7ZN47M
Qwen Team. Qwq-32b: Embracing the power of reinforcement learning, March 2025, URL attps://qwennle hta.eenmmm..ihhubb 1o/blog/ htwpp//ggg///0/// qwq-32b/
Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023.
URL hwtp:::::/hhhgighggegesee
co/ http/daaasss//teee tttes///kuiiummm hemm/OOpeeeeeeeeeee rmee-22.- 2.5.
Robert A Wagner and Michael Fischer. The string-to-string correction problem. Journal of the
ACM (JACM), (11)):68-112) 1974.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu. Brian Lester, Nan Du, An- drew M. Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL https: httpn/pwewe..eeeeeeeeeeeeeeeeeee net/forum? id=gEZzGCozdqR.
Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned LLMs with nothing. In The Thirteenth International Conference on Learning Representations, 2025.
URL https: /ttpss://pppeeeee wpw.revieeee.eeeeeeeeeeee n forum? 1d=Pnk
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source IIm reinforcement learning system scale, 2025a. URL https: //arxiv. wwwp//gg//////ee.e/./oeeeeeeeege 2503. 14476.
Tianyu Yu, Bo Ji, Shouli Wang, Shu Yao, Zefan Wang, Ganqu Cui, Lifan Yuan, Ning Ding, Yuan Yao, Zhiyuan Liu, Maosong Sun, and Tat-Seng Chua. Rlpr: Extrapolating rlvr general domains without verifiers, 2025b. URL ht : aawii.iiivvv///v///aa.aa/aavavvv org/ abs///b///// 2506. 18254.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- Z00: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL hwt.hhtps////..... www.aaaiivvvoo....a....a..aea..a org/ abs / 2503. 18892.
Wen Zhang. Yang Feng, Fandong Meng. Di You, and Qun Liu. Bridging the gap between train- ing and inference for neural machine translation. Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4334 4343, Florence, Italy, July 2019. Association for Computational Linguis- tics. doi: 10.18653/w1/P19 14219 URL hetps: ////aaaaaannttoooyy..ooao wwtp:/ogg//pgeeee..
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging
with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https: //openreviews eww. net. forum? d=uccHPGDl.s
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Luan, Denny Zhou, and Le Hou. Instruction-followingg evaluation for large language models, 2023. URL https: / //laaiii.r//aaaaaaaaaaaaaaaaaa/a attpaa/////11111 07911.
Xiangxin Zhou, Zichen Liu, Anya Sims, Haonan Wang, Tianyu Pang, Chongxuan Li, Liang Wang, Min Lin, and Chao Du. Reinforcing general reasoning without verifiers. arXiv preprint arXiv:2505.21499 2025.
14

---

Published as a conference paper at ICLR 2026
APPENDICES
A DETAILED EXPERIMENTAL SETUP
A.1 IMPLEMENTATION DETAILS
We adopt the OpenRLHF (Hu et al., 2024) framework for efficient training. During SFT, we train models for epochs with a learning rate 2e-5, batch size of 128, a max sequence length of 2048, and cosine learning rate schedule with 10% warmup steps. During GRPO, we set the epoch to 1, the learning rate to 5e-7, the number rollouts to 8, max prompt length and max generation length to 1024 tokens, and maintain the same global batch size of 128. During DPO, train models for epoch with a learning rate 5c-7, a batch size of 128, a max sequence length 2048, and a 3 of 1e-2. All experiments conducted NVIDIA A800 GPUs. We report the average performance of three random runs.
A.2 PROMPT TEMPLATE FOR DATA CONSTRUCTION
Prompt Template of Generating Key Points for Answering the Question
are given ** "Question** Your task identify **all essential key paints** that the assistant notice when answering question. Return your output Python strings, where each string a key point.
*[[ueettion]:* (question)
Figure 4: Prompt template of generating key points for answering the question.
Prompt Template of Generating Keywords
**Reference *nswer** Your task to identify **all essential keywords (less than 3 words for each keyword)** from the reference answer that match each key point. These keywords should be explicitly mentioned or accurately reflected in a good Al-generated answer. These keywords should represent the core facts, concepts, or entities required to assess the correctness and relevance of the response.
You are given a **Question** **Y Points the Question**,
### Output Format: Please return your schema the following JSON
json
"key_points":
{(
"point": "<Key point>",.
"keywords": <a Python list of strings>,
})
))
**[Question]:** (question)
**(eey Points]:** ** (keypoint)
**[Rfference *
(reference)
Figure 5: Prompt template of generating keywords.
15

---

Published as a conference paper at ICLR 2026
Prompt Template of Generating Code for Style Conformity Checking
You given *Referencce Answer**. Your task evaluate whether ** "Al-generated answer** follows similar **format and style** as reference answer. The goal not to assess content correctness or completeness, but to compare the presentation, structure, and formatting features the two answers.
#Mii Key Evaluation Focus:
Your evaluation should focus on *"style-related aspects** such as:
Overall structure and organization (e-g, of sections or bullet points)
Length similarity (in terms word count, within reasonable range)
Paragraph and distribution (roughly comparable, necessarily identical)
of markdown elements like text, headers, lists, code blocks
Visual layout and clarity
**Do NOT evaluate the factual content, relevance, or correctness of the answer."
##h Instructions:
1. **Identify 3-6 Key Style Evaluation Points:** For each point, define **specific and measurable** style- related criterion. Allow for small variations instead of requiring matches.
**Define Python Function for Point:** The function should named "evaluate(answer: str) bool" return:
"True" if the Al-generated answer satisfies the style point (even approximately),
"False" otherwise.
3. **Assign a Weight to Each Point:* Each point should be assigned a weight that reflects its relative importance. All weights must sum to **1.0**
### Output Format: Please return your evaluation schema the following JSON format:
"json
{(
sey_poontss..
{(
"point": "<Key evaluation points",
"explanation": "<Why this point helps assess format/style similaritys", "verification code": "def evaluate(answer: str) >> bool:\/n Your logic here(In return..."
"weight": <float weight>
B)
"total weight":
** "eefereee *
(reference)
Figure 6: Prompt template of generating code for style conformity checking.
A.3 TEMPLATE FOR MATHEMATICAL REASONING
Figure 7 shows the training and evaluation template for mathematical reasoning, where we first require the model think step by step and then output the final answer within "boxed()".
Training and Evaluation Template for Mathematical Reasoning
||m_star||>user (question)
Please reason step by step, and put your final answer within [\boxe{f=<[im_end]]] <| <|im_se hom_start|>aase passissat
Figure 7: Training and evaluation template for mathematical reasoning.
16

---

Published as a conference paper at ICLR 2026
B EVALUATION DETAILS
Table lists the evaluation details for AlpacaEval 2 (Li ct al., 2023), Arena-Hard (Li et al., 2024), MT-Bench (Zheng et al., 2023), IFEval (Zhou et al., 2023), and FollowBench (Jiang et al., 2024). AlpacaEval comprises 805 questions from datasets, and MT-Bench spans 8 categories with a total of 80 questions. Arena-Han Hard is enhanced version of MT-Bench, featuring 500 well-defined technical problem-solving queries. IFEval comprises 541 samples designed to evaluate instruction- following LLMs through diverse, verifiable instructions that include numerous lexical and format- ting constraints. Bench multi-level, fine-grained benchmark for evaluating constraint- following capabiliiies, featuring 820 samples across five constraint types and five difficulty levels. To balance cost and performance, we select GPT-4.I-min as the judge. Evaluation metrics are re- ported in accordance with each benchmark's protocol. For tasks across multiple domains, we align our evaluation settings with OpenCompass (Contributors, 2023).
Table 8: Evaluation details for AlpacaEval 2, Arena-Hard, MT-Bench, IFEval, and FollowBench The baseline model refers to the model compared against.
Benchmark
Exs.
Baseline Model
Judge Model
Scoring Type
Metric
AlpucaEvall
GPT-4 GPT-4-0144
GPT-4.1-mine
Pairwise comparison
Length-controiled rate
Arena-Hard
500
GPT-4.1-min
Pairwise compurison
Win rute Rating of 1-10
MT-Bench
80
GPT-4. I-mini
Single-answer grading
IFEval
S41
Kule-based verification
Accuracy
Follow Bench
820
GPT-4.11 UPr--.1-mini
Rule and LLM verification
Satisfaction rate
C EXPERIMENTAL RESULTS OF LLAMA3.
Table 9 presents results on Llama3. 1-----8BBnn Instruct, as prior work shows that effective GRPO training requires sufficiently strong base model (Liu et al., 2025b). RLVRR consistently outperforms all baselines by more than 2 points, with a comparable improvement observed on Qwen2.5. These findings confirm that approach generalizes robustly across different model architectures.
Table 9: Evaluation results of Llama3.. 1-8B across five open-ended benchmarks and four other tasks.
Method
#Data
Alpaca Eval
Arem Hard
MT Bench
Eval
Follow Bench
Avg.
MMLUU
ARC
MATHI
Human Fvai
Avg.
Instract
30.9
34.3
8.4
76.8
54.2
40.9
69.4
51.9
72.6
69.3
SET
10K
31.2
46.9
8.5
75.6
53.2
43.1
67.7
83.7
20.6
69.9
68.0
SET
51.0
8.5
55.3
67.4
GRPO (Rundom)
17.5
GRPO (BLEU)
10K
30.4
39.6
8.2
69.2
49.8
39.4
69.2
82.0
30.2
72.8
68.6
GRPO (RM)
10K
32.5
50.7
8.7
76.0
$3.7
44.3
69.6
84.2
$2.4
72.1
69.6
GRPO (GRM)
GRPO (RLPPR
48.6
8.3
74.6
54.4
43.3
68.7
52.7
72.3
69.1
DPO
10K
32.0
48.1
8.6
77.3
53.6
41.9
69.1
83.8
$1.0
72.2
69.0
GRPO (RLVRR)
36.7
52.3
8.7
77.7
56.2
46.3
70.2
84.9
52.6
73,0
70,2
D COST ANALYSIS
D.1 COST OF DATA CONSTRUCTION
The data construction phase, responsible for synthesizing verifiable components for content and style reward, operates exclusively offline, meaning it incurs no runtime cost during model training- For context, we estimated the budget for data synthesis using the GPT-4o-mini API, based on the API's pricing of $0.15 per 1M input tokens and $0.60 per IM output tokens. Table lists the breakdown of the estimated costs, which demonstrates that the overall expenditure ($21.36) is both reasonable and manageable.
Can open-source LLM be utilized alternative? In Table 4, explore the impact of LLMs on generating verifiable components during the data construction phase. Our findings in- dicate that substituting the GPT--4o-mini model with less powerful yet open-source alternative,
17

---

Published as a conference paper at ICLR 2026
Table 10: Estimated budget for data construction using the GPT-4o-mini API.
Task
Samples
Avg. Input Token Length
Avg. Output Token Length
Cost ($)
Multiple References
20,000
8.33
Key Points
10,000
1.71
Keywords
30,000
1,156
103
7.06
Style
10,000
853
497
4.26
such as 1amma3-70B-Instruce yields comparable performance while significantly surpassing SFT trained with 10x more data. The Jlama3-70B-Instruue model can be deployed only NVIDIA 3090 GPUs, with the option to further reduce hardware requirements through low-bit quantization". This provides economical alternative for RLVRR without compromising perfor- mance. Overall, our framework demonstrates robustness in leveraging diverse LLMs for verifiable component generation, confirming its adaptability and effectiveness.
D.2 COST OF RL TRAINING
RLVRR incurs negligible computational overhead. As shown Table 11, we report the average runtime per training step 8 NVIDIA A800 GPUs across var- ious reward strategies. RLVRR increases runtime by only 0.71% compared line, comparable to the lightweight BLEU-based re- ward (+0.67%). In contrast, RM introduces sub- stantial 8.28% overhead due to the need to maintain and query learned reward model, while RLPR incurs 6.43% increase from additional reference forward passes. These results highlight that RLVRR achieves verifiability with minimal runtime cost, making scalable choice for real-world RI. training scenarios.
Table 11: Average runtime per step for dif- ferent reward strategies RL training.
Method
Time (s)
Random (%)
the Random Reward base-
Random BLEU
121.56
0.00%
122.38
0.67%
RM
131.62
8.28%
GRM
128.92
6.05%
RLPR
129.38
6.43%
RLVRR
122.42
0.71%
E REWARD CURVES
Figure 8 presents the training dynamics of RLVRR terms of content and style rewards. Both rewards exhibit consistent upward trend in the early stages, indicating effective optimization across dimensions. Notably, the style reward plateaus after approximately 60 steps, suggesting that stylistic improvements saturate relatively quickly. In contrast, the content reward continues increase, albeit more gradually, highlighting the model's sustained ability to refine content quality over time.
0.7
Reward
0.5
0.3
Content
Style
0.2
20
40
60
80
100
Step
Figure 8: Content and style rewards of RLVRR during training, based on Dween.553B--Bast
httpps:/ hhtp:s//////tbtuuooeeee /
18

---

Published as a conference paper at ICLR 2026
F CASE STUDY
In this case study, we analyze the performance of various methods, all based on the Qwen2.5- 3B-Base model, using sample instruction from AlpacaEval 2. Table 12 displays the responses generated by four different methods. The SFT model produces a concise and factually accurate answer, although it lacks detail and context regarding the name change. In contrast, models further trained with BLEU and RM yield incorrect responses, asserting that Facebook Corporation did not change its legal name and providing an inaccurate account of the rebranding process. Our proposed method, RIVRR, demonstrates a notable improvement by providing a response that is both factually accurate and comprehensive. Additionally, the response generated by our method is significantly shorter than those produced by BLEU and RM. This combination of detail, accuracy, and brevity highlights the superiority of our approach in delivering informative and precise answers.
Table 12: Generated responses from different methods for a sampled instruction in AlpacaEval 2.
Instruction
Did Facebook corporation change name?
SFT
Yes, Facebook Corporation changed its
Meta Platforms, Inc. In 2021.
No, the Facehook Corporation did not alter its legal name. Despite the fact that the company has introduced a diverse array of products and services beyond its primary social networking platform, which includes popular applications such Instagram, WhatsApp, Messenger, and various others, official name of the corporation continues Nevertheless, in October 2021, 1, Facebook made an announcement indicating their intention to modify their corporate brand name to better reflect their expanding range of offerings. As of now, the brand name has not been disclosed to the public, and it is anticipated that the complete transition this new name will require significant amount of time.
Facebook, Inc.
BLEU
No, Facebook Corporation did not change its name. Facebook Corporation, also known as Facebook, Inc., is the parent company of the Facebook social media platform, and it has not gone through any significant name change. The company holds other online platforms and services such Instagram and WhatsApp, but the name Facebook Corporation remains the same. There have been some changes the branding, such as the rebranding of What- sApp and Facebook Messenger with the Meta logo, but the parent company's 's name has not changed. The announcement made by Facebook in October 2021 about rebranding its parent company Meta did not materialize, and Facebook Corporation still holds the name Face- book Corporation.
RM
Yes, Facebook Corporation changed its name to Meta Platforms, Inc. In October 2021. The rebranding aims to reflect the company's 's expanded focus beyond social media and into building the "metaverse," virtual space where people can interact and communicate in immersive, 3D environments. The new name "Meta" represents the company's ambition to drive breakthroughs in areas such as virtual and augmented reality.
RLVRR
G LLM USAGE
We utilized large language models to support both manuscript polishing and data construction. In particular, the GPT-4o-mini API is employed to assist with the construction of the training dataset. Further details of this process are provided in Section 3 and Appendix A.2.
19


Reviews:
[{"review_id": "5Hc2HiXY4u", "reviewer": "Reviewer_ExFE", "summary": "The paper introduces RLVRR (Reinforcement Learning with Verifiable Reference-based Rewards), an extension of RLVR (Reinforcement Learning with Verifiable Rewards). While RLVR works well for reasoning tasks (like math and code) where correctness can be verified from the final result (\"verifiable dots\"), it struggles with open-ended generation where there is no single ground truth. RLVRR tackles this challenge by introducing reference-based verifiable reward chains. The core idea is that instead of rewarding only the final verifiable answer, RLVRR derives an ordered linguistic reward chain from reference responses.\n\nThe method decomposes the reward into two verifiable components: (a) content reward that measures inclusion and order of key content elements (keywords or entities) extracted from references, using a two-level hierarchical keyword extraction (key points --> keywords). Alignment is computed via Longest Common Subsequence (LCS) between reference and generated keywords; (b) style reward that checks adherence to stylistic properties using small, verifiable Python functions automatically generated by an LLM (e.g., enforcing markdown format, word length, structure). These are weighted based on LLM-estimated importance. The policy is trained with the GRPO algorithm, optimizing expected rewards with a KL regularization term relative to a reference model.\n\nThe authors evaluate their approach using Qwen2.5 (3B, 7B) and Llama3.1 models. They use 10K RL training samples constructed from a 100K instruction-following corpus. The baselines compared are SFT (10K & 100K), Random, BLEU-based, Reward Model (RM), Generative Reward Model (GRM), RLPR, and DPO. The benchmarks used are focused on open-ended generation: AlpacaEval 2, Arena-Hard, MT-Bench, IFEval, FollowBench, and other domains: MMLU (knowledge), ARC (reasoning), MATH (math), HumanEval (code). Overall, RLVRR outperforms all baselines, including SFT (even with 10× more data) and RLHF-based methods.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is very well-written and easy to understand. In particular, Section 3, the pipeline diagram, and the prompts provided in the Appendix make it easy for readers to follow and grasp the details.\n\n- RLVRR’s \"reward chain\" bridges reasoning-style verifiability with open-ended text generation, which is a key step beyond single-dot verification. It removes dependency on trained reward models, reducing cost, reward hacking, and brittleness.\n\n- RLVRR outperforms SFT, DPO, RLHF-style, and BLEU-based methods across 10+ benchmarks, with minimal compute overhead. It also demonstrates cross-domain robustness, integrates with math reasoning (RLVR), and maintains generation diversity.", "weaknesses": "- One major weakness is that RLVRR captures only rule-based content and style fidelity; it may miss deeper semantic or ethical nuances that require human judgment. For example, the \"content reward\" relies entirely on an LLM judge, which is responsible for extracting critical keywords from the reference responses (the paper claims these keywords capture the \"semantics\" of the reference). However, this does not truly capture semantics for reasoning tasks like mathematical reasoning (GSM8K, MATH, OlympiadBench); it may only work for tasks like essay generation or plain natural language responses. For mathematical reasoning, adhering to certain tokens does not in any way ensure the generated response is correct, and there are no inherent discrepancies. The idea of extracting key points and then keywords is a reasonable abstraction, but it is unclear why it should improve performance on mathematical reasoning tasks. Furthermore, for these tasks, shouldn't the \"verifiable dots\" also be included in the reward? The final answer is at least as important as, if not more important than, adhering to key points or keywords.\n\n- For the style reward, the authors use an LLM to generate a Python verifier function that checks properties like answer length and markdown formatting. These rewards capture surface-level stylistic/formatting patterns rather than the semantic equivalence of the generated and reference responses. It is not clear how these align with tasks like Knowledge, Reasoning, or Math benchmarks. While such patterns may be useful for open-ended instruction-following tasks, they make little sense for Knowledge, Reasoning, or Math tasks.\n\n- The metrics used for the different benchmarks should be clearly defined, and more appropriate/fine-grained metrics for instruction-following should be considered. For example, \"length-controlled win rate\" or using \"GPT-4.1-mini as the evaluation judge\" are weak metrics for evaluating instruction following. They do not effectively capture how well the instruction is followed or, more importantly, whether the \"content\" and \"style\" of the generated responses are adequate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vxmzMtzeMy", "reviewer": "Reviewer_YJen", "summary": "This paper propose a novel RLVRR framework that extend RLVR to domains where reference answers are open-ended and long-form. RLVRR includes two kinds of rewards: content reward that maximize LCS between ordered keywords mentions in reference answer and generated response, style reward that constrains factors like formats and length based on the reference answer style. Comprehensive experiments demonstrate that RLVRR surpass BLUE, RM, GRM, RLPR baselines.", "questions": "1. How does RLVRR handle the flexible natural of languages, as the model could state the same thing in different word choices compared with the reference answer. In this case, the LCS method fails to capture the semantically equivalent  keyword mentions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is more effective compared with competitive baselines like RM and GRM, while also efficient.\n2. Experiments on both base models and instruct models show good performance of RLVRR.", "weaknesses": "1. Lacking the reward quality analysis. For example, how RLVRR achieves better results than the GRM setting that also rely on GPT-4o-mini, is it because the reward quality of RLVRR is better?\n2. It requires proprietary APIs to generate key points information, which might also hinder its scalability.\n3. Is RLVRR more computation efficient compared with the SFT baseline, regarding the training costs. Since SFT is much faster compared with RL which requires slow online response generation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RGr5ByA2d5", "reviewer": "Reviewer_SpgL", "summary": "The paper proposes RLVRR, reinforcement learning with verifiable reference-based rewards, to bring RL with verifiable signals (RLVR) beyond math/code into open-ended generation. Instead of a single “verifiable dot,” RLVRR builds a reward chain from a high-quality reference: a content reward (reference-derived keywords/points checked via rule-based matching) plus a style reward (small, LLM-generated, verifiable Python checks like length/format), aggregated for GRPO training with a KL penalty to a reference model. This yields stable, cheap, and hack-resistant supervision and shows gains across >10 benchmarks with Qwen and Llama models.", "questions": "see above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Splits reward into content (reference-derived key points/keywords) and style (verifiable code checks), avoiding fuzzy learned RMs during training.\n- On open-ended and other tasks, RLVRR improves over SFT (even with 10× more data) and over other baselines, and computation overhead is very small\n- Mixing math RLVR data with open-ended RLVRR produces competitive math and open-ended performance, showing the framework extends the RLVR paradigm.\n- Quite cheap data contruction cost (and even open-source LLMs can replace GPT-4o-mini) to build verifiable components with comparable outcomes.", "weaknesses": "- The content reward relies on information extracted by the reference LLM; performance may hinge on that accuracy/quality. You show a random baseline, but I’m curious how small perturbations/noise in the extracted cues affect results.\n- To cut compute, the reward checks text form rather than semantics. If the policy improves and paraphrases with different wording, this approach may hit a ceiling by penalizing valid semantic matches that use different tokens.\n- What happens if the policy is an instruction-tuned model instead of a base model? I wonder how that changes reward alignment and behavior, especially for formatting/style constraints.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kkBvWygEqC", "reviewer": "Reviewer_kHas", "summary": "This paper presents RLVRR, an approach to extend verifiable rewards to open domain generation by generating reward chains from high-quality references. The approach generates two kinds of rewards - style-based and content-based rewards. Content-based rewards are computed by extracting keywords from the references using an LLM and computing LCS with the response being evaluated, and the style-based rewards are generated as programmatic tests. Experiments on 10 benchmarks with Qwen and Llama models show that this method outperforms SFT-trained with 10 times more data, training with advanced reward models, and unifies the training of structured reasoning and open-ended generation, and generalizes more effectively while preserving output diversity.", "questions": "1. How did you arrive at the range of 3 to 6 functions for the style-based reward for the prompt shown in Figure 6?\n2. Did you do any ablations where you gave example style-based reward functions to the LLM to give more inspiration on what type of functions to generate, and would that help prevent the generation of superfluous/unnecessary functions?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Superior to SFT and alternative reward strategies. Also improves over DPO\n2. Robustness across scales, model families, and initializations\n3. Generalization to diverse tasks, including open-ended generation and knowledge-intensive reasoning, math, and coding tasks.\n4. RLVRR doesn't compromise output diversity despite its reliance on verifiable references.\n5. Low financial cost for data construction and low computational overhead for training (barely more than a random reward baseline).", "weaknesses": "1. Content-based rewards still rely on the mere notion of LCS with keywords, which seems gameable and doesn't capture deeper semantics. While the authors perform ablations that only calculate the fraction of keywords, which are completely vulnerable to reward hacking by models via keyword stuffing, LCS is not as easily hacked. However, what if cases where some sentences are permuted compared to the references, which leads to smaller sequential overlaps, get passed over even if they are virtually equivalent? Doing some reward value analysis with a few curated examples like that could be interesting. \n2. Style-based rewards seem to be focusing on very surface-level characteristics. For example, in Figure 1, one of the stylistic properties evaluated is the markdown formatting, specifically the hierarchy of the title (exactly \"###\"). Could these rewards be overly strict? It doesn't make sense why \"#\" would be better than \"###\" given the question. Moreover, the program just uses \"###\" in answer, but what if the answer was something like \"Decentralization doesn't f###ing work\" (using ### for censoring rather than formatting). \n3. Additional experiments to validate the importance and impact of the LLM-based weighting scheme. Currently, the comparison is only done between no-weighting and weighting, and even there, the differences **don't look statistically significant** except for FollowBench. It would be interesting to compare it with a reasonable random weighting scheme to see if the weighting is much better than random.\n4. Computing \"false positives\" and \"false negatives\" of the reward chain: Doing a study with a small subset of data, where references are specifically engineered to break the reward functions (like the example given in weakness 2) and then seeing if the reward chain catches them or gives a false positive as well as references which are equivalent by the content is reorded (e.g sentnece permutation mentioned in weakness 1) and seeing if the reward chain misses them (false negatives) would be interesting. While the empirical results suggest that such scenarios might be unlikely in organic rollouts, it would be interesting to study whether this method has potential weaknesses that could surface in other scenarios or could be exploited for some sort of data poisoning attack.\n5. The ablations show that keyword quality matters more than quantity. This suggests that the reference quality also indirectly matters (since bad references would yield bad keywords). This makes me wonder if these techniques would be useful at all for learning from noisy references. It feels like the style-based rewards would also overfit to the formatting style of noisy references, and the inferior keywords would quickly degrade performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)