RLAC: Reinforcement Learning with Adversarial Critic for Free-Form Generation Tasks
Authors: Mian Wu
Gavin Zhang
Sewon Min
Sergey Levine
Aviral Kumar
Venue: ICLR 2026 Poster (2026)
Keywords: large language model
reinforcement learning
dynamic critics
language model post-training
open-ended generation
Primary area: foundation or frontier models, including LLMs
TL;DR: RLAC trains a dynamic critic jointly with the generator (RL policy) in an adversarial two-player game, enabling rubric verification for free-form generation tasks without exhaustively enumerating rubrics or manually engineering robust reward models.

Abstract: Open-ended generation tasks require outputs to satisfy diverse and often implicit task-specific evaluation rubrics. The sheer number of relevant rubrics leads to prohibitively high verification costs and incomplete assessments of a response, making reinforcement learning (RL) post-training with rubric-based rewards difficult to scale. This problem is exacerbated by the fact that often the best way to combine these rubrics into one single reward is also highly prompt-specific. We propose Reinforcement Learning with Adversarial Critic (RLAC), a post-training approach that addresses these challenges via dynamic rubric verification. Our approach employs a large language model (LLM) as a critic that dynamically identifies only the most likely failure modes (e.g., a factual error or unhandled edge case), which are then verified by an external validator to optimize both generator and critic jointly. By training both the generator and the critic, this game enhances the critic's error detection and the generator's output quality while reducing required verifications. Our experiments demonstrate that RLAC improves factual accuracy in text generation and correctness in code generation, while also outperforming exhaustive verification and reward model methods. We show that dynamic critics are more effective than fixed critics, showcasing the potential of RLAC for scaling RL post-training to free-form generation tasks.

Published as a conference paper at ICLR 2026
RLAC: REINFORCEMENT LEARNING WITH ADVERSAR- IAL CRITIC FOR FREE-FORM GENERATION TASKS
Mian Wul! Gavin Zhang² Sewon Min
Sergey Levine Aviral Kumar"
Shanghai Jiao Tong University UC Berkeley
'Carnegie Mellon University
tWork done when visiting UC Berkeley
nothern 4 420s stu.eduy en
Project website: https: /tt////maawww..mmmm/m/mmm github.hhh....g 1o/ RO/RLLC--ebsssse
ABSTRACT
Open-ended generation tasks require outputs to satisfy diverse and often implicit task-specific evaluation rubrics. The sheer number of relevant rubrics leads to pro- hibitively high verification costs and incomplete assessments of response, making reinforcement learning (RL) post-training with rubric- rewards difficult scale. This problem exacerbated by the fact that often the best way to combine these rubrics into one single reward is also highly prompt-specific We propose Reinforcement Learning with Adversarial Critic (RLAC), a post-training approach that addresses these challenges via dynamic rubric verification. Our approach employs a large language model (LLM) as critic that dynamically identifies only the most likely failure modes (e.g., a factual error or unhandled edge case), which are then verified by an external validator to optimize both generator and critic jointly. By training both the generator and the critic, this game enhances the critic's 's error detection and the generator's 's output quality while reducing required verifications. Our experiments demonstrate that RLAC improves factual accuracy in text generation and correctness in code generation, while also outperforming exhaustive verification and reward model methods. We show that dynamic critics are more effective than fixed critics, showcasing the potential of RLAC for scaling RL post- tesintraining to free-form generation tasks.
INTRODUCTION
Post-training methods for large language models (LLMs) have progressed dramatically over the past few years, from largely manual supervised fine-tuning (SFT) techniques that rely combination of manual data curation (Radford et al., 2018; Brown et al., 2020; Shengyu et al., 2023) to reinforcement learning (RL) methods that perform general preference-basee Ouyang et al., 2022) or optimize task-specific notions of correctness (Ziegler et al., 2019; Stiennon al., 2020). Despite these remarkable results, RL post-training is limited tasks with clear-cut success criteria (i.c., correctness of an answer or preference of a human user), and it remains unclear how to post-train LLMs with RL on tasks that require producing open-ended or frre--frm outputs that are hard to verify perfectly.
optimization (Christiano et al., 2017;
Perhaps the biggest challenge in building RL post-training methods for free-form generation tasks is the lack of solid reward function: outputs are typically expected to satisfy several task-specific rubrics. principle, task designer could construct reward by combining these rubrics, but both enumerating and verifying them pose major scalability challenges (Min et al., 2023).
For instance, complex code generation requires testing countless edge cases (e.g., empty inputs or specific numbers). Even if such criteria could be enumerated, knowing how to combine them remains difficult (e.g., should correctly handling even numbers outweigh handling primes?). While RLHF-trained reward models or LLM-as-judge approaches (Christiano et al., 2017; Zheng et al., 2023) outsource the job merging rubrics to a learned or prompted reward model. this often leads to reward hacking (Ziegler et al., 2019; Gao et al., 2023; Skalse et al., 2022; Eisenstein al., 2023), since the best combination is highly dependent on the prompt and the model being optimized. How then train L.Mss on free-form generation tasks with multiple (even uncountable) rubrics?

---

Published as a conference paper at ICLR 2026
Tell me bio Michiae!
7ell me Micheel
Cenerator
Rewardi0.77
Generator
Generator
Michael Jackson was a French songwrtte born in Parts, He rose to tame as child star with nis haunding meladies that captivated audionces acroas europo...
Michael Jackson was French songwriter bom in Fanis He rase fame
star haunting melodler
Micheel Jackson wes French congurtter bor Paris He rose fame as a child star with his founting melosles that capciested oudiences across Europe...
captivated ausiences across Europo...
Michael Jackson
Michoel Jaskson was born Paris
Critic
was a French
pongwniee
Micheel Jacksoon was a Frencht songwwiter
Verified
Verified
Reward Model
Reward:00.5
Reward:1
Reward:0
Eumerativve Method
(b) Reward Model
RLAGCours)
Figure 1: Comparison of three post-training paradigms on a biography example ("Michael Jackson"). (a) Enumerative verification explicitly extracts and checks every atomic fact before aggregating scalar reward, which accurate but expensive. (b) Reward-model methods skip verification and directly predict a scalar reward from a learned judge, which is efficient but prone to reward hacking. In contrast, (c) RLAC trains a learned critic to propose one likely-wrong fact (rubric) and verifies it via an external validator, If the fact indeed fails, the critic receives reward and the generator O: otherwise the generator receives and the critic O. This dynamic, adversarial feedback yields prompt-specific, verifiable, and scalable supervision for free-form generation tasks.
We introduce Reinforcement Learning with Adversarial Critic (RLAC), which formulates the problem as an adversarial game between a generator and critic. The critic is learned model that proposes a rubric (e.g., one test case) where the generator's output is likely to fail, and an external validator verifies this. Both models are trained jointly: the critic is rewarded when it correctly pinpoints a rubric that the generator fails (verified by an external validator), while the generator is rewarded when the critic is unable to do so. This formulation eliminates the need to enumerate or verify all rubrics, significantly improving training scalability. At the same time, it ensures that rewards are based on rubrics that are prompt-specific, adversarially chosen, and always on-policy. Figure 1 illustrates how RLAC achieves verification efficiency while maintaining accuracy through adversarial critic-generator dynamics on a biography generation example.
We evaluate Reinforcement Learning with Adversarial Critic on factual text generation and code generation, representing enumerable and non-enumerable verification scenarios, respectively. On 8-sentence biography generation with Qwen3-8B, Reinforcement Learning with Adversarial Critic achieves a FactScore of 0.889, surpassing FaciTune-FF5 FS's 's (Tian et al., 2024) 0.867, while reducing verification calls by 5.7 x. This efficiency gain scales with task complexity, from 4.4x for 4-sentence to 5.7x for 8-sentence generation. In code generation, despite using only 9% of the training data, Reinforcement Learning with Adversarial Critic achieves the highest average scores on both base models: 53.2 on S0wn2.5-Coder-777 prior methods AceCoder-Ri RM and AceCoder-Rule (Zeng et al., 2025).
and 56.6 Qwen2.5-Codee 5-Corr-- Instructt, outperforming
Our primary contribution is Reinforcement Learning with Adversarial Critic (RLAC), a novel post- training paradigm that frames free-form LLM optimization as an adversarial game between a generator and learned critic, with an external validator providing ground- truth feedback. This design avoids exhaustive rubric enumeration and mitigates reward hacking by producing task-specific and on- policy training signals. In experiments, Reinforcement Learning with Adversarial Critic consistently improves factual accuracy while reducing verification costs and surpasses prior methods on code generation, demonstrating scalable gains across both enumerable and non-enumerable verification tasks.
PRELIMINARIES
Our goal is to train an LLM generator that produces a free-form output satisfying requirements of the underlying task, without manually enumerating every rubric for evaluation and grading. In this section, we formalize this problem, introduce notation, and briefly discuss related concepts of reward models (Christiano et al., 2017; Ziegler al., 2019; Rafailov et al., 2023) and enumerative verification (Min et al., 2023; Trivedi et al., 2024; Saha et al., 2025; Wang et al., 2024b; Xic et al., 2025). We illustrate these in Figure 1. We then present our approach in the next section.
2

---

Published as a conference paper at ICLR 2026
Problem setup. We consider free-form generation tasks where outputs must satisfy many task- specific requirements, which we refer to as rubrics. For instance, a biography generation task may require that each factual claim is correct, while code generation task may require the program to handle all edge cases correctly. A math proof may require proving a certain set of intermediate results. To captures these ideas abstractly, let S be distribution over prompts presented to an LLM. Given E S, a generator LLM T (a s) is tasked with producing a textual output E A. We choose to use standard notation typically used in RL (S denoting the state space and A denoting the action space) as we later present an RL training objective. Each instruction is inherently associated with a set of rubrics (denoted as C(s)), where each rubric c e C(s) represents a verifiable property the output should satisfy, such as "the claim about Newton's 's birth year is correct" for biography generation or "the code handles null inputs" for code generation.
instructions that may be
We assume access to a binary verification or reward function R(s, a, c) that returns 1 if a generated output a ~ T (-Is) satisfies the rubric c on instruction 8, and returns 0 otherwise. An output is considered correct only when all rubrics C(s) associated with instruction are satisfied. Our goal is to train T (-|s) to maximize the probability of producing fully correct outputs:
g = arg max EggS
R(s, a. cEC(s)
II
In constrained domains with single, well-defined rubric (e.g., matching reference final answer in math reasoning). we can solve this opimization problem via standard RL. algorithms like PPO (Schul- man et al., 2017) or GRPO (Shao et al., 2024). Note that these RL algorithms require evaluating these rubrics on every sample drawn from the policy. However, such cases are rare in open-ended tasks with diverse rubrics. In these settings, C(s) can be extremely large or even unbounded, making Eq. 1 computationally intractable since every output must be checked against every rubric.
Reward models and enumerative verification. Most approaches to optimizing free-form generation tackle the challenge of diverse rubrics through two paradigms. RLHF (Christiano et al., 2017) trains a single proxy reward model from offline human preference data. While efficient, this optimization is hard because the learned proxy is only as good as its coverage of the preference dataset. When the generator explores beyond this support, the proxy can misalign (Gao et al., 2023), often necessitating additional constraints like KL regularization to avoid collapse. These constraints stabilize training but limit exploration, making it difficult to scale to highly free-form generation tasks (Dong et al., 2024).
Another approach is to enumerate the evaluation criteria and optimize their aggregate, either through prompting (Min et al., 2023; Saha et al., 2025) or via preferences implicitly elicited from hu- mans (Wang et al., 2024b; Mahan et al., 2024). While more faithful to the underlying rubrics (Trivedi et al., 2024), this strategy is fundamentally limited: it assumes the evaluation set C(s) can be exhaus- tively listed, which is unrealistic for complex tasks (e.g., all test cases for a nontrivial program). Even when such enumeration is feasible, iterating over the entire set is computationally prohibitive, turning optimization into intractable verification bottleneck. In practice, the enumeration procedure itself can introduce bias in the aggregated reward (Appendix A).
REINFORCEMENT LEARNING WITH ADVERSARIAL CRITIC
We now introduce our RL post- post-training approach, called Reinforcement Learning with Adversarial Critic (RLAC) for training LLM generators on free-form tasks. Our goal is to provide rewards while avoiding the scalability limits of enumerative verification and the misalignment of static reward models. The core idea is to recast verification of generator response as dynamic process guided by a learned critic. Concretely, we frame training as two-player game: given an output from the generator, the critic proposes rubric the output is likely to violate, while the generator aims to satisfy all such rubrics. An external validator then adjudicates whether the output meets the proposed rubric, and this supervision updates both generator and critic. In this way, verification becomes adaptive and adversarial, tailored to the generator's 's current weaknesses. We now formally derive this approach.
3.1 PROBLEM REFORMULATION
To derive our approach formally, our starting point is the objective of Equation 1, which requires a generation satisfy all rubrics in the set C(s): Since R(s, a, c) is indicator function for each c,
3

---

Published as a conference paper at ICLR 2026
can rewrite the requirement that all rubrics are satisfied minimum over all rubrics follows:
1 (R(s, a, c) = ve E C(s)} = min eEC(s)
a =) 11.
(2)
Intuitively, the minimum selects the worst-case criterion, i.e., the first failure mode encountered by the current model æ. Substituting Equation 2 into Equation I gives:
T arg max ESSS [awnt(1s)) min R(s,a, [[s. cEC(s)
(3)
However, this reformulation by itself does not make the optimization problem simpler: searching over C(s) is infeasible when C(s) is large or infinite (e.g., all possible test cases). To address this, we introduce a critic r²' modeled as a stochastic policy that takes an instruction- -eneerrtnnnnnno pair (s,a) as input and outputs rubric E C(s)in natural language, representing verifiable property that may fail. An external validator then checks the proposed rubric. Then we can rewrite Equation 3 into the equivalent min-max form:
π arg max min xe"
Es (eo) [R(s, O, c.]]
can be shown that the solution from Equation the same as that from Eq. (1), but now we bypass the need to enumerate all criteria over C(s) (Madry et al., 2018).
Pretty much like other mini-max optimization problems, we can solve the above optimization problem by iteratively updating π and π against each other. The optimization goal is to achieve robust generator 7 that does well even according to the most adversarial critic, upon convergence. More details with respect to the practical optimization algorithm will be provided in Section 3.2.
3.2 PRACTICAL INSTANTIATION OF RLAC
We now instantiate the two-player adversarial game from the previous section into a practical approach that we can use to train LLMs. As shown in Figure 1, we parameterize three aask-agnostic components that interact with each other during RL training. Each component instantiated differently based on the domain (as detailed in Section 4).
The generator T², is LLM that is fine-tuned produce output a A for instruction s S. RLAC samples multiple response generations from T for each instruction s. We train 7 to maximize the probability of producing outputs that satisfy all task-specific rubries. The prompt for the generator is included in the Appendix B.I.
Generator.
Critic. Our critic n" is a pre-trained LLM that RLAC finet.une. Specifically, for each instruction and query generation output a, the critic is prompted to generate a natural language output representing rubric e through auto-regressive decoding. The rubric along with the instruction and the generation are then sent to the external validator to obtain a reward signal R(s, a,.) E {0, }}. The prompt for the adversarial critic is included in the Appendix B.2.
Validator.
The validator is an external tool or process that can verify whether a generated response satisfies a rubric provided as input to it. Neither the generator nor the critic ever observes the reference solution any intermediate traces. The validator can be implemented in various ways depending on the domain, such as nule-based checkers or a software tool that evaluates a proposed code on a proposed test-case. Implementation details for specific tasks discussed in the Appendix C.
Updating the generator and critic. At each training step, we sample instructions s E S and have the generator n produce K candidate outputs - ag. For each (s,a,), a,), the adversarial critic n proposes a criterion c, which is then checked by the validator to yield a binary reward r; E {0. 1). This online feedback provides signals for both the generator and the critic. Outputs with r; 1
treated as positives (a+) and those with r; = 0 as negatives (a ), and the generator is updated using the DPO objective (Rafailov et al., 2023) with respect the reference generator m
C( 2 = Ea,B(a [og (log
(a+1s) (a+ls)
7 (a 1s) 3 Blog T [s)
(5)
We adopt DPO primarily for its simplicity and stability, it allows direct policy optimization from binary preference signals without requiring explicit reward scaling or KI.-penalty tuning. Importantly,
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 RLAC
1: Initialize parameters 79, nT; "eet Fiet
2: for each iteration do
3: ## Policy Evaluation for Generator T
4:
for each instruction do
5:
Generate generations d1,... OK (-(sss)
6: Sample a criterion from the adversarial critic for each generation Gj (-1s,a,;)
7:
Construct generator dataset D = {(s, a, R(s, a, G))
8:
## Policy Evaluation for Critic πT.
Optional
9:
for each instruction 8, output a do
10:
Generate N criteria Ci, .... CN ~ (s,sa)
11:
Construct critic dataset Dr (s.a) = [(s, a, R(s, a, o
12:
## Policy Improvement for Generator x
#new f- T
14:
for each update step do
Nnow f- nnew V.(w T
Equation
16:
ref <- T
Policy Improvement for Critic o. new <- T
Optional
18:
19:
for each update step do
20:
new "new V.((TT
news TTff)
Equation 6
21:
T ref π
RLAC is agnostic to the choice of policy optimization algorithm: Any online offline RL objective (e.g. PPO (Schulman et al., 2017), GRPO (Shao et al., 2024)) can be substituted here without affecting the overall framework, since the critic-validator loop provides compatible supervision in all cases.
Similarly, for each (s, a) pair, we sample N criteria from x Criteria rejected by the validator (invalid satisfied by the generator) treated negatives (c-), while valid, unsatisfied ones positives (ct). The critic is then updated with the same DPO objective relative to its reference policy "fe.
(ct(s,,0) (eg(cclss,
(c |s, (.) (= )
L(m"; Tes) B
log
Blog
B Blog T
(6)
In this way, evaluation and improvement are unified: the critic adaptively identifies failure modes, the validator provides ground-truth feedback, and both generator and critic are jointly updated to improve over time.
Algorithm summary. Algorithm I summarizes the practical implementation of RLAC, At a high level, the algorithm follows standard online RL loop that alternates between policy evaluation and improvement. In each evaluation step, we sample generations from the current generator 72, have the critic propose criterion and obtain verification to assign rewards. These rewards are then used to update the generator with the DPO objective (Equation 5). Optionally, we also collect evaluation data for the critic by sampling multiple criteria per instruction-generalie updated with its own DPO objective (Equation 6), allowing to adaptively identify weaknesses in the generator and provide more effective learning signals.
pair. The critic is then
4 EXPERIMENTS
We now evaluate our approach on two free-form generation tasks: factual text generation ($4.1) and code generation ($4.2). Factual text generation presents the muumerable-but-exeeniii regime, where all claims can, in principle, be verified but at a cost that scales with length the text. This tests RLAC's ability maintain verification quality while reducing calls. Code generation, by contrast, represents the hon-enumerable regime, where exhaustive verification is impossible due to infinite comner cases and intractable formal checks (Church, 1936). Here, the goal is to expose critical failures through targeted critic proposals. Together, these tasks span the spectrum from costly-but soilly-uttpoosse but-posssile to fundamentally intractable verification, highlighting the broad applicability of RIAAC
5

---

Published as a conference paper at ICLR 2026
Table Performance comparison on factual text generation. RLAC achieves the highest FactScore across all settings while using fewer verification calls than FactTune-FS.
sentence Generation
8-sentence Generation
Method
Corrt
# Incorr]
FSt
Calls
Corrt
Incorr]
FST
Calls|
Qwen3-4B
Baseline
10.07
6.43
0,610
19.62
12.08
0.619
FactTune-FS
10,66
3,48
0.754
214,911
20.65
5,99
0.775
341,657
ArmoRM
14.54
8.69
0.626
21.02
10.02
0.677
RLAC (Ours)
10.54
3.04
0.776
57,600
21.58
4.84
0.817
48,000
Qwen3-8B
Bascline
12.65
5,53
0.696
22.51
11.97
0.653
FactTunn-FFS
13.31
3.63
0.786
168,735
25.10
3.84
0.867
438,949
ArmoRM
12.96
6.86
0.654
23.31
8.92
0.723
RLAC (Ours)
13.14
3.37
0.796
38,400
24.33
3.03
0.889
76,800
4.1 FACTUAL TEXT GENERATION
Evaluation data & metrics. We follow Min et al. (2023); Tian et al. (2024) in adapting factual text generation task in which the model should produce concise biographies for given individual. We use 170 topies from the Wikipedia Biography Dataset (Lebret al., 2016), split into 120 for training and 50 for testing.
We use factual precision of the output (as defined by FactScore (Min et al., 2023)) as the primary metric, and also report the counts of correct and incorrect facts. To control for length, the model is instructed to generate either four or eight sentences. Since frequent calls the external validator costly, we additionally track the number of validator calls.
Base models & baselines. We compare RLAC against dominant paradigms of post-training: (I) enumerative verification, which relies on explicit checking of all atomic facts, and (2) reward- model optimization, which replaces external verification with learned scalar judge. To represent these paradigms, we evaluate the following baselines and prior approaches: (1) the Qwen3-44 4B and Qwen3-8B base models as starting generators for our study; (2) FactTune-FS (Tian et al., 2024), widely used method for factual text generation to represent exhaustive verification using external validator, FactScore, for all atomic facts; and (3) ArmoRM (Wang al., 2024a), which represents the reward model based method that produces reward score for the generated output.
Both the generator and critic are initialized from the same backbone models (Qwen3-4B and Qwen3- 8B) to ensure fairness. We use FactScore as external validator, i.e.. FactScore checks whether critic-proposed fact appears in the biography and is correct according to Wikipedia. All methods trained with multiple rounds of DPO updates, where the generator produces 10 outputs per prompt and the critic proposes rubrics per output. These values follow common configurations in preference-basee based optimization (Rafailov et al., 2023; Tian al., 2024) and provide balanced trade-off between exploration diversity and verification cost, ensuring fair comparison across methods (see Appendix D for detailed analysis of these hyperparameters).
Table I shows that RLAC achieves the highest factuality scores across model sizes and output lengths, while using significantly fewer verification calls. For instance, on Qwen3-8B with eight-sentence generation, it reaches a FactScore of 0.889, outperforming FactTune-FF (0.867) and ArmoRM (0.723). but with only 77k verification calls compared to 439k for FactTune-F3 This efficiency gap grows with output length: FactTune-FFS requires 4.4x more verification calls in the four-sentence setting (169k vs. 39k) and 5.7x more the eight-sentence setting (439k vs. 77k). This shows that RLAC scales more efficiently the generation complexity increases. Further results demonstrating robustness across multiple random seeds, as well as generalization to the MedicalQA dataset are detailed in Appendices E and G.
Results.
RLAC's improvements throughout training. Figure 2 shows how the generator's accuracy evolves training, measured along three axes: training epoch, number of verification calls, and KL divergence from the base model. In Figure 2(a), RLAC shows slight initial drop in FactScore (from 0.653 to 0.641). At this early stage, the critic has not yet learned identify the most obvious emors, so

---

Published as a conference paper at ICLR 2026
RLAC
Factune-FS
ArmoRM
1.00
1.00
1.00
0.95
0.90
0.90
0.90
FactScore
0.85
0.85
0.80
0.75
0.75
0.75
0.79
0.70
0.70
0.65
0.65
100k 200K
300k 400k 500k
0.0
0.1
0.2
0.4
0.5
Round
Verification Calls
KL Divergence
(a) Training Dynamics
(b) Verification Efficiency
(c) Exploration Behavior
Figure 2: Comparison of training dynamics, verification efficiency, and exploration behavior for RLAC, FactTune-F'S, and ArmoRM on the Qwen3-8B model with 8-sentence generation.
Table 2: Generator's 's test accuracy across critic types.
Method
# Corr
# Incorr
FS
Base
19.62
12.08
0.619
Noisy Validator
19.84
12.83
0.607
Static Critic
17.77
3.77
0.825
Adversarial Critic
21.58
4.84
0.817
the "mistakes" it proposes are often minor or even incorrect. As a result, the generator receives weak targeted training signals, and factuality temporarily degrades. After several rounds, the critic improves at detecting mistakes, which in turn accelerates generator learning. Once this dynamic stabilizes, the generator's ' factuality gradually improves, ultimately reaching 0.889, outperforming FaciTune-FS (0.867). This two-phase process illustrates how RLAC evolves from weak initial supervision highly efficient, targeted verification.
Figure 2(b) shows that RLAC achieves the same level of factuality as FaciTune-FF with far fewer verification calls (e.g., 67K vs. 368K to achieve 84%). This highlights the inefficiency of FactTune- FS, which repeatedly validates already correct facts, whereas RLAC dynamically targets high-risk errors, yielding greater verification efficiency and scalability.
Figure 2(c) measures exploration by tracking the KL divergence from the base model. Such deviation can usually be caused by either (1) improvements from the base model through effective exploration, or (2) reward hacking, which the model overfits the reward model and drafts without real quality gains. For RLAC, KL increases alongside monotonic FactScore gains (0.653 + 0,889), indicating productive exploration. In contrast, RL with a fixed offline reward model (ArmoRM) shows a rise in KL without the corresponding factuality gains, evidence of reward hacking. These dynamics (which are not backbone-specific, see Appendix F) complement Table while both RLAC and FactTune-FTS improve factuality, RLAC achieves comparable or higher FactScore with far fewer verification calls, whereas ArmoRM inflates output length without consistent accuracy due to its static reward.
Ablation study. We compare RLAC with two ablated variants to isolate the factors driving its effectiveness. In the first, we replace the external validator's outputs with random correctness labels assess the role of validator reliability, In the second, freeze the critic model, referred as static critic rather than training it adversarially with the generator, to evaluate the importance of adversarial joint training.
As shown in Table 2, noisy validation destabilizes training and reduces performance below the base model, highlighting the importance of reliable validation. The static critic achieves a superficially high FactScore by generating fewer facts, reducing both correct and incorrect facts, unlike the adversarially trained critic that increases correct facts while reducing errors. This indicates that the static critic inflates precision rather than genuine factual improvement. Figure 3 further illustrates these dynamics. The static critic's ' validator outcomes quickly rise to about 0.81, showing that the generator quickly learns evade its fixed patterns. In contrast, the adversarial critic's outcomes grow much more slowly and reach a lower level of about 0.6 by round 16, indicating that it continues to surface genuine errors and sustain learning pressure. In general, these results highlight that both reliable external
7

---

Published as a conference paper at ICLR 2026
Static Critic
Adversarial Critic
0.8
Mean Retum
10
12
Epoch
Figure 3: Average validator outcomes on suspicious facts proposed by the critic during factual biography generation. Higher values indicate that the critic more often misjudges correct facts (i.e., weaker supervision).
verification and dynamically adapting critic erucial: Without either, the generator fails to achieve meaningful gains in factual accuracy, validating the core design of RLAC.
4.2 CODE GENERATION
Evaluation data & metrics. We evaluate code generation performance using widely studied benchmarks: HumanEval (Base and Plus) (Chen et al., 2021; Liu et al., 2023), MBPP (Base and Plus) (Austin et al., 2021; Liu et al., 2023), BigCodeBench (Zhuo et al., 2024), and LiveCodeBench (V4) (Jain al., 2025). We use Passl as a primary metric. For efficiency analysis, we also report the total number of test cases executed during RL training as the number of validator calls.
the AceCode-88 Moadee-8K-haiis subset (Zeng et al., 2025), consisting of approximately 22K problems. We compare against the following baseline and prior methods: (1) the base model (0wen2.5-Coder-7BBBBB
Base models & baselines. For training data,
without training; (2) an enumerative RI. method AceCoder-Rule, which employs RL with rule-based binary rewards from test execution; and (3) reward model method AceCoder which uses RL with AceCodeRM-7B trained on approximately 300K preference pairs from AceCode-87K dataset.
and Wrren..-CCdeer-BB--.
For all methods, follow the AceCoder experimental setup using the AceCode-87K-hal sub- set (Zeng et al., 2025), which contains about 22K problems and generates k = 8 outputs per prompt. For training efficiency, our method samples 2k problems randomly from this subset and uses critic proposals per generation (see Appendix D). Note that, unlike factual verification, where each check is expensive, unit tests in code generation are cheap to execute. The fundamental bottleneck here is not the per-test cost but the non-enumerability of the test space. Therefore, test-case count is reported only for completeness; the primary evaluation metric remains Pass@k.
Results. Table 6 summarizes results across five widely-used code generation benchmarks. Despite training on only 2,000 problems (9% of the dataset used for RceCoder-RM and AceCoder-Rule), RLAC achieves the highest average scores: 53.2 using Qwen2.5-Coder- Base and 56.6 using Owen2.5-Codee. Coder-7B-Inne InIntruutt consistently outperforming both enumerative method (AceCoder-Rule) and static reward model method (AceCoder-RMM RM) across the majority of benchmarks. Similarly, while AceCoder-Rull executed approximately 7.86 million test cases during training, RLAC required 192 thousand to reach higher final performance, reducing verification cost by 97.5%.
We observe from Table that AceCoder-RM, not only fails to improve performance but can even degrade it under noisy validation. For example, on HumanEval, performance drops from 91.5 to 89.0 despite using the competetive reward model Peccoder-RM-7777 indicating reward hacking.
This fragility arises from the reward model trained on preference pairs from the AceCoder dataset, which itself contains noisy and incomplete test cases (Zeng et al., 2025). During RL training, as the generator's is outputs drift away from the RM's fixed training distribution, these noisy supervision signals are further amplified. The static RM cannot adapt, causing to favor spurious correlations rather than true correctness, leading the generator exploit flaws in the reward signal.
8

---

Published as a conference paper at ICLR 2026
Table 3: Results for HumanEval, MBPP, BigCodeBench Complete and Instruct (BCB-C, BCB-I), and LiveCodeBench, using two different base models. RLAC achieves the highest average score across benchmarks.
HvmanEval
MBPP
BCB-C C
BCB-1
Methad
L.B
Average
Base
Plus
Base
Plus
Full
Hard
Full
Hard
Base: -oonn..--oodr--BBBe
Baseline
83.5
79.3
80.4
69.3
45.8
16.2
40.2
14.2
28.7
50.8
AceCoder-RM
83.5
75.6
80.2
67.2
41.9
14.9
36.8
16.2
25.7
49.1
AceCoder-Rule
78.0
82.3
69.3
48.6
43.2
18.2
52.3
RLAC (Ours)
85.7
80.6
82.4
71.6
50.3
20.9
42.1
16.9
28.7
53.2
Base: Qwen2.5-Coded
7anserr7B-Instruet
Baseline
91.5
84.8
82.8
71.4
49.5
19.6
41.8
20.3
34.2
55.1
AceCoder-Rie RM
89.0
84.1
86.0
72.8
50.4
18.9
42.0
19.6
35,0
55.3
AceCoder-Roe Rule
90.9
84.8
84.1
71.7
50.9
23.0
43.3
19.6
34,9
55.9
RLAC (Ours)
93.3
86.0
83.9
73.0
52.2
42.3
19.6
35.2
56.6
RLAC also suffers from the noisy dataset since we use a simulated 8 as validator mentioned settings. Although the critic is also affected noise, its continuous adaptation allows it stay aligned with the generator's s changing behavior, preserving meaningful supervision. As a result, RLAC consistently improves performance across all benchmarks, even in noisy and imperfect validation environments, showing robustness to noisy validation.
Static Critic
Adversarial Critic
Derection Precision (%6
44
0.930
42
Score
0.925
0.920
0.915
0
2
Round
Round
Detection Precision
HumanEval Pass@1
Figure 4: Ablations on the static critic vs. adversarial critic. Static critic's detection accuracy degrades from 42.3% to 33.9% as the generator exploits its patterns, yielding minimal performance gains (+0.6%) compared to the adversarial critic's continued improvement (+1.8%).
Ablation study. We compare RLAC with variant that replaces the adversarially trained critic with a static critic to evaluate the necessity of dynamic adaptation. As shown Figure 4, the static critic's detection rate, defined by the fraction of test cases generated that correctly expose real errors, drops dramatically from 42.3% to 33.9% over three rounds, as the generator gradually learns to exploit its fixed detection patterns. In contrast, the adversarial critic maintains stable detection rate greater than 39% by continuously adapting to the evolving behavior of the generator.
This degradation directly impacts performance: with the static critic, the generator plateaus 92.1% Pass @ 1, while RLAC reaches 93.3%. Further analysis shows that static critic's test cases critic's 's output degenerated to minor variations of earlier ones, allowing the generator avoid detection by simplifying or reducing outputs rather than truly fixing bugs. These results highlight that dynamic adaptation essential for preventing reward hacking and driving real code correctness improvements.
5 RELATED WORKS
Reward models. One possibility for evaluating free-form and open-ended generations is to encode all criteria into a single scalar through learnt reward model. This is usually achieved through learning from offline dataset of human preferences (Christiano et al., 2017; Ziegler al., 2019;

---

Published as a conference paper at ICLR 2026
Yi et al., 2019; Böhm et al., 2019; Rafailov et al., 2023) or absolute ratings (Cui et al., 2024; Wang et al., 2024c). Multi-objective reward models (Wang et al., 2024a; Dong et al., 2024; Ji et al., 2023) expose several fixed dimensions (e.g., truthfulness, honesty), improving interpretability but still relying on static, globally defined criteria. Our approach differs conceptually: instead of collapsing all rubries into single scalar
fixed multi-objective vector, we learn critic that dynamically proposes a verifiable rubric for each instance and grounds its supervision through an external validator. This yields reward signal that is still scalar for RL optimization, but derived from an objectively checkable criterion rather than a static, unverified proxy, offering better alignment and reliability in open-ended tasks.
Enumerative verifications for free-form generations. To obtain comprehensive and reliable evaluation of free-form generations, the standard practice is to enumerate a set of fine-grained criteria (Zhuge et al., 2024; Min et al., 2023; Saad-Falcon et al., 2024; Chang et al., 2024; Xie et al., 2025). While they can be automatically deposed by LLMs for easier domains (Min et al., 2023; Jing et al., 2024), extensive manual annotations are typically required for more complex domains such as travel planning (Xie et al., 2024), codebase generation (Zhao et al., 2025), and research reproduction (Starace al., 2025). Dedicated computation and actions such as information retrieval (Min et al., 2023) and code execution (Zhuge et al., 2024; Starace et al., 2025) require manual rubric design or domain-specific validators (e.g., retrieval and code execution). Because all rubrics must be checked for each output, verification cost scales roughly linearly with the number of possible rubrics and may still miss unlisted error types. In contrast, RLAC replaces exhaustive enumeration with learned critic that dynamically selects the most informative, verifiable failure mode for each instance. By verifying only this targeted rubric via an external validator, the method retains rubric- level faithfulness while substantially reducing evaluation cost and exposing diverse, on-policy errors that static checklists often overlook. We provide additional discussion of outcome-reward RL for domains with verifiable answers in Appendix J.
LLM--ss---Judae Juddee Because of the common-sense and reasoning capabilities of pre-trained L.MMs, they can directly be prompted to serve as a judge to evaluate free-form generations (Zheng et al., 2023; Yuan et al., 2025; Zhu et al., 2025). Their capabilities in evaluations can be further improved through explicit fine-tuning (Wang et al., 2024b; Yuan et al., 2025). They can also be more interpretable and robust by introducing a long Thouugh (CoT) reasoning to explicitly verify fine-grained criteria (Saha al., 2025; Wang al., 2024b; Trivedi et al., 2024). Beyond rubric-only judging, generative verifiers treat verification itself as next-token generation: they first produce verification rationales or counterevidence, and then score or select candidates (Zhang al., 2025; Singhi et al., 2025; Setlur et al., 2025). These approaches, however, use the judge or verifier only as a static evaluator. They produce fixed judgments or explanations but do not learn adaptively from the generator's 's evolving behaviors. In contrast, RLAC treats the verifier as learned critic policy within an adversarial training loop: the critic dynamically proposes which rubric to verify for each instance, receives direct feedback from an external validator, and updates jointly with the generator. This design transforms audd- from a static scoring module into an active, on-policy agent that allocates verification effort where it is most informative.
6 CONCLUSION
We presented Reinforcement Learning with Adversarial Critic (RLAC), new post-training approach for open-ended tasks requiring diverse, task-specific rubrics, where exhaustive enumeration is infeasi- ble and optiomal reward design is unknown. RLAC formulates training as an adversarial min-max game between generator and a critic, model that dynamically identifies the worst-case rubric for each output and verifies it exterally. By jointly training both models, our approach bypasses the need for exhaustive verification or manual reward design while providing adaptive learning signals that prevent reward hacking. On the factual text generation task and code generation task, RLAC outperforms competitive baselines with significantly lower verification cost. Ablation studies further confirm the critical role of components such as adversarial critic training.
While we evaluate RLAC on two domains, we expect it to generalize broadly to other open-ended generation tasks where multiple evaluation criteria make exhaustive or mubric-b--rubee verification infeasible, such story or scientific text generation. By adaptively selecting the most critical rubric at each step, RLAC makes RL training practical for complex generation tasks that were previously intractable due the combinatorial explosion of rubries or the lack of universal reward functions.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We thank members of the UC Berkeley RAIL lab for their support and feedback on the project. We thank Qi Wang for helpful discussions and for polishing the draft of our paper. This work was done when Mian Wu was visiting UC Berkeley.
REFERENCES
Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. Program synthesis with large language models, 2021. URL https: /aa//aaiivvv.oo../eaaa abs 2108. 07732.
Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, and Aviral Kumar. Digirl: Training -hilh device-control agents with autonomous reinforcement learning, 2024. URL
https: /ttp:////xxvvv/ org.o/////ooooo als 6 11896.
Florian Böhm, Yang Gao, Christian M. Meyer, Ori Shapira, Ido Dagan, and Iryna Gurevych. Better rewards yield better summaries: Learning to summarise without references. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pp. 3108- 3118. Association for Computational Linguisties, 2019. doi: 10. 10.18833/V//D199 1307. URL https: //ii..i.. org/10 1 v1 11/D19-1303
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Vosss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tiee Lin (eds.). Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Infor- mation Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020. URL hettps:/proceedee neura neurips. cc/ httepapprr/22000 10120/har hash/ htee//15cccoddd 14477506bfcb433666 0bbfb8ac142f644 Abst:e444424----aaaaaataae html.
Yapei Chang, Kyle Lo, Tanya Goyal, and Mohit lyyer. Booookscore: systematic exploration of book-length summarization in the of 1lms. In The Twelfth International Conference Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024.
URL httph:tt////mm. /ttp::///ppeeeee http//ppeeeeeeeeeeeeeeeeeeeeeeee net/ Eorum?
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray. Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin. Suchir Balaji, Shantanu Jain, William Saunders. Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam. Vedant Misra, Evan Morikawa, Alee Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, 1lya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code, 2021. URL https: //tt://aaaiiiii org/ abs///b///aa abs//107 033 74.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic. Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances in Neural Information Processing Systems, 00:4299-4303 2017. URL https: tttttss:/pppppe //pppersppppppp/ae .ips wwtpppppp/../////ecccc 2017 file/d
5e2c0adad503cc (ataaaaa555.a1111111a2..00ooeee 90 pddf.
Alonzo Church. A note on the entscheidungsprobllm.. The journal of symbolic logic, 1(1):40 41, 1936.
11

---

Published as a conference paper at ICLR 2026
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, 2021. URL https: / /ttpxxaxvvv..////////........... g/abs/ 0tsss/1110 /2110.11111 141 68.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, and Maosong Sun. ULTRAFEEDBACK: boosting language models with scaled AI feedback. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL https: //openrevie ew net/ forum? dddBoorDDRo....
Ganqu Cui, Lifan Yuan, Zefan Wang, Hanbin Wang. Wendi Li, Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, Jiarui Yuan, Huayu Chen, Kaiyan Zhang, Xingtai Lv. Shuo Wang, Yuan Yao, Xu Han, Hao Peng, Yu Cheng, Zhiyuan Liu, Maosong Sun, Bowen Zhou, and Ning Ding. Process reinforcement through implicit rewards, 2025. https::::/ pttttps://ariiivi aaapaaiiiiioooo org/at s 1/2522001455
Hanze Dong. Wei Xiong, Bo Pang. Haoxiang Wang. Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, and Tong Zhang. RLHF workflow: From reward modeling to online RLHF, Trans, Mach. Learn. Res., 2024, 2024. URL https: /ttp:///pppeeiiieeeeeeeeeeeeeeee net/ forum? d=a 1 3aYUU9eU.
Jacob Eisenstein, Chirag Nagpal, Alekh Agarwal, Ahmad Beirami, Alex D' D'Amour, DJ Dvijotham, Adam Fisch, Katherine Heller, Stephen Pfohl, Balaji Lakshminarayanan, Sanmi Koyejo, and Deepak Ramachandran. Helping or herding? reward model ensembles mitigate but do not eliminate reward hacking. arXiv:2312.09224, 2023. URL https: / aaaaiiivivv.//.i org/ abs 22122 22..... 09244.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In International Conference on Machine Learning. ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 10835-10866, PMLR, 2023. URL https:ttp///....... wwtphhtss//pp/eeeeeeeee www.c//e.eeeccoooooooooooccoccc ings mlm press 2202/ gao.gaa2230 3h. html.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.nee 2025. URL https: / wwwpreveeweeeeeeeeeeeeeeee net/ /
Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. Beavertails: Towards improved safety alignment of LLM via human-prefereme preference dataset. In Alice Oh, Tristan Naumann, Amir Globerson, Kate Saenko, Moritz Hardt, and Sergey Levine (eds.), Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023. URL http::tppppppppe .npeesniiiin nips wtt.ee/ppppaaae hwppppper_aee htlpppaaplessspspp/appapppappp_p files/p aper 5 hhtp//aa///////aa rks html.
oper/2023. 2ttp//2233///a/
htt.://osobb----ttaattt------- Dataset
Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE-bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024, URL https: /wwp/oeeeew...... et/forum? iitrummmi==VFFF d=VTF 6
Liqiang Jing, Ruosen Li, Yunmo Chen, and Xinya Du Faithscore: Fine-grained evaluations of hallucinations in large vision- language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computationall Linguistics: EMNLP 2024, Miani, Florida, USA, November 12-16, 2024, pp. 5042-5063. Association for Computational Linguistics, 1TA.1///22..FFNNINGG.EEE. 2024. doi: 10. tendinggs-emipp 290. 53/v1/202/ findings
URL https: worg/oorgoooo.. 188
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca
12

---

Published as a conference paper at ICLR 2026
Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3:
Pushing frontiers in open language model post- mestttraining, 2025. URL abss2411.1512.
Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 1203-1213, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/1//D1--112 URL htttttst://acaaattoooo
13/DD66-128../
Wenfeng Liang, Xiaohong Zhang, Kai Wang, Rui Chen, Yiming Zhou, Mian Xiao, Rui Zhu, Ziheng Chen, Qing Wang, Hao Song, et al. Deepseek-r1: Incentivizing reasoning capability large language models via reinforcement learning. S.iv:25001122444 2025. URL https::/taaaee //axxiv. org/abs// 2501 12948.
Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and I Lingming Zhang. Is your code generated by chatgpt really correct? rigorous evaluation of large language models for code generation, 2023.
URL wwtphttps//// /ttp://axvvvvi.///e org/ abs/ 2305. 01210.
Michael Luo, Sijun Tan, Roy Huang. Xiaoxiang Shi, Rachel Xin, Colin Cai, Ameen Patel, Alpay Ariyak, Qingyang Wu, Ce Zhang. Li Erran Li, Raluca Ada Popa, and lon Stoica. Deepcoder: A fully open-source 14b coder 03-mini level. -tt-rr-------------------------- -a-ett-03-minnt -ii--lel---eeee te 81 11100cc448003beeeee333333333
notion. si
2025a. Notion Blog.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol ol-preview with a 1.5b model by scaling https: -aw-a--------de-b--------- ---------oddd-------aa----------
radio 675 www.ioniiinno.. site/en.
2025b. Notion Blog.
Aleksander Madry. Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. International Conference on Learning Representations, 2018.
Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models, 2024. URL org/ abs 2410. 12832.
https://aas /ttp:///aaivvvv//i
Sewon Min, Kalpesh Krishma, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. Factscore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pp. 12076-12100. Association for Computational Linguistics, 2023. doi: 1:065501//22.EMMMLPMMMMT EMNLP-MAIN.7A. URL https: doii org/10 1116533vv/2022 101.:22..mmn111.m maan-ppmmiin... 741.
OpenAl, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Jiayi Pan, Xingyao Wang, Graham Neubig, Navdeep Jaitly, Heng Ji, Alane Suhr, and Yizhe Zhang. Training software engineering agents and verifiers with swe-gym, 2024a. URL https: pectss://aret V. ooroor/aaioo /as///bb//// 2412.4222 21139.
Jay Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement digital agents, 2024b. URL https: : /ttp://wwww...a//////aaaaaa//a// 0464 74.
13

---

Published as a conference paper at ICLR 2026
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training: 2018. OpenAI.
Rafael Rafailov, Aditi Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728-53747 2023.
Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. ARES: an automated evaluation framework for retrieval-augee -augmeetee generation systems. In Kevin Duh, Helena Gómez- Adoroo, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico. June 16-21, 2024. pp. 338-354, Association for Computational Linguistics, 2024. doi: 10. 10.18653/V1 1/tp://///22...///// WAW.AAAAAALLLLON.... https:: /doi org/ / 10653/ www.cnaaclllooaa.. 220.
URL
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking-llm--a---judgg... CoRR, abs/2501. 180999 18099, 2025. doi: 10.485 22501. 18099. URL http::///aai..org/11....0 htt////////iii..// 2ttp://0000.ii......... 2501.18011111
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017. URL http:: http//axxiiv//.a////v//...a...aa org/ hbs///b///// /tt.:t/a//aaai..ooo//ab///////// 1707.06347.
Amrith Setlur, Chirag Nagpal, Adam Fisch, Xinyang Geng, Jacob Eisenstein, Rishabh Agarwal, Alekh Agarwal, Jonathan Berant, and Aviral Kumar. Rewarding progress: Scaling automated process verifiers for L.M reasoning. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: //openreview net http://wwww..nt//oouuddda....--.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseckmath: Pushing the limits of mathematical reasoning in open language models, 2024. URL /tttps://aaiiv. org/ /
2402. 03300.
Zhang Shengyu, Dong Linfeng, Li Xiaoya, Zhang Sen, Sun Xiaofei, Wang Shuhe, Li Jiwei, Runyi Hu, Zhang Tianwei, Fei Wu, et al. Instruction tuning for large language models: A survey, arXiv preprint arXiv:2308. 220.22308..7792 10792, 2023. doi: 10.48550/ARXIV 10792, URL https: ///oi. orgiloorg/110 4ww.4485550...... www.wwwwww.............
1079.. 10792.
Nishad Singhi, Hritik Bansal, Arian Hosseini, Aditya Grover, Kai-Wei Chang, Marcus Rohrbach, and Anna Rohrbach. When to solve, when to verify: Compute-optimal problem solving and generative verification for LLM reasoning. CoRR, abs/2504.01005, 2025. doi: 1044550/ARXXV.5544.000 01005.
URL https:tp//.............t. /doi. org/10..///00 11..1...85511 48550/650////
Joar Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. In Sanmi Koyejo, S. Mohamed, A. Agarwal, Danielle Belgrave, K. Cho. and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurlPS 2022, New Orleans, LA. USA, November 28- December 9, 2022, 2022. URL http: wwwpssapppppspppapppeeeeeeaeaeea httppp.ccpppppppe iles/paper/22 onference. html.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating ai's ability to replicate research, 2025. URL
https: /ttpxv/aavvv///////////////.aav. orgo wwt/abss/////oooooooo./o./..sssa /t.os//5544 2504. 01848.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei. and Paul F Christiano. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008-3021, 2020.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen. Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, Chuning Tang, Congcong Wang, Dehao Zhang, Enming
14

---

Published as a conference paper at ICLR 2026
Yuan, Enzhe Lu, Fengxiang Tang, Flood Sung, Guangda Wei, Guokun Lai, Haiqing Guo, Han Zhu, Hao Ding, Hao Hu, Hao Yang, Hao Zhang, Haotian Yao, Haotian Zhao, Haoyu Lu, Haoze Li, Haozhen Yu, Hongcheng Gao, Huabin Zheng, Huan Yuan, Jia Chen, Jianhang Guo, Jianlin Su, Jianzhou Wang, Jie Zhao, Jin Zhang. Jingyuan Liu, Junjie Yan, Junyan Wu, Lidong Shi, Ling Ye, Longhui Yu, Mengnan Dong, Neo Zhang, Ningchen Ma, Qiwei Pan, Qucheng Gong, Shaowei Liu, Shengling Ma. Shupeng Wei, Sihan Cao, Siying Huang, Tao Jiang, Weihao Gao, Weimin Xiong, Weiran He, Weixiao Huang, Wenhao Wu, Wenyang He, Xianghui Wei, Xianqing Jia, Xingzhe Wu, Xinran Xu, Xinxing Zu, Xinyu Zhou, Xuehai Pan, Y. Charles, Yang Li, Yangyang Hu, Yangyang Liu, Yanru Chen, Yejie Wang. Yibo Liu, Yidao Qin, Yifeng Liu, Ying Yang, Yiping Bao, Yulun Du, Yuxin Wu, Yuzhi Wang, Zaida Zhou, Zhaoji Wang. Zhaowei Li, Zhen Zhu, Zheng Zhang, Zhexu Wang. Zhilin Yang, Zhiqi Huang, Zihao Huang, Ziyao Xu, and Zonghan Yang. Kimi k1.5: Scaling reinforcement learning with 1lms, 2025. URL https:ttp/.... http/aawvvvvv..///////..///..t org/ abs 2501 12599.
Katherine Tian, Eric Mitchell, Huaxiu Yao, Christopher D. Manning, and Chelsea Finn. Fine- tuning language models for factuality. In The Twelfth International Conference on Learning Representationss ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL uttps:://openzeiiee net/ Porrumidd-W222PPPa.
Prapti Trivedi, Aditya Gulati, Oliver Molenschot, Meghana Arakkal Rajeev, Rajkumar Ramamurthy, Keith Stevens, Tanveesh Singh Chaudhery, Jahnavi Jambholkar, James Zou, and Nazneen Rajani. Self-rationalization improves Ilm as fine-grained judge, 2024. URL https: //txxaaxvvv/////////////////aa// org/ abs/2410. 05495.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami. Florida, USA, November 12-16, 2024. pp. 10582- 19582-0592. Association for Computational Linguistics, 2024a, doi: 10.18653/1120024FINDDN DEN.OSS-EMMP...... URL httpt:://ooo 9/10.18655 v1 2024. f
p. 620.
Tianlu Wang, Ilia Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu, Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators, 2024b. URL https:://// wrw.v.aa..ooo org.oogg//aooo. abs/2408 02 02666.
Zhilin Wang. Yi Dong, Jiaqi Zeng, Virginia Adams, Makesh Narsimhan Sreedhar, Daniel Egert, Olivier Delalleau, Jane Polak Scowcroft, Neel Kant, Aidan Swope, and Oleksii Kuchaiev. Helpsteer: Multi-attribute helpfulness dataset for steerlm. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computationna Linguistics: Human Language Technologies (Volume 1: Long Pupers), NAACI. 2024, Mexico City, Mexico, June 16-21, 2024, pp. 3371-3384 Association URL
for Computational Linguisties, 2024c. doi: howp::022..aaa11ooooo..... 185. https://doi.oot doii org/ http://...11//////0/00000
naacl
Yuxiang Wei, Olivier Duchenne, Jade Copet, Quentin Carbonneaux, Lingming Zhang, Daniel Fried, Gabriel Synnaeve, Rishabh Singh, and Sida I. Wang. Swe-rl: Advancing 11m reasoning via reinforcement learning on open software evolution, 2025. URL https: //arr///aiiiv///// org/ abs/ 2502. 18449.
Jian Xie, Kai Zhang, Jiangjie Chen, Tinghui Zhu, Renze Lou, Yuandong Tian, Yanghua Xiao, and Yu Su. Travelplanner: A benchmark for real-world planning with language agents. In Forty-first International Conference on Machine Learning, ICMI. 2024, Vienna, Austria, July 2 21-27, 2024. OpenReview.net, 2024. URL https: : wowooeoeevvviieeeeee ew net. forum? d= 15x0=NkACe.
Yiqing Xie, Wenxuan Zhou, Pradyot Prakash, Di Jin, Yuning Mao, Quintin Fettes, Arya Talebzadeh, Sinong Wang, Han Fang, Carolyn Rosé, Daniel Fried, and Hejia Zhang. Improving model factuality with fine-grained satique-based evaluator. arXiv preprint arXiv:2410.18355 2025. doi: 10.485500arxiv.24400 201.i2241..183... URL https: /ttp:i/aaivvvv//i org/ /aws//4111 18359. Version v3, last revised 2025-06-01.
Sanghyun Yi, Rahul Goel. Chandra Khatri, Alessandra Cervone, Tagyoung Chung, Behnam Heday- atnia, Anu Venkatesh, Raefer Gabriel, and Dilek Hakkani-Tine Towards coherent and engaging spoken dialog response generation using automatic conversation evaluators. In Kees van Deemter,
15

---

Published as a conference paper at ICLR 2026
Chenghua Lin, and Hiroya Takamura (eds.), Proceedings of the 12th International Conference Natural Language Generation, INLG 2019, Tokyo, Japan, October 29 November 1, 2019, pp. 65-75. Association for Computational Linguistics, 2019. doi: 10. 18653/8 1/W119606 URL https: / acl anthology orgg/ng/ W1 9- 8608/.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang, Mofan Zhang, Wang Zhang. Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei Ying Ma, Ya- fa-Qin Zhang. Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Ilm reinforcement learning system at scale, 2025. URL https: http://axxvv../// abs 2503.1 4476.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2025. URL https: http://wxwi..r........./o// 01.10020.
Huaye Zeng, Dongfu Jiang, Haozhe Wang, Ping Nie, Xiaotong Chen, and Wenhu Chen. ACECODER: acing coder RL via automated test-case synthesis. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilchvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), ACL 2025, Vienna, Austria, July 27- - August 1, 2025, pp. 12023-12040 Association for Computational Linguistics, 2025. URL https: // h/aaalaantt faclanthology org/ 2025 acl- 587/
Yuexiang Zhai, Hao Bai, Zipeng Lin, Jiayi Pan, Shengbang Tong, Yifei Zhou, Alane Suhr, Saining Xie, Yann LeCun, Yi Ma, and Sergey Levine Fine-tuning large vision-language models as decision-mmking agents via reinforcement learning, 2024. URL https: https://axxiv....r. www.aaaivvvi........ /bs/ 2405. 10292.
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. In The Thirteenth International Conference on Learming Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenRe- view.net, 2025. URL https: /ttp///pprrrreeeeeeee ew net/ wwwruniidcoowpp/oo
Wenting Zhao, Nan Jiang, Celine Lee, Justin T. Chiu, Claire Cardie, Matthias Gallé, and Alexander M. Rush. Commit0: Library generation from scratch. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: wpwpr/vweeeeeeeeeeeeeee/e. net/ wtt.://mdmmmmdaa0 www.dddddwwwwaa....gg.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng. Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li. Eric Xing, et al. Judging llm-as as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems. 36:46595-46623, 2023.
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net, 2025. URL https: http///pnnw..e..e/eeee/e. .n
Mingchen Zhuge, Changsheng Zhao, Dylan Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong. Zechun Liu, Ernie Chang. Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. jgent-as-a-judge: Evaluate agents with agents, 2024.
URL https: /ttp///axvvvv/vv///vv//.../...av org/ abs/2//0...1/////// 10934.
Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kaddour, Ming Xu, Zhihan Zhang, Prateek Yadav, Naman Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Liu, Zijian Wang, David Lo. Binyuan Hui, Niklas Muennighoff, Daniel Fried, Xiaoning Du, Harm de Vries, and Leandro von Werra. Bigcodebench: Benchmarking code generation with diverse function calls and complex instructions, 2024. URL https: : / /aapaaavivvo/////.a//aaa org/ abs/ /20/. 15877.
Daniel M. Ziegler, Nisan Stienmon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences, 2019. URL https:tt////. /ttp///aaivvvv/iv www.ooo/////.......ooo..o......o /1099 08593.
16

---

Published as a conference paper at ICLR 2026
Appendices
A LIMITATIONS OF EXHAUSTIVE FACT DECOMPOSITION AND PER-FACT VERIFICATION
In factual text generation tasks, the rubric for the text can be verified exhaustively, FactScore (Min et al., 2023) and our an LLM to decompose the text into atomic facts, then verify each atomic fact independently using retrieved evidence (e.g., top relevant Wikipedia passages), and finally aggregate these per-fact judgments into a single factuality score.
veriiiittiv baseline FactTune-FS (Tian al., 2024) directly use
However, this approach has drawbacks in terms of cost and accuracy. In this pipeline, the dominant verification cost scales linearly with the number of extracted atomic facts, i.e., the number of per-fact verification calls. In our biography settings, a 4-sentence output contains about 17 atomic facts on average, whereas an 8-sentence output contains about 30, so the cost grows quickly with output length.
Beyond cost, exhaustive fact extraction does not necessarily yield cleaner reward signal. In practice, atomic fact decomposition can introduce overlapping or near-duplicate claims, each of which verified independently and contributes separately to the aggregated score, thereby overweighting single underlying statement and amplifying noise. For example, consider the sentence:
"Aaron Kwok renowned Hong Kong singer and actor known for his versatile
talent in both music and film."
A plausible decomposition into atomic facts may include:
(fz) Aaron Kwok a singer.
(f2) Aaron Kwok is an actor.
(fg) Aaron Kwok is a Hong Kong singer.
(f+) Aaron Kwok is a Hong Kong actor.
(f5) Aaron Kwok is renowned Hong Kong singer.
(fg) Aaron Kwok is renowned Hong Kong singer and actor.
(f>) Aaron Kwok has versatile talent in both music and film.
Several items above are overlapping refinements of essentially the same claim (e.g., (fi) Never- theless, in an exhaustive pipeline they would be verified independently and would each influence the final aggregated score, potentially making the resulting reward noisy and sensitive to decomposition artifacts.
B PROMPTS
This section lists the exact prompts used for the generator model and critic model during data creation and training. They correspond to the input format described in Section 3.2 the main paper.
B. GENERATOR PROMPT
Factual Text Generation
System message: You Al assistant that provides accurate and concise biographies of individuals. Each biography should be exactly four sentences long, highlighting key aspects of the person's 's life, achievements, and significance.
User message: Write a biography of topic.
Code Generation
17

---

Published as a conference paper at ICLR 2026
The generator input exactly matches the problem statement provided in TIGER-Lab/ B without modification.
User message: iprolfeissaaaamee.rrrooooodddddd
B.2 CRITIC PROMPT
Factual Text Generation
System message:
You are a factual checker. Based en your existing knowledge,
identify exactly one sentence that contains the most clearly verifiable factual error in the paragraph. Return your answer in *eexactly three lines**: briefly explaining what is wrong
reason:
sentence:
N the number the most incorrect sentence
(positive integer)
error_fact: F
a brief clause (no more than words) capturing the wrong claim from that sentence
User message:
Here is an example to show the task. Find the sentence that contains the most clearly verifiable factual error in the paragraph about Albert Einstein.
Example paragraph:
[1] Albert Rinstein was awarded the Nobel Prize in Physics in 1921 for his discovery of the photoelectric effect.
[2] was bern in York City, United States, and later moved to Europe where he continued
studies. [3] Einstein developed the theory of relativity, revolutionizing our understanding of space, time, and gravity,
[4] His famous equation describes the equivalence of and energy.
Expected answer:
reason: Einstcin was actually bern in Ulm, Germany, not New York City. sentence:
error_fact: Albert Einstein born York City.
Now apply the same procedure to the paragraph below about (topic).
Paragraph: (numberred_poragraph))
Answer:
Code Generation
System message:
You are a code critic. Analyze code for bugs and generate failing test cases.
Strictly follow the format with <think> and <testcase> tags.
User message:
Analyze the given problem and the generated code to find test case that would cause the code to fail.
Problem: (question)
Generated code:
"''ythhn
(code)
18

---

Published as a conference paper at ICLR 2026
First, think through potential bugs and edge cases in <think> </think> tags.
Then output exactly ONE failing test case inside <testcase> tags using this format:
Option (CALL format)
<testcase> CALL: func_name (argl, arg2, kw=val) </testcase>
Option (SIDIN format)
<testcase> SIDIN:
input here> </testcase>
Do NOT include expected outputs or explanations. apptiional_xammle__blle
C VALIDATOR IMPLEMENTATION DETAILS
This section provides the detailed design of the validator used in our training pipeline, corresponding to Section 3.2 of the main paper.
C.I FACTUAL TEXT GENERATION
We follow strict validation process to ensure both authenticity and factual accuracy. In the first stage, the critic outputs both a suspected erroneous fact and the sentence number containing To prevent exploitation through information injection, we use textual entailment checking to verify that the proposed fact genuinely appears in the specified sentence. In the second stage, for proposals passing authenticity checks, we reuse FactScore's 's atomic fact verification component, which queries Wikipedia knowledge base to provide binary verification individual factual claims, returning true or false based external verification.
C.2 CODE GENERATION
Since the AceCoder dataset lacks reference solutions to prevent data contamination, construct reliable verification anchors by using --one.--Codrr-----nneeee
to generate solutions. To further also generate reference solutions using GPT-4o and observe similar
rule out same-family leakage, improvements (Appendix H). We filter these solutions using original test cases, retaining only those that pass, yielding set of highly reliable anchors (99.7% pass rate) that serve simulated ground test cases. Given
truth for validating critic-propose the reference solution to check that the case is valid (i.e., it executes and yields well-defined output) and to obtain the expected output, and then execute it on the model medee--geerated code to obtain the actual output. We define the validation reward as R(s, a, c) = if the outputs match and otherwise, treating execution failures as detected errors. The AceCoder dataset contains noise in GPT-4o generated test cases, which introduces some bias in our reference-basee validator but reflects realistic imperfections in verification tools.
test case, we first execute it on
D ANALYSIS OF K AND N
This section clarifies the roles of hyperparameters and N RLAC and explains why their values differ between the factual and code generation experiments.
In RLAC, the parameter K controls how many candidate outputs are sampled for each prompt. larger K increases candidate diversity and raises the probability that, for the same instruction, at least one candidate passes all critic checks while another fails at least one. This is essential for constructing non-degenerate preference pairs for DPO, since each pair requires both "chosen" and "rejected" candidate.
The hyperparameter N specifies how many criteria or testcases the critie proposes for each candidate. larger N expands the critic's search space and enables it to discover more potential failure modes. However, this comes with two main drawbacks: fewer candidates pass all checks (which
19

---

Published as a conference paper at ICLR 2026
reduces preference pairs available for training generator), and verification costs increase substantially. Additionally, an excessively large N may introduce redundant checks without meaningful benefits.
For factual text generation, set K 10, following the configuration used in FactTune-FS, main baseline. For code generation, we use K = 8, consistent with the AceCoder setup. Thus, the difference in K is not arbitrary; it adheres to the standard experimental settings established in prior work.
The choice of N is primarily determined by the structure of the underlying verification space. In the factual generation task, the rubric space used by FactScore is large and semantically diverse, and the biography dataset itself is relatively small. Therefore, moderately larger N helps the critic explore broader range of failure modes, In contrast, the code generation dataset contains over 2,000 problems, and the unit tests used criteria are typically deterministic. As result, smaller N is already sufficient to generate a large number of informative preference pairs.
E QUANTIFICATION OF UNCERTAINTY
Table 4: Factual text generation 8-sentence biographies with the Qwen3-4B backbone, with results shown as mean + standard deviation across three runs.
8-sentence Generation
Method
# Corrt
Incorr]
FSt
Calls]
Qwen3-4B
Baseline
19.03 0.41
12.05 0.19
0.616 0.006
FactTune-FS
21.67 = 0.79
5.66 = 0.48
0.793 # 0.017
402,781 + 52,264
ArmoRM
22.51 # 1.00
9.39 # 0.58
0.705 = 0.013
RLAC (Ours)
21.45 = 0.31
4.37 # 0.48
0.831 = 0.016
70,667 = 20,072
We report variability across random seeds for the factual text generation experiment. For each method, we run training with three different random seeds (affecting data shuffling and sampling of generator candidates) and report the mean and standard deviation of the metrics. Table 4 summarizes the results for the 8-sentence biography setting with Qwen3-4B as the base model. We observe that RLAC consistently improves number of correct biographies and the overall FactScore compared to the baselines, while using substantially fewer verification calls than FactTune-FS The standard deviations are relatively small, indicating that the performance gains are robust across independent
F BACKBONE ROBUSTNESS
RLAC
Factune-FS
ArmoRM
1.00
1.00
1.00
0.90
0.90
0.90
0.85
0.85
0.75
0.75
0.75
0.65
0.65
0.65
100k
Round
Verification Calls
KL Divergence
(a) Training Dynamics
(b) Verification Efficiency
(c) Exploration Behavior
Figure 5: Comparison of training dynamics, verification efficiency, and exploration behavior for RLAC, FacrTune-FS, and ArmoRM on the LLaMA-3.1 1--BBBBBBBBBBBB -8B-IInstrut model with 8-sentence generation.
To examine whether the effectiveness RLAC depends on particular model family, we conducted additional experiments using alternative generator/critic backbone, I.LMMA3.11 1-8t-tnsttruc. The
20

---

Published as a conference paper at ICLR 2026
training setup strictly follows the configuration used in the Qwen experiments. Figure 5 shows the resulting training dynamics. We observe the same qualitative behavior as in the Qwen- experiments: RLAC consistently improves factual correctness during training while requiring fewer verification calls than Factune-FS, consistent with the trend observed in the main paper; moreover, in the high-Ke KL. regime, RLAC achieves substantially greater gains in factual accuracy compared the ArmoRM method.
G MEDICALQA
To demonstrate that RLAC generalizes beyond biographical data, we evaluate our method on MedicalQA dataset. Factual generation in the medical domain presents fundamentally different challenge, featuring specialized terminology and distinct semantic structures compared to Wikipedia biographies.
Since the authors of FaetTune-FS did not release their MedicalQA dataset, we constructed our own by randomly sampling 170 common medical conditions from Wikipedia topies, split into 120 for training and 50 for testing. maintain the same 8-sentence generation setting using the Qwen3-4B backbone.
Table 5: Factual text generation 8-sentence Medical Question with the Qwen3-4B backbone.
8-sentence Generation
Method
Corrt
# Incorr]
FSt
Calls
Qwen3-4B
Baseline
34.0
3.38
0.909
FaciTune-FS
34.2
1.99
0.945
524329
ArmoRM
35.1
2.36
0.937
RLAC (Ours)
35.2
1.67
0.955
76800
As shown in Table 5, RLAC achieves the highest factual accuracy (0.955 FactScore) while requiring 6.8x fewer verification calls than FacrTune-FS (76,800 vs. 524,329). These results exactly mirror the efficiency and performance gains observed in the biography experiments, demonstrating the generality and robustness of our approach regardless of the underlying text domain.
H CROSS-FAMILY
Table 6: Results for HumanEval, MBPP, BigCodeBench Complete and Instruct (BCB-C, C, BCB-I), and LiveCodeBench, Using a different model (GPT-4O) to generate reference solutions, the results show that RLAC still achieves similar level of average performance improvement across all benchmarks.
HumanEval
MBPP
BCB-C
BCB-1
Method
LCB
Average
Base
Plus
Base
Plus
Full
Hard
Full
Hard
Base: Sween.55Coder-7B--i
Baseline
91.5
84.8
82.8
71.4
49.5
19.6
41.8
20.3
34.2
55.1
RLAC (Qwen)
93.3
86.0
83,9
73.0
52.2
24.3
42.3
19.6
35.2
56.6
RLAC (GPT-44)
92.1
85.4
83,9
73.2
52.2
23.6
42.5
20.3
35.5
56.5
QUALITATIVE CASE STUDIES
This section presents additional qualitative examples illustrating the behavior of the critic before and after adversarial training.
I.I FACTUAL TEXT GENERATION CASE STUDY
Generator (umput()
21

---

Published as a conference paper at ICLR 2026
[1] Amina Wadud is prominent Islamic scholar and taeologian based the United States.
[2] She is known for her groundbreaking work in Islamic studies, particularly in reinterpreting Islamic texts for modern, gender- inclusive contexts.
[3] Madud has been a leading voice in the movement toward a more inclusive and progressive understanding of Islamic law and theology.
[4] She is professor at the University of Durham and has held acadenic positions at several other Institutions.
[5] Madud widely recognized fer her advocacy women's religious and social rights within Islam.
[6] She is the first woman to lead a Friday prayer in a major American mosque.
[7] Her research and writings challenge traditional interpretations Islamic 1aw and promote gender equality.
|8] Wadua has been an influential figure in interfaith dialogue and religious pluralism.
Initial Critic Output
reason: Amina Wadud is not preminent Islamic schelar based in the United States.
sentence:
error_ fact: Amina Wadud is a prominent Islamic scholar based in the United States.
Trained Critic Output
reason:
The Friday prayer Amina Wadud led took place in church hall in New York, not in mosque.
sentence:
error_fact Amina Wadud led this Friday prayer in a major American nosque in the United States.
1.2 CODE GENERATION CASE STUDY
Code Question
You are tasked with implement in function that simulated the behavior of a drawing application by managing a canvas. The canvas represented by a 20 grid of pixels, where each pixel can be either empty (0) or filled (1). Your function should handle the following operations: 1) Resize the canvas to new width and height, filling the
with empty pixels. 2) Fill the entire clear the canvas by setting all pixels back to eпpty (0). Implement a single function called 'manage_canvas' that takes the following paramoters: 'canvas' (a list of lists representing the initial canvas), 'operationn (a string indicating the operation: 'resize', 'fill', params (a tuple with parameters for the operation). The function should return the modified canvas after perfozming the specified operation. The function should not access external resources.
with
specified color (1 for filled, for empty)
'clear'), and
Generator output
from typing import List, Tuple, Unien
def manage_canvas
canvas: List [List [int]],
operation: str, params: [intn,nnnon float, bool]]
list (listttttt)]]
22

---

Published as a conference paper at ICLR 2026
"Maaage the based the specified operation and parameters **.
if operation - resize';
parans = (now_width, new_height)
new_width, new_he sew_height params
new_canvas [[0] new_width for in range (new_height) ] return new_canvas
elif operation == 'fill';
params (tili_value,)
fill_value, params
return [[fill_value] len for row in canvas]
elif operation == 'clear': return [[0] len (row) for row in canvas]
clsc:
raise ValucError ("Invalid eperation")
Initial Critic Output
<think>
1. The function supports ' 'resize' operation that changes the canvas to nee_wwidth, new hew_height).
2. Typical boundary cases include an empty canvas and zero or negative dimensions. </think>
<testcase> CALL: manage_canvas ((), 'resize', (2, 2)) </testcase>
Trained Critic Output
</think>
1. The code does not handle the 'resize' operation correctly. It initializes new canvas with the correct dimensions but does not copy the existing canvas's data into the new canvas. This could result in a canvas with only empty pixels after resizing,
2. This behavior contradicts the problem description, where 'resize' is expected to preserve existing pixels and only fill newly created areas with 0, as in normal drawing application.
</think>
<testcase> CALL: manage_canvas 11, [1, 111, fresize', (3, 3)) </ testcase>
J EXTENDED RELATED WORK
Outcome-reward RL for reasoning. RL for LLM has been shown to significantly boost model performance domains where the success the final answer can be casily checked (OpenAI et al., 2024; Liang et al., 2025; Team et al., 2025; Lambert al., 2025). This mostly includes the domains of math (Cobbe et al., 2021; ; Cui et al., 2025; Luo et al., 2025b; Yu et al., 2025), coding (Jimenez al., 2024; Pan al., 2024a: Wei al., 2025; Luo al., 2025a) but can be tricky for other domains like agent decision-making (Pan et al., 2024b; Zhai et al., 2024; Bai et al., 2024) and free-form generations (Min al., 2023; Zhuge et al., 2024). However, RLAC is designed relax this requirement so that we can apply RL to more general domains where success cannot be easily verified, such as free-form generations.
23


Reviews:
[{"review_id": "Zi8kS9vCWK", "reviewer": "Reviewer_G4q7", "summary": "This paper tackles a real pain point in training large language models: how do you optimize for tasks where outputs need to satisfy tons of different criteria, and you can't possibly check them all? The authors propose RLDCF, which basically turns training into a game between two models. A generator tries to produce good outputs, while a critic tries to catch the generator's mistakes by proposing specific ways it might fail. An external validator then checks if the critic found a real error. The critic gets better at spotting weaknesses, and the generator gets better at avoiding them. The idea is neat: instead of exhaustively checking every possible criterion or relying on a static reward model that can be gamed, you dynamically focus on the most likely failure points. \n\nThe paper suggesr promising results. For biography generation, they hit a FactScore of 0.889 while doing 5.7 times less verification work than existing methods. For code generation, they claim the best scores despite using only 9% of the training data. However, a potential problem in code generation experiments suggest circular logic: they essentially created \"reference solutions\" using the same model family they're training. Then the model could just be learning what Qwen is already able to do. Qwen-7B-Instruct training shows limited improvement over the base model, could be well within variance, and the authors didn't provide much details on what or how the model is improved.  \n\nThe core idea of adversarial training idea isn't particularly novel, similar approaches have appeared in recent work on generative verifiers. The biography experiments are more solid and the overall problem they're solving matters. But between the circular validation issues, limited novelty, and some unfair experimental comparisons, this feels like a decent idea that needs another round of work with more solid experiments and applications that can justify sufficient contribution to the area.", "questions": "I'm open to change my score if authors can provide sufficient justification or insights in both of the experiments, especially the coding one", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "- Important and natural problems to tackle for rubric based reward modeling and RL training for LLM post training. \n- Theoretical formulation is solid. \n- Strong factual text generation results.\n- Good ablation studies. \n- 4 to 8 sentence comparison shows the method scales well to complexity \n- Presentation is clear", "weaknesses": "- I have doubts about code experiment set up as mentioned above\n- Lack of theoretical or empirical analysis into the method and experiment results \n- Could benefit from more analysis & learnings and more experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TSPRMbwzHJ", "reviewer": "Reviewer_DxZj", "summary": "The paper tackles the important issue of the most likely failure case of generated free-form text by introducing a novel adversarial learning scheme for training a generator and critic. The generator outputs multiple candidate solutions and the critic, given a candidate, finds the case to most likely fail given the generation. They employ a DPO loss for both the generator and critic. They have consistent results on factual text generation and code generation showing improved performance over baseline methods.", "questions": "See Weakness 1 and 2\n\nMinor Suggestions/Weaknesses:\n-L86 has two empty parentheses\n-Missing reference in L248", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "-The paper is well-written and easy to follow\n\n-Paper outperforms baselines methods in both factual text generation and code generation\n\n-The proposed approach is novel, applying adversarial learning to the important, significant problem of identifying failure cases in free-form generation\n\n-I find the use of DPO fairly interesting and novel", "weaknesses": "-For fact verification, I do not necessarily see the need for a critic to specify which fact the check. Can one not separate out all the facts in the generated responses, either programmatically or with an LLM, and then run each one through the fact verifier? I understand from later on in the paper the verification is costly, but verifying each fact would provide more accurate rewards for the generator, correct? For code generation, I understand this simplification is not possible because the test cases cannot be parsed from the generated code.\n\n-If I understand correctly, given a candidate answer a_i to s, the reward that is used to fine-tune the generator is sparse. Therefore, if even one fact is incorrect or one test case is not correct, the whole candidate output is assigned a reward of 0, correct? If so, in the case of code generation, that may be too strict because realistically code is regularly updated to handle unseen test cases, not marked as entirely wrong. Also for fact generation, if one fact is wrong, the sparse reward does not help the generator learn which fact was wrong. I understand the issue may be mitigated by the sampling multiple candidates and using DPO loss, so similar candidate solutions with different rewards can help the generator learn with finer-grained feedback. However, how many times, especially in the beginning of training, do you get a group of candidates with reward = 0 and reward = 1 to provide some distinction to the generator?\n\n-I’m not too familiar with FactScore, so I am not sure what is the bottleneck cost for verification mentioned in L249. But for code generation, I do not see why number of test cases executed is a good metric for efficiency analysis. Test cases, especially for the ones provided in the used datasets, can be quickly executed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gDypYPz4pt", "reviewer": "Reviewer_7h2p", "summary": "Post-training methods for large language models (LLMs) constitute an active area of research. However, reinforcement learning (RL)-based fine tuning is very challenging due to the generative nature of LLM outputs. In particular, the design of efficient reward functions is difficult. \n\nTo address this issue, the authors introduce Reinforcement Learning from Dynamic Critic Feedback (RLDCF), a framework that focuses on post-training as an adversarial game between a generator and a critic. In RLDCF, each instruction/prompt provided to the LLM is associated with a set of rubrics representing task-specific requirements that the output should satisfy.  The objective is then to train a generator that maximizes the probability of providing correct outputs.  \n\nThe critic, modeled as a stochastic policy, aims at providing the worst-case criterion for a given instruction–action pair. The generator is then trained by solving a mini–max optimization problem. Both the generator and critic are implemented as LLMs fine-tuned using a DPO loss.\n\nThe proposed approach is evaluated using text and code generation tasks. For factual text generation, experiments are conducted on the Wikipedia Biography Dataset using base generators Qwen3-4B and Qwen3-8B, compared against three baseline methods. For code generation, the authors employ the AceCode-87K-hard subset, with base generators Qwen2.5-Coder-7B-Base and Qwen2.5-Coder-7B-Instruct, also benchmarked against three baselines.", "questions": "In both experiments N and K are fixed. However, these parameters should have a great impact on the results. Can you discuss this ?\nThese hyperparameters are different in both experiments. Is there a reason for this ?\n\nIn the experiments, can you add some qunatification of uncertainty (std over various runs for instance) ?\n\nIs it possible to highlight the performance of RLDCF on other text generationt tasks to support the applicability of the method ? For instance using datasets used in papers associated with the baselines (medical question answering of [Tian et al., 2024] for instance).\n\nDoes changing the backbone model for the generator and critic have an influence on the experimental conclusions (factuality level/number of calls, dynamics of the KL divergence along epochs) ?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Post-training methods for large language models is a crucial task to improve task specific use of generative models and provide robust LLMs.\n\nThe presentation of the paper is clear, the problem is well motivated and the overall description of the method is good.\n\nExperimental results demonstrate that RLDCF yields competitive results in both text and code generation quality, highlighting the effectiveness of adversarial critic feedback to finetune.\n- In the text generation experiment, RLDCF achieves the same level of factuality as FacTune-FS with fewer verification calls, it also improves KL divergence along epochs with monotonic FactScore gains.\n- In the code generation task, the proposed approach outperforms enumerative method  and static reward model method for most benchmarks.", "weaknesses": "The results are interesting and promising on the two proposed tasks. However, as the paper is mostly experimental, I would expect more discussion on the choice of the methodological choices. For instance on the way the critic and generator are updated. The influence of K (candidate outputs for each instruction) or N (number of criteria sampled from the critic) should be strong on the results. Even if no theoretical guarantees are provided (which is a probably a very hard question), I would expect more discussions on these \"hyperparameters\".\n\nIn the experiments, it is hard to assess the statistical significance of the results are there is no uncertainty quantification (standard deviation of the metrics for independent runs for instance).\n\nAlthough adding additional experiments or simulations is not strictly necessary to demonstrate the soundness of RLDCF, the contribution being primarily methodological, its impact would be strengthened if the empirical evaluation, especially for the factual text generation task, were illustrated with a broader variety of performance results.\n\nMinor weakness: the paper would benefit from a careful proofreading to remove typos (Appendix and Tables wrongly referenced for instance).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)