Reward-Guided Prompt Evolving in Reinforcement Learning for LLMs
Authors: Ziyu Ye
Rishabh Agarwal
Tianqi Liu
Rishabh Joshi
Sarmishta Velury
Quoc V Le
Qijun Tan
Yuan Liu
Venue: ICML 2025 poster (2025)
Keywords: large language models
Primary area: deep_learning->large_language_models
TL;DR: We introduce a new RLHF training method where language models can adaptively prioritize and create useful prompts during training guided by reward signals.

Abstract: Existing reinforcement learning (RL) methods for large language models (LLMs) rely on static prompt sets, where prompts are curated a priori, and sampled in a fixed schedule for training, regardless of their usefulness to the RL process. We design `eva`, the first method that allows LLMs to prioritize and adaptively create useful prompts during RL training by reward signals. In principle, `eva` (Evolving via A symmetric Self-Play) casts language model training as a game between: (1) a creator, who samples and generates training prompts, and (2) a solver, who generates responses to the prompts. `eva` is simple, suits both offline and online RL for LLMs, and sets a new state-of-the-art on challenging benchmarks without extra human prompts: it improves gemma-2-9b-it’s win-rate on Arena-Hard from 51.6% to 60.1% by DPO and 52.6% to 62.4% by RLOO, surpassing claude-3-opus and nearing gemini-1.5-pro, both are orders of magnitude larger. Further ablation studies show `eva` can induce meaningful learning curriculum, and effectively scale RL for LLMs beyond static human prompts.

Reward-Guided Prompt Evolving in Reinforcement Learning for LLMs
Ziyu !e! Rishabh Agarwal' Tianqi Liu' Rishabh Joshi' Sarmishta Velury'
Quoe V. Le' Qijun Tan' Yuan Liu'
Abstract
The Prioritized Prompt Evolving Principle
Existing reinforcement learning (RL) methods for large language models (LLMs) rely on static prompt sets, where prompts are curated a priori, and sampled in fixed schedule for training, re- gardless of their usefulness to the RL process.
LLMs shall adaptively control training prompts by pri- oritizing and generating useful prompts during RL.
evolve prompts
We design eva, the first method that allows LLMs prioritize and adaptively create use- ful prompts during RL training by reward signals.
creator 2x
solver mii
propose responses
In principle, eva (Evolving via Asymmetric Self- Play) casts language model training as a game between: (1) a creator, who samples and gen- erates training prompts, and (2) solver, who generates responses the prompts.
Figure I: RL for LLMs as Asymmetric Self-Play. Conven- tional training methods are restricted to pre-curated prompt sets with fixed training schedules. We show that is crucial to adaptively control the prompt distribution by creating and prioritizing more useful prompts. This is implemented through a simple self-play scheme ( (Algorithm 1), where the creator is the newly introduced prompt generation policy Tx and the solver is the response generation policy Ty|x; both interact with objectives driven by reward signals (6 3).
eva is simple, suits both offline and online RL for LLMs, and sets a new state-of of-the-art on challeng- ing benchmarks without extra human prompts: it improves gemma-2- 2-9b-it's 's win-rate on Arena- Hard from 51.6% 60.1% 1% by DPO and 52.6% to 62.4% by RLOO, surpassing claude-3-opus and nearing gemini-1.. 5-pro, both orders of magnitude larger. Further ablation studies show eva can induce meaningful learning curriculum. and effectively scale RL for LLMs beyond static human prompts.
in enhancing L.LM capabilities (Team al., 2023; 2024a; 2025). However, there is fundamental issue in existing practices: they restrict themselves within pre-curated, static prompt distribution during post-training.
This is sub-optimal and bottlenecks scaling properties w.r.t: (i) training efficiency: the existing paradigm treats all prompts equally, despite their utility depending on the chang- ing states of LLMs in training; as not all prompts contribute equally, relying on a static set with fixed training schedule is inefficient. (ii) model generalizability: once the L.M saturates on the static prompt set, learning stops, preventing the model acquiring new skills or knowledge beyond the predefined distribution.'
1. Introduction
Long-lived artificial intelligence must deal with ever- evolving, open-ended world, however the current training paradigm is restricted to being fairly short-lived and static.
This paper concerns the post-training paradigm of large language models (LLMs), which is typically done in two stages, imitation (i.e., supervised fine-tuning, SFT) and re- inforcement learning (i.e., RL post- traininga. In particular, focus the latter, which has led to remarkable success
We thereby investigate the two questions:
(Signal) Which prompts should prioritized during RL training?
'Google DeepMind University of Chicago. Correspondence to: Ziyu Ye and Yuan Liu <hhazelye, yuanliu @ googll..com comm>. Approved for publication by Google DeepMind as 2024-09-27.
(Algorithm) How can we generate more useful prompts, and use them keep LLMs self-im infrrving in RL?
Proceedings of the nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
"Lteerauure has considered prompt evolving, but is restricted to SFT (Xu al., 2023), and/or evolves uniformly (Yuan et al., 2024). We claim adaptive training w.z.. reward signals are crucial.

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Classical RLHF
The Creator Step
Xo
soleer
ESTIMATE X1
LLM
(1) Di
RM
info(x)) := =u sppraciiee
Open-Ended RLHF: eva
SAMPLE & EVOLVE
weighted sampling
proximal evolving by evol(x))
Xo
cresstor
X1
creatur
X2
by info(s)
Xt = [x]
xtm subset (x)))
Xt+1 evolved (())
0
0
0
Figure 2: (Left) eva generalizes classical RL with open-ended RL for LLMs, by introducing creator policy in training. (Right) The creator strategically controls prompt distributions by generating new prompts after each mini-batch gradient update (for online RL) or a complete dataset iteration (for offline RL), with a simple estimate, sample then evolve procedure, where the usefulness of each prompt is estimated from reward signals. The solver is then trained on mixture of the original and evolved prompts. See the minimax-regret objective driving the game design and practical implementations in & 3.
To address them, we design eva (Evolving via Asymmetric Self-Play), as in Figure 1. 2. Central to eva is an infinite game with minimax-regres objectives, achieved by alternat- ing optimization in creating prompts and solving them. We detail the novelty of this work w.t.t. prior works in $ 5, and summarize our original contributions as:
2. Preliminaries
Classical RL post-training (Ouyang al., 2022) solves reg- ularized optimization for static prompt dataset D;
r(x,y)
max To
- EXWD 3 DKL (- |x)
Tbase(: x]]]
1. Signal: We find reward advantage as an effective signal to identify useful prompts in RL post-training,
Here, Tasse (-1x) is a base policy, x and y are prompts and responses, DD a divergence measure. To approximate the expectation over y, we employ Monte Carlo sampling by generating k responses per prompt, -(1):-.., ((k)) To(-
2. Algorithm: We design eva, the first method that ables LLMs to prioritize and adaptively create useful prompts, for continual RI. training and self-improving.
x) (Sutton et al., 1998; Ahmadian et al., 2024). In this paper, r(-) is the reward assumed to be from an oracle r*(+) and is fixed during post-training (Team et al., 2024a); we use the conventional term RLHF (reinforcement learning from human feedback) interchangeably with RL (post-)training and focus on human preference optimization. i.e., Al align- ment (Leike al., 2018), in our experiments. Nevertheless, the pipeline is compatible with any other reward types.
The rest of technical details are organized as follows: in Problem 1, we formalize the problem of RL post-training as bilevel optimization over creator policy and solver policy, which corresponds to the Stackelberg game (von Stackelberg, 1934) in game theory. In 3.2, we define the objectives for the two players of the game by regret, for which the solver minimizes and the creator maximizes, and use its approximation by reward advantage for creator opti- mization. In $ 3.3, discuss simple practical algorithms of eva in both online and offline settings. In S 4, we run extensive experiments showing eva universally improves the performance of both online RL (e.g., RLOO, OAIF) and offline RI. (e.g., DPO, SPPO, SimPO, ORPO) for L.LMs and is SOTA on various challenging real-world benchmarks.
Conventionally, prompts are scheduled in training either sequentially or via i.i.d. uniform sampling. Prior works explored active selection (Kawaguchi & Lu, 2020; Muldrew et al., 2024) or prioritized sampling (Schaul al., 2015; 5; Lee et al., 2022), but focus solely on existing data. In contrast, eva introduces new prompt creation, enabling training with improved coverage and complexity beyond the initial static prompt set. Depending on how y is generated and labeled, RL post-training methods can be described as: (i) online on- policy, where y are generated on on-policy (Team et al., 2024a); and (ii) offline, where y are pre-generated by existing model checkpoints (Xiong et al., 2024; Pang et al., 2024a). In this paper, for online RLHF, we evolve prompts at each mini- batch, while for offline RLHF, we construct new prompt set after each full iteration over the previous set.
As we enter the new epoch where compute is moving from training data synthesis, the question on "how to put more compute in generating better data" (Nishihara, 2025) is more critical than ever. While existing works (Zelikman et al., 2022; Gulcehre et al., 2023; Chow et al., 2024) pri- marily focus on the exploration in y X, we perform the first systematic study in exploration in (X,Y) ))ii in RL for LLMs, guided by carefully designed reward signals.
2

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Problem (Open-Ended RLHF) We define the problem of Open-Ended RLHF as the bilevel optimization on both a prompt policy (the creator (x(x)) and a response policy (the solver roly | x)):
Φ' E arg max R (T)))), Ttrue ();
(1)
s.t. 0T (Ø) max E Ex) By r(xx,)] B. BKL TO(- | x) || Taase (- xx]
(2)
0
where Trree is the (potentially unknown) true target prompt distribution, 'D is an optional artifact parameter (e.g. the seed prompt distribution), and R(-) is a regularization function on the creator policy Rg, which we discuss in detail at 33. The problem generalizes classical RLHF, and captures the dual objective that (i) response alignment: the solver should perform well on the training prompt distribution while staying close Thase- and (ii) prompt generation: the creator should generate training prompts allowing the solver perform robustly on targel prompt distributions.
3. Method
when Ttrue is unknown a priori; the optimization then falls into a standard decision under ignorance problem (Savage, 1951; Gustafsson, 2022). Several decision rules can be con- sidered, e-g., randomization, that chooses training prompt distribution uniformly (Jiang, 2023); in this paper, we study Minimax Regret Rule, which finds training distribution that minimizes solver's worst-case regret over all possible distributions (see also &S 4.2.1). We abuse the notation r to denote the reward with KL penalty, and Regret is defined as reward differences of Tg and optimal policy mz:
3.1. The Problem: Open-Ended RLHF
Classical RLHF, as in $ 2, samples prompts from static set D, which can have limited prompt coverage and com- plexity (Fig. 11), and may diverge from true scenarios in the open-ended world (Dennis et al., 2020; Parker-Holdes et al., 2022). In Problem 1, we introduce a prompt genera- tion policy (-) to be optimized together with the response generation policy To r-(+).). An optimizable (-) brings few benefits: (i) during training, it allows dynamic adjust- ment of training prompts on- on-the-fly, making it possible to prioritize prompts that are more informative to the current (e)) improving learning efficiency; (ii) at convergence, brings a new prompt distribution beyond the initial static set, making possible for To (- learn knowledge be- yond DD and perform more robustly on the target distribution.
Regret To) Ex~TO(-) [yyaa(ix) r(x,y)
-EE [r(x,y)]]
Problem is then formulated as:
Φ² arg max Rngret(TT.. me.); (3)
db
The creator objective R(-) characterizes the optimization of (-). preventing it from collapsing in trivial prompts and guiding it towards true target prompts, which we discuss a specific implementation by regret maximization in the next.
(x)) E arg min Regret(TT: TO)-
(4)
Note that (i) for solver optimization, Eq. 4 is equivalent to Eq. 2 by definition, and (ii) for creator optimization, Eq. approximates Eq. with worst-case optimal guarantee for the solver's s policy, when Tere is unknown.
3.2. The Game: Minimax Regret Games
Problem can be cast as a sequential game (von Stackelberg, 1934) by two strategic players optimizing each's 's utility:
Remark 1 Under mild assumptions, the (local) Nash equi- librium is a (local) minimax point (Jin et al., 2020) for the above optimization; here, at the (local) Nash equilibrium, the solver follows a (local) minimax regret policy (Jiang, 2023), i.e., the solver's 's regret is worst-case optimal.
Solver (y x). who generates responses that opti-
mize alignment given training prompts.
Creator no (x), who generates training prompts for the solver to perform well in the real world, knowing the solver will optimize over the generated.
The equilibrium finding of the game can be solved by al- ternating optimization (Zhang et al., 2022). Intuitively, this allows for the creation of evolving prompt distributions that challenge the agent progressively for better generalization; the regret objective ensures robustness such evolving cur- ricula by incentivizing agents to perform well in all cases, providing a worst-case guarantec. In optimization, this brings a sweet spot where the creator can create challenging yet solvable prompts (i.e., neither too hard nor too easy) for the solver, as illustrated in & H.3.
natural objective for the creator is to improve the solver's transfer performance (Bengio, 2012) the true target prompt distribution Ttrue: the closer Te is to Ttrues the bet- ter the solver performance is expected, thereby the higher the utility the creator may receive. If Ttrue is known, R(-) can be instantiated by a f-divergence measure for distri- bution matching with Trree- This work considers the case
3

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Regret Minimization for the Solver. Any preference op- timization algorithms can be used a plug-in for the regret minimization for the solver's step in Algorithm 1.
Algorithm 1 eva.
Input: initial policy Too initial set of prompt No 1: for iteration t= 1, 1,2,...du do
Regret Maximization for the Creator. When it is direct for the solver to minimize the regret by policy optimization, the true optimal policy remains unknown during optimiza- tion, and we must approximate it when using it as the utility to incentivize the creator. Similar to heuristics in prior works (Jiang et al., 2021:;aa Parker-Holder et al., 2022), we use the advantage-based estimate for each x:
creator step") "
2:
estimate:
X <- {(.. info(x.)) a E 21-1)
sample:
X t- { drawn w.p. o info(æ;)) X, s evolve(2?
evolve:
solver step "/
3: generate:
V a; E X, {V) (0) To, To. (1 mx))
Regret(x, To) <- r+x, - r(x, y )
(5)
annotate reward:
X¹= + x!nlo U {(v ,1 ((
optimization:
O, < 0,- + n voJxx
where
4: end for
y+ == arg maxy, y):
5: return final solver policy nør
Ybaseline == avgy, r(x,y) y) arg miny, r(x, y),
we sample responses and annotate rewards, then take the responses with the maximal and the minimal reward to construct preference pairs and optimize upon.
and {yrk=1 is set of responses sampled from (e(- | x) and r(:,-) is the reward oracle. We choose arg avgy, r(x. y) for online RLHF, and arg miny, r(x, y) for offline RLHF, based on consistent strong empirical gains observed across extensive experiments. As the policy optimizes, the proxy will approximate the true regret better". We denote the regret estimate (i.e., negated reward advantage) as the infor- mativeness value for prompt X w.r.f. 0,
3.3.2. THE CREATOR STEP
Plainly, the creator finds most useful prompts and generate variants of them to approximate regret maximization.
Step 1: info(-) estimate the informativeness. For each
x in the prompt set X,, we generate responses, annotate rewards and estimate the informativeness of x by Eq. 6.
infoo (x) == Regret mo).
(6)
Directly doing gradient ascent regret could lead to train- ing instability (Zhang, 2023). In this work, we approximate new prompi distributions that maximize regret by 3 steps:
Step 2: sample(.) weighted sampling for an informative subset. By using the informativeness metric as the weight, we sample an informative subset 2:info to be evolved.
Step 3: evolve(-) evolving for high-regret prompts. eva is agnostic to and does not rely on any specific evolving method (see empirical evidence in & H.1). We take Xu et al. (2023) as a default baseline for offline RLHF, with in in-depth and in-breadth instructions for prompt re-writing.
1. Estimate informativeness for each prompt in the set,
2. Sampling a subset of high-regret prompts.
3. Generating new prompts by making variations on those high-regret prompts.
As a side note, we discuss a useful technique below.
The eva way of scalable regret maximization can relate to curriculum RL Paarer-HHolde et al., 2022), which finds environments with high-regret levels, then edits within some distance, or evolution strategies (Schwefel, 1977) which find the most promising species, then mutate and crossover.
Prioritized Generative Buffer. While eva can operate on the full D at once and iteratively train LLMs (i.e,, offline eva), the informativeness can become off-policy. Inspired by Schaul et al. (2015), we design a simple prioritized gen- erative buffer B that extends Algorithm to be on-policy and evolves per mini-batch (i.e., online eva), with:
3.3. The Practical Algorithm
1. Warm-up phase: we start with from D and populate B with evolved prompts until it reaches size B.
Algo is an overview of eva where the creator constructs new training prompts after a full iteration of the solver.
2. Mix-up phase: training continues using a balanced mix of samples from D and B (prioritized by informa- tiveness) per mini-batch. New prompts are evolved and added to B, while older ones are removed.
3.3.1. 1. THE SOLVER STEP
This step is the classical preference optimization (Rafailov et al., 2023). Take DPO as an example, for every prompt,
3. Bootstrap phase: Once D is exhausted, training relies on B, continuing evolving and replacing prompts.
This approximates the expectation over Tă by best observed re- sponses, which introduces bias quantifiable by DKL (To ")" Also, single-sample estimation for To introduces variance. See Liu et al. (2023a) for discussions on sampling effects.
eva is easy to use and flexible to extend. We provide de- tailed instructions for practitioners in the Appendix.
4

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
4. Experiments
MT-Bench
Arena-Hard
AlpacaRial
60.1
55.53
Datasets and models. We use UltraFeedback (Cui et al., 2023) as the training dataset, which contains diverse high- quality prompts that are primarily human-generated. We use the mnstruction-iietuueed GEMMA-2-99 (Team al.,
Rating (0-10)
Winn rate Q8)
Win rater (%)
51.6
8.66
8.57
41.3
38.39
2024b) as the base (0o)', which is a strong baseline for
SF7
sh 3/0 300ee
models of its size. Note that we directly apply RL training without SFT, as the base model is sufficiently capable.
Figure 3: Illustration of gains with one round eva by DPO.
Table I: Online eva results. eva has notable gains and is comparable to default training with even 6x human prompts (gray). Note eva only uses 1x human prompts and contin- uously evolves (nx denotes total prompt size).
Evaluation settings. We use: (i) AlaacaEval 2.0 (Dubois et al., 2024), which assesses general instruction following with 805 questions; (ii) MT-Bench (Zheng et al., 2023), which evaluates multi-turn instruction following with 80 hard questions in 8 categories; (iii) Ward (Li al., 2024), which is derived from 200K user queries Chatbot Arena with 500 challenging prompts across 250 topics.
Optimization Method (+)
Online RLHF
Benchmark (-+)
Arena-Hard
MI-Bench
AE
Method (3) /Metric
WR
avg.
turn
turn 2
L.C-WR
0y: Base Model
41.3
8,57
8.81
8.32
47.11
Optimization algorithms. We evaluate our method across a wide range of six representative RLHF algorithms:
o RLOO (1x)
52.6
8.68
9.02
8.34
54.23
8 RLOD-eva (1x)
57.3
8.87
9.03
8.71
55.02
i RLO0-eva (2x)
60.5
8.96
8.80
57.10
0 RLOO-eva (3x)
62.4
9.09
9.23
8.94
61.04
Online RLHF: RLOO (Ahmadian et al., 2024), OAIF (i.e., online DPO) (Guo et al., 2024).
0g 11: RLOO (6x)
62.7
9.07
9.24
8.90
62,91
Bu-i1: OAIF (1x)
52.1
8.66
8.97
8.35
55.15
8 OAIF-eva (1x)
55.0
8.85
9.04
8.66
55.43
Offline RLHF: (with reference) DPO (Rafailov et al., 2023), SPPO (Wu al., 2024); (without reference) SimPO (Meng et al., 2024), ORPO (Hong et al., 2024).
0 11 OAIF-eva (2x)
60.4
8.93
9.06
8.79
56.49
0- OAIF-ava (3x)
61.7
9.01
9.19
8.82
59.09
Table 2: Offline eva results. We apply eva after iteration of offline RLHF It brings strong gains and can surpass training with human prompts. See more iterations in S 4.2.4.
Oracle reward models. We take ARMORM-SB (Wang al., 2024) be the default reward model for human- preference proxy, with the below for ablation studies:
Optimization Method (-+)
Oflline RLHF
Pointwise: ARMORM-8B (Wang et al., 2024), SKYWORKRM-27B (Liu & Zeng, 2024).
Benehmark (-)
Arena- Heena-Hard
MT-Bench
AE 2.0
Method (L) /etric (-)
WR
avg.
turn
LC-WR
Og: Base Model
41.3
8.57
8.81
8.32
47.11
Pairwise:
PARRRM-0.4B (Jiang et al., 2023), PAIRRM-8B (Dong et al., 2024).
0 DPO
51.6
8.66
9.01
8.32
55.01
6 +eva
60.1
8.90
8.75
55.35
0 new prompts
59.8
8.64
8.88
8.39
55.74
0 SPPO
55.7
8.62
9:03
8.21
51.58
4.1. Main Results
58.9
8.78
9.11
8.45
51.86
6 new www.hhumannnnnnn prompts
57.7
8.64
8.90
8.39
51.78
eva consistenly achieves strong self-improo eell-improveeet. As
0 SimPO
52.3
8.69
9:03
8.35
$4.29
in in Table and 2, eva yields notable performance improve-
60.7
8.92
9.08
8.77
55,85
6 81-21 new huma n prompts
54.6
8.76
9.00
8.52
$4.40
ment across different optimization algorithms, especially on the more challenging and robust Arena-Hard benchmark (Li et al., 2024). For example, eva brings 10.6% gain with DPO in the offline setting, and 9.8% gain with RLOO in the online 3-odee---puss2240000
6 01 ORPO
54.8
9.04
60.3
8.89
9.07
8.71
54,39
01 .2 new human n prompts
57.2
8.74
9.01
8.47
$4.00
setting, surpassing by AH leaderboard and matching gemini-1. 5-pro, while using fully adaptive self-automated joint prompt-response generation. This demonstrates the superior empirical perfor- mance of eva.
as reported
eva curricula can surpass human-crafted prompts. We further show that eva models can match and even outper- form those trained on additional new prompts from Ultra- Feedback (denoted as new human prompts as they are pri- marily sourced from humans (Cui et al., 2023)), while being much more efficient. Interestingly, on MT-Bench, training with new human prompts typically show decreased perfor- mance in the 1st turn and only modest gains in the 2nd turn, whereas eva notably enhances 2nd gains. We hypothesize that eva adaptively evolves novel, learnable prompts that include features of second-turn questions, reflecting gener- alized skills like handling follow-up interactions.
*Unless stated otherwise, each iteration uses 10K prompts (the initial prompt set), referred to as 1x. In offline RLHE, denote 0:-1e= as the one trained with new human prompts from the t-th chockpoint. i_.. denotes the one trained with evolved prompts from the t-th checkpoint without any new human prompts. In online RLHF, training is continual iteration and 0p- (лx) denotes training with 10nK prompts in total, mixed and evolved from the initial.
5

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
4.2. Ablation Studies
Comparing with the inverse advantage (purple): Con- trary to curriculum learning, a line of works conjec- ture that examples with higher losses may be priori- tized (Jiang et al., 2019), which can be done by invert- ing our metric. We find it significantly hurt the align- ment gain, corroborating (Mindermann et al., 2022) that those examples can be unlearnable or irrelevant, meaning our curriculum is effective and practical.
Taking offline DPO as a representative case, we conduct extensive ablation studies on eva, with key findings:
4.2.1 informativeness metric: our regret-based metric outperforms other alternatives.
4.2.2 adaptive evolving procedure: our method outperforms active selection without evolving.
$ 4.2.3 scaling with reward models: the alignment gain of eva scales with reward models.
Among our advantage variants (green): We designed variants of our default advantage-based metric, as in Table 3; the default A remains competitive among its peers. Together, the advantage- based principle pro- vides a robust guideline for sampling and evolving.
8 4.2.4- - continual training: our method has mono- tonic gain with incremental training.
& 4.2.5 curriculum effect: our method creates mean- ingful curriculum over iterations.
The lesson that we must be selective about which are the promising to evolve, otherwise unlearnable, noisy naive prompts may hinder learning. Our regret-inspired metric represents solid baseline.
4.2.1. THE CHOICE OF INFORMATIVNEESS METRICS
Mutric
(nfo(s)
Related Approsimation
4.2.2. THE EFFECT OF EVOLVING
Aaa wonst case optinall advastage
min,, r) =xx.y-(Xxy)) ((x,y))
minimax regret
Ar, average optinall advantnge Ale dueling optimal idvanage
+ F.y n/x y) mayy rix,
Boyenian regret
mux (xxyy ma0y. (XX.3")
min margin regret socoe
Benchmark (-))
Arom-Hard
MT-Bench
AlpacaEval 2.0
Table 3: The reward reward-advantage advantage- based metrics that serve as the informativeness proxies for prompts.
Method (3) Metric (-)
(S)
agg.
bum
turn
LC-WR
(R)
6: DPD
$1.6
8.66
9:01
8.32
R.. Ino eso tee poolvel--reer -greedy
56.1
8.08
8.34
$6.11
53.66
0. [3o coolve--sarpee
55.1
8.69
9:00
8.38
54.22
54.16
Benchmark (-))
Arena Hard
MT-Bench
AE 2.0
1vee wwe-gyeody
59.5
8.72
9.06
6.35
34.52
55.22
Method (L)/ Metrie (-))
WR
avg.
turn
LC-WR...
eVa samole
60.1
8.90
9.04
8.75
55.35
53.53
0...1: DPO
51.6
8.66
9.01
8.32
55.00
0 0:
(und
57.5
8.71
9.02
8.40
53.43
Table 5: Effect of evolving. The blue are those training with only the informative subset and without evolving); we de- note -sample for the default weighted sampling procedure in Algo 1, while using -greedy for the variant from the clas- sical active data selection procedure (cf., a recent work (Mul- drew et al., 2024) and a pre-L..M work (Kawaguchi & Lu, 2020)), which selects data by a high-to-low ranking via the metric greedily. We show evolving brings remarkable aligament gain (green v.s. blue); and as we evolve, sampling is more robust than being greedy.
(vartr)) )
$4.8
8.66
8:20
54.58
eva (avafr))
58.5
8.76
9.13
8.40
55.01
0
(1/wvg(H))
56.7
8.79
9.13
8.45
55.04
0
eva (1/44)))
$2.3
8.64
eva (A2gg) nown!
60.0
8.85
9.08
8.61
56,01
(A5g)
60.0
8.86
9.18
8.52
55.96
(A
60.1
8.90
Table 4: Choice of informativeness metric matters. Our adaptive metric by reward advantage achieves the best per- formances. See also ğ for visualization.
Reward advantage as the informativeness metric out- performs baselines. As in Table 4, eva offers an effective curriculum by the advantage-bal based proxy as the informa- tiveness metric (bottom row):
The design of evolve ( is effective. As in Table 5:
Removing the evolve(-) step: if we only do subset sampling ordered selection, we still have gain, but not as much as with evolving (e.g.. eva brings 4.8% additional wins on Arena Hard).
Comparing with uniform evolving (brown): Existing baselines generate prompts in a uniform manner (Yuan et al., 2024) (cf., the principle of insufficient rea- son (Keynes, 1921; Tobin et al., 2017)). eva concretely outperforms, corroborating (Das et al., 2024) that uni- form learners can suffer from sub-optimality gaps.
Altering the sample(-) step: if we greedily select prompts by the metric instead of using them as weights for importance sampling, the performance will be weaker as we evolve.
The lesson is that simply adaptive training within fixed prompt distribution is not enough; our open-ended RI.HF with generative prompt exploration gives a substantial head- room for eeff-improvement In other words, the RL post- training process should be both adaptive and generative in terms of prompt distribution.
Comparing with other heuristics (gray): Prior prac- tices (Team et al., 2023) tried heuristics like prioritiz- ing prompts with the most variance in its rewards or with the lowest/highest average. We find our advantage based methods (red) outperforms those heuristics.
6

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
4.2.3. SCALING EVA WITH REWARD MODELS
Table 6: Ablation on incremental v.s. scratch training.
Benchmark (-+)
Arena-Hard
MT-Bench
AE 2.0
LC Wia Wia zate (%)
SkyworkRM-277
ArmoRM-B8 56.
PairRM 4B
PairRM-0.4B
52.71
Method (1) (-t
WR (%)
avg. score
LC-WR (%)
55.35
LC Win sale
0y: SFT
41.3
8.57
47.11
55.01
51.86
0a-1: DPO
51.6
8.66
55.01
51,58
0-- eva (scratch)
59.8
8.88
54.59
1 eva (incremental)
60.1
8.90
$5.35
DPO DPO-eva DPO-eva
SPİO SPrO-evaSPrö-eva
Figure eva scales with quality reward models, under pointwise RMs with DPO (left) and pairwise RMs with SPPO (right). Note SPPO handles general preferences thus requires pairwise RMs, and DPO relies the Bradley-Teie Terry assumption, for which pointwise RMs are suitable.
4.2.5 CREATES MEANINGFUL CURRICULUM
Table 7: eva improves prompt quality and complexity.
Prompt Set (L) Metric (-)
Complexity (1-5)
Quality (1-5)
UltraFeedback (seed)
2.90
3.18
Figure 4 presents the length-controlled win rate of eva on AlpacaEval using pointwise and pairwise reward models of varying scales. As the quality of reward models improve, eva brings higher alignment gain. The scaling observation shows the effectiveness of eva in exploiting more accurate reward signals to choose informative prompts for better aligmment. One takeaway interaction with the external world essential for intelligence. The more accurate reward signals observed, the better the agent incentivize themself to improve (cf., (Silver et al., 2021)).
UltraFoedback-vva
3.84
3.59
etraFeeedbakk-ev--ltte
3.92
3.63
UltaaFeehacc-evaa ava-aale---a------
3.98
3.73
Original
2001
keel
2
2000
100
4.2.4. EVA IMPROVES EFFICIENCY & GENERALIZATION
Madi
Extration
Humantien
Writing
Codigg
STEM
Redeplay
Browntng
We run the default incremental training (i.e., trainining from the last checkpoint with the evolved set in each iteration), as in Fig 5 and & H.2. eva presents monotonic gains.
Figure 6: Curriculum effect in training distributions. The prompt distribution of Table 13. eva creates curriculum that prioritizes math / coding prompts over iterations.
The solutions found by eva cannol be recovered by training longer by fixed set (the dashed), nor by naively sourcing new prompts without examining informativeness (the gray dotted), thus our generative data schedule is effective.
Witting
Humanitie
Tolepay
model
grmma--244 2-bb-1
gamma 98 iidpo
We conjecture that behaviors of the dashed/dotted lines re- late to loss of plasticity (Ash & Adams, 2019; Dohare et al., 2023; Abbas al., 2023; Xue al., 2024). Classical works resolve it by the optimization view (e.g., weight perturbing), whereas offers a new data view, potentially mimicing an implicit regularizer for better generalization.
STEM
gyoms-3-48 3-4h-------- ne-e--p----et-eeee
Betractumn
Math
Coding
Figure 7: Curriculum effect in benchmark performance. The radar figure for ratings on MT-Bench. eva prioritizes and gradually improves on coding, math and reasoning over iterations, implicitly reflecting learned curriculum.
DPO
SPRO
Arema Hond win Hote (S)
Acena Hand Win Mate (S)
came pronete
nav buman promepe une protiple
12
In Table 7, we show that there is a gradual improvement in prompt complexity and quality" over iterations with eva. In Figure 6 and 7, we show that eva brings auto-curricula and the creator is incentivized to create new prompts that are informative w.r.f. the current solver policy.
hur
Dur
harr
Figure 5: eva stays robust with more iterations.
Together, those evidences supports the importance of adap- tively evolving prompts jointly with responses, which we believe to be crucial in scaling up next-gen RL post-training.
In Table 6, we ablate eva in scratch training, i.e., training with the full original and evolved set. eva is competitive in incremental training, learning more effectively with less data a nice bonus by minimax regret (Jiang et al., 2021a).
*We use gemi -1. as the generative scorer ($ E).
7

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
5. Related Works
proposing a task, and Bob doing it" (Sukhbaatar et al., 2017;
Samvelyan et al., 2023; Beukman al., 2024; Dennis et al., 2020). We revive the classical asymmetric self-play (Sutton et al., 2011) in optimizing language models. Unlike tradi- tional curriculum RI. (Parker-Holder et al., 2022), which renders environments by specifying levels (Dennis et al., 2020), our approach is generative by nature, as we directly generate states from powerful generative models.
With the history of machine learning, it is not new that self- play and data exploration brings intelligence (e.g., Schmid- huber (1991)). We believe, to our knowledge, eva is among the first empirical works that systematically studied adaptive prompt evolving in RL post-training on large-scale LLM benchmarks. Below, viewing from many different angles, we list eva's 's distinct impact and contribution.
Self-improving algorithms and iterative optimization. This line of work focuses on iteratively generating samples from the response policy and continuously re-training the policy by selected self-generated samples. Major works include ReST (Gulcehre et al., 2023; Singh et al., 2023), STaR (Zelikman et al., 2022). RFT (Yuan et al., 2023), RAFT (Dong et al., 2023), self-improving LLMSs (Huang al., 2022; Yuan et al., 2024); in the context of preference optimization, iterative DPO (Tran et al., 2023; Xiong et al., 2024; Pang et al., 2024b) has proven effective. Most works focus on self-training by improving in y | X, while we jointly optimize both responses and prompts via generative exploration in V)). allowing for continual RLHF.
Self-play in RLHE A growing line of research frames RLHF as symmetric self-play game, where both players are response players (Munos et al., 2023; Wu et al., 2024; Choi et al., 2024; Rosset et al., 2024). However, these methods still rely on a fixed prompt distribution thus is sub-optimal. In contrast, we solve this by asymmetric self-play, enabling evolving prompt distributions; this step is structurely resem- bles adversarial training (Goodfellow et al., 2014b; Ho & Ermon, 2016). (Zheng et al., 2024) is a concurrent work with similar asymmetric setup, however it applies to safety tasks instead of general instruction mnstructionfoolloinng tasks and is incompatible with direct preference optimization.
Prompt synthesis for language models. Major works include Self-Instruel ( (Wang et al., 2022), WizardLM (Xu et al., 2023; Luo et al., 2023), Self-Align (Sun et al., 2024), EvoPrompt (Guo et al., 2023), Magpie (Xu et al., 2024), etc. eva is orthogonal to them since any such method can be plugged in as the evolve(.) for the creator (note we con- tribute a small trick with BoN tree in $ 3). We foeus on the RL post-training phase, while those works are primarily SFT. Importantly, our work proposes an adaptive metric to sample and evolve prompts- - secret sauce is that prompts shall be prioritized w.zt. the informativeness, whereas prior works mostly evolve uniformly (Yuan et al., 2024). Fur- thermore, most works evolves in an offline manner, while eva, to our knowledge, is the first framework that supports online/on-poliy evolving in general RL post-training.
6. Concluding Remarks
Future directions. eva defines new paradigm for RL post-training, opening up many new directions. e.g., (i) jointly optimizing the reward model (RMs) with eva - we assume a fixed oracle RM, as is de facto practice in indus- try (Team et al., 2024a); however, as the policy updates, eva can generate outoo--distribution prompts, necessitating the need for continual RM training (Makar-Limanov et al., 2024); (ii) extending to differentiable creator policies; (iii) extending to reasoning (Poesia et al., 2024): (iv) extending the game with more modality (Bruce et al., 2024), and/or with more players (e.g., rewriters (Kumar et al., 2024)).
This empirical work presents eva, a new, simple and scalable framework for RL post-training of LLMs, that adaptively generates prompt distributions dur- ing training. eva is simple it can be easily plugged into any existing pipeline, and highly effective - it reaches new SOTA on challenging alignment benchmarks. The primary takeaway may be: (i) self-evolving joint training distribu- tions (X, J)) brings significant gain, and (ii) reward advan- tage acts as an effective metric informing the collection and creation of prompts in RLHF Philosophically, eva presents new view of post-training as an infinite game (Abel et al., 2024); eva incentivizes agents to create problems rather than to simply solve, which is key to intelligence, yet LLM trainers may neglect.
Conclusions.
Active and curriculum learning. This line of works re- order training examples for efficiency (Bengio et al., 2009; Mindermann et al., 2022; Kawaguchi & Lu, 2020), with recent LLM-related works (Muldrew et al., 2024; Das et al., 2024) (note those are done much smaller scale com- pared to eva). In contrast, eva breaks free from the static paradigm, not only re-orders data but also generatively creates new data, yielding significant gains (S 4.2.2).
Self-play and curriculum RL. Agents trained on a fixed data distribution are often brittle and may struggle to adapt to the real world (Hughes et al., 2024). Self-play (Samuel, 1959; Goodfellow et al., 2014a; Silver et al., 2016) addresses this by having the agent learn through self-interaction, thus creating more diverse experiences and automatic curric- ula. In asymmetric self-play, the paradigm centers on "Alice
We believe the community should be aware of the surpris- ing effectiveness of adaptive evolving prompts in RL post- training, and scale them together with responses.
8

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Impact Statement
warm-starting neural network training. arXiv preprint arXiv: 2019.
eva enables scalable training of language agents through open-ended training, improving AI alignment with hu- man values, which will ultimately contribute social wel- fare (Pigou, 1920; Arrow, 1952; Rosenfeld & Xu, 2025). This may democratize the development of more generally capable artificial intelligence agents, impacting a wide range of domains from scientific discovery to societal governance.
Ball, P. J., Smith, I.,, Kostrikov, I., and Levine, S. Ef- ficient online reinforcement learning with offline data. In International Conference on Machine Learning, pP- 1577-1594, PMLR, 2023.
Banos, A. On pseudo- games. The Annals of Mathematical Statistics, (9966):133211992 1968.
We recognize that eva relies on self-exploration guided by reward signals. If these signals are inaccurate or misaligned, the trained agents may exhibit undesirable behaviors such as reinforeing biases or hallucinations. Mitigating these risks requires continued research into robust reward models, transparent evaluation protocols, open collaboration within the Al research community, and more. As authors, we will be committed to supporting these efforts by sharing our find- ings and implementations to promote open and responsible research and development.
Bengio, Y.
Deep learning of representations for unsuper- vised and transfer learning. In Proceedings of ICML workshop on unsupervised and transfer learning, pp. 17- 36. JMLR Workshop and Conference Proceedings. 2012.
Bengio, Y., Louradour, J., Collobert, R., and Weston, J. Curriculum learning. In Proceedings of the 26th annual international conference machine learning, pp. 41-48, 2009.
Beukman, M., Coward, S., Matthews, M., Fellows, M., Jiang, M., Dennis, M., and Foerster, J. Refining Minimax Regret for Unsupervised Environment Design. arXiv preprint arXiv:2402.12284 2024.
Acknowledgments
We extend our warm gratitude to Bilal Piot for his thoughtful reviews and valuable suggestions on this paper. We are also grateful to Chenkai Kuang, Dustin Tran, Albert Webson, Hanzhao Lin, Clara Huiyi Hu, Jeremiah Liu, Luheng He, Chenjie Gu, Yong Cheng, Pei Sun, and Heng-Tze Cheng for their fun discussions and interesting notes on Self-Play. We thank David Abel, Yuxin Chen, Minbiao Han, Feiran Jia, Ziniu Hu, Guohao Li, Rylan Schaeffer, Haifeng Xu, Chaoqi Wang and Yifei Wang for their helpful discussions and references on fine-tuning, contrastive learning, data synthesis, open-ended learning, and continual reinforcement learning. We also thank many anonymous reviewers.
Bruce, J., Dennis, M. D., Edwards, A., Parker-Holder, J., Shi, Y., Hughes, E., Lai, M., Mavalankar, A., Steiger- wald, R., Apps, C., et al. Genie: Generative interactive environments. In Forty-first International Conference on Machine Learning, 2024.
Chaiklin, S. et al. The zone of proximal development in Vy- gotsky's 's analysis of learning and instruction. Vygotsky's 'ss educational theory in cultural context, 1(2):9 64, 2003.
Choi, E., Ahmadian, A., Geist, M., Pietquin, O., and Azar, M. G. Iepf-impgo Robust Preference Optimization. arXiv preprint arXiv:2406.01666 2024.
References
Abbas, Z., Zhao, R., Modayil, J., White, A., and Machado, M. C. Loss of plasticity in continual deep reinforcement learning. In Conference on Lifelong Learning Agents, pp. 620-636. PMLR, 2023.
Chow, Y., Tennenholtz, G., Gur, I., Zhuang, V., Dai, B., Thiagarajan, S., Boutilier, C.,, Agarwal, R., Kumar, A., and Faust, A. Inference-Aware Fine-Tuning for Best-of- N Sampling in Large Language Models. arXiv preprint arXiv:2412.1 152877 2024.
Abel, D., Barreto, A., Van Roy, B., Precup, D., van Has- selt, H. P., and Singh, S. A definition of continual re- inforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
Cui, G., Yuan, L., Ding, N., Yao, G., Zhu, W., Ni, Y., Xie. G., Liu, Z., and Sun, M. Ulrafeedback: Boosting lan- guage models with high-quality feedback. arXiv preprint arXh::231000137,, 2023.
Ahmadian, A., Cremer, C., Gallé, M., Fadaee, M., Kreutzer, J.. Üstün, A., and Hooker, S. Back to basics: Revisit- ing reinforce style optimization for learning from human feedback in Ilms. arXiv preprint arXiv:2402. 14740, 2024.
Das, N., Chakraborty, S., Pacchiano, A., and Chowdhury, S. R. Provably sample efficient rihf via active preference optimization. arXiv preprint 2rXiv:2402.10500 2024.
Arrow, Social choice and individual values, volume 12. Yale university press, 1952.
Dennis,
M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., and Levine, S. Emergent complexity and zero-shot transfer via unsupervised environment design.
Ash, J. T. and Adams, R. P. On the difficulty of

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Advances in neural information processing systems, 33: 13049-13061, 2020.
Ho, J. and Ermon, S. Generative adversarial imitation learn-
ing. Advances in neural information processing systems, 29, 2016.
Dohare, S., Hemnandez-Gariia_ J. F., Rahman, P., Mahmood, A. R., and Sutton, R. S. Maintaining plasticity in deep continual learning. arXiv preprint arXiv:2306.13811 2023.
Hong, J., Lee, N., and Thorne, Orpo: Monolithic pref- erence optimization without reference model. arXiv preprint arXiv:2403.0769, 2(4):5, 2024.
Dong, H., Xiong, W., Goyal, D., Zhang. Y. Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint NXiv:2304.06766,, 2023.
Huang. J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv
preprint arXiv:22 10. /6/00, 2022.
Dong, H., Xiong, W., Pang. B., Wang, H., Zhao, H., Zhou, Y., Jiang, N., Sahoo, D., Xiong, C., and Zhang, T. RLHF Workflow: From Reward Modeling to Online RLHF, 2024.
Hughes, E., Dennis, M., Holder, J., Behbahani, F,, Mavalankar, A., Shi, Y., Schaul, T., and Rocktaschel, T. Open-Endedness is Essential for Artificial Superhuman Intelligence. arXiv preprint arXiv:2406.00 04268, 2024.
Dubois, Y. Galambosi, B., Liang, P., and Hashimoto, T. B. Length-controlled alpacaeval: simple way to debias automatic evaluators. arXiv preprint arXiv:2404.0444 2024.
Jiang, A. H., Wong, D. L.-K., Zhou, G., Andersen, D. G., Dean, J., Ganger, G. R., Joshi, G., Kaminksy, M., Kozuch, M., Lipton, Z. C., et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv: 10l0.00210007 2019.
Dwaracherla, V., Asghari, S. M., Hao, B., and Van Roy, B. Efficient exploration for llms. arXiv preprinl arXiv:2402.00396, 2024.
Jiang. D., Ren, X., and Lin, B. Y. LIm-blender: Ensembling large language models with pairwise ranking and genera- tive fusion. arXiv preprint arXiv:2306.025500 2023.
Goodfellow, I., Pouget-Abadis J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances neural informa- tion processing systems, 27, 2014a.
Jiang, M. Learning Curricula in Open- Ended Worlds. arXii preprint arXiv:2312.03126, 2023.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. Generative adversarial nets. Advances in neural informa- tion processing systems, 27, 2014b.
Jiang, M., Dennis, M., Parker-Holder, J., Foerster, J., Grefen- stette, E., and Rocktäschel, Replay-guided adversarial environment design. Advances in Neural Information Processing Systems, 34: 14:1888-1897 2021a.
Gulcehre, C., Paine, T. L., Srinivasan, S., Konyushkova, K., Weerts, L., Sharma, A., Siddhant, A., Ahern, A., Wang, M., Gu, C., et al. Reinforced self-training (rest) for language modeling, arXiv preprint arXiv:2308.08998 2023.
Jiang, M., Grefenstette, E., and Rocktäschel, T. Prioritized level replay. In International Conference Machine Learning, pp. 4940-4950, PMLR, 2021b.
Jiang, Y., Zhou, A., Feng, 7.,, Malladi, S., and Kolter, J. Adaptive data optimization: Dynamic sample selection with scaling laws. arXiv preprint arXiv:2410.11820. 2024.
Guo, Q. Wang, R., Guo, J., Li, B., Song, K., Tan, X., Liu, G., Bian, J., and Yang, Y. Connecting large language mod- cls with evolutionary algorithms yields powerful prompt optimizers. arXiv preprint w.Xiv:2309.08532. 2023.
Jin, C., Netrapalli, P., and Jordan, M. What is local optimal- ity in nonconvex-nonconee minimax optimization? In International conference on machine learning, pp. 4880- 4889. PMLR, 2020.
Guo, S., Zhang, B., Liu, T., Liu, T., Khalman, M., Llinares, F., Rame, A., Mesnard, T., Zhao, Y., Piot, B., et Direct language model alignment from online ai feedback. arXiv 2024.
preprint ar arXiv:2402.04700
Kawaguchi, K. and Lu, H. Ordered sgd: A new stochastic optimization framework for empirical risk minimization. In International Conference on Artificial Intelligence and Statistics, pp. 669 679. PMLR, 2020.
Gustafsson, J. E. Decisions under Ignorance, 2022. Lecture notes.
Hejna, J., Rafailov, R., Sikchi, H., Finn, C., Nickum, S., Knox, W. B., and Sadigh, D. Contrastive prefence learn- ing: Learning from human feedback without rl. arXiv preprint arXiv:2310. /3639, 2023.
Keynes, J. M. treatise on probability. Courier Corporation, 1921.
10

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Kumar, A., Zhuang. V., Agarwal, R., Su, Y., Co-Reyes, J. D., Singh, A., Baumli, K., lqbal, S., Bishop, C.,, Roelofs, R., et al. Training language models to self-correct via reinforcement learning. arXiv preprint arXiv:200.12211,, 2024.
Mindermann, S., Brauner, J. M., Razzak, M. T., Sharma, M., Kirsch, A., Xu, W., Höltgen, B., Gomez, A. N., Morisot, A., Farquhar, S., et al. Prioritized training points that are learnable, worth learning, and not yet learnt. In In- ternational Conference on Machine Learning, pp. 15630- 15649. PMLR, 2022.
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic q g-ensemble. In Conference on Robot Learn- ing. pp. 1702-1712. PMLR, 2022.
Muldrew, W., Hayes, P., Zhang, M., and Barber, D. Active Preference Learning for Large Language Models. arXiv preprint arXiv:2402.08/ 14, 2024.
Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V., and Legg. S. Scalable agent alignment via reward modeling: research direction. arXiv preprint 1rXivv1811.0777 2018.
Munos, R., Valko, M., Calandriello, D., Azar, M. G., Row- land, M., Guo, Z. D., Tang, Y., Geist, M., Mesnard, T., Michi, A., et al. Nash learning from human feedback. arXiv preprint arXiv:2312 2023.
Li, T., Chiang, W.-L., Frick, E., Dunlap, L., Wu, T., Zhu, B., Gonzalez, J. E., and Stoica, I. From Crowdsourced Data to frigh-quality Benchmarks: Arena-Hard and Bench- Builder Pipeline. arXiv preprint arXiv:2406.11930 2024.
Nishihara, R. Take on DeepSeck 2025. LinkedIn post.
Ouyang. L., Wu, J., Jiang. X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A.. al. Training language models follow instructions with human feedback. Advances in neural information processing systems, 01127730-2774444 2022.
Li, Z., Xu, T., Zhang. Y., Lin, Z., Yu, Y.. Sun, R., and Luo, Z.-Q. Remax: A simple, effective, and efficient reinforce- ment learning method for aligning large language models. arXiv preprint arXiv:2310. 10505, 2023.
Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and Weston, Iterative reasoning preference optimization. arXiv preprint arXiv:2404. 19733,, 2024a.
Liu, C. Y. and Zeng, L. Skywork Reward Model Series. ht tps: : htt.g/hggggccccceeee/g//oo co/ Skywork, September co/ cttccoo//// Skkwo
2024. URL htttps://hugglie rk.
Pang, R. Y., Yuan, W., Cho, K., He, H., Sukhbaatar, S., and
Weston, Iterative reasoning preference optimization. arXiv preprint arXiv:2404.19773 2024b.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sam- pling improves preference optimization. arXiv preprint arXiv:2309.00 06057, 2023a.
Parker-Holder, J., Jiang, M., Dennis, M., Samvelyan, M., Fo- erster, J., Grefenstette, E., and Rocktäschel, T. Evolving curricula with regret-based environment design. In Inter- national Conference on Machine Learning, pp. 17473- 17498. PMLR, 2022.
Liu, W. Zeng, W., He, K., Jiang, Y., and He, What makes good data for alignment? comprehensive study of automatic data selection in instruction tuning. arXiv preprint arXiv:2312. 235.1555,, 2023b.
Pigou, A. C. Some aspects of welfare economics. The American Economic Review, 41(3):287-302, 1920.
Luo, H., Sun, Q. Xu, C., Zhao, P., Lou, J., Tao, C., Geng, X., Lin, Q. Chen, S., and Zhang, D. Wizard- math: Empowering mathematical reasoning for large lan- guage models via reinforced evol-instruct. arXiv preprint arXiv:2308. 09583, 2023.
Poesia, G., Broman, D., Haber, N., and Goodman, N. D.
Learning Formal Mathematics From Intrinsic Motivation. arXiv preprint arXiv:2407.00695 2024.
Rafailov, R., Sharma, A., Mitchell, E.,, Ermon, S., Manning, C. D., and Finn, C. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305. 18290, 2023.
Makar-Linn Limanov, J., Prakash, A., Goktas, D., Greenwald, A., and Ayanian, N. STA-RLHF: Stackelberg Aligned Reinforcement Learning with Human Feedback. Co- ordination and Cooperation for Multi-Agent Reinforce- ment Learning Methods Workshop, 2024. URL https: //openzeview net / forumm == Jen Jen
Rosenfeld, N. and Xu, H. Machine Learning Should Max- imize Welfare, Not (Only) Accuracy. arXiv preprint arXiv:2502. 11981, 2025.
Meng, Y., Xia, M., and Chen, D. SimPO: Simple Prefer- ence Optimization with a Reference-Free Reward. arXiv preprint arXiv:2405. 14734, 2024.
Ross, S. and Bagnell, J. A. Agnostic system identification for mnodel-based reinforcement learning. arXiv preprint arXiv: /203. 1007, 2012.

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Rosset, C., Cheng, C.-A., Mitra, A., Santacroce, M., Awadal- lah, A., and Xie, T. Direct Nash Optimization: Teaching Language Models to Self-Improve with General Prefer- ences. arXiv preprint arXiv:2404.03715. 2024.
Sun, Z., Shen, Y., Zhou, Q. Zhang, H., Chen, Z.., Cox, D., Yang, Y., and Gan, C. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2024.
Russo, and Van Roy, B. Learning optimize via cnormattiondirected sampling. Advances in neural infor- mation processing systems, 27, 2014.
Sutton, R. S., Barto, A. G., et al. Reinforcement learning: An introduction. MIT press Cambridge, 1998.
Samuel, A. L. Some studies in machine learning using the game of checkers. IBM Journal of research and development, 3(3):210-229, 1959.
Sutton, R. S., Modayil, J., Delp, M., Degris, T., Pilarski, P. M., White, A., and Precup, D. Horde: A scalable real-time architecture for learning knowledge from unsu- pervised sensorimotor interaction. In The 10th Interna- tional Conference on Autonomous Agents and Multiagent Systems-Volee 2, pp. 761-768, 2011.
Samvelyan, M., Khan, A., Dennis, M., Jiang, M., Parker- Holder, J., Foerster, J., Railcanu, R., and Rocktäschel, T. MAESTRO: Open-ended environment design for multi agent reinforcement learning. arXiv preprint arXiv:2303.083 033766 2023.
Tang, Y., Guo, Z. D., Zheng, Z., Calandriello, D., Munos, R., Rowland, M., Richemond, P. H., Valko, M., Pires, B. Á., and Piot, B. Generalized preference optimization: A unified approach to offline alignment. arXiv preprint arXiv:2402.05749, 2024.
Savage, J. The theory of statistical decision. Journal of the American Statistical association, 46(253):55-67, 1951.
Team, D., Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. DeepSeek- R1: Incentivizing Reasoning Capability in LLMs via Re- inforcement Learning. arXiv preprint 1214425501.12948, 2025.
Schaul, T. Quan, J., Antonoglou, I. and Silve, D. Prioritized Experience Replay. arXiv preprint 2015.
Schmidhuber, A possibility for implementing curiosity and boredom in model-building neural controllers. In Proc. of the international conference on simulation of adaptive behavior: From animals to animats, 1991.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Sorieut, R., Schalkwyk, J., Dai, M., Hauth, A., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.11808 2023.
Schwefel, H.-P. Evolationsstrategien, für die numerische Optimierung. Springer, 1977.
Team, G., Reid. M., Savinov, N., Teplyashin, D., Dmitry, L., Lillicrap, T., Alayrac, J., Soricut, R., Lazaridou, A., Firat, O., et al. Gemini 1.5; Unlocking multimodal understand- ing across millions of tokens of context. arXiv preprint arXiv:2403.05530 2024a.
Shi, R., Zhou, R., and Du, S. S. The Crucial Role of Sam- plers in Online Direct Preference Optimization. arXiv preprint arXiv:2409. 196055,, 2024.
Silver, D., Huang, A., Maddison, C. J., Guez, A., Sifre, L., Van Den Driessche, G., Schrittwieser, J., Antonoglou, I., Panneershelvam, V., Lanetot, M., al. Mastering the game of Go with deep neural networks and tree search. nature, 529(7587):484 2016.
Team, G., Riviere M., Pathak, S., Sessa, P. G., Hardin, C., Bhupatiraju, S., Hussenot, L.., Mesnard, T., Shahri- ari, B., Ramé, A., et al. Gemma 2: Improving open language models at a practical size. arXiv preprint Ttt:22008.001188 2024b.
Silver, D., Singh, S., Precup, D., and Sutton, R. S. Reward is enough. Artificial Intelligence, 299:103535, 2021.
Tobin, J., Fong, R., Ray. A., Schneider, J.. Zaremba, W. and Abbeel, P. Domain randomization for transferring deep neural networks from simulation to the real world. In 2017 IEFE/RSJ international conference on intelligent robots and systems (IROS), pp. 23-30. IEEE, 2017.
Singh, A., Co-Reyes, J. D., Agarwal, R., Anand, A., Patil, P., Liu, P. J., Harrison, J., Lee, J., Xu, K., Parisi, A., et al. Beyond human data: Scaling self-training for problem-solving with language models. arXiv preprint arXiv:2312.06585. 2023.
Tran, H., Glaze, C., and Hancock, B. Iterative DPO Align- ment. Technical report, Snorkel AI, 2023.
Sukhbaatar, S., Lin, Z., Kostrikov, L., Synnaeve, G., Szlam, A., and Fergus, R. Intrinsic motivation and automatic curricula via asymmetric self-play. arXiv preprint 17Xiv:1703.05400/ 2017.
von Stackelberg, H. Markfform und Gleichgewicht, Die Bondelsblatt-iiblottae "Klassiker der Na- tionalókonomie" J. Springer, 1934. URL https: /bo oks. google e cww/books ? d= AAAAAAAAAAAAAAA
12

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Wang, H., Xiong, W. Xie, T., Zhao, H., and Zhang, T. Interpretable Preferences via Multi-Objective Re- ward Modeling and Mixture-of-Expers.. arXiv preprint arXiv:2406. 12845, 2024.
Zhang, G. Deep Learning Dynamics: From Minimization Games. PhD thesis, University of Toronto (Canada), 2023.
Zhang, G., Wang. Y., Lessard, L.,, and Grosse, R. B. Near- optimal local convergence of alternating gradient descent- ascent for minimax optimization. In International Con- ference Artificial Intelligence and Statistics, pp. 7659- 7679. PMLR, 2022.
Wang, Y., Kordi, Y., Mishra, S., Liu, A., Smith, N. A., Khashabi, D., and Hajishirzi, H. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint ar arXiv:2212.10560 2022.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Wu, H. and Liu, X. Double thompson sampling for duel- ing bandits. Advances in newral information processing systems, 29, 2016.
Wu, Y. Sun, Z., Yuan, H., Ji, K., Yang, Y., and Gu, Q- Self-play preference optimization for language model alignment. arXiv preprint 000752405.00665. 2024.
Zheng, R., Guo, H., Liu, Z., Zhang, X., Yao, Y., Xu, X., Wang. Z., Xi, Z., Gui, T., Zhang, Q., et al. Toward Opti- mal LLM Alignments Using Two-Player Games, arXiv preprint arXiv:2406. 10977, 2024.
Xie, S. M., Santurkar, S., Ma, T., and Liang, P. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36: 34201-34221 2023.
Xiong, W., Dong, H., Ye, C., Wang, Z., Zhong, H., Ji, H., Jiang, N., and Zhang, T. Iterative preference learning from human feedback: Bridging theory and practice for rlhf under kl-constraint. In Forty-first International Con- ference on Machine Learning, 2024.
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng, J., Tao, C., and Jiang, D. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
Xu, Z., Jiang, F., Niu, L., Deng, Y., Poovendran, R., Choi, Y., and Lin, B. Y. Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing. arXiv preprint 2Xxx:2406.08445 2024.
Xue, F., Fu, Y., Zhou, W., Zheng, Z., and You, To repeat not repeat: Insights from scaling Ilm under token-crisis. Advances in Neural Information Processing Systems, 36, 2024.
Yuan, W. Pang, R. Y., Cho, K., Sukhbaatar, S., Xu, J., and Weston, J. Self-rewarding language models. arXiv preprint arXiv:2401.10020 2024.
Yuan, Z., Yuan, H., Li, C., Dong, G., Lu, K., Tan, C., Zhou, C., and Zhou, J. Scaling relationship on learning math- ematical reasoning with large language models. arXiv preprint +XXv:2308.01825, 2023.
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. Star: Boot- strapping reasoning with reasoning, Advances in Neural Information Processing Systems, 35:15476-1544 2022.
13

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
A. Details on Reproducibility
Software environments. All experiments are conducted on 8xNVIDIA H100 SXM GPUs. Our codebase primarily relies on trans formers==4. 40. O. For the response generation of GEMMA models the training stage, we use v1lm==0. 5.4 with flashinfer backend for CUDA 12. and torch 2.4. For evolving prompts, we use distilabel==1. use LiteLIM to serve Gemini (default be gemini-1.
and and transformers models (default to be gemma-2-9b- For evaluation on all benchmarks, we use sglang==0.r 2. and openai==1.38 35.11, with gpt-4-1106-previeee as the judge model and pot-4-0311--reviee as the baseline model. Specifically for AlpacaEval 2.0, we use alpaca.oval.gpt furrbooffn as the annotator config.
Hyperparameter settings. We follow the original hyperparameter settings as in (Hong et al., 2024; Meng et al., 2024; Wu et al., 2024), default to be:
Hyperparameter (4) Loss (-))
DPO
ORPO
SimPO
SPPO
learning rate
5e-7
5e-7
8e-7
Se-7
learning rate scheduler
cosine
cosine
cosine
linear
0.05
10
0.001
/
/
5
A
/
0.5
/
no. epochs per iter warmup ratio per iter effective batch size max length max prompt length optimizer
2
1
1
0.1
0.1
0.1
0.1
8
8
32
2048
2048
2048
1024
1024
1024
1024
512
adamw
adamw
adamw
rmsprop
Iterative Training Settings for Offline eva. By default, we train with equal-size prompt subset in each iteration. Unless otherwise specified, we use 10K prompts from the UltraFeedback dataset (Cui et al., 2023) per iteration. The incremental training proceeds as follows (note this is also compatible with online solvers):
0g : Base SFT model.
0--11 initialize with Og: then train w/ the prompt split Xi by self-generated responses from the initial model 0g.
01+2: : initialize with 00-1: trained w/ the prompt split X2 by celf-generated responses from the model 00-+1
For evolving prompts (e.g., evolving Xi to Xi), with the calculated informativeness metric for each prompt, we normalize them as the weight to do weighted sampling for a 25% informative subset to get Xinto, We then iterate over in Xinto and call EvolInst rut (Xu et al., 2023) as the plug-in evolving method (with the number of evolutions 4) using the default mutation templates for (i) in-depth evolving (constraints, deepening, concretizing, increased reasoning steps) and (ūi in- mn-breadth evolving (extrapolation) implemented in tasks/evol www.oliinsuuutuuuttee Next we uniformly select 80% prompts from this evolved dataset and 20% from the original dataset (i.e., the buffer) to form X1. We do not seek extensive parameter search (e.g., the number of evolutions, the evolving ratio) in this stage and encourage future works on exploring this and other plug-in evolving methods. For solver we generate 6 responses per prompt. We use 42 as the random seed.
py of 1istilabel==1.
Training setting online eva Table 1. Plainly put, online eva evolves per mini-batch. In Table 1, we have eva works in three phases with the generative buffer. Below we present an casy-to-understand illustration:
1. Warm-up phase: Default training with data from until the buffer is full. To create the buffer, after each step (training 8 samples in mini-batch) we select the top 50% (which is 4) most informative prompts in the mini batch, then evolve New versions for each one, and we add only the evolved prompts the buffer. (Here, we set Neew 4. thus the buffer increases by 16 at each step, until we reach the preset buffer size which is 3200 currently, i.e., 400 iterations).
2. Mix-up phase: From now on, we do a balanced sampling from the buffer and D to form a mini batch for training, where ratio 50%: each optimization step, we sample half of the mini batch from the buffer by weighted
14

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
sampling via the informativeness score, and take the rest half i.i.d. from D. We similarly evolve new new prompts for the top 50% informative prompts in this batch. We then add the newly generated prompts, and pop the trained ones and the oldest ones from the buffer, to keep fixed buffer size. Then balanced sampling approach resembles the scheme used in (Ross & Bagnell, 2012; Ball et al., 2023).
3. Bootstrap phase:
After we run out of samples from D, we will only sample from the buffer, with the same evolving procedure, and add evolved prompts to the buffer and pop out those trained. We take the top 50% from each mini batch, evolve for bootstrap prompts for each, then pop out with the trained. new
The Hyper-parameete of the subset set sizes as power of 2 is due to the hardware optimization constraints.
B. Additional Ablation Experiments for Online eva
In the following ablation studies, by keeping other syper-parameters the same the above, show that increasing N'ew improves the performance, while balanced sampling with ratio 50% is generally the most robust.
Table 8: The default setting for the first column of each table is 7lee 4 and ratio = 50%. (Left) Varying the number of evolved prompts Vnew- As we increase Feew. we observe a monotonic gain, probably due to the fact that evolving more prompts helps improving the coverage and diversity of training prompts. (Right) Varying the sampling ratio of evolved prompts in each mini-batch. In general, we find the balanced sampling strategy is more robust as the training goes on.
Benchmark (->
Arena Hard (WR (4))
Benchmark (-)
Arena Hard (WR (9))
Setting (-)
new
nnew
8
Setting (-))
ratio = 50%
ratio = 75%
ratio = 25%
0o: Base Model
41.3
41.3
0g: Base Model
41.3
41.3
41.3
0-11: RLOO (1x)
52.6
52.6
0--1: RLOO (1x)
52.6
52.6
52.6
0 0 RLOO-eva (1x)
57.3
$7.6
0 -- RLO0-eva (1x)
57.3
57.0
57.5
0 0-11: RLOO-eva (2x)
60.5
61.2
00-1: RLO0-eva (2x)
60.5
59.9
59.2
0 RLOO-eva (3x)
62.4
63.0
0--i= RLOO-eva (3x)
62.4
62.0
61.3
00-11 RLOO (6x)
62.7
62.7
00-11 RLOO (6x)
62.7
62.7
62.7
C. Additional References
In addition 85,, 5, prior works (Jiang et al., 2024; Xie et al., 2023) have explored signals for adaptive data sampling, which can be integrated into eva's s generative pipeline. For the online solver, an alternative to RLOO (Ahmadian et al., 2024) is ReMax (Li et al., 2023). An important future work for eva is to have efficient online reinforcement learning of reward models on top of the creator-solver game, on which Russo & Van Roy (2014); Dwaracherla et al. (2024) shed light.
D. Visualization on Prompt Selection Metric
In eva, we assign each prompt an informativeness value, which the creator will use as the weight to sample from the seed prompts for prompt synthesis. In $ 4.2.1, we have shown that traditional methods like reward mean and reward variance are less effective our advantage-bant based informativeness proxy. The intuition is simple: advantage/regree-bassee proxy aligns better with the preference optimization objective. We further illustrate that they are statistically different from other choices:
The distribution of informativeness values shows that reward variance is heavily concentrated at lower values, reward mean is more uniformly scattered, and reward advantage achieves a better balance, providing a broader yet also focused sampling range.
Figure 8:
The weak correlation between reward variance and reward advantage shows that variance cannot serve as a substitute for advantage as proxy for informativeness.
Figure 9:
We have discussed the contrastive curriculum hypothesis in 8 H.3 to support using reward advantage in the sense that the induced samples tend to decrease the loss the most in the contrastive optimization. Furthermore, assuming the optimization algorithm can converge to the more optimal responses, neither reward mean nor variance directly capture the learning potential of such responses one may easily construct cases with identical variance yet differ much in reward range
15

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
reward advantage (max min)
1.0
Teward variance
variance v. advantage (max min)
reward advantage (max mean)
0.8
variance advantage (max mean)
density
reward variance
reward mean
0.6
0.4
0.2
0
0.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
normalized value
reward advantage
Figure 8: The probability density distributions of infor- mativeness metrics compared in Table 4 they show different patterns.
Figure 9: The correlation plot for reward advantage (ours) and reward variance they are only weakly correlated.
thus variance fails to distinguish such scenarios. By contrast, reward advantage estimate inherently captures the relative improvement towards better response, and is sensitive differences in reward range; variants of advantage estimate are commonly used in literature.
E. Complexity and Quality of Prompts Over Iterations
As in Table 7, there is a gradual improvement of prompt complexity and quality over iterations with eva. We sample 10K prompts per iteration, and use the below prompts modified from (Liu et al., 2023b) for the complexity and quality evaluation, with gemini-1 flash as the scorer:
fowlowing questions aecording their quality. Your evaluat ahould consider Fall Reievance, Accuracy, popth, Crestivity, and Level oz detail. Score each resporse From to St 11 Poor quality, 2e Rellww average, 3: Average, 4 Goody St Rxxeellees
Helpfulness,
Ranking the rollowing questions according to their dicticulty and complexity. Use rixed scoring eysten: 1: Very simple, 2: Simplo, Hodesate, Dicricult, dieficuls
F. Evolving Instructions
iter
iter 2
i'tey
Figure 10: An illustrative example of tree BoN. Given seed prompt, we uniformly sample evolving strategies from below to apply to it, which results multiple generations; we then proceed with the best one (e.g., by the complexity and quality scorer) the seed prompt for the next generation. We use this setting in the online eva setting trial.
BALBBEAATHLLEE D'gersune', falil 'shift oul', fabsLrac.") DEPEPTE DEPTELLEES D"oosstrainta", 'dospening', "cancretising", 'reasoning", "expansion")
16

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
EVOL, METRODS
La-breadth evolving
"persene";
*Ratzana the saiven Promot written user with comeistely diffezent parsona, backaround, or expertise. "Adjust the tonc, style, phrzsing, or enything you tesl proper to reflect this change. *The changes should make the prompt tsel Like it was authored someone entirely new."
"shitt-in":
"Shitt the high-level ides of the #diven Prompt to explore a difterent subdinsin or context within the same domain.
"Ensure the new topic still chellenges the model to reason or provide knoxiedos relevant to the denain."
"shift-our";
"Shif: the hiah-levei idez of the ddiven Prompt to complesely different topic in differant petting,
*The new topic may challerge the model with cimilar reassaing contextual understanding but in nevel kay."
"mix":
"cobone the high-level concept of the toiven Prempt with alements tron a different domain.
*Iaczoducs nevel scenarios or concexts to create diversity while naintaining relevance to the original idea."
"Tann the #diven Pronsta inzo a nore abstrast or generalized version, zenoving specifis dataila while preservicg its intent.
"Easure the now prompt caceurages broadet, principle-triver reasoning."
dapdepth evolving
"coastraints":
"Add ona more signiricznt constroints requiromente into the 'raiven Promp:t'.
*The added conatrainta must meatingfully alter hew the model would swapond.
*For example, specify additional rules, contexts, or linitationa that denand creative adjustmenta the resporse."
"dsepening";
'It the #diven Proupts contains inquizies about costain issues, increase the depth and breadth of the Lnquiry,
detailed, milti layared, or curgrehensivv responae. the problen into aub-problnaa requize connectiona betveen. unselated concepts." "Por inscance,
MMake the questiin ion require
Repncret ining"; !
"Beplace general concepts in the #livan Prompt with more specific and detailed concepta.
probles defined corcrete, leeving "Po example, replace devicw with la veatable filness Leacher GPS"."
shat change
anbiguity.
),
"reaxoning": !
Addd
reasoning sbaps into 'EGiven Prosgla',
"Expliciily zevzita it demand multi-step reasening jeacity internediate acepa the solution. "ar sescance, if the original prougt 1s single query, make the responae reguire step by step breakdown of logie or saloulations..
Mexgansion": I
"Espand the #alven Prompla by including adeicional pesapect ives, duminn, us layera of cuppexity.
"orr exarple, 1f the original pronpt focuses on single scenario, add relsted scenarios or ask the model corpare di fferent sstuet
INST DEPTH
"Blease Not ae an expert Prompt Reuriter.
"Your objective iee is to sewrite given prosp into more complex version
"to nake thase
languege models le.g., genini) bit harder hansle. An"
rewritten proupt
reasonable
responded by
"Your reariting cannot onit
nen text parta such "You should try your best not to make the Revritten Prompt becone verboze,
table and in #Given Prompi 1f these any."
"The Revriizen Prongt should be roughly the sinilar length little bit than that of velven Prompt 4.A like real human user's prompt; DON'T make sound machine-gencratos."
#eewritzon Frompte must "Speciticall ly, you SHOULD complicate the given prompt using the tollowing metnod: mnethehood va 4 to be formtted
"The revesttor.
structure, sutticiently ditterent tron the original, "Again, nake eure the rexritten pronge presenta more CHANMINTTN "ASK," "Respend with your cevritten pronpt disectly.
should rerlect neaningtul changes
"ensuring
sentence
"#Siven pnippept P'n". be fornstzed
"theuritten Prorpt
) siistrip(
DIST_IN_EREADTH
"Please sot at expert Prompt Dreatoz.Art
"Your vbjective is to generate brand-nex pronge based on the #Giver Pronpté.
"The purpoce of this task is to promots diversity and gonsrelity of training prempts for language modcle, "helping practice with varied challenges Prorpet
perepectives, ie"
complexity
Prorpt&.An"
"The #Craated Premptt must be rezsonable, interpretable, and solvahic by humane. An" "The #Created Prompt most sound like real human user's prompt: DON'T make it sound like mashine-gennrratoo..
"Follow the nethed described "Inethod/Ae" t
quide creation:
formetted
"The crested prorpe should retlect meaningtul changes across its structure, "ensuring the entire sentonce feels sufficicatly different fron the original. created prompt directly.Vn* "#3iven Vnopptt:/niprompp "#Created Foompsfika" >.istrip()
"Respend
tormttees
17

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
G. Additional Experimental Results
In general, eva maintains the downstream performance and robust on reasoning- heavy tasks, and the scaling with reward models more prominent on AlpacaEval, possibly due to training sources for such reward models.
Method (L) Dataset
MUSR-TA
TruthfalQA-Gen
WMDP
GSMSK
GSM-Phus
MMLU-Pr
8g
0o-st: DPO
38.40
34,76
58.45
24.56
18.50
$2.63
0o-11: SPPO
60.80
31.15
58.72
24.79
18.42
52.70
2-42
34.64
58.94
Table 9: Performance on Downstream tasks.
Benchmark (-+)
M'T-Bench
Arema-Hard
AlpucaEsal 2.0
(1)/ (-))
ang. score
24 tum
WR.%)
(%)
Mo-e1: DPO
8.66
9.01
8.32
$1.6
$5.01
51
(ARMO-BE
8,90
8.75
60.1
55.35
55.53
eva-11 (SKYWORKRM--H)
8.73
36.12
56.40
Table 10: Effect of (pointwise) reward models.
Benchmark
MT-Bench
AlpacuEEvll
Method (() Metric (-))
score
1 tum
2 rurn
WR (%6
I.C
WR (%)
Ao-g SPPO
eva-i (PAIRRM-0.44))
8.78
9.11
8.45
58.9
51.86
43.04
eva-i (PAIRRM-SB SB)
8,89
9.08
8.70
60.2
52,71
44.52
Table 11: Effect of (pairwise) reward models.
H. Additional Experimental Results (as Extensions)
H.1. Experiments on Different evolve (.)) Methods
As an addition to Table 2, we have experimented with three different evolve()) methods, including:
SelfInstruct (Wang et al., 2022): Given seed prompts, variations are created based on criteria such as verb diversity and style blending (mixing interrogative and imperative styles). Unlike EvolInstruct (Xu et al., 2023), which generates prompt variations sequentially, this approach generates independentl. We follow the one-shot implementation in self,instruct. py of distilabe===1. ...4. 4.1 and modified the instruction on conciseness so that those newly generated prompts have similar lengths compared to the seed prompts.
EvolQuality and EvolComplexity (Liu et al., 2023b); The two methods use the same evolutionary approach (i.e., sequentially generating), but with slightly different meta-instrustions for prompt generation, where EvolQual- ity asks improve the quality (i.e., helpfulness, relevance, etc) of the seed prompt and EvolComplexity asks to improve the complexity (i.e., increased reasoning steps, etc) of the seed prompt. We follow the implementation in py of distilabel==1.44
evol mool.quaitty/uill. .y and www.eeoooleeett///yeeeeeeee
Panily (-1)
GOMMA
Benehmark
Ammns-Hant
Methed (J) (Metrr (-+)
WRR (56)
sng. Inn
lovelield avolll (xpipcic))
700
levolveld wed. sisattrccc))))
725
Table 12: Results of using different evolving methods.
18

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
eva is effective under different evolving methods. As shown in Table 12, method brings strong performance gain without training with additional human prompts. Among the experimented methods, find EvolComplexity shows better results.
H.2. Experiments on Number of Iterations
As addition to $ 4.2.4, have experimented with the following settings:
10K prompts per iteration with 3 iterations.
20K prompts per iteration with 3 iterations (i.e., all seed prompts used).
60K prompts per iteration with 2 iterations (i.e., all seed prompts are used).
Due to time constraints, we did not perform an extensive haper-yaramenn parameter search; however, we believe the results presented below sufficiently demonstrate the performance gains achieved by eva.
Model Family (-))
GEMMA-2-9B-M
Benchmark (-+)
Arena-Hard
Method (1) /Metric (-+)
WR (%)
avg. len
0g:
SFT
41.3
544
80--1 DPO (10k)
51.6
651
0 01 DPO (10k)
59.8
718
0g >a: DPO (10k)
61.2
802
0 .1: eva (10k)
60.1
733
0 0i- eva (10k)
62.0
787
0g pa: ava (10k)
62.2
774
Table 13: Results of using 10k prompts per iteration (DPO leegth NLL loss).
Model Family (-)
GEMMA-2
Benchmark (-))
Arena-Hard
Method (L) / Metriic (-+)
(%)
avg. len
06:
SFT
41.3
0
DPO (20k)
$3.2
625
0 1::
DPO (20k)
47.0
601
0 8
DPO (20k)
46.8
(20k)
:2:
(20k)
60.0
03.3: (20k)
61.4
791
Table 14: Results of using 20k prompts per iteration (DPO length-penalized NL.L loss).
Model Family (-))
GEMMA-2-9BB-
Benchmark (-)
Arena-Hard
Method (L) MM//Meiic (-))
WR (S)
uvg. len
0a: SFT
41.3
0 DPO (60k)
0 (60%)
59.6
0 0 (60k)
61.9
792
Table 15: Results of using 60k prompts per iteration (DPO length-penalized NLL loss).
eva can bring robust gains with multiple iterations. As shown in Table 13, 14, and 15 below, our method presents persistent performance gain over iterations, and concretely surpasses the performance by default DPO training with true human prompts.
19

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
1. BONUS EXPERIMENTS ON rewriter (+) IN THE LOOP
We present the basic idea here for practitioners build upon. The motivation comes from the hypotheses derived from H.2: the prompts gets harder by evolving, there may be greater demands the solver's capabiliitis compared to earlier iterations. As such, the solver may not be naively treated the same. One may address this by either scaling up response sampling introducing meta-inse veta-innttuuctioss to explicitly enhance the solver's reasoning.
We design a proof-of-concept experiment w.r// the latter by adding rewriter in eva's solver step. Previously, as in Algo. I and 8 3.3.1, for each prompt x, we generate multiple responses, and choose the best as y+ and the worst as y- for preference optimization. Now, we add one more rewriting step that attempts to enhance y+ to be y',, . by applying rewriting instruction (Liu al., 2023b) that asks the solver to alter y+ with imporved helpfulness, relevance, reasoning depths, creativity and details while keeping the similar length. We then train with (x. y-) for preference optimization. Table 16 shows that adding the rewriter yields concrete performance gains over the default training method, while keeping the training budget and only slightly increasing cost for offline data generation.
Model Family (-)
GEMMA-2-9B-IT
Benchmark (-)
Arena-Hard
Method (0/MMtric(
WR (%)
avg. len
0:: SFT
41.3
544
0 DPO (10k)
51.6
651
0 0
+ (10k)
60.1
733
0 1-11: eva with rewriter (10k)
61.9
741
Table 16: Results of adding rewriter in the solver step.
H.3. Understanding the Informativeness Proxy in Different Intuitive Ways
Learning potential. Our metric intuitively identifies the learning potential of prompt by measuring the gap between the best and worst response to it from the solver. We reason, that prompts eliciting both high-reward and low-reward outcomes, reflect learnable tasks where the model is capable of improving but has not yet mastered, thereby implying learning potential (of., (Jiang et al., 2021b))
Worst-case guarantees. The minimax-regret objective, by design, leads to solvers that perform robustly across the prompt space, thus gives the worst-case guarantee. While exact equilibrium may not be attainable with approximation, our empirical results in 4 demonstrate robustness.
Auto-curricula for the players. We visualize the curriculum induced by eva in S 4.2.5. With the stochastic policy, the advantage may be heuristically understood as the reward difference between a base solver and a reference solver. Rather than optimizing separate solvers (Dennis et al., 2020), we sample multiple times from the same policy to create the pair. In this way, the creator is incentivized produce new prompts that are just out of the comfort zone of solvers (Chaiklin et al., 2003):
For overly challenging prompts. both solutions perform poorly, leading low proxy.
For overly easy prompts, the base solution already performs well, again giving low proxy.
The optimal strategy is find prompts that are just beyond the solver's current capability.
Auto-curricula inherent to Contrastive Optimization. Contrastive preference optimization generalizes DPO and a family of algorithms (c.f. (Hejna al., 2023; Rafailov al., 2023; Tang et al., 2024)), many of whose losses monotonically decrease the contrastive ratio increases. To be specific, the DPO (Rafailov et al., 2023) objective for RLHF is:
(T)) =
Σ
- log ( 28-24 -
(y+ X)ED
where we use +, - to denote chosen and rejected responses, and denote the contrastive ratio as:
T8 (y+
x)
T (y- x) x) Wee (y - x)
AX
log
log
(8)
Tref (y+
20

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Here, by Table 3 and Eq. the confrastive ratio be written via the advantage-based proxy:
Amin (x) 3 A :ell rel
By our proxy, we implicitly incentivize the creator to generate prompts that bring the most contrastive responses, which decrease the loss the most. This matches the curriculum learning literature, which prioritizes (in eva, generatively
prioritizes) examples with smaller losses for better convergence and generalization (Bengio al., 2009). We hence suggest the Contrastive Curriculum Hypothesis: In contrastive preference optimization, prioritizing prompts with higher contrastive ratio improves sample efficiency and generalization.
We show initial empirical results on this in $ 4.2.1 and $ 4.2.4.
21

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
I. Examples on Prompts and Model Generations
The initial prompt distribution of UltraFeedback by bart - Sett--arrge-nn--- with 0-shot classification, which is imbalanced. For the reward distribution, the category with lowest average reward has the highest reward gap (i.e., the default informativeness proxy), implying the potential to improve.
Figure 11:
Table 17: Baseline examples of evolved prompts from AlpacaEval.
initial prompt + Write ne the code for distribuzed transaction manager- pseuda firet, Anthen, define incerfaces for all involved actirr and actities. Rational Unified appzoach for this part. laltolay than nove on to the acteal implementation, cixes-by-elast, oode be impl enented in C4 and follow 801TD prineip' es.
step by step
methos-ky-method.. \ntake
(ntakk
evolved +1 + Craft suite of systax for distributed transaction coordinator. Start with sequential breakdown pseudocode fornst. Pollowing establish protocela commincaation and interaction amongst the various participants and compenente, incorporsting the Rational Unifled Process nethodology bryyrrocedd thereafter to conerete oseation, detailing each that the final Ct code adheres to the procepts of SOLID and is arnotaced for elerification and maintainabiläty purposes. vedlved Devise comprehens distributed transaction cocrdinator architacture. Initiate by formulating step-by-step Frameuo-ks dialog cooperation between the numarocs entities and elements, utllizing the tenets of the Rational Unified Precess nethedology. wtw.coooooy.llllaooott. www.laonoonin to the actual generation
function. Ensure
direct ives
Subsequent ly, delireate algorithmic decomposition in pscudocade.
code, meticulously detailing every and corresponding nethod. Guarantee that the suiminating CF code is in strict compllance with SOLID priaciples and is supplemented with descriptive commentary while 41s0 validating the code againat set of unit teats ensure robust functionality.
future elerity upkeep,
Commence constructing algorithm pseudocade net net ously breaks down the sequential stages for dist-ibuted transaction coordirator systen. Progress by outlining the commnication prozocols collaborative mechanisme among nyriad components, adhering the Rational Uni Process appseach. (elodvaacc to crafting the coda in Sk, ensaring each class mothed is elaborated with precision, aligning design principles. Integrate thorough coments for heightened meintalnability and conduct comprahensive unit teating for assured opezational integrity.
SOLID
evolved +4 in cloud-native environment. Begin by datalling the service dissevery, load balancing, and fault tolerance strategies, whils incorporating the Devúps philosophy, Aeapphy-niinProceed by developing the implementation Cava, focuaing clear, nodelar code following objecttoziented progsamning code with clesz documentation for essy unsersstading and perforn rigorous integration testing to guarantes resust
Design detailed flowchart that represents the intricate steps of an rat engine for managing nicroservines
practices. Acconpany your
22

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
1st-turn prompt:
the number divided by 10, remainder What divided by
rameinder
Answer from -p9
Answer from gema-2-9b-1tt-po-eee
Neee''s how to solve thia problem:
Hece'' how to solve this problen:
1. Understand the first statemenz:
1. Underatand the flest statemenL:
"When number is divided by 10, the remsinder is 4" means the number can be represented as 10ck + 4, where k is en inzeger.
"Khen number is divided by 10, the remainder is 4" means the number can be represented as 10ck 4, where k is az integer.
2. Double the number:
2. Double the number:
Twica the number is (+000+ 41 = 20+k
Twice the number is (+0+k 41 = 20+k
3. nivide by 4:
3. Divide by 4:
(20+k 81 divided by 4 is 120+x)/4 8/4
Nhen (20+k 8) is divided by the 20ck
5+x
tern la perfectly divisible 4,
4, Find the semainder:
The inmeinder will the as the remainder when is divided by 4, which is 0.
Since 5uk is multiple of the remoinder when twice the number ia divided by is 2.
Tet me know if you'd ke to try another example!
Therefore, the remainder when twfice the number 1s divided by is D.
2nd-turn prompt:
the
divided by
Answer from gemma-2-9b-it-
Answer from gemma-2-988 -at-------------eeee
Here's 5 how to solve that:
1. We know: twice the number is 20+k +
sere's how to solve that:
(From the pravicus calculation).
1. Twice the number: We know :xice the
2. nivide by 5: (20+k 81 dividad by 5
number la 20+k (from the pravioua steps).
(20+k)/5 + 8/5 4*k + 1.6
2. Divide by 5:
3. Find the remainder: Since 4+k is multiple of 5, the remainder when twice the
20*k 1s divisible by b.
* divided by 5 Leaves semainder of 3, Therefore, the vemainder when exice the number 1a divided by 5 is 3.
number is divided by is 1.
tat re know 1f you have any more number theory puzzles!
Judgment
Judgment gem-a-2-9biit-dpo
Judgment
The assistant's response incorsect.
The assistan: incorrectly calculased remainder twice number divided by assistant calculated
part the visior operation, the remat inda. The correct caleulation should (8 8 51
The asaistant's answer is correct matchea the reference answer. The assistant correctly explains that twice the number that
(20n 8] (20n 3, Therefore, the remaiades when twice the
by is 3, not the assistant stated.
divided by 5, the remainder is 3. assistant's explanation la clear and eaay understand, making it helptul for the uaer. There are no mistakes to correct.
The corrected response would be:
1. Ne knows Twice the numbez 20-k
(Ezom the previcus calculation).
2. Divide by 5: (20 81 divided by 5
Rating: [[10]], "score": 10, "curn": 2, "tszamp": 1722846518. 5543888
gives us a remainder of (20+% 5) (8 $ 5)
3. Find the remainder: the remoinder when twice the number in divided by 5 in 3.
Reging: [[3)], "score": 3, "turn": 2,
"cstamp": 122233111..76890
Table 18: Comparison of model generations on examples from MT-Bench.
23

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Inaqine you ace veiting blog poat compazing tvo popular anactphone nodela. Develog an outline for the blog post, including key points and wubiowttnng to attectioely purpace and cont the festures, parfomance, and user experience the tvo rodela. Please andoee fever than 200 words,
lst-turn peompt:
Acswer Zzon -urma-2-9b-it-- -b-
Ansver from 2ama---9b--t-dporo
I iPhone 14 Peo va. Sarzung balaxy 523: Which Plagahip Beigns Supronatpplteet introduction-lyy heietly introduce both starpphines and their poxition in the narter- State the purpose of the corperiaon: to help readers deaide which phone best soite their beeda.Anln.eit Design and Displaye</p Physicel dixensions, weight, and build quality conpactson. Anr Bezcea slze, reselution, zefrish rato, and display toshnolocy (OLED vo. Super AMCLED) Aar Design asstaeties and unique features (6.0., notch va. punch-hola, color options). /oloeill.. Pazforrance and Canszareijs Procuszor comparizon (Apple Als Bimie Snaptragon Gee 20- RAX and atorage optione. An Cerera speas: repapixels, lene types, sperture, and spectal features (e-g.. telephoto, racro), Ape Trage quality comparison bazed sample shetos and usez revieva, revieva.jelavvd. Boftware
+ Snartphone Showtoom: [Rode] A] vas. [Model
Rf (nlppet,
onseln Rriefly Introdune the tvo enartphone models being compared, highiigghing their popularicy and targst audience, lor State the pargeso of the corpacison: to help readers decide which phone best suits their (ola--II Design and claplayrlas Phyoical dinenaions, wiight, and build quality compariaon. you Sezeen size, reaclution, and display technology je.4., slm, (o))-eer Design sesthesica and user proteramews.oolmm Jolaretl,, Perforsance and Spenaeelpe Processor, RAM and storage emparizzon.pp Inn
Benchnark zcores and real-orrd perfomance snalyels. Rattery life and charging beantiiites.ipeeeeee Camera Capabilitioaslpe hear and front canara space tragapixels, aperture, seaturen) ins Image qullity comperi von in varioux lighting conditions. Aps Video recording cxpabilitlles Intpe Sortware and User Eepertences inn Operating syster and user interface congarison.
and User Experiance Operating systems: 105 16 Androld 13. Ina User interface corpecizon: intcitiveness custonization options, and app ecoxystens. Ans Special festures: Face vs, in-display ringerprint censor, voice asslstants, /rinee, Arinew, Rattery Life and Charging- Battery capacity and seal-world usage corparison. Ans Fast charging speeds and sireless charging eapabilitics. AalareiI Pricc and Valuerrias Compare starting prices and different atoraçe contiqurations. Age Asseas overall value noney based features and pectormncee (nlarwis, Conclusionr Summarize key disterences rengthe the poone specitic oeer types (e-g., photography enthusiasto, cusasa, budger-eansciuus consuners).
eoosysten and avattability.pn Software festures and custonization options. (nla--V,, Prios and Valuewllrr Coppare the pries points of both nodels. Jax Analyze the valce proposition based featurea and performance. Vale Cenclusion=<lle Summarize key cifferences and sinilarision Recermand beat phone apecitic usez pretiles 19.5., pudget-coneiinu photography enchaxiasts, gunare).
phone. Recoseed
2nd-turn peompt: Take your previoux zeaponsw and zephraze it as limerie..
Anaver fron -p--
Ansver from gemma-2-9b--te bb-------------ee
Tvo phonee, shoxlown so grand/ (ilede Atl and [Medel B), hand in hand, JnDestgn, epens, and peice, JnHer11 analyze nios An7o help you choose, underatandalalntet ne know if you'd d like ne to try filling in the bracketed parts with apesifie nodel namcs!
AnTHo phones, Pro and Gelaky, AnFor flagships,
they truty are spacy, spacy.AnMith AnMich careras and coreens, Antad peacessora keen, Jobbich one vins? Read on, be savvy! Vatez as know i2 you'd d like anethoz Limerick!
Judgrent opt-4]
Judgnent for goma--299--t--00
Judgment foz goma-22-9011t--po--o-
The aesistant's x
reeponse is creative and relevant to the user's reguest. The user adted the anstament to rephrase the precious zeaponse, shich vas detailed outline For blog post, into limerich, The amsiatans succenstully did so, meintxining the essence of the griginal response in a Fiterick formtt The assletant also offered to
The assletant's responce 1s oreative and relevent to the caaz's a requeas. the assistant saccussiully rephiased the previous response, which was detailed outline for a blog post comparing txo snartphore rodals, into Linerick. The Hiwerikk le playfut and engaging, and it nelntaine the essence of the original resporse, whith is to corpare the two phones and help the reader nake an informed decision. The accistant alse offers to areate acother linerick it the usez vante, showing willingness to contiaue the Intercation.
forther custonize the Hnerict by filling in the brackered parts with spectfic nodel nanes, showing a willingress to provide additional detail i2 peeded. Bowever, the Linziikk does not fully capture the depth and detail of the osiginal zesponse, which is understandible given the constsoints of the Limesick torras.
Ratisg: [191), "score": 3, "turn": 2, *tatamp":
Rating: ((811, tscore": 6, teuraf:
2, festang":
1722846542, 9431995
1722731104, 0327556
Table 19: (Continued) comparison of model generations on examples from MT-Bench.
24

---

Reward-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
lst-turn peompt: Kach problan conaista of three stacements. Baxed on the Ficas two statezenta, the thisd atatemens nay be true, falln, or unpertain. (pk, Granges cost more than applex.An2. Granges cost Laza than zamanee, Apá, Bansmax cost more than applns and bananwe cost more then amange, Antt the first two statements are troe, then the third staterenn 187
Acaver Eron qurma-2- ib-i
Anaver from 2amna---9mmit-dpo
laree's how to break down the lngiet/nlo s*Stacement
Lace's hov to breas down the logier/elps seStatenent
live Oranges > Apples (Dranges post xore than appllest Ins reStaterent 2iee Dranges i Ranaras (Dranges cost less than bananas) (ryoThle reans the price order 181 Rananae > Oranges > Applesinlns +*Stateeent Sixs Bananas > Apples
lsee Dranges > Apples (Oranges cost more than apples) (es eeStatement 2iea Oranges < Rananaa jOranges cost Tess than bananas) (nlttes statenenta tell us that bananad are the nost exgensive, ther, oranges, and apples are the least expeasive. jnice **Stacemen: Diwe Baranas Apples MD Binanas > OcangestahcsSnnce bananas aze indeed nore expensive than both apples and oranges, statement 3 is .rraa.. Valehatat ne know if you'd a Like tay anothar one!
ARD Pananas Drangeslainstatcmmnn: is espartially
cruzee. Adip- The paxt "Bananas Applcss is tzuc based on oux price erdez, lar The past *Bananao > Oranges" is also true based on our price ordar. Jalalazat me know if you'd d like to another cne!
2nd-turn prompt:
staterent
crue, falas, uncertain? Please explain.
Ansvez fron -um--2----
Answer from gema-2-9B-it--ppreaa
the third atatement is trus, than the Firat staterent semoertainnes cammeetaiiie..pllliiiii Jelpllerets y vagrlelps seStatament Joee Gananax Apples Am Banamax OrangesjpleThia tells or hananes are the rest expentlve, but it dowen's explicitly state the relative prices of apples and oranges. Jnlpe **Statement Live Oranges > Applestelothle statesent cleixs oranges are nore expondive than apples. Anlastateeet 3 decon't comcradiet statenent 1, bet it clse docsa't confian It. leta possible that orangeo varas noxc expensive than apples, but atatenent 3 alona doeac't eive an enough information to be auze.
If the third anatiment brow, the first staterent is astalzese, Anknlleen' e shgrlnlps veStateeentt 3:.e States that banense oost more than both applee AND oranges: Ielothie means the price order 1s1 Ranense Oranges > Apples. saateeeen: 1ie. Btates that oranges cost more than apples. Anyate bananas are the most expensive, and oranges are secand, then excnnes carnot be nors expensive than apples. Anlalntet ze know if you have any sthez logie puzsies!
Judgrant gpt
Judgrant 2pm
Judgmant -bbb--------
assistant's reaponze Lncorrect. the true, statement only
assiatan's zeaponze is cucrect and helpful, axxistant explaing
stabement false, llooweer, that bananaz oost sore than apples and cranges, It does not provide information about the relative costs of apples
is uncertair if third starement true, aligns the reference ansver, asstatant provides olear explanacion shy this is the case, stating that the third stazesent docsn't contradiot statoment, but also docac't conflim it. Thereforo, the asoiszanc's zeapenso is accurate and informative. Jatalto assissant's a zespenao: then the firas statemect is runourtaines. Bare's shys Statement 3: Dananax y Applas AND Dananax Oranges. This tallx us bananax are the rext expenzivee, but it doeentt esplicitly state the relative price of apples and oranges. St Statement 11 Oranges > Agples. This atatement clains cranges are zcre expensive than apples. Statement dowsn's contradict statement 1, but it siso dowen't contim it. Ie's possible that oranges sarea more expensive than spples, but ststement 3 wlone doeer't give Us enough information fon to be sure.
oranges. Therefore,
the first statement could be true (if oranges cost nord than apples), folse (if oranges cost loss than appleal, or uncortain (if erangea and apples cost the aama). The assistane'a a explanation is flaved because is incorructly assunes that the ordez of costa must be Bananax y Oranges Apples, vhen in fast the order could alan Dananex Appless Drangex or Bananaz > Apples Oranges, (pirGornected stnt's Ansver: "f the third statemen: le true, then bananse cost nore then apples and bananas cost nore than orangea. this reans
first
the third stacemant is true,
that cranges could still woes more than applus, making the first statemee: true. Eowever, lefe alse pozsible that oranges and applee have the sore cost, which would nake
the first statement falee, Therefore, based on the th statement alone, the first atatenent is unoestain.
Rating: [110)), "sooxe"; 20, *cuse") 2, "estarp": 1722846027 2333696 22339694
Beting: [[2)]. "scors": 2, "tura": 2, *sstarp":
0922737017.280855
Table 20: (Continued) comparison of model generations on examples from MT-Bench.
25

---

Rewdrd-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
lst-tum peoptt: tine complextoy.
Implasent a furctior to cind the radiac too sosted accays of diffacent sisea with 0:31 space complexisy and ond
Acamez teem quana 2 it dpe
Anseer trom geana 2 26 it opo
pethonlylyeer rind in sorted arreys inins), noMsR) mnyg Finds the racian of to acsted arrays of dictacans inke
Fythanlnlded patpsaattt.aaaaaaarraaaaaa
naes2) i/n ralls Finde, the rediar of two sareed arraye of diffecent lole Ange-in ninsl: The fleee corted arisy-ln ninss: The recond sorced array-bnje Roturnsiy the andian of the Two soreed arrays/ seyolt is in - teninorllly con if v nrle nunal, russit an a nunat, cumt, by a Enaure da stnye lein, inss, bett.ien by or 1 21 // 230a shile inir. de in lixin ieast // 23n 1- helsiden Al IE 1 < and maudl) momallllitn inin 1. F too 2nal,, increem it'n wilf L D and zumst]i 31 mana2lllila inex -
Aegeiin suarl: The firet sacted areay- in suaz2: The sscand corred array. Join Returnsiynn The redian of the tuo corter arroyoi meppli is oi "endooents, ten 1eo Sf nrlp monole susse, to - ninos ross, fo I Rosure n se olote onin, lass, hatczlon 2 Ny Oa+ n - 11 1/ 200 velie imin on inoerla 1- tixin taa // 23n 1 - hel
An it i' and nam211 10 > mo Loin is too srail, inceusse 1oto wiid > 0 and munalll 11 x to ias = - 1 la big, decenaze itlo atmesin in pacteceiaio ie e Doln maxiett naraltt 1l50 alld oita maxders a suarllt - 11/0 steeste naxiads
too big, decrease inlo aizerin I in partectielo if 1 02 rasilats nune211 3150 aide 5 in Dijn soulete nunalli 1310 elzeijo soslete coseeddd - 11, nimatt A1lloye if 2 -n tite reture noclerrtole ie - mije sinirihh
eaxcomel 11, minett - 111 je in ( - 31/0 necurn toxtorell in 1 -n wite vinirigt meeorflle
elle nito wis right my eloeite rigrigie sintumall))), smallilllola rature maatiats sin.eighs) 2.01pint Eeaple stage/oouust B. Manuur 1APaaaddann tindgediian cowperearrrrrgrrrgggrggg (rouet, noneltyggront (rethe nedian of {nowel) and (moee)) tor poppsn/poopppposolaaaaoo in thegual tengrhereege If "unall is longer than trower, svep
nonortylte elrr
sinsigte pil eloerte
nininona: [41, nanattt retarn (axcleft rin.right)
2.flolor Exseple usagelnnuee R, 31nnaree 12Iapracten
arndeedian.eoreeeerryy (nonel) and (nors)) foi Sesttr/pSotppleeetaaaeee
mine2 Vegwrin (hr andian
exkanile thegial tengthoresj 1F "rouet" is longoy than "noees seap
"horsl" alvayo the acrey, Tais
"nundl" alsays sterter array. This
sueplities togin. Aoloz. vetinary Bearcheeely idea hinery seareh, ahoztez actuy Courrt An "imas" oecine point 11" "s "baifaien" cnlcuiates dasized total number alensnta
sirgiiiies the Anlez veBinasy Beazmosely
is use binery seurch shortes areay ('oussl") .n "Lain" "Iaas" detine the swarch cange
cange partition
point "maas" An "heifaien" calcuiates the deaured
aids the parcition each iteration, 1
numbar elseance
of the pastition each inaration,
redias) Anted, lo divider "musel"
mediant, Aplas, seaessstoningssil civides tounet '3" dovidee "ouue2" total nomer of elesenee on the left sice wale.lenn to0ee. Bangursely
partel "Dars troued in) Ao '5' in ostcutared of de tnones" to Anlee, colduuating
coral nomerr
simeenta on the best pa te cospace slenensn at boudariia
ynddinating
"nuradts cas
12 seall, so reed to nove partstion paint "ouee" right Clain 111. "nanales large, ao
parsitions ie 11 in the correct pasitianils trune211 1:1e In toe restl, now "Iaia"
13 nanadEdll
the
15A ts 11 10se2137' Antes werinding the Medianseets correct position, ve findrln "a
large,
Cbax 2000es extinding Hedtanieely
correct parettion to foold (1F io the right postrioo), findrtn "eaxcleee" Tho Paximue elewert on the left side the partition Jn "minueightt The ninimcn etsment on the right
maximu alcment in the Jeit partitiora of both arcaps An "minarighte is The rininus siement in the right partition boch un the total lange is cod, 'rasdett" is toe aedian.la 22 the total lungts is evan, the mutian ds the average of 'ranilte'' and 'ranuright Zize Coepiasity:
sice of the partitionn An II the Loie. mamer et suerenta is eoc, "mondett' the radian.In II woen, the redier ca the avezaqu "manieed" and 'ninaright" 1 Tira Corpbadday C (aipinin,,, m)1) because sa are parterring hinaay search on the aboster areay, dapac Cosplasity 0f31 as ve are only using conatant snount of extea space for variablee.
ottogistnie, ni); becauss ua parfors binsry swarch on the sharter
Sosplecity (()) as un only usa a constars
anoust of exize space Der variabiees
Jodgaan: byy 4p:-4)
Jodgnast for 9p-t-2---
Judipennt for greas-2-----t------uux
aocletant's saswer in correct and provides detasted
explanar of the cods, The asd stant cprreotly states that the complexity otloginints, 1133 and space conpexiit to oil,, aligns
esslatant's answer Lo correct proci detailed explination code. cspiatant correctly tine complusity oriegininos 2131 and space compoesity 012), alugns with rutueente answe.. The casistint prevides stop step supianation cods, vasy helptul underssanding the legic behind aolution. ansierancta code scceest will correctly zorted acraya ditfersnn aszen. Theesfore, coerectiona ressed for usziatant's zeaponas.
reterence answer. assistant halptet ciirecent
provides exurpte zaage tunction, underatanding
it. aszistant'a cefarunce snaves, actiooes the auw ceauts and seata the aaze coeplesity requicerants.
inplcsentation
Rovorns, assiacane's ansver does not address usects origins! quastion about achseving Bin) complextey. aszistant ahould stacicted thst
passible
pesias of aosted ascava of difeneent sissa space
coeplasity cine complexion, stated anaver.
Racing: 1(1013, *acoze": 15, Peaznts Teatamo?: 1722146535 772081
Raring: 11811, "scoceti Peursfi
Textang";
1705751514 3001000
Table 21: (Continued) comparison of model generations on examples from MT-Bench.
26

---

Rewdrd-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
lst-turn peompt: You ace giver two socted Lista of aize n and n. implemanc function to find the ktb analleas
elaxent in the union of the two Lista with linear complexity.
Acower fron germa-2-9 -po
Answez from gomxa-2-90-tt-dpooay
"ppythonlyydeer rind kch snallest Mett, Rrd man Finda the kth mallest elament in the union
arr rinda sindketa,mmileett (listi, lists, R) snallest elenent in the union tho sorted Hate, Angerin Hatt: The first sorted Mett 119t7 The recond sorned Hsttin
""pyshorlojjee
of two sorted listaclple Argecin limi: The tirm
sorted Hattip Hand The secund sorted Ap kr The desired kth sealleer elleent
Returner)y
The kth sxalleet etenent in the unton of the two in vrryal in n i len (liellly (en(11st
destied rask of the snallest elenent, stenent-Voly Voly Returnsryn The kth anallest element in the union the 1ista. werl 1 - Index for listlin 1 Indox foz Lisczia mexged - [I Score the nerqed clcmencz/oja shile (inoti and (antliat2) AA 1I lisei]11 11at2(1): t/n meroed. appera einestn appand (5 1 t- Dale Add any renaining elamenta fron listiyn while i len ( sye aeogeddappend (811 1n Aete e Add any
Is 3 i Dy Cla ninchep (I Aotn i Build xin-heap with the firot clarents of both lista)a for x in zanne inin in, alle]o hwwpppppppuuaaiiaoooooo heap, (1i X, 0)3 I (value, listiindsx, toalt/o
1/a
inin.heap, (11st2(H), 11:jeee +
Extract k anallest elerenta fron the beeple for in
heaps, pespppppppppptnibnegno The top elerent of the heap is the kth ir return heeppopete heep) rolp Exopll
rangele 122/p
reraining elenente from Hart2/n white 3 < Ten dhe appeed apoend (1
t- Aeta retura narqudjk 1] Recurn the kth element (index x-11 po]ed sxappee exagecynlisti - [2, 5, 8, 12]jnlatt2 - D. 3, 7, 9, 101]nk stelpichumalleet - #innkkhhmmllles (1let).
unigeiipliati 12, 5, 8, A2/lala - 11, 3, 7, 9, 10/jpk - mnalth tr (liatl,
1ixt2, legerin tertow [k]th mallesst elazent txt (cthhemateeen hogprl/laaatlllaatttoooooo
11tt7, kroprren cfeehe [e]th lest etement thgp::::enoottoooooooooooo:
intnntsal//atttoneell ' and 'n' store the tengths
1sr
or the inpet liste//n '1' and 12" pointers to track the current positicn each lsst./n ''ninheap''
inn - 11" and 13" are index pointers foz "lisel" and "1lst2" respertively, starting at 0.in "morged" is an enpty List to atore the nerged sosted clasento. setezelngs - The "while" loop continusa an long as both 11" and 'p' aze within the bounds of their sespactive Lista./p - in each itaration, it conpares the slenente at 'limiili]" 1 and (1itt2(5) The stallas elenent is appended to 'merged", and the corresparding indox (11' ar 1971 is Lnarenented. /alaa, eadding Baraining Eleminta:*<aa - After one list in exhaustad, the coraining eleminta from the stber List are directly appended to "rarged" Aelos, celleurning the kth ilerett - rinally, 'rengedik - 113 is returned,
nin-heap used to officiancly
and retzieve
ancllest elarents cocourtered far, latna, r-Building the Mia-Hoap:eela Thhe code ittrates through the first 'sinta, la, af elerants bota lista./e elezent, tuple "/value, liatainees, posstion.saliat)"
each
pussed onto 'mindhap''
tuple helps belongs
keep track of elarect's value, the and its oxiginal pesition. jalas, veBetrctiing smallest Elenatssssl
'for" lcop 12
times asfentively extracting the "ke1" anallest from 'minhhepp' jelos, exrinding the kth mesllestieslp after the loop, top elasent tel hhhppp cortains nkteh melleest eleren Ar "heppgg heappop tteppe renoves and returns this elexent Inlaeetine cooplextty Boilding the nin-heap: ocn- na Extracting x-1 elenentas 0.k 100 I Vo- Overcll: ++
which represents the kth stallesr stenent in the union (remeeber that 1ist indices start fron d) Aolpestineee Corplexttyi<ennte The algorithn sterated through each elarent in zoch lists at most coce daring the nexçing proceso. Ao- The time complexity is therefore Ota a), shich is linear in the total aurber of cienents (6. nt, naking it offlciens for lazca lista.
(01 which
appeexinately +0112 + **** since 't' useally nuch analler than 'n at Ahbeiplttee ne know you have any othez quaationa!
Judgnent tay opt-4)
Judgrent for qum-a-2-9--
Zudgment for -aara---bbbii-dpo-ee
The assistan's unaves in cornect and helptul. It providex Pyshon Function that implamentx the txo pointer approxch to find the keb snallest elenent in the unionn of two sorted Hats, The asslatannts fonction is siightly different fron the reference, but achleves the goal. assistant'a function nerges the lists into 1ist and then returns the kth elanent fear the norged list. The assistant also prevides detailed explanation of hew the Eunesion vosks and confirma thas the tine corplexity linear, waich what asked for. novever, the assistant's sunction extra apace to
The aestetant's a anever is Incorrect, The user asked for a solution with linear complexity, but the asslstan: provided a solution with a tine complexity of oir i (kg y which is not lincar. The assissant's solution usea nin-heap, which is not aecessasy and adds uanacesoazy cerplexity to the sclutioe.
Rating: [[211. "score": 2, "tura": 4, fastang: 1922737030. 5411152
nesped list, which not necuersry and Eat afficient the referance.
Ratine: [[B]], "scora": 8, "Eurn":
"Latamp":
1722846543. 867601
Table 22: (Continued) comparison of model generations on examples from MT-Bench.
27

---

Ruwdrd-Guided Prompt Evolving in Reinforcement Learning for Large Language Models
Zeortuen propee Does Usare males as algusisae wisk bettes time complasaty 22 so, Laplomers
ALEVEC from patat 2000---------
Aeamcc from gamaa 2 Xb- Lt 1p--
We to provices the fen Sieer tle complection
better snlactes te by 103gee tiee compleiity
caing atipicly more aspasattiated appeesse, Salitii sey idsa is to Lesoaaee
Use cortedt oulimm of caat and
searcter straleg Le piteett
Use
1H07, Mar mnp Fivrs the are cnd clence in the ostos
Heto with octoristsoo, sll thee conplrntee Apln Amarte Hiscr The firs
the stoolorely sliee There's officient alspectts with complaxity of Hot nis wishert ustag beap Antenceess the
sorted Har-te 1int2 The recond soroed Ltat in ot The desteed cast the resllart slemer. inte secconeste bth mallers eleners in the onton
implamaccationn tn oo tindant.naalerr..eee. www.eeewwweeeoocnnnnnnaaaa. Hank,
the lamta.te sange leesllreil, lanillanarto
nite Sired
ti saap riade maalast alanace unier too sorted 11
11as2, Hank, itsti, senzce 2ir1 the stceted contels lats,
time complesaty aegec Histt fiest sccted list. .a
atc wille left rigitit particions datt zigar /f 25 partisianntole MAcAtEX Hestppariiiiooe 1]
mallars alanees bale
sustacty
Lo parsisiaat
11)
cospertienss by
te
intes. sompping trackinge
Seaccheesto
"rlght pototars
'pactitionn" paise '1aat', 'paceinices"
coppases
caraising
'partitioos
appessed cornspodiing paiscer
MEceagees", "sirkigiey"
1211 ir
mesttiin
cappacttpen alguzitle maep
Winsigaee's the eaalinst
ouppeciity latele
particionn.
gosstor taceepptt",
fell adurray "iaad" Sirte S CChereine, Dorty
snburay
binery seurdh pertit fon has food Thw Conteettyrro The tieees seerch divioes the rearch space in hate at each step Ap Darefors, the time corplencty orzogratore, nii,, which is stgacdicuncly deseer than lirer time for larpe Liac stace ore Har ench maller shas the othor.
beighe tecteutive
reggert By 4p2-4)
zedpantt for quana 2-1-1---0
pamss- - bt dpp--aa
aaziatane's i fiess caaposss to the usseta quastias is sacoccect. lhe tsec
Favtsion shac siancy sRaCCE appecacE fird the kth sailles:
for ducction to find the its maaliass siatast is the caioe too socted Liste with Liress cosplacity The assistast peoriaes Erectisn that
siemnc the union socted liace. The sexiatarz also decalled
how tasesias
Lise complesity the
suettep stote Sie cupiossiy at dg a A. -91151 This
Linese cupplciity yy in the van repartent, The assistantts secsed respuace to the
weet to the fecoreet The user asked there extsts choettte
with better time comprentty then Uneer, The assl stant peselied frxcecio that
tas the coppleiiy of oas ais wiics to Stsears stt better ther Streer, The
sezsncare should have provided a fivestes that 19sx tieeey seacch sppeoect.
soict tu a tite cooplesdiy of orlogan Logesis, as the cedecacce server Jeogthe liste
by adding check degicaseg ducceior
ccceectly 004s.
the Secgeha
Natregi H181 Tecovets & teurth a. taasagg s1ag SE00E52
Pacsogg 1191)
Pestaoph
Table 23: (Continued) comparison of model generations on examples from MT-Bench.
28


Reviews:
[{"review_id": "jkgIVoQusc", "reviewer": "Reviewer_8RPw", "summary": "The paper proposes Evolving Alignment via Asymmetric Self-Play (Eva), which treats post-training as an infinite game involving two roles: the creator, responsible for generating new prompts, and the solver, which optimizes responses. Eva implements prompt evolution via a regret-based reward objective combined with a prioritized generation buffer, which works for both online and offline RL training.\nEva shows strong performance on Arena-Hard, the win rate of gemma-2-9b-it increased from 51.6% to 60.1% (DPO) and 52.6% to 62.4% (RLOO).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. \n\nThe open-ended RLHF objective (Problem 1) achieves continuous self-training by jointly optimizing prompts and response policies (§3). However, the authors acknowledge estimation bias but do not quantify its impact or provide error bounds.", "experimental_designs_or_analyses": "Yes.\n\nEva is evaluated across multiple RL algorithms:\n\n- Online: RLOO, OAIF\n- Offline: DPO, SPPO, etc. (§4)\n\nTrained using UltraFeedback and tested on three benchmark evaluations, covering both online and offline RLHF scenarios. The choice of reward model (ARMORM-8B) is reasonable, supporting the validity of performance claims.\n\nHowever, continuous training (§4.2.4) only reports monotonic gains, but does not analyze saturation points or changes in prompt quality after multiple iterations.", "relation_to_broader_scientific_literature": "None", "essential_references_not_discussed": "This paper has adequately discussed relevant prior work.", "other_strengths_and_weaknesses": "Strengths:\n- The proposed Eva improves RL post-training performance without additional human prompts\n- The empirical results are strong, gemma-2-9b-it’s win rate on Arena-Hard increased from 51.6% to 60.1% (DPO) and 52.6% to 62.4% (RLOO), surpassing Claude-3 Opus and approaching Gemini-1.5 Pro.\n- Eva-generated curriculum prompts can outperform human prompts. Compared to a baseline using 6× more human prompts, Eva (1× prompts) performs better across multiple metrics.\n\nWeaknesses:\n- Table 4 shows that different approaches' performance are close.\n- The initial prompt distribution (UltraFeedback) and the evolution process may be domain-specific (e.g., Figure 11 shows an imbalanced distribution in AlpacaEval), and cross-domain generalization was not fully tested.\n- Table 16 shows that the evolved prompts mainly focus on technical tasks, therefore they may have limited coverage.", "other_comments_or_suggestions": "See strengths and weaknesses.", "questions_for_authors": "See strengths and weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yqYlFTnm7B", "reviewer": "Reviewer_eTrw", "summary": "The paper \"Evolving Alignment via Asymmetric Self-Play\" presents EVA, an innovative framework that addresses a critical limitation in current RLHF methods by replacing static prompt distributions with an adaptive, generative approach. By framing alignment as a game between a prompt creator and response solver, EVA enables continual self-improvement without requiring additional human data. The empirical results are impressive, boosting Gemma-2-9b-it's performance on Arena-Hard benchmarks significantly and allowing it to compete with much larger models. The method's seamless integration with both online and offline RLHF pipelines, coupled with its robust performance across different configurations, makes it a practical and valuable contribution to the field of language model alignment.\n\n\n# after rebuttal\n\nI will keep my score for accepting this paper.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer's in-depth evaluation and new insights on the game. Below we provide a high-level summary with detailed rebuttal, and will add relevant discussions in the reivisions.\n\n---\n\n**TL;DR**: Under reasonable assumptions, we can evolve to a *local* minimax optimum. (However, due to the nonconvex-nonconcave nature of the neural network optimization, global optimum is generally intractable to find.)\n\n--- \n\n> **Q:** *If both creator and solver evolves, is it possible that they evolve to a bad local optimum? Can the algorithm find the equilibrium, or could you provide some analysis about this?*\n\n**A:** To our knowledge, for the nonconvex-nonconcave minimax optimization problem, finding the global equilibrium is generally NP-hard. In sequential settings, there exist alternating gradient descent algorithms (Jin et al., 2020; Wang et al., 2019) that can achieve exact local convergence to *local* minimax. Thus yes, it is possible for them to be \"bad\" local minimax optimum that are far away from global minimax optimum.\n\nIn simultaneous settings, recent works (Dennis et al., 2020; Parker-Holder et al., 2022; Beukman et al., 2024) have shown that when a Nash equilibrium is reached, the solver follows a minimax regret policy and the solution exhibits robustness properties. \n\nWe believe the existing analysis helps justify the general game-theoretic formulation. In our empirical algorithm, we take the sequential setting, and use approximations for the regret maximization for the creator to avoid instability during training, as discussed in Section 3.2. We take a mixed sampling strategy to avoid the creator to drift too far away in each iteration (as discussed in the Appendix). Moving forward, we believe deriving a tractable algorithm with differentiable creators is a meaningful next step.\n\n---\n\n**References**    \nJin, C., Netrapalli, P., & Jordan. M. (2020). What is Local Optimality in Nonconvex-Nonconcave Minimax Optimization.   \nWang, Y., Zhang G., & Ba. J. (2019). On Solving Minimax Optimization Locally: A Follow-the-Ridge Approach.  \nZhang, G. (2023). Deep Learning Dynamics: From Minimization to Games. Dissertation at University of Toronto.  \nDennis, M., Jaques, N., Vinitsky, E., Bayen, A., Russell, S., Critch, A., & Levine, S. (2020). Emergent complexity and zero-shot transfer via unsupervised environment design.  \nParker-Holder, J., Jiang, M., Dennis, M., Samvelyan, M., Foerster, J., Grefenstette, E., & Rocktäschel, T. (2022). Evolving curricula with regret-based environment design.   \nBeukman, M., Coward, S., Matthews, M., Fellows, M., Jiang, M., Dennis, M., & Foerster, J. (2024). Refining minimax regret for unsupervised environment design.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "Relevant to the people who work on alignment, LLM and AI.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths\n\n1. Open-ended alignment is an important topic, given that the social trends and human opinions are evolved. This paper tackles this problem through the Evolving Alignment via Asymmetric Self-Play, i.e., EVA. \n\n2. Though co-evolving of the creator and solver is promising, the training is unstable. This works tackle this issues by self-play and the regret-based methods. The framework is elegant and powerful. \n\n3. The evaluation is sufficient and comprehensive. \n\nWeakness\n\n1. I am a bit concerning about the creator. If both creator and solver evolves, is it possible that they evolve to a bad local optimum? \n\n2. Still about the game between the creator and the solver. Does the algorithm can find the equilibrium? or could you provide some analysis about this? As if building the problem as a game, the desired solution may be Nash equilibrium. so we can evaluation whether the formulation of the game is reasonable through checking whether the equilibrium is corresponding to the best solution.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see above sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "o8r413VVWB", "reviewer": "Reviewer_QZbz", "summary": "This paper studies a new paradigm for post-training where prompts are sampled adaptively. In particular, this paper proposes eva, in which a creator is addtionally introduced to select prompts for the solver to optimize. It provides extensive empirical results to show the advantage of eva.\n\n## update after rebuttal\n\nI have reviewed the authors' updated formulation, which is now more sound than the submitted version. The empirical ablations also address my concerns regarding effectiveness. Below are additional comments on the rigor of the formulation:\n\n- Gap between Problem 1 and the min-max regret formulation: In Problem 1, the objectives of the two players differ, whereas in the min-max Nash game formulation, they are the same. \n\n- Soundness of Problem 1: The formulation appears reasonable, though the lack of a prior definition for $\\pi_{true}$ remains a subtle issue. I guess the issue is not from the statistical side that we cannot draw samples from $\\pi_{true}$. Instead, the fundamental concern is that there is no clear criterion for defining the of optimality of $\\pi_{true}$. This is analogous to many optimization problems where the optimal solution is unknown a priori but is instead implicitly defined by the objective function.\n\n- Min-max regret formulation: Under the assumption that the solver has strong representation and optimization power, the optimal best response would simply select the action with the highest reward, regardless of the creator’s design. To avoid this trivial case, additional assumptions (e.g., limited representation or optimization power) should be introduced.\n\nOverall, I find the empirical algorithm `eva` reasonable and potentially valuable to the community, though the theoretical formulation should be more rigorous. Given these considerations, I have updated my review score to 4, and I hope that the authors could either refine these aspects or remove unproper formulation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the detailed and thoughtful review, which helps us a lot in shaping a better submission. \n\n---\n**Overview:** We would like to clarify several potential misunderstandings in the review:\n1. **Problem 1 and Minimax Game:**\n    - (**i**) \"*Problem 1 is a max-max collaborative game.*\" --> Incorrect. Problem 1 is a joint optimization problem, whose solution may be approximated in either a min-max or max-max way. \n    - (**ii**) \"*Creator regularization is not implemented.*\" --> Incorrect. Creator regularization is explicitly achieved through regret maximization (as in Section 3.1 and 3.2). \n    - (**iii**) \"*The minimax regret formulation does not actually solve the Problem 1.*\" --> Inaccurate. The minimax regret formulation provides a worst-case optimal solution to Problem 1 (as in the discussions under Remark 1).   \n2. **Adaptive Prompt Sampling:**\n    - \"*It is clear that the optimal strategy is to uniformly select all prompts.*\" --> Potentially misleading. The assumption on unlimited resources can be impractical, as efficiency is a key bottleneck in training large models. Even w/ unlimited resources, uniform sampling is only optimal under strong assumptions (e.g., iid data, perfect RM, ...), which is impractical. We also cited a rich literature on online learning and network optimization showing uniform sampling can be sub-optimal and may lead to worse local minima (as in Table 5, 4.2.1 and 4.2.4).\n\n---\n\n**1. Rebuttal on Problem 1 and the Minimax Game:**  \n(**i**) Problem 1 captures the general objective on optimizing the language model for it to perform well with regard to some potentially unknown prompt distribution (we will add $\\pi _{\\mathsf{true}}(\\cdot)$ inside the regularization to emphasize the discussion in 3.1.)\n$$\n\\max _{\\phi, \\boldsymbol{\\theta}} \\mathcal{J}(\\phi, \\theta) := \\mathbb{E} _{  x \\sim \\pi _\\phi(\\cdot)}  [\\mathbb{E} _{  y \\sim \\pi _{\\boldsymbol{\\theta}}(\\cdot \\mid   x)}[r(  x,   y)]-\\beta_1 \\cdot \\mathbb{D} _{\\mathsf{KL}} [\\pi _{\\boldsymbol{\\theta}}(  y \\mid   x) \\| \\pi _{\\mathsf {base}}(  y \\mid   x)  ]  ] + \\mathcal{R} (\\pi _\\phi(\\cdot), \\pi _{\\mathsf{true}}(\\cdot)  ).\n$$\n\nThis is a joint optimization problem, and itself is **not directly a collaborative (nor competitive) game**. It can be intractable and may be approximated differently in max-max or max-min way by alternating optimization (cf., GAN). With $c$ for creator, $s$ for solver, and $c _{t}$ for target, some choices may be:\n- $\\max _{s} \\max _{c} f _c(s) - \\mathsf{KL}(c, c _{t})$\n- $\\max _{s} \\min _{c} f _c(s) + \\mathsf{KL}(c, c _{t})$\n\nWe believe the current joint optimization formulation is more general, and can induce different practical algorithms.\n\n(**ii**) / (**iii**) When the target true distribution is unknown, the problem is a classical *decision under ignorance* problem (with partial knowledge, it can become decision under uncertainty and strategies like posterior sampling can be applied) (Jiang, 2023; Peterson, 2017), where we need find a propoer decision rule. Here, we seek the *optimal solution under the worst-case*, and this is why we design a game, where **creator regularization** is explicitly achieved by **regret maximization**.  The game provides an approximation to the worst-case-optimal solution of Problem 1. \n\nIt may be subjective to claim \"regularization should not be a central element of game\" and incorrect to derive \"the game is fundamentally flawed\". Here, the regularization is applied to the joint optimization problem, not the minimax regret game. \n\n---\n\n**2. Rebuttal on Online `eva` Settings:**   \nBuffer subset sizes are chosen as powers of 2 for hardware efficiency, as we run on a single machine with 8 GPUs.\n\nWe will add further results in anonymous.4open.science/r/eva-i.\n\n---\n\n**3. Rebuttal on Additional References:**   \nThanks for the wonderful suggestions! We will add them in our revised paper. Some discussions:\n- [2] has been cited and discussed in 4.2.1.\n- [3] is on training reward models, which differs from our main theme. We have cited earlier Thompson Sampling work to reflect this area.\n- We do not assume access to target as in [5].  \n\n---\n**4. Rebuttal on Adaptive Sampling:**   \nPlease see our overview in the beginning. \n\nWe use fixed design for the buffer, which is standard in iterative RLHF (e.g., SimPO). In practice, we found \"knowledge forgetting\" less of an issue (than overfitting), likely due to differences in supervised learning and RL.\n\n---\n\n**References on Fundamentals:**  \nJiang, M. (2023). Learning Curricula in Open-Ended Worlds.   \nPeterson, M. (2017). An introduction to decision theory.   \nOrabona, F. (2023). A modern introduction to online learning. \n\n---\nIt is a great pleasure to have the opportunity to learn from a different mind. We believe the rebuttal have sufficiently addressed the concerns. We sincerely hope the reviewer may reconsider the rating on `eva`, and we are happy to discuss further on any potential future works.", "claims_and_evidence": "The story is imaginative, but its scientific foundation is questionable.\n\nIssue with Problem 1: The problem highlighted in Problem 1 does not make sense to me. Without proper regularization for the creator, the creator may simply choose the simplest prompt for the solver, leading to an equilibrium that is essentially meaningless. Although the paper addresses this issue in Section 3.1 and introduces actor regularization, it still does not resolve my concerns. Regularization should not be treated as a central element of game design. Therefore, I believe the game is fundamentally flawed in its design. Moreover, if regularization is indeed crucial, the paper should discuss it more thoroughly in the main text. Currently, I see no significant discussion of this in the main text.\n\nDiscrepancy between Problem 1 and the Formulation: There is a notable gap between Problem 1 and the formulation presented in Section 2. In Problem 1, the game is collaborative, with both layers aiming to maximize the same objective (i.e., a max-max formulation). However, in Section 2, the paper shifts to a min-max game formulation, where the creator appears to act adversarially. This inconsistency suggests that the formulation does not actually solve the problem proposed in Problem 1.", "methods_and_evaluation_criteria": "Not applicable", "theoretical_claims": "Not applicable", "experimental_designs_or_analyses": "Yes, I reviewed the experiment details and found that the proposed approach, as described in Appendix A, appears to be rather heuristic. For instance, it uses a specific number of prompts (e.g., 4 and 16) without clear justification. This raises concerns about the generalizability of the approach: is the demonstrated performance highly sensitive to this hyperparameter? Without further analysis or ablation studies, it is difficult to assess whether the results are robust or overly dependent on this particular configuration.", "relation_to_broader_scientific_literature": "The problem of adaptive prompt selection studied in this paper is quite interesting; however, its scientific value remains questionable.", "essential_references_not_discussed": "This paper provides a comprehensive review of previous works. However, I would like to highlight several key points and relevant literature that should be discussed.\n\nThe core idea of regret maximization for the creator bears strong similarities to the following two works, which deserve discussion [1, 2].\n\nThe paper should also consider discussing the work [3], which presents an information-theoretic approach to data collection.\n\nThe gradient estimator used for the actor appears to be similar to the ReMax algorithm [4].\n\nTo my knowledge, there are two main paradigms for adaptive sampling:\n- Information-Seeking (Min-Max Formulation): This approach may demonstrate advantages over uniform sampling in compute-limited settings.\n- Transferability-Based Sampling: This approach aims to select prompts from a large source pool based on representative samples from the target domain. For example, see [5]\n\n[1] Jiang, Yiding, et al. \"Adaptive data optimization: Dynamic sample selection with scaling laws.\" arXiv preprint arXiv:2410.11820 (2024).\n\n[2] Mindermann, Sören, et al. \"Prioritized training on points that are learnable, worth learning, and not yet learnt.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Dwaracherla, Vikranth, et al. \"Efficient exploration for LLMs.\" arXiv preprint arXiv:2402.00396 (2024).\n\n[4] Li, Ziniu, et al. \"ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models.\" arXiv preprint arXiv:2310.10505 (2023).\n\n[5] Xie, Sang Michael, et al. \"Data selection for language models via importance resampling.\" Advances in Neural Information Processing Systems 36 (2023): 34201-34227.", "other_strengths_and_weaknesses": "The results are presented in a well-organized format; however, certain parts lack clear explanation in the main text, making it difficult to fully assess the scientific value of the work. For instance:\n\n- The details of creator regularization in Section 3.1 are not sufficiently elaborated.\n\n- The creator optimization step in Section 3.3.2 is not clearly explained.\n\nAdditionally, the paper fails to discuss simple baselines (e.g., uniform sampling) in the formulation and does not provide a thorough analysis of why and under what conditions the proposed approaches are expected to outperform these baselines. Addressing these gaps would significantly strengthen the paper's scientific rigor and practical relevance.", "other_comments_or_suggestions": "I find that Remark 1 does not provide any new insights beyond reiterating the definition.", "questions_for_authors": "From my understanding, the min-max formulation is advantageous in compute-limited settings, as it can quickly identify samples whose loss decreases rapidly. However, from a theoretical standpoint, it is clear that the optimal strategy for the creator is to uniformly select all prompts if the solver is provided with sufficient computational resources to optimize under each prompt. In such cases, uniform sampling is expected to performs well. I note that the experiments are conducted with relatively small sample sizes (e.g., 10k). I am curious about the performance advantage of the proposed approach over uniform sampling when the data size is significantly larger (e.g., 1M, as seen in commercial-level LLM products).\n\nAdditionally, I am unclear about the design of the prompt buffer:\n\n- Is it incremental (e.g., 10k → 10k + 10k = 20k → 20k + 10k = 30k) in each iteration?\n\n- Or is it fixed (e.g., 10k, 10k, 10k) in each iteration?\n\nI feel confused about this setting. In the latter case, I suspect there may be a knowledge forgetting issue, as previous prompts are discarded in later stages of training.\n\nFinally, I would like to see ablation studies on the hyperparameter choices for online EVA mentioned in Appendix A. The current details provided are too heuristic and lack justification, making it difficult to assess the robustness and generalizability of the approach.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0Yaxaofmsz", "reviewer": "Reviewer_irNB", "summary": "This paper presents eva, a new minimax algorithm for RLHF which pushes beyond the static prompt set used by the majority of RLHF algorithms. In eva, the creator is trained to generate prompts that are solvable by the the solver. The largely empirical work focuses on evaluating a large number of design choices through extensive experimental evaluation.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the detailed review and insightful questions. Below we provide a high-level summary with detailed rebuttal, and will add relevant discussions in the reivisions.  \n\n---\n\n> **Q2 & Q3**: *Could the authors provide more details on how they evaluate `eva`? What subset of prompts are used from ultrafeedback? How does `eva` compare as the number of prompts in the initial set increases?*  \n\n**A**:  As discussed in Section 4, we evaluate `eva` on off-the-shelf benchmakrs including [AlpacaEval 2.0](github.com/tatsu-lab/alpaca_eval), [MT-Bench](github.com/lm-sys/FastChat/blob/main/fastchat/llm_judge), and [Arena-Hard](github.com/lmarena/arena-hard-auto). We train `eva` with the train split of UltraFeedback; following standard practices in iterative RLHF, we shuffle the full training set and divide it into equal-sized subsets for each iteration. In Appendix F.2., we have experimented with varying number of prompts in the initial set from 10K to 20K and 60K, and show that `eva` can consistently bring robust gains across multiple iterations.  \n\n> **Q1 & Q4**:  (**i**) Could the authors detail how the reward models are trained? (**ii**) What data is used for training the reward model *v.s.* the data used for training the policy model (*i.e.*, the solver)? (**iii**) Does this create a problem where the reward model must have more data to be sufficiently generalizable to generated prompts?\n\n**A**:   (**i**) As discussed in Section 3 and 4, we assume a fixed reward model as the oracle for human preferences during training. We have evaluated our method under different reward models ([ArmoRM-8B](https://arxiv.org/pdf/2406.12845), [SkyworkRM-27b](https://arxiv.org/pdf/2410.18451), [PairRM-0.4B](arxiv.org/abs/2306.02561), and [PairRM-8B](huggingface.co/RLHFlow/pair-preference-model-LLaMA3-8B)). (**ii**) The data used for training the reward model can be different from the data used to train the solver. For example, the prompts used to train ArmoRM-8B and PairRM-8B are selected from UltraFeedback, HelpSteer, OpenOcra, UltraInteract, Capybara and DIBT-10K; the prompts used to train SkyworkRM-27B are 80K prompts selected from  HelpSteer2,  OffsetBias, WildGuard (adversarial), and Magpie. (**iii**) We believe continual training of reward models is an important future work to enhance the robustness of `eva`. As the creator generates more prompts that diverge from the initial training prompt set (in our case, subsets of UltraFeedback), a fixed RM may be struggle to generalize reliably, especially if its training data does not cover the evolving prompt distribution. While our experiments show that `eva` remains effective under multiple fixed RMs, we view continual refinement of reward models (e.g., by online updates or co-training with the evolving policy) as important future works for improving the long-term robustness of `eva`.  Note that reward models may be trained more efficiently and generalize better than policy models, as they only produce scalar scores or rankings, thus it does not necessarily require *more* data (or prompts) than the policy to remain effective.\n\n\n> **Comments:** *It is nicer when papers write full works, e.g., \"with\" instead of \"w/\". Line 229 col 2 \"and can compete default training\". Is there a missing word here?*  \n\n**A**: Thanks so much for the suggestions! We will carefully revise each and every abbreviation and make sure the writing is clear. And yes, for Line 229, we meant that `eva` with only 1× raw data throughout training can compete with default training -- which uses 5× more human prompts and no evolved data -- by achieving comparable or better results.", "claims_and_evidence": "The claims made in the paper surrounding eva's performance are largely substantiated by extensive experiments.\n\nHowever, one component that I believe would be important to understanding eva's claimed performance would be the different prompt/datasets used to train the reward model versus used for RL. I have listed some questions surrounding this in the \"questions\" section.", "methods_and_evaluation_criteria": "The presentation of the method is clear and the work provides sufficient justification for the objective presented in Eq. 2 through connections to minimax regret. \n\nThe chosen evaluations make sense as they are standard evaluations used for RLHF / Alignment.", "theoretical_claims": "The work does not make any theoretical claims.", "experimental_designs_or_analyses": "The work is thoroughly evaluated on a number of different axes. I commend the authors for their work in setting up all of these experiments.\n\nOne aspect that was less clear to me were the differences between the prompt datasets used to train the reward model versus prompt datasets used for the RL component. From looking at experiments in Appendix F, it seems like eva's gains versus DPO degrade substantially when DPO is given larger preference datasets. \n\nI think it is most fair to evaluate DPO and eva with access to the same base prompt dataset. If the reward function used by EVA to select prompts is trained on all the data, this seems like an unfair advantage in the evaluation.", "relation_to_broader_scientific_literature": "Though prior work exists on both prompt evolution and evolution based reward design, eva presents an exciting proof of concept for dataset expansion for post-training RL. This demonstration is at the highest scale I have seen, and I believe would be of value to the community.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "* I can tell the authors were crammed for space, but I think it is nicer when papers write full works e.g. \"with\" instead of \"w/\" (Line 198 col 2)\n\n* Line 229 col 2 \"and can compete default training\". Is there a missing word here?", "questions_for_authors": "* Eva appears to require having an explicit reward model. Could the authors detail how that reward model is trained? \n\n* Could the authors provide more details on how they evaluate EVA? What subset of prompts are used from ultrafeedback? \n\n* How does eva compare as the number of prompts in the initial set increase? \n\n* Could the authors do a better job clarifying what data is used for training the reward model used for EVA versus the data used for RL? It seems like these might be different in places. Does this create a problem where the reward function must have more data to be sufficiently generalizable to generated prompts?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes Eva, a scalable reinforcement post-training framework that replaces static human prompts with an adaptive, asymmetric self-play mechanism. Eva treats post-training as a game with a creator generating prompts and a solver optimizing responses. It also uses a regret-based reward objective and a prioritized generation buffer, working for both online and offline RL training. Evaluated on Arena-Hard and UltraFeedback datasets, Eva demonstrates significant performance gains for models like Gemma-2-9b-it and outperforms baselines using 6× more human prompts.

Reviewers acknowledge several strengths of the paper. As a practical, unified approach compatible with both online and offline RLHF pipelines, Eva improves RL post-training performance without extra human prompts. Empirical results via extensive experiments across RL algorithms and model scales show significant win-rate increases on the Arena-Hard benchmarks. The integration of regret-based optimization and prioritized buffers is also appreciated.

On the other hand, the paper has some drawbacks. Reviewer QZbz questions the scientific foundation, pointing out issues in game design and inconsistencies in problem formulation. Reviewer eTrw also worries about the creator evolving to a bad local optimum and the algorithm's ability to find an equilibrium. There are also concerns regarding the limited analysis of cross-domain generalization, heuristic hyperparameter choices, potential unfair comparisons with DPO baselines, and ambiguities in prompt buffer design.

The authors have submitted their responses to the reviews. Subsequently, all the reviewers have confirmed that they have read the authors' responses, and some of them have updated their overall assessments based on these responses.
Decision: Accept (poster)