Confronting Reward Model Overoptimization with Constrained RLHF
Authors: Ted Moskovitz
Aaditya K Singh
DJ Strouse
Tuomas Sandholm
Ruslan Salakhutdinov
Anca Dragan
Stephen Marcus McAleer
Venue: ICLR 2024 spotlight (2024)
Keywords: rlhf
overoptimization
constrained RL
Primary area: reinforcement learning
TL;DR: We study overoptimization in composite reward models and introduce several approaches based on a constrained MDP formulation of the problem to address it.

Abstract: Large language models are typically aligned with human preferences by optimizing reward models (RMs) fitted to human feedback. However, human preferences are multi-faceted, and it is increasingly common to derive reward from a composition of simpler reward models which each capture a different aspect of language quality. This itself presents a challenge, as it is difficult to appropriately weight these component RMs when combining them. Compounding this difficulty, because any RM is only a proxy for human evaluation, this process is vulnerable to *overoptimization*, wherein past a certain point, accumulating higher reward is associated with worse human ratings. In this paper, we perform the first study on overoptimization in composite RMs, showing that correlation between component RMs has a significant effect on the locations of these points. We then introduce an approach to solve this issue using constrained reinforcement learning as a means of preventing the agent from exceeding each RM's threshold of usefulness. Our method addresses the problem of weighting component RMs by learning dynamic weights, naturally given by the Lagrange multipliers. As a result, each RM stays within the range at which it is an effective proxy, improving evaluation performance. Finally, we introduce an adaptive method using gradient-free optimization to identify and optimize towards these points during a single run.

Published as a conference paper at ICLR 2024
CONFRONTING REWARD MODEL OVEROPTIMIZATION WITH CONSTRAINED RLHF
Ted Moskovitz' Gatsby Unit, UCL
Aaditya K. Singh Gatsby Unit, UCL
DJ Strouse Google DeepMind
Tuomas Sandholm Carnegie Mellon University
Ruslan Salakhutdinov Carnegie Mellon University
Anca D. Dragan University of California, Berkeley
Stephen McAleer Carnegic Mellon University
ABSTRACT
Large language models are typically aligned with human preferences by optimizing reward models (RMs) fitted to human feedback. However, human preferences are multi-faceted, and it is increasingly common to derive reward from composition of simpler reward models which each capture a different aspect of language quality. This itself presents a challenge, as it is difficult to appropriately weight these com- ponent RMs when combining them. Compounding this difficulty, because any RM is only a proxy for human evaluation, this process is vulnerable to overoptinization, wherein past a certain point, accumulating higher reward is associated with worse human ratings. In this paper, we perform, to our knowledge, the first study on overoptimization in composite RMs,, showing that correlation between component RMs has significant effect on the locations of these points. We then introduce approach to solve this issue using constrained reinforcement learning as a means of preventing the agent from exceeding each RM's threshold of usefulness. Our method addresses the problem of weighting component RMs by learning dynamic weights, naturally expressed by Lagrange multipliers. As result, each RM stays within the range at which it is an effective proxy, improving evaluation perfor- mance. Finally, introduce adaptive method using gradient-free optimization to identify and optimize towards these points during single run.
INTRODUCTION
In the last several years, Large Language Models (LLMs) have made impressive advances in natural language processing. These models, which are typically pretrained on massive amounts of text data from the Internet to predict the next token given the current context, are often known as foundation models (Bommasani et al., 2021) for their ability to be adapted to a variety of downstream applications, such as chatbots (Brown et al., 2020; OpenAl, 2023; Touvron et al., 2023) code generation (Ahmad et al., 2021; Wang et al., 2021; Rozière et al., 2023). This adaptation, or finetuning, is often performed via reinforcement learning from human feedback (RLHF; Knox and Stone, 2008; Christiano et al., 2017; Stiennon et al., 2020). RI.HF treats the pretrained language model as a decision-making agent whose "actions" tokens and whose goal is maximize reward model (RM) trained to emulate human preferences over output text. As these models become more prevalent in society, there are many concerns regarding their safe deployment (Hendrycks et al., 2023; Bubeck et al., 2023; Legg, 2008), including biases against marginalized or underrepresented groups (Bender et al., 2021), proliferation of false information (Lin et al., 2021), and leakage of sensitive information (Carlini et al., 2021). These concerns are collectively known as the alignment problem: how can we ensure that the behavior of these models is aligned with human preferences?
"Corresponding author: ted8gatssy..o wwt.ssyyccccccc... www.uc.l............ ac.uk
Additiional affiliations: Strategy Robot, Inc., Strategic Machine, Inc., Optimized Markets, Inc.

---

Published as a conference paper at ICLR 2024
Current approaches to alignment within RLHF center around the collection of vast amounts of human rating data and the training of larger, more powerful RMs (Ouyang et al., 2022; Gao et al., 2022). However, fundamental issue with any RM is that ultimately, it is only human preferences. Gao et al. (2022) drew attention to this fact, showing that maximizing a reward model beyond certain point can actually begin to decrease ground truth performance (ie., lead text-based agent to produce outputs which are judged as qualitatively worse). This phenomenon is known as reward model overoptimization. Examples of overoptimization include producing overly wordy responses or hallucinating information in an effort to give the impression of expertise. One simple, yet expensive, approach to mitigating this issue is to periodically evaluate the model with fresh human rating throughout finetuning and stop early when ratings decline.
imperfect proxy for
is also increasingly common to derive reward from composite RMs: fixed combinations of several RMs each designed to capture different aspect of text quality (Ramamurthy et al., 2022; Glaese et al,, 2022; Yuan et al., 2023; Bakker et al., 2022; Wu et al., 2023). Such composite RMs are useful because they allow for more fine-gran fine-grained measurement of agent behavior and each component can be retrained or swapped out without affecting the others. Despite these advantages, this approach also presents its own challenges. Determining the weighting among RMs requires hyperparameter optimization to find the combination that produces the best correlation with ground truth evaluation, and the risk of overoptimization means that the best weighting is contingent on set training duration. Furthermore, when the reward is constructed from several RMs, information about each individual RM is lost, and the agent cannot attribute changes in reward to any single model. In particular, component rewards may even oppose one another, such as an RM which measures safety (and thus may deny certain user requests) versus another rewarding helpfulness (Bai et al., 2022). Worse, early stopping to avoid overoptimization in composite RMs is problematic, as different components will have different values at which they stop being effective proxies for human evaluation. In this paper, we propose simple approach address these challenges: identify the points of overoptimization, which we term proxy points, and then use constrained optimization to ensure that each component RM reaches, but does not exceed, its associated proxy point. Rather than use fixed weighting among components, our method dynamically adapts weighting to modulate the influence of each RM on the learning process. The core idea behind our approach is to use these constraints to prevent the agent from overoptimizing its (composite) RM beyond the proxy points.
As in existing methods (Gao et al., 2022), we rely on some access to ground-truth queries. We propose two ways of using these queries to identify proxy points. In the first approach, we train multiple runs and track each reward model value, periodically querying the ground-truth reward model. This approach then finds an optimal joint proxy point by fitting surface to this data and maximizing it. While effective, this approach requires multiple runs to fit the surface used to find proxy points. In the second approach, we speed up this process by only using one reinforcement learning run. As this run is training, we can periodically query the ground-truth reward model and use this data run derivative-free optimization algorithm to find the next candidate proxy points. To summarize, we make the following contributions:
We provide analysis of reward model overoptimization in the context of composite reward functions, showing that the correlation between RMs has a significant influence on proxy points.
We propose several constrained RL approaches which incorporate these points into the optimization objectives, preventing overoptimization and improving evaluation performance.
We show that a derivative-fre optimization method can be used to dynamically find these proxy points during a single run, significantly saving computation.
2 PRELIMINARIES REINFORCEMENT LEARNING FROM HUMAN FEEDBACK
RL Problem Formulation In reinforcement learning (RL; Sutton and Barto, 2018), an agent seeks to take actions in its environment in order to maximize reward. Mathematically, this problem is typically formalized as a Markov decision process (MDP; Puterman, 2014), defined as a tuple M 2 (S, A, P, r, 7, p). where S is the state space, A is the action space, P x A A P(S) is the transition kernel (where P(X) denotes the set of distributions over X), S x A S - RR is the reward function, 7 E [0. 1) is the discount factor, and p E P(S) is the initial state distribution. In practice, the agent's 's experience is typically broken into discrete segments, or "episodes" of maximum length T. At the beginning of each episode, the environment resets and an initial state is sampled 80 p(-). At each time step i 0, .......? T - 1, the agent selects an action a conditioned on
2

---

Published as a conference paper at ICLR 2024
Evaluation vs. Intent
Evaluation vs, METEOR
Evaluation vs. KL[m, Mo]
0.66
METEOR
Ereluation Score
0.60
0.44
0.05
042
635
0.00
0:56
650
knane
0:56
044
64s
046
0.19
620
6:11
0:33
0.00
0.25
8.50
0.75
1.00
1.25
150
175
Intent
METEOR
KL[m 7l
Figure 3.1: Individual RMs are imperfect proxies for evaluation score. Evaluation score initially increases as individual RMs and the KL divergence grow before falling at proxy points, denoted by dashed lines. Results are averaged over 5 seeds, with shading showing standard error.
its current state 8t using stationary policy x(ae)ss), where S + P(A). Each episode can be summarized as a trajectory r = (sp, ao 81,...,877 ST). The agent's ' goal is to find a policy with maximum expected return R(T), where R(T) Et -T-1 7"r(st. St+1). The expected return under policy is known as the value 1" (s) & [[((T) 80 s] or the action-valus if conditioned on both states and actions (5, A [[(((T)|s0 = 0g a]. The optimization problem faced by the agent, then, max; v", where v" & Eso~p())' (so) is the average value over initial states.
Integrating Human Feedback The origin and nature of the reward is a fundamental question when formalizing problem using RL. Using human evaluation to delineate good agent behaviors from bad has history that extends beyond language models. Knox and Stone (2008) used human ratings of actions to construct a reward model for the game Tetris, while Christiano et al. (2017) proposed mechanism for using human feedback to express preferences over trajectories collected in Atari and MuJoCo. In language modeling, each action is viewed as adding a new token to the current context string (Ziegler al., 2019; Stiennon et al., 2020; Bai et al., 2022; Ouyang et al., 2022), which can be viewed as the state. The LM then the policy, with action space A being the vocabulary of possible tokens, and state space S being the set of all sequences of tokens up to maximum length T. Transitions are deterministic, with each action token simply appended to the current state. Given a pretrained LM No, RLHF often consists of three stages (Casper al., 2023): 1) collecting human feedback on model utterances (typically in the form of ranked preference data), 2) training RM to model score utterances in alignment with human feedback (typically initialized from a separate pretrained LM) and 3) finetuning the LM with RL using the learned R.M. While early work in RLHF for LLLMs (Stiennon et al., 2020) focused on single reward model, more recent work has shown performance benefits of using weighted combination of simpler RMs (Wu et al., 2023).
Overoptimization Recently, Gao et al. (2022) performed an empirical study a phenomenon with deep ramifications for alignment: RM overoptimization. Their core finding is that after certain point, increasing an LLM agent's 's value with respect to a given RM will actually begin to decrease its quality on the actual preferences it trying to learn. (Gao al. (2022) use "gold standard" RM to stand in for human ratings for convenience.) The root of this issue is that any RM is only a proxy for the agent's 's true measuring stick human evaluation so as predicted by Goodhart's 's Law (Goodhart and Goodhart, 1984), an agent trained to maximize it will eventually learn behaviors which the true objective would discourage. Our approach to addressing this issue based on simple two-stage process: first, find the points where the available rewards stop being useful proxies, and second, train an agent to only maximize reward up until that point.
FINDING PROXY POINTS
Setting In order to conduct an in-depth analysis given our available computational resources, we focus on a single setting as a case study: dialogue generation with the DailyDialog (Li et al., 2017) dataset, which consists of transcripts snippet of conversation, and from this context, it must predict the next utterance. We describe this setting in detail in Appendix A. As a base LLM, we follow prior work (Wu et al., 2023) and use GPT-2 (Radford et al., 2019) here and throughout this paper. For the reward, we combination of two component rewards, each meant to capture a different element of desired behavior, to demonstrate our approach most directly. The first, met is the METEOR score (Banerjee and Lavie, 2005) between
conversations between humans. As input, the agent receives
3

---

Published as a conference paper at ICLR 2024
the generated utterance and reference output, which is computed based on a number of features, including word- matchin,, synonym- synonym-matching, and phrasing. The second, rint, measures how well the intent of the generated utterance matches that of the reference output. is computed using fine-tuned RoBERTa model (Liu et al., 2019) which classifies text into different "intent categories" such 'inform,' "question," or 'direct.' The typical approach (Ramamurthy et al., 2022) is to linearly combine these RMs to form a composite reward:
ft = a met pmett + a iint
(3.1)
where the coefficients (amet a ) are fixed. As is standard in RLHF applied to language models, an additional KL penalty was added to discourage deviation from the initial model To:
Correlation Between Proxy Points
Proxcy Pont
KL log πt (agls)) Tn (8)
Tt = F. at
(3.2)
0.40
The coefficient of effectively acts as a La- grange multiplier, increasing the KL exceeds some threshold and decreasing otherwise. We discuss this in more detail in Appendix B.
035
Valuutoo
600
0.45
Evaluation and Proxy Points In an ideal world, evaluation performance for all agents across all runs could be measured by collect- ing a large number of human ratings. However, this is expensive, so we instead selected a num- ber of metries other than METEOR and intent score which measure the lexical quality and di- versity of text outputs and averaged them to serve as evaluation metric (details in Appendix A). Our choice is in line with prior work that uses held out metrics as the ground truth for convenience of iteration (Gao al., 2022). We call the value at which further increasing the proxy reward results in decreased tround-truth performance the proxy point 0*, To identify proxy points, we trained PPO agents (Schulman et al., 2017) to maximize only one reward or the other (without KL regularization) and plotted the resulting evaluation scores against the METEOR and intent scores in Fig. 3.1. In both cases, the evaluation score initially inereases before falling. Gao al. (2022) also observed that, in general, maximization of reward causes the KI. divergence between the trained and pretrained policies to increase, and therefore we also expect evaluation score to initially increase before decreasing as the KL grows as well, also shown in Fig. 3.1. One additional phenomenon that makes optimization of composite RMs challenging is that the component RMs may be correlated. We hypothesized that this interaction would influence the proxy points of the component rewards. To test this, plotted the evaluation scores as a function of the METEOR and intent rewards for each run shown in Fig. 3.1 in Fig. 3.2 and fit polynomial surface to the data, using kernel density estimation only fit the surface over regions with sufficient data (further details in Appendix A). The maximizing point (0) (0intentr 0f "eutens:Oimsteer". indeed differs from the proxy points found by only considering one RM at time. It is also important to note that the predicted maximizing point is of the fitted surface, rather than any point attained by one of the individual runs.
Metsor Score
644
346
Intont Score
039
Figure 3.2: Correlation influences proxy points.
4 CONSTRAINED RLHF
Once one has identified proxy points for the component reward models, the next question is how to train agents to maximize these rewards until they hit their critical values. We propose that useful approach to doing this is to reformulate the optimization objective using constraints.
Adding Constraints to RL In constrained reinforcement lcarning, an agent seeks to maxi- mize its value while adhering constraints on its behavior. Mathematically, this problem is formalized as constrained MDP (CMDP; Altman, 1999), which is defined a tuple Me A (S. A, P. ro, Y, p. {r.) [0.),N Here, S, A, P. ro- y, and are all as defined for standard MDPs (with To the reward function), with r; S x A + R. i IN...,N being constraint reward functions and 0g E R, i = 1..... .N associated constraint thresholds. Note that the subscripts on To:N are indices over reward functions, not time steps. For clarity, we will hereafter refer ro the
4

---

Published as a conference paper at ICLR 2024
"task reward" rather than just the reward. Rather than simply maximize value with respect to ru, the CMDP optimization problem is given by
max to S.L. U/ 2 0i, i 1,
(4.1)
That is, CMDPs represent behaviors which one would like constrain in the form of value estimates with respect to reward functions which measure these behaviors. The 2 symbol in Eq. (4.1) can easily be reversed if the constraint(s) encode behaviors which should be limited, and the inequality constraint(s) can be replaced with cquality constraint(s). While there are many possible formulations, we default to the canonical form in Eq. (4.1) ) for the purposes of exposition.
Proposed Method
Given our possible objectives, we can now consider how to optimize them. One popular approach to solving constrained problems such as Eq. (4.1) to use Lagrangian relaxation (Everett, 1963; Altman, 1999):
N Σ Me(UT 0.) a ((T,
max min D0 T 120
(4.2)
where the weights on the value of each RM μ [M1,.... AN] Ro are the Lagrange multipliers associated with each constraint. In the case that we use equality constraints rather than inequality constraints,
the variable rather than μ.. Optimization then proceeds by collecting experience using the policy and updating the policy and Langrange multipliers using gradient bessent-ascenn We stress that the Lagrange multipliers are not fixed hyperparameters, but rather are learned as part of the optimization process. The negative gradient with respect to is simply the constraint violation: VHLL(TT p)) = A: V.T To see how policy optimization works, we can rewrite the Lagrangian
C(T, p)) 10
(So, ao) 1ō
+
Σ
Hig (so, ao)
Σ
140,
(4.3)
ap a)~70-130))
i=1
N
(800 ao)]
Σ
Mr0i,
samo(t) ap/o0(1130)
where we define (s,a) A ()) + EN H.g. (s, a) as the mixed q-values of policy æ given the current Lagrange multipliers μ. Note that this value is non-stationary, as the same policy will have a different value the weightings proceeds as normal with respect to the mixed v-values. As is frequently done deep RL to reduce variance, we can replace the mixed q-values with mixed advantages Ag a q a) (s). with (s) Ea (s, a). We can optimize this objective with any policy gradient approach, in our case PPO. Detailed pseudocode is provided in Algorithm 1.
each constraint value change. Policy optimization then
Formal Guarantees While our focus is primarily empirical, we briefly comment on the theoretical properties of the above approach. Lagrangian relaxation converts the CMDP problem into min-max game. If the values are decomposed as u." = (Ti, d,), where (s, a) & (1 7) Σizo Pr(s, = s, a: a/ıT is the policy's cumulative, discounted state-action occupancy measure, and opti- mization is performed over dg, then the problem convex-concave and gradient descent-ascent (under basic assumptions) guarantees convergence of the average iterates to a saddle point, i.e., Ek) -
(d", n²) as the number of iterations K + oo (Freund and Schapire, 1997). However, in large-scale problems it is difficult to optimize directly over dz, and we instead update the policy directly. In this case, the problem is convex in μ but non-concave in Efroni et al. (2020) show sublinear regret bounds with respect to both policy optimality and constraint satisfaction using optimistic approach, and Ding al. (2020) show convergence rate for the averaged iterates for general smooth policy classes of O(1 K) for the policy and (11K))/4) for the constraint violation using natural policy gradients. There is significant work on primal-dual policy optimization for CMDPs, which discuss further in Appendix C.
K "K-EK
Ck=1
K
5

---

Published as a conference paper at ICLR 2024
Method
Objective
Intuition
PPO (no KL)
max, E; auy max E, a.2 s.t. UKL 2 8KL
Max. values
PPO
Max. values & stay close to To
New Methods
PPO-SAT
Find E [Tl 0, Vi}
Find 'feasible" π s.t. values hit targets
p-PPO
max, UKL s.t. v2 2 0: Vi
Stay close to To s.t. RMs high enough
All-PPO
max, E, a.y s.t. U, S 0 Vi, UKLL 0KL
Max. RMs but not too much Stay close to Tn & ensure RMs hit targets
S-PPO
max, 2Kl s.t. U; = 0g Vi
Table I: A summary of the approaches we consider.
Choosing a Constrained Objective Given this approach, we can now consider possible con- straint formulations, all of which should embody the intuition that the agent should maximize each component reward only until its corresponding proxy point. This naturally suggests that the proxy points should be used as thresholds in the constrained objective. However, there are a number of possible formulations to consider when casting RLHF as a CMDP with this goal in mind. Once the proxy point for given RM is reached, the agent has two options: continue to update the Lagrange multiplier on that RM to ensure that values remain at that point (via equality constraints), or simply stop optimizing/un tnightttt that RM entirely, i.e., set the multiplier zero, only re- weighting it if the constraint is violated (via inequality constraints). This latter approach carries that risk that the value with respect that RM will continue to increase (past the proxy point) as other RMs continue to be optimized, but may be empirically effective if this is not the case and optimization is simplified by having a source of non- -on-stationarity eliminated. In both of these cases, each component RM is assigned a constraint threshold, but the question of how to set the task reward remains. We propose x(orle.) log [o. the main task reward. Gao et al. (2022) liken the KL to resource which the agent spends, such that it should try to maximize its reward while limiting its divergence from the original policy as much as possible. Using the negative KL as the task reward carries the intuition of keeping the policy as similar as possible to the pretrained policy, subject the constraint that each RM hits the point beyond which it stops aligning with the true objective. Note that the requirement that the agent hits these thresholds is crucial, as it prevents the agent from fully maximizing the negative KL reward (i.e., remaining at the pretrained policy). In addition to these, there is another possible constrained approach wherein the agent simply maximizes the combined reward as in standard PPO (with KL regularization), but constrained so that each individual RM does not violate its respective threshold. All of these methods can be implemented by different settings of Algorithm 1. Finally, one could try to formulate the problem as one purely of constraint satisfaction: find any feasible policy whose values with respect to each of the RMs hit the appropriate proxy points. This could be implemented via a reward function that penalizes deviations from these points, e.g. TSAT = - Σ; (r; 8.) However, this approach (Algorithm 2) faces the same problem as standard PrO-namelly how to best set the weights o; These approaches are summarized in Table 1.
the KL. reward IKL
Hacks Here, we describe several practical modifications the "ideal" algorithm which we found to improve empirical performance. In practice, the noise and con-stationarity that primal-dual opti- mization in RI. must contend with can lead to instability in the updates for the Lagrange multipliers. To handle this in practice, we follow prior work (Stooke et al., 2020; Zahavy et al., 2022; Moskovitz et al., 2023a) and use a sigmoid function to bound the Lagrange multipliers between 0 and I. This results in mixed advantages which are a convex combination of the task and constraint advantages:
(s, = N Σ Eo(m))
A5 + Σ (s,)))7(s0a)
(4.4)
This equation has the intuitive interpretation of placing more weight on optimizing constraint reward Tiso when #450 is high (indicating a constraint violation), and more weight on task reward To when Mi:N are low (indicating that constraints are satisfied). When we use equality constraints rather than inequality constraints, we replace the sigmoid with a tanh function (bounding the Lagrange multipliers between -1 and 1). When updating the Lagrange multipliers, we found that using low or momentum in the optimizer (we use SGD with momentum parameter of 0.1) was helpful for

---

Published as a conference paper at ICLR 2024
0.0
0.70
Evaluation Score
065
0.66-
0.60
0.62
0.0
Orig. Train Time
0.58
0
20000
40000
60000
80000
100000
120000
0
100000
200000 Training Steps
300000
Training Steps
PPO
PPO-SAT
μ--PO
All-PPO
E-PPO
Figure 5.1: Constrained RLHF improves evaluation performance. (Left) Two constrained methods, μ-PPO and (-PPO produce the best performance over the course of training. (Right) Balancing RMs using constraints makes performance more robust to longer training time.
performance, as otherwise (J4) or tanh({;) could be overly "sticky," remaining high for too long when constraints became satisfied and vice to replace the value estimates in the constraint violation calculations with the sum of rewards to-go (for the appropriate reward function) for the remainder given episode. This is because we found that early in training, value estimates are inaccurate, which can cause the agent to incorrectly believe it is either adhering to or violating the constraint, leading incorrect weighting of rewards via the Lagrange multiplier and slower overall learning.
Another hack which we found be useful was
5 EXPERIMENTAL EVALUATION
We now evaluate these possible approaches in the same setting as described in Section 3. The primary questions we would like to answer are as follows. (1) Do constrained methods result in better evaluation performance compared to PPO (and PPO-SAT)? (2) Do these approaches successfully enforce the desired constraints? (3) Do the thresholds determined by the proxy points lead to the best performance? Unless otherwise noted, all experiments run for 5 random seeds, and any shading in plots denotes standard error. Code for all methods is available here: g github. com/ Ledmoskove tz Ca medRL4IMS
Does constrained RLHF improve performance? In Fig. 5.1, we indeed find that two constrained approaches. μoPPOO and E-PPO achieve better evaluation performance than other methods, with E-PPO performing slightly better at the end of training. To ensure fairness across methods, to set the fixed RM weightings used to train PPO and PPO-SAT, we selected the best settings found after 10 initial runs of each approach, the same as the total number of runs used to find proxy points used for the constrained methods. We conjecture that the strong performance of μ- and {-PPO due to the beneficial effects of jointly optimizing the policy and Lagrange multipliers (RM weightings). For example, even setting the weightings to be the optimal Lagrange multipliers and fixing them throughout training is not guaranteed to converge observed empirically by Moskovitz et al. (2023a). Notably, AlI-PPO did not perform as well as the other constrained methods, which we believe process (Appendix Fig. D.3). This is common in constrained problems with "paradoxical" objectives (Moskovitz et al., 2023a). Another benefit of continually modulating the weightings among RMs is that the weightings themselves are not hyper-oplimized to particular training duration. We trained both PPO and E-PPO using their hyperparameter settings optimized over runs with 128,000 steps for 3 times as long over 3 seeds and confirmed that the constrained approach was more stable (Fig. 5.1).
saddle point (Szepesvári, 2020), phenomenon
due to increased instability in the optimization
Are constraints successfully enforced? To verify that the constrained algorithms are working expected, we plotted the intent and METEOR rewards across training for μ-PPO, All-PPO, and E-PPO in Fig. 5.2. We can see that, as required by the constraints, p-PPO (approximately) reaches at least high the proxy point thresholds, All-PPO remains below them, and C-PPO approximately
7

---

Published as a conference paper at ICLR 2024
Intent Constraint Sadisfcction
METEOR Constraint Satisfaction
0.55
0.24
METEOR Score
0.22
Intant Store
0:45
0.20
0.40
0.18
0.16
0.30
20020
40000
40000
85000
100000
120000
20200
40000
60000
50000 100000 120000
Training Steps
Training Steps
PPPP
AII-POO
E-PPO
Figure 5.2: Constraints are satisfied. Pt-PPO reaches or exceeds the required intent (left) and METEOR (right) thresholds (dashed lines), All-PPO remains below them, and {-PPO hits them.
Enluation Score
0.65
0.60
0.55
Proxy Point Thresholds
Low Thresholds
Independent Proxy Points
0.50
High Thresholds
Correlated Praxy Points
20000
40000
60000
80000
100000
120000
20000
40000
60000
50000
100000
120000
Training Sceps
Training Steps
Figure 5.3: Using proxy points as thresholds leads to the best performance. (Left) Using thresholds that are 10% lower or higher reduces performance compared proxy point thresholds. (Right) The proxy points that account for the correlation between RMs are more effective than those estimated independently.
hits them. Pt-PPO continues increase above the intent proxy point, which may contribute to its slightly worse final performance compared to E-PPO in Fig. 5.1.
Are proxy points the best thresholds? We compared the performance of E-PPO using the proxy points identified in Section 3 against the same method using thresholds that were 10% lower and 10% higher. The left panel of Fig. 5.3 shows that making thresholds lower causes initial performance to increase more quickly, as once the casier-to-reach thresholds are met, the agent able to begin tightening the KL with respect to the pretrained policy earlier. However, performance plateaus at a lower level. When thresholds are too high, the KI. reward is ignored and the proxy rewards are optimized beyond the point at which they are useful, leading to worse performance. We also compared the performance of E-PPO using the correlated proxy points found Fig. 3.2 against the independent proxy points found by only considering one RM at time (Fig. 3.1).
5.1 IMPROVING THRESHOLD IDENTIFICATION
One downside of all methods considered so far is the need for multiple runs to either select fixed weighting of RMs or identify proxy points. would save significant compute- and reduce environmental impact, particularly for larger models-if it were possible to identify thresholds over the course single training run. Assuming we are allowed a limited number of queries to the evaluation metric over the course of training, one approach to accomplishing this would be to use gradient- optimizer to update the constraint thresholds to reach better performance. In order to limit the required number of policy updates between threshold updates, used local hill-climbing algorithm, Nelder-Mead (Nelder and Mead, 1965), which iteratively updates simplex of thresholds based the evaluation performance at each point. Once set of thresholds is proposed,
8

---

Published as a conference paper at ICLR 2024
Nelder-Mead Search Reduces Computation
Nelder-Mead Proxy Point Trajectory
0.72
0.8
0.4
Score
Final Evaluation Score
0.70
METEOR Threshold
022
0.68
030
0.5
0.2
0.66
PPO
0.66
Inical Simplex Incermediace Simplex
9.4
0.64
E-PPO
14
NM-PPO
Simplex
62
06
088
12
1.4
0.38
0.40
0/12
0.16
0.8
050
0.52
Total Training Steps
Incent Threshold
Figure 5.4: Nelder-Mead threshold search saves computation. (Left) Final evaluation performance versus total number of training steps (including hyperparameterss searches). We allowed NM-PPO twice as many training steps for a single run, 256,000. (Right) An example threshold simplex trajectory overlaid on a contour plot of predicted evaluation performance from Fig. 3.2. The search converges to a local maximum.
use E-PPO converge to those points and then evaluate the model once they're 're reached. Details are provided in Appendix A.4. We plotted the final evaluation performance of this variant of our approach, which we term NM-PPO (Algorithm 4), versus total number of training steps (including runs used for hyperparameter optimization) of PPO and E-PPO in Fig. 5.4. We found that NM-PPO obtains strong performance over the course of single run, significantly saving in computation. Furthermore, the trajectories of simplexes proposed by Nelder-Mead closely follow the predicted evaluation performance found in Fig. 3.2, converging to local maxima of the surface. Fig. 5.4, the trajectory converges to local maximum rather than the global maximum, though other runs did indeed find the global optimum as predicted by Fig. 3.2 (Appendix Fig. D.5). One caveat with respect this result is that the feasible region of threshold pairs is relatively small. There is therefore a moderate chance that the initial simplex already contains at least one threshold pair which produces reasonable performance. Further experimentation is required problems with larger feasible regions and more than two component RMs.
DISCUSSION
In this work, we studied reward model overoptimization and the influence of correlation on proxy points in composite RMs. Then, we introduced set of approaches for identifying and using these points as thresholds within constrained optimization approach to RLHE. One weakness shared by all saarrachhs--ncoossraan to the true objective/evaluation metric required. Though in resource-rich settings this could be feasible (e.g., by occasionally freezing training and querying human evaluators using Al feedback), ideally, this would be dispensed with entirely. However, doing so is beyond the scope of this work. One weakness of gradient descent-ascent applied to primal-dual policy optimization is that it does not guarantee that the final policy and Lagrange multiplier(s) converge to a saddle point, only their averages. It would be an interesting direction for future work to apply an approach which does have such guarantees, such as ReLOAD (Moskovitz et al., 2023a). For optimizing the constraint thresholds during single run, it would be interesting explore alternative optimizers Nelder-Mead, such as Bayesian optimization. Another interesting direction for future work would be to study the usefulness of CMDP formulation for avoiding degenneration/collapse of model outputs, as while deterministic optimal policy always exists for standard MDPs, CMDPs may demand optimal policies which are stochastic (Szepesvári, 2020). similar idea was explored using a maximum entropy formulation by Khalifa et al. (2020). In general, further testing of our methods is necessary more domains and with composite RMs with more components. We believe there are additional interesting avenues to explore in mitigating overoptimization, such multi-objective RL (Abdolmaleki et al., 2020) with constraints added to supervised learning (Rafailov et al., 2023). More broadly, we believe constrained optimization offers important toolbox for approaching the alignment problem.
and constrained alike-is that at least some minimal degree of access
Acknowledgemente Ted Moskovitz is funded by the Gatsby Charitable Foundation. Tuomas Sandholm is supported by the Vannevar Bush Faculty Fellowship ONR 900014-23-1-2280 National

---

Published as a conference paper at ICLR 2024
Science Foundation grants RI-2312342 and RI-1901403, and ARO award W911NF2210266. Stephen McAleer funded by a CI Fellowship. The authors would like to thank Vivek Veeriah, Tom Zahavy, Misha Laskin, and Dave Abel for helpful discussions.
10

---

Published as a conference paper at ICLR 2024
REFERENCES
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportuni- ties and risks of foundation models. arXiv preprint arXiv:2108.0722 077288, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
OpenAl. Gpt-4 technical report, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0922 2023.
Wasi Uddin Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Wei Chang. Unified pre-training for program understanding and generation. arXiv preprint ar arXiv:2103.06333: 2021.
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8696-8708, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10L18865/v1/2222 1tt6:////10011...mmmmam.......o 2021. emnlp-main. 685.
URL htwph:ttppe.cccc /ttp///llaaaaa http///aaaaaatttogco/oe org/
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, Jérémy Rapin, et al. Code Ilama: Open foundation models for code. arXiv preprint arXiv: 12950, 2023.
W Bradley Knox and Peter Stone. Tamer: Training an agent manually via evaluative reinforcement. In 2008 7th IEEE international conference on development and learning, pages 292-297. IEEE, 2008.
Paul FF Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. Advances neural information processing systems, 30, 2017.
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021 2020.
2020.
Dan Hendrycks, Geoffrey Hinton, Yoshua Bengio, Sam Altman, Ilya Sutskever, Bill Gates, and Grimes. Statement on ai risk, May 2023.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.12712 2023.
Shane Legg. Machine super intelligence. University of Lugano, 2008
Emily M Bender, Timnit Gebru, Angelina McMillan-Major and Shmargaret Shmitchell. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on faimess, accountability, and transparency, pages 610-623, 2021.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109, 109.09999. 2021.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models, In 30th USENIX Security Symposium (USENIX Security 21), pages 2633-2650, 2021.
11

---

Published as a conference paper at ICLR 2024
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. arXiv preprint arXiv:2210.10760, 2022.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimization. arXiv preprint arXiv:2210.01241, 2022.
Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, et al. Improving alignment dialogue agents via targeted human judgements. arXiv preprint arXiv:2209.1433 2022.
Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023.
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Camphell-GGllinghaan, Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, et Fine-tuning language models to find agreement among humans with diverse preferences. Advances in Neural Information Processing Systems, 35:381766 176-3811,,8,, 2022.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine- Fine-grained human feedback gives better rewards for language model training. arXiv preprint arXiv:2306.01 01693, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Richard Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 11999999899, 2019.
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng. Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan. Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov. Xin Chen, Lauro Langosco, Peter Hase, Erdem BIyIk, Anca Dragan, David Krueger, Dorsa Sadigh, and Dylan Hadfield-Menell. Open problems and fundamental limitations of reinforcement learning from human feedback, 2023.
Charles AE Goodhart and CAE Goodhart. Problems of monetary management: the UK experience. Springer, 1984.
Yanran Li, Hui Su, Xiaoyu Shen, Wenjie Li, Ziqiang Cao, and Shuzi Niu. Dailydialog: A manually labelled multi-turn dialogue dataset. arXiv preprint 1rX.v:1710.03957, 2017.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
12

---

Published as a conference paper at ICLR 2024
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pages 65- 2005.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907, 1692, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint Xiv: 1707:07777 2017.
Eitan Altman. Constrained markov decision processes, 1999.
Hugh Everett. Generalized lagrange multiplier method for solving problems of optimum allocation of resources. Oper. Res., 113):399-417, jun 1963. URL hwwp::ttt/s/ttt /ttp::///ooooo/////////oo//////. /opre.
11.3 3.999.9.
Yoav Freund and Robert E Schapire. A decision-theoretic generalization of on-line learn- ing and an application to boosting. Journal of Computer and System Sciences, 55(1): 119-139, 1997. URL https: //t::///wwwwww www.://ccennee......... www.e//cccirottt.cc///eeeeeeeeee S0C22000097915042
Yonathan Efroni, Shie Mannor, and Matteo Pirotta. Exploration-exploitation in constrained mdps. 2020. URL https: http///axxvvv.o/// orgg abs /22//2203.0222
Dongsheng Ding. Xiaohan Wei. Zhuoran Yang. Zhaoran Wang, and Mihailo R. Jovanovié. Provably
efficient safe exploration via primal-dual policy optimization, 2020. URL httts:///axxii
org/ abs/ 2003. 00534.
Adam Stooke, Joshua Achiam, and Pieter Abbeel. Responsive safety in reinforcement learning by pid lagrangian methods, 2020. URL https: org /ttp////.2..2 03964.
Tom Zabavy, Yannick Schroecker, Feryal Behbahani, Kate Baumli, Sebastian Flennerhag, Shaobo Hou, and Satinder Singh. Discovering policies with domino: Diversity optimization maintaining near optimality, 2022. URL https: www.iv.voogggo..oo....oooooo.. abs 2205.13521.
Ted Moskovitz, Brendan O' O'Donoghue, Vivek Veeriah, Sebastian Flennerhag, Satinder Singh, and Tom Zahavy, Reload: Reinforcement learning with optimistic ascent-descent for last-iterate convergence in constrained mdps. arXiv preprint arXiv:2302.01275, 2023a.
Csaba Szepesvári. 2020.
Constrained mdps and the reward hypothesis, Mar wtt.gsmm..llosoo wlo.spoo.....oo.oooooooo........ com/ 20mm/200033////// constrainedd mdps aadaaandddaaee wew---eeawaddteereeeeeee hypothesistsssssseeeeee html.
URL
John A Nelder and Roger Mead. A simplex method for function minimization. The computer journal, 744):308-313, 1965.
Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. A distributional approach to controlled text generation. arXiv preprint arXiv:2012.1 1635, 2020.
Abbas Abdolmaleki. Sandy Huang, Leonard Hasenclever, Michael Neunert, Francis Song, Mar- tina Zambelli, Murilo Martins, Nicolas Heess, Raia Hadsell, and Martin Riedmiller. A dis- tributional view on multi-objective policy optimization. Hal Daumé and Aarti Singh, editors, Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 11-22. PMLR, 13-18 Jul 2020. URL ngs ml p.press////e............r v11 9/ aa/abdolmme ma 1 eki20a hhmll
https:://procee
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.1 18290, 2023.
Matt Post. A call for clarity in reporting bleu scores. arXiv preprint Xixiv: 1ee.180000777 2018.
13

---

Published as a conference paper at ICLR 2024
Chin- Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 2004.
Kavita Ganesan. Rouge 2.0: Updated and improved measures for evaluation of summarization tasks.
arXiv preprint arXiv: 1oXiv:1803.00 803.01937, 2018.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-33113 3-13318, 2002.
Joshua Maynez, Priyanka Agrawal, and Sebastian Gehrmann. Benchmarking large language model capabilities for conditional generation. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pages 9214-9213, Toronto, Canada, July 2023, Asso- ciation for Computational Linguistics. doi: 10.186653/1/22223.ccll //aclantho ogy- or org/2023. cocccllllngee 511.
URL https:
Ciyou Zhu, Richard H Byrd, Peihuang Lu, and Jorge Nocedal. Algorithm 778: L-bfgs-b: Fortran subroutines for large-scale hound-constriined optimization. ACM Transactions on mathematical software (TOMS), 234))555--560 1997.
Pauli Virtanen, Ralf Gommers, Travis E Oliphant, Matt Haberland, Tyler Reddy, David Coumapeau,,
Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, al. Seipy 1.0: fundamental algorithms for scientific computing in python. Nature methods, 17(3):261-272. 2020.
Vivek S Borkar. An actor-critic algorithm for constrained markov decision processes. Systems & control letters, 54(3):207-213, 2005.
Shalabh Bhatnagar and K Lakshmanan. An online actor cetor-critic algorithm with function approximation for constrained markov decision processes. Journal Optimization Theory and Applications, 153 (3):688-708, 2012.
Joshua Achiam, David Held, Aviv Tamar, and Picter Abbeel. Constrained policy optimization. In Doina Precup and Yee Whye Teh, editors. Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 22-31. PMLR, 06-11 Aug 2017. URL https: http::://proceeeli ngs. mlr. wwwpprresvvvve wwtsss//vveeeee acth//cchmmit am17a. html.
Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, and Mohammad Ghavamzadeh. A lyapunov-based approach safe reinforcement learning. In S. Bengio, H. Wal- lach, H. Larochelle, K. Grauman, N. Cesa-BBinnchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.. URL httpss:/prroeedings Afee1140399a0 75777755d66666 65ee666eeeeeee11116660000---00-0 2ff.554400033375556666//62221210
2018.
http:///ng....iiooooo//////////p cc/paper//018/E11.
Santiago Paternain, Luiz Chamon, Miguel Calvo-Fullana, and Alejandro Ribeiro. Constrained rein- forcement learning has zero duality gap. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc. E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems. volume 32, Curran Associates, Inc., 2019. URL paper/2019 Ei 1
neurips. cc/ pdf.
Chen Tessler, Daniel J. Mankowitz, and Shie Mannor. Reward constrained policy optimization. In International Conference on Learning Representations, 2019. URL https::
net/ n 2tt.:/ooummn1---22veA8--
Dan A. Calian, Daniel J. Mankowitz, Tom Zahavy, Zhongwen Xu, Junhyuk Oh, Nir Levine, and Timothy Mann. Balancing constraints and rewards with meta-gradient d4pg, 2020. URL https:
//arxivv.. org/ abs/2010...0../ 06344.
Yuhao Ding and Javad Lavaei. Provably efficient primal-dual reinforcement learning for cmdps with non-stationary objectives and constraints. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 7396-7404, 2023.
14

---

Published as a conference paper at ICLR 2024
Sindhu Padakandla, Prabuchandran KJ, and Shalabh Bhatnagar. Reinforcement learning algorithm for non-stationary environments. Applied Intelligence, -035900-600,, 2020.
Wang Chi Cheung, David Simchi-Levi, and Ruihao Zhu. Reinforcement learning for non-stationary markov decision processes: The blessing of (more) optimism. In International Conference Machine Learning, pages 1843-1854. PMLR, 2020.
Erwan Lecarpentier and Emmanuel Rachelson. Non-stationary markov decision processes, a worst-case approach using model-baset reinforcement learning. Advances in neural information processing systems, 32, 2019.
Khimya Khetarpal, Matthew Riemer, Irina Rish, and Doina Precup. Towards continual reinforcement learning: A review and perspectives. Journal of Artificial Intelligence Research, 75:1401-1476, 2022.
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst lifelong non- stationarity. arXiv preprint arXiv:2006. 10701, 2020.
Annie Xie, James Harrison, and Chelsea Finn. Deep reinforcement learning amidst continual structured non-stationarity. In International Conference on Machine Learning, pages 1 11393-114 11403. PMLR, 2021.
Brendan O' O'Donoguue. Efficient exploration via epistemic-risk-seekiie policy optimization. arXiv preprint arXiv:2302.09339 2023.
Jean Tarbouriech, Tor Lattimore, and Brendan O' O'Donoghue. Probabilistic inference reinforcement learning done right. In Sixteenth European Workshop Reinforcement Learning. 2023.
Ted Moskovitz, Spencer R Wilson, and Maneesh Sahani. A first-occupancy representation for reinforcement learning. In International Conference on Learning Representations, 2021a.
Ted Moskovitz, Samo Hromadka, Ahmed Touati, Diana Borsa, and Maneesh Sahani. A state representation for diminishing rewards. arXiv preprint arXiv:2309.037111 033100 2023b.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, Rafal 16zefowicz, Scott Gray, Catherine Olsson, Jakub Pachocki, Michael Petrov, Henrique Pondé de Oliveira Pinto, Jonathan Raiman, Tim Salimans, Jeremy Schlatter, Jonas Schneider, Szymon Sidor, Ilya Sutskever, Jie Tang, Filip Wolski, and Susan Zhang. Dota 2 with large scale deep reinforcement learning, CoRR, fbss/1912.06680 2019.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL RL with importance weighted actor-learner architectures. In The International Conference on Machine Learning (ICML). JMLR, 2018
Sham Kakade and John Langford. Approximately optimal approximate reinforcement learning. In IN PROC. 19TH INTERNATIONAL CONFERENCE ON MACHINE LEARNING, pages 267-274, 2002.
Ted Moskovitz, Michael Arbel, Ferene Huszar, and Arthur Gretton. Efficient wasserstein natural gradients for reinforcement learning. In International Conference on Learning Representations, 2021b. URL httpttt:////peeee//r/e
net/ Forum? Id=OHgn
Aldo Pacchiano, Jack Parker-Holder, Yunhao Tang, Krzysztof Choromanski, Anna Choromanska, and Michael Jordan. Learning to score behaviors for guided policy optimization. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 7445-7454 PMLR, 13-18 Jul 2020. URL httpttt/ppoceediieeeeee
mlr http:///ww.....e/e////ccccccccca ano20a. html.
John Schulman, Sergey Levine, Pieter Abbeel, Michael I. Jordan, and Philipp Moritz. Trust region policy optimization. In Francis R. Bach and David M. Blei, editors, ICML, volume 37 of JMLR Workshop and Conference Proceedings, pages 1889-1897. JMLR.org, 2015. URL http:// proceedi ngs l /tg.eede//// schu 1ann1 5. html.
15

---

Published as a conference paper at ICLR 2024
Sergey Levine. Reinforcement learning and control probabilistic inference: Tutorial and review, 2018.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor, 2018.
Abbas Abdolmaleki, Jost Tobias Springenherg, Yuval Tassa, Remi Munos, Nicolas Heess, and Martin Riedmiller. Maximum posteriori policy optimisation, 2018. URL
Alexandre Galashov, Siddhant M. Jayakumar, Leonard Hasenelever, Dhruva Tirumala, Jonathan Schwarz, Guillaume Desjardins, Wojciech M. Czarnecki, Yee Whye Teh, Razvan Pascanu, and Nicolas Heess. Information asymmetry in kl-regularized RL. CoRR, abs/1905.01240, 2019. URL
http://arxivvooo org/
Yee Whye Teh, Victor Bapst, Wojciech Marian Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Proceedings the 31st International Conference on Neural Information Processing Systems, NIPS' 17, page 4499 4509, 2017.
Ted Moskovitz, Michael Arbel, Jack Parker- Parker-Holder, and Aldo Pacchiano. Towards an understanding of default policies in multitask policy optimization. In International Conference on Artificial Intelligence and Statistics, pages 10661-10686. PMLR, 2022.
16

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL DETAILS
A. SETTING
We use the same general experimental setting as Ramamurthy et al. (2022). The context window was of length 5, and separating the conversations in this way resulted in 35k training, 3k validation, and 3k test utterances. As Ramamurthy et al. (2022), we use top-k; k = 20 sampling for decoding. The inputs to the model are concatenated snippets of human conversation in which changes of speaker are denoted by a special end-of-utterence (<EOU>) token. The intent classifier reward was derived from a finetuned RoBERTa model (Liu et al., 2019) which awards score of if the classified intent of the model's s utterance matches that of the reference/ground truth utterance and 0 otherwise.
A.2 THE EVALUATION METRIC
As we note in the main text, our objective in constructing an evaluation metric was to find one for which Goodhart's Law holds with respect to both the METEOR and intent reward functions, not to directly model human preferences. We therefore chose three metries measuring lexical quality and three metrics measuring text diversity from among the metrics available in the RL4LMS codebase published by Ramamurthy et al. (2022). Specifically, the lexical metrics we used were SACREBLEU r, (Post, 20, ROUGE2 x, (Lin, 2004; Ganesan, 2018), and BLEU Ng (Papineni et al., 2002), and the diversity metrics we used were unique 3 Zas vocab_ Im. For each metric, we individually normalized the
3-nouunct
Tn. and max_pred_ @pre__engthh score between 0 and I (based on the range of observed values across all runs of PPO METEOR and PPO Intent), then averaged the resulting lexical scores and resulting diversity scores, before averaging the two average category scores. More precisely:
eval_score 1
Zg I. + T6
Tx Iv + Imm
(A.1)
3
3
Given the use of GPT2 on a relatively structured Dialog dataset, we opted for text overlap metrics for relevance, supplemented by diversity metries Khalifa al. (2020). Future work on larger models and more involved tasks (e.g., summarization Stiennon et al. (2020)) could consider more sophisticated evaluation methodology, taking inspiration from works such as Maynez al. (2023). For this work, we focused on simple evaluation metric designed to facilitate our study of overoptimization, rather than to emulate human ratings or rank different PLMs.
A.3 FITTING THE EVALUATION SCORE SURFACE
The overall procedure is described in Phase of Algorithm 3, where F is the function class for the evaluation score estimator. In our case, F was the space of polynomials of degree 10. To avoid predicting high evaluation scores over regions of the METEOR x intent space with little or no data points, we employed kernel density estimation with a Gaussian kernel to create a mask which hid parts of the fitted surface over low-density data regions (with a threshold density of 50/square unit). This approach is purely heuristic and could likely be greatly improved on in future work. To find the argmax of the surface, we tried two approuches. First, we used the built-in implementation of L-BFGS-B B/ Zhu et al. (1997) in the SciPy optimi mi ze package (Virtanen et al., 2020) using the results from KDE to provide bounds. Second, since the polynomial surface approximated using a discrete mesh we also simply picked the coordinates corresponding to the plotted maximum. There was negligible difference between methods, so we used the second approach in practice.
A.4 NELDER HELDER-MEAD PPO DETAILS
We provide detailed pseudocode of our approach in Algorithm 4. In practice, we found several implementation details to be important for ensuring good performance. First, the initial simplex was crucial. Rather than initialize thresholds randomly across the entire range of possible METEOR and intent values, we initialize them based on random perturbations of the evaluation of the ini- tial/pretrained policy (i.e., what the METHOR and intent scores are at the beginning of finetuning). This was very helpful, as otherwise Nelder-Mead would propose threshold pairs that were effectively not feasible for the policy to achieve, e.g., a very high METEOR threshold with a very low intent threshold. Second, we capped the number of iterations allowed for one evaluation/threshold setting at 1/8 of the total allowed training steps. Without this, the agent would often waste most of its run trying to hit challenging/infeen /innesssiifeeeeeeee thresholds. If the thresholds couldn't 't be reached in that time, the evaluation score was computed wherever the agent was at that time. Third, the agent cached the eval scores of previously-reachen threshold pairs-if if Helder-Mead proposed a threshold pair that had
17

---

Published as a conference paper at ICLR 2024
been reached before (or is within clementwise tolerance of +5% of previously-reachen pair) then it just returns the evaluation score it measured previously rather than updating the policy to return to 1 2, p 0.5, 4.5-these settings
it. The Nelder-Mead hyperparameters untuned, and could likely adjusted to improve performance.
A.5 COMPUTATIONAL RESOURCES
All experiments were performed on single NVIDIA A100 GPU, with each run taking between and 10 hours with the exception of for Nelder-Mead PPO, which took approximately 20 hours.
A.6 ALGORITIM HYPERPARAMETERS
Hyperparameter
PPO
PPO-SAT
μ-PPO
All-PPO
E-PPO
Steps per Update (M')
1,280
1,280
1,280
1,280
1,280
Total Steps (KM))
128,000
128,000
128,000
128,000
128,000
Batch Size (B)
64
64
64
64
Epochs per Update (L)
5
5
5
Learning Rate (17)
le-6
1e-6
le-6
le-6
1e-6
Initial KL Coefficient (ao)
0.2
0.2
0.2
0.2
0.2
Target KL
0.5
0.5
0.5
Discount Factor (r)
0.99
0.99
0.99
0.99
0.99
GAE À
0.95
0.95
0.95
0.95
0.95
Clip Ratio (c)
0.2
0.2
0.2
0.2
0.2
Rollouts Top-k
20
20
20
20
20
Value Function Coefficient (a.)
0.5
0.5
METEOR Coefficient (a (met)
0.5
0.5
Intent Coefficient (n
1.0
1.0
METEOR Proxy Point (8* (ftnetor)
0.23
0.23
0.23
Intent Proxy Point (8* intent)
0.48
0.48
0.48
METEOR Value Coefficient
0.5
0.5
0.5
Intent Value Coefficient
0.5
0.5
0.5
KL Value Coefficient
0.2
0.2
Lagrange Multiplier Function
sigmoid
sigmoid
tanh
Table Experiment Hyperparameters.
B THE KL REGULARIZATION COEFFICIENT
As introduced by Ziegler al. (2019), it is common in RLHF with PPO to adapt the KL coefficient KL with the following update:
[KL[T(-||s) (s)))))))))))))) mo()) OKL
= clip
-..2,
ØKL
KL 0++1
KL a + KLe)).
where gKL is a hyperparameter which effectively acts as an upper limit on the KL from the initial policy, and 1 acts like learning rate. The KL coefficient then follows the path of Lagrange multiplier with gKL as its constraint threshold, as the constraint violation (L(T(-Isx): mo(+)s))] gKLL exactly the gradient with respect such Lagrange multiplier.
C ADDITIONAL RELATED WORK
In addition to the discussion in the main text, there is long history work on CMDPs. Borkar (2005) first studied actor-critic approaches in this context, and Bhatnagar and Lakshmanan (2012) were the first to consider constrained policy optimization with function approximation. More broadly, Achiam et al. (2017), Chow et al. (2018), Paternain al. (2019), Tessler et al. (2019), Calian et al. (2020), Efroni et al. (2020), Stooke et al. (2020), Moskovitz et al. (2023a), and Ding and Lavaei (2023) all study the problem of integrating constraints into RL. More generally, important factor
18

---

Published as a conference paper at ICLR 2024
Algorithm 1: Constrained PPO for Dialogue Generation
1: Require: Dataset D ((x", ) initial policy parameters v(1,, initial parameters for value functions % ON constraint thresholds 0f!) ,...,0% ...,, ,0
initial Lagrange multipliers
μ()), squashing function f E {a, tanh)
2: for step I..... K do
3: // Sample experience
4: Uniformly sample M' < M contexts xm² ~ (()
5: Generate predicted "trajectory" responses
T
y = -a1....., ~ px II T(a)ls))
where $1 xm
6:
Compute generalized advantage estimates:
(A.). (6:). + 71((6))+11 + +... (()) 1= ON,...,
where (6.), - (s ats s141. ym (8.4.)))) u1(s))
7: Store advantages and trajectories buffer B
8:
// Update
9:
for epoch L L..... do
10:
for trajectory batch {((Au.N) )b. (ōo)b..... (8N) 9o)) U(B) in B do
Compute mixed advantage estimates:
N
N
(A =
N
(1((t)
(Av)
(R)
14
(K)
(A))
12:
Compute the policy loss:
LPPO
Σ
min winten ena (v: (AAW))) clip((pa((s 1 e (,,11)))4 +A)
BT
b= 0
x (k) (am))me) mmc)
where Pbe (1;(()))))
Tick رد |ass| [mse]"
13:
Compute the value function losses:
B
C,
16:)
i= 1,0,...., N
BT
t=
14:
Update the policy and value functions via SGD LpPo + au E Cv, Update the Lagrange multipliers via SGD on Ly:
15:
Cp.
( 0.)5 (1.,²)
(K)
BT
16:
end for
17:
end for
18: Reset buffer B Ø
19:
end for
in using Lagrangian approach to solving CMDPs is the introduction of non-stationarity into the reward function. RL with non-stationary rewards is an active area of interest in RL (Padakandla et al., 2020; Cheung et al., 2020; Lecarpentier and Rachelson, 2019), particularly in the context of continual RL (Khetarpal et al., 2022) often with some form of temporal structure introduced in the non-stationarity (Xie et al., 2020; 2021). An interesting in additional to primal-dual optimization
19

---

Published as a conference paper at ICLR 2024
Algorithm PPO-SAT for Dialogue Generation
1: Require: Dataset D {(x",, )) initial policy parameters v(), initial parameters for
value function ((1), proxy point estimates 01,...,0:: ON. RM coefficients ,N,,,,,,
2: for step k= l..... do
3: // Sample experience
4: Uniformly sample M' < M contexts xT' U(D)
5: Generate predicted 'trajectory' responses
T
mm' (T) px II 1=1
T(a)l)))
where 81
xmn
6:
Compute generalized advantage estimates:
A = 6 + 78441 + .... (7)2--+t10-- 8r
where 6, -
-is (ri(ri(8e, at 81+1, ymh 0,2 + yo (81+1) - Gsras) (81).
7: Store advantages and trajectories in buffer B
8:
// Update
9:
for epoch é = ..... L do
for trajectory batch { {((As. (6o) (6N).. L(() in B do
11:
Compute the policy loss:
(ppO(()()))
clip((po((b())) -
- e, +)A())
BT
b=1 1=
(e (oselese)
where pbe (1)(k)
(k (aos|ser)
12:
Compute the value function loss:
B T
(o(())
BT
2
bt
1=1
t=
Update the policy and value functions via SGD on Lppo + anLo
14: end for
end for
16: Reset buffer B +- Ø
end for
in which hon-stationarity is introduced by the agent itself is in the use of epistemic uncertainty for more efficient exploration, manifested in the form of non-stationary exploration bonuses reward (O' D'Donoguue, 2023; Tarbouriech et al., 2023). Non-stationarity may also be introduced as means of modeling more naturalistic reward structures for studying animal behavior (Moskovitz al., 202 la; 2023b). Finally, another area of related work is regularized policy optimization, whereby the standard maximizin policy optimization objective augmented with a regularization term, typically divergence measure with respect to some reference policy (Berner et al., 2019; Espeholt et al., 2018). 8). In the single-task setting, the updated policy is typically regularized to stay close to its current setting, which has close connections to natural gradient (Kakade and Langford, 2002; Moskovitz et al., 2021 b; Pacchiano et al., 2020), trust region (Schulman et al., 2015), and variational inference (Levine, 2018; Haarnoja et al., 2018; Abdolmaleki et al., 2018 8) approaches. In the multitask setting, the policy is typically regularized towards some default policy which encodes behavior thought to be useful across family of tasks, and which may be far from the current policy (Galashov al., 2019; Teh et al., 2017; Moskovitz et al., 2022). This setting is quite similar in this sense to KL regularization as used in RLHF.
20

---

Published as a conference paper at ICLR 2024
Algorithm 3: Two-Phase Approach
1: Require: Proxy RMs F1:N T1. :
.N. Evaluation RM r", policy gradient algorithm Alg.
constrained algorithm CAlg (e.g,, Algorithm 1)
2: Phase 1: Proxy point identifi ficaiioi
3: Evaluation RM dataset D <
4: for ,., do
5: Fit Alg on collect K measurements {(11.... ,N,, rb)kk across training
6: DU (x,
7: end for
8: Fit evaluation RM predictor f
N
K
f argmin FrEF NK
(ru fr(1))) ((x)))))...,(()))))))
R=1
9: Proxy point: 0* + argmax Py ft (TI,...,NNNN)
10: Phase 2: Constrained optimization
n <- CAlg(O")
12: Return x²
Algorithm Nelder-Mead Proxy Point Search
1: Require: Evaluation RM Tenats initial simplex thresholds {0, (0n.)) N+ ) reflection coefficient ax, expansion coefficient 7, contraction coefficient p. shrinkage coefficient a
2: Fit E-PPO using initial thresholds, compute (veva((0)))
3: while not converged do
4: Sort threshold by evaluation score (01,.... 1N+1)
5: Compute the centroid of the N-best thresholds 0
1 N Zi=1
6: Reflect the worst point 0, = + a( N++1)
7:
Fit EPPOO the reflected thresholds and compute voval (0,)
8:
if va (01) S venat(0,) < ( then
9:
ON+1 0,
10: GOTO Line 3
end if
12:
if
w "a < ox
cval (o) then
13:
Expand: 0 = (0, 0)
14:
Fit E-PPO the expanded thresholds and compute o (0.)
15:
if o
(0.) < U ( then
ON 0.
17:
GOTO Line 3
else
19:
0x 0,
GOTO Line
21:
end if
22:
end if
23:
Contract: 0 0 p(0 - 0)
24:
Fit EPPO on the contracted thresholds and compute v (0.)
25:
if vvval (0.) < 2i77 v (1) then
26:
ON+1 = 0.
27:
GOTO Line 3
28:
end if
Shrink: 0, 0 + o(0, 01). j 2..,.,
30:
end while
21

---

Published as a conference paper at ICLR 2024
rouge_ rouge
24094
0 100
0.15
0.080
1770
Score
3.0000
0.085
22028
0.160
0.00
0.155
3.0044
0.065
1000CO 120000
20000
10000
20002
2000
rougo_ rougel
rougo rouge2
rouge rougelssn
0.135
0.445
0.145
Score
28120
0.155
28115
0.155
0.30
20000 40000 E0000
B3030 LOCOCO 120000
20000
4000s 500C0 HEOG) 1EC0CO
20000
20002
4000c st600 1oo0
(00000 20000
bort Score
perpiexity
cond uodd_ontreyy--nopunet
2.0
135
4.7
635
20000 40000
ED0EO
93030 LDCECO 120000
20000 4050E Training Steps
SEDEO
HEOM
IDCECO 20000
20002 40002
SE0EO
(OC0CO 20000
Training Staps
Training Staps
PPO
PPO-SAT
μ-PPO
AI-PPO
EPPPO
Figure D.I: Performance of the tested methods across various metries.
D ADDITIONAL RESULTS
D. ADDITIONAL METRICS
22

---

Published as a conference paper at ICLR 2024
Nelder-Mead Performance with RMs
0.72
PPO
NM-PPO
Final Evaluation Score
0.71
0.69
0.68
0.67
0.66
0.65
0.8
1.0
1.2
1.4
1.6
1.8
2.0
2.2
Total Training Steps
Figure D.2: Nelder-Mead retains effectiveness with three component RMs. To evaluate whether NM-PPO able to scale effectively to more than two component RMs, we added BLEU score as third RM and compared the performance to PPO. Because of the additional RM, we allowed PPO five additional tuning runs (15 total) to adjust the coefficients on the component RMs. Both NM-PPO and PPO (with the tuned coefficients) were then run for 3 seeds, with the means and standard emrors plotted above. can see the NM-PPO is again able to outperform PPO while using 1/3 the number of total training steps.
23

---

Published as a conference paper at ICLR 2024
Intent Lagrange Multiplier
METEOR Lagrange Multiplier
0.5
Lagraige Mutsipler
0.5
0.0
94
0.3
0.00
0.2
-222
9.1
-4.0
60
-075
20000
40000
60000
80000
100000
120000
20000
40000
60000
80003
100000
120000
Intent Return
METEOR Return
M
0.6
0.24
Return
0.5
0.22
0.20
63
9.18
20000
40000
60000
80000
100000
1200000
20000
40000
60000
80003
100000
120000
Training
Training Steps
AIL-PO
U--PO
E-PPO
Figure D.3: Paradoxical objectives and intermediate thresholds can worsen oscillations. Running gradient sesscent-ascent to the saddle point. In practice, this can mean that the Lagrange multiplier(s) and value(s) can oscillate wildly over the course of training, even if their averages converge. The problem is worse for constraint thresholds which are intermediate- those that are neither high nor low relative to the range of individual reward function (Moskovitz et al., 2023a), but can be hidden by averaging. Above are example runs of All-PPO, p-PPO, and E-PPO, showing this problem is worse for All-PPO, likely due to its paradoxical objective (maximizing an objective which its constraints demand minimize). This instability likely leads to worse performance. We can also see that E-PPO's ' METEOR Lagrange multiplier oscillates as well, though with long periods of stability, which likely mitigates the effect on performance.
min-max game only guarantees that the average of the iterates converges
Intent Constraint Satisfaction
METEOR Constraint Satisfaction
0.26
0.50
0.24
METEOR M Score
Inoont Score
0.48
0.22
0.20
0.44
0.18
0.42
0.16
Orig Training Time
200000
300000
100000
200000
300000
Training Steps
Training Steps
Figure D.4: E-PPO constraint satisfaction for long training runs. EPPPO is largely able to satisfy constraints with respect to its component RMs over the course of many training steps. However, is possible for it to lose stability. As in other plots, solid curves and shading denote the means and standard errors across runs. In the right hand panel, the dotted curve indicates one training run which became unstable after roughly twice the length of the standard training time, with its METEOR score falling below the constraint threshold. This corresponded with a drop in evaluation performance, which reduced the overall average in 5.1.
24

---

Published as a conference paper at ICLR 2024
Nelder-Mead Proxy Point Trajectory
0.68
0.24
0.64
Predicted Evaluation Score
METEOR Threshold
0.60
0.56
0.20
0.52
0.44
0.16
Initial Simplex
0.14
Intermediate Simplex
0.36
Final Simplex
Intent Threshold
Figure D.5: Another example Nelder-Mead simplex trajectory. In this case, Nelder-Mead con- verged to the global maximum proxy point setting.
NM-PPO Example Runs
0.725
Passed over
0.700
Evaluation Score
0.625
0.600
0.575
Best Simplex
0.525
50000
100000 Training Steps
150000
200000
250000
Figure D.6: Several example NM-PPO evaluation curves. This plot shows the evaluation scores across training of three runs of NM-PPO. There several salient features to note. First, because of the nature Melder-Mead optimization, performance is not expected to be monotonic. Rather, the algorithm noisily hill-climbs The algorithm returns the best simplex point, which may not be the final one tested. Second, not every evaluation point in the curves above corresponds to a query the algorithm (i.e., upon reaching its target proxy point values), but is rather just measured for plotting purposes. This lets us see that occasionally, on the way to the next set of proxy points query, the optimization process may induce the policy to "pass over" region of high performance without being aware of it. This is what occurred the circled point the plot above.
adjusts the estimated proxy points comprising its current simplex.
25

---

Published as a conference paper at ICLR 2024
Evaliotion Surface: 20 quertesiran
Boluntoo Surface: 10 quertesirun
Evaliation Surface: 5 quertesruu
50:
620
LEGK
MET
314
Iriend Score
Figure D.7: Predicted evaluation surface as a function of evaluation queries per run. To visualize the effect of the number of queries to the evaluation function per run on the estimate of the evaluation function surface, we created contour plots of the predicted surface using the procedure described in Appendix A.3 for three different query frequencies: 20 (the default), 10, and 5. This was done using the same data used to generate Fig. 3.1. We can see that the basic structure of the predicted surface is preserved even down to 5 queries/run, and in particular that the global maximum is consistently located in the upper right region of the plots.
Evaluation
0.75int, 0.25met
1.0int 1.0int_0,25me met5
Evaluation Score
1.iint 1.0int0..5mec 0.5met
1.0int_ [inn__1.0mmm 1.0met
0.5int 0.5met
0,25int 0.75met
0.5int m.75
0.75met
0.75int_
0.25int 0.25met
0.56
0
20000
40000
60000
80000
100000
120000
Training Staps
Figure D.8: Sweep over possihle RM weightings for PPO.
PPO (t mM "l
Evaluntion
1.0.1.0...0)
0.85
(075.05.0755
(05,0.75005
(0.75.0.5.03)
(075.0.25.02)
Golation Score
(025.0075.0.55)
(05,05,0.5)
(.77...75.0075)
(025,075.003)
0.55
(075.
0025,0.25.0255
0.50
(05,025,055
0.45
(02500.5.025)
40000
100000
120000
(05,05.0755
Training Stops
(05, 05..0.5,7775
Figure D.9: Sweep over possible RM weightings for PPO with RMs (Intent, METEOR, and BLEU).
26

---

Published as a conference paper at ICLR 2024
D.2 SAMPLE OUTPUTS
For reference, we provide several random prompts and model outputs for the various methods. The key takeaway is that high METEOR and intent scores do not necessarily correlate with high evaluation scores and vice versa.
They didn <ROU> It s the truth <EOU> Are you okay ? <EOU> I really
Prompt:
accept
<EOU> Are you serious
wanted to get into NYU <BOU>
Reference: I really sorry to hear that <BOU>
PPO:
what were you trying to hide. <EOU> I'm m not hiding
anything. <EOU> METEOR:0.23, Intent: 1, Eval: 0.62
PPO-SAT: What do you mean? Did something happen to you?
<BOU> METEOR:0.19, Intent: Eval: 0.62
μ-PPO:
What do you intend on doing in college?
<BOU>
don't know if I'1l METEOR:0.22, Intent: 0, Eval: 0.66
All-PPO: What do you mean u don't t accept people.
<BOU>
ME-
TEOR:0.23, Intent: 1, Eval: 0.61
{-PPO: What did you do? <EOU> METEOR:0.23, Intent: 0, Eval: 0.65
NM-PPO: If wasn't a big enough place to a professor, could have stayed there. <EOU> METEOR:0.22, Intent: 1, Eval: 0.66
can't I'm waiting for Paul but he's s late <BOU> I'1l say the movie's starting in the minute <EOU> I know wonder what's keeping him <BOU> Looks like he won't show up
Prompt:
I
He's done this before hasn't t he ? <EOU> Yeah number of
times <BOU>
Reference: What's he trying pull anyway <BOU>
You're re right. I think he's been looking for Paul. <BOU> He said. METEOR:0.22, Intent: 1, Eval: 0.64
PPO:
PPO-SAT: So we're waiting.
you want to stay a bit longer
you can with Faul. <EOU> METEOR:0.20, Intent: 1, Eval: 0.62
--PPO: You' re looking forward to meeting him tomorrow.
<BOU>
I'11 find out. <EOU> METEOR:0.22, Intent: Eval: 0.66
All-PPO: So we' re waiting.. <EOU> And he just didn't show
<BOU> METEOR:0.24, Intent: Eval: 0.64
{-PPO: So I'm waiting. <EOU> He's s been doing this for long
time. METEOR:0.22, Intent: 1, Eval: 0.69
NM-PPO: Oh no. The second time, it was. can't t even get into that stuff. <BOU> METEOR:0.21, Intent: 1, Eval: 0.65
Prompt: What's s wrong with you ? <EOU> have sore throat and headache <BOU> Do you feel tired ? <EOU> Yes , my whole body
feels weak And really feel terrible <EOU> I'm sorry to hear that I'll have to examine you <BOU>
Reference: Ok go ahead <BOU>
PPO: What's s the matter? You're always so busy. <BOU> TEOR:0.20, Intent: 0, Eval: 0.64
ME-
PPO-SAT: What is your name?
<BOU> Moth <BOU> Moth <BOU> ME-
TEOR:0.17, Intent: 0, Eval: 0.60
.-PPO: What's wrong with your food? <BOU> No matter what..
I'1l have to check METEO Intent: 0, Eval: 0.67
All-PPO: What is the most important step of your quest? <BOU>
Making sure that your quest succeeds METEOR:0.20, Intent: 0, Eval:
0.63
27

---

Published as a conference paper at ICLR 2024
<-PPO: What is your condition? Do you have any fever? <BOU> No METEOR:0.20, Intent: 0, Eval: 0.70
NM-PPO: I don't feel good. I real ly dont feel good at all. <BOU> No way. METEOR:0.21, Intent: 1, Eval: 0.66
28


Reviews:
[{"review_id": "uYnKcSjzru", "reviewer": "Reviewer_uRgS", "summary": "The authors analyze the problem of reward model over parameterization in the context of composite reward functions. They propose a method of identifying the points of over parameterization or proxy points taking into account the correlations between reward models. Once these proxy points are determined, the authors propose several constrained RL approaches which incorporate these points into the optimization objective.", "questions": "1. **Proxy Points** \n    1. How and how many times is the ground truth evaluation function queried to determine the maximizing proxy point? (For results in Fig 3.2 and 5.1)\n    2. How is line 9 in algorithm 2 computed?\n    3. Why is KL regularization not used to determine the proxy points? (results in Fig 3.1)\n2. In Fig 5.1 (right) the performance of $\\xi-$PPO drops after about 180k training steps, are the constraints still satisfied during this stage? \n3. **Minor Points** \n    1. It would be clear if you could plot $\\xi-$PPO and $\\mu-$PPO in figure D2 for reference. It will help in understanding the instability of All-PPO.\n    2. Does All-PPO have an KL constraint?\n    3. It would be helpful if  you could plot the training curve for  NM-PPO? It would be helpful to see how the changing threshold affects the training process. \n    4. Sample outputs of NM-PPO are missing in section D.2\n    5. Could you explain the sample outputs in section D2? its hard to understand the performance of the different algorithms. It would be helpful if you could include the evaluation score and the rewards obtained as well.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The analysis of the over parameterization of composite reward functions in interesting and of importance to LLM-Alignment\n2. The proposed derivative free optimization method, NM-PPO is novel and computationally efficient.   \n3. The paper is generally well written and easy to follow.", "weaknesses": "**Determining the joint maximizing point seems heuristic**\n\nTo determine the joint maximizing proxy point, the evaluation scores as a function of the METEOR and intent rewards for each run shown in Fig. 3.1 are plotted and a surface is fit over them. But these evaluations were done by maximizing each reward individually, without account of any interaction. Thus using these to determine the joint maximizing point seems heuristic at best. \n\nIn other words, if the ultimate objective is to determine the weighting among RMs to find the combination which produces the best correlation with ground truth evaluation, and to fine-tune the language model until it reaches  the proxy points of this composite RM, the proxy points obtained using the method outlined in the paper may not align with this desired outcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mxxo7KAROf", "reviewer": "Reviewer_kPJr", "summary": "This paper provides a novel analysis of overoptimization in the context of RLHF with multiple proxy reward functions, identifying their termed \"proxy points\" of overoptimization. The proxy point is a threshold at which further increasing the proxy reward results in decreased ground-truth performance. Their primary contribution is to introduce constrained RL approaches that incorporate these points into optimization objectives. Additionally, they demonstrate how a derivative-free optimization method can be used to dynamically identify the proxy points in a single run.", "questions": "- On the Validity of Evaluation Metric (Equation A.1):\n  - How did the authors determine that the evaluation metric chosen in Equation A.1 accurately represents the efficacy of the proposed method?\n  - Were there other potential metrics considered? If so, could the authors elucidate the rationale behind the chosen metric over others?\n  - Are there any design principles or guidelines followed when setting this metric? The significance of this setting in the proposed method seems critical, and a clearer explanation would be beneficial.\n\n- Regarding mixed advantages right after Equation (4.3): Shouldn't the term v_µ(s) encompass the second term from Equation (4.3)?\n\n- I understand that PPO-SAT adheres to the Lagrangian multipliers method for equality constraints and thus the weights of RMs will be adjusted based on how well the constraints are met. However, in Section 5, the authors mentioned that the RM weightings in PPO-SAT are fixed. Can you please provide clarity on the workings of PPO-SAT?\n\n- In Figure D.2, for All-PPO, why is the tanh function used for constraints that seem to be inequality constraints? This appears to be inconsistent with the explanation provided in Section 4 of the main text.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "**Originality**\n\nThe paper presents a unique framework that introduces the concept of \"proxy points\" to address overoptimization in an environment with multiple proxy reward models and a given ground-truth reward model. This novel idea of defining a threshold for proxy rewards, ensuring that they don't exceed the proxy point, is a commendable original contribution to the field of RLHF. \n\n**Quality**\n\nThe authors have validated the effectiveness of the proposed approach through empirical experiments. The application of a local hill-climbing algorithm, the Nelder-Mead threshold search, is also shown and validated through numerical experiments.\n\n\n**Significance**\n\nIn the broader perspective of RLHF, managing overoptimization is a critical challenge. By estimating the thresholds that may cause a drop in performance and incorporating them in the optimization process, this paper makes a significant stride in improving the robustness and reliability of reinforcement learning models.", "weaknesses": "- One significant limitation, as acknowledged by the authors themselves, is the assumption of the availability of the ground-truth reward model in RLHF. While such an assumption of the gold reward model aids in understanding and analyzing overoptimization as in Gao+ 2022, its actual use within RLHF algorithms seems impractical.\n\n- The evaluation metric described in Section A.2 is aimed at respecting both the METEOR and intent reward functions in light of Goodhart's Law. While the authors have meticulously chosen metrics from two categories - lexical quality and text diversity, there seems to be a lack of in-depth discussion about its relevance and applicability in the context of LLM-based RLHF. Drawing insights from works such as the one by Maynez et al. (Benchmarking Large Language Model Capabilities for Conditional Generation, ACL 2023) could have provided a clearer perspective on how well the proposed metric aligns with the requirements and challenges of LLM.\n\n- The lack of detailed descriptions and elaborations of the specific methodologies presented in Table 1 raises concerns about the paper's clarity. One might find it challenging to grasp the proposed approach without sufficient explanations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EQ0hw7bfpg", "reviewer": "Reviewer_xdFb", "summary": "In this work, the authors improve the ability to optimize LLMs so that they are aligned with human preferences, but not overly optimized to a specific reward model (inferred via reinforcement learning from human feedback or some oracle model) or set of reward models. In this work, the authors identify the points of overoptimization, termed proxy points, and use constrained optimization to ensure that each reward model reaches, but does not surpass the proxy point. The authors explore two different approaches to determining proxy points and display that the proposed technique can avoid overoptimization and improve performance. The authors present detail regarding related work, their approach for determining proxy points, the proposed method, and a thorough evaluation section.", "questions": "- Could you comment on how the results may differ for newer, more high-performing LLMs (e.g., GPT-4)?\n\n- Could you also address the questions in the weaknesses above?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "+ This paper is extremely well-written and contains sufficient related work and explanation to understand the proposed techniques and prior work.\n+  To the best of my knowledge, the proposed technique to avoid LLM overoptimization toward composite RMs is novel.\n+ The analysis is very detailed and showcases the proposed technique works as expected\n+ As LLMs are a very popular topic now and actively being deployed, this approach tackles an important issue in optimizing and improving LLM performance toward high-performance behavior.", "weaknesses": "- Could you comment on how easy it is to identify the proxy point? In larger or multi-topic datasets, would determining a proxy point be more difficult?\n- While the paper does contain much information, a lot of important information is in the appendix that would benefit from also appearing briefly in the paper. On that thread, it would be beneficial to highlight important details in D.2. Is there a specific feature of the sample outputs you are attempting to highlight?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqalAgKcV0", "reviewer": "Reviewer_FHce", "summary": "This paper shows that over-optimizing reward models can lead to decreased performance of LLMs. The authors remedy this through constrained reinforcement learning, where the LLM is prevented from exceeding each reward model's usefulness threshold.", "questions": "- I'm curious if it's necessary in practice to satisfy all constraints. For instance, I wonder how performance would be affected if only the intent constraint was satisfied, or if only the METEOR constraint was satisfied (while still identifying proxy points through considering all RMs together)\n- I'm a bit confused on why $\\mu$-PPO still performs well even though the proxy point thresholds are exceeded. I do see that the authors acknowledge this as a reason why $\\mu$-PPO may perform worse than $\\xi$-PPO, but I wonder why $\\mu$-PPO still performs well. I'm especially confused about this because in Figure 5.3 the authors show that using thresholds that are higher than the proxy point thresholds leads to much worse performance for $\\xi$-PPO. Can the authors provide an explanation for this?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors address the reward model over-optimization problem with a simple constrained optimization solution.\n- The authors identify that the interaction between component RMs can influence the proxy points of the component rewards.\n- The authors experiment with a number of different constrained optimization solutions (e.g. $\\xi$-PPO, $\\mu$-PPO, etc.)\n- The authors present NM-PPO to mitigate the need for multiple runs to identify proxy points.", "weaknesses": "- The method requires multiple runs to identify proxy points for each reward model. This is somewhat addressed by NM-PPO, but NM-PPO still requires some queries to the evaluation metric during training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)