L3Ms — Lagrange Large Language Models
Authors: Guneet S. Dhillon
Xingjian Shi
Yee Whye Teh
Alex Smola
Venue: ICLR 2025 Poster (2025)
Keywords: LLM
alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: Supervised fine-tuning (SFT) and alignment of large language models (LLMs) are key steps in providing a good user experience. However, the concept of an appropriate alignment is inherently application-dependent, and current methods often rely on heuristic choices to drive optimization. In this work, we formulate SFT and alignment as a constrained optimization problem: the LLM is fine-tuned on a task while being required to meet application-specific requirements, without resorting to heuristics. To solve this, we propose Lagrange Large Language Models (L3Ms), which employ logarithmic barriers to enforce the constraints. This approach allows for the customization of L3Ms across diverse applications while avoiding heuristic-driven processes. We experimentally demonstrate the versatility and efficacy of L3Ms in achieving tailored alignments for various applications.

Published as a conference paper at ICLR 2025
L3Ms LAGRANGE LARGE LANGUAGE MODELS
Guneet S. Dhillon 1x, Xingjian Shi?, Yee Whye Teh 1, Alex Smola 2
University of Oxford, Boson AI
{guneet. chi lon, y www. ox ac. uk, (xingjian,
aii
ABSTRACT
Supervised fine-tuning (SFT) and alignment of large language models (LLMs) are key steps in providing a good user experience. However, the concept of an appropriate alignment is inherently often rely on heuristic choices to drive optimization. In this work, we formulate SFT and alignment as a constrained optimization problem: the LLM is fine-tuned task while being required to meet application-speccficc requirements, without resorting to heuristics. To solve this, we propose Lagrange Large Language Mod- els (L3Ms), which employ logarithmic barriers to enforce the constraints. This approach allows for the customization of L3Ms across diverse applications while avoiding heuristic-driven processes. We experimentally demonstrate the versatil- ity and efficacy of L3Ms in achieving tailored alignments for various applications.
peplication-deeendee
and current methods
INTRODUCTION
Large language models (LLMs) are used for a wide range of tasks: as chatbots (Brown et al. 2020
OpenAl 2024), for code generation (Ahmad et al.) [021]] Wang et al.] [202]] Rozière et al. 2024). for medical assistance (Yang et al. 2022; Moor et al. 2023), and so on. The key ingredients for their impressive downstream performance are supervised fine-tuning (SFT) and alignment; the for- mer fine-tunes the LLM to a task of interest, while the latter instills it with preferential properties. Arguably, the right combination of preferential properties is highly application-dependent. For ex- ample, scholar would want a chatbot to be honest and factual for assistance with their work. In con- trast, a fiction writer might prefer the opposite behavior to help create fantastical imaginary worlds. There is also plenty of (anecdotal) evidence in support: some LLMs refuse provide information on how to "kill" a process in Unix, recommending the use of less violent strategies for dealing with wayward computer programs instead Therefore, need frameworks for L.MM customization.
Consequently, Bai et al. (2022); Rame et al. (203)); Wu et al.] (2023); Ji et al. (2023); Zhou et al.) (2024) fine-tune LLMs on varying combinations of such preferential properties. In practice, one tends to resort to trial and error to find the right combination of preferences for their particular application. doing so, one verifies if certain minimum baselines are satisfied, such as ensuring the factual correctness of statements or confirming that response lengths are capped at 100 words. Since there isn't 't way enforce such requirements directly, current methods resort to heuristics. Additionally, existing pipelines carry out SFT and alignment sequentially and must ensure that the I.LM does not forget relevant task information learned during the SFT stage. This is achieved by penalizing the LLM for drastic deviations, with the strength of the penalty determined heuristically.
In this work, we formulate SFT and alignment in LLMs as a constrained optimization problem. In particular, we fine-tune LLM to minimize the task objective (the objective function) while simultaneously satisfying application-speciinne the SFT and alignment stages and mitigates the reliance on heuristics altogether. Furthermore, propose Lagrange Large Language Models, a.k.a. 1.3Ms, to solve such constrained optimization problems. Specifically, we do so by employing logarithmic barriers and gradually enforeing the constraints during the fine-tuning procedure. Lastly, we empirically demonstrate how one can pick and choose constraints and tailor L3Ms to a range of applications without resorting to heuristics.
minimum requirements (the constraints). This merges
Work done while at Boson AI redd http://///.oeeeeeecccc/////////r uses _o__to__00 u
cladee 21 21_ref

---

Published as a conference paper at ICLR 2025
Prompt
What are the four main types of macromolecules found in living organisms?
1.
Nucleic acids (DNA and RNA)
Response
2. Proteins
(short response L3M)
Carbohydrates
4. Lipids
Nucleic acids, including DNA and RNA, which carry genetic information
2. Protcins, which are responsible for the structure, function, and regulation of living organisms
Response
(long response L3M)
3. Lipids, which provide energy and form the membranes that surround cells Carbohydrates, which are used as an energy source as structural components of cells.
Prompt
Who discovered the law of universal gravitation?
Response
Sir Isaac Newton is credited with discovering the law of universal gravitation in the late 17th century.
(short response L3M)
Sir Isaac Newton is widely credited with the discovery of the law of universal gravitation, which states that every particle in the universe attracts every other particle with a force proportional to their masses and inversely proportional the square of the distance between them.
Response
(long response L3M)
Table 1: Example responses from length constrained L3Ms. We provide example responses from L3Ms with varying length constraints. We include the prompt along with the generated responses from two L3Ms; one constrained to have short responses and the other constrained to long ones.
For example, Table provides the generated responses from two such L3Ms; both are fine-tuned for instruction-folloiingg but one is constrained to be concise, while the other is to be verbose.
In summary, contributions are follows:
We formulate SFT and alignment LLMs as constrained optimization problem: an LLM is fine-tuned on task while simultaneously satisfying custom requirements (cf. Section
2. We propose L3Ms, a family LLMs fine-tuned using the above framework (ef. Section
3. We experimentally demonstrate how L3Ms can be customized different applications and their specific requirements while avoiding heuristic-dmt driven processes (cf. Section 6
2 OPTIMIZATION FOR LLMs
Training an L.MM proceeds in multiple stages (Ouyang al. 2022), which we discuss below.
2.1 TRAININGG
The pre-training stage instills the LLM with generic knowledge of language. It entails regenerating text/token sequences by minimizing their perplexity, i.e., the negative log-likelihood of the sequence normalized by its length. More formally, the perplexity sequence z is defined as:
la (x) = log To (x)
le log To (I.(
where xy and Jzz denote the i-th token in the sequence and its prefix, respectively. Additionally, the function TT(-) denotes the predicted probability distribution of the L.MM over token sequences, where the LLM is parameterized with weights 0. Then, the pre-training objective is given as:
min Ex-g(+) [lo (x)]
and the expectation is replaced by empirical average over large corpus with trillions of tokens.
2

---

Published as a conference paper at ICLR 2025
2.2 SUPERVISED FINE-TUNING (SFT)
Next, one fine-tunes the LLM to a task of interest, such as instruction-followe summarization, or translation. The data (prompt, response) pairs (2r, y), and the LLM is fine-tuned regenerate the responses. Thus, one minimizes the perplexity (or a related loss) on the response given the prompt:
min (x.))))p(() [oo (y/==)]
for distribution p()) over (prompt. response) pairs that reflects the task-related data.
2.3 ALIGNMENT
This stage aligns the LLM to generate responses with preferential properties. A common setup is to learn preference reward functions that represent properties such as helpfulness and harmlessness (Bai et al.) 2022). followed by reinforcement learning to adapt the LLM to maximize the said re- wards. This is called reinforcement learning from human feedback (RLHF; Knox & Stone 2008 Grifhth et al. 2013 Christiano et al., 2017). Note that the preference reward functions need not always be learned; they could also be engineered rule-based, such the length of the response.
Given a single preference reward function r(.), the alignment objective is given as:
max (zx))=pp)) (y/x=)] y-
This maximizes the rewards for the LLM's responses prompts sampled from the task distribution. To prevent over-optimization of the reward and avoid drastic deviation away from the SFT model, is common practice to add a regularization penalty, such as the KL divergence (Gao et al. 2023)
We are interested in & 2 1 different preferential properties. In such a scenario, one could learn individual preference reward functions (-)'s and optimize the LLM to maximize their combination. In particular, Li et al. (202T): |Rame et al. 2023); Wu et al. (203):
) Ji et al. (2023) use a linear combination of the rewards, substituting the single reward in Eq. 2 with Σ i=1 (yr:(ula)), for some choice of a; 2 0. Alternatively, one could learn a single reward function by choosing the data proportions of the different properties used to train it. For example, Bai et al. (2022) use 3:1 proportion of helpfulness to harmlessness data to train a single preference reward function. Note that the proportions can also be represented as weights a; 2 with Σ a; = 1. Therefore, the choice of o s. combined with the strength of the regularization penalty, together the alignment. They are chosen based on the good judgment of the practitioner in a somewhat heuristic manner.
2.4 SHORTCOMINGS
Although the above pipeline is commonly used to train LLMs for deployment, it has shortcomings.
Firstly, how does one choose the weights o 's? as's? preference properties? Choosing between properties such truthfulness, verbosity, and humor depends on the application at hand. However, even when the application is known, the weights are chosen through trial and error: trying different combinations and inspecting the achieved rewards. For instance, if one wants the responses to be under 100 words for a summarization task, one might repeatedly examine the length of the responses and adjust the weights a; 's to achieve that. This inherently involves verification against set of minimum baselines being satisfied. Can we avoid heuristics and enforce such minimum requirements for the preference rewards in a principled way?
Or equivalently, what is the right combination of
Secondly, recall that the original task objective is given in Eq. [ However, optimizing special purpose objective such as Eq. (2) can lead to decrease in performance on the original task. Whilst penalizing the deviation away from the SFT model mitigates this to some extent, the strength of the penalty is again determined heuristically. Can we ensure the performance on the original task?
RELATED WORK
3.1 ONE SIZE DOES NOT FIT ALL
Early work on LLM alignment assumed homogeneity of preferences (Bakker et al., 2022). However, the reality is quite different: human preferences vary widely and highly application-specifcc
3

---

Published as a conference paper at ICLR 2025
Rame et al. 2023 Casper et al. 2023). Consequently, Bai et al. (002 Rame et al. (002)); Wu et al. )023); Ji et al. (002); Zhou et al. (2024) linearly combine several fine- ime-grained preference
rewards, where each combination reflects unique customization. However, the right combination of preferential properties for particular application is determined heuristically by trial and error. In contrast, formulate alignment constrained optimization problem. In doing so, custom requirements are naturally imposed as constraints. This alleviates the need for heuristic choices.
3.2 CONSTRAINED ALIGNMENT FOR LLMS
Independent of our work, Moskovitz et al. )2024;; Dai et al. (024) also introduced constrained optimization for LLM alignment but for varying reasons. Motivated avoid over-optimization of preference rewards. Moskovitz et al. (2024) find "proxy points", values of the reward functions be- yond which the performance of the LLM is negatively impacted. They constrain the average rewards to be in the vicinity of these proxy points. Dai et al. (2024) trade-off between helpfulness and harm- lessness from a safety perspective. They maximize the LLM's helpfulness while constraining its average harmlessness reward. Our motivation, on the other hand, is to tailor to custom preferences through sets of different constraints while simultaneously learning to perform the task of interest.
Our work is different in two important ways. Firstly, both Moskovitz et al. 2024 Dai et al. (2024) consider the alignment process in isolation, because which their objective is either to maximize one of the preference rewards minimize the deviation away from the SFT model. Instead, merge the SFT and alignment stages and minimize the task objective directly. This ensures that the LLM learns task-solving capabilities without any deviation penalty. Furthermore, by avoiding the computation of the deviations from the SFT model, we save both memory and time during the fine- tuning process: we no longer need to load the SFT model and evaluate it on generated responses.
Secondly, both both Moskovitz et al. (2024); Dai et al. (2024) obtain a solution to the constrained problem by computing a saddle point of the Lagrangian. This is achieved by formulating minimax game where the LLM minimizes the Lagrangian, and the Lagrange multipliers adapt to maximize it. How- ever, the non-convexity of the objective makes this nontrivial. Instead, we choose the logarithmic barrier approach as the associated Lagrange multipliers satisfy the KKT complementary slackness condition by design (cf. Section 5.2) rather than letting the learning procedure do so (which in our experience is extremely sensitive to the choice of the learning rate). We empirically observe that while both methods satisfy the constraints, our approach using log barriers minimizes the task objective better (cf. Section 6.3). As side effect, we avoid introducing new learning parameters.
Furthermore, we provide experimental validation of our approach using more preferences than Moskovitz et al. 20244) Dai et al. (2024) and 4.5x larger LLMss than Moskovitz al. (002)).
4 CONSTRAINED OPTIMIZATION FOR LLMS
Our goal is to reduce the task objective for the LLM to solve the task of interest while also en- abling custom alignment by having the LLM meet the application-specffic minimum requirements on different preference rewards. To do so, we propose the constrained optimization problem:
min E(nn)r(() [o (y]x))
(3)
subject to ((r,)) [rs (y|=)] 2 b, for all i e (1,2....,k) (-
(4)
Here, the objective is the same that of SFT in Eq. and the constraints are enforced to satisfy the custom requirements; this merges the SFT and alignment stages. The b,'s signify the minimum baselines for each preference reward function r.(-), and the constraints enforced on average.
Compared to the previous approach, we no longer rely on heuristics to find a set of weights a; 's to satisfy the minimum requirements; we can do so directly through the constraints. Furthermore, with the same objective as SFT, we can directly maintain task performance without any deviation penalty. Additionally, note that whenever constraint is satisfied, its influence vanishes. For example, if the LLM is naturally harmless and Thamnles (y)xx) > bramlesss then the constraint is not active, and the L.M will not be penalized. In contrast, the previous approach would further penalize the LLM.
4

---

Published as a conference paper at ICLR 2025
Notation
For ease of notation, rewrite the constrained problem in Eqs. 3 and (4) as:
min L (0) subject C, (8) S 0 for all i
(5)
0
with the objective I(0)= ((( Do (y) a:] and constraints C:(0) F ((.))-p(-)) [be (r:l/))))))-
4.1 TYPES OF CONSTRAINTS
While we write the constraints in Eq. 4 as expectation/average constraints, other forms exist. For instance, uniform constraints impose minimum reward every generated (prompt, response) pair:
Ts (y|=)) by for all (ær, ) p()) y ~ To (-1z) and E (1.2..... .k.
(6)
Additionally, chance constraints bound the probability of the inequality holding away from zero:
P (a.)>p(() [r, (3/z)) 2 b,] 2 1 1 for all i E {1,2...., .k..
These constraints are not equivalent, but they are related. We can rewrite Eq. in the form of average constraints using E as the threshold and taking the expectation of the indicator 1 (ra(ylar) > b:}- Moreover, Eq. (6) implies Eq. ( but the converse is not true. Unfortunately, Eq. (6) difficult to achieve in practice, especially when the data distribution is unknown.
We continue using expectation/average constraints, but similar discussions can extend to other types.
4.2 LAGRANGE MUL.TIPLIERS
We can introduce Lagrange multipliers X 2 for the constraints and obtain the Lagrangian:
(0) (0) Σ (0).
(8)
There is a rich literature connecting the Lagrangian with constrained optimization. Notably, the KKT conditions (Karush 1939 Kuhn & Tucker 195T) provide sufficiency conditions for global optimality under convexity, where the solution is obtained by finding the saddle point the La- grangian. However, these conditions are not enough for highly non-convex scenarios such ours.
Nevertheless, the Lagrangian is instructive in understanding the relative importance of the con- straints. For active constraint, i.e., one satisfied with equality, the corresponding Lagrange mul- tiplier can be non-zero; the larger its value, the more important the constraint. Conversely, for an inactive constraint, i.c., one satisfied with strict inequality, the corresponding Lagrange multiplier must vanish to O. This is known as complementary slackness and is one of the KKT conditions.
4.3 LOGARITHMIC BARRIER
practical way to enforce constraints is with barrier functions. Consider the (relaxed) log barrier:
B Bu,s (2) - log (-2), [ + ya log 8, z
2
and hence O,B (2) max (-2,
with parameters n1, 0. This is convex, continuous, and differentiable function, which is valid for x E IR. Importantly, for s = n2² this barrier function converges to the characteristic function xfz < 0} as μ 0, i.e., it takes the value 0 when < 0 and otherwise (Tal et al. 1992 Nash et al. 1994, Hauser & Saccon 2006 Feller & Ebenbauer 2017); the condition = μ² is sufficintt but not necessary (Kevvddee al. 2022). This convergence to the characteristic function is visually depicted in Fig. [ showing the change in the log barrier function as we gradually decrease j..
We can now the log barrier enforce the constraints in Eq. 5 and simply add them to the objective. We obtain an unconstrained objective, with controlling the strength of the constraints:
k
1
G, (8) (0) +
Ban (C, (8))
(10)
k
5

---

Published as a conference paper at ICLR 2025
0.01
Fau
0
Figure 1: The relaxed logarithmic barrier. We depict the convergence of the relaxed logarithmic barrier B3 B,,,a(4 (2) to the characteristic function xf2 S 0} μ 0. We gradually decrease μ from
(blue) to 0.01 (red). Consequently, B, (z) gets closer to 0 for < and increases to co otherwise.
LAGRANGE LARGE LANGUAGE MODELS (L3Ms)
Thus far, we have formulated the SFT and alignment stages as a constrained optimization problem in Eq. (5). We proceed to find solutions for the same by solving the unconstrained objective in Eq. (10).
We call the family of models obtained this way L3Ms, i.c. Lagrange Large Language Models.
5.1 OPTIMIZATION PROCEDURE
Since the log barrier converges the characteristic function y - want to find the minimizer of gr,(0) for very small j.. However, doing so directly leads to instabilities as the objective function ill-conditioned. Instead, it is common practice to follow minimizer for fixed jr, reduces μ,, and repeats (Curtis et al. 2024). Specifically, the procedure instantiated with initial values 6g Ho, and
iterative procedure: one finds the
7 < 1. the t-th iteration, 14 + 7/4-1 is reduced and 0g +- arga min G.. (f) (with initialization 0.-1). In doing so, the constraints are gradually enforced, nudging the LLM to satisfy them over optimization procedure while avoiding instabilities. As {1r} Y 0, the weights {0.} converge to the minimizer of the constrained problem.
is impossible to minimize Gr, (0) exactly many practical applications. Instead, at each iteration, one take single optimization step toward the solution. Doing is amenable to stochastic gradient methods and mitigates computational overhead: the optimization proceeds as normal while the value of is reduced over the course of the procedure. One can guarantee the convergence of this procedure to the optimal solution in some settings; for example, Curtis et al. (2024) prove convergence when dealing with box constraints. However, convergence in scenario like ours not guaranteed. Nevertheless, we will experimentally demonstrate its use for our constrained problems.
We employ stochastic gradient methods and derive the gradient of our objective function directly:
&
OgC, (0)
ae9, (0) = dgL (0) +
(11)
max (-C: (0),
This follows immediately from Eqs. 9 and 10). Note that the gradients (g):s are also known as policy gradients in reinforcement learning literature. We discuss our strategy for estimating these gradients Appendix B and refer readers Schulman al. (2016) for a more detailed review.
5.2 CONNECTION TO LAGRANGE MULTIPLIERS
The log barrier and the Lagrangian are intrinsically connected; this becomes evident when compar- ing Eq. (II) with the (gradient the) Lagrangian in Eq. 8 In particular, we define the multipliers:
Å max C. (0),

---

Published as a conference paper at ICLR 2025
corresponding to the gradients OgCi(0)'s in Eq. (11) They can be interpreted as Lagrange multi-
pliers: for active constraints, Å 1/kp is non-zero; for inactive constraints, X -p/KC vanishes to 0 as + O. Hence, the KKT complementary slackness condition is satisfied by design.
5.3 MEMORY AND TIME COMPLEXITY
L3Ms differ from traditional LLMs only in the fine-tuning process. In fact, L3Ms require less memory and are faster to fine-tune compared to traditional LLMs. This comes from merging the SFT and alignment stages as we did in our constrained optimization formulation in Eq. (). We minimize the task objective directly and avoid the need to compute deviations away from the SFT model (for instance, impose the KL divergence penalty). As result, save loading the SFT model into memory and evaluating it on generated responses at each optimization step. Also, the L3M's log barrier parameter j adjusted during fine-tuning itself, without needing any extra steps.
5.4 IMPLEMENTATION DETAILS
Alternating the objective and gradient clipping Gradient clipping is simple yet effective way to ensure stable training of large models (Goodiellow et al. 2016 Zhang et al. 2020). We employ this technique, albeit with the modification of clipping the gradients of both the task objective and the constraints separately, as they have varying magnitudes. We achieve this by alternating between reducing the task objective and enforcing the constraints by flipping a fair coin to select one or the other. While this doubles the number of steps to achieve the same effect, it does not increase the amount of work done as now only one part of the objective or the other is evaluated at each step.
Length normalization The gradient of our objective function in Eq. 11] involves the LLM's log- likelihoods on the generated responses through the gradients (0)ss 's (cf. Eq. (12). To avoid response length bias, we length-normalize the log-likelihoods, akin to the definition of perplexity.
Estimating the mean preference rewards We need estimate the expectations involved in the gradient of our objective function in Eq. (TT.. The expectations in the numerators can be estimated with the per-mini-bateh Monte Carlo averages (Mohamed et al. (020).. However, C,)))'s in the denominator need careful consideration. Note that: (i) C,) does not involve the gradient, so its estimate can include information from previous mini-batches to reduce the estimate's variance, and (ii) since the weight 0 is updated during fine-tuning, C,(0) is non-stationary. Hence, we use an expo- nential moving average estimate for the mean (offset) preference rewards (0) in the denominator.
EXPERIMENTAL RESULTS
To illustrate the customization of L3Ms, we empirically evaluate them (i) satisfaction of the im- posed constraints, and (ii) minimization of the task objective. Our code, based on the Transformers library (Wolf et al. 2020 is available at: httpt:t////// tttpt/ggtttt.t/../ coom//Gneett-Dia olnooon/33 3m
6.1 SETUP
We use LLaMA-7B (Touvron et al. 2023) for all our experiments, as a lightweight LLM pre- trained on large corpus. We are interested in the task of instruction-followe for which we use UltraChat (Ding et al. 2023), a large-scale dataset of instructional conversations. We run all experi- ments NVIDIA H100s. Further details of our experimental setup are included in Appendix A
We refer to the LLM fine-tuned to minimize the SFT objective only (without the alignment stage) as the SFT model. We fine-tune LLMs using the minimax approach to find a saddle point of the Lagrangian, as proposed by Moskovitz et al. (2024); Dai et al. (2024). and refer to them as MMs. Lastly, we refer to the LLMs fine-tuned using our proposed approach L3Ms.
In what follows, we use different preference reward functions and vary the custom constraint require- ments. All results are obtained held-out test dataset (not seen during training validation).
7

---

Published as a conference paper at ICLR 2025
LLM type
Length
Perplexity
400
-00
SFT
121.6
0.8
2000
L3M 50
100
81.3
0.804
890
LL type
SFT
L3M [100
150
120.7
0.804
300
1000, 1030
1.M for (1000 150g
L3M 50
75
64.4
0.807
250
1.5M for (150, 07%]
L3M 75
100
88.2
0.808 1:3
Resporee
200
1100, 120
1075, 103
L3M [100
125
111.7
0.810
for [128. 153
L3M [125
150
126.5
0.3
100
LBM for 1076, 087.5
p007
L3M 75
87.5]
82.9
0.811 1:03
(12.5)
LBM for 1112.6 120)
L3M 87.5,
100
92.7
0.809 1:03
L3M
[100 112.5]
104.8
0.810
L3M [112.5, 125
117.3
0.810 +033
Figure 2: Length constrained L3Ms. We report the response lengths (in tokens) and task perplex- ities of the SFT model and the L3Ms with varying length constraints. Left: The mean response length with the mean and standard deviation of the task perplexities. Right: The distribution of the response lengths. The notches indicate the medians and their 95% confidence intervals, the boxes show the 25% quantiles, and the whiskers denote the 1.5x interquartile ranges. The white circles mark the means, and the black dashed lines depict the constraints imposed on the different L3Ms.
6.2 LENGTH CONSTRAINED 1.3MS
Consider tasks in which the lengths of the responses need to be contained in the range [low hight to control verbosity; for example, in summarization tasks (Makino et al. 2019). In this case, the natural choice for the reward functions compute the response length and its negation: ri (y(xx) = lyl and (2(1(x) w/ Furthermore, these rewards are to be controlled with the minimum requirements of hlw and lnihh. respectively. Note that these reward functions are perfectly negatively correlated.
If we naively average the rewards, any unconstrained formulation of alignment (including RLHF) will be ineffective, as the loss will always vanish due to the anti-correlation. We could use a weighted average and tune the weights heuristically, but this is tedious. Instead, we use the constrained for- mulation and directly constrain the rewards 71 (y/z) = |y| 2 how and r2(y(() = |yl 2 -high-
We fine-tune several L3Ms with varying length constraints. We illustrate the distributions of the gen- erated response lengths (in tokens) and report the perplexities achieved on the task-related data in Fig. 2 We observe that the mean response lengths are in the required range in each case, satisfying the imposed average constraints. Additionally, the task perplexities increase slightly as the con- straints on the response lengths are made more stringent. However, there is little to degradation relative to the SFT model, with all mean task perplexities being within 0.02 standard deviations.
The examples included in Table | are generated from such L3Ms. While all the responses correctly answer the prompts, their lengths vary, corresponding to the length constraints imposed on them.
6.3 HELPFUL AND HARMLESS L3Ms
Next, consider the Helpful and Harmless (HH) preferences that have been extensively used in the LLM alignment literature Ti et al. 2023 Wang et al. 2024 Zhou et al. 2024 Guo et al. 2024). Specifically, we utilize the datasets by Bai al. (2022) to train two preference reward functions, respectively. These learned reward functions are negatively correlated (Bai et al.) 2022 Dai et al.) 2024). Furthermore, note that the numerical outputs of both these reward functions are interpreted as ratings such that a higher numerical value indicates higher helpfuunnesshhrrmlesnnss and vice versa.
We fine-tune several L3Ms with varying HH constraints. We compare our L3M approach of using log barriers with that of the minimax optimization by Moskovitz et al. (2024); Dai et al. (2024) to find a saddle point of the Lagrangian (MMs). In our experience, learning the L Lagrange multipliers with the latter is extremely sensitive to the choice of the learning rate. Moreover, an additional SFT model during fine-tume toeiiunnng. as is done by Moskovitz et al. (2024); 4) Dai et al. (2024), our implementation of MM minimizes the task objective directly (as is done by L3MMs as well).
avoid loading
8

---

Published as a conference paper at ICLR 2025
Constraints
Perplexity
Helpfulness
Init SFT
Helpful
Harmless
MM
L3M
0.804_0.
MM
4
0,809
2
L3M
4
0.822
0.818
3
0.822
0.814
2
0.825 +0.3
0.820 0.3
0.817=03
0
0.816_0.3
2
Harmlessness
Figure 3: Helpful and harmless L3Ms. We report the helpful-harmless rewards and task perplex- ities achieved by the different LLMs. Left: The helpful-harmless rewards attained by the LLM at initialization (at the bottom-left in blue), the SFT model (at the top-left in orange), the MMs (in green), and the L3Ms (in red). We depict the imposed constraints black, with the dotted gray lines connecting LLMs to their corresponding constraints. Note that constraints are satisfied if the ob- tained reward point is at the top-right of its corresponding constraint point. For example, the shaded region denotes the feasible region for the constraint point (3, 3), with the shade gradient denoting the distance from the constraint boundary (light to dark shows an increase in distance). Right: The mean and standard deviation of the task perplexities for MMs and L3Ms, along with their corresponding constraints; the task perplexity at initialization is 1.316 and that of the SFT model is 0.805
Fig. 3ishows the achieved task perplexities and helpful-harmler rewards for the different LLMs. At initialization, the helpful-harmless rewards are both low, with high task perplexity of 1.316 This is improved upon by the SFT model, reducing the perplexity to 0.805 and attaining a high helpfulness reward (due the task data instilling mnstruction-folloiing capabilities). Furthermore, MMs sacrifice task performance to oversatisfy the constraints, with mean task perplexities 2 0.822. Conversely, L3Ms satisfy the imposed helpful-harmless reward constraints with consistently lower task perplexities (the mean perplexities are in the range 0.804-0.820). We attribute this to the L3Ms having better Lagrange multipliers by design rather than learning them as in MMs (cf. Section (.2
Note that here we evaluate the LLMs reward constraint satisfaction and task objective minimiza- tion. Furthermore, one can increase the constraint minimum baselines obtain higher rewards.
CONCLUSIONS
In this work, we formulate SFT and alignment in LLMs as constrained optimization: we minimize the task objective while simultaneously imposing application-specifi constraints on preferences. This enables the customization of LLMS to different preferential properties while maintaining per- formance the task of interest. Consequently, we propose Lagrange Large Language Models (L3Ms) to solve this constrained optimization problem by incorporating the constraints in the ob- jective using the logarithmic barrier. We include experimental results to illustrate the customization qualities L.Ms,, which can fit to different preferences, providing a personalized user experience.
REFERENCES
Wasi Ahmad, Saikat Chakraborty, Baishakhi Ray, and Kai- Wei Chang. Unified pre- pre-training for pro- gram understanding and generation. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer,

---

Published as a conference paper at ICLR 2025
Dilek Hakkani-Tuu Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Asso-
ciation for Computational Linguistics: Human Language Technologies, pp. 2655-2666 Online, June 2021. Association for Computational I. Linguistics. doi: 10. (0.18653/v1/ 3/2/3//11122222/ wttp://.2221111lmnn// maacl-main.211. URL https:://cclc www.ooanntttlolooooo wowhoolooy..ooo wrg.oogg.///o... naac -a 211
T
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli. Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodds, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Ka- plan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022. URL (RLLtt::///rxii org/ abs/ 2204 05862
Michiel Bakker, Martin Chadwick, Hannah Sheahan, Michael Tessler, Lucy Camphell-Gillinghee Jan Balaguer, Nat McAleese, Amelia Glaese, John Aslanides, Matt Botvinick, and Christopher Summerfield. Fine-tuning language models find agreement among humans with diverse prefer- ences. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems, volume 35, pp. 38176-88189 Curran Associates, Inc., 2022. URL
neur.ps...ee.pp...e... paper_ forr_ffllii httpaaappeeeeeeeepeeeeeeeeeeeeee http:////ee./epeeeeeeeeee//////p
1e/
pdf 3
Ralph Allan Bradley and Milton E. Terry Rank analysis of incomplete block designs: The method
of paired comparisons. Biometrika, 39(3-))3333444555,,,1, 12 1952. ISSN 0006-3444. doi: 10.1093/
3iomet/39.3-44322
/9. 3- 4. 324
14
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neural Information PwwpeammoPPoeeeee Processing Systems, volume 33, pp. 1877-1901, Curran Associates, Inc., 2020. URL /ttps////p http://www....opooop/o////////// 1 142f64a-Faper.c
Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Tong Wang, Samuel Marks, Charbel-Raphael Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Biyik, Anca Dragan, David Krueger. Dorsa Sadigh, and Dylan Hadfield and fundamental limitations of reinforcement learning from human feedback. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
Open problems
/ttp:s:///peeeeeeeeee www.enewwiieewweeeeeeeeeee n
orum?id-bx 44p
Survey Certification. 4
Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. I. Guyon, U. Von Luxburg, S. Bengio, H. Wal- lach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Process- ing Systems, volume 30. Curran Associates, Inc., 2017. URL / ne striss.cc/paper Satdl9-Paper.poe pdf
Frank E. Curtis, Vyacheslav Kungurtsev, Daniel P. Robinson, and Qi Wang. A stochastic -raadittt--raddent-- based interior-point algorithm for solving smooth bound-constrained optimization problems, 2024.
6
Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe RLHF: Safe reinforcement learning from human feedback. In The Twelfth Interna- tional Conference on Learning Representations, 2024. URL URL/attps:////// http://::///ppeeie.....o
orum?i Dremp:.d=TyFFPOKYX
10

---

Published as a conference paper at ICLR 2025
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversa- tions. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.). Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3009-30095 Singapore, December URL
2023. Association for Computational Linguisties. doi: 10. 1181188111222233 2ttp///22...mmmmmmaaa....
coanlp-mmaii 183
14
Christian Feller and Christian Ebenbauer. Relaxed logarithmic barrier function based model predic- tive control linear systems. IEEE Transactions on Automatic Control, 62(3):123--1233 2017. doi: 10.1109/TAC.2016.55555
5
Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization. In An- dreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learning. volume 202 of Proceedings of Machine Learning Research, pp. 100835-1866 PMLR, 23-29 Jul 2023. URL
/202/ gao2 aaaaahhhhhhh 3h. html
lan Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT Press, 2016. URL
org
Shane Griffith. Kaushik Subramanian, Jonathan Scholz, Charles L Isbell, and Andrea L Thomaz. Policy shaping: Integrating human feedback with reinforcement learning. In C.J. Burges. L. Bot- tou, M. Welling. Z. Ghahramani, and K.Q Weinberger (eds.), Advances in Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013. URL /ttttss:://rrceiiiigggg
s. s.neurips cct.ppccpppppee
dfb08da98 Paper pdf 3
Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, and Maosong Sun. Controllable preference optimiza- tion: Toward controllable multi-objective alignment. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natu- ral Language Processing, pp. 1437-1454, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10.18655 0000655/112222444 www.eemmllmamaa.ee 85. URL https:
//aclanthology www.aannthooooooooo/oooeo2//0.0 emn p main 85/
g
John Hauser and Alessandro Saccon. A barrier function method for the optimization of trajectory functionals with constraints. In Proceedings of the 45th IEEE Conference Decision and Con-
trol, pp. 864-869, 2006. 5
Jamingg Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang. and Yaodong Yang. BeaverTails: Towards improved safety alignment of LLM via human-preference dataset. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and
S. Levine (eds.), Advances in Neurul Information Processing Systems, volume 36, pp. 24678- 24704. Curran Associates, Inc., 2023. URL WtRhhttps: http://:s/pppccceeeeeeeeeeeeeeie er faaes//ppper/223//LLeeeeBea http2222211444411111111c1cc12110
Daaaaaattsss and www._peenhhhhr.eemee fd
William Karush. Minima of functions of several variables with inequalities as side conditions.
Master's 's thesis, Department of Mathematics, University of Chicago, Illinois, USA, 1939. 5
Hoel Kervadec, Jose Dolz, Jing Yuan, Christian Desrosiers, Eric Granger, and Ismail Ben Ayed. Constrained deep networks: Lagrangian optimization via log-barrier extensions. In 2022 30th European Signal Processing Conference (EUSIPCO), pp. 962 962-966, 2022. doi: 10.239 19/23II
(0550093.0022.999997 5
W. Bradley Knox and Peter Stone. TAMER: Training an agent manually via evaluative reinforce- ment. In 2008 7th IEEE International Conference on Development and Learning, pp. 292-297, 2008. doi: 10. 1001000VLLN..OOO 46..004..40855.... 3
Harold W. Kuhn and Albert W. Tucker. Nonlinear programming. In J. Neyman (ed.), Proceedings of the Second Berkeley Symposium on Mathematical Statistics and Probability, pp. 481 1994, Berkeley, California, USA, 1951. University of California Press.
11

---

Published as a conference paper at ICLR 2025
Kaiwen Li, Tao Zhang, and Rui Wang. Deep reinforcement learning for multiobjectiive optimiiztin.. IEEE Transactions on Cybernetics, 51(6):3103-1111 2021. doi: 10.1109/TCYB.22202227777
Takuya Makino, Tomoya Iwakura, Hiroya Takamura, and Manabu Okumura. Global optimization under length constraint for neural text summarization. In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computa- tional Linguistics, pp. 1009- 1048, Florence, Italy, July 2019. Association for Computational Lin- guisties. doi: 10. 10.18855/11/11-1111 1099. URL https: worlllantoooggge www.:/orgg/ssS.///
Shakir Mohamed, Mihacla Rosca, Michael Figurnov, and Andriy Mnih. Monte Carlo gradient estimation in machine learning. Journal of Machine Learning Research, 21(132):1-62, 2020.
URL hattp://e
346. html 7
Michael Moor, Oishi Banerjee, Zahra Shakeri Hossein Abad, Harlan M. Krumholz, Jure Leskovec, Eric J. Topol, and Pranav Rajpurkar. Foundation models for generalist medical artificial intelli- gence, Nature, 616(7956):254 265, Apr 2023. ISSN 1476-4687.
doi:
81-4. URL https: doi . org 10. 1038/04044 11:445866-0000 -23-866-22--200 05881-2
Ted Moskovitz, Aaditya K Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca Dragan, and Stephen Marcus McAleer. Confronting reward model overoptimization with con- strained RLHF In The Twelfth International Conference Learning Representations, 2024.
URL https: / http:::://peeeeviieeee wtt.rriwwwwneeetoneiiieeeeee
net wwr.moormmmoooooom (oooouumiigggfuuOOnOFo
7,
Stephen G. Nash, R. Polyak, and Ariela Sofer. A Numerical Comparison of Barrier and Modified Barrier Methods For Large-Scale
Optimization, pp. 319-338. Springer US, doi: 1001/999 11118881--1113-3-2-4166 URL
Boston, MA, 1994. ISBN 978-1-4613-3632-17
https::/doi.... htt/:oi..oggg//// 000.0.1000000 1001 1117888-1413-333
16 5
OpenAI. GPT-4 technical report, 2024. URL httpt::tttaaateett /ttp///xwwvvoog///////3/333
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kel- ton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances Neural Information Processing Systems, volume 35, pp. 27730-27744. Curran Associates, Inc., 2022. /2tp///2222///////
ortpr/oeeerrrrgrrrrrrrsscc/oreee paper /ttpppperr..aeaaepppe////////// URL htt.httss::/// htt.:htttss///rrrceeeeeie neerissss cc/
pdf
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In A. Oh, T. Naumann, A. Gloherson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 53728-53741. Curran Associates, Inc., 2023. paper_ wwwp::/wweeppeeeaeeeaeeaaaaaeape httpp/pppe///////ee/////
URL http:::::/.p.....ceeee..e.eaaaaa 85b4 555400e666c6677777aee33322222233 PtteeeecccePPPPPeePeee Paper wonferencee..eee pdf
14
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Jean-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: Towards Pareto-optimal alignment by inter- polating weights fine-tuned on diverse rewards. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 71095-71134 134. Curran Associates, Inc., 2023. URL /
S. ccc 8 8-Paper-Conference.
3
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Ev- timov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usumier, Thomas Scialom, and Gabriel Synnaeve. Code LLaMA: Open foundation models for code, 2024.
URL htl.:ttps:// xixv org/ abs/ 1023088.129550 12950
12

---

Published as a conference paper at ICLR 2025
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference Learning Representations (ICLR), 2016. 6 15
A Ben Tal, M Tiibulevkki,, and Yusefovich. Modified barrier methods for constrained and minimax problems. Technical report, Optimization Laboratory, Israel Institute of Technology, 1992. 5
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models, 2023. URL https: /ww.ivaiivvv..///a//eeeeeaaeaaea
www.ii.oog/gaaae....eoeeee org/ 2302 13971
14
Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. CodeT5: Identifier-awarr unified pre-trained encoder-decoder models for code understanding and generation. In Marie-France Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pp. 8696 8708, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10..1853//1/22 1/2/3///12221../// 10.18655/112221.mmmllll wew.2.2mmmmmmmmmnmm65... URL https::///// http://aallatttl aalanntthoooyy otgg//2// em
n1 mlp-main 685
Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alexander Nicholas D' D'Amurr, Sanmi Koyejo, and Victor Veitch. Transforming and combining rewards for aligning large lan- guage models. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 51161-51177. PMLR, 21-27 Jul 2024. URL htttps://proceeeeee ngs. ml m.pprsss v235 2235/wang24ay ht.l
g
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the- -frttr--rttttttt natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10.18653/v1/202 emmllp -demos.6. URL htaps:://cllaathoolloooo orteeorg////0/ emnlp demps..eem..... 6
714
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorf, and Hannaneh Hajishirzi. Fine-grained human feedback gives better rewards for language model training. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 59008 59033. Curran Associates, Inc., 2023. URL http////tts////pooeeddeno....eee
er
Paper
Conferrenceepddf
3
Xi Yang, Aokun Chen, Nima PourNejatian, Hoo Chang Shin, Kaleb E. Smith, Christopher Parisien, Colin Compas, Cheryl Martin, Anthony B. Costa, Mona G. Flores, Ying Zhang, Tanja Magoc, Christopher A. Harle, Gloria Lipori, Duane A. Mitchell, William R. Hogan, Elizabeth A. Shenkman, Jiang Bian, and Yonghui Wu. A large language model for electronic health records. doi: 10. 10.108//447446000 022-440-22200022222 URL http:httss///tm //tp::////..... wowgloorg/ooo 100 038 / 41 746 022 02002222200222222222222222222222
upj Digital Medicine. 5(1):194, Dec 2022. ISSN 2338-63522
Jingzhao Zhang, Tianxing He, Suvrit Sra, and Ali Jadbabaie. Why gradient clipping accelerates training: A theoretical justification for adaptivity. In International Conference on Learning Rep- resentations, 2020. URL /ttl:tttt///::prreeeee
net/ forum?iddBBUnXXUV77
Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. Beyond one-preference-fitss filts-ll alignment: Multi-objective direct preference optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics ACL 2024, pp. 10586-10613, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. doi: 10. 00.18653/v1/2024 https://acl antho ogg.o0g/ fi miinningee redings-ace 630
faeee..finiings- ac1.630. URL
g
13

---

Published as a conference paper at ICLR 2025
A EXPERIMENTAL SETUP
In addition the experimental setup discussed in Section 6.1 we provide further details here.
Task data We use UltraChat (Ding et al, 2023). large-scals dataset of instructional conversa- tions, as our task data to induce instruction pnstuuctionfoolowinng capabilities. Since each sample contains a sequence of multi-turn question-answer pairs, we randomly sample one of the answers as the re- sponse and treat the preceding dialogue the prompt. We then filter such (prompt, response) pairs to maximum token length of 512. Consequently, we obtain 340k training samples, 1.7k validation samples, and 1.7 test samples, split randomly since the dataset does not contain train-val-test splits.
Hyper-parameter We fine-tune LLMs for epoch on the task data, with a mini-batch size of 64. We use Adam with a learning rate of 10 and a cosine learning rate scheduler (with 5% of the epoch used for warmup). We set weight decay to 0. and the gradient clipping maximum norm to We utilize 16-bit (mixed) precision training and gradient checkpointing. We exponentially decay the log-barrier parameter μ during fine-tuning from I to 10 and use a smoothing factor of 0. for the exponential moving average. Lastly, we use top-p sampling (p set to 0.9) for response generation. Apart from this, we use the default hyper-parameter in the Transformers library (Wolf et al. 2020
A.1 LEARNING PREFERENCE REWARD MODELS
While some preference reward functions are engineered or rule-based, others are learned. Such preferences can often be difficult to quantify. Alternatively, it is easier to compare responses with respect to the preference, e.g., ranking them from most to least helpful. Consequently, the data for learning preference reward models consist of tuples of the form (r, ,+.U-), where the prompt x accompanied by two responses V+ and y-, with preference for the former response over the latter.
The preference reward model denoted by (-))) (parameterized by Φ). Assuming the Bradley-Tery model (Bradley& & Terry 1952), the model's predicted probability for preferring y+ over y- is:
Pro (y+ y-1x) (r& (v+1z) To (z-1z)).
with the standard logistic function a.)) Then, the model minimizes the negative log-likelihood:
min E(E.) [- logg Pro (++ > y- y-(z)]
Taking inspiration from Rafailov et al. (202)), we initialize the preference reward model rs(+) as pre-trained LLM and set the reward be the pre-trained model fully, not just its backbone. As the preference reward model is fine-tuned, its nog-likelihoodd/ewarrkk are updated to differentiate the preferred responses from the rejected ones.
likeliioidoood. In this way, we utilize
Helpful and harmless data We use the Helpful and Harmless (Bai et al. 2022) preference data to learn two reward models, respectively. We obtain 161k training samples and 9k test samples after filtering the (prompt. response) pairs to maximum token length of 2024; 3/4-th are for helpfulness while the remaining 1/4-th are for harmlessness, We further set 5% of the training data for validation.
Hapee--psrmeeter We initialize all reward models with LLaMA-7B [Touvron et al. 2023). We fine-tune for 2 epochs with mini-batch size of 64. We use Adam with a learning rate of 1 and a cosine learning rate scheduler (with 10% of the epoch used for warmup). We set weight decay to 0.1 and the gradient clipping maximum norm to 1. We utilize 16-bit (mixed) precision training and gradient checkpointing. Apart from this, we use the default hyper papemeersstterrs in the Transformers library (Wolf et al. 2020 We validate after every 10% of the epoch and save the best model.
14

---

Published as a conference paper at ICLR 2025
B POLICY GRADIENT
We are interested in the policy gradients (0)ss Note that while computing gradients with re- spect to the parameters of a distribution in an expectation, can use the log-derivative trick:
OgE [f f fæf (x) Oepe (x) / drf
fdrf (z)
Pa (3)
Oepe (x)
pa
= = dof (x) (x) 0g log (x) = ( [f (x) Og log pa (x))
Applying the above to the policy gradient (gC((0) yields:
(cs (y[z)] OgC; (8) (a, v~
((.))p(() (cu (ylæ) Ög log To (y]z)), go-aa(-12)
(12)
where c2 (y|rx by - (2) This is the simplest form of the policy gradient and can be estimated as
Monte Carlo averages. We refer readers to Schulman et al. (2016) for a review other estimates.
15


Reviews:
[{"review_id": "6c9Q2RWCe9", "reviewer": "Reviewer_uf9P", "summary": "This paper presents Lagrange large language models (L3Ms), which formulate supervised fine-tuning and alignment of large language models as a constrained optimization problem. L3Ms use logarithmic barriers to enforce constraints, allowing for customization across diverse applications without relying on heuristics. The paper demonstrates the versatility and efficacy of L3Ms in achieving tailored alignments.", "questions": "-Are there any plans to incorporate other evaluation metrics in addition to perplexity to better evaluate the performance of L3Ms?\n\n-Will the authors explore the application of L3Ms to a wider range of tasks, such as computational and reasoning tasks? How would the method be adapted to optimize for task-related metrics specific to these tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The formulation of SFT and alignment as a constrained optimization problem and the use of logarithmic barriers in L3Ms is a novel approach that has the potential to improve the customization of LLMs for different applications.\n\nThe paper provides a solid theoretical foundation for the proposed method, including discussions on constraint types, Lagrange multipliers, and the connection to the KKT conditions.\n\nThe experimental results on length-constrained L3Ms and helpful and harmless (HH) L3Ms show that the proposed method can effectively customize LLMs according to different constraints while maintaining performance on the original task.", "weaknesses": "The experiments mainly consider length constraints and helpful and harmless preferences. It would be interesting to see how the proposed method performs on a more diverse set of tasks, such as computational and reasoning tasks, and whether it can optimize for task-related metrics specific to those tasks.\n\nThe paper primarily focuses on perplexity as an evaluation metric. It would be beneficial to include other metrics to provide a more comprehensive assessment of the performance of L3Ms.Are there any plans to incorporate other evaluation metrics in addition to perplexity to better evaluate the performance of L3Ms? If so, which metrics are being considered and why?\nWill the authors explore the application of L3Ms to a wider range of tasks, such as computational and reasoning tasks? How would the method be adapted to optimize for task-related metrics specific to these tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ogFHUABJpu", "reviewer": "Reviewer_PMWz", "summary": "This paper presents a novel approach to the supervised fine-tuning (SFT) and alignment of large language models (LLMs), formulating it as a constrained optimization problem. The authors propose Lagrange large language models (L3Ms), which use logarithmic barriers to enforce application-specific requirements, eliminating the need for heuristic-driven processes. This approach allows for the customization of L3Ms across diverse applications. The paper demonstrates the versatility and efficacy of L3Ms through experimental results, showing their ability to achieve tailored alignments for various applications. This work contributes a new perspective and methodology to the fine-tuning and alignment of LLMs, enhancing their adaptability and effectiveness across different applications.", "questions": "See the above weaknesses.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper innovatively formulates SFT and alignment as a constrained optimization problem, offering a fresh perspective that avoids traditional heuristic methods.\n2. L3Ms, with their logarithmic barriers, demonstrate technical excellence and adaptability across diverse applications, ensuring robust performance.\n3. By providing a systematic approach to fine-tuning LLMs, this research addresses a critical need and offers practical benefits for both academic and industrial use.", "weaknesses": "1. The paper's experimental section may lack comprehensive evaluations, which could limit the full demonstration of L3Ms' capabilities and robustness across a wide range of scenarios.\n2. The choice of datasets used in the experiments appears to be restricted, potentially affecting the generalizability of the findings and the ability to assess the method's effectiveness on varied data types.\n3. Utilizing LLMs with only 7 billion parameters might restrict the scalability and performance of L3Ms, as larger models are often necessary for handling complex tasks and achieving state-of-the-art results.\n4. Writing Issues: The paper has some writing issues, including incorrect formula numbering, which can cause confusion and disrupt the flow of the paper. Improving the clarity and accuracy of the writing would enhance the overall quality and readability of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Yc0bC0TcIb", "reviewer": "Reviewer_oGFz", "summary": "This work introduces a new method for fine-tuning and aligning large language models (LLMs) by treating these processes as a constrained optimization problem. Instead of relying on heuristic methods, the authors propose using Lagrange Large Language Models (L3Ms) that incorporate constraints directly into the optimization through logarithmic barriers. This approach allows models to maintain performance on primary tasks while being customized to meet specific user or application-related preferences. The paper provides experimental evidence showing that L3Ms are effective.", "questions": "- Can authors provided more discussions on computation overheads introduced in this work in comparison to the general SFT and other related works (https://openreview.net/forum?id=TyFrPOKYXw, https://openreview.net/forum?id=gkfUvn0fLU).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Clear problem statement. This work discusses the shortcomings of previous works on constrained RLHF and proposes corresponding improvements to solve it, which seems convincing to me.\n- Good optimization problem conversion. I find it quite interesting and instructive how this work converts a general target of balancing task performance and multiple specific requirements into a solvable object function of Lagrangian and introduces the necessary components of the logarithmic barrier to make it practically work.", "weaknesses": "- My main concern is with the experimental results in Figure 3, where MM appears to achieve a better performance frontier than L3M. Although L3M results in lower perplexity (PPL), it's unclear how significantly this affects real task performance, as the improvement seems marginal. Could the authors provide further discussion on this?\n- The constraints used in the experiments are somewhat limited in terms of variety. It's unclear how the proposed algorithms would perform in scenarios with more complex or demanding constraints.\n- The experimental setups seem relatively simple, which may not fully capture the real-world performance of the fine-tuned LLMs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OtQM2c2ii0", "reviewer": "Reviewer_tPT4", "summary": "The paper suggests to model SFT and alignment into one constrained optimization problem and proposes a method called L3M, which is to optimize such constrained optimization problems via approximate Lagrangian.", "questions": "- Why the log barrier parameter $\\mu$ should be decayed during training? (Lines 715-716) \n\n- As the log barrier parameter $\\mu$ is a hyper-parameter of L3M, what the relationship between its value and L3M performance?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper suggests to formulate SFT and alignment into one constrained optimization problem and optimize its approximate Lagrangian.", "weaknesses": "- The core objective Eq.(10) $\\mathcal{G} _ \\mu(\\theta)=L(\\theta)+\\frac{1}{k}\\sum _ {i=1}^k\\mathcal{B} _ {\\mu,\\mu^2}(C _ i(\\theta))$ is equivalent to the lowerbound of multi-objective RLHF objective with induced reward models and a SFT term. Given a reward model $r _ i(x,y)$, a monotonically increasing function $f _ 1(x)=\\mathcal{B} _ {\\mu,\\mu^2}(x)$, and a monotonically decreasing function $f _ 2(x)=b _ i-x$, we can induce a new reward model $r _ i'(x,y)=-f _ 1(f _ 2(r(x,y)))=-\\mathcal{B} _ {\\mu,\\mu^2}(b _ i-r _ i(x,y))$. As $\\mathcal{B} _ {\\mu,\\mu^2}(x)$ is convex,\n$$\n\\begin{split}\n\\min &\\mathbb{E} _ {x,y}[-r _ i'(x,y)]\\\\\\\\\n=&\\mathbb{E} _ {x,y}[\\mathcal{B} _ {\\mu,\\mu^2}(b _ i-r(x,y))] \\\\\\\\\n\\geq&\\mathcal{B} _ {\\mu,\\mu^2}(\\mathbb{E} _ {x,y}[b _ i-r(x,y)])\\\\\\\\\n=&\\mathcal{B} _ {\\mu,\\mu^2}(C _ i(\\theta)),\n\\end{split}\n$$\nwhere the inequality holds by jensen’s inequality. It can be extend into multi-objective scenario with a SFT term $L(\\theta)$:\n$$\n\\begin{split}\n\\min &L(\\theta)+\\mathbb{E} _ {x,y}\\left[\\frac{1}{k}\\sum _ {i=1}^k-r _ i'(x,y)\\right]\\\\\\\\\n=&L(\\theta)+\\frac{1}{k}\\sum _ {i=1}^k\\mathbb{E} _ {x,y}[-r _ i'(x,y)]\\\\\\\\\n\\geq&L(\\theta)+\\frac{1}{k}\\sum _ {i=1}^k\\mathcal{B} _ {\\mu,\\mu^2}(C _ i(\\theta))=\\mathcal{G} _ \\mu(\\theta),\n\\end{split}\n$$\nwhich is Eq.(10) of the paper.\n\n- For baselines, as Line 355-358 writed: “L3Ms have the same time complexity as traditional approaches as we combine the SFT and alignment stages into one”. Traditional SFT+alignment, having the same training budget with L3M, should be considered as a baseline.\n\n- For benchmarks, more instruction following tasks should be evaluated to demonstrate the generalization of L3M across diverse distribution.\n\n- For evaluations, 1) ppl (perplexity) can not directly reflect the instruction following ability of an aligned LM. 2) As reward hacking is a considerable problem in RLHF, the score of reward models which are involved into L3M training is not a valid metric. Other metrics should be considered to evaluate the alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)