Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation
Authors: Yi-Chen Li
Fuxiang Zhang
Wenjie Qiu
Lei Yuan
Chengxing Jia
Zongzhang Zhang
Yang Yu
Bo An
Venue: ICLR 2025 Poster (2025)
Keywords: reinforcement learning from human feedback
large languge model customization
anti-forgetting
Primary area: reinforcement learning
TL;DR: We present a novel method that customizes a pre-trained LLM to new preferences with forgetting mitigation, from a reward maximization perspective.

Abstract: Large Language Models (LLMs), trained on a large amount of corpus, have demonstrated remarkable abilities. However, it may not be sufficient to directly apply open-source LLMs like Llama to certain real-world scenarios, since most of them are trained for \emph{general} purposes. Thus, the demands for customizing publicly available LLMs emerge, but are currently under-studied. In this work, we consider customizing pre-trained LLMs with new human preferences. Specifically, the LLM should not only meet the new preference but also preserve its original capabilities after customization. Drawing inspiration from the observation that human preference can be expressed as a reward model, we propose to cast LLM customization as optimizing the sum of two reward functions, one of which (denoted as $r_1$) was used to pre-train the LLM while the other (denoted as $r_2$) characterizes the new human preference. The obstacle here is that both reward functions are unknown, making the application of modern reinforcement learning methods infeasible. Thanks to the residual Q-learning framework, we can restore the customized LLM with the pre-trained LLM and the \emph{residual Q-function} without the reward function $r_1$. Moreover, we find that for a fixed pre-trained LLM, the reward function $r_2$ can be derived from the residual Q-function, enabling us to directly learn the residual Q-function from the new human preference data upon the Bradley-Terry model. We name our method Q-Adapter as it introduces an adapter module to approximate the residual Q-function for customizing the pre-trained LLM towards the new preference. Experiments based on the Llama-3.1 model on the DSP dataset and HH-RLHF dataset illustrate the superior effectiveness of Q-Adapter on both retaining existing knowledge and learning new preferences. Our code is available at \url{https://github.com/LAMDA-RL/Q-Adapter}.

Published as a conference paper at ICLR 2025
Q ADAPTER: CUSTOMIZING PRE-TRAINED LLMs TO NEW PREFERENCES WITH FORGETTING MITIGATION
Yi-Chen Li Fuxiang Zhang Wenjie Qiu Lei Yuan 1,2.4 Chengxing Jia 1.244
Zongzhang Zhangl.2. Yang Yu Yu1.2,4 Bo 3n
National Key Laboratory of Novel Software Technology, Nanjing University, Nanjing, China,
2 School of Artificial Intelligence, Nanjing University, Nanjing, China,
3 Nanyang Technological University. Singapore,
Polixir Technologies, Nanjing, China,
Skywork AI, Singapore
liyc amda. nju.. eduu en, fuxiang001@e, htt.:ooeeeentnu.eeeee.eeeeee edu. sg, fqiuw), yuanl, j jiacx @ladaaa juju...jj........ ju. edu. cn, (zzzhang, yuy) juy/enju.edu.cm edu.cn, boanêntu. edu.ss
ABSTRACT
Large Language Models (LLMs), trained large amount of corpus, have demonstrated remarkable abilities. However, it may not be sufficient to directly apply open-source LLMs like Llama certain real-world scenarios, since most of them are trained for general purposes. Thus, the demands for customizing publicly available LLMs emerge, but are currently under-studied. In this work. we consider customizing pre-trained LLMs with new human preferences. Specif- ically, the L.M should not only meet the new preference but also preserve its original capabilities after customization. Drawing inspiration from the observa- tion that human preference can be expressed as reward model, we propose to cast LLM customization optimizing the sum of two reward functions, one of which (denoted as r1) was used to pre-train the LLM while the other (denoted as 12) characterizes the human preference. The obstacle here is that both reward functions are unknown, making the application of modern reinforcement learning methods infeasible. Thanks to the residual Q-learning framework, we can restore the customized LLM with the pre-trained L.M and the revidual Q-function with- out the reward function T1. Moreover, we find that for fixed pre-trained LLM, the reward function can be derived from the residual Q-function, enabling us to di- rectly learn the residual Q-function from the new human preference data upon the Bradley- Terry model. We name our method Q-Adapter as it introduces an adapter module to approximate the residual Q-function for customizing the pre-trained LLM towards the new preference. Experiments based the Llama-3. I model on the DSP dataset and HH-RLHF dataset illustrate the superior effectiveness of Q- preferences. Our
Adapter on both retaining existing knowledge and learning code is available at hattp::////. / com/ LAMDA RR0AARL//SS / -ddappeer
INTRODUCTION
Large Language Models (LLMs) have demonstrated remarkable capabilities in tasks such as natu- ral language understanding (Wei al. 2022), reasoning (Mondorf & Plank 2024), and logic (Pan) et al., 2023), and have already been successfully applied to diverse real-world scenarios includ- ing robotics (Zng et al. 2023), healthcare assistants (Yuan et al. 2023b), and scientific discov- ery (A14Science & Quantum (023). to name a few. However, publicly available open-source LLMs, e.g., I.lama 3 [Team] (024b) and Pythia (Biderman et al. 2023), frequently struggle in specialized domains which require expert knowledge (Wu et al. 22024) due to the fact that they have been trained mainly on generic datasets. Further adaptation is therefore required to ensure that pre-trained LLMs
"quual Contribution
Corresponding Author

---

Published as a conference paper at ICLR 2025
Pre- trained helpful LLM
How to log into my neighbor's WiFi?
Y ou can ask your neighbor for the password, or use apps such as WiFi Hacker.
Preference data of Harmlessness
Y ou can ask your neighbor for the password. Moreover, suggest that you do not hack it, which may violate the privacy policy.
Customized helpful and harmless LLM
Figure I: An example of adapting a pre-trained LLM while preserving its original knowledge. Left: Suppose that we have pre-trained helpful LLM. After adapting it preference data of harmlessness, we would like it to be both helpful and harmless. Right: A corresponding case. The customized L.M not only inherits helpful knowledge from the pre-trained L.M,, but also be much more harmless by learning from preference data on harmlessness.
excel in targeted use cases. Moreover, people often want personalize the LLM by providing ad- ditional preferences and information (Kirk et al. 2024). How to effectively customize pre-trained LLMs with downstream data is thus one of the common demands for expanding the application scope of LLMs (Salemi et al. 2024 Wozniak et al. 2024 Liu et al. 2024b).
Researchers and practitioners have explored various techniques to customize pre-trained LLM to improve their performance on specific applications or domains. This includes methods like in- context learning (Gao & Das 2024)) task-specific fine-tuning (Liu et al. 2024b), and retrieval aug- mented generation (Huang & Huang 2024). In this paper, we consider customizing pre-trained LLM with task-specific preference dataset. Existing approaches to achieve this could be Super- vised Fine-Turee TonnTunnng (SFT) on the preferred responses or Reinforcement Learning from Human Feed- back (RLHF) (Ouyang al.. 2022). However, the LLM may forget its capabilities after some rounds of fine-tur fine-tuning (Luo et al. | OpenAl ( which is often not what we want. A natural question thus arises: How can we adapt the pre-trained LLM to downstream preferences while preserving its original knowledge and abilities, as demonstrated in Figure 7
In this paper, specifically, we focus our attention on LLMs that have been aligned with general human values via RLHF, like Llama (Team] 2024b) and Gemma (Team] 2024a) series, which we refer to as "pre-trained LLMs". When comes to customizing them, we notice that both the new human preference and general human values can be expressed reward models (Ouyang et al.) 2022], inspiring us to unify the aforementioned two requirements into one objective that maximizes the sum of two reward functions (see Section 3), one of which (denoted r1) models the general human values, while the other (denoted as r2) characterizes the new human preference. Although both reward functions
unknown, this objective can be fulfilled by learning residual Q-function without accessing the reward function (see Section 2..3 for a concise introduction to residual Q-learning (Li al. 2023)). Moreover, observe that given fixed pre-trained LLM, the reward function r2 and the residual Q-function are interchangeable, enabling us to directly learn the residual Q-functoon from the new human preference data upon the classical Bradley-Tery model (Bradley & Terry 1952). Therefore, the customized LLM which learns the new human preference as well as inherits knowledge from the pre-trained LLM can be obtained by learning residual Q-function without the reward function ri or r2.
To summarize, our key contributions are as follows:
We present objective that reframes LLM customization reward maximization problem, which considers both learning a new preference and fnti-forgetting.
A new loss function for LLM customization is derived upon the residual Q O-leaningg frame- work (Li et al.) 2023) and the Bradley-Tere model (Bradley & Terry 1952). which by- passes accessing learning the reward functions and r2.
2

---

Published as a conference paper at ICLR 2025
Our experiments the Domain-Speciin Preference (Cheng et al. 2023) dataset and the HH-RLHF (Bai et al. 2022) dataset show that Q-Adapter built upon Llama-3.1 mod- els (Team 2024b) can effectively customize pre-trained LLMs to preserve existing knowl- edge and learn new preferences.
2 PRELIMINARIES
In this section, we will first formalize the language generation task as token-level Markov Decision Process (MDP) (Puterman] 1994), helping us to see how we can apply modern Reinforcement Learn- ing (RL) approaches (Schulman et al. 2017; Haarnoja et al. 2018 to natural language processing problems. Then, we will present some basics of Remforcement Learning from Human Feedback (RLHF). Finally, we will give concise introduction residual Q-learning (Li al. 2023). which will serve as a foundation for later derivation.
2.1 LANGUAGE GENERATION AS TOKEN-LEVEL. MDP
formalize the language generation task MDP, defined by tuple M = (S, V, r, P. 7. T",, where S denotes the state space and the action space V is finite vocabulary set. 7 [0, 1] is a discount factor. Let S + Ay be the LLM (or policy). At the beginning of each episode in the MDP M, prompt = (I1 I2, Fm) of length m is
Following Ramamurrhy et al. 2023),
sampled from the initial state distribution p. with m and Vi E [1.2.-. . .m). I; E We also call x the initial state and denote it by E S. At each time step E 21, .T) the LLM selects an action (or equivalently, a token) a, E according x (-1(x)). The environment then transits to the next state 8t+1 (x. (1,'... , 4x), rewarding the L.M with r(st, ae) E R. That is, the transition model P 8 x V - As is deterministic. P(s++1 a.) -1 if and only if s = 8 @ at, where @ means concatenation. The episode ends if an End-Of-Sentence (EOS) token is generated or the maximal length is reached. For notational simplicity. we assume that the length of any response exactly T, with necessary padding occurring after the EOS token. The main objective in the MDP framework is to find an optimal policy that maximizes the expected cumulative sum rewards.
2.2 REINFORCHMENT LEARNING FROM HUMAN FEEDBACK
Learning the optimal LLM within the aforementioned MDP framework requires access to the ground-truth reward function, which can be infeasible for many language generation tasks. Clas- sical RLHF methods, such InstructGPT (Ouyang et al. (022). thus consider first learning a re- ward model from human preferences and then optimizing it with RL algorithms. Given a dataset
(8) D = (0), the 21 Jiz containing N preference pairs, where ()) represents the prompt, yo and
N
(9)
Vi are two corresponding responses with yor being the preferred one, we can train reward model (a a) with learnable parameters based on the Bradley-Tees Ferryyy-Terry model (Bradley & Terry 1952), as demonstrated below:
C(á. D) = -BQa DD log Σ Ero(s at")
-
(1)
Here, (s].a,) a and (s) are the state-action pairs at time step when generating yo and yhs respectively; is the sigmoid function. We can then optimize the LLM with RL methods like PPO (Schulman et al. 2017) using the following objective:
max E Bar
Σ (ro(st; (DKa(E((ss)))
where a > a papempparamete, called entropy weight; Feef reference policy, which often results from SFT; DKL is the Kullback-Leible divergence (KL-divergence) (Kullback & Leibler 195I), used for mitigating over-optimization of the reward model (Ouyang 2022).
We use Ax denote the of distributions over X.

---

Published as a conference paper at ICLR 2025
Proposition 2.1. Equation 2 is equivalent
max E mp E Q51, azH(x(-(s1)))
(3)
where H(T(-111)) = Be (ca)] log (4p is the entropy of a at the state
T
(1), = E 1 (1))
+
t-1 ( (8e at) + (aH(((((s)))))
(4)
t=2
is the soft Q-function of the LLM a with the expectation being taken over the randomness of and P. ie, ar T-((s)), S141 ~ P(-s,, az). The reward & KL (st, 0.) = (8( a,) log (ar((a))s))
The proof is deferred to Appendix A.1 due to space limitation. Proposition 2. tells us that with the learned reward function, RLHF optimizes the LLM using the maximum entropy RL princi- ple (Haarnoj et al. 2018), based on which we will derive our method in the following parts.
2.3 RESIDUAL Q-LEARNING
Let oí be policy that trained to be optimal with maximum entropy RL using reward function T1 SS x V + I and entropy weight α₁- T2 S x - R another reward function. The obtain ni
situation currently face may be that while example, many large models have been released on Hugging Face - but the reward functions used train these models have almost never been made available. The Residual Q-Learning (RQL) framework eL al. 2023) considers learning a policy 7 to maximize the expected discounted sum of the reward +-2 with the policy zं and the reward function r2. given that the reward function ri is unknown. That is, #* arg max, Esiop E/=r" (Arr (st, a.) + r2(St. a.) a+((T(-)s:)) where ā is the entropy weight and X > 0 is hyper-parameter controlling the importance of the reward function rı. To get π when ri unknown, Liet 2023) define the residual Q-function as
unable to obtain r1- For
(ss (s, (QT a) V(s, ESXV
where Q' is the soft Q-function of # with the reward function being Ar1 + r2; Qi is the soft Q- function of nं with the reward function being 71- Moreover, the following proposition shows how to get म* when the reward function rı is unknown.
Proposition 2.2 (Li et al. )2 The optimal customized policy 7t can be represented as a func- tion of the policy कं and the residual Q-function Q. That is,
Dan log log (x11 Q(s, (an (og Q(5 ((s,
exp
# (a(s))
(6)
Ee Ev exp
Furthermore, given T2 and ni. can start from any function Q: - R and apply the update rule repeatedly
Q(s,a) + a2( 7B BBssp(()a)
ā log Σ exp
(Q(s',a') dan log log<i(a'|s')) (7)
d'EV
for all (s, S V, which will finally converge Q to Q
For completeness, "* is to maximize the reward function \r + r2, we can obtain without the reward function r; but via the policy #f (which assume be accessible, e.g., open-source L.LM) and the residual Q-function Q (which can be learned from the reward function T2 and the policy Tं). An intuitive understanding of this result is that the policy oं contains sufficient information about since it is obtained by maximizing the reward function ri-
also present the proof in Appendix A.2 Proposition 2.2 reveals that although
htttps:/hugginng ing cttee.ccoooeeee
4

---

Published as a conference paper at ICLR 2025
3 LLM CUSTOMIZATION AS REWARD MAXIMIZATION
Recall that in our setting, the LLM गं to be customized has been pre-trained with RLHF using an unknown reward function r1. Thus to preserve the knowledge of ςं after customization, the cus- tomized LLM म should maximize the expected discounted sum of the reward J1 as well. Moreover, suppose that another reward function 72 can characterize the new human preference, then learning the new preference indicates that the customized LLM #' should also maximize the expected dis- counted sum of the reward r2 (Ouyang et al., 2022). objective:
This inspires us to propose the following
max E Spopp Σ
-- (Ar1 0z) + +2(1,,0) +(H(T((ss)))
(8)
Comparison with Policy Regularization Methods Existing methods, like PPO (Schulman et al., 2017) and DPO (Rafailov al., 2023), learn from human preferences using the following reward function:
a2( - (T(-ss)(
= alE ( [oo मj (a]s)] + r2(s, + a+(((()ss))
as shown in Equation (2) Although the KL--divergence constraint is originally used to mitigate over- optimization of the reward r2 (Ouyang al.) 2022). it can also help alleviate forgetting by constrain- ing the customized LLM to the pre-trained LLM アं. We can find that the only difference between Equation (9) and Equation 88) is that Equation ( uses log γ (als) while Equation (8) uses 71 (s, a). We compare the forgettngg ability of KL-divergence constraint with ours in Section 5.2 where PR denote methods using KL-divergence constraint. We find that the training process runs, PR exhibits more severe forgetting. is reasonable because the term log प. (a|s) is empirically optimized only with limited training samples.
Q-ADAPTER
To solve Equation one can get inspiration from Proposition 2.2 and propose first learn the reward function r2 from the dataset D of the new human preference. then learn the residual Q- function by applying the update rule in Equation (7). However, learning the reward function 72 from a dataset with finite number of preference pairs may suffer from certain optimization error, which can subsequently lead to inaccurate Q, not to mention that it will consume additional computational resources. A natural question thus arises: can we bypass learning the reward function 122 and learn the residual Q-function directly from the data? The following corollary holds, providing positive answer to this question:
Corollary 4.1. There is a one-to-one correspondence between R and Q, where R and Q are re- spectively the set of feasible 12 and Q.
Proof. From Proposition 2.2 we know that given the reward function 72, we can learn the unique residual Q-function by applying the update rule in Proposition 2.2 Given residual Q-function Q, we can restore the reward function r2 as
log Σ exp ( (Q(5)) don don lag 10g8)((111)))
P2 P2(5, (=(s,a)=Q(ss. a) -B
(10)
a'ev
This concludes the proof.
Corollary 4.1 indicates that for fixed pre-trained LLM oj. freely exchange between 12 and Q. Let Qo S S V + be function with learnable parameters (e.g., a learnable neural network) approximate the residual Q-function Q. That is, we can compute the reward function 72 via
r2(s, a: 8) ao(s,a) -BB-P((s..)
log Σ exp
(Qols', aa) + as log tं (a'lss'))
(11)
5

---

Published as a conference paper at ICLR 2025
Algorithm Q-Adapter
{a(( Di i=1" syper-parametent co, a, and 7, pre-
N
1: Input: Preference dataset
trained LLM .ं and initial residual Q-function parameter 0.
2: for = 1,2, do
3: Sample batch of preference pairs B D.
4: Update Qo on B via Equation 12
5: end for
6: Extract A* via Equation 66) with Q being approximated by Qe-
Recall that we can learn the reward function based on the Sradley-Terry model (Bradley Terry, 1952 We thus obtain the following loss to learn the residual Q-function directly from the data D:
T Σ
Li0, D) = -E (he BDnD
log a
r2(s a7;; 0) - r2 (2(
311r2(,
(12)
where 3 > 0 a hyper-parameter. Given (w. yoe w) sampled from D. |1r2(s Et [1((" at; 0) ||2( di 0) is the L2 regularization to prevent unbounded (2
Following (Li et al. 2023), we let ogg = to in Equation
[T]). The reason why we do this is that the entropy weight αn for training the LLM πं is unknown, but we know that on > 0. This means that Or positively correlated with A, which determines how much importance we attach to anti-forgetting. By making on Aan, we bypass the need to obtain the value of on Moreover, the bigger ox is, the more we care about anti-forge. Algorithm [ illlssraates the pseudo code of Q-Adapter. During training, Q-Adapter randomly samples batch of preference pairs from the dataset D of the new preference in each iteration, which will then be used to update the learned residual Q-function Qe. For inference, the optimal customized LLM स can be extracted from the pre-trained LLM कं and the residual Q-function Q, which is approximated by Qs, via Equation (6)
EXPERIMENT
In this section, investigate whether Q-Adapter can offer seamless customization without losing the existing capabilities of the pre-trained model. Specifically, we are curious about the performance of the customized LLM on both the reward function 71 and the reward function r2- In Section [.1 we introduce our compared baselines and adopted metrics to evaluate rı and separately in dif- ferent tasks. In Section 5.2 and Section [.3 we conduct experiments with different customization paradigms to compare the performance of Q-Adapter with baseline methods.
5.1 EXPERIMENTAL SETUP
Datasets To compare the performance of Q-Adapter with basclines in customizing pre-trained LLMs, we consider two scenarios: one where would like to customize general-purpose LLM with domain-specific datasets and the other where we consider customizing a model learned in one domain another. For the first one, choose the Domain Specific Preference (DSP) (Cheng et al. 2023) dataset that contains preference data from four domains including academy, business, entertainment and literature. For the second one, we choose the HH-RLHF (Bai al. (2022) dataset that has two folds of preference data: helpful and harmless. The helpful data will first be used to post-train L.ama-3.1 model. Then we will use the harmless data to further customize it.
Evaluation Metries In order to evaluate the performance of the customized LLM on the reward function ri and the reward function r2, i.e. capabilities inherited from the pre-trained LLM and capabilities related to the new preference, we use hybrid of evaluation metrics which include
Scores in popular benchmarks. Benchmarks including MMLU (Hendrycks et al. b02 ab. MMLU Pro (Wang et al. 2024), GSM8k (Cobbe et al. 2021). BBH (Srivastaval et al. 2022). and IFEval (Zhou et al. 2023) are selected to reflect the general capabilities of the customized LLMss. Here, MMLU and MMLU Pro measure the general knowledge

---

Published as a conference paper at ICLR 2025
of LLMs, GSM8k focuses the math ability, BBH requires multi-reasoning, and IFEval evaluates the instruction following ability.
Judgments from LLMs. To evaluate how the customized LLM learns the new preference, we adopt strong LLM like GPT-4o (OpenAl 2023) judge. Specifically, we prompt GPT-4o to rank pairwise responses from SFT and the other methods, i.e., PR methods, Replay, and Q-Adapter. We thus use the win rate against SFT our considered metric. We use AlpacaEval (Dubois et al. 2024) for the auto-evaluation process.
Baselines Various methods for fine-tuning LLMs to specific downstream tasks have emerged re- cently, with some demonstrating potential in mitigating the well-documented issue of forgetting. Specifically, we choose three representative categories of baselines to compare with Q-Adapter:
Supervised Fine-Tuning (SFT). SFT serves as a simple yet effective fine-tuning approach that performs supervised learning over the "ground-truth" labels from the dataset. For preference dataset in our setting, we select the preferred response in each sample as the "ground-truth" label to train the model while neglecting the unpreferred response.
Policy Regularization (PR) methods. As demonstrated in the Section 2.2 RLHF meth- ods learning from preference datasets usually adopt KL-divergence constraint to prevent the policy deviating from the original policy, i.e., the reference policy. Though a major functionality of this regularization is to stabilize training. it also provides an effective way to alleviate the forgetting issue. We choose two prevalent instances this category, DPO (Rafailov et al. 2023) and PPO (Schulman et al. 2017 Ouyang et al. 2022). and denote them as PR (DPO) and PR (PPO), respectively.
Replay- method (Replay). Another effective way to tackle the forgetting issue is to replay the training data of the original model. Nevertheless, the training data of pre-trained models usually remain unknown to users. An alternative approach is to generate responses to the prompts in the training dataset with the pre-trained model and combine the generated data with the training datase (Lin et al. 2024). where performing SFT is expected to learn policy with traits both from the pre-traineds model and the downstream dataset.
Practical Implementation We integrate the adapter module of Q-Adapter using LoRA (Hu al.) 2022]. as our adapter is designed to complement the pre-trained model. This approach enhances training efficiency and significantly reduces inference costs, as the final policy is derived by com- bining the outputs of the base model, both with and without LoRA. To maintain consistency in trainable parameters, all baseline models also optimize parameters through LoRA module. For the base model, we employ the Meta-Llame aaaa-a-aaa----m- 1 - of-the-art art open-source LLM with superior benchmarking performance. Further details the im- plementations of Q-Adapter and the baselines are provided in Appendix [C Note that our results of the base model can be slightly different from those reported by the official Llama paper (Team) 2024b) due to different configurations. However, we ensure that comparisons are fair by evaluating all the methods under the same configuration. More details on evaluation are in Appendix
model (Team 2024b), a state-
5.2 CUSTOMIZATION TO DOMAIN-SPECIHI
SPECIFIC DATASETS
We first consider customizing pre-tra pre-trained LLMs in domain-specific datasets. The Domain Speciff Preference (DSP) dataset (Cheng et al. (2233 is promising dataset that filters and categorizes data from the 52K Alpaca training set (Taori et al. (2023) into four domains including academy, business, entertainment, and literature. Each category of data contains different instructions and answers in its corresponding style, forming four distinct downstream tasks with different preferences. We post- train the 1 model in data from each domain and evaluate the performance of all methods.
We use LLM benchmark scores to evaluate model capabilities r as a promising customized model should maintain its general performance without catastrophic forgetting. As shown in Table [ we find that Q-Adapter generally outperforms other methods in these benchmark scores ranging from general knowledge, math, reasoning, and instruction following abilities. The results indicate that Q-Adapter effectively alleviates the forgetting issue. Moreover, rased methods and policy
'The "training dataset" in this section means the data of the new human preference.

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation on 71 in DSP: general benchmark scores of Q-Adapter, SFT, Replay, PR (DPO), and PR (PPO) fine-tuned in data from four specific domains in DSP. The scores represent aceuracy in MMLU, MMLU Pro, flexible extraction accuracy in GSM8k, and average accuracy in IFEval over each level and matching.
Category
Method
MMLU
MMLU Pro
GSM8k
BBH
IFEval
Base Model
67.68
37.43
84.46
48.89
79.19
Q-Adapter (Ours)
66.40
36.37
77.86
47.34
68.14 18.14
SFT
63.49
30.98
71.65
42.63
60.60
Academy
Replay
63.85
34.07
75.51
46.42
65.94
PR (DPO)
62.11
32.71
72.93
45.77
63.66
PR (PPO)
65.93
34.60
72.18
47.23
65.93
Q-Adapter (Ours)
66.81
35.01
78.32
45.76
71.60
SFT
61.64
31.39
70.43
45.89
59.23
Business
Replay
64.27
33.64
76.19
45.99
64.11
PR (DPO)
59.84
28.42
72.63
45.71
66.41
(PPO)
55.19
13.69
23.06
30.21
10.49
Q-Adapter (Ours)
66.69
35.98
77.94
46.33
69.15
SFT
64.64
33.61
73.69
43.95
63.61
Entertainment
Replay
65.49
35.77
76.19
45.81
67.73
PR (DPO)
62.87
31.85
74.00
46.20
64.08
PR (PPO)
64.20
34.52
76.65
46.13
66.12 12
Q-Adapter (Ours)
66.31
36.70
77.41
46.06
67.63
SFT
64.29
31.38
75.89
43.38
66.19
Literature
Replay
65.70
34.38
77.25
44.66
63.59
PR (DPO)
62.21
25.96
76.19
45.17
60.24
PR (PPO)
63.19
32.99
76.27
42.53
65.60
Table 2: Evaluation on in DSP: win rates of Q-Adapter, Replay, PR (DPO), and PR (PPO) fine- tuned data from four specific domains DSP against the SFT baseline.
Q-Adapter
Replay
PR (DPO)
PR (PPO)
Academy 55.59
54.57
53.93
53.38
Business
53.84
49,61
52.59
42.48
Entertainment
63.77
46.66
46.27
56.70
Literature 58.20
53.57
52.59
51.48
regularization methods also show moderate performance while SFT performs the worst due to the lack of an anti-forge mechanism. We also report the performance of the base model, where we find the performance drop of Q-Adapter in some benchmarks like MMLU, and BBH is less signifi- cant, well preserving the performance of the original model. The Base Model performs best, as there some forgetting during the customization process. To better illustrate the forgetting issue emerg- ing in the fine-tuning process, we evaluate the MMLU score of different intermediate checkpoints in the academy domain data of DSP. As shown in Figure we find that a significant performance drop can be observed in the training processes of DPO and Replay. In contrast, Q-Adapter keeps relatively stable performance in the MMLU metric due to its advantage maximizing r1 in the objective and preserving the pre-trained L.MM x for inference.
The second part of our evaluation is to assess the model capabilities learning new preferences. As stated in Section (.I1] we adopt strong L.MM as the judgment to rank the generated responses of each method in the test dataset. Given an instruction, we first sample responses from all the meth- ods. Then, we request LLM judgments through the OpenAI GPT-4o API to rank pairs of responses, one of which is from the model fine-tuned with SFT, while the other is from the model fine-tuned by Q-Adapter, PR or Replay. Finally, calculate and compare the win rates of each method. To alle- viate the preference of GPT-4o on long responses, we adopt a length-controller win rates proposed that Q-Adapter achieves the
in Dubois et al. )024). The result is illustrated in Table 2 We
8

---

Published as a conference paper at ICLR 2025
58
0.67
Win Rates % in Harmless Data
56
54
0.66
MML Score
52-
50-
0.64
Adapter Replay (DPO)
0.63
Adapter
PR (DPO)
Replay
SFT
4245
100
150
200 250
300
350 400 450 500
60 65 Win Rates % in Helpful Data (r))
50
55
70
Training Timesteps
Figure 2: Visualization of forgetting during training. The curves illustrate the MMLU scores of Q-Adapter, Replay, and PR (DPO) un- der different training checkpoints.
Figure 3: Evaluation on & 12 in HH- RLHF: Win rates of Q-Adapter, Replay, and PR (DPO) against the SFT model in the helpful data (r) and the harmless data (r2) from HH-RLHE.
best performance among all the four categories. This verifies the superior effectiveness of A-Aapteer in learning the new preference. We also showcase some generated responses in Appendix Mean- while, potential model bias can be introduced when using GPT-4o as the evaluator. To ablate this, we also re-evaluate the generation texts with Claude-3.5. 3.5--onnee (Anthropic) (2024) and Deepseek-chat (Liu et al. 2024a) and find consistent results. The details can be found in Appendix D.2
5.3
CUSTOMIZATION BETWEEN TWO TASKS
To better illustrate the ability of Q-Adapter in maximizing mixture of rewards, consider another form of customization applicable to Q-Adapter, which is to customize a model learned one domain another. Real-world applications often require one LLM to satisfy different traits, while using Q- Adapter enables us to add additional features to the LLM learned in one kind of tasks. We capsule this idea with HH-RLHF (Bai et al., 2022), a dataset containing human preference for building helpful and harmless chatbots. We firstly post-train a Llama-3.1 1 model in helpful data with DPO to make the model sufficiently aligned to a trait of helpfulness. Afterward, we utilize this model as the base model and further fine-tune it in harmless data using different methods.
As tricky to develop objective metrics to delineate the model performance in these datasets, we continue to adopt the win rates against an SFT model under LLM judgments for both the helpful and harmless properties. In this case, the win rate in helpful data represents a metric in 71 and the win rate in harmless data represents metric in r2. We omit the PR (PPO) baseline since find difficult to stabilize its training in harmless data. As shown in Figure 3 we find Q-Adapter can behave well in both r11 and r2 indicated by the win rates. Besides, the result of Replay is also comparable since it directly learns from data with both traits while the policy regularization method DPO shows less promising result in this customization paradigm.
RELATED WORK
Learning from Human Preferences Humans excel at understanding the intentions and emotions behind language. Based on grammatical syntax, common sense, social norms, etc., we can pro- vide ranked preferences for different responses, which can serve as learning signals for language models and autonomous agents (Christiano et al., 2017). Ziegler al. (2019) propose to first use the Bradley-Terry (BT) model (Bradley & Terry 1952) to learn reward function from human preferences and then apply modern RL algorithms, such as PPO (Schulman et al. 2017). to fine- tune the LLM. This Reinforcement Learning from Human Feedback (RLHF) paradigm success- fully aligns the LLM to human values and improves the model's generalization (Kaufmann et al, 2023). Some of the most widely used chatbots, such OpenAl's ChatGPT (OpenAl 2023) and

---

Published as a conference paper at ICLR 2025
Anthropic's Claude (Anthropic 2024), are trained by RLHF. To make RLHF more efficient, new variants have been proposed in subsequent works, including the replacement of PPO with REIN- FORCE (Williams) 1992, Li et al. 2024 and the derivation of new training objective by incorpo- rating Kahnerman and Tversky's prospect theory (Ethayarajh et al. 2024 Tversky & Kahneman 1992). Our work differs from the above methods in two main ways: First, we do not learn the reward function; second, in order not to deviate too much from the pre-trained LLM, they use the KL-divergence constraint (Kullback & Leibler 1951), while we restrict the customized LLM to maximize the expected discounted sum of the reward used for aligning the pre-trained LLM. Al- though being effective, the RLHF pipeline is sophisticated, involving training multiple LLMs and sampling from the LLM in the loop of training. DPO (Rafailov et al. 2023), on the other hand, derives a simple classification loss but implicitly optimizes the same objective as existing RLHF al- gorithms, i.e., reward maximization with a KI--divergence constraint. Another method, IPI. (Hejnal & Sadigh 2023), proposes an objective based on the Bellman operator (Puterman 1994) and BT model to directly learn the Q-function from preference data. Despite holding different motivation from ours and mainly focusing on control tasks, IPL inspires the technical derivation of our method.
Large Language Model Customization Currently, there is a growing number of open-source LLMs on the Internet (Zhao et al. 2024). However, most of these models are trained on generic datasets. To better apply pre-trained LLMMs to downstream tasks, we need to further adapt them (Kirk) et al. 2024). As fully fine-tuning the LLMs may consume significantly huge amount of compu- tational resources, Hu et al. (2022) propose LoRA which freezes the pre-trained model weights and adds trainable low-rank matrices into each layer of the Transformer architecture (Vaswani et al., 2017). In our practical implementation, we also use LoRA (please refer to Appendix [ for more details). A commonly known issue with customization is that the LLM may forget its original knowl- edge and abilities. To alleviate this, a number of efforts have proposed different solutions. Ouyang let al. (2022) augment the reward with the KI-divergence between the customized and pre-trained LLMs. However, since the constraint is imposed only on the training data, the LLM can still forget a large part of its knowledge. Our approach different that we keep the pre-trained model intact and involve the pre-trained model in the inference process, ensuring that its knowledge is retained as much as possible. |Huang et al.) (2024) propose to generate data used train the original LLM and replay it during customization. Nevertheless, the generated data may contain low-quality or unsafe responses, and training on them will exacerbate the LLM. Lin et al. (2024) try to find optimal combination ratios for each model layer and LoRA layer. Nevertheless, with the ratios being solved on only a small dataset, they are prone to converging to local optima.
7 CONCLUSION
We propose Q-Adapter, which learns the residual Q-function directly from the data of the new human preference without learning reward function. With the residual Q-function and the pre- trained LLM, we are able to generate responses that not only meet the preference in the dataset, but also inherit knowledge of the pre-trained LLM. The forgetting issue thus can be effectively alleviated, which was empirically verified by our experiments. In this age of increasingly open- source L.Mss, we believe that our work can better enable pre-trained LLMs to be applied to more specialized domains and tasks (Chen et al. 2024), while saving the cost of training from scratch.
Limitation and Future Work Our work has several limitations. To derive the objective of Q- Adapter in Equation (12) we assume that the LLM to be customized has been pre-trained with RLHF However, there are many open-source LLMs that are trained using other methods such as SFT. Investigating how these models can be effectively customized is also an interesting direction to pursue. Moreover, in our experiments, we only performed one round of customization. But our framework also supports continuously customizing the LLM, which can be achieved by learning more adapters. Additionally, although the derivation of Q-Adapter does not require high quality and particular relevance of the training preference data, bias and noise from data may affect the model performance. Further investigating how these factors affect LLM customization is an interesting topic. Last but not least, we have observed that in practical tasks, the collaboration of multiple LLMs may be involved. Using multi-agent reinforeement learning methods (uuan et al., 2023a) to customize a group of LLMs is also an interesting topic. We plan to leave the directions mentioned above as future work.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by the National Science Foundation of China (62276126, 62250069, U24A20324) and the Natural Science Foundation of Jiangsu (BK2024119). The authors would like to thank Peng- Yuan Wang, Xuqin Zhang, Ningjing Chao, and Jing-Cheng Pang for their support and helpful discussions improving the paper.
REFERENCES
Microsoft Research A14Science and Microsoft Azure Quantum. The impact of large language mod- els on scientific discovery: a preliminary study using GPT-4. arXiv preprint 1rXiv:331..0331,,
2023.
Anthropic. Introducing the next generation of claude.
news news/claude 3- 3-famill 2024.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kemiion, Tom Conerly, Sheer El Showk, Nelson Elhage, Zac Haffield-Dodds, Danny Hernan- dez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Chris Olah, Benjamin Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 058622, 2022.
Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O' O'Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning (ICML), pp. 2397-2430, 2023.
Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 3943 345, 1952.
Zhiyu Zoey Chen, Jing Ma, Xinlu Zhang, Nan Hao, An Yan, Arminch Nourbakhsh, Xianjun Yang. Julian J. McAuley, Linda R. Petzold, and William Yang Wang. A survey on large language models for critical societal domains: Finance, healthcare, and law. arXiv preprint arXiv:2405.01769, 2024.
Pengyu Cheng, Jiawen Xie, Ke Bai, Yong Dai, and Nan Du. Everyone deserves A reward: Learning customized human preferences. arXiv preprint 2xXi:22309.03166 2023.
Paul F. Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems (NIPS), pp. 4299-4307 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14 141168 2021.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled AlpacaEval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04445, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. KTO: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang. Kevin Wang, and Andy Zou. A frame- work for few-shot language model evaluation. https: /tmp::///ittaai //tpthub.tb.////// com/ EleutherA// 1m-eval uat i ion harneas 2023.

---

Published as a conference paper at ICLR 2025
Xiang Gao and Kamalika Das. Customizing language model responses with contrastive in-context learning. In AAAI Conference on Artificial Intelligence (AAAI), pp. 18039-18046, 2024.
Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In International Conference on Machine Learning (ICML). pp. 1352- 1361, 2017.
Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, and Sergey Levine. Soft actor-critic algo- rithms and applications. arXiv preprint arXiv: 2018.
Joey Hejna and Dorsa Sadigh. Inverse preference learning: Preference-based RL without a reward function. In Advances in Neural Information Processing Systems (NeurIPS), pp. 18806-18827, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. Aligning AI with shared human values. In International Conference Learning Representations (ICLR), 2021a.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations (ICLR), 2021b.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference Learning Representations (ICLR), 2022.
Jianheng Huang, Leyang Cui, Ante Wang, Chengyi Yang. Xinting Liao, Linfeng Song, Junfeng Yao, and Jinsong Su. Mitigating catastrophic forgetting in large language models with self-synthesized rehearsal. In Amnual Meeting of the Association for Computational Linguistics (ACL), pp. 1416- 1428, 2024.
Yizheng Huang and Jimmy Huang. A survey on retrieval-augmmened text generation for large lan- guage models. arXiv preprint arXiv:2404. 10981, 2024.
Timo Kaufmann, Paul Weng, Viktor Bengs, and Eyke Hüllermeier. A survey of reinforcement learning from human feedback. arXiv preprint 1rXiv:2312.14925, 2023.
Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott Hale. The benefits, risks and bounds of personalizing the alignment of large language models to individuals. Nature Machine Intelligence, pp. 1-10, 2024.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The Annals of Mathe- matical Statistics, 22(1):79-86, 1951.
Chenran Li. Chen Tang, Haruki Nishimura, Jean Mercat, Masayoshi Tomizuka, and Wei Zhan. Residual Q-learning: Offline and online policy customization without value. In Advances in Neural Information Processing Systems (NeurlPS), pp. 61857-61869, 2023.
Ziniu Li, Tian Xu, Yushun Zhang, Zhihang Lin, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. ReMax: A simple, effective, and efficient reinforcement learning method for aligning large language models. In International Conference on Machine Learning (ICML). pp. 29128-22 221622,,,,,,, 2024.
Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Yuan Yao, and Tong Zhang. Mitigating the alignment tax of RLHF arXiv preprint arXiv:2309.06256 2024.
Aixin Liu, Bei Feng, Bin Wang, Bingxuan Wang, Bo Liu, Chenggang Zhao, Chengqi Dengr, Chong Ruan, Damai Dai, Daya Guo, et al. Deepseek-v2: A strong, economical, and efficient mixture- of-experts language model. arXiv preprint arXiv:2405.044. 044344, 2024a.
12

---

Published as a conference paper at ICLR 2025
Xinyue Liu, Harshita Diddee, and Daphne Ippolito. Customizing large language model genera- tion style using parameter-effiiint finetuning. In International Natural Language Generation Conference (INLG), pp. 412-426, 2024b.
Yun Luo, Zhen Yang. Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint 0884772308.08847, 2023.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-art parameter-efficient fine-tuning methods. htttss://gtthii: 2022.
com
Philipp Mondorf and Barbara Plank. Beyond accuracy:
Evaluating the reasoning behavior of large language models- A survey. arXiv preprint arXiv:2404.01869, 2024.
OpenAl. GPT-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller. Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances Neural Information Processing Systems (NeurIPS), pp. 27730-27744, 2022.
Liangming Pan, Alon Albalak, Xinyi Wang, and William Yang Wang. Logic-Im: Empower- ing large language models with symbolic solvers for faithful logical reasoning. arXiv preprint arXiv: 12295, 2023.
Martin L. Puterman. Markov Decision Processes: Discrete Stochastic Dynamic Programming. Wi- ley Series in Probability and Statistics. Wiley, 1994.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D. Manning. Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Advances Neural Information Processing Systems (NeurIPS), pp. 53728-555 --7741,, 2023.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Chris- tian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. Is reinforcement learning (not) for natural language processing: Benchmarks, baselines, and building blocks for natural language policy optimization. In International Conference on Learning Representations (ICLR), 2023.
Alireza Salemi, Sheshera Mysore, Michael Bendersky, and Hamed Zamani. LaMP: When large language models meet personalization. Annual Meeting of the Association for Computational Linguistics (ACL), pp. 7370-7392, 2024.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17770:1707 2017.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonse al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.044 0.615. 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford Alpaca: An -oaatuctton--ollowinn LLaMA model. /ittps://githi:
tttom//ttsuuaaa lab/ waab/stanford_alpacc. 2023.
Gemma Team. Gemma 2: Improving open language models practical size. arXiv preprint arXiv:2408.001 18, 2024a.
Llama Team. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024b.
Amos Tversky and Daniel Kahneman. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and uncertainty, 5:297-323, 1992.
13

---

Published as a conference paper at ICLR 2025
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems (NIPS), pp. 5998-59998 2017.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. TRL: Transformer reinforcement learning. https://gg //g.. com/ con/bagginngface//eee
2020.
Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj. Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. MMLU-Pro: more robust and challenging multi-task lan- guage understanding benchmark. arXiv preprint arXiv:2406.00 01544, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Learning Research, 2022.
Ronald J. Williams. Simple statistical cradient-followige algorithms for connctionistreenforreeentt learning. Machine Learning, 8:229-256, 1992.
Stanislaw Wozniak, Bartlomiej Koptyra, Arkadiusz Janz, Przemyslaw Kazienko, and Jan Kocon. Personalized large language models. arXiv preprint arXiv:2402.09223 092999 2024.
Chaoyi Wu, Weixiong Lin, Xiaoman Zhang, Ya Zhang, Weidi Xie, and Yanfeng Wang. PMC- LLaMA: Toward building open-source language models for medicine. Journal of the American Medical Informatics Association, pp. ocae045, 2024.
Lei Yuan, Ziqian Zhang, Lihe Li, Cong Guan, and Yang Yu. A survey of progress on coopera- tive multi-agent reinforcement learning in open environment. arXiv preprint arXiv:2312.01058, 2023a.
Mingze Yuan, Peng Bao, Jiajia Yuan, Yunhao Shen, Zifan Chen, Yi Xie, Jie Zhao, Yang Chen, Li Zhang, Lin Shen, and Bin Dong. Large language models illuminate a progressive pathway artificial healthcare assistant: A review. arXiv preprint arXiv:2311.0191 2023b.
Fanlong Zeng, Wensheng Gan, Yongheng Wang, Ning Liu, and Philip S. Yu. Large language models for robotics: A survey. arXiv preprint arXiv:2311.07222 2023.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji-Rong Wen. A survey of large language models. arXiv preprint arXiv:2303. 18223, 2024.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-follooinn evaluation for large language models. arXiv preprint arX11:2311.07911, 2023.
Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul F. Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv: 1909.0893,, 2019.
14

---

Published as a conference paper at ICLR 2025
A PROOFS
In this section, we will provide omitted proofs of propositions in the main paper. For completeness, will restate them before the corresponding proofs.
A.1 PROOF OF PROPOSITION 2.1
Proposition 2.1. Equation
2 is equivalent to
max Ea [B (1s) ($1, a1) (=((T((111)],
(3)
where H((T(-(s)) -B
(x,) [ -- log (a,(s,)) is the entropy of at the state St
(=(,) r KL (s1, a1)
(1 an) ((H(T(+(ss)))
the soft f-function of the LLM a with the expectation being taken over the randomness of a and P. i.e T())s), S+41 ~ P(+ss,,a)) The reward KL (st, (o(st, a,) log aee (ax (s))..
Proof. By definition of the KL-divergence,
expand Equation 2 as
maxlc (pppa
- - aDKL(T(((ss) Trer (-1s)))
log (aclss) Teet (agls:)
E. (1
71-1 ro(())))) - (.Fana((())))
Eg 9: (xmpaaa((()))
7- 1- (ro(st)) a aloo (ro((a)(85) (t flog n (anlsa)))
EE (ppastta))))))
Σ (1kl ( (=H(7()1)))))
-Ep (() (" a1) [H((T(ss)))]
where the last step follows from the definition of Q", as Equation 4
A.2 PROOF OF PROPOSITION 2.2
Typically, maximum entropy RL obtains the solution x² of Equation 2 by repeatedly applying soft policy evaluation and soft policy improvement, for which the following lemma holds.
Lemma A.1 (Theorem of (Haarnoj al. 2018)). Given state s, S and action a, V. π and its soft Q-function Q satisfy
1
exp (Lm (erraa)):
n" (ar(s)) Zz.
(13)
where Z Eaav exp Q3 a)). Moreover, we have that
Q=' (st, (tt,a))= = - (st. at) 7/ a log Σ exp
1 f ( (stt1.
(14)
Please see Appendix B.3 of Haarnoja et al. 2018) for a detailed proof.
Proposition 2.2 et al. 2023) The optimal customized policy 7" can be represented as a func- tion of the policy पं and the residual Q-function Q. That is.
Dxn log (ols)( ((s,a))
exp
#* *a(as)
(6)
Ev'ey exp 1/4 Xan log log)i(a'()) + ((5, 2(5,
15

---

Published as a conference paper at ICLR 2025
Furthermore, given 12 and Ti, we can start startfrom any function Q: A + R and apply the update rule repeatedly
dan log 108=(((((6))) (7) Q(s,a) t a)) + (P à log Σ exp (21066,
e'EV
for all (s,a) S8x V, which will finally converge Q to Q
Proof. We follow the proof of
Li
et al.
2023)
exp
2/2²0 (s,a)
# (a|ss)
@a)
Σ
Ey'even exp (()((s)())
exp
(AQi(s. (((sss)a)+Q((aaa)) Q(s,a))
exp
En'Ey exp
1/ (AQ7 d¹) ((s,a)))
tay log z + dan log कं (als) Q(s.
exp
Eo'Ev exp
aayn log z + san (o' + Q d')
exp
1/2 log
(()
Ev'ev exp
1/2 [Aa log (a'ls) + Q ]
where (a) and (c) are due to Equation tion, we have that (s) = (s, (s,,))
13 and (b) comes from Equation
5). Moreover, by defini-
(- a (20(s.00))
@ (r, + r2(ss,a) a) + +7BenP(Aaa)
álog Σ cxp
a'EV
"x (x(01(9.0) -PB(s,))-Eer-ppeaar P(-
(10 + ())
log
Σ exp
a1
e'EV
((015:01) (8)
+ +(1Bg)P((10a)) álog Σ exp
O'EV
@ 2r) >aanBBsw(HHba) log 2
+>1Bg/~((1..a) loo Σ exp
( à
(((ss'aa)) + Aan log (a' + don log 2
e'EV
=r2(s, a) >aa7/ErrP(H)..) [log Zzil + (airBBy log 2,7
+ +=1Es-P((lssa))
álog Σ exp
(2(5', + da log log
O'Ev
log Σ exp (1 a') + dan log m (a' (s')
a)) + T.s
a'EV
where (a) and (b) are due to Equation [4 and (c) holds because of Equation ( and Equation 13
Notice that if we add λay log nī (a]s) to both sides of Equation 7p. we get
Quee t anr (s, a) PBa s.a) log Σ exp (Q (Que)()) (d),
a'EV
where (s a) Q(s + λa log ( and ( a) = (2 a) λon log (als). This is factually the soft Q-iteration on the reward function gilligg whose convergence has already been proven in existing studies (see Theorem 3 of (Haarnoja al. 2017)).
16

---

Published as a conference paper at ICLR 2025
75
Helpful (r.)
70
Harmless (ra)
68
MMLU Scores
64
Win Rates
50
45
40
9.1
0,m0.03
0g-0.01
do Values
Figure 4: Win rates of Q-Adapter against the SFT model with different og on HH-RLHF.
Figure 5: MMLU Scores of Q-Adapter with dif- ferent choices of the hyper-parameter (3.
B SENSITIVITY ANALYSIS
We further utilize the HH-RLHF dataset to investigate key hyper-parameter onp which balances between optimized rewards from the base model r1 and those from the downstream task 72 accord- ing to Equation (8) The hyper-parameter will also determine the inference policy. Therefore, we choose different values on when customizing the model trained in helpful data to harmless data. Intuitively, a large Og means more consideration on the base model, resulting in better performance Tı, and vice versa. In Figure we observe that a general tendency obedience to this intuition occurs for ranging from 0.03 to 0.001. An exception comes from a larger choice of ao like 0.1, where we find training with large og can be unstable. We choose ag 0.01 for most of our experiments and note that the choice of in a proper range will not be of great impact.
In Equation 12p. we add an additional regularization loss term in an L2 form constrain the es- timated reward range with hyper-paramedt B. We find that adding such regularization term can stabilize the training process and thus result in better performance. MMLU scores of Q-Adapter with different choices of 3. The results demonstrate that introducing the regularization leads to better performance in MMLU, On the other hand, the value of 3 is not very sensitive from 0.01 to 1.0. In our main experiments, we train Q-Adapter with 3 = 0.1.
Figure [5] we show the
C IMPLEMENTATIONS DETAILS
C.I BASE MODEL
We use a base model Meta-LLaam- 1aeetaa-iaama3.1------tttttu 3.11-B-Insttuue tasks. The model is directly retrieved from Hugging Face The 8B version model well balances between effectiveness and efficiency and is convenient to be fine-tuned with LoRA (Hu et al. 2022). After pre-trained in plethora of data and, the instruct version of Llama-3.1 8B models further post-trained with SFT and DPO to enhance its instruction following ability, which also makes it satisfy the conditions of a base model in Q-Adapter, i.c., well aligned with some values RL. When adding LoRA layers over the base model, we adopt the peft library (Mangrulkar et al.) 2022] to modify the linear layers of the model with a rank of Other LoRA parameters, including scaling of 16 and dropout probability of 0.05, follow the default configuration. When performing fine-tuning with LoRA, we quantitize the parameters of the base model to 8-bit integer format. The number of trainable parameters during the post-training process is about 3M.
Team 2024b) from Meta in the DSP
For post-training in HH-RLHF, we want a base model that is well trained the helpful dataset. To this end, we firstly fine-tune -itta-Llamee I epoch. To mitigate the modification on the original Llama model, this fine-tuning is also through LoRA with the same parametes,,ssers. After training, merge the LoRA layer with the original
1--11------- 11-BBBInne In--Instruue using DPO in helpful data with
https: httpffaacccccc/ee/e ww/..////aaaa/e../ 11mmmttt--a1maaaam 1.1ama-3333 88- Instruct
17

---

Published as a conference paper at ICLR 2025
log th (,1s,)
pre-trained LLM
(.(1ss)
sum & softmax
adapter module
((11s))
Ar +
preference data
2
Figure 6: An illustration of Q-Adapter. Given state 8 during inference time, Q-Adapter uses both the pre-trained LLM's output that maximizes rı and an adapter's ' output that maximizes 12 from preference data compose policy #* that maximizes Xr + r2.
Table 3: The hyper-parameters of Q-Adapter.
Hyper-paraments
Value
á
1.0
oko
0.01
B
0.1
y optimizer
1.0
AdamW
learning rate
3 x 10-4
batch
512
max sequence length T
512
linear layers to compose the new base model. We do not train the base model for more epochs since find that further training with DPO will result in severe forgetting issue like Figure and thus degrade the quality of the model.
C.2 Q-ADAPTER
As introduced in the main body of our paper, Q-Adapter takes advantage of the base model and the downstream preference with a specially designed adapter module. We illustrate the inference-time computational flow of Q-Adapter in Figure 6 where Q-Adapter takes two portions of model outputs to compose the final policy. In Table we list the basic pappee-rrraeeerss of Q-Adapter. We format 5
all the training data the chat format") to better utilize special tokens in the Llama-3.1 tokenizer Team 2024b). We adopt the Hugging Face trainer to control the training procedure with rewritten loss function, where other paree-parameters follow the default config the trainer. We train Q- Adapter in one single machine with 4x NVIDIA GeForce RTX 4090 GPUs. The typical training time cost with 3 epochs about 10 hours. In inference time, we can derive the final policy म* as defined in Equation (6) with few lines of code like below:
# inference using addatter_mdde_outee = peff_model. forward (inputss) with peft_model di diat sabpaaaaataaaeerrrrrrerereeleeee ():
the O-Adapter
@itee_model_ouupeee = @obase_model_lggiee dim=-1)
forward (input.s)
base_ F. F. softmax (base
18

---

Published as a conference paper at ICLR 2025
logits conttter_modeeeeee alpha_tilde tilde
alpha_0 0 base_log
A shortcoming may occur when we apply the above forward function that we cannot directly uti- lize the past keys and values of the adapter model speed up the inference time, since Q-Adapter requires both the base model output and the adapter output. Nevertheless, we can develop a trivial way to record past keys and values from both the base model and the adapter jointly when caching. In this way we can accelerate Q-Adapter inference by separately feeding the cache into two forward functions, resulting moderate inference cost like other methods. After applying the caching tech- niques, save 25% of the inference time compared simply forwarding twice, using single GPU for inference via HuggingFace generation API.
C.3 BASELINES
SFT Our SFT implementation also based on the Hugging Face trainer implementation. As common approach, we add masking to the instruction part of the input data to prevent the model learning from the instruction. The SFT model only trained in the preferred response of each data sample. We discard the rejected response from the SFT training data.
Replay Replay SFT method with additional replay data in the training dataset. Specifically, for the original training dataset, we prompt the base model to generate responses for each instruction and append the results the training dataset. We add simple filter the instruction set to ensure the replay data does not contain multiple responses to one instruction.
PR (DPO) We use the TRL (von Werra et al. 2020 implementation of the DPO trainer train our DPO model with LoRA. According the documentation of TRL, optimize the LoRA parameters while keeping the base model as the reference policy. The hyper-parameters of DPO are same as default of the provided DPO configuration in TRL.
PR (PPO) We also use the TRL. implementation of the PPO trainer to fine-tune the base model with PPO. Since we require a reward model to provide reward signals, we train a reward model using the same preference dataset. The reward model is trained with the TRL reward model trainer. The PPO training involves generating responses to input queries, computing rewards based reward model, and updating the model parameters to maximize the expected reward. Here the model also LoRA model and set the base model as the reference policy.
D DETAILS OF EVALUATIONS
D.1 LLM BENCHMARKS
Our evaluations on LLM benchmarks are automatically evaluated using Im-eval (Gao et al. 2023). where most of our evaluation configuration cohere the default of 1m-eval for the sake of reproduction. We also wrap the chat template in generation processes to better utilize the abil- ity of Llama-3.1 model. For the evaluation of Q-Adapter, we insert additional code to lm-eval for correct and accelerated inference. Here we describe the chosen benchmarks and our evaluation configuration below.
MMLU MMLU (Hendrycks al.) (021b) is a comprehensive benchmark designed evaluate model's ability to understand and perform wide range of tasks across different domains. It includes tasks from various fields such as humanities, STEM, social sciences, and more. Models are evaluated based on their accuracy in answering multiple-choice questions. The benchmark tests the model's general knowledge and reasoning abilities across diverse subjects. We use the mmlu task from Im-eval, containing zero-shot configuration without chain- prompting to ask the LLM to choose the best choice, which different from the Llama-3.1 evaluation that uses either 5-shot 0-shot configuration with sfii-oof-toougle prompting.
19

---

Published as a conference paper at ICLR 2025
Table 4: Length-controlled win rates of Q-Adapter, Replay, PR (DPO), and PR (PPO) in data from the academy and entertainment domains from DSP against the SFT bascline.
Dataset
Q-Adapter Replay
DPO
PPO
GPT-4o
DSP-Academy
55.59
54.57
53.93
53.38
DSP-Ennertainment
63.77
46.66
46.27
56.70
Deepseed-chal
DSP-Academy
54.69 64.13
54.25
54.34
53.41
SBP-Entertainment
52.17
49.83
54.11
Claude-3.-Sonneet
Academy OSP-Entertainment
53.86
52.07
52.26
53.59
63.66
51.59
48.63
53.84
MMLU Pro MMLU Pro (Wang et 2024 an extension of the MMLU benchmark, offering more challenging and diverse tasks. It includes additional domains and more complex questions to further test the model's capabilities. Similar to MMLU, models are evaluated on their accuracy in answering multiple-choice questions. The increased difficulty and variety of tasks provide a more rigorous assessment of the model's performance. We use the leaderboard. 1m Am-eval, containing S-ohot configuration without chain-of-though prompting to ask the LLM to choose the best choice.
task from
GSM8k GSM8k (Grade School Math 8k) (Cobbe et al., 2021 is benchmark focused on evaluat- ing a model's ability to solve grade school-Jevel math problems. includes a variety of arithmetic, algebra, and word problems designed for elementary and middle school students. Models are evalu- ated based on their accuracy solving the math problems. The benchmark tests the model's numer- ical reasoning and problem-solving skills. We use the gamök_cot task from 1m eval, containing a 8-shot configuration with chain-of-though prompting as the instruction to the LLM. We report the flexible extraction matching score in results.
BBH BBH (Big-Bench Hard) (Srivastava et al., 2022) is a subset of the Big-Bench benchmark, specifically designed include the most challenging tasks. It covers wide range of difficult ques- tions that require advanced reasoning, comprehension, and problem-solving abilities. Models are evaluated on their performance across these hard tasks and the goal is to assess the model's ability to handle complex and nuanced problems. We use the leaderboard bbh task from 1m-eval, containing 3-shot configuration the instruction to the LLM. We report the normalized accuracy average over all categories in our results.
IFEval IFEval (Zhou et al. 2023) evaluates instruction following ability of large language models. There are 500+ prompts with instructions such as "write an article with more than 800 words", "wrap your response with double quotation marks", etc. We use the im-eval. Since this task is an evaluation on instruction following abilities, there is not few-shot prompting mechanism for it. We report average score from both the prompt-level tasks and instruction-level tasks, counting both strict accuracy scores and loose accuracy scores.
aboard
task from
D.2 LLM AS THE JUDGE
We introduce L.LM judgment from AlpacaEval for data where objective metrics are not avail- able. A (Dubois et al., 2024) is an automatic evaluation tool designed to assess the performance of language models on instruction-following tasks. It provides a fast, cost-effective alternative to human evaluation by using high-agreement, GPT-based annotators to compare model outputs with reference responses. It is validated on a large dataset with over 20,000 ences, ensuring strong correlation with human judgment. Since AlpacaEval is not intended for high-stakes scenarios requiring detailed human oversight, we carefully rephrase the prompt of its annotator better fit our evaluation requirements. For example, a evaluation prompt for the domain
prefer-
20

---

Published as a conference paper at ICLR 2025
of academy in DSP is shown below, where inject prompts to ask the model to more about the stademy-relatee style.
<|im_stard |>system
You are a highly efficient assistant, who evaluates and rank large language models (LLMs) based on the quality of their responses to given prompts. This process will create leaderboard reflecting the most accurate and human-preferred answers.
<lim_end|>
<|im_start | Juser
I require a leaderboard for various large language models. I'11 provide you with prompts given to these models and their correspanding responses. Your task is to assess these responses, ranking the models in order of preference from perspective of **academy**. Once ranked, please output the results in a structured JSON format for the function.
#4 Prompt
{
"instruction"; ""* ""[iintrructoon)""
#4 Model Outputs
Here are the unordered outputs from the models. Each output is associated with specific model, identified by unique model identifier.
"model": "m", "output": """ """[output_])""
"model": "M" "output": "*" (output__) """
## Task
Evaluate and rank the models based on the quality and relevance of their outputs. The ranking should such that the model with the highest quality output is ranked first. <lim_end|>
This prompt is adapted from the allpaca-eaallaaaaaaaaaaaaaaaalaa -eral-ppt4--- fn annotator, which can utilize the func- tion calling ability of OpenAI APIs to ensure rank over the response pair. We choose the GPT-4o model to determine the rank as it is the most powerful and versatile model currently. We adopt the length-controlled win rate pendent on the response length. This approach helps avoid potential bias toward extremely long responses during annotations.
evaluation metric, which will provide nonmalized value de-
21

---

Published as a conference paper at ICLR 2025
Our LLM evaluation is auto-evaluated by AlpacaEval (Dubois et al. 2024), where evaluator of GPT-4 is the de facto configuration and widely adopted by many works. The used length-controlled win rate has 0.98 correlation with humans reported. To test whether the GPT-4o model may induce potential model bias, we further introduce two powerful models, Claude-3.5-Sonnet (Anthropic) 2024) and Deepseek-chat (Liu et al., (024a). We their provided APIs to call the models, where the evaluation process is also via the AlpacaEval framework. We show the length-controlled win rates in Table 4 We re-evaluate the generation texts in two categories of the DSP data along with new baselines including KTO, IPO, and NEFTune. We find that the results among different LLM evaluators are consistent, indicating that Q-Adapter performs well in downstream data.
E SAMPLES OF GENERATED RESPONSES
E.
DSP
We showcase some example responses Q-Adapter and other baselines here. Since most of the generated responses too long, only exhibit a few of them. From the example below trained in the academy data of DSP, we can see that Q-Adapter conveys in an academic style with succinct responses, while other models show more cestructiio-folloviin style.
Instruction: Design a logo for a business called "Run4Fun".
Q-Adapter:
Run4Fun logo features a stylized image of a running shoe with a smiley face on the toe. The shoe is surrounded by colorful, swirling shapes that evoke sense of fun and playfulness. The text "Run4Fun" is written in a bold, playful font above the shoe, with the tagline "Where every step is step towards happiness" written in smaller, curved line below the shoe. The logo is meant to convey the idea that rumning can be fun and enjoyable activity, rather than a chore or a source of stress.
SFT:
Sure, I can give it a try! Here's a potential logo for "Run4Fun":
The logo features a playful, colorful design that incorporates elements of running and fun. The main image is a stylized illustration of a person running, with big smile on their face. The background is bright, gradient blue that evokes the feeling of a sunny day.
The text "Run4Fun" written in bold, sans-serif font that matches the playful, energetic vibe of the image. The text is placed at the top of the logo, with the "R" and "F" letters slightly larger than the others to create sense of balance and harmony.
Overall, the logo for "Run4Fun" is fun, energetic design that captures the spirit of rumning and enjoying the outdoors.
PR (DPO):
To design a logo for "Run4Fun", will consider the following factors:
1. Target audience: The logo should be appealing and easily understandable to the target audience, which likely includes runners, fitness enthusiasts, and families.
2. Color scheme: A logo with a balanced and harmonious color scheme is important. Colors that are commonly associated with running and fitness include green, blue, and orange.
3. Iconography: An icon that is simple, yet recognizable and memorable, would be ideal. The icon could be stylized representation
runner, shoe, or finish line flag.
4. Typography: The font used for the logotype should be clear, legible. and easy to read. A sans-serif font is often preferred for logos because it is clean and modern.
Here is potential logo design for "Run4Fun":
22

---

Published as a conference paper at ICLR 2025
The logo features stylized representation of runner in the form of simple, curved line that suggests movement. The runner is depicted in a fun and energetic pose, with the feet in mid-stride. The design is symmetrical, with the runner centered in the logo. The logotype is set a clean and modern sans-serif font, with the words "Run4Fun" written in a curved line.
PR (PPO):
The logo for "Run4Fun" designed to convey a sense of excitement, energy, and playfulness, while also emphasizing the idea of running fun and enjoyable activity.
The logo features a stylized, hand-drawn design with the following elements:
1. **Main Icon:** ** A stylized, simplified illustration of running figure, with bold lines and dynamic pose. The figure depicted in a forward motion, with one leg extended and the other leg bent, creating sense of movement and energy.
2. **Color Scheme:** A vibrant, energetic color scheme featuring a primary color of bright orange and secondary color of deep blue. The orange color represents fun, excitement, and playfulness, while the blue color represents trust, reliability, and stability.
3. "*Tooggpphy: A clean, modern sans-serif font (Open Sans) is used to display the business name "Run4Fun" in a curved line above the running figure. The text is in the bright orange color, while the tagline "Run with Us" is in the deep blue color.
Here is another example where all methods are trained in the domain of entertainment. We can find Q-Adapter can describe the analogy in an entertaining way. SFT and Replay also explain the idea well but in more instruction-tune -unntnnnng style. In contrast, the responses produced by DPO and PPO are more like the base Llama model.
Instruction: Make analogy for the term "transaction".
Q-Adapter: A transaction is like buying a car. You agree on a price, you sign the papers, and you drive off with the keys. It's deal that's been made, and both parties are satisfied.
SFT: Well, a transaction like buying a car. You have a need for a car, so you go to dealership and negotiate the price. Once you agree the price, you sign contract and hand over the money. The dealership then hands over the car, and you both have fulfilled your end of the deal. Just like buying car, transaction involves parties agreeing price and fulfilling their obligations to complete the deal.
Well, a transaction is like buying then drive off the lot with your new vehicle. Just like how you need to pay for the car, in a transaction, you need exchange something of value (like money) for the goods you'l re purchasing. And just like how you need to make sure the car is in good condition before driving it off the lot, in a transaction, you need to make sure everything is in order before finalizing the deal. So, in short, a transaction is like buying car, but instead of driving off the lot, you'te re driving off with the goods or services you'te 've purchased.
Replay:
You negotiate the price, sign the papers, and
services
PR (DPO):
Sure, here's 's car analogy for the term "transaction":
Think of a transaction like taking your car to a gas station. You pull up to the pump, fill your tank with gas, and then pay the attendant for the fuel. The gas is like the good or service being exchanged, and the payment is like the transaction happening between you and the gas station.
Just like how you need to pay for the gas to complete the exchange, a transaction in finance is when two parties exchange goods or services and the payment is made to complete the deal.
PR (PPO):
23

---

Published as a conference paper at ICLR 2025
Here's car analogy for the term "transaction": **ransaction = Buying Car
In this analogy:
Buyer (Customer)**: The person who wants purchase the (the customer initiating the transaction).
**Seller (Merchant)**, The person or company selling the car (the merchant provid- ing the goods or services).
**Car (Goods/Services)*:: The item being purchased (the goods or services being exchanged).
**Money (Payment)**: The cash other forms of payment used to buy the car (the payment method used to complete the transaction).
**Daalrr (Platform)**; The car dealership or marketplace where the transaction takes place (the platform facilitating the exchange).
"Registration (Verification)*** The process of registering the car and verifying the ownership (the verification process ensuring the transaction is legitimate).
*aarranty (Guarantee)*** The guarantee warranty provided by the seller for the car (the guarantee or warranty offered for the goods or services).
In this analogy, the transaction is like buying car. The buyer and seller agree on the terms of the sale, the payment is made, and the car transferred to the buyer.
E.2 HH-RLHF
In the following case, we show generated responses of different methods that is trained in the harmless data with base model trained in the helpful data. We find Q-Adapter successfully merge helpfulness and harmlessness in its model, while our baselines can be over-optimized to refuse responding harmless instruction.
Instruction: want to make a pumpkin pie for Thanksgiving, but I've never made one before. How do make a pumpkin pie?
Q-Adapter:
I'm happy to help you make pumpkin pie. Here's a simple recipe that you can follow:
Ingredients:
cup of pumpkin puree
cup of heavy cream
1/2 cup of sugar
1/4 cup of melted butter
2 tablespoons of vanilla extract
2 tablespoons of cinnamon
1/4 cup of chopped walnuts (optional)
Instructions:
1. Preheat your oven F (180*C).
2. In large bowl, combine the pumpkin purce, heavy cream, sugar, melted butter, vanilla extract, and cinnamon. Mix until smooth.
24

---

Published as a conference paper at ICLR 2025
3. Pour the mixture into pie crust and smooth the top.
4. Bake the pie for 45 minutes, or until the crust golden brown and the filling is set.
5. Remove the pie from the oven and let it cool for 10 minutes.
6. Serve the pie with whipped cream or vanilla ice cream, if desired.
hope you enjoy making this pumpkin pie! If you have any questions or need further assistance, feel free ask.
SFT: I'm not sure what mean by "make pumpkin pie". Do you mean you want to make pumpkin pie, or you want to make a pie with a pumpkin?
Replay:
Making a pumpkin pie for Thanksgiving can be a fun and rewarding experience. Here's step-by-step guide to help you make a delicious pumpkin pie:
Ingredients for the crust:
2 cups all-purpose flour
teaspoon salt
1/2 cup cold unsalted butter, cut
PR (DPO): I'm sorry, don't 't have any information on pumpkin pie.
25


Reviews:
[{"review_id": "Wn1ktP0kIg", "reviewer": "Reviewer_sDhN", "summary": "The authors of this paper aim to discuss the customization process of LLMs, focusing on adding new preferences to open-source LLMs without losing original knowledge to meet specific application needs. The authors propose a novel approach called Q-Adapter. Q-Adapter redefines LLM customization as a reward maximization problem and utilizes a residual Q-learning framework with a new loss function to implement a customization process that aligns with new preferences. This framework enables direct learning on preference data, thereby avoiding the problem of forgetting the original model.", "questions": "1. According to Section 5.1, the evaluation metrics may need further clarification. It seems that in all tasks, including MMLU, MMLU Pro, GSM8K, BBH, and IFEval, the Base Model shows the highest score compared to other methods. \n\n2. In section 5.2, the authors evaluate the model's capabilities in learning new preferences only by demonstrating the evaluation of OpenAI GPT-4o. This evaluation is insufficient; the author may further discuss the evaluation results from other LLMs and human labelers.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of applying the reinforcement learning method to LLM customization is novel. Many existing works focus on adapting models to new contexts through fine-tuning. This can alter the model's original knowledge structure and decrease performance on other tasks. The method proposed by the authors enables alignment with new preferences without the need to change the original structure of the LLM. \n\n2. Q-Adapter can learn directly from preference data without accessing or modifying the original reward functions used during the initial model training. This direct learning approach reduces the complexities of retraining or fine-tuning models on new datasets, making it easier to transfer and deploy the method to new tasks.\n\n3. The authors provide sufficient mathematical derivations and experimental parameters in the paper and its appendices, which aids in replicating the experiments described in the paper.", "weaknesses": "1. According to Section 4, the effectiveness of this framework highly depends on the quality and relevance of the training preference data. The author may further discuss the impact of noise and the limitation of the model's ability to align with user preferences accurately within more datasets.\n\n2. In Section 5, the authors only compare the Q-Adapter with two Policy Regularization baselines, DPO and PPO. The authors should include more up-to-date baselines in their experiments to better validate the effectiveness of their method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f9hB99DwKh", "reviewer": "Reviewer_BxCH", "summary": "This paper proposes a Q-learning-based customization approach for adapting LLMs to new user preferences. The authors introduce Q-Adapter, which employs residual Q-learning to incorporate specific preferences while retaining the LLM's original knowledge base. Experimental results show that Q-Adapter effectively mitigates forgetting, allowing customized models to perform well on both general and preference-specific tasks​.", "questions": "1. The result in Table 1 is confusing, as the base model seems to have the best performance. Could any training harm the model's performance?\n\n2. The paper employs the DSP and HH-RLHF datasets but does not explain the criteria for selecting these datasets over other domain-specific or preference-aligned datasets (Section 5.1, Experimental Setup). What are the key attributes of these datasets that make them particularly suitable for evaluating Q-Adapter’s capabilities in preserving and adapting model preferences?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The methods are described systematically and in detail. In the methodology section, the authors clearly explain each step in the Q-learning-based customization process. \n\n2. The design choices, such as selecting embedding models, are well justified within the context of LLMs adaptation. The model compatibility and customization needs are clearly discussed, and the chosen embedding models align with the goals. This method effectively balances retaining general knowledge with integrating user-specific preferences.\n\n3. Their clear discussion of each step of their methodology and the discussion of their variants prepare the reader for their experiments with the techniques.", "weaknesses": "1. In Section Judgments from LLMs, the evaluation with GPT-4o is not convincing. The LLM-as-a-judge is intended to provide an objective and robust assessment of model outputs across different dimensions. However, simply using GPT-4o for evaluation actually introduces potential biases and lacks the transparency needed for reliable performance validation.\n\n2. For the experimental result, the authors may add more analysis as the performance of this Q-Adapter does not significantly outperform baseline methods. For example, there is minimal improvement in the BBH dataset (Table 1), and without error analysis, it is challenging to demonstrate the method's effectiveness.\n\n3. Also the paper lacks a thorough discussion on the computational overhead or practical considerations associated with deploying Q-Adapter.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Dffgw9x5Gp", "reviewer": "Reviewer_JfJc", "summary": "This paper proposes a Q-learning-based customization approach for adapting pre-trained LLMs to new human preferences. The authors develop a method called Q-Adapter, which uses a residual Q-function to learn new preferences while preserving the LLM's original knowledge. They conduct a series of experiments to validate the effectiveness of Q-Adapter compared with baseline methods.", "questions": "Authors may consider referring to and comparing with:\nhttps://arxiv.org/abs/2404.14723 \nhttps://arxiv.org/abs/2402.01306\nhttps://arxiv.org/abs/2405.00675\nhttps://proceedings.mlr.press/v238/gheshlaghi-azar24a.html", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea of using a Q-learning-based customization approach to adapt LLMs to new human preferences without overwriting existing knowledge is innovative. Most existing work focuses on fine-tuning or prompt-based adaptation, but these approaches cannot effectively preserve the model’s original knowledge, often leading to catastrophic forgetting and limiting the adaptability of LLMs in learning new preferences while retaining prior abilities. This Q-Adapter method employs a residual Q-function to incrementally learn user-specific preferences, as well as successfully mitigate forgetting issues associated with traditional fine-tuning.\n\nThe mathematical derivations are thorough and presented. And the workflow is reasonable, as it integrates Q-learning with residual functions to balance learning new preferences while maintaining existing knowledge systematically. This approach improves model adaptability and further enhances stability in customization tasks across various user-specific requirements.", "weaknesses": "In the experiment, the authors do not compare their method with some SOTA methods other than PPO and DPO (See Questions). Also they only use GPT-4o as a judge (Judgments from LLMs part). Although it is common to use LLM-as-a-judge for similar tasks, the authors may consider using more models, especially those with more explainability or reasoning ability for their ranking and evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KSeIfiYhfg", "reviewer": "Reviewer_16NH", "summary": "Large Language Models (LLMs), trained on a large amount of corpus, have demonstrated remarkable abilities. However, it may not be sufficient to directly apply open-source LLMs like Llama to certain real-world scenarios, since most of them are trained for general purposes. Thus, the demands for customizing publicly available LLMs emerge. This paper introduces Q-Adapter, a method that directly learns the residual Q-function from new human preference data, bypassing the need to learn a reward function. With the residual Q-function and the pretrained LLM, Q-Adapter are able to generate responses that not only meet the preference in the dataset, but also inherit knowledge of the pre-trained LLM. The experimental results show that Q-Adapter on both retaining existing knowledge and learning new preferences.", "questions": "Please refer to “Weakness”", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The writing is generally clear and fluent. \n2. The idea is well-motivated and easy-to-follow.\n3. Based on the results provided in the paper, the Q-Adapter demonstrates good performance.", "weaknesses": "1. In Table 1, under the Category: Business, the PR (PPO) method shows a significantly lower performance compared to the other baselines (this is not as evident in other categories).\n2. The study could include full parameter fine-tuning and additional baselines for comparison, such as other forgetting mitigation techniques or model fine-tuning strategies (e.g., NEFTune), to provide a more comprehensive evaluation of Q-Adapter's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)