Q-Adapter: Customizing Pre-trained LLMs to New Preferences with Forgetting Mitigation

Authors: Yi-Chen Li, Fuxiang Zhang, Wenjie Qiu, Lei Yuan, Chengxing Jia, Zongzhang Zhang, Yang Yu, Bo An
Primary area: reinforcement learning
Keywords: reinforcement learning from human feedback, large languge model customization, anti-forgetting

Abstract:
Large Language Models (LLMs), trained on a large amount of corpus, have demonstrated remarkable abilities. However, it may not be sufficient to directly apply open-source LLMs like Llama to certain real-world scenarios, since most of them are trained for \emph{general} purposes. Thus, the demands for customizing publicly available LLMs emerge, but are currently under-studied. In this work, we consider customizing pre-trained LLMs with new human preferences. Specifically, the LLM should not only meet the new preference but also preserve its original capabilities after customization. Drawing inspiration from the observation that human preference can be expressed as a reward model, we propose to cast LLM customization as optimizing the sum of two reward functions, one of which (denoted as $r_1$) was used to pre-train the LLM while the other (denoted as $r_2$) characterizes the new human preference. The obstacle here is that both reward functions are unknown, making the application of modern reinforcement learning methods infeasible. Thanks to the residual Q-learning framework, we can restore the customized LLM with the pre-trained LLM and the \emph{residual Q-function} without the reward function $r_1$. Moreover, we find that for a fixed pre-trained LLM, the reward function $r_2$ can be derived from the residual Q-function, enabling us to directly learn the residual Q-function from the new human preference data upon the Bradley-Terry model. We name our method Q-Adapter as it introduces an adapter module to approximate the residual Q-function for customizing the pre-trained LLM towards the new preference. Experiments based on the Llama-3.1 model on the DSP dataset and HH-RLHF dataset illustrate the superior effectiveness of Q-Adapter on both retaining existing knowledge and learning new preferences. Our code is available at \url{https://github.com/LAMDA-RL/Q-Adapter}.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730423903780

thread_type:
official_review

<thread_object>
creator: reviewer_16nh

<content>
summary:
Large Language Models (LLMs), trained on a large amount of corpus, have demonstrated remarkable abilities. However, it may not be sufficient to directly apply open-source LLMs like Llama to certain real-world scenarios, since most of them are trained for general purposes. Thus, the demands for customizing publicly available LLMs emerge. This paper introduces Q-Adapter, a method that directly learns the residual Q-function from new human preference data, bypassing the need to learn a reward function. With the residual Q-function and the pretrained LLM, Q-Adapter are able to generate responses that not only meet the preference in the dataset, but also inherit knowledge of the pre-trained LLM. The experimental results show that Q-Adapter on both retaining existing knowledge and learning new preferences.

strengths:
1. The writing is generally clear and fluent. 
2. The idea is well-motivated and easy-to-follow.
3. Based on the results provided in the paper, the Q-Adapter demonstrates good performance.

weaknesses:
1. In Table 1, under the Category: Business, the PR (PPO) method shows a significantly lower performance compared to the other baselines (this is not as evident in other categories).
2. The study could include full parameter fine-tuning and additional baselines for comparison, such as other forgetting mitigation techniques or model fine-tuning strategies (e.g., NEFTune), to provide a more comprehensive evaluation of Q-Adapter's performance.

questions:
Please refer to “Weakness”
</content>
</thread_object>

</thread>
<thread>
creation date:
1730583263217

thread_type:
official_review

<thread_object>
creator: reviewer_jfjc

<content>
summary:
This paper proposes a Q-learning-based customization approach for adapting pre-trained LLMs to new human preferences. The authors develop a method called Q-Adapter, which uses a residual Q-function to learn new preferences while preserving the LLM's original knowledge. They conduct a series of experiments to validate the effectiveness of Q-Adapter compared with baseline methods.

strengths:
The idea of using a Q-learning-based customization approach to adapt LLMs to new human preferences without overwriting existing knowledge is innovative. Most existing work focuses on fine-tuning or prompt-based adaptation, but these approaches cannot effectively preserve the model’s original knowledge, often leading to catastrophic forgetting and limiting the adaptability of LLMs in learning new preferences while retaining prior abilities. This Q-Adapter method employs a residual Q-function to incrementally learn user-specific preferences, as well as successfully mitigate forgetting issues associated with traditional fine-tuning.

The mathematical derivations are thorough and presented. And the workflow is reasonable, as it integrates Q-learning with residual functions to balance learning new preferences while maintaining existing knowledge systematically. This approach improves model adaptability and further enhances stability in customization tasks across various user-specific requirements.

weaknesses:
In the experiment, the authors do not compare their method with some SOTA methods other than PPO and DPO (See Questions). Also they only use GPT-4o as a judge (Judgments from LLMs part). Although it is common to use LLM-as-a-judge for similar tasks, the authors may consider using more models, especially those with more explainability or reasoning ability for their ranking and evaluation.

questions:
Authors may consider referring to and comparing with:
https://arxiv.org/abs/2404.14723 
https://arxiv.org/abs/2402.01306
https://arxiv.org/abs/2405.00675
https://proceedings.mlr.press/v238/gheshlaghi-azar24a.html
</content>
</thread_object>

</thread>
<thread>
creation date:
1730700751904

thread_type:
official_review

<thread_object>
creator: reviewer_bxch

<content>
summary:
This paper proposes a Q-learning-based customization approach for adapting LLMs to new user preferences. The authors introduce Q-Adapter, which employs residual Q-learning to incorporate specific preferences while retaining the LLM's original knowledge base. Experimental results show that Q-Adapter effectively mitigates forgetting, allowing customized models to perform well on both general and preference-specific tasks​.

strengths:
1. The methods are described systematically and in detail. In the methodology section, the authors clearly explain each step in the Q-learning-based customization process. 

2. The design choices, such as selecting embedding models, are well justified within the context of LLMs adaptation. The model compatibility and customization needs are clearly discussed, and the chosen embedding models align with the goals. This method effectively balances retaining general knowledge with integrating user-specific preferences.

3. Their clear discussion of each step of their methodology and the discussion of their variants prepare the reader for their experiments with the techniques.

weaknesses:
1. In Section Judgments from LLMs, the evaluation with GPT-4o is not convincing. The LLM-as-a-judge is intended to provide an objective and robust assessment of model outputs across different dimensions. However, simply using GPT-4o for evaluation actually introduces potential biases and lacks the transparency needed for reliable performance validation.

2. For the experimental result, the authors may add more analysis as the performance of this Q-Adapter does not significantly outperform baseline methods. For example, there is minimal improvement in the BBH dataset (Table 1), and without error analysis, it is challenging to demonstrate the method's effectiveness.

3. Also the paper lacks a thorough discussion on the computational overhead or practical considerations associated with deploying Q-Adapter.

questions:
1. The result in Table 1 is confusing, as the base model seems to have the best performance. Could any training harm the model's performance?

2. The paper employs the DSP and HH-RLHF datasets but does not explain the criteria for selecting these datasets over other domain-specific or preference-aligned datasets (Section 5.1, Experimental Setup). What are the key attributes of these datasets that make them particularly suitable for evaluating Q-Adapter’s capabilities in preserving and adapting model preferences?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730704155814

thread_type:
official_review

<thread_object>
creator: reviewer_sdhn

<content>
summary:
The authors of this paper aim to discuss the customization process of LLMs, focusing on adding new preferences to open-source LLMs without losing original knowledge to meet specific application needs. The authors propose a novel approach called Q-Adapter. Q-Adapter redefines LLM customization as a reward maximization problem and utilizes a residual Q-learning framework with a new loss function to implement a customization process that aligns with new preferences. This framework enables direct learning on preference data, thereby avoiding the problem of forgetting the original model.

strengths:
1. The idea of applying the reinforcement learning method to LLM customization is novel. Many existing works focus on adapting models to new contexts through fine-tuning. This can alter the model's original knowledge structure and decrease performance on other tasks. The method proposed by the authors enables alignment with new preferences without the need to change the original structure of the LLM. 

2. Q-Adapter can learn directly from preference data without accessing or modifying the original reward functions used during the initial model training. This direct learning approach reduces the complexities of retraining or fine-tuning models on new datasets, making it easier to transfer and deploy the method to new tasks.

3. The authors provide sufficient mathematical derivations and experimental parameters in the paper and its appendices, which aids in replicating the experiments described in the paper.

weaknesses:
1. According to Section 4, the effectiveness of this framework highly depends on the quality and relevance of the training preference data. The author may further discuss the impact of noise and the limitation of the model's ability to align with user preferences accurately within more datasets.

2. In Section 5, the authors only compare the Q-Adapter with two Policy Regularization baselines, DPO and PPO. The authors should include more up-to-date baselines in their experiments to better validate the effectiveness of their method.

questions:
1. According to Section 5.1, the evaluation metrics may need further clarification. It seems that in all tasks, including MMLU, MMLU Pro, GSM8K, BBH, and IFEval, the Base Model shows the highest score compared to other methods. 

2. In section 5.2, the authors evaluate the model's capabilities in learning new preferences only by demonstrating the evaluation of OpenAI GPT-4o. This evaluation is insufficient; the author may further discuss the evaluation results from other LLMs and human labelers.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732162874690

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Official Comment By Authors

comment:
We sincerely thank all the reviewers for your time and efforts in reviewing our paper. In this global response, we would like to first summarize our revisions (highlighted in `magenta` color in the revised PDF). Clarifications to several concerns commonly raised by reviewers are then followed. 

## Revisions

+ Add explanations on selecting DSP and HH-RLHF for experiments (Section 5.1).
+ Add analysis on why the Base Model performs the best in Table 1 (Section 5.2).
+ Add evaluations from Claude-3.5-Sonnet and Deepseek-Chat (Section 5.2 and Appendix C).
+ Add a concise discussion about the impact of noise on model performance (Section 7).
+ Add details on the computation requirements for deploying Q-Adapter (Appendix B.2).

## Global Responses

> **global_Q1**: Comparison with More Baselines (W2 of Reviewer sDhN, W1 of Reviewer BxCH, Q1 of Reviewer JfJc, W2 of Reviewer 16NH)

We have added more baselines including IPO, KTO, and NEFTune. IPO and KTO are two methods that adopt policy regularization to mitigate forgetting similar to DPO, while NEFTune is an enhanced SFT approach. We show the anti-forgetting capabilities of new baselines and Q-Adapter in the following table. The results demonstrate that Q-Adapter maintains the best performance among all methods. We find KTO generally shows similar performance to previous policy regularization baselines like DPO, and NEFTune can achieve slightly better performance than SFT. We also evaluate the performance of these additional baselines in downstream data through LLM judgments and put the results in the next question.

| Category | Method | MMLU | MMLU Pro | GSM8k | BBH | IFEval |
| --- | --- | --- | --- | --- | --- | --- |
| DSP-Academy | Q-Adapter | **66.40** | **36.37** | **77.86** | **47.34** | **68.14** |
|  | IPO | 62.55 | 34.46 | 73.39 | 45.13 | 50.19 |
|  | KTO | 65.31 | 33.27 | 74.33 | 44.56 | 66.60 |
|  | NEFTune | 64.67 | 34.29 | 74.68 | 44.87 | 60.95 |
| DSP-Entertainment | Q-Adapter | **66.69** | **35.98** | **77.94** | **46.33** | **69.15** |
|  | IPO | 63.09 | 32.65 | 72.55 | 40.04 | 55.71 |
|  | KTO | 65.00 | 33.92 | 75.66 | 45.15 | 65.44 |
|  | NEFTune | 63.77 | 31.19 | 74.00 | 44.63 | 58.50 |

> **global_Q2**: Evaluation with Other LLMs (Q2 of Reviewer sDhN, W1 from Reviewer BxCH, W1 of Reviewer JfJc)

Our LLM evaluation is auto-evaluated by AlpacaEval, where an evaluator of GPT-4 is the default configuration and wildly adopted by many works. The used length controlled win rate has 0.98 correlation with humans as reported by [1]. As the reviewers proposed to use other models for evaluation, we further introduce two LLMs, Claude-3.5-Sonnet and Deepseek-chat. We use their provided APIs to call the models, where the evaluation process is also via the AlpacaEval framework. We show the length-controlled win rates as follows.

| GPT-4o | Q-Adapter | Replay | DPO | PPO | IPO | KTO | NEFTune |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DSP-Academy | **55.59** | 54.57 | 53.93 | 53.38 | 51.64 | 55.07 | 52.11 |
| DSP-Entertainment | **63.77** | 46.66 | 46.27 | 56.7 | 49.37 | 50.27 | 55.75 |

| Claude-3.5-Sonnet | Q-Adapter | Replay | DPO | PPO | IPO | KTO | NEFTune |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DSP-Academy | **53.86** | 52.07 | 52.26 | 53.59 | 52.75 | 53.65 | 52.79 |
| DSP-Entertainment | **63.66** | 51.59 | 48.63 | 53.84 | 51.58 | 48.07 | 54.54 |

| Deepseed-chat | Q-Adapter | Replay | DPO | PPO | IPO | KTO | NEFTune |
| --- | --- | --- | --- | --- | --- | --- | --- |
| DSP-Academy | **54.69** | 54.25 | 54.34 | 53.41 | 54.9 | 54.85 | 53.95 |
| DSP-Entertainment | **64.13** | 52.17 | 49.83 | 54.11 | 50.02 | 44.61 | 53.02 |

We re-evaluate the generation texts in two categories of the DSP data along with new baselines including KTO, IPO, and NEFTune. We find that the results among different LLM evaluators are consistent, indicating that Q-Adapter performs well in downstream data.

[1] Dubois, Y., Galambosi, B., Liang, P., & Hashimoto, T. B. (2024). Length-controlled AlpacaEval: A simple way to debias automatic evaluators. *arXiv preprint arXiv:2404.04475*.

> **global_Q3**: Why does the base model perform the best in Table 1? (Q1 of Reviewer sDhN, Q1 of Reviewer BxCH)

It is expected when we customize LLMs in downstream data that has large discrepancies with these general benchmark. As shown in Figure 2, the forgetting issue clearly occurs in general benchmarks like MMLU, similar to previous findings [1]. Therefore, the major motivation of Q-Adapter is to investigate how to mitigate such forgetting. In general, the average decrease in benchmark scores of Q-Adapter is 30.0% less than the best baseline (Replay) and 43.2% less than the average performance of all baselines in Table 1, indicating the anti-forgetting capability of Q-Adapter during customization.

[1] Lin, Y., Lin, H., Xiong, W., Diao, S., Liu, J., Zhang, J., ... & Zhang, T. (2024). Mitigating the alignment tax of RLHF. In EMNLP.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732531839850

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Kindly Request for Feedback

comment:
Hi, reviewers! As the discussion period comes to an end, we want to follow up to see if the response addresses your concerns. If you have any further questions, please let us know. Thank you again!
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: This paper introduces Q-Adapter, a novel Q-learning-based approach for customizing Large Language Models (LLMs) to align with new user preferences without overwriting their original knowledge. The method employs a residual Q-function to learn from preference data, enhancing model adaptability and stability across various user-specific requirements. The experimental results demonstrate Q-Adapter's effectiveness in retaining existing knowledge while learning new preferences.

Strengths:

Q-Adapter innovatively applies reinforcement learning to LLM customization, allowing for the integration of new preferences without compromising the model's original knowledge base.

The method provides a systematic balance between retaining general knowledge and adapting to specific user preferences, improving model versatility.

All the reviewers give the positive feedbacks， thus I recommend the acceptance of this work.