HyperDPO: Hypernetwork-based Multi-Objective Fine-Tuning Framework

Authors: Yinuo Ren, Tesi Xiao, Michael Shavlovsky, Lexing Ying, Holakou Rahmanian
Primary area: optimization
Keywords: Direct Preference Optimization, Multi-Objective Optimization, Hypernetwork, Alignment

Abstract:
In LLM alignment and many other ML applications, one often faces the *Multi-Objective Fine-Tuning (MOFT)* problem, *i.e.* fine-tuning an existing model with datasets labeled w.r.t. different objectives simultaneously. To address the challenge, we propose the *HyperDPO* framework, a conditioned one-shot fine-tuning approach that extends the Direct Preference Optimization (DPO) technique, originally developed for efficient LLM alignment with preference data, to accommodate the MOFT settings. By substituting the Bradley-Terry-Luce model in DPO with the Plackett-Luce model, our framework is capable of handling a wide range of MOFT tasks that involve listwise ranking datasets. Compared with previous approaches, HyperDPO enjoys an efficient one-shot training process for profiling the Pareto front of auxiliary objectives, and offers post-training control over trade-offs. Additionally, we propose a novel *Hyper Prompt Tuning* design, that conveys continuous importance weight across objectives to transformer-based models without altering their architecture, and investigate the potential of *temperature-conditioned networks* for enhancing the flexibility of post-training control. We demonstrate the effectiveness and efficiency of the HyperDPO framework through its applications to various tasks, including Learning-to-Rank (LTR) and LLM alignment, highlighting its viability for large-scale ML deployments.

Decision: reject
Reviewer scores: 5, 6, 8, 3

Reviews and discussion:
<thread>
creation date:
1730289510744

thread_type:
official_review

<thread_object>
creator: reviewer_f9mp

<content>
summary:
The submission describes a system for multi-objective model finetuning, where a base model is trained on several new objectives from listwise ranking data. The main component is an input-conditioned network that takes additional inputs weights for the different objectives, and it trained with a random mixture over these. At prediction time, the weights can be set to achieve different trade-offs between objectives. Experiments for learning-to-rank and LLM-alignment show improved performance compared to baseline approaches. The supplemental material provides further (ablation) studies as well as an additional formulation in which not only the objective weights but also the temperature of their softmax can be adjusted at prediction time.

strengths:
- timely and relevant topic

The submission addresses the largely unexplored problem of how LLMs can be fine-tuned in a multi-objective way, i.e. a network should be adapted with respect to not just one but multiple, potentially contradictory, targets and the trade-off between should be adjustable at prediction time. This matches well to real-world tasks, where LLMs should perform well in different metrics, e.g. utility, safety and fairness, and one cannot afford to train individual models for any desired trade-off. 

- theoretical derivations as well as practical experiments

The submission presents mathematical derivations of their main components as well as experimental evaluations and ablation studies (but see below for shortcomings in presentation). 

- experiments show improvement over baselines

The experimental evaluation shows that the proposed method tends to find Pareto-better solutions than baselines, and that it also takes less time for training (but again, see below).

weaknesses:
While overall I believe the submission has merits, I found its presentation to leave several aspects unclear or even misleading. 

1) the manuscript uses the term *hypernetwork* throughout the text, but the system it describes is not actually hypernetwork-based. A hypernetwork is a network that outputs the parameters of another network [Ha et al, "Hypernetworks", ICLR 2017], see also e.g. the recent survey [Chauhan et al, "A brief review of hypernetworks in deep learning", AI Review 2024]. In contrast, the system described in the submissions uses an input-conditioned network as in [Ruchte & Grabocka. "Scalable Pareto front approximation for deep multi-objective learning", ICDM 2021]. These two approaches are simply different, in their theoretical properties (see e.g. [Galanti & Wolf, "On the Modularity of Hypernetworks", NeurIPS 2020]) as well as in practice.

2) From the writing I was not able to extract the core technical contribution. Section 1.1 list the HyperDPO system as the main contribution, but that consists of several parts which existed (at least in similar form) previously. Does the main contribution lie in the differences? Or in the combination of techniques? Or in their application to the finetuning task? 

To make my concern clearer: I believe for the reader it is most helpful if the "Preliminaries" section bundles prior work, and the "Methodology" section introduces the new scientific contribution on top of it. In the manuscript, the Methodology section indeed it introduces the multi-objective finetuning task, which I assume is a new formulation that can be seen as a generalization of standard LLM finetuning (thought as a special case of general multiobjective learing). But the Section discusses the classical Plackett-Luce model, and rederives the ListNet loss from [Cao et al, 2007]. Next, input-conditioned multi-objective learning [Ruchte & Grabocka, 2021] is introduced. Inserting the loss into the formulation and ultimately arrives at HyperDPO. It would be good to mark more clearly, which of the intermediate steps the authors consider scientific contributions and which are just provided for the convenience of the reader. The post-training control (via Proposition 3.1) appears new, but its relevance is not made clear here, and I have questions about the proof. 

Other contribution mentioned in Section 1.1 are the Hyper Prompt Tuning, which however is first described in the experimental Section 4.2 at much lower level of detail than I would expect from a core contribution. The final contribution of a "temperature hypernetwork" is even only discussed in the supplemental material. 

3) The experiments are only partially convincing. 

The learning-to-rank experiments of Section 4.1 show promising results, but the settings is not the one used for motivating the method. The setup of the LLM alignment task is described rather briefly, e.g. the description of dataset, objectives and optimization process are too short to allow for reproducibility. The visualized Pareto curves and hypervolume values look promising, but absolute differences are small. In the LLM experiments, no error bars are provided, and generally statistical significance is not established or discussed. The training times would need more explanation or analysis as well. The times for the baselines as well as HyperDPO will depend on several factors, such as how many w-vector to sample and the total number of epochs. A justification is needed why the provided analysis is fair.

On a more fundamental level I am unsure what claim the experimental evaluation support (also related to 2). Section 4 compares the proposed input-conditioned network to simpler baselines, but [Ruchte & Grabocka, 2021] already did that (on different tasks, of course). Can the objective (13) be expected to be different from other MO tasks, so new evidence is required? Then experiments that highlight how (13) relates to alternative objectives would make the evidence stronger. If the multi-objective way of training is indeed the core contribution, I would have expected a comparison to other multi-objective methods from the literature, in particular actual multi-objective hypernetworks, such as [Navon et al, "Learning the Pareto Front with Hypernetworks", ICLR 2021] or [Hoang et al, "Improving Pareto Front Learning via Multi-Sample Hypernetworks", AAAI 2023]. 

(Please note that I do not ask for new experiments of this type to be reported in the author response)

Additional comments:
- the reference section has some issues: e.g. acronyms and conferences are often incorrectly capitalized, arXiv versions are cited where published version exist, some publications years seem incorrect (e.g. NeurIPS 2024 proceedings don't exist yet).
- floating point notation, such as "1.473e-01", should be change to mathematical notation, "0.1473e"

questions:
The concerns I list in "weaknesses" make it hard to hard to judge the overall contribution and soundness of the work. Consequently, I gave low scores for these and the overall rating so far. I plan to revisit the scores after the author response.

* "hypernetwork"

I find *hypernetwork* not the correct term for the method you describe, given its use in the literature so far (see above). If you agree, please edit the manuscript accordingly. Otherwise, in case you have a different definition of hypernetwork in mind, please provide a reference for this. 

* Contribution

Could you please clarify which aspects/parts of HyperDPO exactly you consider new scientific contributions? Or is it more of a systems-type contribution, where the novelty lies in the combination of existing components?

* Experiments

Related to "Contribution": what is the exact scientific claim for which the experiments provide evidence? 

* Proposition 3.1

I have problems understanding the proof of Proposition 3.1 (page 16). Specifically, how does it follow from a comparison of $s_{\theta,c\beta}(y,w|x)$ with $s_{\theta,\beta}(y,w|x)$ that the numerators of both expressions have to be the same? Or do I misunderstand something else here?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730597090770

thread_type:
official_review

<thread_object>
creator: reviewer_nrft

<content>
summary:
This paper aims to tackle the multi-objective fine-tuning (MOFT) problem, where LLMs are trained towards satisfying multiple objectives simultaneously. As a main contribution, a method named HyperDPO is proposed. HyperDPO extends the Direct Preference Optimization (DPO) method with a hypernetwork that take weights of objectives and the input data. In this work, authors assumes the total utility (under MOO setting) is calculated via linear scalarization.  To demonstrate the effectiveness of HyperDPO, authors select three baseline methods, DPO-LS, DPO Soup, and MO-DPO. The authors select the hypervolume (HV) as the evaluation metric to compare the Pareto front. Experiments are conducted on two tasks, namely learning to rank and LLM alignment. Authors reported the results of fine-tuning GPT-2 and Alpaca-7B. The results demonstrate that HyperDPO could obtain higher HV with less training time, compared with baseline methods.

strengths:
This work propose the HyperDPO method to address the MOFT problem. The technical contribution of this work is to incorporate a hypernetwork (through Hyper Prompt Tuning (HPT)) to DPO, such that the weights of objectives can be considered when fine-tuning LLM. While both Hypernetwork and DPO have been recently studied, the integration of both methods, espcially, using weights as inputs to the hypernetwork, seems to be a noval integration. In terms of writing, this paper is clearly written. In addition, it illustrate the theoretical foundation with clear detailed mathematical proofs. As DPO is a efficient way to fine-tune LLM, while source code is not available at the moment, HyperDPO might be an efficient way to fine-tune LLM under multi-objective setting.

weaknesses:
This paper is overall well-written, however, the results could be further enhanced to better support the claim.  The first issue is that the results only include the hypervolume metric as the evaluation of pareto fronts (PF). Other important metrics including sparsity is not reported. In authors' experiments, they fine-tune two models, GPT-2 and Alpaca-7B. While results demonstrate HyperDPO perform better for both experiments, results in Figure 4 show that the performance of evaluated methods seems to be related to the base model. For example, while the PF of DPO-LS on GPT-2 is dominated by the one of HyperDPO, DPO-LS on Alpaca-7B performs quite close with HyperDPO. Similar pattern also reflects on HV and IQR. Therefore, it is not certain whether hyperDPO is generally better than other methods on different LLMs. More comparisons and analysis on when HyperDPO is a better method are expected. 

In terms of citations, the inline citations could be improved. Generally, "(author, year)" in a sentence could not be used as a word.

questions:
Here are my questions that requires further clarification from authors. 
1. Why the results of MO-DPO are not reported in the Table 2?
2. How many times/random seeds that the experiments are conducted? In the results, HV and training time are not reported with STD, while the average main score is reported with STD, why here is a difference?
3. Can HyperDPO be extended to non-linear utility functions in the context of MOO?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649060615

thread_type:
official_review

<thread_object>
creator: reviewer_jnjv

<content>
summary:
This paper focuses on Direct Preference Optimization (DPO) with multiple objectives. On top of the Plackett-Luce (PL) model, it proposes a generalized DPO problem that considers the Pareto front of multiple objectives. Afterward, by applying the hyper-network techniques, the authors propose a corresponding optimization algorithm through one-shot training, and offer a post-training control over the temperature hypermeter.

strengths:
1. This work extends the DPO problem into the multiple-objective setting, which is important for the alignment of LLMs. 
2. The proposed method connects the learning-to-rank problem and the LLM alignment. This might be instructive for developing more effective alignment methods.

weaknesses:
1. The main concern is that the contributions of this work might be overclaimed. After formulating the multi-objective DPO problem, this work uses Ruchte & Grabocka's algorithm to profile the Pareto front. 
2. Another contribution of this work is Prop. 3.1, which shows a linear transformation property between the optimal solutions under different $\beta$. However, according to the proof, this property is based on $\mathcal{L}\_{ListNet}$ (Eq. (12)) instead of $\mathcal{L}\_{Hypernet}$ 
 (Eq. (13)). Therefore, such a property ignores the Pareto fronts.
3. As one of the main contributions, the linear transformation property should have been validated in the experiments. Besides, how to apply this property in real scenarios should be further discussed.

questions:
Please refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731124891399

thread_type:
official_review

<thread_object>
creator: reviewer_8vxc

<content>
summary:
This work extends Direct Preference Optimization (DPO) to the multi-objective settings. 
This is achieved by replacing the BTL model to PL model that can handle list-wise rankings. 

The work also profiles the Pareto Front with hypernetworks. The Pareto Front is a set of solutions involving trade-offs such that no objective can be improved further without hurting other objectives.

strengths:
Fine tuning with multiple objectives is an important problem with many real world applications such as alignment of LLMs with multiple goals in helpfulness, trust and safety etc.

weaknesses:
The technical contribution of this work is quite incremental. It is a combination of known techniques.
- The generalization due to PL was originally proposed in Liu 2024.
- Using a hypernetwork to profile the Pareto Front was proposed in Navon 2020.

questions:
In the experiment, how would you describe the significance of the improvements?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732048403111

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Responses (1/3)

comment:
We sincerely thank all the reviewers for their constructive feedback and for recognizing the contributions of our work. We are heartened by the positive remarks highlighting our framework as "*timely and relevant,*" "*novel,*" "*clearly written,*" offering "*improved performance compared to baseline approaches,*" and that "*might be an efficient way to fine-tune LLM under multi-objective setting.*" Such endorsements affirm the value and potential impact of our research. Below, we address two common concerns raised by multiple reviewers, ensuring our commitment to clarity and precision in our scientific communication.

---

## Regarding the Core Contributions of the Work 

We acknowledge the concerns raised by reviewers regarding the core contributions of the work. We would like to clarify the contributions of our work as follows:

### Formulation of the Multi-Objective Fine-Tuning (MOFT) Problem

Our newly-formulated MOFT problems are motivated by and designed to extract commonalities among several existing problems, such as Learning-to-Rank (LTR) [1], Multi-Objective Optimization (MOO) [2], and LLM alignment [3], especially Direct Preference Optimization (DPO) [4]. As scattering methods and techniques have been proposed for these problems, we point out that as long as the preference across different items according to one query are quantified in the form of logits, the problems and their fine-tuning can be unified under a single framework, with a general solution that can be applicable to a wide range of tasks and generalizable to multi-objective settings.

As commented by Reviewer F9MP, our work "*addresses the largely unexplored problem of how LLMs can be fine-tuned in a multi-objective way,*" and "*matches well to real-world tasks*". Given the pervasiveness of such problems where one would like to fine-tune an existing model that may be time-consuming to retrain with multiple objectives, we believe our formulation of the MOFT problem would serve as a novel, relevant, and solid foundation for future research in this area.

### Introduction of HyperDPO Framework

With the common characteristics and challenges among the problems clearly identified, we are subsequently able to adapt the state-of-the-art techniques from LTR, MOO, and LLM Alignment to the MOFT problem. 

- The HyperDPO framework are grounded in the Plackett-Luce model for listwise ranking, powered by the hypernetwork structure for Pareto front profiling, and particularly designed for fine-tuning. 
- As we mentioned in the manuscript, the HyperDPO framework can be seen as the generalization of LTR, MOO, and DPO problems, and there would be no wonder that our framework is capable of handling a wide range of tasks, as demonstrated by the superior performance on both LTR and LLM alignment in our experiments, which are of very distinct natures.
- We have also explored several theoretical properties of the HyperDPO framework in Proposition 3.1, so as to illustrate the rigorousness and rationale behind our framework.

As commented by Reviewer F9MP, the integration of hypernetwork and DPO "*seems to be a novel integration*" and by Reviewer JnjV, "*connects the learning-to-rank problem and the LLM alignment*", and "*might be instructive for developing more effective alignment methods*", we believe our hypernetwork-based method is a novel and useful way for MOFT problems, and we are excited to share our unique experimental findings with the community.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a Multi-Objective Fine-Tuning framework named HyperDPO, which applies to various machine learning tasks, including Learning-to-Rank and aligning large language models (LLMs). Initially, the paper received mixed reviews. After the rebuttal phase, the final feedback remains inconsistent, with one positive (8), two borderline (6, 5), and one negative review (3). To make a final decision, I thoroughly read and discussed the comments with SAC, with special attention to the negative one. Overall, I agree with the three non-negative reviews. That is, it is a promising direction to integrate hypernetwork to DPO, and this paper is validated by detailed mathematical proofs. However, the shortcomings of this paper are also significant, and I decide to reject this paper due to the following reasons.

- As pointed out by Reviewer ``F9MP``, although the proposed method outperforms the baseline methods in the context of fine-tuning, prior work has shown similar performance gain in other tasks [Ruchte & Grabocka, 2021]. Hence,  more evidence is necessary to demonstrate that the gain comes from the newly proposed components. Otherwise, it is hard to evaluate the effectiveness of the proposed method.
- Some important concepts, such as hypernetwork, are confused. Specifically, hypernetwork has appeared in [Ha et al, ICLR 2017], which is inherently different from that proposed in this paper. It is recommended to replace the term with the other one, and the description of the proposed method should be modified accordingly. Hence, a significant revision is required.

To sum up, I recognize the value of HyperDPO (or CondDPO, according to the advice of Reviewer ``F9MP``). However, According to the careful and professional suggestions proposed by Reviewer ``F9MP``, I think a major revision is necessary to make this paper qualified for a top publication venue. I encourage the authors to revise these problems carefully and resubmit the paper to a future venue.