Multi-Objective Alignment of LLMs with ORPO using Self-Judgement

Authors: Jayant Parashar, Suchendra M. Bhandarkar
Primary area: foundation or frontier models, including LLMs
Keywords: LLMs, preference optimization, self-judgement, LLM-as-a-judge, supervised fine-tuning, multi-objective optimization, multi-task learning, supervised learning

Abstract:
The alignment of Large Language Models (LLMs) is achieved through fine-tuning with human preference data, where preference optimization has become a critical part of the process. Many methods have scaled LLM performance by incorporating self-judgement, highlighting the importance of unifying LLM-as-a-judge with the alignment process. One such method, called Self-rewarding LLMs, iteratively samples new data from the model to improve alignment using self-judgement. Since this additional data is generated by the LLM, we argue that similar improvements can be achieved without new data. We propose a method that reuses alignment data in the form of a self-judgement classification task and defines a multi-objective optimization problem. Our self-judgement task is derived from a simple transformation of the primary alignment data, asking the LLM to select the superior response. It introduces no new data beyond the existing alignment data. Thus, we claim the improvements are due to positive interference between the two tasks. We focus on a direct preference optimization method called Odds-Ratio Preference Optimization (ORPO). We conduct a thorough study of linear scalarization on two objectives and introduce two alternative approaches that vary the emphasis on alignment versus self-judgement objectives. Our results on Mistral 7B indicate a promising direction for fine-tuning LLMs on multiple objectives, particularly for improving performance on related tasks without additional natural language data.

Decision: reject
Reviewer scores: 3, 5, 3, 3, 3

Reviews and discussion:
<thread>
creation date:
1730652028819

thread_type:
official_review

<thread_object>
creator: reviewer_sjfv

<content>
summary:
This paper proposes a multi-objective alignment method based on ORPO. More specifically, the first objective is the same as the log odds alignment objective proposed in the ORPO paper, the authors propose to add a second objective where the LLM should predict which response is preferred by humans. They test their approach on Alpaca Eval and ablate different design decisions.

strengths:
The ablation study of the proposed method appears to be thorough. The authors systematically compare various strategies for combining the two objectives, including conditional, epoch-by-epoch, and task-by-task approaches, and report detailed results for each.

weaknesses:
This paper would benefit from substantial improvements across several dimensions:

**Presentation:** The framing of the paper is somewhat misleading. The primary contribution appears to be the proposal of combining two objectives and exploring various ways to integrate them. However, certain claims, such as the statement on line 97, position this work as an alternative to iterative fine-tuning. The intent of sampling from the current policy in iterative fine-tuning is generally to produce more on-policy data, which can help bridge the performance gap between offline and online methods. I would accept this framing if the paper presented a novel approach for obtaining on-policy data without direct sampling from the policy itself. As it stands, this framing seems to overstate the contribution.

**Motivation:** The motivation lacks clarity. The authors suggest that because LLMs are often used to judge their own outputs, sampling from the model may not be necessary. This argument is unclear; it's possible that verification tasks are fundamentally easier than generation tasks—an LLM may perform well as a judge but poorly as a generator, thereby effectively evaluating its own inputs and improving performance. Additionally, the paper does not seem to empirically test this hypothesis. If the aim is to demonstrate that the proposed approach can replace iterative fine-tuning, then a direct comparison with iterative fine-tuning methods is essential, showing that similar or better performance can be achieved with comparable or less data.

**Experiments:** The experimental design has some gaps. Key baselines, such as the performance of the supervised fine-tuning (SFT) model or the pretrained model, are missing. To evaluate the effectiveness of the proposed method, comparisons with established alignment methods like Direct Preference Optimization (DPO), Proximal Policy Optimization (PPO), and other conventional techniques are necessary.

questions:
- What is the y-axis in Figure 3?
- Line 283 refers to equation 8, but I don't see any equation 8
- In equation 6, $\theta'$ is said to be frozen weights of the self judgement task, it is the parameters of the same model, right? Also, in the same equation, shouldn't you use a different prompt for this task and the alignment task? (it seems that both tasks are using $x$)
- Why is the second task called self judgment in the context of your work? It doesn't seem to be that you are generating the responses from the same LLM, so it is not really **self** judgement, is it?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730670903193

thread_type:
official_review

<thread_object>
creator: reviewer_lm5q

<content>
summary:
The authors propose a new alignment method for LLMs that uses multi-objective optimization to combine (1) self-judgment and (2) odds-ratio preference optimization (ORPO; Hong et al. 2024).  This paper explores a few general techniques for performing multi-objective optimization for these two objectives.  They present some promising preliminary experiments.

strengths:
Exploring techniques for combining the strengths of various learning paradigms through multi-objective optimization is an interesting avenue to explore.  Similarly, looking at methods to exploit LLMs as judges is also interesting.

weaknesses:
I found this paper hard to follow.  The clarity of the presentation and overall pitch of the paper needs to be improved.

The method's empirical validation is weak. I do not see a meaningful comparison to other methods.

I do not see a good case for why the paper needs to combine specifically ORPO with LLMs-as-judge.  What about other competing alignment objectives?

questions:
What happens if you replace ORPD with a more popular objective like DPO?  Are you open to adding this comparison to the paper?

Multi-objective optimization techniques should support more than two objectives.  Why did you limit the experiments to just two objectives? 

---
Suggestions:

 - Please learn the difference between textual (`\citet`) and parenthetical (`\citep`) citations.

 - I suggest moving the long related work section to a later section of the paper or an appendix.  This will help refocus the paper in my opinion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730679195970

thread_type:
official_review

<thread_object>
creator: reviewer_azdc

<content>
summary:
The paper proposes multi-objective optimization of self-judged preference data and available alignment data. The self-judged data is collected on a dataset where the human annotated preference labels are already available. The authors propose that in addition to optimizing the model on annotated preference data we should also optimize on self-judged preference data. The authors use two multiobjective optimization algorithm on these two objectives. Additionally they use one algorithm called CONJOINED MULTI-OBJECTIVE LOSS where they optimize against the self-judgement objective. The experiment result does not show any significant gains compared to the baselines.

strengths:
1. The authors clearly explain their approach and it's connection to previous approaches. The novel contributions have been clearly explained.
2. For self-judgment data collection the authors have addressed the issue of various types of biases. To mitigate the biases they have created about 50 prompts to get the correct judgement as per the model.

weaknesses:
1. On Figure 3 (right) the Pareto frontier is visualized for different scalar weights. The solutions be smoothly moving on the multiobjective-loss space as the weights are changed. However, the solutions are all over the place. This indicates that the optimization have not been performed well or the random effects (of batch order etc) are too high. This puts into question the validity of the results from the first set of results. I would suggest authors perform thorough optimization through longer training. Otherwise, the authors should perform multiple runs with the same weight and show the averaged results. 
2. The proposed approach relies on annotated alignment data. This data already has the gold annotations. Using a small LLM (Mistral-7B) to annotate this again (ie self-judged data) and using this annotation as a separate ground truth does not make sense. The model may just be wrong on it's self-judgement. Unless this self-judgement annotations are collected on a large scale in an unsupervised fashion (as done on the paper Self-Rewarding Language Models), we cannot rely on this self-judgement annotations.
3. The authors have not clearly explained why the two objectives (performance on alignment data and performance on self-judged data)  may be contrary to each other. One way to show that would be to provide the loss for self-judged data for the baseline model. If better performance on the baseline means better performance on self-judged data than the two objectives are always aligned and there is no point in doing multi-objective optimzation.
4. For the CONJOINED MULTI-OBJECTIVE LOSS, the self-judgement objective is optimized against. This is a direct contradiction to the multi-objective optimization that is being argued by the authors.

questions:
1. Can you explain why log-sigmoid is used in equation 3?
2. Is there any numerical stability issues during training? As odds ratio are calculated it to require the probabilities which can be very small and cause numerical stability issues. Does it happen?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730683122062

thread_type:
official_review

<thread_object>
creator: reviewer_tktp

<content>
summary:
The paper proposes a method to fine-tune LLMs by reusing existing alignment data for self-judgment tasks, eliminating the need for self-sampling. This approach utilizes the ORPO method as the primary baseline for supervised fine-tuning. The training process is approached as a multi-objective optimization problem, with comparisons among scalarization, conjoint, and conditional optimization methods. Experiments are conducted on the Mistral 7B model, benchmarked against the Llama 3 70B model on the Alpaca Eval dataset. The results show mixed to slight improvements in alignment in contrast to the ORPO baseline fine-tuned solely on the alignment task.

strengths:
- The integration of ORPO with a multi-objective optimization framework to simultaneously address alignment and self-judgment tasks is innovative and promises efficiency improvements in LLM fine-tuning.
- The introduction, related work and methodology sections are clearly written, providing a solid understanding of the theoretical background and operational framework. This helps in appreciating how the proposed method diverges from and potentially improves upon existing techniques.

weaknesses:
- The experiment and results section reads a bit rushed and lacks examples that reflect the discussed issues such as verbosity and overfitting.  These could be included in an appendix for reference. 
- The findings are limited by the use of only one model and one evaluation dataset. Expanding the testing to include models of varying sizes could better determine the method's general applicability and robustness.
- The identification of verbosity as a significant issue without an applicable solution reduces the potential utility of the proposed approach. More testing on prompt engineering or response format control could help mitigate the verbosity bias ( see questions for more details).

questions:
1. The paper tests the proposed method only on the Mistral 7B model and on one evaluation dataset. I understand that finetuning and evals can be costly, testing on other models within the 2B-13B size range could help determine if verbosity is inherent to the method itself or specific to Mistral 7B model?
2. With verbosity identified as a major issue, have you tried prompt engineering that could effectively reduce the bias in model responses, such as asking the model to respond briefly or follow a specific response format?
3.  Regarding the overfitting issue, would separating the alignment dataset into an alignment subset and another non-overlapping self-judgment subset mitigate this problem? 
4. Have there been tests where only self-judgment training was conducted (w1=0, w2=1). As a contrast to w1=1, w2=0? I understand that this would not fine-tune on alignment data, but only on model’s reasoning why it chose A over B (or vice versa). 
5. It is not clear how a longer context length might be an issue. The example in Figure 1 shows minimal differences between the two cases, as the self-judgment dataset wraps a rather brief prompt around the rather lengthier alignment data.
6. It is not clear why some training runs have 2 epochs while others have 3. 
7. By equation 8, which is missing, do you mean equation 7 for the gradient of Conjoined method?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688030773

thread_type:
official_review

<thread_object>
creator: reviewer_2rx7

<content>
summary:
This work presents an approach of multi-objective alignment of LLMs for both answering and self-judging abilities. Specifically, the model is tuned with both aligning and self-judgment objectives, and various ways to combine the two objectives are investigated. The tuning itself does not require sampling new data, which makes the process data and computational efficiency. Experiments on Alpaca Eval show the effectivenss of the proposed method in some way.

strengths:
- The direction of self-rewarding is interesting, which makes the model being able to judge its own outputs and get progress.
- The idea of reducing dependency on high-quality data is attractive.

weaknesses:
- Most of the ideas in this work seem to be from previous works and this work seems to focus on a combination of various methods. This is fine if this work aims to be an empirical one, however, the experiments and analyses are lacking, and there seems to be little insights on which factors work and why.
- There should be more baseline methods to prepare the proposed method, which is important to demonstrate certain choices of the proposed method.

questions:
- I'm wonderin why specifically using ORPO? It seems that there are no direct comparisons to other methods such as DPO, more explanations and experimental results should be provided.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper explores multi-objective optimization to align LLMs on answering and self-judging abilities by leveraging existing alignment data and ORPO as a baseline. It is innovative and promises efficiency improvements in LLM fine-tuning. However, all reviewers agree that this paper requires significant presentation, motivation, and experimental evaluation revisions. Therefore, I recommend rejecting this work.