Learning Loss Landscapes in Preference Optimization

Authors: Carlo Alfano, Silvia Sapora, Jakob Nicolaus Foerster, Patrick Rebeschini, Yee Whye Teh
Primary area: reinforcement learning
Keywords: Preference optimization, mirror descent

Abstract:
We present a framework to discover preference optimization algorithms specialized to particular scenarios, in a theoretically sound and computationally efficient setting. We start by designing a novel family of PO algorithms based on mirror descent, which we call Mirror Preference Optimization (MPO). MPO recovers existing methods like Direct Preference Optimization (DPO) and Odds-Ratio Preference Optimization (ORPO) for specific choices of the mirror map. Given specific properties of preference datasets, such as mixed-quality or noisy data, we show that we can efficiently search the MPO class to find specialized algorithms that outperform current baselines. Namely, we leverage evolutionary strategies and preference datasets generated on MuJoCo environments to systematically evaluate and optimize MPO algorithms on hand-crafted scenarios. We demonstrate the resulting PO algorithms successfully transfer to a Large Language Model (LLM) alignment task, where they demonstrate superior robustness in handling mixed-quality datasets.

Decision: reject
Reviewer scores: 6, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730100730153

thread_type:
official_review

<thread_object>
creator: reviewer_cdrz

<content>
summary:
This paper generalizes a recent variant of DPO, namely ORPO, by introducing two maps into the problem definition of ORPO. Then these maps are parametrized by a neural network that is trained by OpenAI-ES approach. An empirical study shows better performance than the original ORPO in some settings of noisy preference datasets and competitive performance in other settings.

strengths:
It generalizes a recent and promising variant of DPO, namely ORPO, by introducing mirror maps.

Empirical evaluation shows that optimizing the parameterized mirror maps by OpenAI-ES sometimes results in better performance.

weaknesses:
The position of this study among related studies has not been clarified. Several studies have investigated the preference optimization under noisy preferences and proposed variants of DPO, such as cDPO and rDPO. I see that the current paper is based on ORPO and hence the direct comparison may not be fully fair, but at the same time, I do not see the advantages of the current approach. On the other hand, there are several studies modifying the loss or the divergence in DPO such as IPO and f-DPO. The novelty of this study is not clearly stated among these related works. The literature review should cover these topics, however, in this paper, only f-DPO is mentioned in the related work as a generalization of DPO. 

The motivation is not stated clearly. Why is the generalization of ORPO as in (10) and searching the introduced maps expected to improve the performance of ORPO for noisy preference datasets? I could not see the logic behind it. Because the logic is not provided, I do not see whether the experimental results are really associated with the generalization. Does this generalization contribute to the good performance? From the current experiments, I do not see whether the proposed approach is better than simply introducing some scalar factors to all terms in ORPO and optimizing them.

Reproducibility is not high. First, the proposed learning scheme for mirror maps is not clearly stated. It requires estimating the cumulative reward for each obtained policy, but how is it estimated in an offline manner (stated in Line 102)? Second, the code to reproduce the experiments is not available. Some details, such as the architecture of the policy network and temperature parameters.

Comparison to other approaches is missing. The proposed approach is compared only to the original ORPO, which is not designed to address noisy preference datasets. No comparisons with DPO variants such as rDPO specialized for noisy preference datasets are provided.

questions:
Please address the questions given in the weaknesses section.

Additional comments are listed here:

Additional comments are listed here:

In the experiments in Table 1, what is the difference between setting a high temperature and adding a noise in this setting? They look like the same.

How is the temperature parameter in Table 2 set?

How are the obtained mirror maps interpreted? If it is transferable, shouldn’t the result be analyzed?

In line 222, it was not clear to me how these terms canceled out. Could you elaborate on the derivation?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730268987584

thread_type:
official_review

<thread_object>
creator: reviewer_93e1

<content>
summary:
This paper presents an empirical study on preference optimization (PO) algorithms, focusing on how preference dataset properties affect performance. The authors introduce a novel PO framework based on mirror descent, which generalizes existing methods like DPO and ORPO. A new framework for algorithm discovery, and the demonstration of generalization to LLM tasks have been proposed.

strengths:
The paper proposes a novel framework for preference optimization (PO) based on mirror descent, which generalizes existing methods like DPO and ORPO. This approach of using mirror maps to explore the space of PO algorithms and discover new loss functions is relatively new in the field.
The idea of using evolutionary strategies to search for the best mirror maps and thus new PO algorithms allows for an automated and potentially more efficient way of finding algorithms that can handle different dataset properties compared to traditional methods of manually designing algorithms.
The application of the discovered PO algorithms from MuJoCo environments to LLM fine-tuning tasks shows the potential and generalization of algorithms discovered in a simpler, more computationally tractable environment to a more complex and practical domain like LLM tuning.

weaknesses:
The discovered loss functions are complex and parametrized by neural networks. There is no attempt to provide an intuitive interpretation of what these functions represent or how they relate to the properties of the preference datasets. This lack of interpretability makes it difficult to understand and potentially modify the algorithms for specific applications.
The theoretical derivations, such as Theorem 3.1, are based on simplified assumptions. The impact of relaxing these assumptions or considering more complex scenarios where they may not hold is not explored.
The LLM fine-tuning experiment uses a modified dataset to simulate mixed-quality data, but it would be beneficial to validate the proposed algorithms on real-world, large-scale LLM datasets.
While the paper identifies certain failure modes of ORPO and shows how the proposed algorithms address them, the discussion could be more comprehensive. For example, exploring why these failure modes occur in the first place and what implications they have for the broader field of preference optimization could provide deeper insights.

questions:
1) In the Hopper and Ant/TLA environments, why were the specific numbers of rows chosen for the datasets (5120 for Hopper and 1280 for TLA)? How do these choices impact the representativeness and generalizability of the results?
2) The evolutionary strategies used to search for the best mirror maps and loss functions are computationally expensive. Have you considered any techniques to speed up this process or approximate the optimal solutions more efficiently?
3) The discovered loss functions are complex and parametrized by neural networks. How can we ensure the stability and reliability of these functions in different application scenarios?
4) The generalization of the discovered PO algorithms from MuJoCo to LLM tasks is demonstrated, but it is not clear how the differences in the nature of the two domains (e.g., discrete vs. continuous actions, different data distributions) are accounted for. Can you provide more insights into this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730447027523

thread_type:
official_review

<thread_object>
creator: reviewer_t1gu

<content>
summary:
This paper proposed a new Preference Optimization (PO) framework . 

Overall, I think the novelty of this paper is limited.

strengths:
This paper provide a theoretical way to study preference optimization (PO) and give both theoretical and empirical results.

weaknesses:
Experimental results are a bit too weak. Others see questions.

questions:
1. Line 144. "A known issue of DPO is that it pushes probability mass away from the preference dataset and to
unseen responses". Is that true?

2. Line 133. Besides it computational costs, PPO is known to be prone to reward overoptimization. Is that true?

3. In which case DPO is better than RLHF, and which case RLHF is better than DPO. Is there an affirmative answer?

4. Line 240 to 242. Why these functions are considered?

5. As far as I understand, the proposed work try to change different Bregman distance. This is very similar to an ICLR work called "f-PO: Generalizing Preference Optimization with f-divergence Minimization". So, is there equivalence between each other? For example, by choosing a propose f divergence will yield a corresponding Bregman distance. I admit that choosing the Bregman distance is a clever idea.  However, what is the novelty becomes compared with FPO?

6. This paper mention "online " occurs a lot of times. What does it mean in preference optimization?

7. The experiments seems a bit weak. What is the performance on multi-modal model or 3B model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707234983

thread_type:
official_review

<thread_object>
creator: reviewer_vf5c

<content>
summary:
The paper investigates how preference optimization (PO) algorithms perform under varying conditions of data quality. Through empirical experiments primarily in MuJoCo environments, the authors examine the weaknesses of state-of-the-art PO methods like Direct Preference Optimization (DPO) and Odds-Ratio Preference Optimization (ORPO) when working with noisy or low-quality data. They propose a novel PO framework based on mirror descent to generalize existing PO methods and employ evolutionary strategies to identify optimal loss functions that could address identified performance issues. Results demonstrate that these new loss functions improve PO algorithm performance, even in fine-tuning tasks with large language models (LLMs).

strengths:
1. The paper introduces a novel framework for loss optimization using mirror descent, which effectively addresses common issues in preference datasets.
2. The study includes well-designed experiments across various data settings.
3. The flexibility of the method across different application domains.

weaknesses:
1. While the methodology is empirically validated, a deeper theoretical analysis, particularly around robustness and generalizability, would strengthen the findings.
2. The paper lacks a discussion on the computational cost of training with evolutionary strategies, especially for large models or in settings where data or compute resources may be constrained. The framework鈥檚 reliance on the computational complexity of evolutionary strategies might make it less practical.
3. The experimental settings focus primarily on MuJoCo, which, while diverse, may limit insights into real-world applications. Expanding beyond MuJoCo could improve the generalizability of results.

questions:
1. How does the computational cost of this new framework compare with conventional PO methods in large-scale settings? Could efficiency trade-offs limit practical applications?
2. Are there specific criteria for choosing mirror maps or evolutionary strategies based on the characteristics of the preference dataset?
3. Could the approach be extended or adapted to reinforcement learning tasks beyond MuJoCo and language model fine-tuning? How would it perform in such settings?
4. More peer algorithms should be included for experimental comparison.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces Mirror Preference Optimization (MPO), a framework that generalizes existing preference optimization methods, including Direct Preference Optimization (DPO) and Odds-Ratio Preference Optimization (ORPO) via mirror descent. The authors use evolutionary strategies to discover specialized loss functions tailored to noisy or mixed-quality datasets. Empirical evaluations in MuJoCo environments show improvements over baselines, and the discovered loss functions transfer successfully to large language model fine-tuning tasks.

**Strengths**
* MPO offers a principled generalization of existing PO methods
* The experiments in MuJoCo show consistent improvements over baselines, and the transfer to LLM tasks is promising.
* The use of ES for discovering loss functions helps automate algorithm design.

**Weaknesses**
* Limited comparisons with specialized DPO variants (e.g., cDPO, rDPO) hinder a comprehensive assessment of novelty.
* Lacks robust theoretical guarantees on generalizability.
* Validation is focused on MuJoCo with limited real-world testing.

The paper presents an interesting and potentially impactful approach, but the weaknesses — particularly the lack of baseline comparisons, theoretical analysis, and broader experimental validation — reduce its overall significance. While the approach shows promise, these issues currently outweigh the paper's strengths.