Title: Meta-Learning Objectives for Preference Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Preference Optimisation

Evolution Strategies

Meta-Learning

Mirror Maps
Abstract: Evaluating preference optimization (PO) algorithms on LLM alignment is a challenging task that presents prohibitive costs, noise, and several variables like model size and hyper-parameters. In this work, we show that it is possible to gain insights on the efficacy of PO algorithm on simpler benchmarks. We design a diagnostic suite of MuJoCo tasks and datasets, which we use to systematically evaluate PO algorithms, establishing a more controlled and cheaper benchmark. We then propose a novel family of PO algorithms based on mirror descent, which we call Mirror Preference Optimization (MPO). Through evolutionary strategies, we search this class to discover algorithms specialized to specific properties of preference datasets, such as mixed-quality or noisy data. We demonstrate that our discovered PO algorithms outperform all known algorithms in the targeted MuJoCo settings. Finally, based on the insights gained from our MuJoCo experiments, we design a PO algorithm that significantly outperform existing baselines in an LLM alignment task.


Peer reviews:
comments: SUMMARY:
This paper proposes a unified formulation of the existing Preference Optimisation (PO) algorithm. Significantly, it replaces the original KL divergence with the Bregman divergence and introduces an evolutionary term to control the percentage of training progress from SFT to PO.  Based on the unified formulation (meta-learning objective, such as 1step/2-step MPO and Temporally-Aware MPO, TA-MPO), they evaluate existing PO algorithms in a synthetic experiment environment, MuJoCo, in which you can control the reward structure as the reward datasets are well-defined. Beyond the synthetic setting, the authors also show the results when transferring to LLM. Specifically,  they implement the learning objective (TA-MPO) and compare with existing PO algorithms, such as DPO, SimPO, etc. The experimental results show the effectiveness of TA-MPO.

STRENGTHS AND WEAKNESSES:
Strengths:
1. This paper proposes a unified theoretical formulation of existing preference optimisation algorithms, which provide a comprehensive and fundamental perspective to compare and improve the PO algorithms.
2. The meta-objective shows effectiveness in both synthetic and LLM PO settings.


Weakness:
1. The verification in LLM settings is rather limited compared with the synthetic settings, so it is unclear if the objective can work well and be generalisable in real LLM training. Also, the performances of TA-MPO (1), TA-MPO (2) vary a lot.

QUESTIONS:
1. As far as I understand, this meta-objective (I) replaces KL with Bregman divergence (2), introducing an adaptive factor transition from SFT to PO, so what are the separate effects for the two designs?
2. Why TA-MPO (1), TA-MPO (2)  perform very differently? any hypermeter suggestions in different practical settings?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors introduce a generalized class of direct preference alignment approaches that build on some theoretical techniques from information geometry. Specifically, in contrast to classical RLHF objective (Eq. 4), they replace the KL-divergence term with a Bregman divergence term (Eq. 7) based on \"mirror maps\" (the details of which are included in 2.2. While I wasn't familiar with these concepts, I found this part easy to follow). They key advantage of this formulation over a standard KL-divergence term is summarized starting on line 140: \"When the Bregman divergence is employed as a regularization term in optimization problems, tuning the mirror map allows us to control the geometry of the updated of the parameters to be optimized, determining when to take large or small updates...\"\n\nMost interestingly, many mirror maps can be defined that induce different Bregman divergences, each of which will enforce different types of regularization (this includes a temporal-aware variant, the learns to switch between different losses). They show that DPO and ORPO, two popular direct preference alignment approaches, employ a particular set of mirror maps (their full objective and derivation of DPO and ORPO importantly relies on Theorem 3, which follows a similar derivation to what's used in the original DPO). What's more, they propose learning new mirror maps using meta-learning via the OpenAI-ES method in order to discovery preference-tailored losses. \n\nTheir experiment setting is non-standard, and first involves what they call \"MuJoCO experiments\" and variations of the ant simulation (clarifying what MuJoCO is and stands for early on would be helpful for those who are not familiar). In brief summary, they use the MuJoCO environment to generate different ant simulation scenarios with specific properties (e.g., stylistic variations, systematically injected noise) and synthetically generate preference data from these scenarios. The intuition (as I understand) is use such data to demonstrate how their approach is able to adapt to the pecularities of each preference distribution and learn the corresponding mirror maps via OpenAI-ES. They find that their approach is able to effectively discover/meta-learn improved losses (Table 1). To show that their approach also works in a more conventional LLM preference optimization setting, they report small scale experiments in Table 2 where their approach is competitive with a comprehensive set of standard approaches (CPO, ORPO, DPO, SimPO).\n\nSTRENGTHS AND WEAKNESSES:\n# strengths \n\n-- A new theoretical framework for direct preference optimization from information geometry (MPO) that broadens our understanding and the scope of current approaches. This approach not only provides an original view on existing losses, but provides a means for defining many new classes of losses. \n\n-- An empirical demonstration that such a framework can yield improved losses, especially through meta- and evolutionary learning. I could imagine that this part about learning the parameters of their MPO could influence many follow up studies, especially following up on their small scale \"LLM transfer\" study in Section 5.  \n\n# weaknesses \n\n-- (minor) Given the non-standard nature of their MuJoCO expeirments, a figure illustrating their data generation process and a running example would be very helpful, since the details are otherwise hard to follow. \n\n-- (minor) I also found figure 5 to be confusing, an intuition for why the absolute value of the gradient is meaninful to show seems to be missing. Please clarify this.\n\nQUESTIONS:\n-- Why are the length controlled win-rates put in the parethenses in Table 2, shouldn't this be the main metric to report? Please clarify. \n\n-- Can you explain why you only use these 7k preference mixtures? Do you think this would be the same if you scaled this? Have you tried any scaled experiments?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies preference optimization (PO) algorithms for aligning both reinforcement learning agents in simulated control environments and large language models (LLMs) using human feedback. Motivated by the high cost and instability of evaluating PO algorithms directly on LLMs, the authors design a MuJoCo-based diagnostic benchmark, simulating preference datasets with varying quality and noise. The authors empirically show that their discovered objectives outperform baselines on MuJoCo TLA tasks, particularly in noisy and mixed-quality datasets, and that a variant (TA-MPO) yields competitive or superior results in LLM alignment.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The authors alleviate an important evaluation challenge in PO by introducing a controlled and interpretable MuJoCo benchmark to probe PO algorithms’ behaviors under varying data noise and quality, enabling better comparisons than LLMs alone.\n\n2. The MPO framework is interesting, extending and generalizing well-known PO algorithms (DPO, ORPO) via mirror maps and Bregman divergences. \n\n3. The approach shows robustness and improved performance in high-noise and mixed-quality data cases.\n\n4. This paper provides gradients for SimPO, ORPO, and discovered MPO losses, illustrating where these losses induce significant parameter updates; such visual inspection aids interpretability of the algorithms’ behavior.\n\n\nWeaknesses\n\n1. While the paper claims practical transferability to LLM alignment, the experiments in Table 2 only cover AlpacaEval. The degree of improvement, although sometimes statistically significant, is not overwhelming. It’s not clear how robust TA-MPO will be across other important alignment tasks, adversarial data, or human preferences beyond the datasets tested.\n\n2. As noted in Section 5, algorithms discovered with parameterized $g$ overfit the MuJoCo setting and transfer poorly to LLMs, raising concerns about generalizability and task-specific tuning rather than finding universal improvements.\n\n3. Table 2’s reliance on an automatic judge (Llama-3.1-70B-Instruct) rather than diverse human annotators or more comprehensive human-based preference collection somewhat weakens claims of improved LLM alignment in practice.\n\nQUESTIONS:\nPlease refer to Strengths And Weaknesses\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

comments: SUMMARY:
This paper proposes a new framework, Mirror Preference Optimization (MPO), for aligning models with human preferences. Instead of evaluating Preference Optimization (PO) methods directly on large language models (LLMs), which is costly and noisy, the authors construct a controlled benchmark using MuJoCo environments to simulate preference datasets. They formalize a general class of PO algorithms using mirror descent and employ evolutionary strategies to discover algorithms tailored to various dataset properties (e.g., noise, quality). These discovered algorithms outperform existing PO methods in MuJoCo settings. Leveraging insights from these experiments, they propose a temporally-aware MPO (TA-MPO) algorithm, which shows improved performance in real LLM alignment tasks.

STRENGTHS AND WEAKNESSES:
**Strengths**

Proposes a unified framework (MPO) that generalizes DPO/ORPO and allows algorithm discovery via mirror descent and ES.

Experiments on MuJoCo are systematic and cover different noise and data quality settings.

The discovered algorithms outperform baselines in these synthetic benchmarks.

**Weaknesses**

Related work is incomplete—missing citations like SPO and several recent LLM-specific PO methods.

Evaluating PO methods on simple benchmarks is not novel and has been done before.

The claimed transfer from MuJoCo to LLM lacks clear insight or analysis showing what actually generalizes.

LLM experiments are limited and only form a small part of the paper.

QUESTIONS:
1. Can you explain the insights that allow transfer from MuJoCo to LLMs?
2. Can you do more LLM experiments?
3. Why do you use Bregman divergence instead of f-divergence?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-03