Learning to Optimize Multi-Objective Alignment Through Dynamic Reward
  Weighting
Authors: name: Yining Lu, name: Zilong Wang, name: Shiyang Li, name: Xin Liu, name: Changlong Yu, name: Qingyu Yin, name: Zhan Shi, name: Zixuan Zhang, name: Meng Jiang
Organization: University of Notre Dame DM2 Lab

Prior works in multi-objective reinforcement learning typically use linear
reward scalarization with fixed weights, which provably fail to capture
non-convex Pareto fronts and thus yield suboptimal results. This limitation
becomes especially critical in online preference alignment for large language
models. Here, stochastic trajectories generated by parameterized policies
create highly non-linear and non-convex mappings from parameters to objectives
that no single static weighting scheme can find optimal trade-offs. We address
this limitation by introducing dynamic reward weighting, which adaptively
adjusts reward weights during the online reinforcement learning process. Unlike
existing approaches that rely on fixed-weight interpolation, our dynamic
weighting continuously balances and prioritizes objectives in training,
facilitating effective exploration of Pareto fronts in objective space. We
introduce two approaches of increasing sophistication and generalizability: (1)
hypervolume-guided weight adaptation and (2) gradient-based weight
optimization, offering a versatile toolkit for online multi-objective
alignment. Our extensive experiments demonstrate their compatibility with
commonly used online reinforcement learning algorithms (including GRPO,
REINFORCE, and RLOO), effectiveness across multiple mathematical reasoning
datasets, and applicability to different model families, consistently achieving
Pareto dominant solutions with fewer training steps than fixed-weight linear
scalarization baselines.