NeurIPS 2025

Alignment of Large Language Models with
Constrained Learning
Botong Zhang
University of Pennsylvania
bzhang16@seas.upenn.edu
Shuo Li
Amazon
lishuo1@seas.upenn.edu
Ignacio Hounie
University of Pennsylvania
ihounie@seas.upenn.edu
Osbert Bastani
University of Pennsylvania
obastani@seas.upenn.edu
Dongsheng Ding∗
University of Tennessee, Knoxville
dongshed@utk.edu
Alejandro Ribeiro
University of Pennsylvania
aribeiro@seas.upenn.edu
Abstract
We study the problem of computing an optimal large language model (LLM)
policy for the constrained alignment problem, where the goal is to maximize
a primary reward objective while satisfying constraints on secondary utilities.
Despite the popularity of Lagrangian-based LLM policy search in constrained
alignment, iterative primal-dual methods often fail to converge, and non-iterative
dual-based methods do not achieve optimality in the LLM parameter space. To
address these challenges, we employ Lagrangian duality to develop an iterative
dual-based alignment method that alternates between updating the LLM policy via
Lagrangian maximization and updating the dual variable via dual descent. In theory,
we characterize the primal-dual gap between the primal value in the distribution
space and the dual value in the LLM parameter space. We further quantify the
optimality gap of the learned LLM policies at near-optimal dual variables with
respect to both the objective and the constraint functions. These results prove that
dual-based alignment methods can find an optimal constrained LLM policy, up to
an LLM parametrization gap. We demonstrate the effectiveness and merits of our
approach through extensive experiments conducted on the PKU-SafeRLHF and
Anthropic HH-RLHF datasets.
1
Introduction
Large language models (LLMs), built upon the transformer architecture [43], have become a core
tool for a wide range of natural language processing tasks (e.g., code generation [17], translation [50],
robotics [37]). Central to these remarkable capabilities ...