From Data to Rewards: a Bilevel Optimization Perspective on Maximum
  Likelihood Estimation
Authors: name: Abdelhakim Benechehab; name: Gabriel Singer; name: Corentin Léger; name: Youssef Attia El Hili; name: Giuseppe Paolo; name: Albert Thomas; name: Maurizio Filippone; name: Balázs Kégl
Published: 2025-10-08 23:45:37

Abstract: Generative models form the backbone of modern machine learning, underpinning
state-of-the-art systems in text, vision, and multimodal applications. While
Maximum Likelihood Estimation has traditionally served as the dominant training
paradigm, recent work have highlighted its limitations, particularly in
generalization and susceptibility to catastrophic forgetting compared to
Reinforcement Learning techniques, such as Policy Gradient methods. However,
these approaches depend on explicit reward signals, which are often unavailable
in practice, leaving open the fundamental problem of how to align generative
models when only high-quality datasets are accessible. In this work, we address
this challenge via a Bilevel Optimization framework, where the reward function
is treated as the optimization variable of an outer-level problem, while a
policy gradient objective defines the inner-level. We then conduct a
theoretical analysis of this optimization problem in a tractable setting and
extract insights that, as we demonstrate, generalize to applications such as
tabular classification and model-based reinforcement learning. We release the
code at https://github.com/abenechehab/nll_to_po .