ICML 2025

Preference Learning for AI Alignment: a Causal Perspective
Katarzyna Kobalczyk 1 Mihaela van der Schaar 1
Abstract
Reward modelling from preference data is a
crucial step in aligning large language models
(LLMs) with human values, requiring robust gen-
eralisation to novel prompt-response pairs. In
this work, we propose to frame this problem in
a causal paradigm, providing the rich toolbox
of causality to identify the persistent challenges,
such as causal misidentification, preference het-
erogeneity, and confounding due to user-specific
factors. Inheriting from the literature of causal
inference, we identify key assumptions neces-
sary for reliable generalisation and contrast them
with common data collection practices. We illus-
trate failure modes of naive reward models and
demonstrate how causally-inspired approaches
can improve model robustness. Finally, we out-
line desiderata for future research and practices,
advocating targeted interventions to address inher-
ent limitations of observational data.
1. Introduction
The remarkable success of LLMs lies partially in their abil-
ity to align with human values, producing responses that
are helpful, harmless, and honest. A central method for
achieving this alignment is reinforcement learning from hu-
man feedback (RLHF), with the LLM’s behaviour shaped
by reward models derived from datasets of human prefer-
ences (Ouyang et al., 2022; Bai et al., 2022). The reward
modelling, aka preference learning stage seeks to address a
seemingly straightforward question: Given two responses
to the same prompt, which one aligns better with human
objectives? However, the challenges of this stage are of-
ten underestimated, assuming that simple regression-based
models fitted to observational datasets can generalise effec-
tively to unseen texts. Meanwhile, evidence suggests this
optimism may be misplaced (Tien et al., 2023; Skalse et al.,
1Department of Applied Mathematics and Theoretical Physics,
University of Cambridge, United Ki...