ICLR 2025

Published as a conference paper at ICLR 2025
MORAL ALIGNMENT FOR LLM AGENTS
Elizaveta Tennant
University College London
University of Bologna
l.karmannaya.16@ucl.ac.uk
Stephen Hailes
University College London
s.hailes@ucl.ac.uk
Mirco Musolesi
University College London
University of Bologna
m.musolesi@ucl.ac.uk
ABSTRACT
Decision-making agents based on pre-trained Large Language Models (LLMs)
are increasingly being deployed across various domains of human activity. While
their applications are currently rather specialized, several research efforts are un-
derway to develop more generalist agents. As LLM-based systems become more
agentic, their influence on human activity will grow and their transparency will de-
crease. Consequently, developing effective methods for aligning them to human
values is vital.
The prevailing practice in alignment often relies on human preference data (e.g., in
RLHF or DPO), in which values are implicit, opaque and are essentially deduced
from relative preferences over different model outputs. In this work, instead of
relying on human feedback, we introduce the design of reward functions that ex-
plicitly and transparently encode core human values for Reinforcement Learning-
based fine-tuning of foundation agent models. Specifically, we use intrinsic re-
wards for the moral alignment of LLM agents.
We evaluate our approach using the traditional philosophical frameworks of Deon-
tological Ethics and Utilitarianism, quantifying moral rewards for agents in terms
of actions and consequences on the Iterated Prisoner’s Dilemma (IPD) environ-
ment. We also show how moral fine-tuning can be deployed to enable an agent to
unlearn a previously developed selfish strategy. Finally, we find that certain moral
strategies learned on the IPD game generalize to several other matrix game en-
vironments. In summary, we demonstrate that fine-tuning with intrinsic rewards
is a promising general solution for aligning LLM agents to human values, and
it might repr...