ICML 2025

Learning Utilities from Demonstrations in Markov Decision Processes
Filippo Lazzati 1 Alberto Maria Metelli 1
Abstract
Although it is well-known that humans com-
monly engage in risk-sensitive behaviors in the
presence of stochasticity, most Inverse Rein-
forcement Learning (IRL) models assume a risk-
neutral agent. As such, beyond piq introducing
model misspecification, piiq they do not permit
direct inference of the risk attitude of the ob-
served agent, which can be useful in many ap-
plications.
In this paper, we propose a novel
model of behavior to cope with these issues. By
allowing for risk sensitivity, our model allevi-
ates piq, and by explicitly representing risk at-
titudes through (learnable) utility functions, it
solves piiq.
Then, we characterize the partial
identifiability of an agent’s utility under the new
model and note that demonstrations from multi-
ple environments mitigate the problem. We de-
vise two provably-efficient algorithms for learn-
ing utilities in a finite-data regime, and we con-
clude with some proof-of-concept experiments to
validate both our model and our algorithms.
1. Introduction
The ultimate goal of Artificial Intelligence (AI) is to con-
struct artificial rational autonomous agents (Russell &
Norvig, 2010). Such agents will interact with each other
and with human beings to achieve the tasks that we assign
to them. In this vision, a crucial feature is being able to cor-
rectly model the observed behavior of other agents. This
allows a variety of applications: descriptive, to understand
the intent of the observed agent (Russell, 1998), predictive,
to anticipate the behavior of the observed agent (potentially
in new scenarios) (Arora & Doshi, 2021), and normative,
to imitate the observed agent because they are behaving in
the “right way” (Osa et al., 2018).
Nowadays, Inverse Reinforcement Learning (IRL) pro-
1Politecnico di Milano, Milan, Italy. Correspondence to: Fil-
ippo Lazzati <filippo.lazzati@polimi.it>.
Proceedings of th...