Risk-Sensitive Theory of Mind: Coordinating with Agents of Unknown Bias using Cumulative Prospect Theory
Authors: Mason O. Smith
Wenlong Zhang
Venue: ICML 2025 poster (2025)
Keywords: Human-autonomy teaming
reinforcement learning
risk
cumulative prospect theory
Primary area: reinforcement_learning->multiagent
TL;DR: A novel framework called Risk-Sensitive Theory of Mind to coordinate with agents of unknown bias.

Abstract: Humans are often modeled as rational actors by interactive agents when they are in fact frequently observed to make biased decisions. This erroneous assumption may cause an agent’s model of the human to fail, especially when interaction occurs in bias-inducing settings that prompt risky decisions. To address this, this paper formulates a risk-sensitive multi-agent coordination problem and presents the novel Risk-Sensitive Theory of Mind (RS-ToM) framework that allows an autonomous agent to reason about and adapt to a partner of unknown risk-sensitivity. In simulated studies, we show that an agent with an RS-ToM is able to better coordinate with such a partner when compared to an agent that assumes their partner is rational. Thus, we observe significant improvements to team performance, coordination fluency, compliance with partner risk-preferences, and predictability. The presented results suggest that an RS-ToM will be able to model and plan with partners that exhibit these risk-sensitive biases in the real world.

Kisk-Sensitive Theory of Mind: Coordinating with Agents of Unknown Bias using Cumulative Prospect Theory
Mason O. Smith
Wenlong Zhang
Abstract
Humans are often modeled rational actors by interactive agents when they in fact frequently observed to make biased decisions. This erro- neous assumption may cause agent's model of the human to fail, especially when interaction occurs in bias-inducing settings that prompt risky decisions. To address this, this paper formulates risk-sensitive multi-agent coordination problem and presents the novel Risk-Sensitive Theory of Mind (RS-ToM) framework that allows tonomous agent to reason about and adapt to a partner of unknown risk-sensitivity. In simulated studies, we show that an agent with an RS-ToM able to better coordinate with such partner when compared to an agent that assumes their partner is rational. Thus, we observe significant improve- ments performance, coordination fluency, compliance with partner risk-preferences, and pre- dictability. The presented results suggest that an RS-ToM will be able to model and plan with part- ners that exhibit these risk-sensitive biases in the real world.
Optimistic
Beller play Pessimiatic
Align with Optimistic Strategy
Align with Pessimistic Strategy
Puddle
Agent
Partner
Plates
Pot
Delivery
Figure 1. Illustration of (robot) agent reasoning about how its (human) partner may hold irrationally optimistic or pessimistic perceptions about entering a puddle, where they risk slipping and losing their held item, and the best way to align its strategy to improve team coordination in Risky Overcooked task.
1. Introduction
risk-averse) about their prospects when faced with risky decision (Kahneman & Tversky, 2013). For example, con- sider the task in Fig. which demonstrates two strategies. An optimistic human would likely prefer the purple strategy that risks traversing puddle states since optimism discounts the negative consequence of slipping. Conversely, pes- simistic human would likely prefer the orange strategy and take an unnecessarily long detour to avoid any risk at all. An autonomous agent that can reason about these preferences can then deploy strategies that best aligned with (biased) human values improve coordination.
Autonomous agents have traditionally modeled their hu- man partners as noisy-rational actors (Baker & Tenenbaum, 2014) and think in approximately the same way that ma- chines do. While convenient, this can be an erroneous as- sumption as humans often display behavior that systemically deviates from rational strategies. This referred to bi- ased decision-making (Kahneman & Tversky, 1974). Most relevant to the proposed work, humans tend to be either irrationally optimistic (i.e., risk-seeking) or pessimistic (i.e.,
School of Manufacturing Systems and Networks, Arizona State University, Mesa, Arizona USA. Correspondence to: Mason O. Smith mmosmith @da
Humans said to have risk-sensitive preferences that vio- late axioms of rational decision-making (Briggs, 2023) that noisy-rationality relies upon. When violated, this leads to the inability for an agent to understand human preferences and accurately predict their actions. Thus, team perfor- mance can suffer when inadequate consideration of human
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Risk-Sensitive Theory of Mind
risk-sensitivity is given in settings that prompt such biases to occur (Kwon et al., 2020). Consideration of risk-sensitivity may also impact interpersonal dynamics like trust (Smith & Zhang, 2025), a critical feature of -utooomy in- teraction (HAT). In fact, risk-sensitivity was chosen as the target for improving an agent's model of their partner due to its unique connection to trust: trust requires both risk and truster (human) valnerability to be present (Lee & Moray, 1992). Therefore, every trusting situation can prompt risk- sensitive biases to occur. Automation that is then able to infer and reason about this relationship may then be af- forded a more nuanced approach to closing the loop with trust calibration in HAT.
While these approaches are good step towards better aligning autonomy and biased human values, they all train policies under the median CPT parameter estimates from (Tversky & Kahneman, 1992) which generally describe risk- averse behaviors. Thus, in practice, they align with a single type of risk-sensitivity prescribed to the human and emulate single, fixed mode of bias. This fails to capture a powerful aspect of CPT that distinguishes from other measures like CVaR: CPT can model variety of risk-sensitive modes including risk-seeking tendencies. This may be important to consider as human risk-sensitivity can vary between in- dividuals and contexts (Kahneman & Tversky, 2013) and learned coordination strategies can drastically change based on how CPT is parameterized (Ferreira et al., 2021). Thus, an agent afforded the ability to reason over several modes of risk-sensitin behaviors (i.e. has an RS-ToM) can adapt online to different contexts and personalize on individual level during interaction.
Motivated by these opportunities from improving upon the assumption of noisy-rationality, this paper formulates risk- sensitive multi-agent coordination problem where an au- tonomous agent must coordinate with partner of unknown risk- risk-sensitivity. We then attempt to solve this problem by reasoning about the partner's biased preferences as different than our own, referred to as having a Theory of Mind (ToM) (Devin & Alami, 2016), and having the agent align with its partner's s preferences. An ToM classically reasons over dis- crete set of intentions subtasks (Wu et al., 2021) as com- pact way to define preferences within the context of a single task. We take a more abstract perspective that can ubiqui- tously model preferences in any risky task on a continuous spectrum by varying model parameters. Therefore, when specifically reasoning about the partner's risk-sensitive pref- erences, we refer to this as an Risk-Sensitive Theory of Mind (RS-ToM). Here, we seek to evaluate 1) how to overcome learning challenges of risk-sensitive coordination problems in complex, multi-agent settings, 2) what, if any, improvements to team coordination are present in contrast to the aforementioned assumption of noisy-rationality,, and 3) what are the implications of the simulated results on HAT in terms of team coordination and trust?
Works on personalized risk-sensitive models are limited. In (Kwon et al., 2020), the authors collected a data-prion to formulate a personalized model of human risk-sensitivity with CPT in a collaborative cup stacking task. Notably, they showed that robot endowed with a personalized model can improve team performance and perceived trustworthiness of the robot by better aligning with risk-sensitive prefer- ences. Similarly, (Sun et al., 2019) learned CPT-based utility functions using inverse reinforcement learning (IRL) in a roundabout driving task. Constraining IRL with learned CPT parameters unlocks personalized explanations of irra- tional behaviors not available under noisy-rationality that lead to improved human- preddictteiiiionn accuracy and coordina- tion. Alternatively, the authors in (Cheng et al., 2023) used interactive querying to estimate CPT parameters. The main limitation that these works share is thr reliance on a human data-prior which may be expensive or infeasible to collect, especially when it requires human risk be present. Addi- tionally, these approaches can only account for differences between humans, not how each's 's risk- risk-sennttyvity changes between contexts or over time.
Related Works. An RS-ToM draws inspiration from risk- aware control settings that bound an agent's 's future exposure to risk by applying pessimistic risk measures like condi- tional value-at-risk (CVaR) (Hakobyan et al., 2019; Tan et al., 2022). More closely related to the proposed work is the subset of approaches that apply more generalized risk measures like Cumulative Prospect Theory (CPT) that better align with empirical observations of human risk-sensitivity (Tversky & Kahneman, 1992). Works that apply CPT in sequential decision tasks tend to impart prototypical risk- sensitivity onto an agent such that it behaves more "human- like" and its actions can be better understood by humans TRamasubramanian et al., 2021; Prashanth et al., 2016). Closest in formulation to an RS-ToM, the authors of (Danis al., 2023) applied CPT in multi-agent setting using Nash Q-Learning to generate risk-sensitive joint strategies in simple gord-world navigation task.
Lastly, works on value-based multi-agent reinforcement learning (MARL) with CPT tend to implement tabular al- gorithms (e.g., value iteration (Ferreira et al., 2021; Kwon et al., 2020), Q-learning (Danis et al., 2023), or SARSA (Ramasubramanian et al., 2021)) to solve low-dimensional learning problems. This limits task complexity and the abil- ity to learn in real-world settings. Thus, there is gap in existing value-based methods that are able to solve these problems as deep learning approaches are not well studied in this space.
Our Approach.
As prior works have shown that adaptation to human risk- preferences can benefit interaction, this paper focuses on
2

---

Risk-Sensitive Theory of Mind
algorithmic contributions that address the aforementioned challenges (i.e., data prior requirements enable personal- ization and the limited task complexity afforded with tabular methods) where we seek to validate with real humans in future studies. To the best of the authors' knowledge, we are the first to formulate an RS-ToM that affords adaptation to a partner with unknown risk-sensitivity in a zero-shot fashion and without the need for costly data-priors. We achieve this by first drawing from existing work in risk-sensitive reinforcement learning (Danis et al., 2023) to train several candidate policies conditioned on a space of CPT param- eters. This is a similar goal to using IRL to recover sub- optimal value functions (Rothkopf& & Dimitrakakis, 201 Bergerson, 2021) where we instead take the forward ap- proach that avoids data requirements by prescribing agents CPT-based evaluation of task objectives. Then, when interacting with a partner of unknown risk-sensi,,vity, agent will infer which candidate policy best describes the current partner's behavior and deploy the appropriate policy in response. Doing so, we enable the autonomous agent to align its preferences with different types of sensitiiityv so that it can better coordinate. To that end, the contributions of this paper are follows:
model and novel replay memory for improved tractabil- ity. The last component is what affords an agent with an RS-ToM and will describe inference over and response to space of pre-trained risk-sensitive policies. The remainder section will formulate these three steps while a high-level summary of this framework can be found in Fig. 2.
2.1. Cumulative Prospect Theory
Let X' denote the support of discrete random variable with probability pa of having utility X; s.t. [R. E X. Following (Tversky & Kahneman, 1992), we then define the transformations on utility and probability.
Definition 2.1.
Let u ()) and u' (.) denote the utility trans- formations applied to what the agent perceives as gains and losses, respectively, and be defined as:
" (xx) (Xi - -))"
(I)
2 (x. = E b1"
where n. E [0,1] account for exponential discounting of utility it moves farther from the reference point (-o0, and f E [0, oo) asymmetrically weights losses. Definition 2.2. Let w'() (-) and w (.) denote the probability transformations applied to what the agent perceives as gains and losses, respectively, and be defined as:
1. To our knowledge, we are the first to integrate CPT into deep multi-agent risk-sensitive reinforcement learn- ing (MARSRL) to induce biased behavior in complex risky-decision tasks.
w (p=) = (V + - (1--i) p:)0'1216
2. We formulate novel memory buffer and apply a level- k quantal response equilibrium to overcome tractability issues in this deep multi-agent setting with CPT.
(2)
w" (pr) (pf (1 - 1:30
3. We then leverage our MARSRL algorithm to enable an RS-ToM and demonstrate its ability to achieve su- perior team performance, coordination fluency, risk taking aligmment, and partner predictability relative to noisy-rational baseline when interacting with a partner of unknown risk-sensitivity in our novel Risky Overcooked benchmark task.
where 5",5" E [0, apply inverted S-shaped transforma- tions on probabilities that overestimates small probabilities and underestimates large probabilities. Examples of these transformations and more intuitive descriptions of each pa- rameter can be found in Appendix B.
In order to deal with more than two prospects, expectations are carried out over the cumulative probability distributions.
The remainder of this paper is as follows. Section 2 pro- vides a formulation of the RS-ToM. Section 3 details the experimental procedure including description of the Risky Overcooked task (Sec. 3.1), analysis methods (Sec. 3.2), and results (Sec. 3.3). Results and implications are discussed in Sec. and concluding remarks are given in Sec. 5.
Definition 2.3. Let X be arranged in increasing order of X4 s.I. X, < X2 S ... < XK and be the index where X, S b. Also, define FB Ek Pi V k I and Σ Ps V k > I as the cumulative probability distributions for gains and losses, respectively. We then define CPT-expectation of X as (2);
2. Risk-Sensitive Theory of Mind
Pxpe(X) = K-
An RS-ToM is derived of three components. First, we will define CPT as the risk measure that induces risk-sensitive evaluation of prospects. Next, we will define the approach to the MARL problem and how it is extended to MARSRL using CPT where we leverage knowledge of the transition
Σ 1=++1
a (X))( (FF)) w² (F(+))))
+ +u' (Xx) (px)
(u (8))) (pz) Σ ni (2))(w (F.) w (F(-1)) (3)
3

---

Risk-Sensitive Theory of Mind
MARSRL (Offline)
Inference (Online)
Prospect Replay Memory
DDQN Qest
{irs No, xA)
Belief Update (9)
viVi
Next State Prospects
~ Kayec (8)
(s,,,,s'))
(3())
aA
a
Qe à)
TD-Target Prospects ((T,P:) v/3
(()
Environment
QRE
(8,188
r CPT-Value
Environment
Psx(T)
C
Rollout
Loss
CPT
Candidate Policies
Rollout
Figure 2. Illustration of the RS-ToM framework. The offline MARSRL algorithm trains a space of risk-scnsitive candidate policies conditioned on the CPT-value transformation (T) The red lines show the novel replay memory that stores all possible next states sf and their probability pc. During the update of Qept. the value (S)) of each prospective state 6 is calculated from the expectation over the level-k quantal response equilibrium (QRE) policy and the forward pass of DDQN Qept (s)). An agent arbitrary fisk-sensitivity can he generated by tuning the CPT parameters that defines the transformation in the red shaded region. Thus, any number of candidate policies can be trained such that an agent can reason about its partner's risk-sensitivity online, in zero-shot fashion, and without the need for prior interaction data by performing belief update over the space of pretrained candidates.
where u", at R- R4 are continuous, have a bounded moments s.t. u (x,)) 0 v X S b and u (2)) O Y & >> b and are monotonically non-decreasing oth- erwise. Also, we assume the probability weighting fune- tions w²,
over all possible next states sf S:
Q(s.a) = Ex 7V |T]]
= =r(s, +
Σ
P(s a))) T)
(5)
".""" w",w" (Prashanth et al., 2016) and non-decreasing such that they satisfy w' (0), wi (0) and w" (1), w (1)
1] + [0, I] are Lipschitz continuous
ES
We can then apply dynamic programming methods itera- tively update Q(s, a) such that:
2.2. MARSRL Algorithm
Q(s,a) + Q a) + Q(s,a))
(6)
Definition 2.4. An MDP is the tuple M := (S,A,P,) where S S x S-k is a finite set of joint-states for agent i and their partner -k, A: Ag X A. is the finite set of joint- actions that are sampled from the joint-policy P(ss' s,a) S x A x S is the probability of transitioning to state s' when action a is taken in state and r(s, a) is the stochastic reward function.
where T == r(s,a) + E.Ees P(s 8, s,)) | T) is refereed to as the temporal difference target (TD-Target) and a is the learning rate. In multi-agent settings, we recover the joint-policy x(a 8) = (и,, T-x), where p(a 8) = TTy(ak s)=-x(a= s). by solving each transition single-stage, common- common-payoff game Q(s) using the Quantal Response Equilibria (QRE) (see Appendix A.4 for details), This reinforcement learning algorithm and its following risk- sensitive version can be arbitrarily extended to n > 2 agents with appropriate factorization of the state-action space and equilibrium solution.
Ta T-k-
The goal of standard reinforcement learning is then to find policy æ that maximizes an objective Σ' B[)'r(s,,) a) T] over some time horizon T where E (0, 1) is a discounting factor weighting more immediate rewards as more important (Sutton et al., 1998). We specify a policy by the quality of a state-action pair ((s, a) where π (s, a) xx earp)Q((s, a)) and \ defines the decision temperature of policy. Given that the value of a state can be written as V(s m) EoeA TTa s)Q(s,a), a), we can define Q(s, a) as:
CPT- The previous formulation can now be modified with CPT by treating the TD-target the random variable X, in (3) by defining:
T = T a(), + +
P: = PPs 8,
((s,a) = r(s, +VV(s' T))
(4)
where pt is the chance of observing as the TD-target and is the index of each possible next state s given action a was taken in state s. While other works employ a sample- based approximation of CPT-value to maintain a model-free approach (Ramasubramanian al., 2021; Danis et al., 2023;
Given that states stochastically evolve according to PPss' 8, a), we can generalize (4) to the following expected value

---

Risk-Sensitive Theory of Mind
Prashanth al., 2016), we instead leverage knowledge of the environment for improved tractability. Thus, we directly compute the CPT-value Pope (T) over the finite set of possi- ble TD-targets T s.t. (r J using (3) and rewrite the update from (6) in its risk-sensitive form:
Algorithm Deep MARSRL with CPT
Input: exploration rate < 1, soft update rate a << 1, and parametrized CPT-value functional Pepe(.)
Initialize: Policy network Q target network Q and replay memory D = 0
for episode = 1, ,,... M do
as) t Qepe(s, a) + a ((Pep((T) - (cpe(s,a)) (8)
Observe initial state
while t T do
Sample action Tepe
Deep Learning: While similar work has been conducted in simple, multi-agent settings using tabular methods (Da- nis et al., 2023; Ferreira et al., 2021) and more complex, single-agent settings using deep learning methods (Rama- subramanian al., 2021; Rajabi al., 2022), our work focuses complex multi-agent tasks which calls for deep learning. We therefore solve (8), by applying Algorithm which is based on a Double Deep Q-Network (DDQN) (van Hasselt et al., 2016) that keeps track of separate policy O% and target networks Qept defined with weights on & and o, respectively. Selected hyperparamters for learning can be found in Appendix A.3.
Observe next state s' and reward r
Get next state prospects s' := (1s : S} Store transitions in memory D t (s, a. r,s Ak Sample a minibateh of N transitions from D for all (8, a, E minibatch do opt Tept). p.} (13%, pi} E $
T = {ff + + yV ($ Cn := (Pcpa(T) - ( ())
end for
N Perform gradient decent on I with 1 E Cn Soft target update & t o0 + (1 o)Φ 5'
end while end for
However, CPT induces additional computational challenges in deep setting. In contrast to a rational expectation, CPT increases complexity by a factor of +n2 +nloggn nlog(n) where is the number of prospects included the expectation'. Also, CPT-value requires computation of all possible next states for each minibatch sample' which is only exacerbated by computation of equilibrium solutions. To overcome this, we first apply the QRE as an efficient approximation of mixed Nash equilibrium solution. We then formulate a novel replay memory D that, instead of storing the traditional ob- served transition (s, a, r, s'), stores all transition prospects in the form of (s, a, r, where Ps) E s' is all possi- ble next-states. Thus, given we update every timestep, we compute possible next states once instead of N times.
2.3. Inference
The inference step is what imparts the agent with an RS-ToM as it allows it to track its partner's 's risk-sensitive preferences independently from its own. Bayesian infer- ence performed over the past 10 observations O
{s, (s. .... {s, a- to form belief b(.) about which joint-policy π E π the human -k is following:
b( O) + PPO T (((T))
(9)
Σ FE
POO T)((T)
To further aid tractability in the later Risky Overcooked task, we apply curriculum learning and reward shaping strategies (see Appendix A.2 for details) to deal with sparse rewards. Moreover, due the symmetrical roles of the task, leverage self-play methods (i.e. single Q-function for both agents) where transition prospect is stored from each agent's 's perspective D <- (s, a, s')"), V k at every timestep to improve sample efficiency.
where b( O) is the updated belief over all possible joint- policies after observing O. Here, we add a decaying weight to the likelihood IPO T) of observations further in the past'. The artificial agent i then discretely adopts joint- strategy believes is most likely s.t. क. E argmax,, b(=[O). This what frames the approuch to coordination risk- sensitive alignment problem*.
We establish the convergence of Algorithm 1 by showing the update of Qept will be a contraction given appropriate choice of learning parameters. Also, we rely on a conserva- tive upuer- assumption on l that ensures convergence of the Bellman operator (see Appendix A.5 for details).
3. Experiments
The following experiments investigate RS-ToM's ability to adapt to a partner of unknown sissisensitivity. Due to the novelty of an RS-ToM, there a lack of feasible baseline algorithms that can adapt partner risk-sensitivity in a zero-
rational expectation has linear complexity n whereas sorting prospects has at best nlog(n) complexity (Alkharabsheh et al., 2013), calculation of Fx is n² and (1)-() all have complexity. TThis leads to N more next state computations than tabular methods per Q-value update where N is the minibatch size.
Thii mirrors future human studies where recent observations are more representative of possibly dynamic risk-preferences. 'More sophisticated methods such (Wang et al., 2025) required for value alignment with n > 2 agents.
5

---

Risk-Sensitive Theory of Mind
shot manner in complex settings. This stems from existing works for adapting to risk-sensitive preferences relying on data-priors collected from partner (Kwon et al., 2020; Sun et al., 2019) and state of the art MARSRL algorithms (Da- nis et al., 2023; Ferreira et al., 2021) not being tractable in complex tasks like Risky Overcooked. However, we can evaluate performance in contrast with the standard assump- tion of noisy-rationality (RAT) to glean insight on expected outcomes when interacting with real humans.
low-level actions and lossless state (see Appendix A. for details) that drastically increases complexity.
Design of Risky Overcooked layouts can be used to frame coordination in two ways where we select one of each fram- ing for later experiments. First, the riskyrely framing pro- poses a choice between the level of acceptable risk and the reliance between partners. This is achieved by designing subtasks require puddle state traversal or coordinate hand- offs between agents. This framing couples risk- iisk-senitiivity closely with trust, as reliance is a behavioral outcome of trust (Lee & See, 2004), which makes it a promising setting for future human experiments. In the selected Risky Coor- dination Ring (RCR) layout, seen in the left half of Fig. 3, agents can either pass items through the center counter space to avoid all puddles (A and in Fig. 3) or traverse puddles to complete subtasks independently (C and D in Fig. 3). Here, we see that a risk-averse strategy facilitates strong reliance between partners by always passing objects due to pessimistic perceptions of slipping while a risk-seeking strategy favors avoiding costs of imperfect handoffs due to optimistic perceptions.
3.1. Risky Overcooked
While there exists tasks such as traffic control (Prashanth al., 2016) and multi-agent navigation (Danis al., 2023; Tian et al., 2021), there appears to be no standardized risky decision benchmark that simultaneously incorporate shared awareness (mental models), a strong interdependence be- tween teammates, optional reliance on one's partner, and explicit risk from the environment. This gap in standardized platforms is a common theme in HAT (Smith et al.. 2024) that seek to address by proposing Risky Overcooked. Risky Overcooked is an open-source, risky coordination benchmark that extends the popular Overcooked environ- ment (Carroll et al., 2019) by incorporating risky decisions into team coordination.
A Alternatively, the risk/detour framing proposes a choice between the level of acceptable risk and the additional nav- igation costs incurred by taking a detour around puddle. In the selected Risky Multi-Path (RMP) layout, seen in the right half of Fig. 3, agents have three route choices: 1) a direct and high-risk route that traverses two puddles, 2) a slightly longer route that only requires traversal of one pud- dle, or 3) a long detour that requires no risk to be taken as it avoids all puddles. Here, we see that a risk-averse strategy tends to avoid puddle traversal by incurring higher naviga- tion costs (ES and F in Fig. 3) while risk-seeking strategy prefers to incur additional risk in favor of more direct routes (G and H in Fig. 3).
In the standard Overcooked game, two agents must coor- dinate their behaviors cook soup as fast as possible by completing three subtasks: 1) bring three onions to a pot at which time the soup will begin cooking, 2) bring a plate to the pot with a cooked soup to obtain a plated soup, and 3) bring the plated soup to a delivery window to receive a reward of Tsoup = 20. Agents cannot pass through each other, can only carry one object at a time, and can choose to place objects on any free counter space.
Overcooked task does not explicitly elicit risky decisions from the environment. Therefore, we propose the modi- fication seen in Fig. 1 where we add risky puddle states. When an agent enters risky state, they have Pp chance of slipping where they will lose whatever resource they are holding. The rationale of how values for Pp are selected can be found in Appendix B.2. This frames the consequence of slipping as a forgone gain where we add a small time- cost Tıe = -0.2 at every timestep to elicit perceptions of loss if soups are not cooked fast enough. These risky states are intended to induce diverging and unique strategies for agents of different risk-sensitivittis. Figure 3 shows how risk-averse teams tend to pessimistically avoid puddle states while risk- risk-seeking teams optimistically traverse them (see Appendix B.3 for more examples).
3.2. Analysis
A space of three joint-policy candidates ा containing risk- seeking ny, risk-ncutral (i.c. rational) Tys and risk-averse A strategies are trained with the MARSRL algorithm us- ing CPT parameters motivated by median estimates from (Tversky & Kahneman, 1992) (see Appendix B.1 for further description). We then evaluate the difference between an agent with an RS-ToM and a RAT assumption when inter- acting with a partner of unknown risk-sensitivity. We also provide results contrasted against an Oracle baseline where the ground truth risk-sensitivity of the partner is known to show theoretically optimal results. The rest of this sec- tion will describe the four measures used to evaluate these experiments.
Due to risk-sensitivity presenting itself as a choice between routes, we cannot rely on the mid-level action planners and state featurizations (Carroll et al., 2019) commonly used to simplify Overcooked. Instead, we are required to use
Trravesal of puddles is caused by the other agent blacking the risk-averse route when picking up dish.
6

---

Risk-Sensitive Theory of Mind
Onions
Dishes
Onions
Dishes
Averse (TT
Aversc (IT
Seeking (TT
Seeking (TT
Passealle
les
(C)
(D)
(G)
(H)
Risky Coordination Ring (RCR)
Risky Multi-Path (RMP)
Figure 3. Illustration of successful generation of various joint strategies with the proposed MARSRL algorithm by showing the state visitation frequencies of the onion and dish objects carried by either agent where higher frequencies are indicated by increasingly red centroids in each tile. The top row shows the learned Averse strategy while the bottom row shows the Seeking Ts strategy for both the RCR and RMP layouts. Soup visitation frequencies (excluded) always follows path that avoids puddles due the high cost slipping being irrational, regardless of risk-sensitivity, The 2 3 area in (G) shows the space where agents may freely pass by each other.
Team performance in the Risky Overcooked task will be evaluated the cumulative reward of the team Dr. The time-cost penalty Ite is removed during calculation for sim- pler analysis where Er is always positive.
The predictability of the risk-sensitive partner Pv is ob- served evaluate the afforded ability for the agent to pre- dict future actions of the risk sensitive partner and plan coordinated strategies. This is defined as the probability that the risk-sensitive partner takes the maximum likelihood action under the agent's (inferred) model of partner's risk- sensitivity. We are primarily interested in improving the predictability of the risk-sensitive partner and therefore do not evaluate their predictability of the other agent.
Coordination fluency is the "coordinated meshing of joint activities of members of a sellhsynchro team" and will be evaluated with concurrent activity C-ACT (Hoffman, 2019) defined the proportion of task time where both agents are active simultaneously. A high C-ACT indicates a fair work balance and good synchronization of strategies. In Risky Overcooked, an agent is considered inactive while remaining in same state or during periods that result in the return previous state without completion of subtask".
3.3. Results
The mean values of N 1000 trials for the Oracle, RS-ToM, and RAT algorithms interacting with Seeking or Averse partner across the RCR and RMP layouts can be found Table 1. Bold values indicate the superior perfor- mance between RS-ToM and RAT.
The number of risks taken by the team Nrisk accounts for how well an agent aligns their partner's risk-taking pref- erences. When interacting with a human, they would likely judge an agent's strategy as inappropriate if the amount of acceptable risk realized by the team is not properly cal- ibrated to their perceptions. For example, risk-averse human may view an agent taking the optimal amount of risk as "too risky" while risk-sceking human may view it "too safe."
A 2 ANOVA, with factors of partner risk risk-sensitivity (Seeking and Averse) that is unknown to the ego agent and used to evaluate significance within each layout. Both main effects and all interaction effects were sigmificant for all measures. A Tukey post-hoc test revealed all but five significant com- parisons that will mentioned when relevant. Unless stated otherwise, tests had significance level of < 0.02. As ideal values of measures depends on the layout, results are normalized with percent change from RAT to RS-ToM for easier discussion.
2) the ego agent's algorithm (RS-ToM RAT),
This accounts an agent moving back forth having to retrace their steps in order to get out of the way of their partner in a tight corridor. In both cases, the behavior is not productive and equivalent to agent waiting in the same state.

---

Risk-Sensitive Theory of Mind
Table 1. Numerical Results from Risky-Overcooked Experiments
Σr
C-ACT
Nrisk
Pi
Layout Partner
Oracle
RS- ToM
RAT
Oracle
RS-ToM
RAT
Oracle
RS-ToM
RAT
Oracle
RS-ToM
RAT
Sceking
151
150
98
0.64
0.64
0.44
25.60
25.93
26.96
0.76
0.76
0.41
RCR
Averse
146
146
70
0.48
0.47
0.26
0.00
0.02
8.42
0.75
0.75
0.26
Seeking
125
122
33
0.40
0.40
0.15
73.08
72.59
29.84
0.65
0.65
0.26
RMP
Averse
100
101
44
0.51
0.51
0.22
14.79
15.07
35.24
0.65
0.65
0.33
Comparison to Oracle. In all experiments, the RS-ToM algorithm performed nearly identically to the Oracle. This is due to the RS-ToM's 's inference correctly converging to the ground truth quickly as seen by Fig 4. The RAT algorithm saw significant losses in all cases except for Nrisk in the RCR with a Seeking partner. Rationale for this is explained in the later "Risk Taking Alignment" results.
(+59%) which is best attributed product of the layout de- sign. Notably, there was larger gains in RCR with Averse partner (+81%) than with a Seeking one (+44%) due to the Averse strategy attempting to avoid puddles through coor- dinated handoff's to the partner (A and B in Fig 3) that are never completed in RAT. Thus, the Averse partner is left with high inactivity when waiting for this to occur.
Risk Taking Alignment. The number of risks taken by the team Nrisk shows that an RS-ToM allows the risk-sensitive partner to follow their preferred risk-taking tendencies. This can be seen by the near identical Nrisk between RS-ToM and Oracle (i.e. the ideal risk-taking tendency to the part- ner). In contrast to Oracle, RAT decreased Nrisk when interacting with a Seeking partner and increased Nyisk with Averse partner. The only exception is for RCR with Seeking partner which is attributed to discoordination in a confined space requiring backtracking into puddles in a way not present in RMP.
RCR
RMP
1.0
Seoking (mn)
0.5
0.0
1.0
Averse (MT))
b(t; (+)
0.5-
biT, bbmt (0)
0.0
0
50
100
150
200
50
100
150
200
Timestep (t)
Timestep
Predictability. The predictability of the risk-sensitive part- ner Ph over doubled (+ 121%) with an RS-ToM on average across all experiments. In RCR, RAT showed more diffi- culty predicting an Averse partner's action while the Seeking partner is more difficult to predict in RMP. Here, Pxx saw +81% % and +188% increase for Seeking and Averse partners, respectively, in RCR and a +147% and +97% increase in RMP. There were non-significnn between Seeking and Averse partners when using RS-ToM in both the RCR (p 0.42) and RMP (p- = 0.82) layouts. This is attributed to equal decision temperatures À for Seek- ing and Averse policies once the correct policy was inferred.
Figure 4. Bayesian inference b(T]| O) converging to correct belief about risk-sceking iTs (top row) risk-averse Ta (bottom row) partner in RCR and RMP layouts.
Team Performance. On average across both risk-sensitive partners, an RS-ToM improved upon the reward Er of RAT by factor + +77% in RCR and +189% in RMP. In RCR, gains were more apparent with an Averse partner (+109%) than a Sceking one (+53%) while the inverse is true in RMP with a percent gain of +127% and +273% for Averse and Seeking partners, respectively. The discrepancy in RCR is primarily caused by the risk/rely framing where failure to pass objects through the center counter space (A and B in Fig 3) comes at a high cost which affords greater opportunity for performance gains with an RS-ToM. Discrepancy in RMP can be attributed to added congestion of risk-seeking paths due to little utilization of the open x 3 space that allows partners to pass each other (see in Fig 3). Post-hoc testing revealed non-significant (p- 0.06) difference in rewards between an Averse and Seeking partner with RS-ToM in RCR but significant difference in RMP.
post-hoc test comparisons
4. Discussion
The experiments with an RS-ToM show the ability to learn diverse, risk-sensitive behaviors in high-dimensional tasks and the ability to correctly adapt an agent of unknown risk-sensitivity online. This enables us to build upon prior works that support risk-sensitive human interaction by af- fording interaction more realistic scenarios (i.c., complex tasks and personalization without a data prior). As CPT parameters describe different biases, this also provides us with an interpretable method for generating arbitrary risk- sensitive behavior in multi-agent, risky decision settings.
Coordination Fluency, On average across all experiments, an RS- ToM improved upon RAT by a factor of 104%. Gains for both partners were larger in RMP (+150%) than in RCR
8

---

Risk-Sensitive Theory of Mind
We then validate the RS-ToM by comparing it the con- ventional assumption of noisy-rationality. We evaluated these algorithms in two different cases (i.e., interacting with risk-seeking or risk-averse partner) where the partner's 's risk- sensitivity was unknown to the agent. Results ubiquitously show that interaction can be improved with RS-ToM and that the standard assumption of rationality in RAT is insuffi- cient for effective coordination to occur. The near identical performance of an RS-ToM and Oracle indicates that the inference step was able to quickly converge to the correct belief about partner risk-sensitivity. Thus, the RS-ToM was able to recover the optimal policy effectively allowing com- parable performance to the ideal strategy.
However, the discrete set of Risky Overcooked strategies is helpful in clearly showing RS-ToM's ability to adapt to risk-sensitive partners, as three candidate policies are sufficient for describing these high-level behaviors. Never- theless, increasing the resolution of the candidate policies may be helpful for scaling to tasks with more a diverse set of risky-decisions (e.g., continuous control problems affording continuous variation in risk-sensitive strategies). Although RS-ToM can be arbitrarily scaled with additional can- didate policies as the task demands, meta-model methods that can continuously interpolate between candidates (Hong et al., 2023) or transfer learning (Weiss et al., 2016) may be a promising direction for increasing RS- ToM resolution.
While these results are performed with an artificial partner, insights on the consequences of an RS-ToM when interact- ing with real humans may be extrapolated. Similar transfer between simulation and human studies is seen in (Smith & Zhang, 2025), albeit with a less dramatic effect due to the human's ability to adapt to misaligned risk-sensitive models. The results showed that RAT tended to cause risk taking strategies that were contrary to the partner's risk preferences (i.e., Averse saw an increase in Nrisk while Seeking saw a decrease) whereas RS-ToM was able to quickly infer and align to its partner's 's preferences to avoid this. Consequently, an RS-ToM would likely avoid deploying strategies that could be perceived as inappropriate (i.e., "too risky" or "too safe") in a way that could damage interpersonal features like trust (Lee & Moray, 1992). The improvements to coor- dination fluency with an RS-ToM is also likely to promote appreciation and confidence by the human (Hoffman, 2019). Thus, we suspect that an agent with RS-ToM would be perceived as a better teammate.
5. Conclusion
Standard approaches to HAT model the human being noisy-rational which can lead to poor coordination when they are subject to risk-sensitive biases. To address this, our work presents a novel RS-ToM framework that gener- ates personalized adaption to a partner of unknown risk- sensitivity without the need for a data prior. a simulated study, we show that an RS-ToM is able to better align itself with its partner's risk preferences and coordinate its behav- ior in way that significantly outperforms noisy-rationale Thus, these results imply that an agent endowed with an RS-ToM would be able to facilitate more effective teaming when paired with real humans.
Software and Data
Training of policies used in the RS-ToM was conducted in Python 3.11 using PyTorch 2.3.1's GPU acceleration. Implementation of the Risky Overcooked environment, the RS-ToM algorithm, and pre-trained models can be found at https: / /gitttu.......//// risky_overcooked/
Limitations and Future Work. The RCR and RMP layouts were selected to maximize diversity across framings. How- ever, the observed context-dependen effects may suggest that there are unobserved sensitvsi behaviors that may emerge with novel layout designs. Moreover, although RCR and RMP were sufficient for demonstrating the RS-ToM algorithm in simulation, future studies with real human partners may want to deploy a wider variety of layouts to ensure that their unique riskisens has the opportunity to present itself in a significant way.
A Acknowledgements
This material is based upon work supported by the Air Force Office of Scientific Research under Award No. FA9550-23- 1-0283.
While we believe that modeling human risk-sensitivity is an essential factor in risky settings, such behaviors could feasi- bly be described by alternative human models like limited planning horizon due to the discrete set of strategies in Risky Overcooked (i.e., traverse puddle, take detour, or handoff object). This overlap will likely fade as biases become more differentiable either through performance of multiple Risky Overcooked layouts or with increased decision complexity. However, multi-modal approach (Kryven al., 2024) to modeling human bias may be valuable future direction.
Impact Statement
This paper presents work whose goal is to advance the field of human-aware reinforcement learning. There are many potential societal consequences of our work, most important of which is machines that understand biased humans so that they can effectively predict, coordinate, and maintain trustworthy relationships in the real world.
9

---

Risk-Sensitive Theory of Mind
References
Hoffman, G. Evaluating fluency in human-robot collabora- tion. IEEE Transactions Human-Machine Systems, 49 (3):209-218, 2019.
Alkharabsheh, K., Alturani, I,, Alturani, A., and Zanoon, D. Review on sorting algorithms a comparative study. International Journal of Computer Science and Security (IJCS)), 7:120-126, 2013.
Hong. J., Dragan, A., and Levine, S. Learning to influ- ence human behavior with offline reinforcement learning. arXiv preprint arXiv:2303. 02065, 2023.
Baker, C. L. and Tenenbaum, J. B. Modeling Human Plan Recognition Using Bayesian Theory of Mind. Plan, Ac- fivity, and Intent Recognition: Theory and Practice, 1()): 177-204, 2014.
Hu, J. and Wellman, M. P. Nash q-learning for general-sum stochastic games. Journal Machine Learning Research, 1(Nov):1039-111 1069, 2003.
Bergerson, S. Multi agett inverse reinforcement learning: Suboptimal demonstrations and alternative solution con- cepts. arXiv preprint arXiv:2109.01 01 178, 2021.
Kahneman, and Tversky, A. Judgment under uncertainty: Heuristics and biases. Science, 185(4157): 1224415):1124-1133 124-1131,, 1974.
Briggs, R. A. Normative theories of rational choice: Ex- pected utility. In Zalta, E. N. and Nodelman, U. (eds.), The Stanford Encyclopedia of Philosophy. Metaphysics Research Lab, Stanford University, Winter 2023 edition, 2023.
Kahneman, D. and Tversky, A. Prospect theory: An analysis decision under risk. In Handbook of the Fundamentals of Financial Decision Making, chapter pp. 99-127. World Scientific, 2013.
Kryven, M., Yu, S., Kleiman- Weiner, M., Ullman, T., and
Tenenbaum, J. Approximate planning in spatial search.
Carroll, M., Shah, R., Ho, M. K., Griffiths, T., Seshia, S., Abbeel, P., and Dragan, A. On the utility of learning about humans for human-ai coordination. Advances in Neural Information Processing Systems, pp. 5175-5186, 2019.
PLOS Computational Biology, (0(11):1-2) 11 2024.
Kwon, M., Biyik, E., Talati, A., Bhasin, K., Losey, D. P., and Sadigh, D. When humans aren't 't optimal: Robots that collaborate with risk-aware humans. In International Con- ference Human-Robot Interaction, pp. 43-52, 2020.
Cheng, Z.. Coache, A., and Jaimungal, S. Eliciting risk aver- sion with inverse reinforcement learning via interactive questioning, arXiv preprint arXiv:2308.068: 084427, 2023.
Lee, J. and Moray, N. Trust, control strategies and allocation of function in human-machine systems. Ergonomics, 35 (10):1243-127,, 1992.
Danis, D., Parmacek, P., Dunajsky, D., and Ramasubrama- nian, B. Multi-agent reinforcement relioaggettnnnnoecemmee learning with prospect theory. In Conference Control and its Applications (CT), pp. 9-16, 2023.
Lee, J. D. and See, K. A. Trust in automation: Designing for appropriate reliance. Human Factors, 46(1):50-800 2004.
Devin, S. and Alami, R. An implemented theory of mind to improve human-robot shared plans execution. In Inter- national Conference on Human-Robol Interaction (HRI), pp. 319-326, 2016.
Lin, K.. Jie, C., and Marcus, S. I. Probabilistically distorted risk-sensitive affnite-hhrizon dynamic programming. Au- tomatica, 97:1 2018.
McKelvey, R. D. and Palfrey, T. R. Quantal response equi- libria for normal form games. Games and Economic Behavior, 10(1):6-38, 1995.
Ferreira, P. L., Santos, F. C., and Pequito, S. Risk sensi- tivity and theory of mind in human coordination. PLOS Computational Biology, 17(7):1-22, 2021.
Niclsen, F. On the jensen-shannon shannon symmetrization of dis- tances relying on abstract means. Entropy, 21(5), 2019.
Goeree, J. K., Holt, C. A., and Palfrey, T. R. Chapter 1: Stochastic game theory for social science: primer on quantal response equilibrium. In Handbook of Experimen- tal Game Theory. Edward Elgar Publishing, Cheltenham, UK, 2020.
Prashanth, L., Jie, C., Fu, M., Marcus, S., and Szepesvari, C. Cumulative prospect theory meets reinforcement learning: prediction and control. In International Conference on Machine Learning, pp. 1406-1415, 2016.
Hakobyan, A., Kim, G. C., and Yang. I. Risk-aware motion planning and control using evar-constrained optimization. IEEE Robotics and Automation Letters, 4(4):3924-39) -3911, 2019.
Rajabi, A., Ramasubramanian. B., Al Maruf, A., and Poovendran, R. Privacy-preserving reinforcement learn- ing beyond expectation. Conference on Decision and Control (CDC), pp. 4713-4713, 2022.
10

---

Risk-Sensitive Theory of Mind
Ramasubramanian, B., Niu, L., Clark, A., and Poovendran, R. Reinforcement learning beyond expectation. In Con- ference Decision and Control (CDC), pp. 1528-1535, 2021.
Wu, S. A., Wang, R. E., Evans, A., Tenenbaum, J. B.,
Parkes, D. C., and Bayesian inference for coordinating multi-agent collabo- ration. Topics in Cognitive Science, 13(2):414 432, 2021.
M. Too many cooks:
Rothkopf, C. A. and Dimitrakakis, C. Preference elicitation and inverse reinforcement learning. In Machine Learn- ing and Knowledge Discovery in Databases, pp. 34 48. Springer Berlin Heidelberg. 2011.
Smith, M. O. and Zhang, W. What if i'm wrong? team performance and trustworthiness when modeling risk- sensitivity in human-robot collaboration. Transactions on Human-Rober Interaction, 14(2):1-30, 2025.
Smith, M. O., Amatya, S., Amresh, A., Gorman, J., John- son, M., Cooke, N., and Zhang, W. Research needs in human-autonomy teaming: Thematic analysis of priority features for testbed development. In 2024 33rd IEEE In- ternational Conference on Robot and Human Interactive Communication (ROMAN), pp. 1183-1190, IEEE, 2024.
Sun, L., Zhan, W., Hu, Y., and Tomizuka, M. Interpretable modelling of driving behaviors in interactive driving sce- narios based on cumulative prospect theory. In Intelligent Transportation Systems Conference (ITSC), pp. 4329- 4335, 2019.
Sutton, R. S., Barto, A. G., et al. Introduction to reinforce- ment learning, volume 135. MIT press Cambridge, 1998.
Tan, V. Y., Prashanth, I.., and Jagannathan, K. A survey of risk-aware multi-armed bandits. In International Joint Conference Artificial Intelligence, pp. 5623-5629, 2022.
Tian, R., Sun, L., and Tomizuka, M. Bounded risk-sensitive markov games: Forward policy design and inverse reward learning with iterative reasoning and cumulative prospect theory. In Proceedings of the AAAI Conference on Artifi- cial Intelligence, volume pp. 6011-6020, 2021.
Tversky, A. and Kahneman, D. Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5(4):297-323, 1992.
van Hasselt, H., Guez, A., and Silver, D. Deep reinforce- ment learning with double g-learning. In Conference on Decision and Control (CDC), volume 30, 2016.
Wang, X., Le, Q., Ahmed, A., Diao, E., Zhou, Y., Baracaldo, N., Ding, J., and Anwar, A. Map: Multi-human-valus alignment palette. In The Thirteenth International Con- ference on Learning Representations, 2025.
Weiss, K., Khoshgoftaar, T., and Wang, D. A survey of transfer learning. Big Data, 3(9):1 40, 2016.

---

Risk-Sensitive Theory of Mind
A. Implementation
A.I. State and Action Space
The learning problem in Overcooked is commonly simplified in two ways (Carroll et al., 2019). First is by using mid-level action planners that allow agents to make decisions over high-level actions such as get an onion and serve the dish, and rely on algorithms such as A" search to solve the low-level routing problem. The problem is further simplified by creating custom state featurization that specifies state with distance (dx, dy) the closest onion, dish, soup, and their dispensers. Therefore, we lose information on all other objects in the environment. Both of these approaches are incompatible with Risky Overcooked as the low-level routing problem that A' solves is where the risky decision is present (e.g., traverse puddle or take a detour). Also, we need full observability of all objects in the environment as an risk-sensitive agent may want to navigate to an object that is not the closest if a puddle is in the way.
We define the action space for each agent as A₂ := {up, down, left, right, interact, wait) where the size of the joint- action space is |A| 36. Also, we formulate a lossless state encoding vector to enable full observability of the en- vironment. This first consists of features describing each agent: location (z, y). one-hot encoding of orientation {up, down, ft, righ!), and a one-hot encoding of held object {onion, dish, soup). As result, the total length of the feature vector describing both agents is 18.
Next, we have to encode objects that may be placed on any reachable counter space. Thus, every counter is given a one-hot encoding of { {onion, dish, soup) resulting in a total feature vector length of 3e where e is the number of reachable counters.
Lastly, we encode the status of each pot with a single value where we assign sequential pot status as {0- : empty pot, contains one onion, 2 : contains two onions, contains three onions, 4 soup is ready). Thus, the total length of the feature vector is where is the number of pots.
This lossless state encoding varies with the layout where the total length can be expressed as |s| = 18 + 3c z. This implics that there may be scalability issues in extremely large environments, especially as more open counter spaces are added. For the RCR and RMP layouts, we have |s| = 41 and [s] = 68, respectively.
A.2. Handling Sparse Rewards
Risky Overcooked is an even more sparsely rewarded environment than the original Overcooked task due to the possibility of losing held objects. To address this during learning, we first apply a naive curriculum that consisted of learning the following subtasks in order: I) deliver the soup, 2) pick up the cooked soup from the pot, 3) grab the dish before picking up the soup, 4) pot the third onion, 5) pot the second onion, 6) pot the first onion, and 7) full task. This curriculum is equivalent to having the agents learn the task backwards such that the steps for obtaining the primary payoff 20 for delivering the soup only requires one subtask to be learned where later subtasks are already solved. During sampling, a decaying probability of sampling previous curricula was added so subtask strategies were not forgotten.
Also, a series of small, intermediate rewards are provided to agents upon completion of subtasks. This helps reduce the sparsity of rewards in the environment and speeds up convergence of the MARSRL algorithm. A linearly decaying weight added to these shaping rewards that converges to as training progresses. When an agent places an onion in a pot or picks up a dish when a pot contains finished soup, the agent will receive F 3 reward. When an agent successfully picks up a soup by bringing plate pot with finished soup, they receive F = 5 reward. As we attempt to explore behaviors involving reliance and coordinated handoff' (i.e., risk/rely framing), we do not want to only reward the agent that completes the subtasks and encourage greedy and independent behaviors. Conversely, if agents shared all shaped rewards, an agent that did not participate in a subtask would receive frivolous reward signals that do not support effective learning. Thus, an agent receives a shaped reward only if they participated in that subtask. For example, both agents will receive the shaped reward if l picks up an onion and places on an open counter so -k can pick it up and place in the pot. However, is required to have interacted with the object relevant to the completed subtask some point in order be rewarded.
A.3. Hyperparameter Selection
An exhaustive grid search over the values in Table was used to optimize model hyperparamters, The search was ran with a rational agent in the RCR layout and final values were selected based on the model with the maximum mean reward over the past 10 iterations to select for both optimality and stability of the policy. Selected hyperparamters used during training of models are indicated by the bold values in Table 2.
12

---

Risk-Sensitive Theory of Mind
Table 2. Hyperparamters Tuning and Selection
Hyperparameter
Value(s)
Discount factor (t)
{0.95, 0.97, 0.99)
Learning rate (a)
{5e-3, 1e-3, 5e-4]
Target soft update rate (a)
(5e-3, 1e-3, Se-4)
Minibatch size (N)
{128, 256)
Num. hidden layers
(3.5, 7
Hidden layer size
{128, 256}
Memory size
(2e4, 5e4, le5}
A.4. Quantal Response Equilibrium
A QRE (McKelvey & Palfrey, 1995) is game theoretic solution to multi-agent choice problems. Here, players choose strategies using the expected utility of that strategy and assume that other players do the same. In other words, both players' payoffs are computed conditionally on their beliefs about the probability distribution of the other player's actions. This creates a recursive set of belief's where the QRE is obtained when utility functions converge as the level of recursion goes to co. Similarly, the QRE converges to Nash equilibrium solution as the decision temperature À -> (Goeree et al., 2020). However, humans do not have the resources to compute this infinite sequence due to bounded rationality. Therefore, we apply a yo-level reasoning paradigm where we terminate the recursive sequence after yp iterations. While traditionally refereed to as "level-k" reasoning, we use y instead to avoid confusion with agent indexes & or -k. When & O, we assume that a player has a uniform probability distribution U over their controllable actions. Consequently, if şo = 3, we get the following sequence of calculations:
P (Ax) x B[Q(s, (x " (A+)],
P4p (A-a) E[Q(s, (00ss, a+)) (A+)), p (Ax) x E[Q(s, 4n 1P (A+)],
- U,
Thus, an equilibrium strategy (T*(s), T-x (s)) to a single stage game Q(s) can be computed where we assume that spk = (P-k-
A.5. Convergence Proof
In order to establish convergence of Algorithm 1, we will first prove that rational (non-CPT) updates performed on the Q-values in (6) under a QRE solution constitute a contraction mapping and that Qopt is fixed-point. This proof follows closely with (Hu & Wellman, 2003) with minor modifications to the equilibrium solution and formulation as a two-player game. Once this proof concluded, an extension the convergence of the risk-sensitive version in (8) will be demonstrated using the results in (Lin et al., 2018).
We first rely on two assumptions about infinite sampling and decaying of learning rate. Also, we assume that each stage of the game is structured such that an optimal solution can be consistently found.
Assumption A.1. Each joint state-action pair is observed infinitely often Assumption A.2. The learning rate a satisfies [0, 1). Σ a, oc, and E af < oc.
Assumption A.3. Every stage game Qr(s) V 1, s has globally optimal point?.
Assumption A.3 may seem strong at first glance when using QRE as it can converge to arbitrary Nash equilibrium solution, and not necessarily the global optima, as the decision temperature \ + co (Goeree et al., 2020). However, we support Assumption A.3 with the fact that Risky Overcooked is a common-payoff game and that the QRE will most frequently converge to the globally optimal point in this case. To simplify notation, we denote the T (= (s()(ss)) as shorthand for Eng, Te(aa (')=- d')'('s, d'e d'x)) that describes the value for agent as a result of the QRE solution (TT T- (s')) to the stage game ((=").
In (Hu& & Wellman, 2003), the authors assume that every stage game may also he saddle point which relaxes this assumption. However, the common-payoff structure of Risky Overcnoked prevents saddle points from existing.
13

---

Risk-Sensitive Theory of Mind
Definition A.4. A joint strategy (mī((), m4. (s) of stage game Qe(s) is a global optimal point if every agent receives their highest payoff at this point. That is,
> TM (s) V Th, T-k E II
Definition A.5. A payoof- game is game in which for all joint strategies (TT((), T=*(())) it is the case that agents receive the same payoff such that T (s))) = Ta (s)))==(s)Q-=((())
By Definition A.5, we have ( existence of globally optimal point common-payofi games.
(+)())) which satisfies Definition A.4 and guarantees the
Proposition A.6. Given the existence of a globally optimal point and a QRE that terminates at a level-O agent with uniform strategy TU, the globally optimal point is the most likely equalibrium solution as X + oc.
Proof. Consider a uniform stage game Q(s, ag. a-x) 9 A ag. Q-k except at the global optima dg ai such that Q(s, (g. > q. Then, given that -k is a level-0 agent in a QRE and assumed to have a uniform strategy To by we have Ea., Tu(a- ssQ(s, 0g. a-x) 2 Eas To (a.x [s)(s, ag.a.k) V az making the QRE guaranteed to select the globally optimal action o as À + since (akak 8) x expp>> a-; (a-x s)Q(s, au, d-x)). Subsequent level-k iterations will continue to select the pure strategy (at, a²+) as this is a strietly dominant strategy given this initial condition. If we instead let all all other values of Qs,, ax; a-x)) be randomly sampled from a distribution with a mean of g. we expect the same to hold for most cases, especially when Q(s, ag. ).2.) > similar argument holds for mixed equilibrium strategies where the global optima (xt; ) is the most probable solution. In the case that (T m4k) is a strictly dominated strategy, the globally optimal solution is guaranteed to be selected.
Thus, support the Assumption A.3 with the fact that QRE is more likely to converge to global optima rather than other Nash equilibrium solutions with sufficiently high À in an ambiguous case.
The majority of the remaining proof will be working towards showing that (6) meets the conditions of the following theorem to provide a means for showing convergence.
Theorem A.7. Let the mapping Tr Q - Q satisfy the following condition: there exists a S 7 < 1, such that ITQ < T: < >llQ QlI v Q. E Q and Te admits unique fixed point Q" Q such that Q" T[ Then, the Q-iteration operator (T,Q) ak. a-x) = r(s, ake a-k) + )== performed during the Q-value update converges to Q' with probability 1.
where Q is the space of all possible Q-functions. Here, we can equivalently define (T:Q)(s, as, a-x) = r(s,, axs a-s) +
(V Tk, T-&). The simplest condition of Theorem A.7 show is that Q" fixed point.
Lemma A.8. For a two-player stochastic game, [[T,Q"] Q*
Proof. Following from (5), with an equilibrium (T)(s), T (s)) on a shared optimal Q-function Q", we have
ak, a+) r(s, ak. ax) + Σ PPss' s, ak ax) Tg.
s'ES
0k + Σ IP(8' 8, an, (4)
s'ES
= Σ P(s') 8, ax a.)) (eox)))((s,),
ass as, + + (1)
s'ES
E[T,Q" (s,
for all 8, agg a-g. Thus, B[T+Q")=''
We must now prove that Tú is a contraction which requires that every stage game encountered during learning has a unique solution and that the learner consistently chooses this solution when updating its Q-values (i.e., Assumption A.3).
We then define the distance between two Q-functions.
Definition A.9. For Q, Q E Q, define
||Q - Q11 = max [1Q(s) - (ss]]
max max [Q(s, az. a-x) Q(s, ax a+)
0k.00k
14

---

Risk-Sensitive Theory of Mind
Given Assumption A.3, establish that T, is contraction mapping.
Lemma A.10. |T:Q Q:Q1 S +|1Q Q1I v Q.Q&Q
Proof. For any Q-functions Q.Q E Q
||TiQ T:Q11 = max (
mmx m (0
We then seek to prove that
(T nx --) + 110(s) Q())ll
Consider that both (TT) (s) and (fx +s) satisfy Assumption A.3 by being the global optimal solutions we have
Q and Q, respectively. If
Th (s) (+)
S nW (00
Σ
a-k) QQ ak a-x)))
Σ
TA (8, ((). a-k) ag
max
ax ax, Q(0, axs ax))
ag:a-
Σ
TH TT(55a))T-466 a= Q(s)]l
(s)(s) (s)l
If m=())===(s)((ss) < tn (s) then (+Q()) (+)))++(()(((s) I (s)))) +t) which affords similar proof to the above. Thus. Te is a contraction mapping under the given assumptions. Theorem A.11. Under Assumptions A.I.A.3, the sequence Qe updated by:
Q4t1 aie a-x) (1 - Qe(s, ou, a-x)) + a (ora + you
where (Tx (s'), (s)) is the level-k quantal response equilibrium solution for a stage game Q+(5), converges to the Q-value Q"
Proof. Proof of convergence to Q" follows directly from Theorem A.7 which requires that Tr is contraction operator, established in Lemma 10, and that the fixed-point condition [[] = Q" is satisfied according to Lemma A.8. Therefore, the process (1 ar Q + oz [T.Qa] converges to Q"
We will now extend the previous proof of convergence of the rational (non-CPT) algorithm its risk-sensitive variant. Theorem A.12. Let the utility weighting functions u' and u' be according to Definition 2. and assume they invertible, differentiable, and have monotonically non-increasing derivatives. Also, let probability weighting functions w" and w be according to Definition 2.2 and monotone. Then, the following is satisfied [[T.Q TpQ < 7/1Q Qcpe|] 4 Qcpt; Qcpt
Qept-
Proof. This proof follows similarly to (Ramasubramanian et al., 2021) where we leverage the fact that utility weighting func- tions are monotonically non-decreasing (Definition 2.1). Thus, given a policy is improved such that (s)="(( (s' (6)) e (8) > T& (x( Ccpt (s), then we have
u' (r(s, ak a+)) + ( (sppp 2 u' (r(s, ax, axx) + (sT(("')==(s)))) (s))
(10)
where we denote the above inequality as u 2 u Since the probability weighting functions in Definition 2.2 are also monotonically non non-decreasing, we can then write the continuous form of the calculation of the CPT-value in (3) as
w (P(u 2))dz > wi (P((i > 2))dz
(11)
15

---

Risk-Sensitive Theory of Mind
where similar argument holds for u" and w". Thus, the operator TeQept is monotone. We can then show that
(Ti (TrQepe)(s, ak, a-b)
Popt (r(s; ax, Cx, a-x) TTN((")TTRR( (p))
Pept ((s, ap, a-k) + Σ
Th(s', TT (s', a-x (Qppe(s)) + Qepe(s') Qope(s'))
S Pept (T,, a-2)
Σ 02
Ta(d, an ( a-k (epe
where e = - Q('))l- We then rely on Theorem in (Lin et al., 2018) and its proof that provides an analysis of integrals composing (epe(-) to obtain (T, 0k, a-x) S (TiQ) (s, ag, a-x) qe which results in ||T:Q TiQ s rllQ - Q1 Q Q E Q as a consequence. Thus, we have shown that both the QRE and CPT-value modifications constitute a contraction mapping. This concludes the proof.
1. UPPER-BOUND ON
helpful here to mention a potential challenge when choosing CPT parameters way that ensures convergence. Specifically, the loss aversion parameter introduces gain the TD-target which may introduce means for the Q- function to diverge under certain conditions. This section will provide heuristic upper-bound on such that we can maintain the critical assumption that 7 < 1 in Theorem A.7.
Consider the worst-cas cast scenario where every TD-Target prospect t; = + V(s') with probability p; = ((('' a) is perceived as a loss (i.e. I b V i). Thus, the loss's utility function u (T;))) = L/T; b/" where we assume b = 0 for simplicity, can be expressed as:
u (T:) = f|T/"
() + 7V((s' T.) r(s + + Vs's Th, T-))"""""
Letting fo = b-- we can satisfy the same condition for convergence in the worst-case scenario given that no 1. For example, we would have 900 = 2.23 with é 2.25, ` = 1, and 0.99. This would provide no means for future values to decay where the Q-function would quickly grow towards -oo or co. However, given that т, can be perceived as a gain or a loss, this constraint on parameters l and rÿ may be relaxed but there is no theoretical grantees since the expected & would need be calculated from the cumulative CPT expectation (3). naive estimate can be made with
K-L K
î1 Rd K
10
where the number of TD-targets perceived as and is the number perceived as gains. There no feasible method for determining theoretically before algorithm runtime. However, this motivates the use of the pragmatic CPT-value that only applies CPT expectation during choices that involve risky outcomes where |r] 1. This further relaxes the constraints on l and 1f" since the
Pp
np
f/2
where # the proportion of risky decisions under some time horizon T. Then, 22 a 7 + oc, the condition for the Bellman equation should be met. Given the sparsity of dscisincsssons in Risky Overcooked and reasonable choices of and 17, this condition is almost surely satisfied.
B. Experiment Details
B.I. fisk-Sensitive Parameter Selection
The goal of parameter selection is to create sufficient divergence in risk-sensitive strategies (c.g., traverse risky states versus passing objects through the center counter in RCR) while still being cognitively motivated. Therefore, we take the median
16

---

Risk-Sensitive Theory of Mind
CPT parameter estimates in (Tversky & Kahneman, 1992) and fine them obtain clear definitions for risk-averse and risk-seeking candidate policies. A summary of the selected parameters be found in Table 3 while the prospect curves illustrating these parameters can be found in Fig. 5. The remainder of this section will describe the rationale behind these selections.
Table 3. CPT Parameter Selection
Utility Function
Weighting Function
Policy
n¹
of
Risk-Secking
0.44
1
0.88
0.61
0.69
Risk-Neutral
1
1
1
Risk-Averse 2.25
0.88
1
0.61
0.69
For the loss-aversion parameter, the median estimate is l 2.25 where > creates risk-averse tendencies while L < 1 creates seek-se tendencies. Therefore, we select / 2.25 for the risk-averse candidate policy and its inverse f = 0.44 for the risk-seeking candidate policy. For the exponential utility discounting parameters, 1j < 1 will generate risk-averse tendencies while 7j < generates risk-seeking tendencies. Given that the median estimates were found to be '7" = 1 = 0.88, we can select r7" = 0.88 and 7j = 1 such that there is exponential discounting of gains and none on losses to emulate an agent perceiving gains as less important than losses. This results in consistently risk-averse utility transformation in Fig. 5 across all possibly utility values. Similarly, we select rÿ" = 1 and rj" = 0.88 to emulate perceptions that weights losses as less important than gains. This generate the risk-seeking utility transformation Fig. By selecting the nominal value of 1 for the 1) that would generate the opposite of the intended risk-sensitive behaviors, in conjunction with proper selection of clear definition of risk-averse
can ensure that there is conflicting utility weightings that would otherwise interfere with risk risk-seeking behaviors.
The interpretation of the probability weighting parameters 8* and 8 is context dependent and leads to infeasible analysis across tasks (i.e. Pp changes between layouts). Therefore, the median estimates of = 0.61 and 8- 0.69 were used identically for both risk-averse and risk-seeking agents in the decision weighting transformation in Fig. 5. To obtain a noisy-rational candidate policy, we simply set all CPT parameters to their nominal value of that creates an identity transformation for all functions.
Risk-Seeking
Risk-Averse
Risk-Seeking & Averse
10
10
1
(3%)
0 (Xt)
w (p,)
Perceived Utility u(x))
(X%)
Perceived Utiliyy u(a))
a (xt)
Decision Weight w(o)
w (p,)
Rational
Rational
Rational
-10
-10
0
10
-10
10
Utility Xi
Utility X
Probability
Figure 5. Prospect curves for the selected CPT parameters used generate the risk- is and risk-averse candidate policies. The prospect curves in all plots for the risk-neutral up candidate policy show by the black dashed line in all figures and is equivalent to the identity transformation.
B.2. Tuning Risk Levels
The probability of slipping Pp (i.e. risk level) was tuned to maximize mutual dissimalarity between the risk-seeking ms. risk-averse A. and resk-neutral (i.e. rational) mo policies. We specify "mutual" here to express the desire to weight cases where a Pp generates one extremely dissimilar policy and two similar ones less than a Pe that generates three equally dissimilar policies with a possibly lower cumulative dissimilarity measure. Doing so allows to optimize each layout to
17

---

Risk-Sensitive Theory of Mind
amplify the divergence of the tested candidate policies, perform better inference, and generate insightful experiments.
We implement the Jensen-Shannon divergence (JSD) (Nielsen, 2019) as similarity measure which is symmetrized version of the Kullback Leibler divergence measure where the JSD between policies and can be expressed as:
DuS(m [12) = 2 ( + (
(12)
where M ( + π2) is the mixture distribution and Dx is the Kellback-Leible divergence between и and M. For each of the three combinations of policies {(TS, Ta), (TT No), (TA, mo)), ()s(TTl)z2) D computed. To maximize mutual dissimilarity, additional JSD beyond the minimum is discounted:
Σ dED
log(d min min(Dinnii-).. + + min(D)
(13)
where a E [0, 1] is a measure of mutual dissimilarity and = 0 indicates three identical policies. Over discrete space of risk levels we then find the Pp that maximizes a. A total of 5, 000 random states were sampled where the average was used select the final Pp- This procedure was conducted for each layout.
As seen Fig. P, 0.4 was selected for RCR and Рp 0.15 was selected for RMP. Several pp achieved comparable score leading the possibility of alternative risk levels being viable for simulation. However, for conciseness, strictly selected the maximum value.
RCR
RMP
0.6
0.6
0.4
Figure Mutual dissimilarity scores a different risk-levels Po the RCR (left) and RMP (right) layouts. Orange hars indicate the selected risk-level that maximizes
B.3. Timeseries Snapshots
To supplement the cumulative expression of joint-policiee in Fig 3, this section will more directly examine how strategies differ with risk-sensitivity using illustrative examples. In Fig. 7, four brief timeseries snapshots of these strategies. In all snapshots, agents attempting to either bring onions dishes to the pot in order to complete subtasks 1) or 2), respectively, mentioned in Sec. 3.1.
Seeking (T,,
Averse (T
Seeking (T
Averse (T ()
(A)
(B)
(C)
(D)
Risky Coordination Ring (RCR)
Risky Multi-Path (RMP)
Figure Illustration timeseries snapshots of the risk-sccking as and risk-averse na strategies in the RCR and RMP layouts. More
transparent chefs and objects indicate timesteps further in the past.
18

---

Risk-Sensitive Theory of Mind
In (A), that both agents are optimistically traversing puddle states with onions in order avoid possible coordination costs involved with handing off onions through the center counter tile. In contrast, (B) has the opposite preferences where pessimistic tendencies cause agents to view the consequences of slipping in a puddle to outweigh these coordination costs. Therefore, agents will form joint strategy that passes onions through the center counter tile.
In (C), an optimistic team will prefer to incur more risk by traversing puddle states order to take a more direct route and avoid the added navigation costs of the long, risk-free detour. Conversely, in (D), agents prefer to incur these navigation costs and take the long detour as they pessimistically perceive the cost of slipping be great.
19


Reviews:
[{"review_id": "8gky6z6jnV", "reviewer": "Reviewer_2s5A", "summary": "The authors consider the problem of a two-agent cooperative sequential decision task, in which one of the agents is a human, and address the problem of learning to coordinate behavior in the second agent. The idea is to model human behavior as following cumulative prospect theory, i.e. scaling probabilities and values according to a specific family of functions. By inferring which such function from a set of predefined parameters most closely describes human behavior, the agent can leverage this model to solve the multi-agent choice problem using DDQN. This involves computing the expected action of the human agent with level-k reasoning. Transitions are stored in a buffer for computational efficiency. The algorithm is applied to synthetic data with different simulated human agents in a new variant of the overcooked task and it is shown that performance in coordination is close to optimal as quantified by an oracle baseline and significantly improved with respect to a model of the other agent assuming noisy optimality.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your effort in providing comments and respond to them below.\n\n\n> **C1:** The reviewer asked why we did not include a limited human study.\n\nWe recommend you check the response to Reviewer 3GLy’s C1 for a full description of the summary we provide here.\n\nWhile “...capturing human behavior better than alternative methods” is the end goal of this research, that is not exactly the claim we are making in this particular paper. The claim we are making is that *we can overcome the challenges in state-of-the-art methods that already show adaptation to human risk-preferences improves human-robot/AI interaction*. Specifically, our approach overcomes:\n\n1. the limited task complexity afforded by tabular RL methods (Danis et al., 2023; Kwon et al., 2020) through our “Deep MARSRL with CPT” algorithm. \n2. the requirement for data priors to enable personalization (Kwon et al., 2020) through our generation of risk-sensitive candidate policies offline.\n\nWe believe the simulations support this claim. Consequently, the algorithmic contributions are independently significant from human studies, fill an important gap in the existing literature, and fits the scope of ICML well. We are conducting these human experiments in an ongoing study that warrants its own paper. This way, we can maintain the focus on AI in the current ICML paper and shift towards a more human-centered perspective in the next work.\n\n> **C2:** The reviewer asked how the level-k reasoning affects the convergence proof.\n\nPlease check out Proposition A.6 and the supporting Definition A.4 and Definition A.5. Here, we leverage the fact that this is a common-payoff game to show there exists a globally optimal point to every stage game (Assumption A.3) and that the quantal response equilibrium converges to this globally optimal game solution as $\\lambda\\rightarrow\\infty$.\n\n\n> **C3:** “Maybe citing some work on inverse RL may be adequate, particularly IRL work that allowed the agent to be suboptimal with respect to the known rewards such as work involving soft policies or work connecting IRL to preference elicitation?”\n\nWhile our related work is focused on RL to support the contribution of our RL algorithm, we acknowledge the relevance of IRL and did previously include a CPT-IRL paper that related to this work (Sun et al., 2019). This was to contextualize an alternative to our approach and illustrate a major hurdle that our work seeks to address: personalized adaptation to risk-preferences (e.g., learned from IRL) requires a human data prior which may be costly or infeasible to collect, especially when human risk is present. This is one of the primary advantages of using a cognitively valid model of human behavior (i.e., CPT) in that we do not need a data prior to achieve human models in risky settings. \n\nTo address this comment, we added review of additional work on IRL to better contextualize the problem:\n\n1. (Rothkopf & Dimitrakakis, 2011) A more standard approach to Bayesian IRL for preference elicitation when observing suboptimal demonstrations.\n2. (Cheng et al., 2023) Applies a more interactive approach risk-sensitive to IRL through interactive querying.\n3. (Bergerson, 2021) The authors identify several approaches to multi-agent IRL that identify suboptimal human traits (i.e. noise, biases and heuristics) but the existing work shows limited attention on integrating cognitive models like prospect theory.\n\n> **C4:** The reviewer suggested notational edits to Definition 2.3 and pointed out a few typos.\n\nWe have updated these items. \n\n\n> **C5:** The reviewer asked what the purpose of the decaying weight of previous observations in the belief update was for.\n\nThis is intended to accommodate human agents shifting their risk preferences online where newer observations are more representative of their risk preferences at the current moment. Using a forgetting factor like this is a common approach in recursive inference to ensure stability and representativeness of the inferred parameter (Liu et al., 2016). While the simulated agents in this paper are fixed, the decay is intended to emulate settings with real humans to maintain external validity. We will add an explanation of this into Sec. 2.3.\n\n\n> **References**\n\nBergerson, S. Multi-agent inverse reinforcement learning: Suboptimal demonstrations and alternative solution concepts. Computing Research Repository, abs/2109.01178, 2021.\n\nCheng, Z., Coache, A., and Jaimungal, S. Eliciting risk aversion with inverse reinforcement learning via interactive questioning, arXiv preprint arXiv:2308.08427, 2023. \n\n\nLiu, C., Zhang, W., and Tomizuka, M. Who to blame? learning and control strategies with information asymmetry. In 2016 American Control Conference, pp. 4859–4864, 2016.\n\nRothkopf, C. A. and Dimitrakakis, C. Preference elicitation and inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases, pp. 34–48, 2011.", "claims_and_evidence": "The main motivation for the proposed model is that it captures human behavior better than alternative methods, as it models human choices with cumulative prospect theory. However, the evaluation is done exclusively on synthetic data. Human experiments would have been more convincing. For the synthetic data, the claims are convincing, both the inference of the agent's risk attitude as well as performance.", "methods_and_evaluation_criteria": "The newly devised risky overcooked task makes sense for evaluating cooperation with potentially nonneutral risk attitudes.", "theoretical_claims": "The proofs involve standard contraction mapping arguments. Adapting to cumulative prospect theory, one can leverage the fact that the utility weighting functions are monotonically non-decreasing. However, I am not sure how moving from quantal response equilibrium to an approximate action from the agent using level-k reasoning affects the argument.", "experimental_designs_or_analyses": "I could not check any code or simulations but the evaluations in terms of risk attitude and performance seems sound.", "relation_to_broader_scientific_literature": "The paper sits somewhere between multi-agent RL, cognitive science, inverse RL, and alignment in human-machine interaction.", "essential_references_not_discussed": "Maybe citing some work on inverse RL may be adequate, particularly IRL work that allowed the agent to be suboptimal with respect to the known rewards such as work involving soft policies or work connecting IRL to preference elicitation?", "other_strengths_and_weaknesses": "Definition 2.3 could be easier to read by writing the bounds on the sums in the initial definitions in terms of l instead of two different ks, especially because the expectation then uses l.", "other_comments_or_suggestions": "Typos:\n“Humans like are said to have“\n“they are holding where rationale for how values for pρ can be found “\n“risk-sensitivity We also”", "questions_for_authors": "How does moving from quantal response equilibrium to an approximate action from the agent using level-k reasoning affects the proof of convergence?\n\nWhy did you add a decaying weight to the likelihood P(O|π) of observations further in the past?\n\nWhy did you not include at least a limited human study as the motivation is to better model human behavior?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ALaYth6dFB", "reviewer": "Reviewer_pniM", "summary": "This paper proposes a new model of risk-sensitive multi-agent coordination towards better aligning autonomous agents with human utilities. The authors define a risk-sensitive ToM that affords adaptation to a partner with unknown risk-sensitivity in in a zero-shot\nfashion by pre-training several policies with different risk preferences, and online matching observed behavior to these policies.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments.\n\n> **C1:** The reviewer commented on the generalization of our algorithm beyond these toy scenarios and that alternative models may give rise to the same behavior. They suggest that we mention this in limitations. \n\nWe generally agree that the observed behaviors could arise from alternative models (e.g., limited planning horizon). However, \n\n1. it is unlikely that alternative models would consistently differentiate risk-averse/seeking policies and that the novelty of our approach has distinguishable value across use-cases and in more complex tasks.\n2. As for generalizing beyond toy scenarios, we chose to extend Overcooked for this very reason: it is a well accepted baseline for coordination tasks that is simple enough for validation but rich enough to allow diverse coordination strategies. Once validated, we can scale to more complex domains where higher differentiability is available. In the second paragraph of Limitations and Future Work, we mentioned “increasing the resolution of the candidate policies may be helpful for scaling to tasks with a more diverse set of risky-decisions available (e.g., continuous control problems affording continuous variation in risk-sensitive strategies)”.\n\nTo address these two components, we add the following discussion to the Limitations section: \n\n- “Due to the limited number of strategies (e.g., traverse puddle or detour), alternative models, like limited planning horizon and heuristics, could potentially give rise to the observed behaviors. However, these similarities are likely to fade as decisions become more complex and afford more variation in behaviors. While we believe that modeling risk-sensitivity is an essential factor in risky settings, humans are subject to many biases and heuristics. Therefore, a multimodal approach (Kryven et al. 2024) that captures risk-sensitivity along with other human planning characteristics will be pursued in future work.”\n\n> **C2:** The reviewer suggested discussing why we took our RL approach instead of the “Bayesian ToM previously used with Overcooked.”\n\nAs you pointed out, the Bayesian ToM approach was used in (Wu et al. 2021) where they formulate their ToM as reasoning over intentions and sub-tasks. However, we reason over risk-sensitive preferences in more complex tasks and our RL-approach is more generalizable since it \n\n1. Defines continuous variation over a larger latent space (i.e. CPT parameters)\n2. Does not need manual re-definition of mental states (i.e. sub-tasks) when working on a new task\n3. Applies deep RL to overcome state space complexity issues that exist in their tabular bounded real-time dynamic programming (BRTDP) approach to policy solutions.\n\nWe will add discussion of this article when introducing ToM and Overcooked.\n\n> **C3:** The reviewer mentioned that the focus on pre-training is a downside and asked \"How would this scale?”\n\nWe view the focus on pre-training as an advantage where we address excessive pre-training and scaling in the following ways:\n1. Pre-training is more affordable/scalable than current approaches that require collection of human data priors for personalization (Kwon et al., 2020), especially when risk to the human is present. Additionally, we can aggregate an increasing resolution of an RS-ToM over time which affords arbitrary complexity that meets the specific needs of the more sophisticated tasks.\n2. We do not need to continue training to adapt to more humans. Once we have completed the pre-training, we have a fully deployable model for all types of human partners. It is very scalable in this sense.\n3. The future work section includes an approach to reducing pre-training requirements by using model interpolation methods but we will also add discussion of transfer learning (Weiss, 2016) to better address the challenge of scaling.\n4. This algorithm can also scale to teams with more than two agents. See Reviewer 3GLy’s response to C2 for details.\n\n> **C4:** The reviewer suggested references of (Wu et al. 2021) and (Kryven et al. 2024).\n\nWe have incorporated discussion of these papers in the sections mentioned in the previous response.\n\n> **C5:** “Did not see SI”\n\nWe included the SI in the same PDF as the main paper. \n\n\n> **References**\n\nHoulihan S. D., Kleiman-Weiner M., Hewitt L. B., Tenenbaum J. B. and Saxe R. Emotion prediction as computation over a generative theory of mind, Philosophical Transactions of The Royal Society. 2023.\n\nKryven, M., Yu, S., Kleiman-Weiner, M., Ullman, T., and Tenenbaum, J. Approximate planning in spatial search. PLOS Computational Biology, 20(11):1–21, 11 2024.\n\nWeiss, K., Khoshgoftaar, T.M. & Wang, D. A survey of transfer learning. Journal of Big Data, 3, 9. 2016.\n\nWu, S. A., Wang, R. E., Evans, J. A., Tenenbaum, J. B., Parkes, D. C., and Kleiman-Weiner, M. Too many cooks: Bayesian inference for coordinating multi-agent collaboration. Topics in Cognitive Science, 13(2):414–432, 2021.", "claims_and_evidence": "Claim: The proposed model can collaborate with humans better than an AI partner without RS-ToM\nSupported, within the scope of this task. \n\nMight not generalize to beyond toy scenarios, in general the same behavior could arise not only from risk sensitivity, but also from other approximations to sequential decision-making (see suggested refs).", "methods_and_evaluation_criteria": "Yes. I am very well familiar with Overcooked domain for Cooperation.", "theoretical_claims": "Skimmed the math, seems fine.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The lit review is mostly good, but a few references are missing. \n\n1. The authors should discuss the advantages and the reasons for choosing a RL approach over Bayesian ToM  previously used with Overcooked. The authors claim to achieve zero-shot understanding of the human partner without prior collection of human data, however the downside to this method is the need for excessive per-training. How would this scale?\n\n2. The authors take assumption of noisy-rationality as a baseline, however in field of modelling human planning, it is well established that people may be using a variety of approximations that are more elaborate than injecting noise, most notably limited planning horizon, heuristics, simplified space representations. \nIn the original BToM papers (e.g. Baker et al 2017) high-temperature noise is simply a convenient catch-all for non-alignment with the optimal policy, given that they use trivial scenarios such that will not allow to differentiate between the various approximations. I do not suggest reviewing this entire field of human planning models, but I suggest carefully framing the limitations.", "essential_references_not_discussed": "Overcooked as a model of Cooperation with ToM:\nhttps://onlinelibrary.wiley.com/doi/full/10.1111/tops.12525 \n\nThis paper evaluates prospect theory in human sequential decision making, alongside limited planning horizon, and a model of numerosity in distance/area perception, showing that prospect theory may indeed generalize to spatial planning, yet is not an exclusive cause of such deviations.\nhttps://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1012582", "other_strengths_and_weaknesses": "The paper is well written and original.\n\nI'm leaning to accept, but I'd like to see a better discussion of limitations and related work.", "other_comments_or_suggestions": "Edit: I have increased my rating by a point, to Accept.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iU9xs9C94y", "reviewer": "Reviewer_3GLy", "summary": "This paper proposes a multi-agent RL method where agents model the other agents under the cumulative prospect theory (CPT) model. This allows better coordination with risk-averse or risk-seeking agents (an agent may have both tendencies depending on the context), which includes human agents.\n\nThe proposed method consists of three components: first a CPT-transformation is applied on the state-action values and transition probabilities. Second, Nash equilibria under these transformed values are found. Finally, Bayesian inference is performed to identify which equilibrium the other agent is going for.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. \n\n> **C1:** The reviewer is concerned that “the claim is to better model humans' decisions under CPT-related biases which will then lead to better collaboration, there is no evidence about it in the paper due to the lack of human experiments”\n\nWe would like to point out that there is “similar transfer between simulation and human studies” (Smith & Zhang, 2025) showing a side-by-side comparison between simulation and human studies to support external validity.  (Kwon et al., 2020) shows that modeling risk-sensitivity with CPT leads to improved performance and robotic perceptions. Thus, we see that there is existing support for modeling human decisions with CPT leading to better collaboration in simple settings but this is not directly our claim.\n\nThe claim we are making in this paper is that *we can overcome AI challenges in state-of-the-art methods that already show adaptation to human risk-preferences improves interaction*. Specifically, we are addressing:\n\n1. the limited task complexity available with tabular RL methods (Danis et al., 2023; Kwon et al., 2020) by using our “Deep MARSRL with CPT” algorithm affords scaling to scenarios more aligned with real world resolutions \n2. the requirement for data priors to enable personalization (Kwon et al., 2020) using our generation of risk-sensitive candidate policies offline to resolve costly data collection when human risk is present.\n\nThe simulations show the ability to generate diverse risk-sensitive behaviors in more complex coordination tasks (i.e. infeasible to solve with tabular methods) offline and correctly adapt to an agent of unknown risk-sensitivity online which supports this claim. Consequently, we believe the algorithmic contributions are independently significant from human studies, fill an important gap in the existing literature, and fit the scope of ICML well .\n\nWe agree on the importance of supporting the “better model humans' decisions” claim that you mentioned. This paper paves the way for subsequent human studies. We have already started human experiments where we intend to collect 60 subjects where we employ more sophisticated experimental designs to characterize human behavior in response to RS-ToM and Rational agents. We believe that this next study warrants its own paper due to the possibility that combining these works would cause muddled contributions and decrease the amount of depth we can explore in each due to the page limit constraints.\n\n> **C2:** The reviewer is not clear how this algorithm extends to n>2 agents and believes it should be framed as a human-robot collaboration instead of a multi-agent problem.\n\nThis work is motivated by the human-robot interaction paradigm. However, the MARSRL algorithm could be applied to generate risk-sensitive multi-robot interactions if desired. Additionally, the RS-ToM framework can work for n>2 agents by factorizing the state and action (e.g., $a_1$ x $a_2$ x…x $a_n$) in Algorithm 1 and extending the equilibrium solution (QRE with level-k reasoning) to initialize a level-1 player (see Appendix A.5) to assume all other agents have random policies where the remaining procedure remains the same. We will add an explanation of how to extend to n>2 agents into the end of Sec. 2.2.\n\nYou also make an important point about how to align with humans “with different equilibria”. This is an ongoing and active research field called multi-value alignment where more sophisticated multi-objective optimization methods, such as the MAP algorithm (Wang et al., 2024), must be applied. We will discuss this at the end of Sec 2.3.\n\n>**C3:** The reviewer suggested stating the assumptions about (a) requiring access to the transition model and (b) a completely collaborative scenario earlier.\n\nWe will add (a) to the beginning of Sec. 2 and (b) to the beginning of Sec. 3.1.\n\n> **C4:** The reviewer mentions that Def. 2.4 should be a multi-agent MDP.\n\nWe agree and will state that Def. 2.4 defines a multi-agent MDP when the joint action and state spaces are factored out.\n\n> **C5:** The reviewer is skeptical about our statement “median CPT parameters …generally describe risk-averse behavior”\n\nYou are correct in that both averse/seeking behaviors emerge from median parameters depending on context. We say “generally” to refer to the fact that, under random prospects, we expect the loss to be disproportionately represented in the choice since $\\ell = 2.25$. Therefore, these parameters generate risk-averse preferences in most cases. We refer you to Appendix B.1 for additional discussion.\n\n> **References**\n\nLi, C., Wang, T., Wu, C., Zhao, Q., Yang, J., and Zhang, C. Celebrating diversity in shared multi-agent reinforcement learning. In Advances in Neural Information Processing Systems, volume 34, pp. 3991–4002. 2021.\n\nWang, X., Le, Q., Ahmed, A., Diao, E., Zhou, Y., Baracaldo, N., Ding, J., and Anwar, A. Map: Multi-human-value alignment palette, arXiv preprint arXiv:2410.19198, 2024.", "claims_and_evidence": "It is not clear to me if this algorithm easily extends to n>2 agents. In the current setup, one of the agents is modeled as the biased agent (whose behavior can be modeled via CPT) and the other agent adapts to it. This makes sense for human-robot collaboration scenarios. However, what if there are multiple humans? In such cases, different humans may go for different equilibria, and it is not clear if the robot(s) will be able to decide how to coordinate with them. So maybe claiming this is a multi-agent algorithm is a little misleading, and it should be framed as a human-robot collaboration algorithm where there is only one human and one AI agent.\n\nSimilarly, the problem formulation implicitly assumes the agents share the reward function, meaning this is a completely collaborative scenario. While I am fine with this assumption, I think this should be noted earlier, because multi-agent systems would normally include competitive or mixed-sum settings, too.\n\nWhile the main claim is to better model humans' decisions under CPT-related biases which will then lead to better collaboration, there is no evidence about it in the paper due to the lack of human experiments. The simulation experiments with artificial agents do not provide enough evidence, because they are controlled experiments that are already expected to work (\"designed to work\" in a sense).", "methods_and_evaluation_criteria": "The proposed method makes sense but some assumptions should be stated more explicitly.\n\n- One of them is the assumption about n=2 agents as I mentioned before.\n- Another one is the shared rewards assumption, again as I mentioned before.\n- It seems to me that approximating the Nash equilibria and the procedure described after line 165 (right column) require access to the transition model, i.e., access to the probability distribution over the next state given the current state and the action is needed. This should be stated earlier.\n\nAlso, while the paper starts defining an MDP, it includes multiple agents that are decentralized, so it should really be a multi-agent MDP.", "theoretical_claims": "There is no theoretical claim in the main paper -- some \"somewhat theoretical findings\" are in the Appendix which I didn't review.", "experimental_designs_or_analyses": "While I like the paper a lot in general, the lack of human experiments prevents me from giving an accept score. Because the target biases are human specific and it is impossible to evaluate the value of the paper without having human experiments.", "relation_to_broader_scientific_literature": "While talking about the prior work, the paper states some of them \"train policies under the median CPT parameter estimates from (Tversky & Kahneman, 1992) which generally describe risk-averse behaviors. I am skeptical about this statement. As far as I know (but I am happy to be convinced otherwise), the median parameters model both the risk-averse and risk-seeking behaviors depending on outcomes being negative (e.g., people buy insurance to avoid large losses) or positive (e.g. people buy lottery tickets to earn lots of money).", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "There are several typos in the paper. Some of them are:\n\n1) \"a RS-ToM\" --> \"an RS-ToM\"\n2) \"in in a zero-shot\" --> \"in a zero-shot\"\n3) \"in this a deep multi-agent setting\" --> \"in this deep multi-agent setting\"\n4) \"Figure. 2\" --> \"Figure 2\"", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WvX6vM6Vx3", "reviewer": "Reviewer_ah7K", "summary": "This paper presents a new method for learning agent strategies that support human decision-making under risk. The agent has the ability to learn strategies in situations where it does not know how risk-averse its partner is. The paper adopts prospect theory as a model for human decision-making and proposes a framework for determining strategies using reinforcement learning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and feedback. We have addressed them below.\n\n> **C1:** The reviewer asked “(Kwon et al., 2020; Sun et al., 2019; Danis et al., 2023; Ferreira et al., 2021) What is the reason why these methods cannot be applied? Even if they cannot be applied straightforwardly, I would also like to see the results of comparing them with methods that have been simply customized from these conventional methods.”\n\nThere are two main reasons discussed at the beginning of Sec. 3 but we have provided some additional elaboration below:\n\n1. (Kwon et al., 2020; Sun et al., 2019) generate diverse risk-sensitive behaviors by relying on data-priors collected from a human partner. In a simulation study, we simply do not have this option but, more importantly, a key contribution of our algorithm is that we do not need a data prior to generating adaptation to risk-preferences of different human partners, because we have access to a cognitively valid decision model (i.e., CPT).\n\n2. (Danis et al., 2023; Ferreira et al., 2021) rely on tabular methods like Q-learning algorithms which simply do not scale to large state-action spaces. We actually tried this and it did not go well because the Q-table must allocate memory for every state-joint-action pair. Other works have used low-level motion planners to simplify this problem and solve over high-level sub-tasks but this is not available to us due to “risk-sensitivity” of an agent being present in their low-level navigation policy (see Appendix A.2 for details). Let's look at the smaller of the two layouts (RCR) shown in our paper as an a motivating example:\n    - 2 players have 5 actions each for a total of 5^2 = 25 actions\n    - 2 players can each have 6 x and 6 y coordinates, 4 orientations, and 4 holding states {empty, onion, dish, soup} leading to a total of (6\\*6\\*4\\*4)^2 = 331,776 combinations.\n    - Each of the 7 reachable counter spaces have 4 possible states {empty, has onion, has dish, has soup} for a total of 4^7=16,384 combinations\n    - Each of the 2 pots has 5 variations {nothing, 1 onion, 2 onion, 3 onion, finished soup} for a total of 5^2=25 combinations\n    - This brings us to a 25 \\* 331,776 \\* 16,384 \\* 25 = 3.39e12 element array. \n    - With a 16 bit float precision, this requires 54 terabytes for each of the 3 models\n\n    As you can see, this quickly becomes infeasible to manage which requires a function approximation method when using this lossless state. We spent significant efforts to develop this novel deep learning algorithm including development of the novel replay memory which is one of our contributions to the AI community.\n\nWe plan on adding “there is a lack of feasible baseline algorithms …[because] state of the art MARSRL algorithms (Danis et al., 2023; Ferreira et al., 2021) are not tractable in complex tasks like Risky Overcooked due to their tabular formulation creating infeasible memory requirements under the high number of permutations in the state space.” into Sec. 3 to remove this ambiguity in the revised paper.\n\n> **C2:** “it is limited to simulation testing, and it has not been verified whether it is useful in actual decision-making scenes.”\n\nReviewer 3GLy had a similar comment which we recommend you check out the response to C1 for a full description of the summary we provide here. Here, there is a slight clarification about the claim of this paper that we intend to make more clear when listing the contributions. \n\nWhile testing “...whether it is useful in actual decision-making scenes” is the end goal of this research, that is not exactly the claim we are making in this particular paper. The claim we are making is that *we can overcome the challenges in state-of-the-art methods that already show adaptation to human risk-preferences improves human-robot/AI interaction*. Specifically, our approach overcomes:\n\n1. the limited task complexity afforded by tabular RL methods (Danis et al., 2023; Kwon et al., 2020) through our “Deep MARSRL with CPT” algorithm. \n2. the requirement for data priors to enable personalization (Kwon et al., 2020) through our generation of risk-sensitive candidate policies offline.\n\nWe believe the simulations support this claim as we are able to generate diverse risk-sensitive behaviors in more complex coordination tasks (i.e. infeasible to solve with tabular methods) offline and correctly adapt to an agent of unknown risk-sensitivity online. Consequently, the algorithmic contributions are independently significant from human studies, fill an important gap in the existing literature, and fits the scope of ICML well. We are conducting these human experiments in an ongoing study that warrants its own paper. This way, we can maintain the focus on AI in the current ICML paper and shift towards a more human-centered perspective in the next work.", "claims_and_evidence": "The research is based on the assumptions presented in previous literature, methods that have been proven effective in the field of agents, and evaluation experiments conducted in previous studies. As a whole, it is very convincing.", "methods_and_evaluation_criteria": "It is a reasonable fusion of human decision-making theory (prospect theory) and reinforcement learning. The evaluation experiment in this paper is based on the well-known setting (the Overcooked environment) by Carroll et al., and the game is extended in a way that involves risk in decision-making. I think it is appropriate as the most primitive simulation experiment to analyze decision-making under risk. On the other hand, as the author mentions in the Limitations, it has only been verified in a few simple scenarios. In addition, it is limited to simulation testing, and it has not been verified whether it is useful in actual decision-making scenes.", "theoretical_claims": "The main proof is included in the reference materials from page 11, but I have not verified the accuracy of the proof.", "experimental_designs_or_analyses": "(Kwon et al., 2020; Sun et al., 2019; Danis et al., 2023; Ferreira et al., 2021) \nWhat is the reason why these methods cannot be applied?\nEven if they cannot be applied straightforwardly, I would also like to see the results of comparing them with methods that have been simply customized from these conventional methods.", "relation_to_broader_scientific_literature": "In this research, behavioral economics (prospect theory) is used to model the human decision-making process. In other words, the basic behavior of agents and partners is based on these findings. This research can be positioned as interdisciplinary research in the fields of economics, the social sciences, artificial intelligence, and computer science related to human behavior.", "essential_references_not_discussed": "I believe the papers essential to this paper are included in the references.", "other_strengths_and_weaknesses": "- The structure of the paper is excellent. It was easy to understand the content.\n- I can appreciate the interdisciplinary approach of economics and computer science.", "other_comments_or_suggestions": "I don't understand why the methods of (Kwon et al., 2020 Sun et al., 2019 Danis et al., 2023 and Ferreira et al., 2021) cannot be applied to the current experimental setting, so I would like an explanation.", "questions_for_authors": "I have questions mainly about experimental methods and comparative methods. \nSee \"Other Comments Or Suggestions\" and \"Experimental Designs Or Analyses\".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper investigates a two-agent sequential decision-making problem, where one agent learns to coordinate with a human partner whose behavior is modeled using cumulative prospect theory. The authors propose a risk-sensitive reinforcement learning approach and demonstrate its effectiveness through empirical evaluation in simulated environments.

A key limitation of the work is the absence of experiments involving actual human subjects. Although the paper focuses on modeling human-specific biases, the presence and impact of such biases are not directly validated within the experimental settings presented.

Following extensive discussions with the reviewers, a consensus was not fully reached on this issue. Nonetheless, there is agreement that the paper offers substantial methodological and algorithmic contributions, and that the proposed approach is convincingly validated in simulation. It is strongly recommended that the limitation regarding the lack of validation with real human participants be clearly acknowledged in the camera-ready version.
Decision: Accept (poster)