Discovering User Types: Mapping User Traits by Task-Specific Behaviors in
Reinforcement Learning

Lars L. Ankile * 1 Brian S. Ham * 1 Kevin Mao 1 Eura Shin 1 Siddharth Swaroop 1 Finale Doshi-Velez 1
Weiwei Pan 1

Abstract
When assisting human users in reinforcement
learning (RL), we can represent users as RL
agents and study key parameters, called user
traits, to inform intervention design. We study
the relationship between user behaviors (policy
classes) and user traits. Given an environment, we
introduce an intuitive tool for studying the breakdown of “user types”: broad sets of traits that result in the same behavior. We show that seemingly
different real-world environments admit the same
set of user types and formalize this observation as
an equivalence relation defined on environments.
By transferring intervention design between environments within the same equivalence class, we
can help rapidly personalize interventions.

1. Introduction
Mobile Health (mHealth) applications, like a physical therapy (PT) app that recommends personalized exercises to auser working to regain ankle mobility, are gaining popularity
as cost-effective interventions. In these applications, personalization can be achieved by inferring the user-specificinternal obstacles to reaching health targets, then designing
treatments for those obstacles (Shin et al., 2022). In thispaper, we provide a set of novel tools for studying the relationship between user-specific obstacles and user behavior,
thereby generating insights for treatment design.
We model user-internal obstacles as discrepancies between
the real-world environment, formalized as a Markov Decision Process (MDP), and the user’s perceived environment,
another MDP. We want the user, as a Reinforcement Learning (RL) agent, to adopt a target policy in the real-world
environment (e.g., perform recommended daily exercises until full recovery), but since the user plans in their perceived
Equal contribution 1 Harvard University, Cambridge MA, USA.
Correspondence to: Lars L. Ankile <larsankile@g.harvard.edu>.

environment, their perceived optimal policy can deviate drastically from the target (e.g., prematurely terminate the PT
program due to the perceived infeasibility of full recovery).
For a real environment, we characterize the user-perceived
environments using MDP parameters that map to wellstudied human traits—which we call user traits—in the
behavioral sciences. In particular, for many mHealth applications, a user’s confidence in their physical capabilities
and their ability to perform long-term planning (their degreeof myopia) both significantly impact their success in prescribed fitness regimens (Picha et al., 2021b). In our work,
we model myopia as the discount factor and confidence
as the dynamics (specifically, the perceived probability ofpositive outcomes) of the user’s MDP (Section 3).
Given a real environment, we introduce a tool for visualizing the relationship between user traits (the user’s MDP
parameters) and the corresponding user behavior (the user’s
possible policies). Specifically, within the environment, westudy the breakdown of “user types”—regions in the spaceof all possible user traits that define the same user behavior—
and visualize these types as two-dimensional behavior maps
(Section 4). Behavior maps shed light on the extent to which
it is possible to infer user traits by observing user behavior.
Finally, we show that seemingly different real-world environments admit the same behavior maps. We formalize
this observation as an equivalence relation defined on realworld environments (Section 5). We map several environments commonly used in the RL literature (that also model
mHealth tasks) to just a small set of equivalence classes,
where the sets of user behaviors are similar across different
environments within each class (Section 6.2). This result
allows us to provide guidelines on intervention design invarious complex environments by lifting insights from anequivalent and simpler toy environment (Section 6.4).

2. Related Work*Interactive Learning with Implicit Human Feedback Workshop at
ICML 2023, Honolulu, Hawaii, USA. 2023. Copyright 2023 by
the author(s).

Inferring a user’s parameters from demonstrations.
Like us, some works (Evans et al., 2016; Shah et al., 2019;
Zhi-Xuan et al., 2020) model humans as RL agents with different perceived MDPs. However, inferring an agent’s MDP

parameters from demonstration is a difficult and nonidentifiable problem (Shah et al., 2019). This paper shows that,while user parameters cannot be exactly recovered from behavior data in most settings, we can infer general rules about
the relationship between user parameters and user behavior.
These rules can help us design mHealth interventions.
Equivalence in Inverse RL (IRL). In IRL, when parameters of an MDP cannot be uniquely identified, we infer classes of these parameters, typically rewards (Ziebart,2010) or transitions functions (Reddy et al., 2018; Golub
et al., 2013), that are equally likely under the behavior data
provided by one user. In this work, we study the behaviors
of multiple users and equate different environments (MDPs)
in which the partitioning of the set of users by behavior issimilar.
Equivalence of MDPs. Notions of equivalence between
MDPs allow for knowledge transfer between different environments (Soni & Singh, 2006; Sorg & Singh, 2009).
For example, bisimulation-based equivalence definitions
are used in MDP minimization, where large state spaces
are reduced to speed up planning (Givan et al., 2003). Relaxed versions of bisimulations, e.g., MDP homomorphism(Biza & Platt, 2018), stochastic homomorphism (van der Pol
et al., 2020), and approximate homomorphisms (Ravindran& Barto, 2004) allow optimal policies in simple MDPs to
be lifted to desirable policies in more complex and comparable MDPs. More general definitions of MDP equivalence
can be defined through other methods of state aggregation(e.g., value equivalence) (Li et al., 2006). While these notions of equivalence are defined over the set of MDPs, we
decompose an MDP into task-specific and user-specific components and consider equivalences between the task-specificcomponents of MDPs while varying the user-specific ones.

3. Formalizing Users as RL Agents
We formalize an RL environment for an mHealth application
as a Markov Decision Process (MDP). An MDP is a 5-tuple,
M = ⟨S, A, T, R, γ⟩, consisting of a set of states S, a
set of actions A, a reward function R : S × A × S →R, a transition function T : S × A × S → [0, 1] and a
discount rate γ ∈ [0, 1]. For simplicity, in this paper, weonly consider discrete state spaces.
An optimal RL agent acts in M according to a policy
πM : S → A, giving a 
cumulative reward (expected reTPπturns): JM=E
γ t rt , where rt is the random variable
t=0

representing the reward received at time t. The optimal pol∗icy for M maximizes the expected returns: πM= max J π .
π∗
We want the user to adopt the optimal policy πM
in M.
However, the user plans in their perceived environment,

Muser = ⟨S user , Auser , T user , Ruser , γ user ⟩ and adopts the pol∗user
icy, πM. Discrepancies between
user , that is optimal for M
the real environment and the user’s perceived one can lead∗to drastic differences between the target policy, πM, and the∗adopted one, πMuser .
In this work, we shall assume that the perceived environment differs from the real only in the transition function(modeling the user trait confidence) and the discount rate
(modeling the user trait myopia). Specifically, we define
a world as a tuple W = ⟨S, A, R⟩ of states S, actions A,
and reward function R. This captures the real environment
and the task in an application of interest (see Fig. 2 forexample-grid worlds). Since the user’s perceived states,
actions, and rewards match the real environment, we set
S user = S, Auser = A and Ruser = R.
Furthermore, since we are interested in the set of optimal
policies generated by varying the user’s perceived environment Muser , we do not keep track of the real transition
function T and the real discount rate γ. Instead, the user’s
policy depends only on their (fixed) perception of the environment, T user , and their (fixed) discount rate γ user . The real
T is useful only when the user is learning (updating T user )based on data generated by T .
We use γ user ∈ [0, 1] to represent the user’s level of myopia.
To represent the level of confidence, we parameterize theuser’s transition Tpuser function with p ∈ [0, 1], which is thelevel of stochasticity in the environment transitions that theuser perceives. Other parameterizations of T user are possible,
but this one aligns with the intuition that a user with low
confidence is unsure whether their actions a ∈ Auser willlead to desired outcomes s′ ∈ S user .
In Section 4, we model how and why users with distinct
traits behave differently (i.e., adopt different policies) in thesame real-life setting. For example, two people with different levels of myopia would judge different PT behaviors
to be optimal in their respective MDPs. However, we firstconnect our formalization of user traits (their level of myopia γ user , and their confidence level p parameterizing Tpuser )to well-studied constructs in psychology and behavioral
science.

Mapping RL to Behavior Science. Myopia corresponds
to the concept of temporal discounting in psychology. In
user MDPs, we represent temporal discounting with γ user ∈[0, 1). This captures people’s tendency to undervalue future
rewards, often leading to unhealthy behavior (Story et al.,2014). However, we note that in RL, discounting is exponential by default, which does not capture the phenomenon
observed in humans called preference reversal (Ainslie &Haslam, 1992; Shah et al., 2019) (which hyperbolic discounting is more suited for).

D

Confidence level p

0.99

Equivalence Classification
C

1

0.4

0

0.4

A

0
1
Discount factor

B
0.99

Figure 1. Example behavior map (Big-Small world). The twocolors indicate the two possible behaviors (see Fig. 2c for theworld and the behaviors). Annotations describe the procedurefor deriving the equivalence class. The x-axis varies over the
discounting factor, γ; the y-axis varies over the confidence level, p.
“Extreme” users, i.e., corners of the map, are labeled as circles. The
number of “behavior switches” when tracing each edge between
extreme users (from A to B, to C, to D, and back to A) are labeled
as squares.

In behavioral science, confidence, also known as selfefficacy, measures an agent’s belief in their capability toperform a task (Picha et al., 2021a). Intuitively, this is theuser’s perceived probability that their intended outcome
can be achieved through action. In user MDPs, we represent the user’s confidence level with p ∈ [0, 1], whichis the level of stochasticity in the transitions. Concretely,
Tpuser (s, a, s′ ) = p for a user’s intended outcome s′ fromperforming action a in state s. We divide the remaining
1 − p probability equally among the alternate outcomes:
T user (s, a, ŝ′ ) = 1−p. Our current instantiation of confi|Ŝ|
dence is simple, and it is equivalent to adding epsilon-noise
to the real-world transition matrix. However, the transition
Tpuser can be a function of p in more complex ways.

4. Behavior Maps: A Tool for Understanding
User Traits and User Behaviors
In the previous section, we formalized the user’s MDP∗Muser and their optimal policy πuser. We now introduce behavior maps, a tool for studying the relationship between theuser-specific parameters (Tpuser , γ user ) and the corresponding∗
optimal user policy πuser.
Given a world W, we denote the set of possible (deterministic) policies, π : S → A, as ΠW . We note that in many
real-life applications, distinct policies may functionally de-scribe the same type of behavior (e.g., if we are interested
in overall adherence, skipping PT exercises every Tuesday
can be considered functionally equivalent to skipping every
Monday). Thus, we work with a concept that generalizes
the notion of policy; we define a “user behavior”, denotedB ⊂ ΠW , as a set of policies considered equivalent in theapplication domain. We study how differences in user traits
lead to different user behaviors.
To do this, we introduce a behavior map of the world W asa mapping of user traits to the corresponding user behaviors
in W. That is, the behavior map BW maps (γ user , p) to theuser behavior B that contains the optimal policy for the user
MDP Muser = ⟨S, A, R, Tpuser , γ user ⟩.
In Fig. 1, we show an example of a behavior map. We seethat it classifies the user parameter space into regions whereparameters map to the same user behavior. In this world,
there are only two behaviors (indicated by color), and theuser’s behavior depends on the value of their user traits (thetwo axes).
Applications of Behavior Maps. We demonstrate thatbehavior maps can inform the design and deployment ofinterventions on user traits (for example, interventions toincrease γ user ). Specifically, they can help us (1) determine
to what extent user traits are identifiable through behavioral
observations; (2) warm-start an intervention strategy forinteracting with new users.
Identifiability of User Traits. Since behavior maps tell us
which set of parameters gives the same user behavior, they
allow us to anticipate the limits of what we can infer about auser (using Inverse Reinforcement Learning (IRL) or related
methods) by observing their behavior in a given world. For
example, in worlds with the behavior map in Fig. 1, we candistinguish between users with low and high discount factors because users have different optimal policies (different
colors). On the other hand, the difference in confidence
does not generally correspond to a difference in behavior.
Therefore, we cannot generally distinguish between users
with different confidence levels. However, we find that behavior maps can inform intervention design, even when theparameters of individual users cannot be exactly inferred.
Warm-start Intervention Strategy. Given a world and a
new user, behavior maps can help identify interventions that,a priori, is likely to be more impactful. In particular, themore variation there is in user behavior along a given axis,
the more likely an intervention on the corresponding trait
will change the user’s behavior. For example, in Fig. 1, weknow that an intervention on γ user is more likely to change
the user’s behavior than an intervention on Tpuser .
Although useful, directly computing the behavior map for acomplex application such as PT requires solving user MDPs
for a range of user parameters and can thus be computationally costly. Instead, to get the same insights, we reduce thePT world W to a simpler toy world W ′ , for which we caneasily compute BW ′ . We define an equivalence relation thatallows us to make this reduction.

5. A Behavior-Based Equivalence Relation
This section uses behavior maps to draw analogies between
seemingly different worlds.
Suppose that two different applications, such as PT anddieting, have the same behavior map, such as the one from
Fig. 1. Then, in both applications, we know that confidence
does not impact user behavior and that users with “low”
gamma have one behavior, while users with “high” gamma
have another. In this way, we consider PT and dieting
equivalent worlds because intervention design principles
can be transferred from one to the other. For example, inboth cases, the initial intervention strategy should focus onγ user instead of Tpuser . Note that this transfer can work incases where the state and action spaces differ between the
two applications because the behavior maps depend on highlevel behaviors (not exact states and actions). For example,
in PT, the behaviors may be a set of exercises. In dieting,the behaviors may be a set of food choices. In either case,there is a desired behavior (e.g., choosing nutritious foods
or choosing the right exercises) and an undesired behavior.
We are only concerned with what interventions will help theuser go from undesired to desired behaviors, not that theactions defining those behaviors match exactly.
Moreover, we can transfer between worlds with similar
but not necessarily identical behavior maps. For example,
we might see that both PT and dieting have two possible
behaviors, where users with lower γ user act differently fromusers with higher γ user . However, what is considered to be“low” or “high” γ user need not match exactly between the
two applications: in PT, the range for “low” γ user could be[0, 0.3] and in dieting the range could be [0, 0.2]. If we knew
both applications had similar behavior maps, we could stilltransfer the knowledge that the initial intervention strategy
should focus on γ user instead of Tpuser . We could also transfer
the knowledge that users with different γ user are identifiable,
while users with different Tpuser are not.

the axes of each behavior map BW is scaled to the unit
interval, that is, BW is a map over I 2 , where I = [0, 1].
Thus, the decision boundary classifying different user behaviors in BW is a 1-dimensional submanifold in I 2 definedby the map gW : [0, 1] → I 2 satisfying some additional
constraints. Although we consider the case where the decision boundary is connected here, our definition extends
straightforwardly to cases where it is not.
Definition 5.1 (World Equivalence Induced by Behavior
Map). We define an equivalence relation, ≡map , on the setof discrete worlds W by
W ≡map W ′ ,W, W ′ ∈ Wwhen (1) the number of behaviors in BW and BW ′ are equal,and (2) there is a continuous map h : I 2 × [0, 1] → I 2 , suchthat ht : I 2 × {t} → I 2 is bijective, where h0 is the identity
map, and where h1 satisfies h1 ◦ gW = gW ′ .
Note that we can simply say that h is an ambient isotopy
between the decision boundaries in W and W ′ .
The idea behind Definition 5.1 can be made more intuitive.
We consider each behavior map as a diagram in which (i)ni number of vertices (each representing a switch between
behaviors) is placed on the i-th edge, and where (ii) eachpair of vertices is connected by a curve defined by a decisionboundary that separates two user behaviors (see Fig. 1). We
say that two maps are equivalent if they are labeled by thesame number of distinct behaviors, and, as diagrams, they
are topologically equivalent: the decision boundary in one
behavior map can be continuously deformed, by using themap h, to look like that in the other.
For the set of worlds studied in this work, we note thatwhether two worlds are equivalent boils down to counting
the number of behavior switches along the edges of their
behavior maps (counterclockwise, starting from the bottomedge). We can focus exclusively on the edges because ourworlds do not induce behavior maps with decision boundaries that behave differently in the middle (e.g. Fig. 19). By
counting the number of behavior switches along the edges,
we can represent the set of worlds in the same equivalenceclass as a count vector (see Fig. 1).
5.2. Intervention Transfer Between Equivalent Worlds

Thus motivated, we call two behavior maps equivalent if
the shapes of the decision boundaries between user behaviors in the behavior maps are the same and use an equivalence definition invariant to stretching or translation of theseboundaries. We formalize this in Definition 5.1.

Recall that our primary motivation for defining an equivalence relation on worlds is to develop intervention strategies in simple settings and transfer them to more complex
analogous ones. This section provides the formalism fortransferring interventions between equivalent worlds. In
Section 6.1, we will introduce a set of simple worlds towhich many commonly studied RL environments can be
reduced through our equivalence.

In the following, we assume, without loss of generality, that

Given a world W, we represent a single intervention on

5.1. Equivalence Between Behavior MapsWall World
Start

-10

-10

-10

Goal
(100)

0

-10

-10

-10

0

0

-10

-10

-10

0

0

-10

-10

-10

0

0

-10

-10

-10

0

0

0

0

Through Wall
00
AroundWall

Cliff World
0

0

0

0

0

Safe
0 Dangerous
0
0

0

0

0

0

0

0

0

0

Big-Small World

0

0

0

0

0

0

0

0

Start -

-

-

-

-

-

Goal

(a) A 6 × 5 Wall world where agents can (b) A 4 × 8 Cliff world where agents canpass directly through a costly wall (orange) walk close to the cliff and risk ruin (blue) oror take the longer, safer path around it (blue). keep space but walk farther (orange).

Start

0

0

Close
0 small0 R
Far large R

0

0

0

0

0

0

0

0

0

0

0

0

0

0

0

Small R

0

0

0

Big R

(c) A 5 × 5 Big-Small world where agentscan walk straight down to a small reward(orange) or farther to a bigger reward (blue).

Figure 2. Each atomic world has two qualitatively distinct behaviors (shown with blue and orange arrows). Each diagram shows what theworld looks like for one setting of the parameters, and other sizes are usually also valid.

a user’s myopia and confidence level as a real-valued pair(∆γ , ∆p ) ∈ I 2 that is added to the user’s current parameters.
Thus, a sequence of interventions defines a (piece-wiselinear) path, which we call an intervention strategy anddenote by τW , in the behavior map BW . Our goal is to map
an intervention strategy τW in BW , that realizes a behavior
change, to a strategy τW ′ in an equivalent map BW ′ thatrealizes an analogous behavior change.
We first observe that the continuous map h in Definition 5.1
induces a mapping from the set of user parameters related
to one world W to the user parameters related to W ′ , defined by h1 : I 2 → I 2 . Hence, every path τW defines apath τW ′ = h1 ◦ τW . Since h continuously deforms thedecision boundary of BW , it preserves the number of times
τW intersects the decision boundary in BW . In particular,
if τW represents an intervention strategy that achieves N
number of behavior changes in BW , then τW ′ is a strategy
that achieves the same number of behavior changes in BW ′ .
Note that, unlike knowledge generalization approaches in
RL wherein one computes a mapping between all parameters of two MDPs, our approach to intervention transfer
between two worlds by-passes explicit mappings between
the state and action sets of W and W ′ . Instead, we rely onh, the mapping between user parameter and policy spaces.
In practice, explicitly computing h can be difficult. In thenext section, we show that we can derive a more general setof heuristics for intervention design in a complex world by
reasoning about an equivalent simple world.

6. Atomic Worlds: Simple Representatives ofEquivalence Classes
Under Definition 5.1, we seek the simplest representative,
called atomic worlds, for each equivalence class. User behaviors can be characterized in atomic worlds, and the insights transferred thereafter to more complex equivalentworlds. We describe three atomic worlds and reduce commonly studied worlds in RL literature to our atomic worlds.
6.1. Atomic Worlds
We visualize an instance of each of the following worlds in
Fig. 2 and their corresponding behavior maps in Fig. 3.
The Big-Small world is an atomic world that captures a tradeoff between choosing a smaller, more convenient reward anda bigger reward that is more difficult to reach. In mHealth,this world reflects scenarios in which smaller immediate
rewards, such as the time saved by skipping PT for theday, preclude larger but delayed rewards, such as a fully
rehabilitated ankle.
The Cliff world captures settings in which a harmful absorbing state may be reached due to an action going awry. For
example, deciding the intensity of the PT regimen can bemodeled as a Cliff world. A high-intensity regimen could
accelerate recovery but also risk re-injuring the patient.
The Wall world captures the choice between a short, costly
path to the goal and a longer, free path to the same goal.
This can model the trade-off in choosing the type of physicaltherapy: virtual therapy may be more affordable, while inperson therapy is more costly and targeted.
In the above, we note that different aspects of user decisionmaking (e.g., choosing the intensity vs choosing the type(d) Chain world example
behavior map belonging
to equivalence class [1, 0,
1, 0].

0.99(e) RiverSwim world example behavior map belonging to equivalenceclass [1, 0, 1, 0].

0.4

Confidence level p
0.71
0.99

Continue
Finish0.71
Discount factor

0.99(f) Gambler’s Ruinworld (varying pC )example behavior map
in equivalence class [1,0, 1, 0].

0.4
0.4

0.71
Discount factor

0.99(c) Big-Small world example behavior
map belonging to equivalence class [1,0, 1, 0].
Gambler's (pF) Map
Cafe Map
Continue
Finish0.71
Discount factor

0.99(g) Gambler’s Ruinworld (varying pF )example behavior map
in equivalence class [1,1, 0, 0].

Confidence level p
0.71
0.99

0.71
Discount factor

Gambler's (pC) Map

Confidence level p
0.71
0.99

Confidence level p
0.71
0.99

Upstream
Downstream

Close small R
Far large R

0.4(b) Cliff world example behavior map belonging to equivalence class [1, 1, 0, 0].

RiverSwim Map

Big-Small Map

Donut
Vegan/Noodle

0.4

0.4

0.99

0.4

0.99

0.71
Discount factor

0.4

0.4

Confidence level p
0.71
0.99

0.71
Discount factor

0.4

0.4

Confidence level p
0.71
0.99
0.4

Chain Map

Dangerous
Safe

0.99

(a) Wall world example behavior map belonging to equivalence class [2, 0, 2, 0].
Exercise
DisengageConfidence level p
0.71
0.99

0.71
Discount factor

0.4

Cliff Map

0.4

Confidence level p
0.71
0.99

Around Wall
Through Wall

0.4

Wall Map

0.71
Discount factor

0.4

0.99

(h) Café world example
behavior map belonging
to equivalence class [1, 0,
1, 0].

Figure 3. Seemingly different worlds (bottom row) are equivalent to one of our atomic worlds (top row).

of therapy) in the same mHealth application (PT), can map
to different equivalence classes. We hypothesize that more
complex worlds (e.g., larger portions of the user decisionmaking process in PT) can be captured by compositions ofsimpler atomic worlds. In future work, we are interested
in characterizing the set of complex worlds that can bestudied through decomposition into atomic worlds. Further
discussion can be found in Section 7 and Appendix C.
6.2. Atomic Worlds Capture Commonly Studied RL
Environments
We compare the behavior maps corresponding to four typesof RL environments commonly studied in the literature:
Chain, RiverSwim, Gambler’s Fallacy, and Café worlds (details on each world are in Appendix A), and illustrate thatthe set of worlds they define reduces to the three atomic
worlds we identify in Section 6.1. We note that these RL
environments are diverse in their state and action spaces;
more interestingly, they are diverse in how they map to
real-life tasks. Thus, we expect that many useful mHealthapplications can be modeled by known atomic worlds orstraightforward combinations of atomic worlds (see Section 7 and Appendix C for more details), allowing us totransfer intervention design from familiar, simpler settings

onto unexplored and more complex ones.
Under our definition, Chain (Fig. 3d), RiverSwim (Fig. 3e),Gambler’s Fallacy V1 (Fig. 3f), and the Café worlds(Fig. 3h) are equivalent to the Big-Small world (Fig. 3c);these are worlds in which the user chooses between a readily available but small reward (i.e., disengaging in Chain,swimming downstream in RiverSwim, and performing theFinish action in the Gambler’s Ruin world) and a greater
but more time-consuming reward. Gambler’s Fallacy V2(Fig. 3g) is equivalent to Cliff World—both worlds have a“catastrophic absorbing state,” i.e., a nonzero risk of ending
up in a terminal state with a negative reward.
6.3. The Equivalence Definition Is Robust to Parameter
Perturbations in World Definitions
We want a world to remain within its equivalence class despite minor parameter adjustments (e.g., the world for amonth-long PT program should be in the same class as thatfor a 2-month program). This is evidence that our equivalence definition captures essential rather than incidental
qualities of applications.
In Fig. 4, we verify that the Big-Small world remains within
its equivalence class despite parameter changes, such as theConfidence level p
0.71
0.99

Close small R
Far large R

0.4

Reward ratio=0.1, Width=70.71
Discount factor

0.4

0.99

Confidence level p
0.71
0.99
0.4

0.99

Close small R
Far large R

0.71
Discount factor

0.4

0.99

Reward ratio=0.5, Width=7

Confidence level p
0.71
0.99

0.71
Discount factor

0.4

Reward ratio=0.5, Width=9

Close small R
Far large R

0.4

Confidence level p
0.71
0.99

Close small R
Far large R

0.4

Reward ratio=0.5, Width=30.71
Discount factor

0.4

0.99

Figure 4. A Big-Small world stays within its equivalence class formany different parameter combinations. The example behavior
maps have different values for the world width and the ratio of thesmall reward to the big reward, while the rest of the parameters are
fixed as height = 7 and Big far R = 300.

world’s width or the ratio of the big to a small reward. In
Appendix B, we provide additional evidence of how ourequivalence classes withstand perturbations across moreparameters for all 8 worlds investigated.
6.4. Heuristics for Intervention Transfer
Many real-world applications may be roughly mapped to
an atomic world through domain knowledge rather thancomputing an explicit map h, as in Section 5.2. For example,
behavior scientists can often describe the types of expecteduser behavior, e.g., “how many different behaviors are there
for users with very low confidence?”. Absent a map h, wecannot transfer an intervention strategy in precise terms.
However, the broader insights we obtain from studying the
behavior maps of atomic worlds can be easily transferred.
For example, conclusions we reach on the identifiability ofuser traits and the effectiveness of a particular warm-startintervention strategy (see Section 4) apply to all worlds
within the same equivalence class.

per. The existence of such classes may be especially relevant when we try to capture multiple distinct aspects of an
mHealth application in a single world. In future work, we
intend to explore the space of possible equivalence classes
more exhaustively.
World Compositions. Complex real-life scenarios are unlikely to neatly map to a singular atomic world; however,
we conjecture that some worlds may fall into compositionsof atomic worlds. Some initial experiments with composite worlds indicate that the composition of the Big-Smalland Cliff worlds leads to a behavior map that combines
the atomic worlds’ respective maps. See Appendix C for
examples of these experiments. This finding further supports the generality of our equivalence classes as seeminglycomplicated scenarios can be broken down into atomic
worlds that each capture a unique aspect of the application.
Other User-Intrinsic Obstacles. While we focus on myopia (γ user ) and confidence (Tpuser ) in this paper, we areinterested in modeling a wider range of user-intrinsic obstacles, as differences between the real and user-perceived
MDP. For motivation, works like Evans et al. (2016), undera different model of the user’s decision-making process, capture behaviors that cannot be parameterized as combinationsof γ user and Tpuser in the Café world. This observation raises
the question of whether our formal framework can capture
behaviors observed under other paradigms of sequential
decision-making (e.g. hyperbolic discounting, replanning,
etc).
Real World Dynamics vs. User Perceived Dynamics.
We note that the definition of behavior maps does not relyon the environment’s true dynamics T since the user’s policy
is computed based on their perceived dynamics Tpuser . In
reality, if T and Tpuser are significantly different, it would bereasonable to assume that the user iteratively updates Tpuser
as they interact with the real world.
The Topology of Behavior Maps. For the set of worlds inthis work, verifying that any two are equivalent reduces tomatching the number of behavior changes along the edgesof their behavior maps. That is, the decision boundaries of
their behavior maps have no interesting topology. See Appendix D for a discussion on intervention transfers between
worlds whose behavior maps are topologically distinct inmore nuanced ways. Future research could characterize
the set of worlds for which the decision boundaries of the
behavior maps are not as “well-behaved”.

7. Discussion & Future Work

8. Conclusion

Exhaustive World Search. We expect there to be manyequivalence classes outside the three identified in this pa-

In this work, we propose a novel tool, the behavior map, tostudy the relationship between user traits and user behaviors

for worlds in which the user acts as an RL agent. We define
an equivalence relation between worlds based on the shapesof their corresponding behavior maps. We show that intervention strategies can be transferred between equivalentworlds. In particular, we demonstrate that many seemingly
different RL environments map to one of a few equivalenceclasses, each represented by a simple atomic world. We further argue that many real-world applications can be mapped
to atomic worlds by leveraging domain knowledge in behavioral science and psychology. Finally, we show how broad
insight into intervention design for simple worlds can belifted to complex ones in the same equivalence class.

Acknowledgements
This material is based upon work supported by the National
Science Foundation under Grant No. IIS-2107391. Any
opinions, findings, conclusions, or recommendations expressed in this material are those of the author(s) and do
not necessarily reflect the views of the National Science
Foundation.

References
Ainslie, G. and Haslam, N. Hyperbolic discounting. Choiceover time, 1992.

Soni, V. and Singh, S. Using homomorphisms to transfer options across continuous reinforcement learning domains.
In AAAI, volume 6, pp. 494–499, 2006.

Biza, O. and Platt, R. Online abstraction with mdp
homomorphisms for deep learning. arXiv preprint
arXiv:1811.12929, 2018.
Evans, O., Stuhlmüller, A., and Goodman, N. Learning thepreferences of ignorant, inconsistent agents. In Proceedings of the AAAI Conference on Artificial Intelligence,
volume 30, 2016.
Givan, R., Dean, T., and Greig, M. Equivalence notions
and model minimization in markov decision processes.
Artificial Intelligence, 147(1-2):163–223, 2003.
Golub, M., Chase, S., and Yu, B. Learning an internal
dynamics model from control demonstration. In International Conference on Machine Learning, pp. 606–614.
PMLR, 2013.
Li, L., Walsh, T. J., and Littman, M. L. Towards a unified
theory of state abstraction for mdps. In AI&M, 2006.
Picha, K. J., Valier, A. S., Heebner, N. R., Abt, J. P., Usher,
E. L., Capilouto, G., and Uhl, T. L. Physical therapists’assessment of patient self-efficacy for home exercise programs. International Journal of Sports Physical Therapy,
16(1):184, 2021a.
Picha, K. J., Valier, A. S., Heebner, N. R., Abt, J. P., Usher,
E. L., Capilouto, G., and Uhl, T. L. Physical therapists’assessment of patient self-efficacy for home exercise programs. International Journal of Sports Physical Therapy,
16(1):184, 2021b.

Sorg, J. and Singh, S. Transfer via soft homomorphisms.
In Proceedings of The 8th International Conference on
Autonomous Agents and Multiagent Systems-Volume 2,
pp. 741–748, 2009.

Story, G. W., Vlaev, I., Seymour, B., Darzi, A., and Dolan,
R. J. Does temporal discounting explain unhealthy behavior? a systematic review and reinforcement learningperspective. Frontiers in behavioral neuroscience, 8:76,
2014.

van der Pol, E., Kipf, T., Oliehoek, F. A., and Welling,
M.
Plannable approximations to mdp homomorphisms: Equivariance under actions. arXiv preprint
arXiv:2002.11963, 2020.

Ravindran, B. and Barto, A. G. Approximate homomorphisms: A framework for non-exact minimization inmarkov decision processes. College of Information andComputer Sciences, University of Massachusetts, 2004.
Reddy, S., Dragan, A., and Levine, S. Where do you think
you’re going?: Inferring beliefs about dynamics frombehavior. Advances in Neural Information Processing
Systems, 31, 2018.
Shah, R., Gundotra, N., Abbeel, P., and Dragan, A. On thefeasibility of learning, rather than assuming, human biases for reward inference. In Chaudhuri, K. and Salakhutdinov, R. (eds.), Proceedings of the 36th International
Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5670–5679.
PMLR, 09–15 Jun 2019.
Shin, E., Swaroop, S., Pan, W., Murphy, S., and DoshiVelez, F. Modeling mobile health users as reinforcement
learning agents. arXiv preprint arXiv:2212.00863, 2022.

Zhi-Xuan, T., Mann, J., Silver, T., Tenenbaum, J., and Mansinghka, V. Online bayesian goal inference for boundedly
rational planning agents. Advances in neural information
processing systems, 33:19238–19250, 2020.

Ziebart, B. D. Modeling purposeful adaptive behavior withthe principle of maximum causal entropy. Carnegie Mellon University, 2010.

A. Descriptions of Each World from the Literature
In this section, we present the MDPs for the MDPs from the mHealth literature we study in this work, i.e., the Chain
World, RiverSwim World, and Gambler’s Ruin in Fig. 5, Fig. 6, and Fig. 7, respectively. Blue arrows indicate the behavior
corresponding to the blue behavior in the corresponding behavior maps and, likewise, for orange arrows.

Figure 5. In the Chain world, users can choose to “exercise,” or progress step-by-step to reach the desired goal. At each stage, they also
have the option to “disengage,” which results in a smaller reward and the termination of their progression.

Figure 6. In the RiverSwim world, the user can choose the rightward “upstream” action, which has a chance of successfully advancing
the user toward the larger reward but also a failure probability of staying in place or falling behind. They can also choose the leftward“downstream” action that deterministically moves the user toward the small reward on the far left.

Figure 7. In the Gambler’s Ruin (Bandit Problem) world, users can choose the “continue” action, which can either move the user one stepleft toward the dead-end state or one step right toward the goal state. They can also choose the “finish” action, moving them directly to thedead-end or goal state.

Cafe World
Veg.
(200)
0

Veg/Noodle
Donut

0

Donut
(50)

0

0

0
0

0

0

0

0

0

0

0

0

0
0

0

0

0
Donut
(50)

0

0

0

0
Noodle
(100)

0

0
0
0

Figure 8. In the Café world, users start at the bottom of a 13 × 8 grid and must choose where to get food. The choices are two donut
stores, a noodle shop, and a vegan café. The rewards of 50, 50, 100, and 200 represent the long-term satisfaction someone might feel fromeating the food. An important dynamic in this world is that users must pass the donut stores to reach the noodle shop or vegan café, andthe noodles are closer to the start than the vegan café. In our initial experiments, we look at the choice between the unhealthy choice(donuts) versus the comparatively healthy choice of noodles or vegan food. We indicate the paths users take when making the unhealthy
choice in orange and the healthy choices in blue. In Appendix C, we look at the dynamics of the behavior maps when all three choices areevaluated separately.

B. Parameter Perturbations for Each WorldIn the following, we present more comprehensive investigations into the invariance of the different worlds to changes in
the world parameters under our definition of equivalence. Different worlds have different sets of parameters to perturb andranges for which they remain invariant.

Cliff World Equivalence Class Invariancewidth=4.00
Safe
Dangerous

width=6.00
Safe
Dangerous

width=8.00
Safe
Dangerous

width=10.00
Safe
Dangerous

height=4.00

height=6.00

height=8.00

height=10.00

Safe
Dangerous

Safe
Dangerous

Safe
Dangerous

Safe
Dangerous

reward_mag=100.00

reward_mag=233.33

reward_mag=366.67

reward_mag=500.00

Safe
Dangerous

Safe
Dangerous

Safe
Dangerous

Safe
Dangerous

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

Default: height=5, width=9, reward_mag=100.0, neg_mag=-100000000.0, latent_reward=00.4

0.56 0.71 0.83 0.99 0.4

Discount factor

0.56 0.71 0.83 0.99 0.4

Discount factor

0.56 0.71 0.83 0.99 0.4

Discount factor

0.56 0.71 0.83 0.99

Discount factor

Figure 9. This array of graphs depicts behavior maps within the Cliff world across variations of three parameters: height, width, and reward
size. These maps are placed in the same equivalence class under our definition, indicating their robustness to parameter perturbations.

Big-Small Equivalence Class Invarianceheight=3.00
Close small R
Far large R

height=6.00
Close small R
Far large R

height=8.00
Close small R
Far large R

height=11.00
Close small R
Far large R

width=3.00

width=6.00

width=8.00

width=11.00

Close small R
Far large R

Close small R
Far large R

Close small R
Far large R

Close small R
Far large R

big_reward=100.00

big_reward=200.00

big_reward=300.00

big_reward=400.00

Close small R
Far large R

Close small R
Far large R

Close small R
Far large R

Close small R
Far large R

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

Default: height=7, width=7, big_reward=300, small_reward_frac=0.5

0.4

small_reward_frac=0.05
Close small R
Far large R

0.56 0.71 0.83 0.99 0.4

Discount factor

small_reward_frac=0.30
Close small R
Far large R

0.56 0.71 0.83 0.99 0.4

Discount factor

small_reward_frac=0.55
Close small R
Far large R

0.56 0.71 0.83 0.99 0.4

Discount factor

small_reward_frac=0.80
Close small R
Far large R

0.56 0.71 0.83 0.99

Discount factor

Figure 10. This array of graphs depicts behavior maps within the Big-Small world across variations of multiple parameters, such as
world size and magnitude of rewards. While the graphs are not identical, all these maps are still in the same equivalence class under our
definition, indicating their robustness to parameter perturbations.

L
RRRR

L
RRRRRRL
RRRRRRRRL
RRRRRRRRRR

disengage_prob=0.10

disengage_prob=0.37

disengage_prob=0.63

disengage_prob=0.90

L
RRRR

L
RRRR

L
RRRR

L
RRRR

lost_progress_prob=0.10
L
RRRR

lost_progress_prob=0.30
L
RRRR

lost_progress_prob=0.50

lost_progress_prob=0.70

L
RRRR

L
RRRR

goal_mag=4.00
L
RRRR

goal_mag=7.67
L
RRRR

goal_mag=11.33

goal_mag=15.00

L
RRRR

L
RRRR

disengage_reward=-1.00
L
RRRR

disengage_reward=-4.00
L
RRRR

disengage_reward=-7.00

disengage_reward=-10.00

L
RRRR

L
RRRR

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

Chain World Equivalence Class Invariance

Default: width=5, disengage_prob=0.3, lost_progress_prob=0.1, goal_mag=15, disengage_reward=-1, burden=-1
width=5.00
width=7.00
width=9.00
width=11.00

0.4

burden=-1.00
L
RRRR

0.56 0.71 0.83 0.99 0.4

Discount factor

burden=-2.00
L
RRRR

0.56 0.71 0.83 0.99 0.4

Discount factor

burden=-3.00
L
RRRR

0.56 0.71 0.83 0.99 0.4

Discount factor

burden=-4.00
L
RRRR

0.56 0.71 0.83 0.99

Discount factor

Figure 11. This array of graphs depicts behavior maps within the Chain world across variations of six parameters, including world sizeand disengagement probabilities. These maps are placed in the same equivalence class under our definition, indicating their robustness toparameter perturbations.

Gamblers World (pC) Equivalence Class Invariancewidth=6.00

D
RR

D
RR

width=7.00
D
RRR

width=8.00
D
RRR

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

Default: width=5, prob=0.8, gamma=0.9, big_r=10, small_r=0

width=5.00

0.4

big_r=5.00

big_r=37.00

D
RR

D
RR

0.56 0.71 0.83 0.99 0.4

Discount factor

big_r=68.00
DRR

0.56 0.71 0.83 0.99 0.4

Discount factor

big_r=100.00
D
RR

0.56 0.71 0.83 0.99 0.4

Discount factor

0.56 0.71 0.83 0.99

Discount factor

Figure 12. This array of graphs depicts
 behavior maps within the Gambler’s Ruin world across the width and reward size variations while
holding the failure probability pF constant. These maps are placed in the same equivalence class under our definition, indicating theirrobustness to parameter perturbations.

Gamblers World (pF) Equivalence Class InvarianceD
RR

width=6.00
DRR

width=7.00
D
RRR

width=8.00
D
RRR

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

Default: width=5, prob=0.8, gamma=0.9, big_r=10, small_r=0

width=5.00

0.4

big_r=5.00
DRR

0.56 0.71 0.83 0.99 0.4

Discount factor

big_r=37.00
DRR

0.56 0.71 0.83 0.99 0.4

Discount factor

big_r=68.00
DRR

0.56 0.71 0.83 0.99 0.4

Discount factor

big_r=100.00
D
RR

0.56 0.71 0.83 0.99

Discount factor

Figure 13. This array of graphs depicts behavior
maps within the Gambler’s Ruin world across the width and reward size variations while
holding the “continue” probability pC constant. These maps are placed in the same equivalence class under our definition, indicating
their robustness to parameter perturbations.

Riverswim World Equivalence Class Invariancewidth=6.00
LR
RRRRL

width=9.00
LR
RRRRRRRL

width=11.00
LR
RRRRRRRRRL

width=14.00
LR
RRRRRRRRRRRRL

big_r=2.00
LR
RRRRRL

big_r=6.00
LR
RRRRRL

big_r=11.00
LR
RRRRRL

big_r=15.00
LR
RRRRRL

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

Default: height=1, width=7, prob=0.8, gamma=0.9, big_r=10, small_r=1

0.4

small_r=1.00
LR
RRRRRL0.56 0.71 0.83 0.99 0.4

Discount factor

small_r=2.00
LR
RRRRRL0.56 0.71 0.83 0.99 0.4

Discount factor

small_r=4.00
LR
RRRRRL0.56 0.71 0.83 0.99 0.4

Discount factor

small_r=5.00
LR
RRRRRL0.56 0.71 0.83 0.99

Discount factor

Figure 14. This array of graphs depicts behavior maps within the RiverSwim world across the width and reward sizes variations. These
maps are placed in the same equivalence class under our definition, indicating their robustness to parameter perturbations.

Wall World Equivalence Class Invarianceheight=2.00
Around Wall
Through Wallheight=3.00
Around Wall
Through Wallheight=4.00
Around Wall
Through Wallheight=5.00
Around Wall
Through Wallwidth=5.00

width=6.00

width=7.00

width=8.00

Around Wall
Through Wall

Around Wall
Through Wall

Around Wall
Through Wall

Around Wall
Through Wallreward_mag=100.00

reward_mag=150.00

reward_mag=200.00

reward_mag=250.00

Around Wall
Through Wall

Around Wall
Through Wall

Around Wall
Through Wall

Around Wall
Through Wall

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

Default: height=4, width=6, neg_mag=-10, reward_mag=200

0.4

neg_mag=-20.00

neg_mag=-16.67

neg_mag=-13.33

neg_mag=-10.00

Around Wall
Through Wall

Around Wall
Through Wall

Around Wall
Through Wall

Around Wall
Through Wall

0.56 0.71 0.83 0.99 0.4

Discount factor

0.56 0.71 0.83 0.99 0.4

Discount factor

0.56 0.71 0.83 0.99 0.4

Discount factor

0.56 0.71 0.83 0.99

Discount factor

Figure 15. This array of graphs depicts behavior maps within the Wall world across variations of world size and reward magnitude. These
maps are placed in the same equivalence class under our definition, indicating their robustness to parameter perturbations.

Cafe World Equivalence Class Invariancevegetarian_reward=100.00

vegetarian_reward=150.00

vegetarian_reward=200.00

vegetarian_reward=250.00

Donut
Vegan/Noodle

Donut
Vegan/Noodle

Donut
Vegan/Noodle

Donut
Vegan/Noodle

donut_reward=25.00

donut_reward=41.67

donut_reward=58.33

donut_reward=75.00

Donut
Vegan/Noodle

Donut
Vegan/Noodle

Donut
Vegan/Noodle

Donut
Vegan/Noodle

noodle_reward=50.00

noodle_reward=83.33

Donut
Vegan/Noodle

Donut
Vegan/Noodle

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

0.4

Confidence level p

0.56 0.71 0.83 0.99

Default: vegetarian_reward=200, donut_reward=50, noodle_reward=1000.40.56 0.71 0.83 0.99 0.4

Discount factor

0.56 0.71 0.83 0.99 0.4

Discount factor

noodle_reward=116.67
Donut
Vegan/Noodle

0.56 0.71 0.83 0.99 0.4

Discount factor

noodle_reward=150.00
Donut
Vegan/Noodle

0.56 0.71 0.83 0.99

Discount factor

Figure 16. This array of graphs depicts behavior maps within the Cafe world, presented in (Evans et al., 2016), across variations of relative
reward for the types of eating options: Donuts, noodles/veggie. These maps are placed in the same equivalence class under our definition,
indicating their robustness to parameter perturbations.

C. Initial World Composition Experiments
In this section, we present two behavior map perturbations that indicate that more complex worlds can be decomposed as a
combination of several smaller atomic worlds.

Big-Small & Cliff Composition. The first, seen in Fig. 17a, is a Cliff world with an added option of disengaging. Thisdisengagement state is modeled as a state immediately below the start state in Fig. 2b. The disengagement is associated witha small positive reward, which can, e.g., be interpreted as the user’s sense of relief for not having to engage in physicaltherapy anymore (which is obviously smaller than the faraway reward of being fully rehabilitated). The compositionality
comes from the observation that the user now has two choices: (1) to engage or disengage, and (2) if they engage, be safe, or
take risks. The first choice is similar to a Big-Small world (disengage for a small reward or engage for an expected bigger
reward farther away).

Big-Small & Big-Small Composition. The second composition, whose behavior map is shown in Appendix C, is theCafé world with the choices between donuts, noodles, and vegan. Intuitively, the agent is now faced with two separate
decisions, where both are the choice between a small reward near and a relatively larger reward farther away.

0.4

0.71
Discount factor

0.4

0.99

(a) In the Cliff world with the possibility for disengagement,
the agent is effectively faced with first the choice between
a small and big reward, and then the choice of strategy for
traversing the cliff (safe or risky). This effect is evident fromthe new decision boundary that crosses from the top edge’s
left side to the bottom edge’s right side, just like a Big-Smalldecision boundary.

0.4

Disengage
Dangerous
Safe

Confidence level p
0.71
0.99

Confidence level p
0.71
0.99

Cliff Map

0.4

Cafe Map
Vegan
Donut
Noodle0.71
Discount factor

0.99(b) In the Café world, the agent is effectively faced with two,
sequential Big-Small worlds. This can be seen by considering
the boundary between the orange and blue areas as the first decision between a far and big reward (noodle/vegan) and a nearand small reward (donuts). If the user avoids the donuts, they
are faced with the choice between another far and big reward(vegan) and a near and (relatively) small reward (noodle).

Figure 17. Two worlds that appear to be straightforward compositions of two atomic worlds. Characterizing these compositions andunderstanding whether and how they can be useful is an interesting avenue for future research.

D. Consideration on the Interior of Behavior Maps
We have argued that the most important part of behavior maps is the extreme regions, i.e. the behavior along the edges.
One way to argue this is by using literature from the behavioral sciences, which has been one motivating factor. Anotherobservation one can make is the following. Let worlds (a) and (b) in Fig. 18 be two different worlds that belong to the sameequivalence class [1, 0, 1, 0]. Since n1 = n3 = 1, there can be no ambiguity about how the vertices are connected. However,we place no restrictions on where along the γ-axis the vertices are. If we decide the blue region is the desired behavior, but
we observe the user in the orange region (regardless of where), the optimal intervention will be the same in both worlds (a)and (b).
In the more complex case shown in Fig. 19, both worlds (a) and (b)P are in the same equivalence class [2, 0, 2, 0], despite
having very different middle regions. This disparity arises since i ni ≥ 4 and there will be more than one valid way toconnect the vertices. If we again imagine that blue is the desired behavior and orange is observed, these two worlds will stillshare the optimal behavior, as indicated with gray arrows in Fig. 19.
We have not proven this exhaustively;
Patomic worlds where this observation does not hold might still arise. From our initial
experiments, however, worlds with i ni ≥ 4 appear rare.
(a) Behavior Map

Optimal intervention(b) Behavior Map

Optimal intervention

Figure 18. Two different worlds with equivalent and simple behavior maps. Gray arrows indicate the optimal intervention for an agent
that exists in the orange region. Despite having their decision boundaries in different locations along the γ-axis, the best intervention is
the same.
(a) Behavior Map(b) Behavior Map

Optimal intervention
Optimal intervention

Figure 19. Two different worlds with more complex and differing behavior maps still belong to the same equivalence class. Despite
having very different interior regions, in many cases, the optimal intervention on an agent located in the orange region would be the same.