Multi-intention Inverse Q-learning for Interpretable Behavior Representation
Authors: Hao Zhu
Brice De La Crompe
Gabriel Kalweit
Artur Schneider
Maria Kalweit
Ilka Diester
Joschka Boedecker
Venue: Accepted by TMLR ()

Abstract: In advancing the understanding of natural decision-making processes, inverse reinforcement learning (IRL) methods have proven instrumental in reconstructing animal's intentions underlying complex behaviors. Given the recent development of a continuous-time multi-intention IRL framework, there has been persistent inquiry into inferring discrete time-varying rewards with IRL. To address this challenge, we introduce the class of hierarchical inverse Q-learning (HIQL) algorithms. Through an unsupervised learning process, HIQL divides expert trajectories into multiple intention segments, and solves the IRL problem independently for each. Applying HIQL to simulated experiments and several real animal behavior datasets, our approach outperforms current benchmarks in behavior prediction and produces interpretable reward functions. Our results suggest that the intention transition dynamics underlying complex decision-making behavior is better modeled by a step function instead of a smoothly varying function. This advancement holds promise for neuroscience and cognitive science, contributing to a deeper understanding of decision-making and uncovering underlying brain mechanisms.

Published in Transactions on Machine Learning Research (09/2024)
Multi-intention Inverse Q-learning for Interpretable Behavior Representation
Hao Zhu
Ceuh@es..niifreeee de
Deportment of Computer Science BBBBTT/Brminlnnks--BP Brann
Boainnnosssoao Tools, University of Freiburg
Brice De La Crompe
brice. de. da..a..comma....... wow..mauhhooooi...k./a/aabrrr
Institute of Biology III &
Brain Tools, University of Freibury
Gabriel Kalweit
kalweity Ccs www.::wwwwww....ro....... de
Department of Computer Science, University of Freiburg
Collaborative Reseorch Institute Intelligent Oncology (CRIION)
Artur Schneider
hatttr/cchhssaddoohoooooeeeeeeee de
Institule of Biology III & IMBIT//BrainLinkssBBB Bruin Tools, University of Freibury
Maria Kalweit.
Department of Computer Science, University of Freiburg
Collaborative Reseorch Institute Intelligent Oncology (CRIOON)
Ilka Diester
ika. dia.diester Boanddesttr@@bbooooiimmmi/rrrrrr
Institute of Biology III & IMBIT//Brcinlinkk- Brain Tools, University of Freiburg Bernstein Center Freiburg
Brainn
Joschka Boedecker
Department of Computer Science HWIBIT//BranLLinkee Tools, University of Freihurg Collaborative Research Institute Intelligent Oncology (CRITON)
Reviewed on OpenReview: https: openreview. / net/ formfo hrKHHmLUUF
Abstract
In advancing the understanding of natural decision-makinng processes, inverse reinforcement learning (IRI.) methods have proven instrumental in reconstructing animal's intentions un- derlying complex behaviors. Given the recent development of continuous-tine multi- intention IRL framework, there has been persistent inquiry into inferring discrete time- varying rewards with IRL. To address this challenge, we introduce the class of hierarchical inverse Q-learning (HIQL) algorithms. Through an unsupervised learning process, HIQL divides expert trajectories into multiple intention segments, and solves the IRL problem independently for each. Applying HIQL to simulated experiments and several real animal behavior datasets, our approach outperforms current benchmarks in behavior prediction and produces interpretable reward functions. Our results suggest that the intention tran- sition dynamics underlying complex decision-making behavior better modeled by step function instead of smoothly varying function. This advancement holds promise for neu- roscience and cognitive science, contributing to a deeper understanding of decision-making and uncovering underlying brain mechanisms.
1 Introduction
Characterizing decision-making behavior stands as fundamental objective within the field of behavioral neuroscience Niv 2009 Wilson & Collins 2019). Prior research has formulated variety of mathematical

---

Published in Transactions on Machine Learning Research (09/2024)
inchuding generalized linear models and models based on reinforcement learning. These forward models facilitate the understanding and comparison of secision-making strategies employed by both human and animal subjects. Additionally, they offer a low-dimensional behavioral representation suitable for regression analysis with ncural activities (Hat- tori et al. 2019 Hamaguchi et al. 2022). Forward models require an empirically defined reward function that guides subjects optimizing their behavior during decision-making. However, defining a comprehensive and suitable reward function can pose challenges in complex behavioral tasks (Alyahyay et al. 2023 Rosenberg 2021 h. Inverse reinforcement learning (IRL) [g et al., 2000 Arora & Doshi 2021 is popular ap- proach to recover reward function that induces the observed behavior, assuming that the demonstrator was maximizing its long-term return. Along with the significant successes of IRL in autonomous driving (Kalweit)
behavioral models across diverse tasks (Ashwood et al. 2022b; Beron et al. 2022
et al.
et al. 2020 Nasernejad et al. 2023). ; robotics (Kumar et al. 2023 Chen et al.] 2023), and healthcare do-
mains Coronato et al. 2020 Chan & van der Schaar 2021), it appears to be emerging as a valuable tool
for constructing mathematical bohavior models in neuroscience rescarch, as excmplified by Yamaguchi et al.
20188; ; Kwon et al. (2020); Alyahyay al. (2023).
Classic IRL methods seek to identify single, fixed reward function for a specific scenario. In contrast, Ashwood ct al. (2022a) suggested that animal's goals can evolve over time due to factors like fatigue, sa- tiation, and curiosity. Under this assumption, they proposed the dynamic inverse reinforcement learning (DIRL) framework, which parametrizes the animal's 's reward function as a smoothly time-varying linear com- bination of a small number of spatial reward maps, which are referred to as 'goal maps'. By assuming the existence of multiple goal maps with time- vareing weights, DIRL allows the instantancous reward function to vary conlinuously in time. This innovative framework achieved state-of-the-art performance in animal behavior prediction. Nevertheless, demands have emerged regarding an IRL framework incorporating dis- crele lirnevarying reward functions, particularly following the proposal by Ashwood et al. (2022b) that natural behaviors can be represented through a Markov chain characterized by alternating between discrete intentions.
To address this requirement, we propose the novel class of hierarchical inverse Q-learning (HIQL) algorithms, which extend the fixed-reward inverse Q-learning (IQL) framework from Kalweit al. (2020) solve multi- intention IRL problems. Based on the assumption that the intention transition dynamics follows Markov process, HIQL integrates an cnncctation-maximizatttt multiple intention segments in an unsupervised manner, and then solve the IRL. problem independently for each segment. We then compare the performance of HIQL and the shate-of-theear model DIRL (Ashwood) ct al. 2022a) in a simulated gridworld environment and a dataset of trajectorics from real mice navigating a 227-node-labyringg
(EM) approach to first divide expert trajectories into
(Rosenberg el al. 202T). The results show that our HIQL, algorithm outperforms DIRL, in behavior prediction in both benchmarks, and produces interpretable reward functions. Finally, we applied HIQL in real mice decision-making dataset from a dynamic two-armed bandit task (De La Crompe et al. 2023), and mathematically characterized exploitation and exploration behavior of animals during value-based decision-making.
2 Related work
Various approaches have been introduced to address multi-intention IRL problems. Notably, several frame- works based on parametric (Babes et al. 2011 Likmeta et al.] 2021), or mon-paramttric (Choi & Kim 2012 Bighashdel et al. 2021) approaches allow for learning from multiple agents with distinct reward functions. Based on the assumption that the expert sticks to the same intention through one episode, these meth- ods iterate between clustering expert demonstrations according to different intentions and solving the IRL problem for each cluster. However, unlike HIQL, these frameworks do not accommodate the scenario where intentions might alternate within the same episode.
In contrast, several approaches formulate MI-IRL problem as finding the maximun-likelihodd partition of each trajectory, where each segment was generated from different locally consistent reward functions. To solve the problem, Dimitrakakis & Rothkopf (2012); Michini & How (2012) extended the Bayesian IRL. franeworks (Ramachandran & Amir (2007) to multi-intention scenario by introducing a Dirichlet process prior different. reward functions. The resulting Dirichlet process mixture model (DPM) be solved via
2

---

Published in Transactions on Machine Learning Research (09/2024)
probabilistic inference, e.g., with Markov chain Monte Carlo Neal 2000). These algorithms do not require the number of intentions as a hyperparameter due to the nonparametric nature of DPMs. However, because of the nonparametric setting, DPMs inadequately model the temporal persistence of states and often create redundant states and rapidly switch among them, which eventually leads to poor interpretability of the results [Fx 2009). This problem was addressed by Surana & Srivastava) (2014), where they exlended the DPM to a sticky hicrarchical Dirichlet process hidden Markov model. Nevertheless, solving the IRL problem as Bayesian inference is still computationally intensive, which could be intractable even for moderately sized finite-state IRL problems. Under the same problem formulation, Nguyen et al. (2015) proposed probabilistic graphical model, generalizing the algorithm from Babes et al. (2011). This approach avoids the computationally intensive Bayesian inference problem, but is restricted to the case of linearly-solvable environments. Our parametric HIQL algorithm also adapls the 'trajectory segmentation + IRL' problem formulation, but on the other hand, avoids the aforementioned limitations present in the other approaches. Particularly, we propose that the parametric sctup, manually specifying the number of intentions allows the researchers to decide the level of abstraction on intentions according to their expertise and scientific interest, i.e., whether focus on the fine-grained difference between intentions (cf. 5.3
The statc-of-thc-art framework, DIRL (Ashwood et al. 2022a), can be considered as an extension of the maximum entropy IRL algorithm Ziebart et al. 2008 2010) to non-stationary rewards, where the reward function is assumed to be parameterized as a time-varying linear combination of a small number of non- linear spatial reward maps with Gaussian random walk prior over weights. Under this assumption, the borderline between intentions in DIRL blur, i.e., the intention transition is a continuous process, instead of a step-like procedure. However, such continuous time-varying reward assumed by DIRL is contradicted by the findings from Ashwood et al. (022b), where it was proposed that humans and animals switch between multiple discrele strategies during decision-making. Although in theory, one can try to approximale such step-like intention transition dynamics with DIRL by assigning a larger variance a to the Gaussian to make the random walk less smooth, we show empirically ($5.1) that there are only minor influence from this hyperparameter a on the learnt reward, indicating that it would be challenging for DIRL to capture such siep-like intention transition function during natural decision-making behavior. Instead, HTQT, assumes that the intention transition dynamics follows Markov process, which intrinsically discrete and aligns better to the discrete switching intentions than DIRL (ef. $5.1 and $5.2p.
Last but not the least, most of the aforementioned algorithms are model-based, relying on known tran- sition dynamics of the environment, whereas in many scenarios, the environment model is unknown. As an improvement, HIQL can also perform model-free learning, enabling the application in a wider range of environments.
3 Background
Markov decision processes. A Markov decision process (MDP) can be denoted by a tuple (S,A,Prr,y) where S and A denote the state- and action-space, respectively; The function P: AxAxS + [0, 1] is the state
transition function with P(s, a.)') s') Ps's s, a) and 1 ('PPss a. = 1; The function r: S x A - R defines
the reward function, and 7 E [0, 1] denotes the discount factor. Additionally, the function म: S x A - [0, 1] with T(s a) = P(a s) and 1" (s-)) = 1 is used to represent the policy according to which actions are selected in the MDP.
Inverse Q-learning. Given the set of expert demonstrations D in an MDP, where each trajectory & D is denoted with a sequence of state-action pairs: = {(s0. (o)).aa). ..... (sn an)}, the IRL problem consists in determining reward function that best explains the observed expert behavior in the form of demonstrated trajectories. Note that both the word 'demonstration" and 'trajectory' will be used subsequently. The former refers to one action execution and state transition of the expert, while the latter corresponds to a sequence of actions and state transitions for an entire episode. Unfortunately, IRL is generally ill-posed because infinitely many reward functions are consistent with the expert's observed behavior. To resolve this issue, various approaches have been proposed (see Arora & Doshi (2021) for a detailed review). Particularly,
3

---

Published in Transactions on Machine Learning Research (09/2024)
Kalweit et al. (2020) formulated the IRL problem maximum likelihood estimation (MLE) problem:
maximize subject to
Ee-D llog (E T.]] (s, exp (Q(s, a) log Σ exp ((s, -),, for all e 8, a a ((s,a) r(s.a) + + Eres P(s, a, s') max ((s',a"), for all E a,
(1)
where is the oplimization variable and D is the problem data. The objective of problem when the expert policy x (s(saa) is equivalent to (s) Hence,
is maximized
exp(Q(s, π (s, a) En'Es exp(Q(s,u())
((s,a) = Q(s, + log( a)) (s,((T((ss))
(2)
for all actions E A and e A-a where A_a AMa). Using the Bellman optimality equation in (2), the immediate reward of action in state can be expressed by the immediate reward of some other action b E A-a, the respective log-probabilitiss and future action-values (Kalweit et al. 2020):
r(s,a) a) = p(8, + card(A-a) DEA_a
E (r(x,b) - b)),
(3)
where card(A-a) denotes the cardinality of A-a; and
u(8 a) log( n (x, a))
E e'ES
P(s,a, a, max Q(s',
(4)
a'EA
Let cArd(A)= m, for each state S formulating for all actions A results in system of linear equations about rss.:):
a(s,a1) r(s, a2)
u(s,ai) _ EsEA u(s;b) u() m-1 Eved (s,
m-I
(5)
r(s,, am)
(lss0mi) m-1 EsEA M(s
Thus the unknown reward function r can be obtained by solving (5) for all s E S via least squares (Kalweit
et al. 2020). This approach is called the inverse action-value iteration (IAVT) (Algorithm [ a model-basod algorithm that solves the IRL problem analytically in closed-form. When the transition dynamics P is unknown, stochastic approximation can be applied to (3) and (4p such that can be obtained a sampling- based manner (Kalweit et al. 2020). The resulting inverse Q-learning (TQL) algorithm (Algorithm 2 is model-free extension of LAVI. The class of IQL algorithms solve the MDP underlying the demonstrated behavior only once, leading to speedup of up to several orders of magnitude compared to the popular max- imuun entropy IRL algorithm (Ziebart et al. 2008) and some of its variants. In addition, it can accommodate arbitrary non-lincar reward functions.
4 Hierarchical inverse Q-learning
We formulate the multi-intention IRL problem under the following assumptions:
Assumption 4.1. Each expert demonstration is generated according to the Boltzmann optimal policy under one of the reward functions in a K-dimensional finite set R = TKk,, with each corresponding to one specific intention.
Assumption 4.2. The probability that one demonstration is generated under reward function r E R. is controlled by a Markov chain with initial state distribution IT and transition ion matrix A, where the vector IT and each row of the transition matrix Ag. i I,....K represent some probability distribution over the set R, i.e., E K-1 and A₁ E K-1, where A AK {a E R K x z T. =1] is a (K dimmnnssio
probability simplex.

---

Published in Transactions on Machine Learning Research (09/2024)
The resulting hierarchical decision process of expert following these assumptions from the perspective of a re- searcher is depicted graphically in Figure ! where the first two rows represent the MDP, and the last row repre- sents the Markov chain for intention Iransition dynamics. Solving IRI. problems on such decision network with pa- rameter space θ (II, RJ consists in determining 1) set of reward functions, and 2) the reward function in- dex for each demonstration that best jointly explain the observed expert behavior. An expectation-maxxiizaatee maximizationn (EM) algorithm can be devised to iteratively learn O For convenience, we introduce 7j = {zo...... 2n to be the pre- dicted sequence of reward function indexes for trajectory Ç E D. Then each iteration of the EM process can be written predicted reward function indexes rp.
St
P
81
St11
T(-1
A
Pt
A
Cr+1
Figure 1: Graphical representation of expert's decision process.
MLE problem where the likelihood is obtained by marginalizing the sequence of
maximize
J J ( = Eg-D., [log P (5.7 e*)],
(6)
where ++ is the optimization variable and D the problem data.
Theorem 4.3. Solving problem c is equivalent to solving a sequence of optimization problems:
maximize (over IIT) EęND Σ Di=l P( 1) subject to
(7)
IT+ 2 0, 11
maximize (over A+) EAND Σ K E P(z - = 1 (5, log Atg subject to
(8)
At 20 0, 1 A = 1,
K,
and
maximize (over r
E En P( = (sn
subject to
(s, exp (Q(s, a) - log Σ exp Q(s
(s. 1), for all s S, a A Q(s,a)= = (,, (s.a) + Eves P(s, a,) s') max Q(s',a')) for all s slsse, S, a
(9)
Proof. See Appendix A.1
In practice, to evaluate the objective functions of 7
88 and 19) the Baum-Welch algorithm (Baum
Petric 1966) can be appliod to obtain the required postcrior probabilities P( = &, θ) and P(2 2t = j &, θ) (ef. Appendix A.2 Then according the Gibbs' inequality, the optimum of and are achieved by
II.
E P(20 = O)). END
i= 1,...,K, K,
(10)
and
A EgnD, P(2-1 i,
&. e)] EE-D, [P (2 = (
i= i..., K, j= 1..,,,,
(11)
To solve problem mooe that it has the same structure as thus it can be addressed by the class of IQL algorithms (Algorithm I and 2 with slight adaptations. Specifically, for cach i = 1,..., K, subset of expert demonstrations from will be sampled w.r.t. the posterior probability P(* = O),,, then either Algorithm D or Algorithm (depending on whether the transition dynamics P available) will be applied 2
this subset of demonstrations so that rt can be obtained. Assembled, we call the above process of solving multi-intention IRL problems the hierarchical inverse Q-learning (HIQL) algorithm. The corresponding pseudo-code is listed in Algorithm 3
5

---

Published in Transactions on Machine Learning Research (09/2024)
5 Experiments and discussion
We evaluate the performance of HIQL on a simulated gridworld environment and a real mice behavior dataset obtained from a 127-node labyrinth navigation task (Rosenberg ct al. 2021]. and compare to DIRL (Ashwood et al. 2022a). We then show the polential of performing model-Iree learning and detecting exploration behavior with HIQL on a mice reversal-learning task (De La Crompe et al. 2023)
5.1 Gridworld benchmark
The simulated gridworld euvironment is 5 5 map (Figure 2), where an agent can choose between going up, down, left, right, to stay in place per time step. The expert starts at origin (0,0) and any of its actions can achieve the intended state with 90% probability, while the other 10% lead to random directions. It has two possible policies, where 7 prefers going to the destination (4, 4) and ahhnndon prefers returning to origin (0.0). The expert starts an episode always with the ng0al policy. While moving to the destination, the expert will encounter barriers '#' at some states. Every time the expert runs into barrier, has 30% probability of switching to different policy. Each episode ends when the expert reaches either the origin or the destination, and at the 8th time step, if the episode has not finished, the expert will have a 50% probability of switching to x zabandon (ef. Appendix C.1). Such intention transition dynamics of the simulated expert as described above is better aligned to a step-function, instead of smoothly varying function.
0-
#
1
#
2
#
#
#
3
4-
Figure 2: The gridworld environ- ment.
We compared between the performance of HTAVT (the model-based variant of HIQL) and DIRL with 1 2 intention(s). Note that for the single intention case, HIAVI falls back to normal LAVI and DIRL falls back to maximum causal entropy IRL (Ziebart et al.) 2010). Trying to enable DIRL to better capture such step-like intention transition dynamics, we varied the hyperparameter that controls the smoothness of the time-varying reward in DIRL. - the variance a for the Gaussian of the random walk prior, from 0.01 to 10. A larger a value corresponds to a less smooth intention transition dynamics in DIRL. The whole expert demonstration dataset consisted of 1024 trajectories. All evaluated algorithms were fit to multiple sub-datasets with different number of expert trajectories, and each dataset was analyzed using 5-fold cross-validation. We list the detailed information about model training in Appendix C.1
In general, HIAVI outperformed DIRL in predicting the expert behavior as indicated by the log- log-likel on the test dataset (Figure Ba). Notice that HIAVI already had a better performance compared to DIRL even in the collapsed single intention model, indicating that LAVI serves as a better inner-loop (IRL problem) solver than maximum entropy IRL. This is probably because TAVT learns the unknown expert reward closed-form by solving a system of linear equations instead of performing parameter estimation as maximum entropy IRL docs, which introduces approximate error (Kalweit et al. 2020 Comparing the single- and multi-intention variants, both HIAVI and DIRL had an increase in prediction performance. Particularly, the HIAVI almost. achieved the expert level as the number of trajectories increased, while the performance improvement for DIRL was only minor. Besides, different a values of DIRL seem not to have a significant influence on the DIRL performance when there is enough expert trajectorics in the datasct.
Next we compared HIAVT and DIRL abilities capturing the expert's intention transition dynamics (Fig- ure 3bb and recovering the expert policy (Figure 3c/. The analysis was performed based on the learnt policy using the whole dataset with all 1024 trajectories (with cross-validation applied). As measure of performance in policy recovery, used the expected value difference (EVD) metric (Levine et al., 2011 EVD is defined as the mean square error between the state-value under the true reward function for the expert policy and the state-value under the true reward for the optimal Boltznann policy w.r.t. the learnt reward. It provides an estimation the sub-optimality of the learnt policy under the true reward function. For HIAVI and DIRL with 2 intentions, the inferred intentions were assigned to the best-fit ground truth intentions, and the two learnt policy was evaluated under the corresponding true reward individually obtain the respective state-values Under the single-intention setup, since only one learnt policy could be
6

---

Published in Transactions on Machine Learning Research (09/2024)
(b)
0.0
1.0
log-likelihood
ground
ground
HIVAI intention)
P('abandon'
HIAVI intentions)
00.5
HUAI intentions)
DIRL (2 0.01)
DIRL
0.5
DIRL (2
-1.0
DIRI (2 maps, 0.01)
DIRL (2 maps,
DIRL (2 maps, 0.1)
DIRL (2 maps,
DIRL (2
-1.5
0.0
DIRL (2 maps, 10)
500
1000
2
6 8
10
12
14
# trajectories
time
(c)
ground truth
HIAVI
DIRL
(1
intention)
(2 intentions]
[1 map)
(2 maps, 0.01)
(2 maps. 0.1)
(2 maps,
(2 maps, 10)
'ooa''
4.2
9.5
0.0
2.7
'abandon'
4.2
9.5
0.0
0.5
1.8
0.0
0.1
0.0
1.8
0.0
2.1
0.0
0.0
0.1
HIAVI
DIRL
2 maps
intention
2 intentions
1 map
0 0.01
0 0.1
'goal'
13.96 + 0.23
5.58 0.47
47.10 + 0.00
27.63 + 6.55
2.97
0.67
11.08 3.56
37.97 +: 1.29
'abandon'
45.55 + 0.07
6.39 + 1.80
48.17 + 0.00
45.90 + 0.15
45.48 + 0.49
48.15 + 0.05
48.15 0.06
Figure 3: Results for the gridworld benchmark. (a) Comparison of HIAVI and DIRL datasets with different number of expert trajectories, represented log-likelihood the test dataset (mean + standard error, 5-fold cross-validation). (b) Predicted intention dynamics from HIAVI and DIRL using the outputs from the best cross-validation fold, represented as the posterior probability of the 'abandon' intention and averaged across all trajectories (mean + standard error across 1024 trajectories). (c) Visualization of the ground truth and learnt state-value functions from the best cross-validation fold (top), and the corresponding EVDs (bottom, mcan standard crror, 5-fold cross-validane validation) from HIAVI and DIRL.
provided by the algorithms, the same output learnt policy was evaluated twice repeatedly under the different truc reward functions, respectively. Aligning with the results indicated by Figure Bal HIAVI had smaller EVD values than DIRL even under single-intention setup (Figure Bch. Particularly, the learnt policy from single-intention HIAVT exhibits mixed characteristic of both 'goal' and 'thirsty' intentions but is more close to the former, whereas the learnt policy from single-intention DIRL deviates from both of the two intentions. This explains why HIAVI already had a better prediction log-likelihood than DIRL under the single-intention setup. Then by introducing the second intention, the best performance recovering expert policy under both intentions was achieved by HIAVI (5.58 + 0.47 for gooal and 6.39 + 1.80 for abandon)) While by selecting an appropriate value, DIRL could reconstruct the 'goal' policy at very high level (with 2.97 + 0.67 EVD when 0.1), it struggled to learn the policy under 'abandon' intention. As result, the HIAVI predicted posterior probability of the expert remaining in the 'abandon' intention at different time steps in a single episode matched exactly with the ground truth (Figure 3b whereas DIRL seemed to overestimate this probability at the beginning (where the expert reached the barrier for the first time), and underestimate it after the 8th time step.
To sum up, the comparison between HIAVI and DIRL on the gridworld benchmark suggests that HIAVI better capture step-like intention transition dynamics and reconstruet the underlying expert policies

---

Published in Transactions on Machine Learning Research (09/2024)
(a)
(b)
random
DIRL
10
BIC
4 maps
1 intention
HIAVI
DIRL
HUM
Intentions
10a
2
3
2.0
1.5
1.0
0.5
0.0
# intentions/mman
log- likelihood
(c)
(d)
(e)
intention 1: 'tired'
Intention 2: 'thirsty'
1.0
P(tthissty))
intention p
0.98
0.02
0.5
0.124
0.876
0.01
0.0
P(tiieed)
intention
0
10
20
time
Figure 5: Results for the navigation benchmark of the water-restricted cohort. (a) Comparison HTAVI, DIRL, and random policy, represented as log-likelihood on the test dataset. (b) BIC function of the number of intentions in HIAVI. (c) Learnt policy (red arrows and crosses) in the environment and corresponding state occupancy (grey colormap) under different intentions. (d) Predicted intention dynamics from HIAVI, averaged across all 200 trajectories. Solid and shaded curves denote the mcan and standard error across 200 trajectories.
(e) Inferred intention transition matrix from HIAVI.
than DIRL. Although in theory one can try to approximate such step-function with DIRI, by assigning a larger variance a to the Gaussian to make the random walk less smooth, it has been shown empirically in this experiment that it's not trivial in practice since the inffuence of this hyperparameter o on DIRL would only be limited.
5.2 Real-world mice navigation benchmark
The expert demonstrations in this real-world dataset were originally col- lected by [Rosenberg et al. (2021) from a 127-node labyrinth navigation task. As shown Figure 4 each black dot represents one state the en- vironment and the subjects can select from 4 actions: left, right, reverse, and stay at each state. One of the 127 states is occupied with water resource (the blue square), and the optimal path from the labyrinth en- trance to the water port is depicted as blue line. Two cohorts of 10 mice moved freely in dark through the labyrinth over the course of 7 hours, where one cohort of animals was under water restriction while the other was not. Difference in water restriction condition resulted in different an- imal behavior between these two cohorts in the environment. For model evaluation, HIAVI and DIRL with varied intentions from 1 to 4 were trained independently the water-restricted and water- unteetrerett ex- pert demonstration datascts (with 200 and 207 animal trajectories, respectively). 20% of the trajectories from each dataset were held out as a test set. We list the detailed information about trajectory preprocessing and model training in Appendix C.2
Figure 4: The labyrinth environ- ment.
We first compared between HIAVI and DIRL on the trajectorics collected from the group of water restricted animals. First of all, as the number of intentions increased, the log-likelihood value for HIAVI predictions increased significantly, whereas the improvement of DIRT performance rather minor (Figure 5a This
8

---

Published in Transactions on Machine Learning Research (09/2024)
(a)
(b)
random
map
10
DIRL
BIC
4 maps
intention
HIAVI
DIRL
HBM
Intontions
10a
2
3
2.0
1.5
1.0
0.5
0.0
# intentions/ 'maps
log- likelihood
(c)
(d)
(e)
intention 1: 'exploring'
intention 2: 'tired'
1.0
intention t
I-
+0.441
0.559
P
U1
0.5
2.0.054
0.946
P( l''exploring'
2
0.01
0.0
intention t
10
time
Figure 6: Results for the navigation benchmark of the water-unrestricted cohort. (a) Comparison of HIAVI, DIRL, and random policy, represented as log-likelihood on the test dataset. (b) BIC as a function of the number of intentions in HIAVI. (c) Learnt policy (red arrows and crosses) in the environment and corresponding state occupancy (grey colormap) under different intentions. (d) Predicted intention dynamics from HIAVI, averaged across all 207 trajectories. Solid and shaded curves denote the mcan and standard error across 207 trajectories. (e) Inferred intention transition matrix from HIAVT.
led to a distinguishable outperformance of HIAVI compared to DIRL in this benchmark. To check whether the better performance of HIAVI compared to DIRL was due to a larger number of papameters, we then calculated the Bayesian information criterion (BIC) for these two approaches (Figure 5b). It shows that the BIC values for HIAVI are consistently lower than those for DIRL when the number of intentions exceeds one. To analyze the recovered expert policies by HIAVI, we selected the number of intentions K = 2 where the lowest BIC value was achieved. The learnt mice policy under intention 1 ('tired') displays a preference of moving out from the water port towards the maze entrance and stay, while the policy under intention 2 ('thirsty") guides the mice directly to the water port along the optimal track. Correspondingly, under the 'tired' intention, the highest state occupancy was observed at the entrance of the environment, while under "thirsty', it was observed at the water port (Figure 5ch. To visualize the predicted intention transition dynamics, we computed the posterior probability over mice's intentions across all trajectories. The recovered temporal intention dynamics shows high probability of the 'thirsty' intention at the boginning but later drops gradually, the 'tired' intention gradually becomes dominant (Figure 5d Besides, the intention transition matrix learnt by HIAVI (Figure 5eb suggests that while both the 'thirsty" and 'tired' intentions are stable, the probability that the animals switch from "thirsty' to 'tired' is relatively larger than the other way around. These observations indicate the following interpretations on the mice behavior within this cohort, which also align well with our intuition: The water-restrieted mice are eager to find the water port as the trial starts, and then gradually return back the labyrinth entrance as they obtain enough water. Once they have decided to return, the probability to go back to search for the water port would be quite low.
When comparing the dataset collected from the unter-unrestricted animals, HIAVI also had larger test log-likelihood and a lower BIC value than DIRL (Figure 6a and 6b). Interestingly, for this dataset, increasing the number of intentions seemed not to improve the performance of DIRL at all, which might be a result of a more random behavior the animals due to the lack of motivation to find water. We then selected HIAVI with K according to BIC (Figure 6b) to analyze the recovered policy and intention dynamics. For this cohort of animals, the inferred policy under two intentions exhibits 'exploring' and 'tired' behavior. The policy under "exploring' tends encourage the animal lingering in the labyrinth, whereas the policy under
9

---

Published in Transactions on Machine Learning Research (09/2024)
(b)
(c)
(d)
-0.88
log. likelihoood
0.8
global
P(win-stay)
log- se-likllihodd
0.4
4000
Swwitch)
Piicse switch
-0.40
0.7 -
00.42
0.2
BIC
3000 -
0.6
-0.44
train test
2000
0.0-
0.4
exp ottation
disongaged
eepploration
0.46
F.
history length
# intentions
#
intentions
intention
(e)
(I)
(s)
1.0
1.00
contect
3666
Pisploratioon"
P('explcitation)
Picorect)
0.74
0.10
0.75
0.5
P P(ddsengaged")
0.56
0.05
P( P('exoloration"
0.50
0
10
15
trial #
intention
Pi (llexploitation)
Figure 7: Results for the mice reversal-learingg dataset. (a) Log-likelihood (mean + standard error, 5-fold cross-validation) as a function of history length of single intention HIQL. (b) Change in test set log-likelihood as a function of the number of intentions in HIQL with 4, 3, relative to the fQ-learning model (labeled 'F.). Each trace represents single mouse, averaged over vooss-vllidttion. Solid black indicates the mean across animals, and the dashed curve indicates the example mouse. (c) BIC as function of the number of intention in HIQL with é, = 3. (d) Learnt mice policy represented with the probability of switch, win- stay, and lose-switch. Fach grey curve denotes one mouse. (e) Average task performance and the predicted intention dynamics. Solid and shaded curves denote the mean and standard error across 9 animals. (f) Overall lask performance (gray) and the performance under different intentions, mean + standard error across animals. (g) Relationship between the probability of the "exploitation' intention, 5 trials before block switch and the probability of the 'exploration' intention, trials after block switch, mean + standard error across 9 animals.
'tired' intention steers the animal back to the maze entrance (Figure Gch. Correspondingly, the posterior probability of 'exploring" initially dominates at the beginning of the trial but is generally surpassed by the 'tired' intention over time (Figure (d). The inferred intention transition matrix (Figure Gol suggests that the 'exploring' intention unstable, meaning that it has larger probability of switching to "tired' intention compared to staying al 'exploring' In contrast, after switching to the 'tired' intention, animals tend to keep this intention until the end of trial. These observations again consistent with intuition. Since the water-sceking motivation for this group of vater-unrestricten mice is very low, they would prefer to explore the labyrinth, rather than search for the water port..
In conclusion, the above results show that HIAVI outperforms DIRL in predicting animal behavior on this labyrinth navigation benchmark for both two cohorts of mice, and is able to generate interpretable reward functions underlying different intentions. On the other hand, the significant difference bet ween the prediction performance of HIAVI and DIRL indicates that the intention transition dynamics during natural decision- making behavior may be better described with a step-function, rather that a smoothly varying function assumedd by DIRL.
5.3 Application to mice reversal-learning behavior
Finally, we apply the HIQL algorithm behavioral data recorded from a group of mice engaged dynamic two-armed bandit reversal-learning task. The dataset initially reported by De La Crompe al. (0233
10

---

Published in Transactions on Machine Learning Research (09/2024)
At the beginning of the task, water-restricted mice may choose from two available spouts, left (L) and right (R), with random one of them assigned water as extrinsic reward. After reaching an online performance of 75% correct in a 15-trials sliding average window and a minimun 20-trials block, the rewarded spout is changed. To formulate the MDP, we define the action space as: A= = fleft, right). Every state s E S is defined with a tuple of truncated history information: 8, = (ft-1...... Pe-e, a1-1,-... a.-1)...;((ta))) where the positive integer la denotes the history length, E A denotes history action, and yp E (correct, error) represents trial feedback, i.e., the extrinsic reward. Such MDP formulation allows us to avoid explicitly establishing a partially observable MDP. Different from the first two experiments, we applied the model-free variant of HIQL, in this dynamic reversal-learning task since the environment model is unknown.
We begin from selecting the hyperparameter &n using single intention HIQL (equivalent to vanilla IQL). The dataset for all subsequent analysis consists of 64 sessions (trajectories) from 9 different animals and a 5-fold cross-validation was applied over the trajectorics. We compared the log-likelihood on training and test sets of multiple IQL fitting with different l, (Figure 7a). The log-likelihood on test sets shows a bell-shaped curve as l, increases, indicating an overfit on the training set when &, > 3. Note that there is an abnormal drop on training set log-likeliig ligliiklihood at large Eys. This can be explained with the insufficient sampling given the fixed set of expert demonstrations, since the size of the state space grows exponcntially as the history length lA increases. The best test log-likelihoodd was achieved by l = 3, which was selected for subsequent steps. Next, to determine the number of intentions K under which mice demonstrated the trajectories, we fit multiple HIQL with varying number of intentions. In this step, we additionally applied a forgetting Q- learning (fQ-learming) model (Beron et al., 2022), which has been widely recognized as a prominent forward behavioral model for the reversal-learning task, serving a baseline for comparative analysis. We found that the multiple intention HIQL fitting substantially outperformed the single intention models (Figure 7b The BIC w.r.t. different K indicates that both K = 2 and K = 3 are reasonable values (Figure (c).. Taking biological interpretability into account, we selected K = 3 for the following discussion. We list additional details about the above procedure of fitting HIQL. to this mice reversal-learning dataset in Appendix
C.3
The inferred mice policies from HIQL represent dif- ferent strategies in the task under three intentions (Figure 7 The policy under intention ('exploita- tion') displays a strong preference of 'win-stay' and 'loseswwicch' strategy, which is the optimal policy in this deterministic reward bandit task, i.e., stay on the same side for the next trial if it is rewarded in the current trial, and switch to the other one otherwise. On the other hand, under intention 2 ('disengagod') the policy exhibits a preference for exploitation when the previous trial was successful, but following error trials, it employs a random action selection, indi- cated by the ca. 0.5 probability of exccuting a 'lose- switch?. Lastly, under intent ion 3 3 ('exploration") the subject consistently favors selecting the option oppo- site to the one chosen in the precoding trial, irrespec- live of whether they had won or lost in that particular instance. As shown in an example session (Figure Sa the 'exploration' intention predominantly occurs at the onset of a block and only lasts for a few trials, whereas the other two intentions may persists over multiple consecutive trials. These results suggests that compared to 'exploitation' and 'disengaged', the 'exploration' intention is not stable, which aligns well with the learned intention transition matrix (Figure Sb). Additionally, it becomes evident that error trials tend to coincide with the trials where the posterior probability of 'disengaged' and 'exploration' intentions are dominant. The intention dynamics averaged across all blocks closely resembles those observed in the example session (Figures 7ep. As each block begins with the animal's performance at a relatively low level, there is decline in the posterior probability associated with the 'exploitation' intention, accompanied by an increase in the probabilities of the other two intentions associated with suboptimal exploratory strategies. Then the subjects' performance steadily improves, the 'exploitation' intention progressively reasserts its dominance. In contrast to the cohort's 's general correct rate of 0.74 + 0.02, the subjects performed significantly better
(a)
(b)
1.0-
0.132
0.008
wwww
Intention
cornect
3MP
0.7
0.547
0.5
0.196
0.37
reploction
20
411
intention C
trial
Figure 8: (a) Predicted intention dynamics for an ex- ample session. Dots and triangles indicate mice ac- tion. (b) Inferred intention transition matrix for this example mouse.
11

---

Published in Transactions on Machine Learning Research (09/2024)
within the "exploitation' intention, achieving correctness rate of 0.86 + 0.01, whereas they only attained lower correctness rates of 0.64 + 0.03 and 0.56 + 0.06 in the two other intentions (Figure 7 Interestingly, it was observed that the mean posterior probability of the "exploration' intention at the beginming of a new block showed a positive correlation with the average probability of the 'exploitation' state at the end of the preceding block (Figure Tgh- These results confirm the difference between the two suboptimal intentions 'disengagod' and 'exploration': the first strategy explores passively upon error, while the second strategy involves a deliberate, exploration-oriented action selection when the subject is highly engaged and possess a good understanding of the environment. Although the 'exploration' intention is suggested to be unstable, it provides the opportunity to characterize a type of learning strategy based on error, which has been shown experimentally by Kononowicz et al. 2022).
In summary, by applying HIQL in a real-world mice reversal learning dataset, we demonstrate the possibility of performing model-free learning and producing interpretable behavior characteristics of animal experts via HIQL. We also detocted typical exploration strategy of mice during this value-basod Docision-making fask via mathematical modeling, which has only be achieved in perceptual decision-making tasks (Ashwood] et al. 2022b). Last but not the least, the model training procedure that we proposed in this experiment enables fits to individual subjects without running into the challenge of post hoc aligmment of intentions from different individuals. Although the mice subjects exhibited similar behavior during our experiment, limiting the magnitude of observed differences in individual solutions, we anticipate that this method will prove beneficial in tasks where subjects exhibit greater variance in policy preferences.
6 Conclusion
This paper proposes the class of EM-based hierarchical inverse Q-learning (HIQL) algorithms, extending the IQL algorithu (Kalweit et al. 2020) for multi-intention IRL problems. Empirical results demonstrate that HIQL, outperforms the stale-of-the-art on both synthesized and real-world datasets, and is able to produce interpretable behavior characteristics. We also mathematically characterized typical exploration behavior of rodents during valuo-based decision-making using HIQL.
Compared to the state-of-the-at algorithm for characterizing animal behavior, DIRL, (Ashwood et al. 2022a the advantages of our HIQL algorithm are two fold. First, the assumptions about the underlying intention transition dynamics in HIQL align better with those observed in real-world behavioral experiments that animal and human alternate between discrete strategies during decision making (Ashwood et al. 2022b), rather than optimize their policy over contimuous time-varying reward as assumed by DIRL. Second, the inner-loop IRL problem solver adapted by HIQL, the class of IQL algorithms, were shown to have better performance in learning the expert's unknown reward and higher computing efficiency than the maximum entropy IRL algorithm used by DIRL (Kalweit et al. 2020).
As future work, we plan to relax the Markov assumption about the intention transition dynamics in HIQL to incorporate Markovan intention dynamics, and also extend HIQL for an unknown number of reward functions (Dimitrakakis Rothkopf 2012 Michini & How 2012 Surana & Srivastava, 2014].
Acknowledgments
This work has been funded as part of BeainLinks--raiinn Brain'Tolss, which is funded by the Federal Ministry of Economics, Science and Arts of Baden-Wiin Württemberg within the sustainability programme for projects of the Excellence Initiative IΤ; as well as the Bernstein Award 2012, the Research Unit. 5159 "Resolving Prefrontal Flexibility" (Grant DI 1908/11-1) and the Deutsche Forschunggemcinschaff (Grants DI 1908/3-1 and DI 1908/6-1) to I.D., and CRC/TRR 384 "IN-Code" to I.D. and J.B.
References
Mansour Alyahyay, Gabriel Kalweit, Maria Kalweit, Golan Karvat, Julian Ammer, Artur Schneider, Ahmed Adzemovic, Andreas Vlachos, Joschka Boodecker, and llka Diester. Mechanisms of premotor-mote cortex interactions during goal directed behavior. bio pp. 2023 01, 2023.
12

---

Published in Transactions on Machine Learning Research (09/2024)
Saurabh Arora and Prashant Doshi. A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence, 297:103500, 2021.
Zoe Ashwood, Aditi Jha, and Jonathan W Pillow. Dynamic inverse reinforcement learning for characterizing animal behavior. Advances in Neural Information Processing Systems, 35:29663 -26666, 2022a.
Zoo C Ashwood, Nicholas A Roy, Iris R Stone, International Brain Laboratory, Anne E Urai, Anne K Churchland, Alexandre Ponget, and Jonathan W Pillow. Micc alternate between discrete strategies during perceptual ual decision-making. Nature Neuroscience, 25(2):201 212, 2022b.
Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael L Littman. Apprenticeship learning about multiple intentions. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 897 904, 2011.
Loonard E Baum and Ted Petric. Statistical inference for probabilistic functions of finite state Markov chains. The Annals of Mathemalicul Statistics, 37(6))1554 1563, 1966.
Celia C Beron, Shay Q Neufeld, Scott W Linderman, and Bernardo L Sabatini. Mice exhibit stochastic and efficient action switching during probabilistic decision making. Proceedinggs of the National Academy of Sciences, 119(15):e211336111 2022.
Ariyan Bighashdel, Panagiotis Meletis, Pavol Jancura, and Gijs Dubbelman. Deep adaptive multi-intention inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases. Research Track: European Conference, ECML PKDD 2021, Bilboo, Spain, Seplember 13 17, 2021, Proceedings, Springer, 2021.
Part I 21, pp.
Alex J Chan and Mihaela van der Schaar. Scalable Bayesian inverse reinforcement learning. arXiv preprint WVXiv2212.06483. 2021.
Jiayu Chen, Tian Lan, and Vancet Aggarwal. Option-aware adversarial inverse reinforcement learning for robotic control. In 2023 IEEE International Conference on Robotics and Aulomation (ICRA), pp. 5902 5908. IEEE, 2023.
Jaedeug Choi and Kee-Eung Kim. Nonparametric Bayesian inverse reinforcement learning for multiple reward functions. Advances in Neural Information Processing Systems, 25, 2012.
Antonio Coronato, Muddasar Nacem, Giuseppe De Pietro, and Giovanni Paragliola. Reinforcement learning for intelligent healthcare applications: A survey. Artificial Intelligence in Medicine, 109:101964, 2020.
Brice De La Crompe, Megan Schnock, Florian Steenbergen, Artur Schncider, and Ilka Diester. FreiBox: A versatile open-source behavioral setup for investigating the neuronal correlates of behavioral flexibility via 1-photon imaging in freely moving mice. Eneuro, 10(4), 2023.
Christos Dimitrakakis and Constantin A Rothkopf. Bayesian multitask inverse reinforcement learning. In Re- cent Advances in Reinforcement Learning: 9th European Workshop, EWRL 2011, Athens, Greece, Septem- ber 9-11, 2011, Revised Selected Papers 9, pp. 273 284. Springer, 2012.
Martin Ester, Mlas-PPtee Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In KDD, volume 96, pp.
1996.
Emily Beth Fox. Bayesian Nonpurumetric Learning of Complex Dynasnical Phenomena. PhD thesis, Mas- sachusetts Institute of Technology, 2009.
Kosuke Hamaguchi, Hiromi Takahashi-Aoki, and Dai Watanabe. Prospective and retrospective values inte- grated in frontal cortex drive predictive choice. Proceedings of the National Academy of Sciences, 119(48): e2206067119, 2022.
Ryoma Hattori, Bethanny Danskin, Zeljana Babic, Nicole Mlynaryk, and Takaki Komiyama. Area-specificity and plasticity of bistory-dependent value coding during learning. Cell, 177(())1858-18)) 1872, 2019.
13

---

Published in Transactions on Machine Learning Research (09/2024)
Gabriel Kalweit, Maria Huegle, Moritz Werling, and Joschka Boedecker. Deep inverse Q-learning with constraints. Advances in Neural Information Processing Systems, 03:114291144388 2020.
Tadeusz Wladyslaw Kononowicz, Virginie van Wassenhove, and Valérie Doyère. Rodents monitor their error in self-generated duration on a single trial basis. Proceedings of the National Academy of Sciences, 119(9): e2108850119, 2022.
Sateesh Kumar, Jonathan Zamora, Nicklas Hansen, Rishabh Jangir, and Xiaolong Wang. Graph inverse reinforcement learning from diverse videos. In Conference on Robot Learning, pp- 55-66. PMLR, 2023.
Minhae Kwon, Saurabh Daplardar, Paul R Schrater, and Xaq Pitkow. Inverse rational control with partially observable contimuous nonlinear dynamics. Aduances in Neural Information Processing Systems, 33:7898- 7909, 2020.
Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with Gaussian processes. Advances in Neural Information Processing Systems, 24, 2011.
Amarildo Likmeta, Alberto Maria Metelli, Giorgia Ramponi, Andrea Tirinzoni, Matteo Giuliani, and Mar- cello Restelli. Dealing with multiple experts and non-stationarity in inverse reinforcement learning: An application to real-life problems. Machine Learning, 110:2541 2576, 2021.
Bernard Michini and Jonathan P How. Bayesian nonparametric inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Dalabases: European Conference, ECML PKDD 2012, Bristol, UK, September 24-28, 2012. Proceedings, Part II 23, pp. 148-163. Springer, 2012.
Payam Nasernejad, Tarek Sayed, and Rushdi Alsalch. Multiagent modcling of pedestrian-vchicle conflicts using adversarial inverse reinforcement learning. Transporlmelrica A: Transport Science, 19(3):206108) 1081, 2023.
Radford M Neal. Markov chain sampling methods for Dirichlet process mixture models. Journal of Compu- tational and Graphical Statistics, ((2):249-266) 2000.
Andrew Y Ng, Stuart Russell, et al. Algorithms for inverse reinforcement learning. In International Confer- ence on Machine Learning (ICML), volume 1, pp. 2, 2000.
Quoc Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. Inverse reinforcement learning with
locally consistent reward functions. Adoances in Neural Information Processing Systems, 28, 2015.
Yael Niv. Reinforcement learning in the brain. Journal of Mathematical Psychology, 53(3):139-154 154, 2009.
Decpak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In IJCAI, volume 7, pp. 2586 2591, 2007.
Matthew Rosenberg, Tony Zhang, Pictro Perona, and Markus Meister. Mice in a labyrinth show rapid learning, sudden insight, and efficient exploration. Elife, 10:e66175, 2021.
Amit Surana and Kunal Srivastava. Bayesian nonparametric inverse reinforcement learning for switched Markov decision processes. In 2014 13th International Conference on Machine Learning and Applications, pp. 47 54. IEEE, 2014.
Robert C Wilson and Anne GE Collins. Ten simple rules for the computational modeling of behavioral data. Elife, 8se45547, 2019.
Shoichiro Yamaguchi, Honda Naoki, Muneki Ikeda, Yuki Tsukada, Shunji Nakano, Iuue Mori, and Shin Ishii Identification of animal behavioral strategies by inverse reinforcement learning. PLoS Compulational Biology, 1ee1eee10 2018.
Brian Ziebart, Andrew Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In AAAT, volume pp. 1433-1438. Chicago, II, USA, 2008.
Brian D Ziebart, J Andrew Bagnell, and Anind K Dey. Modeling interaction via the principle of maximum causal entropy. 2010.
14

---

Published in Transactions on Machine Learning Research (09/2024)
A Theoretical and technical details
A.1 Proof of Theorem 4.3
Proof. The objective function J J@ [)) from problem G can be writlen as:
J (et (6)
[lo P (5.n [ fnDD,
E CDD
P(n E, e) (5, 1,0t)
]
[ T (su, ao)
E E~D
Σ P(n g. log
A+
(St. ae)
Σ P(n 0)
P(nle, Σ log At 21
E 8~D
+ E {WD
"
+ E S~DD
Σ P(n E, () Σ
log क (0t,
K
K
K
E $~D
(2)
E
P &, () log A
gnD
7
P(s)=il(.9)
K
+ E E~D
P (,0)4(z))og log (sr, ac)
i=1 1=
P(==))).))
E
P P = =i -i log 11
(A.1a)
E~D
K
K
+
E
P = i, = -il. 1) log A
(A.1b)
i=1
1=1
P(.. = 1, प.. (st.
E
(A.1c)
CDD
where 1; is the indicator function with 4(x) = for all == and 0 otherwise. Thus maximizing J (@+ + over θ (problem (6) is equivalent to separately maximizing (A.1a) over IIt, (A.1b) over A+, and (A.cc
over R+ = (ri...... } The first two optimization problems can be formally written as:
maximize (over II=)
E Σ P(zo = | (0
subject
I1+ > 1 =1
and
maximize (over A+) subject to
Σ Σ Eh P( =1 = (), log A A 0,
E
A 1, 1=1...,,K K.
Note that AAlc has similar structure as the optimization objective of IRL (problem ) and can be further separated for each R, thus maximizing A.1c) over R+ is equivalent to independently solving IRL.
15

---

Published in Transactions on Machine Learning Research (09/2024)
problem for each Pt, with the tth demonstration weighted by P( i g)
maximize (over r7) EqWD E P( = 1 () nH (st, subject to
" (s, exp (Q a) - log E exp Q(+ )), for all 8 a A ((),0) = (s, + Eves P(s, a, 8') max Q(s', a'), for all E S, a E
9
Similar to
11 the constraints on policy ア, are introduced here to make the IRL problem tractable.
A.2
Computing required posterior probabilities
To obtain the posterior probability terms P(zo i {), P(z-1 -, E, and P(=. f required for evaluating the objective functions of (8) and (9. we apply the Baum-Welch algorithm (Baum
& Petrie 1966) as follows.
Let 0g R be the forward probability which denotes the posterior probability of observing the expert demonstrations up until time and the tth demonstration was generated under reward function Ta i.e.,
(i) = P( 2t = i =) 1, (80. an)
(A.2)
t t= 0 t > 0
(A.3)
at Ar (st. a,)
for all 1,... K. Similarly, we define the backward probability 3 E R denoting the posterior probability of observing the expert demonstrations after time t, given that the tth demonstration was generated under reward function rz:
(1) = P6 2 =,
(A.4)
j=m (((Ti)))TTTr
n
(A.5)
t == n,
for all i 1 K. With A.3 and A.5 we can efficiently obtain the forward and backward probabilities for each demonstration in a recursive form. Finally, we can compute the posterior probabilities
(1) af 3e
P( (()=)
(A.6)
and
(i) (8141 (4+1)3+4155
(18-1=i,2 = j|E,0)
(A.7)
Σ P= Ex=1 ar (t)) Try (8411, ((41)8442(1)
16

---

Published in Transactions on Machine Learning Research (09/2024)
B
Algorithms
Algorithm Inverse action-valus iteration.
given expert demonstrations D.
1: initialize v, Q.
2: Obtain expert policy nB by counting the state-action visitation D.
3: repeat
4: for all S do
5: Calculate p(s; a) = log(t"(ss,)) a))
=VEs P(s,a, a, s¹) max Q(6', d') for all c A.
6: Update -) solving
least squares.
7: Update (( a(s,a) Evees Pls. ma Q(s',a') for all
8: end for
9:
until stopping criterion is satisfied.
A Algorithm Inverse Q-learning.
given expert demonstrations D; learning rate ar, ag, and agh.
1: initialize r, Q, Qhh, and state-action visitation counter
2: repeat
3: for all E do
4: for = O..... do
5: Ohtain (Ft, ((s,, a) E Exees p(ss. for all a
Observe state-action pairs (st, s(41) from &, and update p(st, p(st, +1.
6:
7:
Update (s),)) (1 agh QQ (sr.at) ast 7maxx Q(s(+1,a).
8:
Calculate a() (st, (s,(s),)) for all a A.
Update a() = (1 a.)r(s at) + a- a()), + card(d_n,) Esed (r(s), bl).
9:
Update Q(s., end for
aq (s) ag(r(s) + maxaed ((+++1,)).
end
until stopping criterion is satisfied.
Algorithm 3 Hierarchical inverse Q-learning.
given expert demonstrations D and reward set cardinality K.
1:
initialize II, A, Ti....,KK- repeat
2:
3: Calculating the Boltzmann policy कe; for all i = I,.... K.
4: for all & E D do
5: for i = I,... j =1, 1,..., K, .,..., do
6:
Calculate (il)) and (1) according to (A.3 and A.5
7:
Calculate P(a: = O) and P
(.6)
according to
A.6 and
A.7
8:
end for end for
9:
10:
Update and A according to 10 and
for 1,...,K do
initialize expert demonstration subeet D' for all & do
for O...., do
Observe state-action pairs (st, (ec,0e,84111 from E sample ~ 24(0, 1).
u < P(2, = &, 9) then
D' D' U [(t, [++1].
end if end for 20: end for
19:
21: Update by applying Algorithm to D' if P known, else using Algorithm
22:
end for
23:
until stopping criterion is satisfied.
17

---

Published in Transactions on Machine Learning Research (09/2024)
C Datasets and model training
C.1
Gridworld benchmark
The following pscudo-code shows the expert policy for each episode that we used to obtain expert demon-
strations the gridworld environment $5.1
initialize s = (0, 0), == 78oal O.
repeat
4: 8~ al),
5: if has barrier '#' then
6:
Switch to another policy (30%).
7:
else t = 8 then π abaandon (50%).
end if
10:
t= + 1.
11:
until (0,0) (4,4) is reached.
The whole expert demonstration dataset consisted of 1024 trajectories. All evaluated algorithms were fit to multiple sub-datasets with different number of expert trajectorics, and each dataset was analyzed using a 5-fold cross-validation. Note that since the EM process does not guarantee to find the global optimum, for each training, HIAVI was randomly initialized repeatedly for 10 times and the best performed initialization (evaluated by test set log-likelihood) was selected for analysis. The initial intention distribution IT initialized uniformly, and the intention transition matrix A was initialized as: A 0.95 I + (O, 0.05 I), where N' denotes the normal distribution and I E R2x2 is the identily matrix. This initial A was then normalized so that each row added up to 1. The discount factor of the MDP was set to be = 0.9.
C.2 weal-world mice navigation benchmark
For comparability with the results in Ashwood et al. 2022a)), we obtained their pre-processed mouse trajecto- rics for water-restricted and water-unrestricted animals from /tttps://giiaii cwm/97aditi/ htt///ddiiddddmmmiccccc irl The original recorded animal trajectories from Rosenberg el al. (001)) are provided with MIT open source license at the following repository: httpps: /tt///tttuuuu...//// httpg//tt.ooccoomaa.rr.aaaaa www.aaasseeeeeeeeeeeeeeeeeeaeeee -20g 2002-002------ 1021-Repositoryy For the pre-processe pre-processing, Ashwood al. (2022a) used a clustering algorithm (based on DBSCAN (Ester al, 1996)) for aligning trajectories across animals and bouts to reduce variability. After the pre- pre-processing, they obtained 200 trajectories from the water-resh restereeestritted animals and 207 trajectories from the water-unrestricted animals. 20% of trajectories from each cohort were held out as a test set.
We used the source code and the best performed set of hyperparameters provided by Ashwood et al. 2022aa) to train DIRL on the animal trajectories. All HIAVI algorithms were trained for 10 repeated runs with different initializations, and the results from the initializations with the highest test set log-likclihood was selected for analysis. The initial intention distribution II was initialized uniformly, and the intention transition matrix A was initialized as: A 0.95 x + [O00, 0.05 I), where N denotes the normal distribution and I R K is the identity matrix. This initial A was then normalized so that each row added up to 1. The MDP was defined to have a deterministic state transition function P and the discount factor was set to be 0.99.
C.3 Application to mice reversal-learning behavior
The expert demonstrations for this dynamic reversal-learning task was collected from cohort of mice consisted of 9 mice in total. Behavior recordings for each subject were repeated for at least 7 independent. sessions with an average ca. 87 trials per session.
We employed a multi-stage fitting procedure select hyperparamoters and to allow us to fit HIQL indi- vidually to each animal. In the first stage, concatenated the data from all animals in single dataset
18

---

Published in Transactions on Machine Learning Research (09/2024)
together. We then performed multiple IQL (single intention HIQL) with different history length la 1, 5 on the concatenated dataset. Out of the 5 different values, We chose the la that resulted in the best test set log- liglliied for subsequent stages. In the second stage, we run multiple HIQL with different num- ber of intentions K = 2.....5 again to the concatenated dataset to obtain a global fit. For each fitting, we performed 10 different initializations and the best performed initialization was selected. The initial in- tention distribution II was initialized uniformly, and the intention transition matrix A was initialized as: A 0.95 I + (0(0,, 0.05 I), where N denotes the normal distribution and I R is the identity matrix. This initial A was then normalized that each row added up to 1. In the last stage of the fitting procedure, we wanted to obtain an independent but aligned HIQL. fit for each animal, so we initialized the parameters for each animal with the best global fit parameters from all animals together, omitting the ne- cessity to permute the retrieved intentions from each animal so as to map semantically similar intentions lo one another. 5-fold cross-validation was used to split the training and test dataset. The discount factor fot this experiment were sot be 7 0.99.
Code availability
Implementation for the class of HIQL algorithms can be found httpp://sttabb.cccmm/aaaoo1/11/1
19


Reviews:
[{"review_id": "hQ1ZVAGY0A", "reviewer": "Reviewer_NGQK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The ideas are well-motivated and novel. Modeling animals' intentions across time with a Markov process is intuitive.\n* Overall the paper is well written. The story and theoretical justification are clear.\n* The experiments clearly show that the proposed method is better than the baseline DIRL.\n\nWeaknesses\n* Notations in Sec 4 and Figures should be more consistent:\n    * The predicted sequence of reward functions are represented by z1,...,zn in the text but r1,...,rn in Fig 1\n    * What is P in Fig 1?\n    * Why is delta^(K-1) but R^K\n* Some details of the experiments are missing: \n    * In Fig 3c, how are the two state value functions(goal and abandon) calculated in the 1 intention/1 map case?\n    * In the Gridworld benchmark, how many trajectories are used to evaluate the policies?\n    * What's the runtime of this algorithm?\n    * It would be clearer if P('tired') in Fig 6d and 5d share the same color.\n* It would be helpful if authors discuss why HIQL is better than DIRL even in the 1 intention case.\n* Please clarify what's 'win-stay' and 'lose-switch' in sec 5.3.\n* It is a little unsatisfying that in three of the experiments, there are only 2 3 intentions for the agents.\n* An in-depth analysis of why HIQL is so much better than DIRL would help readers intuitively understand the advantages of this method. \n* Are the prediction errors for DIRL mainly happening during intention transitioning? Or is DIRL having a hard time predicting the correct intention overall?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method to infer discrete time-varying rewards with IRL called HIQL. Unlike previous IRL methods that only aim to infer a single reward function, HIQL is motivated by the observation that animals' intentions of tasks over time often change due to changes in their physical conditions. The authors mainly compare HIQL with one previous method, DIRL, which models the reward function as a smoothly time-varying linear combination of reward maps. In the experiments, they first demonstrate HIQL's superior performances in a simulated gridworld environment and a real mice behavior dataset. They analyze HIQL's performances in terms of predicting the expert's behavior and recovering the expert's intention transition dynamics. In the end, they show some evidence of recovering exploration behavior on a mice reversal-learning task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses for recommended changes.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SMrN7E1VIe", "reviewer": "Reviewer_CkFD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n1. The HIQL algorithm itself, which makes clever use of the recently developed IAVI algorithm to allow for efficient inference of latent, non-stationary reward processes.\n2. The extensive experimental analysis using both synthetic data and data from real behavioral science experiments reflecting the primary use-case for this algorithm.\n\n**Weaknesses:**\n\n1. The technical presentation of HIQL could be significantly improved.\n2. It is not immediately clear whether the improved log-likelihood of the reward processes learned with HIQL over DIRL are due to a better fit with underlying decision making process that controls the behavior of the mice, or is simply the result of HIQL having more parameters than DIRL allowing for a tighter fit.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents *hierarchical inverse Q-learning* (HIQL), a novel algorithm for inverse reinforcement learning that assumes the reward function is non-stationary, and varies according to a discrete-time Markov process.  HIQL extends the IAVI algorithm (which estimates stationary reward functions from expert trajectories), to the setting where reward functions change at discrete points in time.  The primary motivation for the work appears to be the analysis of the behavior of human and animal subjects, rather than imitation learning.  In addition to experiments with synthetic data, they demonstrate the effectiveness of HIQL in behavioral analysis by applying it to two datasets collected from real-world experiments with mice completing navigation tasks.  Qualitative and quantitative results suggest that the reward models learned with HIQL better fit the observed behavior, and in the synthetic case, better capture the underlying dynamics of the reward function.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. **Critical** - The paper really describes two algorithms, HIAVI and HIQL, for the cases where the transition dynamics are known and unknown respectively.  The distinction between these algorithms, and their connection to the corresponding algorithms of Kalweit et al. (2020) should be made explicit in sections 3 and 4.\n2. More detailed pseudocode for HIQL and HIAVI would helpful in understanding the algorithms, as Algorithm 1 is somewhat informal in its description of each step. (These could be included in the appendices.)\n3. As HIQL is a form of expectation maximization, it could be made more explicit that the reward function sequences $\\xi$ are the latent parameters that are to be marginalized over.\n4. It was not immediately clear in sections 5.2 and 5.3 whether the mouse data sets were collected specifically for this study, or were used in previous studies.  While this ambiguity may have been needed to maintain anonymity, any previous published uses of the data should be cited.\n5. While the log-likelihood comparisons in figures 5a and 6a suggest that the Markov chain model is a better fit the the mouse dataset, it is unclear whether this simply reflects the larger number of parameters in the models learned by HIQL/HIAVI.  Comparing the BIC values for these two model space would be useful.  Section 5.3 already reports BIC values for HIQL with different numbers of parameters, so this should be straightforward.\n6. As IAVI is a relatively new algorithm, a more detailed description of the method in Section 3 would be helpful.", "broader_impact_concerns": "I identified no future ethical concerns arising from this work that would need to be addressed in the text.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TDnjB6ENfk", "reviewer": "Reviewer_7Eh8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well written and fairly easy to follow. The notation is not overbearing and the algorithms are clearly explained.\n- HIQL is well placed into related work and the difference to previous methods is clear.\n- The empirical results are analysed extensively and all design decisions are motivated well across all three experiments.\n\n\nWeaknesses:\n- The language around \"demonstrations\", \"trajectories\", \"reward functions/maps\", and \"intentions\" could be clearer in the text. Figure 1 clarifies that a \"demonstration\" is one state transition within an expert trajectory. However, it was just slightly confusing from the text in Section 4, if a demonstration was a full trajectory or one transition. For example \"Inverse Q-learning. Given the set of expert demonstrations $\\mathcal{D}$ in an MDP, where each trajectory $\\xi \\in \\mathcal{D}$ is denoted with a sequence of state-action pairs: $\\xi=\\{(s_1, a_1), \\ldots,(s_n, a_n)\\}$, the IRL problem consists in determining a reward function $r$ that best explains the observed expert behavior in the form of demonstrated trajectories.\". One can infer that a demonstration is a state transition from the definition of $\\eta=\\{z_1, \\ldots, z_n\\}$ but the paper could be improved by just quickly defining that a demonstration is a state transition. \n- Similarly, the almost interchangeable use of \"reward\", \"intention\", and \"reward map\" makes the paper slightly harder to understand, as \"intention\" is not clearly defined. For example, why did the authors not reuse the word \"reward map\" as used in DIRL? Should \"intention\" always be used when reward functions change discretely and \"map\" when the reward changes continuously?\n- Figure 3c (1 intention) could be explained better. “Since the single-intention variants assume all trajectories were demonstrated under one intention, the EVD was analyzed twice on the ground truth reward for different intentions with the same learnt Boltzmann policy“. Does that mean that the authors ran HIAVI with 1 Intention multiple times, and hand-selected the resulting policies most akin to 'goal' and 'abandon'? Or how is it possible that you run HIAVI \"for different intentions\", when the intentions are learned unsupervised? Is HIAVI more likely to learn 'goal' or 'abandon' if the intention=1?\n- While the authors provide information about the mice dataset in the appendix, the reviewers preference is that specific information, such as the number of expert trajectories would be present in the main body of the paper. For example, when the authors report standard error, it is important that it's clearly stated if the standard error is reported over how many trajectories (and potentially over different seeds for initial state and state transition distribution?)\n- The experiments do not appear particularly complex, compared to other IRL problem settings as mentioned in the Introduction (self-driving, robotics, etc.). While the reviewer does not deem it necessary, obviously the paper could be made much stronger by demonstrating HIQL on datasets with 10s or 100s of intentions with much larger state spaces.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new class of algorithms, Hierarchical Inverse Q-learning (HIQL), including a model-based (HIAVI) and a model-free (HIQL) algorithm. The general goal is to accurately determine the underlying reward function of a set of expert demonstrations. The authors suggest that reward functions (intentions) during one trajectory change according to a Markov Chain. HIQL approximates the initial state distribution and state transition matrix of the Markov Chain using Expectation-Maximisation on the expert trajectories. It then selects the most likely policy under the state and transition distribution using Inverse Q-Learning. The authors provide theoretical results (a derivation of the objective function) and empirical results (evaluating HIAVI on a toy example and real-world mice data, and HIQL on a real-world mice bandit problem).\n\nHIQL is different from previous methods as it assumes that intentions change discretely within a trajectory, in contrast to previous methods, which typically assumed continuous transitions between intentions. The authors show that HIQL performs distinctively different and better than previous methods, such as Dynamic Inverse Reinforcement Learning (DIRL).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, it is the reviewers opinion that the paper is sufficient in quality and further improvements will not affect the decision substantially. If the authors could address the questions in the Weaknesses section, the clarity of the paper could be further improved. \n\nAdditionally, more complex experiments with many intentions and high-dimensional state spaces would obviously improve the potential impact of the paper drastically. However, the results are sufficient to support the claims in the paper and provide interesting insights for the TMLR community.", "broader_impact_concerns": "The reviewer does not see any ethical concerns with the method proposed. The authors could include an ethics section in the appendix, describing the potential ethical concerns of inferring intentions of human behaviour.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Taking all information available, I think the paper is novel and original, makes concrete scientific claims that are well supported by evidence, and is ready and interesting to be shared with the TMLR audience. All reviewers agree with this assessment and recommend acceptance as is. No issues were brought up in the reviewer-AE discussion (since there was nothing left to discuss after the rebuttal).

This is a clear case with all information available already in the reviews and authors' responses. I do not have anything to add and clearly recommend accepting the paper.
Decision: Accept (poster)