Contextual Policies Enable Efficient and Interpretable Inverse Reinforcement Learning for Populations
Authors: Ville Tanskanen
Chang Rajani
Perttu Hämäläinen
Christian Guckelsberger
Arto Klami
Venue: Accepted by TMLR ()

Abstract: Inverse reinforcement learning (IRL) methods learn a reward function from expert demonstrations such as human behavior, offering a practical solution for crafting reward functions for complex environments. However, IRL is computationally expensive when applied to large populations of demonstrators, as existing IRL algorithms require solving a separate reinforcement learning (RL) problem for each individual. We propose a new IRL approach that relies on contextual RL, where an optimal policy is learned for multiple contexts.
We first learn a contextual policy that provides the RL solution directly for a parametric family of reward functions, and then re-use it for IRL on each individual within the population. We motivate our method within the scenario of AI-driven playtesting of videogames, and focus on an interpretable family of reward functions. We evaluate the method on a navigation task and the battle arena game Derk, where it successfully recovers distinct player reward preferences from a simulated population and provides substantial time savings compared to a solid baseline of adversarial IRL.

Published in Transactions on Machine Learning Research (06/2024)
Contextual Policies Enable Efficient and Interpretable Inverse Reinforcement Learning for Populations
Ville Tanskanen
Deportment of Computer Science, University of Helsinki, Finland
Chang Rajani
Department of Computer Science, University of Helsinki, Finland
Perttu Hämäläinen
pertuu.aamalainend@aatttoo
Aolto University, Finland
Christian Guckelsberger
chrristinnguccklsseereee..r
Department of Computer Science, Aalto University, Finland
School of Electrical Engineering and Computer Science, Queen Mary University of London, UK
Arto Klami
arto Klami@hhllsinii
Department of Computer Science, University of Helsinki, Finland
Reviewed on OpenReview: https: // openreview. net/
net/forum? forum? id= 4CURCG6ITe
Abstract
Inverse reinforcement learning (IRL) methods learn a reward function from expert demon- strations such as human behavior, offering practical solution for crafting reward functions for complex environments. However, TRL computationally expensive when applied to large populations of demonstrators, as existing IRL algorithms require solving a separate reinforcement learning (RL) problem for each individual. We propose new IRL approach that relies on contextual RL, where an optimal policy is learned for multiple contexts. We first learn contextual policy that provides the RL solution directly for parametric family of reward functions, and then re-use it for IRL on each individual within the population. We motivate our method within the scenario of Al-driven playtesting of videogames, and focus on an interpretable family of reward functions. We evaluate the method on navigation task and the battle arena game Derk, where successfully recovers distinct player reward preferences from a simulated population and provides substantial time savings compared to solid baseline of adversarial IRL.
1 Introduction
Modelling human behavior is one the grand goals of AI, with versatile applications from Computational Cognitive Science to Human-Computer Interaction (Oulasvirta et al., 2022). User modelling video games constitutes an example application the latter domain and serves as the use-case for our contribution. Models of human behavior can implicitly inform game design choices, e.g. design more engaging and satisfying games. Moreover, they be explicitly employed for simulating player behavior, e,g. with the goal of reducing the need for costly, slow and tedious human playtesting sessions (Gudmundsson et al. 2018
Chang et al. 2019 Stahlke et al. 2020 Roohi et al. 2018).
Reinforcement learning (RL) provides one basis for building behavioral models. RL algorithms (utton interacting with an environment learn an optimal behavioral policy which maximizes a sum of future rewards. Whether not humans truly behave as reward- maximizing agents, RL has been shown to be useful framework (Silver al. 2021), with empirical evidence
& Barto 2018) leverage an agent's experience

---

Published in Transactions on Machine Learning Research (06/2024)
1) Contextual Reinforcement Learning
2) Fast Inverse Reinforcement Learning
User
User 2
User
User 4
Figure 1: Efficient population-level IRL in 2-D navigation task (see Section for details). Left: We first find contextual policy for family of reward functions. These weight the preference of four target areas (red dots) without using the expert trajectories. The colored lines indicate the optimal trajectories for a range of preferences, all starting from the center for clarity. Right: For a set of expert trajectories (here one per user), we infer the user's reward preference (size of the target area) using a gradient- based algorithm that does not require solving any RL problems. Instead, the algorithm only needs the pre-computed contextual RL solution. This enables largo-scale learning of users' preferences.
in behavioral modeling in many domains, including Al-driven playtesting (Bergdahl et al., 2020 Shin et al. 2020 Kristensen al., 2020 Roohi al., 2021 de Woillemont et al. 2021).
The key challenge in developing behavioral models with RL is in defining the reward function that matches individual's goals. In some cases such reward functions can be derived directly from cognitive theories (Holmgard et al., 2014a b), but a more general approach is to learn the reward function from observations of how the target individual interacts with the environment. The task of inferring the reward function from such expert demonstrutions is called inverse reinforcement lcarning (IRL) (Abbeel & Ng. 2004), a sub-field of imitation learning (IL) (Osa et al. 2018 Pelling Gardner 2019). This formulation allows learning models even for an individual given sufficient samples of their past behavior. Games serve as an excellent domain for studying and applying IRL, as they constitute worlds of arbitrary complexity and game companies often extensively log the gameplay of potentially millions of players.
Tn the current IRL literature, the rewards are often represented as flexible neural networks Finn et al. 2016b; Fu et al.) 2018) and the learning is often based on adversarial algorithms that contrast simulated and expert trajectories (Ho & Ermon, 2016 Fu et al. 2018 Wang et al. 2021). Despite notable progress in the core algorithms, all current solutions still scale poorly for large populations of demonstrators. In particular, vast majority of the mothods require repeatedly solving the optimal policy within the IRL algorithms. When applied to large populations, repeatedly solving RL problems within iterative algorithms for each user is clearly wasteful. Earlier works that propose IRL methods that avoid using nested RL solutions still retain high cost for populations since they require training user-specific models instead of using a shared model
(Klinn et al. 2013 Sharma et al. 2017 Garg et al. 2021 Pirotta & Restelli 2016).
Closest to our work, Ramponi et al. 2020 extended the gradient- based method Pirotta Restelli (2016) for populations by assuming that the demonstrators can be grouped to few (in their experiments 2-3) clusters where all demonstrators within cluster share the exact same reward. This improves efficiency when the clustering assumption holds, but does not help in learning individual rewards in general.
2

---

Published in Transactions on Machine Learning Research (06/2024)
We contribute method to speed up IRL for large populations based on the concept of contertual RL (cRL) (Hallak et al.) 2015). Standard RT algorithms learn policy for specific enviromment with fixed (but often stochastic) transition dynamics and reward functions, whereas cRL methods learn policies for contextual environments where the dynamics and rewards are context-dependent. For instance, Eimer et al. (2021) learnt a contextual policy for a PointMass environment with parameterized friction, enabling the agent to behave optimally with arbitrary future friction coefficients.
Our core idea is to interpret parameterization of the reward function context. This allows first learning cRL solution for all possible reward preferences in advance, without yet accessing any expert demonstrations, We can then solve the IRL. problem for individual users by leveraging the cRL solution. We thus separate the RL and IRL problem, overcoming the need to repeatedly execute an RL algorithm within the inner loop of IRL. This reduces the cost of the per-user computation at the expense a fixed precomputation of the cRL policy. To the best of our knowledge, no prior work has used cRI. for efficient population- IRI, even though cRLs have recently been considered in related tasks. De Woillemont al. (2021) employed cRL without explicit reference to the concept. for AT playtesting, creating alternative player personas manually with the contextual policy. Merel et al. (2017) and Waan et al. (2017), in turn, used contexts for improving imitation of the demonstrations, without paying attention to the reward functions underlying the bchavior. Finally, Moon et al. (2022 2023) used cRLs as part of a behavioral model, but they do not solve the problem with IRL but rather rely on general-purpose inference methods. Contextual RT. with reward as the context has been called multi-objecctiv RL (MORL) in some works (Castelletti et al. 2012 Parisi et al., 2016 Yang
et al. 2019). Many of these works focus on (using the terminology of cRL) learning the contextual policy and its theoretical aspects without considering the IRL task (Castelletti et al. 2012 Parisi al., 2016). Closest
to our work in this literature is Yang et al. (2019) where they introduce an efficient algorithm for learning contextual policies, and show how to recover hidden preferences (similar to IRL) with scalar rewards coming from simulator, only using the simulator and not demonstrations (differing from IRL).
Our approach is applicable to all behavioral modelling tasks, assuming a sufficiently low-dimensional pa- rameterization for the reward function. We construct the reward function by specifying a set of sub-reward functions that are hypothesised motivate the user's behavior, and consider cases where the sub-rewards are defined in advance interpretable basis of behavior. Game companies often analyze their players through pre-defined metrics that characterise e.g. how often they use specific features of the game to obtain insights about their behavior a population level. Such analysis, however, is necessarily limited to aggregate sum- maries and does not generalize for complex enviromments with no clear interpretation between rewards and game features or when reaching a particular reward requires a long sequence of actions. RL resolves these challenges as a natural basis of modelling sequential and context-dependent decisions, and the pre-existing metries can directly be used as sub-rewards, roquiring no additional developer effort. The full reward func- tion is parametrized as a weighted combination of these sub-rewards, which is naturally low-dimensional and interpretable. However, the new IRL algorithm itself is general and applicable also for other reward functions.
We evaluate the approach in two simulation studies. We first employ it in a custom-made navigation task (Figure to investigate the method's reliability in inferring the weights of a growing number of sub-rewards. We also compare our method to adversarial inverse reinforcement learning (AIRL) (Fu et al. 2018), showing that we require only 1% of the environment interactions for each user compared to AIRL, and hence show that dramatic speedups are possible for large populations even after accounting for the pre-training cost of cRL. Finally, we apply the method for distinguishing between players using either aggressive passive playing strategics in Derk (Norón 2020), vidco game designed for developing RL algorithms.
2 Background
2.1 Reinforcement Learning
Reinforcement learning algorithms are formulated using Markov Decision processes (MDPs) Sutton & Barto 2018). A MDP describes an environment and task using a tuple (S, P(s'|s, a).r), where is the state space, A is the action space, P(s'|s, a) is a distribution called transition dynamics that describes the evolution
3

---

Published in Transactions on Machine Learning Research (06/2024)
of the system from any state E S, to future state s' E S, with an action E A. The reward function S x A x S + R describes the utility of transition (s,
The goal is to find a policy т S + + A that maximizes the expected sum of (discounted) future rewards, also called expected return J. For stochastic environments and policies parameterized by 0 this corresponds to maximizing
(()) pepasp((ss- age)to(ee)
Σ 2'r(ss ar $411)
(1)
where y the discount factor. We use finite and hence consider episodic RL tasks, An episode a collection of state-action-reward tuples starting from an initial state so and terminating either at time limit or a terminal state 8y-
The RL problem can be solved with numerous algorithms. For instance, when the transition dynamics are unknown we often employ model-free algorithums (e.g. Schulman et al. (2017)) and if the dynamics are known then model-based algorithms can be used (c.g. Chua et al. (2018)). Our work is agnostic w.r.t. the specific RL algorithm since our IRI. approach merely requires running a suitable RT. solver as an intermediate step, and hence the RL algorithm can be decided based on the properties of the environment.
2.2 Contextual Reinforcement Learning
Contextual reinforcement learning is about learning policics in contextual MDPs (cMDPs) (Hallak et al. 2015). It is used commonly in environments where the agent naturally faces multiple contexts, for instance different weather conditions in autonomous driving, or in difficult environments that can be parametrized in way that curriculum learning methods can be employed to reach better final performances. Learning cRL policies is harder, as the environment becomes more complex due to the additional dependency on a context, but is proven to be doable (Jiang et al. 2017 Modi et al. 2018 Belogolovsky et al. 2021
Formally, a contextual MDP is a tuple (C,S,A,MMc)) where S and A are state and action spaces like before, but a space of conterts and M(c) is function from context space a MDP the transition dynamics Pa(s' (, and the reward function re depend on the context
a), where
We use special case of the general cMDP where only the reward function Te is affected by the context and the transition dynamics are not; this means the context only encodes information about the reward function (and hence a given user, e.g. a player) rather than being a property of the environment (e.g. the game). This differs from many previous uses of cRL where the environment itself is contextual.
2.3 Inverse Reinforcement Learning
While the goal of RL is to find an optimal policy given the reward function, inverse reinforcement learning (IRL) refers to learning the reward function from expert demonstrations. We assume that every expert has a single reward Fw and the observed demonstrations correspond to a policy maximizing it. The reward function Tw assumed to be parametrized by vector w, and the goal is learn w such that a policy maximizing the corresponding return would be similar to the policy that generated the demonstrations. We assume access (( and a
to a set of N trajectories (r) {(s) ((t)) ..... 2 are the state and action of the ith trajectory at timestep i, each trajectory being possibly of different length T, The goal is to maximize, w.r.t. the weights w, the likelihood of N independent expert demonstrations
1) where t; is the ith trajectory and
N
exp (T.(
G P(n. .(TT....TT/w)
ww/ = II
(2)
Z
where 2= exp EEi Tw (w Tw (, (141)) dr is the normalizing constant integrating over every possible tra- jectory sequence {so, a1, $1, ST... We also write (w)(T) = En Tw (w a, st-1) ) for compactness. The form of the likelihood comes from the Boltzmann rationality assumption w.r.t. the demonstrations (Ziebart et al., 2008 Finn et al. 2016b); we assume the experts choose actions so that the ones yielding high returns are favored over others by an exponential factor.
4

---

Published in Transactions on Machine Learning Research (06/2024)
Algorithm 1: Standard IRL
Algorithm 2: cRL + IRL
Initialize randomly;
for j 0 to R. do
// pretrain CRL
for = to do for 0 to R do
// IRL. loop
2
w Dir(a))
// RL loop
To(:, w) + Improve Io(s w) w.r.t. 0 using equation and Tw
4
To + Improve No w.r.t. & using equation
and Tw
end
end
5 Initialize w randomly;
1 f- no 7)
8[(w) 8w
for i = 0 to I do // IRL with pretrained cRL w <- 1f [2(w)
end
end
Since the integral in 2 is intractable, the objective of equation 2 is typically replaced by a surrogate loss
((w) (r)) (T.)) (r) (T)),
(3)
whose gradient is equivalent to that of - log G, and where where sg(.) is the stop-gradienl function. Here denotes the expert policy and (T|Ww) denotes the policy that is optimal w.r.t. the current parameters The procedure can be expressed as standard IRL algorithm, presented in Algorithm [ where RL policy is optimized for each IRL step to estimate the first expectation in equation 3 [cf. Algorithm [ lines 3-5). Both IRL and RL are solved with iterative algorithms with I and R denoting the respective number of iterations. The exact updates depend on the algorithm.
In Section we will compare our method against one commonly used IRL method, which relies on adversarial training. In adversarial IRI, two models are trained: policy my like before and discriminator D, whose purpose is to classify trajectories (or state-action-next-ssatee triplets) into two classes: those generated by the expert and those generated by the current policy. In adversarial inverse reinforcement learning (AIRL) Fu et al. 2018 the discriminator takes the specific form of
exp(fu.s(s, (s, D(s, (') exp(Sv.o(s a, 8')) + (a(a))
(4)
where (v.o(s, a,,')) = (v a) - Despite the adversarial formulation, the algorithm is closely
related to the objective equation 3 (for details, (Finn et al., 2016a) and (Fu et al. 2018)) and follows approximately Algorithm [ The differences are that policy is trained to maximize R(s,a, a, 8) log(1
D(s, a, s')) log(Dss, a, s")",, that is, to fool the discriminator, while learning reward function is interchanged with discriminator training.
3
Method
3.1 Problem Setup
Our task to solve the IRL problem for a population of users u E (1....,U)) U,, based on expert trajectories (r of each user. That is, we need to solve the optimization problem of equation U times. many scenarios U can be very large and for each user we may have a fairly large number of expert trajectories N,, Hence, we aim to solve this problem in computationally efficient manner. Furthermore, where the reward functions are (at least to some extent)) interpretable. In our video game use case, this would provide developers insights on the player population to aid both game design and analytics.
seek solution
3.2 Approach
3.2.1
Overview
We propose an abstract general pipeline, where we combine cRL and IRL to overcome the computational burden of IRL. We first train a single contextual policy that is (sufficiently) optimal for any reward function characterized by the weights This allows the inner loop of Algorithm 1 to be moved to pre-training
5

---

Published in Transactions on Machine Learning Research (06/2024)
phase, and the IRL loss in equation 3 to be calculated using the pre-trained contextual policy. Note that user trajectories are not used in the cRT, part, but only revealed to the IRL, part of the approach. Combining IRL with cRL can in principle speed up all IRL methods that require optimal policies for multiple reward fanctions. We provide technical details for one instance of the principle to demonstrate how it works, but note that other choices for the cRL and IRL training could be made. The procedure is outlined in Algorithm and described in detail below.
3.2.2 Interpretable Reward Parameterization
We focus on linearly weighted sub-reward functions
so that the reward function becomes
K
(8, a, 8' = Σ k=1
war as,
The sub-reward functions are assumed be predefined by domain experts, and the IRL part only needs learn the weights w that offer interpretability by providing information about user preferences among the sub-reward functions. This parameterization is similar to the early TRL works (Abbeel & Ng 2004 Ziebart
jet al. 2008), ; but we additionally assume the weights w lo be probability vectors, i.e. Σ Wy = =1, 80 that
the individual weights can be interpreted as proportional importance of the sub-rewards.
Even though we run all experiments with this reward family motivated by the sub-rewards are pre-defined by the company as aspects they want to use as basis for understanding the individual user behavior, we note that the method itself is compatible with more general reward functions as long as we can solve the corresponding cRL problem. We discuss the fcasibility of this in more detail in Section i
video game
where
3.2.3 Contextual Policy Learning
A contextual policy depends not only on the states but also on the reward weights w that define the contexts here, making the policy т S x W + A a function of stale and reward weights as explained in Section 2.2. Importantly, in our formulation the weights w only influence the (user-dependent) reward function and not the rest of the environment.
As a specific choice for our pipeline, we choose to solve the cRL problem with the algorithm by de Woillemont et al. (2021) that iteratively improves the policy re(s, w) for randomly selected contexts w. For each episode we sample new weights w and then improve the policy T w) for this set of weights (lines 2-5 in Alg, 2) Note that other sampling strategies could be employed for training cRL as well, for example ones similar to Fimer et al. (202)) that try to sample contexts that are sufficiently difficult for the current policy.
We use neural networks (of two hidden layers in our experiments) for defining the contextual policy TØ(s, w).. For updating the policy, we can use any iterative RL algorithm and we use the Proximal Policy Optimization (PPO) (Schulman et al. 2017) that has been widely adopted by the RL community. Even though w is kept fixed during an episode, the policies for other weights still change due to updating the shared policy parameters 0. This process is continued until the policy no(s, w) converges for all w. In practice, we stop when the episode returns converge.
The weights w need to be probability vectors and are sampled from a Dirichlet distribution w ~ Dir(a). In principle, any posilive a would be applicable, but we found that a - 0 works well practice I and provides an easy choice with no tunable hyperparameters. That is, we only select a single sub-reward at a time, but the cRL algorithm still learns to interpolate between these archetypal cases due to the continuous nature of neural networks that we use as policies.
Figure (left) shows an example of the interpolation capabilities in a 2-D navigation task. During train- ing, only trajectorics aiming for a single goal were rewarded, but the solution still efficiently interpolates between the different goals. Note that this choice is made only for training the cRL, and the approach makes
In early experimentation found that allowing non one-hot vectors (a 0) led to the policy getting stuck local optimum where it only prefers one or few goals and others are ignored.

---

Published in Transactions on Machine Learning Research (06/2024)
assumptions on the distribution of for the demonstrations. In all of the experiments the observed demonstrations correspond to w with multiple non-zero weights.
3.2.4 Gradient-based IRL
The converged contextual policy me can be used provide optimal behavior w.r.t. any weights w (that define reward function Tw) by using the policy with fixed weights. This allows directly computing the IRL loss given by equation 3 in Line 7 of Algorithm 2 without requiring optimization of the policy.
The log-likelihood of equation is maximized with gradient updates, using the standard gradients that now take the form
w log G = (r) V (wTw(T)) (r-x(r) - [W)
(6)
N
The derivation is provided in the Appendix A In practice we impose the mizing the unconstrained weights v E R and transforming them by a softmax-operator w; = E
w by opti- exples)
شド= exp(1s)
The softmax reparamcterization introduces non-identifiabilit (adding constant value for each entry of v does not influence w) which may in some cases introduce difficulties in optimization, but this formulation work well in our empirical experiments. Alternative techniques for accounting the constraint could be used as well.
The TRL problem is solved separately for each user, in the standard approach, but the critical difference is that now only need standard gradient descent, completely skipping the inner loop of R iterations of the RL solver in Algorithm 1 The computationnl complexity for U users and / IRL iterations (per user) hence changes from OUUI(L + R)) to O(Re + UIL), where L refers to the iterative computation that is required for the IRL loss, with substantial practical speedup for large U.
4 Experiments
We rum experiments two environments, an m-D navigation task and the video game Derk (Norén 2020 The first alfords straight-forward visualization and technical verification of the proposed approach, whereas the latter can demonstrate its applicability to more complex environments, use simulated expert trajectories so that we have ground truth for verifying the results, but the method is directly applicable to real user trajectories as well.
particular video games. We
4.1 Experiment 1: m-D Navigation
4.1.1 Purpose
We demonstrate that (a) the proposed approach is able to recover the true weights w from expert trajectorios, and (b) that this works for sufficiently large K, the number of sub-reward functions. We also show that our method learns the models for each individual in fraction of the environment interactions compared AIRL FFu et al. 2018), method shown be highly competitive in recent IRL comparisons (Wang et al. 2021
We use environment that allows for casily modifying the number of sub-reward functions and mcasuring the quality of the recovered weights. Since we are assuming K manually defined sub-rewards as a basis for interpretable reward functions, our aim is in the ballpark of approximately K and cases with K >>>>> are not considered relevant.
4.1.2 Environment and Task
We use an m-dimensional navigation (ask, where an agent must navigale lowards a goal with discrete steps, similar to the environment used by Eimer et al.] (2021). We extend this traditional environment by allowing multiple goals, each offering a distinct, proximity- sub-reward function. We assume that the users may have different personal preferences for these sub-rewards, and hence the preferences define an optimal goal state in the environment. Figure] already illustrated 2-D version of this environment and showed how
7

---

Published in Transactions on Machine Learning Research (06/2024)
1.0
Goal
1.75
0.8
0.8
Baseline
1.50
AIRL warmstart
1.25
weigh:
0.6
0.6
0.4
0.75
0.2
0.50
0.25
0,0
0.00
200
400
500
200
1000
1200
10
106
1000
10000 100000
IRL Reration
# goals
environment Interactions (for IRL)
Figure 2: Left: An example convergence plot for real weights [0.16, 0.02, 0.57, 0.25] (horizonal lines) and the corresponding recovered weights. The plot shows that, with modest number of sub-rewards, our method recovers good approximation of the behavior in a relatively small number of iterations. Middle: Symmetric KL divergences between the real weights and the recovered oncs, averaged over 20 randomly sampled real weight vectors w for varying numbers of sub-rewards. We can correctly identify the weights up to K = 4 sub-rewards, after which the performance slowly deteriorates (and becomes theoretically unidentifiable K 7). We also show naíve bascline prediction of & proving natural upper bound for the symmetric KL. Right: Sample complexity comparison of the proposed cRL IRL method, AIRL and AIRL with warmstart for the IRT, part of learning. The figure shows 0.25, 0.5 and 0.75-quantiles of symmetric KT. over the 20 weight vectors related to 3 goals. Using cRL + IRL lowers the sample complexity compared to AIRL by two orders magnitude. It also outperforms AIRL with warmstart by a similar magnitude.
the preferences influence the optimal policies. We note that, even though the environment may seem easy, it is relatively challenging from the RL perspective due to the high-dimensional action space and sparse state representation. We conduct experiments using m = 5 dimensions, which translates to 2m x 4 x 2 = 80 possible discrete actions to choose from. Hence, solving the RL problem alone is non-trivial.
The environment can formally described as (see Appendix B.4 for details):
States: s, = [xda), (9x), ...., x (9k)) w). where (a) and (se) are the m-dimensional vectors denoting the locations of the agent and the kth goal respectively, and w are the reward weights.
Actions: a = [ai, a2, as), a1 for direction (discrete, 2m outcomes), a2 for the stride (discrete, 4 outcomes) and ag does not affect the reward or the next state (nuisance).
Sub-rewards: (s), ae, S141) -1 20
The set of goals {øx [g were chosen so that they are maximally far apart from each other on the medimensional 0.5-radius sphere. This ensures distinct goals and identifiability of rewards, so that for an environment of m dimensions we can identify the behavior uniquely for a maximum of goals.
Goal locations:
Termination: 150 steps, enough for reaching any goal.
We trained a cRL model for each choice of K, the number of goals. We then sampled 20 weight vectors (w] (00 (denoting a user) for cach K f 8), and generated 128 trajectories with each of them. The weight vectors un (gen) used for generating the trajectories were sampled from Dir(1.0). These trajectories were then used as demonstrations for the IRL part investigate how IRL performance is affected by increased number of sub-reward functions.
For the AIRL comparison, we use K = 3 goals with the same 20 weight vectors and trajectories, and compare the sample complexity required for a similar accuracy in weight recovery. To acquire linearly weighted rewards with ATRL, we define (s) to be the linear combination of weights and sub-rewards, and keep h as flexible neural network (like Fu et al. 20018 D.1).
We also compare to an intuitive extension of AIRL that attempts leverage already trained models speed up the learning for
We not aware of specific methods proposed for this, but introduce
8

---

Published in Transactions on Machine Learning Research (06/2024)
here simple variant that initialises the policy and discriminator of AIRL to sensible checkpoint coming from the earlier ATRL run (specifically a checkpoint saved at 2000 environment iterations). This can be though of as observing part of the player population beforehand and using the solution learned for them as an initialization for future user-specific AIRL models. We call this method AIRL wormstort. Note, however, that the specific method is not to be interpreted as an optimal way of re-using previous computation, but is merely intended as an additional bascline.
4.1.3 Results
We inspect the results from three perspectives. Figure 2 (left) shows example convergence of the weights over the IRL iterations, showing how the algorithm converges already within approximately 100-200 iterations and correctly identifies the preferences for a case of four sub-rewards. The weights were smoothed in a 50- step sliding window for visual clarity. The convergence for other ground truths and numbers of sub-reward functions in K [ 7 is similar (see Appendix B for more examples).
Figure 2 (middle) quantifics the accuracy for varying numbers of sub-reward functions. We mcasure the goodness of recovered weights by calculating symmetric Kullback-Leibhe (sKL) divergence (a.k.a the Jef- freys divergence, Jeffreys (1998); Kullback & Leibler (1951)) DAKI. (ur(sen), ud) between a generating weight vector (gen) and the recovered weight vector w. The sKL between two probability vectors p and q of length K' is defined by DAKL = Ex p; log Pi 9i
+ -E q log 2t, For comparison, we also plot the expected sKL
pi" Fy--Dir(a)) (P 1 denoting the symmetric KT. for baseline predictions for weights, natural upper bound for the divergence. Symmetric KL measure was chosen as our weights carry a probability interpreta- tion, and the KT. divergence is naturally comparable between varying length vectors. Other common metries could have been used, and the results were similar for example for Euclidean distance between log- weights....
The results show that we can accurately estimate the true weight vectors up to performance starts to deteriorate because the problem becomes harder but we still clearly improve the bascline of mean prediction. For K 2 the problem is longer identifiable for the environment since multiple weight vectors can explain the same trajectories equally well (see Appendix [ and the performance indeed drops. In other words, we verify that we can solve the IRL problem when it is solvable, but also highlight that the designer neods to be carcful in determining the sub-rewards in a way that is appropriate for the scenario. For scenarios with non-identifiable sub-rewards (e.g. the same action always provides the same two sub-rewards) is impossible to recover unique weights.
4. For K > the
5 dimensional
However, even in those cases we can verify that the policies TT., ()) induced by the recovered weights are meaningful, by comparing the actual returns. The results presented in Appendix D confirm that the policies using the recovered weights achieve similar performance to the policies using the ground truth weights (gen). even for large K's.
Finally, Fig. 2 (right) compares the convergence of the proposed method against AIRL Fu et al 2018 a representative example of how IRL problems are solved the literature. Averaged over the 20 ground truth weight vectors, our method converges in approximately 100 environment interactions for each user, whereas AIRL requires approximately 100 times more interactions due to repeatedly solving RL problem within the algorithm (cf. Section 2 That is, we solve approximately hundred IRL problems at the same time it takes to run AIRL once. The AIRL warmstart method that attempts to leverage previous computation in an alternative way leads to better initial performance, but takes just as long as standard AIRL to converge. For fair comparison we need to account for the pre-training cost needed for cRL, We used million interactions for that, but in practice the method converged after 500K interactions (sce Appendix B.4]. This is approximately the same cost as solving 50 IRL problems with AIRL. For U < 50 AIRL would hence be faster, but for U > 50 get substantial speedup.
9

---

Published in Transactions on Machine Learning Research (06/2024)
12
population1
population2
10
Recovered
0.8
Roal Population
0.4
2-
0.0
1.0
1.0
Weight(damaaee)
Recovered
Figure 3: Left: Screenshot of the Derk environment Norén 2020 used for the experiments. The red agent is controlled, cyan is its team, and purple is the opposing tcam. Middle: Weights for the damage sub- reward "dmg for population of 45 simulated players in the Derk game. The two populations of players are easily recognizable from the recovered weights. Right: Cross-plot between recovered and real weights for the damage reward ddng (correlation coefficient 0.9).
4.2
Experiment 2: Population IRL for Derk
4.2.1 Purpose
To demonstrate the feasibility of the method for video game player modelling with large populations, we use the Multiplayer online battle arena (MOBA) style game Derk [Nornn 2020), developed for testing RL algorithms in a realistic environment, and apply the proposed method to discover clusters of player behaviors similar to Babes al. (2011). We design two sub-reward functions, corresponding to dealing damage and reliably infer the weights of (simulated) users based
healing, and show that of expert trajectories for each user. Note that the clustering assumption is here made only to simplify the data generation process and analysis; the model still learns individual rewards for each user. For this specific setup also hods that explicitly cluster the population could be used (Ramponi et. al. 2020), but they explicitly leverage the clustering assumption that our method does not make.
reasonable amount
4.2.2 Environment and Task
Derk [Norén 2020 involves two teamss each having one lower to protect and three agents capable of moving, healing and attacking other agents see Figure 3 (left) for screenshot of the game. We provide here a high- level description of the environment used in experiments, with the details Appendix B.5
We learn the policy for single agent, treating the other agents part of the environment. A 64-dimensional state vector describes the state of the environment, including e.g. the agents' positions, orientations and whether an attack is available. We sot a time limit of 150 steps and assume that all agents have the same abilitics: a melee attack that requires being near the enemy, and a healing ability that can be used for healing from a distance. Finally, we discretized the action space to be a discrete-valued vector with 5 dimensions for movement (turning and advancing), focus, attack and heal.
We use K = 2 intuitive sub-reward functions: 'dmg rewarding for dealing damage the enemies and Theal for healing a teammate. Learning the weights on these sub-reward functions would allow a game designer to e.g. identify whether there are players who tend to ignore safely of the teammates and focusing on personally killing the opponents. Notably, the healing action is only rewarded in case the player is sufficiently close to an injured teammate, and hence even users with high preference for the healing action need to get involved in combat. This makes identifying the preference for healing difficult.
The bots that were not the target of the optimization (Figure 3 left, cyan and purple) were controlled by a separate bot policy. During the cRL training, this was the same policy as the one being trained, copied the bots after every 20 update steps. This is form of self- play, that has been successfully used for example set of weights every episode.
by Silver al. (007)). The bot policy also sampled
10

---

Published in Transactions on Machine Learning Research (06/2024)
During the IRL training we used a different bot policy to ensure that the approach is not overfitting to the policy used for the cRT: simple heuristic where the bots walk toward the nearest enemy at every timestep and melee attack them if it is closer than a fixed threshold. The cRL converged in 11 hours and each user's IRL computation took approximately 11 minutes on single machine with P100 GPU.
4.2.3 Expert Trajectories
We generate U = 45 distinct weight vectors and simulate different playing style with each one of them. To do sO, we first learn a cRL solution that provides a policy T(s for all w and then use the policy to simulate the Irajectories.
More specifically, we sample N1 = 20 weights from ((oop)) ~ (eta(127.2, 31.8) and N2 = 25 weights from (pop2)) 99ta(9.6, 17.8) to represent two sub-populations of users (Figure 3 middle). These distributions were chosen by fixing the means to be distinet enough and variances that the distributions have some overlap, and then solving for the parameters. For each simulated user we generate 128 trajectories (rounds); used for solving the IRL problem.
4.2.4 Results
We evaluate the method by comparing the estimated weights for the damage reward the true ones used for generating the expert trajectories. The Pearson correlation between these two is p = 0.9 and the mean absolute error is 0.08, confirming that can estimate the true weights with sufficient accuracy. The results are illustrated in Figure 3 The middle sub-plot shows histograms of the generated and recovered weights, confirming that we can recover the sub-populations that are interpreted as players who focus on killing the opponents (population 1) and players that focus on keeping their own team alive (population 2). The right sub-plot shows a cross-plot of the individual weights.
5
Discussion
Our main contribution is the new approach and hence the experiments focus on validating its behavior. The Derk game even in the simplified format considered here is a complex environment that resembles well the typical intended use case and more complex than e.g. many single-player mobile games, but the main limitation of the experiments is the reliance simulated users. We focused on introducing the cRL++RRI approach and demonstrating its fcasibility; obtaining data from commercial game for user experiments is subject to more applicalion-orreeteed future work.
Despite the focus on large-scale population tasks, we only used 45 users in the Derk experiment and 20 users (for each K) in the navigation experiment to avoid an excess waste of energy - in both cases the cost scales linearly as a function of the users as the IRL problems are solved independently and rumning the experiments explicitly for larger populations would not provide any additional value. For the navigation environment, we showed that the proposed method is orders of magnitude better than the bascline method AIRL in terms of sample complexity after the cRL pre-training. We decided not to train the baseline method for Derk which is considerably slower due to rendering the screen after each step, but note that the per-user speedup would be dramatic. In this environment already a single standard RL solver takes approximately 2 hours and hence AIRL and other standard IRL methods that solve it repeatedly would take considerably longer, yet we get the solution for each user in 11 minutes. The exact speedup depends on the context, but should be of similar order for all environments and against all standard IRL algorithms that need to solve RL problems within IRL loop.
From the perspective of interpretability, it is worth noting that we cannot always learn unique solution. For the navigation task we specifically constructed the environment and the sub-rewards so that the problem is identifiable (see Appendix B.4] but for general environments and sub-rewards this cannot be ensured, and even degenerate solutions are possible. For example, if one of the sub-rewards is constant we can optimize oquation 3 by setting weight to one (sce Appendix Ap. Recently, [ao et al. (2021), Kim et al. (2021) and Metelli al. (2023) studied the identifiabilty of IRL in general, and while their analysis is done for discrete state-action spaces, it is likely to generalize for our setup: Additional assumptions on the environment
11

---

Published in Transactions on Machine Learning Research (06/2024)
(e.g. sub-rewards, transition dynamics, expert trajectories and/or discount factors) are required to attain proven identifiability. Our reward function family was motivated by personal discussion with game developers who already have concrete sub-rewards (e.g. different types of resources) for their games, and who want to investigate the players from the perspective of these sub-rewards. We hence assumed the rewards have been chosen with reasonable care, for instance avoiding degenerate scenarios, and that we have observed sufficiently many interactions to infer the reward preference. We leave rocommendations on how the sub-rewards should be designed if interested in uniquely defined solution as future work.
When interpretability of the users's reward functions is not important and the goal is to simulate users' behavior, then more flexible reward structure, such as neural network, may be preferable because of its stronger expressive power. Algorithm 2 generalizes directly for such reward functions and does not require any changes in implementation, but proper experimentation would naturally be needed to validate how it works in practice. Learning the contextual policy for non-linear rewards more challenging, but [Sodhani et al.. (2021) have shown that the cRL problem can be solved even in cases where the context is a 768 di8-dimensional encoding of natural language, and methods like curriculum learning (Klink et al. 2020 Eimer al.] 2021 could be useful for learning the cRL policy in these situations. In work, the context corresponds to the parameterization of the reward function, and hence these results suggests that we could use (reasonably-sized) neural networks as a family for the reward functions and could still learn the cRL. However, the distances d(w, v) between real and recovered parameters would no longer be informative as quality measures due to non-identifiabilityy of neural networks, and hence the generated behaviors would need to be compared instead. Devising a suitable metric for these kinds of trajectory comparisons is a core challenge in many imitation
learning problems (Ho & Ermon 2016 Fu et al. 2018 Wang et al. 2021 Ciosek 2022
6 Conclusion
We presented novel integration of contextual reinforcement learning and inverse reinforcement learning to overcome prohibitive computational complexity of IRL for large populations. This was achieved by replacing repeated and uscr-specific RL solvers within an IRL algorithm with general contextual RI. solution for a family of possible reward functions, trained only once without needing access to the user demonstrations. We showed this lowers the total sample complexity needed for solving the IRL. problem for large user populations by orders of magnitude. The approach is agnostic to the specific RL algorithms. In this work we introduced one practical instance and studied how the number of sub-reward functions affects its performance. We also empirically demonstrated that we can correctly identify simulated player profiles in realistic MOBA game environment.
Our work opens new possibilities for inverse reinforcement learning at a scale applicable in industry. We here considered an application case in game industry where companies want to offer personalized experiences for their players and understanding their behavior as individuals the first step for customizod games. However, any domain where designer might want to study and understand user behavior and/or optimize the experience of individual user specific user group can benefit from the introduced method.
Acknowledgments
This work was supported by Business Finland (project MINERAL) and the Academy of Finland (Flagship programme: Finnish Center for Artificial Intelligence, FCAI). The authors wish to thank the Finnish Com- puting Competence Infrastructure (FCCI) for supporting this project with computational and data storage resources.
References
Picter Abbeel and Androw Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1, 2004.
Monica Babes, Vukosi Marivate, Kaushik Subramanian, and Michael Littman. Apprenticeship learning about multiple intentions. In Proceedings of the 28th international conference on machine learning (ICML- 11), pp. 597-904, 2011.
12

---

Pierre Le Pelletier de Woillemont, Rémi Labory, and Vincent Corruble. Configurable agent with reward input: A play-style continuum generation. In 2021 IEEE Conference on Games (CoG), Copenhagen, doi: LLLL19//CG522122002000 Published in Transactions on Machine Learning Research (06/2024)
Stav Belogolovsky, Philip Korsunsky, Shic Mannor, Chen Tessler, and Tom Zahavy. Inverse reinforcement learning in contextual mdps. Machine Learning, 110(9):2295-2333 2021.
Joakim Bergdahl, Camilo Gordillo, Konrad Tollmar, and Linus Gisslén. Augmenting automated game testing with deep reinforcement learning. In 2020 IEEE Conference on Games (CoG), pp. 600 603, Osaka, Japan, 2020. IEEE. doi: 10. Ww..109/Coooal /CoG47366..... 01147356.2020.9922 2020.923155..
Haoyang Cao, Samuel Cohen, and Lukasz Szpruch. Identifiability in inverse reinforcement learn- ing. In M. Ranzato, A. Beygelzime, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 12362 12373. Curran Associates, Inc., 2021. URL htttps:/ www.eedddnnnnn..is........ wtt.a//p..ccppprrr..aaaap/////// Patp:/999bbpppp..ppppp
Andrea Castelletti, Francesca Pianosi, and Marcello Restelli. Tree-based fitted q-iteration for multi-objective markov decision problems. In The 2012 International Joint Conference Newral Networks (IJCNN) pp. 1 8, 2012. doi:
Kenneth Chang, Batu Aytemiz, and Adam M Smith. Reveal-more: Amplifying human effort in quality assurance testing using automated exploration. In 2019 IEEE Conference on Games (CoG), pp. 1-8, London, United Kingdom, 2019. TEEE. doi: 10. (0.1109/CTG (TG.2009.884809..
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in handful of trials using probabilistic dynamics models. In Advances in Newral Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://procceodngs.. http:///ngg..uuippp..cppppppppep pdf
Kamil Ciosek. Imitation learning by reinforcement learning. In International Conference on Learning Rep- resentations, 2022. URL B00/:fic//a555t57704404411177783
Denmark, August 17-20, 2021, pp. 1-8. IEEE, 2021.
URL https:
7/doi org/10. 1111/11199/22222 2021. 9619127
Theresa Eimer, André Bicdenkapp, Frank Hutter, and Marius Lindaner. Self-paced context evaluation for contextual reinforcement learning. In International Conference on Machine Learning, pp. 2958. PMLR, 2021.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. A connection between generative adversar- ial networks, inverse reinforcement learning, and energy-based modols. arXiv preprint 11216.1110860 611.08882.. 2016a.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference machine learning, pp. 49-58. PMLR, 2016b.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adverserial inverse reinforcement learning. In International Conference on Learning Representations, 2018.
Divyansh Garg, Shuvam Chakraborty, Chris Cundy, Jiaming Song, and Stefano Ermon. Iq-learn: Inverse sofl-q learning for imitation. Advances in Neural Information Processing Systems, 34:4028 4039, 2021.
Stefan Freyr Gudmundsson, Philipp Eisen, Erik Poromaa, Alex Nodel, Sami Purmonen, Bartlomiej Koza- kowski, Richard Meurling, and Lele Cao. Human-liks playtesting with deep learning. In 2018 IEEE Conference on Computational Intelligence and Games (CIG), pp. 1 8, Maastricht, 2018. IEEE. doi: CCt.//II........
Assaf Hallak, Dotan Di Castro, and Shie Mannor. Contextual markov decision processes, 2015. URL https://arre wwt.:/...g.///..eeee///o.osssese 15tp./102..0000000 02259
13

---

Published in Transactions on Machine Learning Research (06/2024)
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neurol information processing systems, 29, 2016.
Christoffer Holmgård, Antonios Liapis, Julian Togelius, and Georgios N Yannakakis. Evolving Personas for Player Decision Modeling, In Proc. Conference on Computational Intelligence and Games (CIG), pp. 1 8. IEEE, 2014a.
Christoffer Holmgård, Antonios Liapis, Julian Togelius, and Georgios Yannakakis. Generative Agents for Player Decision Modeling in Games, Proc. Int. Conference on the Foundations of Digital Games (FDG), pp. 1-8. ACM, 2014b.
Harold Jeffreys. The theory of probability. OUP Oxford, 1998.
Nan Jiang, Akshay Krishnamurthy, Alekh Agarwal, John Langford, and Robert Schapire. Contextual decision processes with low Bellman rank are PAC-learnable. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1704- 1713. PMLR, 06-11 11 Aug 2017. URL https://proceeddingg
mlr htt.e//ddinna......rree/////////
html
Kuno Kim, Shivam Garg, Kirankumar Shiragur, and Stefano Ermon. Reward identification in inverse rein- forcement learning. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Con- ference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 5496 5505. PMLR, 18 24 Jul 2021. URL https:: https:/pprocedings mlr. ppess/v139/ hasss//v3///im11... http//iimmcc...m..
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL https:
7/arxiv. orgrxiv..rrg/ee htrgg/bb//1112// 1ttp//411..99999 6980
Edouard Klein, Bilal Piot, Matthieu Geist, and Olivier Pictquin. cascaded supervised learning approach inverse reinforcement learning. In Joint European conference on machine learning and knowledge discovery in databases, pp. 1-16. Springer, 2013.
Pascal Klink, Hany Abdulsamad, Boris Belousov, and Jan Peters. Self-paced contextual reinforcement learning. In Proceedings of the Conference Robot Learning, volume 100 of Proceedings of Machine Learning Research, pp. 513-5 529. PMLR, 30 Oct-01 -0- Nov 2020. URL https://proceddings / v100/klink20a. html
Jeppe Theiss Kristensen, Arturo Valdivia, and Paolo Burelli. Estimating player completion rate in mobile puzzle games using reinforcement learning. In 2020 IEEE Conference on Games (CoG), pp. 636-639, Osaka, Japan, 2020. IEEE. doi: 241109//GG44336622200000
S. Kullback and R. A. Leibler. On Information and Sufficiency. The Annals of Mathematical Statistics, (1):79 86, 1951. doi: 10.1214/aoms/1177//
URL https::////at http::::/aoi.org10...o 121/ 12211////00m/177779999444
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL http::///ppareeeeeaaaaae/r/rrttt
Josh Merel, Yuval Tassa, Dhruva TB, Sriram Srinivasan, Jay Lemmon, Ziyu Wang, Greg Wayne, and Nico- las Heess. Learning human behaviors from motion capture by adversarial imitation. ar preprinl arXiv: 10ki::1077.020 02201, 2017.
Alberto Maria Metelli, Filippo Lazzati, and Marcello Restelli. Towards theoretical understanding of inverse reinforcement learning. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), Proceedings of the 40th International Conference on Machine Learn- ing, volume 202 of Proceedings of Machine Learning Research, pp. 11555-24591. PMLR, 23-29 Jul 2023. URL httpp::/prroceedings. mlr.
ppesss/2202/metell1223 httpppoeeeaaaal....rrrree/////e/
Aditya Modi, Nan Jiang, Satinder Singh, and Ambuj Tewari. Markov decision processes with continuous side information. In Algorithmic Learning Theory, pp. 597-618. PMLR, 2018.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Hee-Seung Moon, Seungwon Do, Wonjae Kim, Jiwon Sco, Minsuk Chang, and Byungjoo Lee. Speed- ing up inference with user simulators through policy modulation. In Proceedings of the 2022 CHT Conference on Human Factors in Computing Systems, CHI 22, New York, NY, USA, 2022. Associ- ation for Computing Machincry. ISBN 9781450391573. doi: 00.114//3411009255000223 URL https:
//doi.org/10 /0og//0..00 www./1141///1..... 3502003
Hee-Seung Moon, Antti Oulasvirta, and Byungjoo Lee. Amortized inference with user simulations. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI '23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9781450394215. doi: 10.1145/354456
3581439. URL htttps://ddi.org/1110 1145/3544548... 3581439
John Fredrik Wilhelm Norén. Derk gym environment. https:// http://ggmdee.kgmmeee com 2020.
Takayuki Osa, Joni Pajarinen, Gerhard Neumann, J Andrew Bagnell, Pieter Abbeel, Jan Peters, al. An algorithmic perspective imitation learning. Foundations and Trends& in Robotics, 7(1-2):1-179, 2018.
Antti Oulasvirta, Jussi PP Jokinen, and Andrew Howes. Computational rationality as theory of interaction. In CHI Conference Human Factors in Computing Systems, pp. 1 14, 2022.
Simone Parisi, Matteo Pirotta, and Marcello Restelli. Multi-objective reinforcement learning through con- tinuous pareto manifold approximation. Journal of Artificial Intelligence Research, 57:187-222 2016.
Chris Pelling and Henry Gardner. Two human-like imitation-learning bots with probabilistic behaviors. Tn Proceedings of the Conference on Games, pp. 1 7. IEEE, 2019.
Matteo Pirotta and Marcello Restelli. Inverse reinforcement learning through policy gradient minimization.
In Proceedings of the AAAT Conference on Artificial Intelligence, volume 30, 2016.
Giorgia Ramponi, Amarildo Likmeta, Alberto Maria Metelli, Andrea Tirinzoni, and Marcello Restelli. Truly batch model-free inverse reinforcement learning about multiple intentions. Silvia Chiappa and Roberto Calandra (eds.), Proceedings of the Twenly Third International Conference on Artificial Intelligence and Statistics, volume 108 of Proceedings of Machine Learning Reseorch, pp. 2359 2369. PMLR, 26 28 Aug 2020. URL
httpg//a.....ppaeaaa///a/aaaaaaa html
Shaghayegh Roohi, Jari Takatalo, Christian Guckelsberger, and Perttu Hämäläinen. Review of intrinsic motivation in simulation- based game testing. In Proceedings of the Conference Human Factors in Computing Systems, pp. 1 1-13, 2018
Shaghayegh Roohi, Christian Guckelsberger, Asko Relas, Henri Heiskanen, Jari Takatalo, and Perttu Hämäläinen. Predicting game difficulty and engagement using ai players. Proceedings of the ACM on Human- Camman-Computer Interaction, 5(CHI PLAY))1-17, 2021.
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High-dimensional con- tinuous control using generalized advantage estimation. arXiv preprint arXiv:1506. 150600248,, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv: 06347, 2017.
Mohit Sharma, Kris M Kitani, and Joachim Groeger. Inverse reinforcement learning with conditional choice probabilities. aix preprint XXX::1709007599 2017.
Yuchul Shin, Jaewon Kim, Kyohoon Jin, and Young Bin Kim. Playtesting in match 3 game using strategic plays via reinforcement. learning. IEEE Access, 8:51593-51600, 2020. doi: 10. 1..1109/ACCESS SOCCEESSS.0..22....2 2020.2980380.
David Silver, Julian Schrittwieser, Karen Simonyan, Ioannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354 359, 2017.
15

---

Published in Transactions on Machine Learning Research (06/2024)
David Silver, Satinder Singh, Doina Precup, and Richard S. Sutton. Reward is enough. Artificial Intellignnce, 299:103535, 2021. ISSN 0004-3702 doi: http:://bbbi.org//1000///.aaaai 2001://i.rrrmm222111333333. 103535. 8ciencedirect http////saaacccrtcaa/a////////00
Shagun Sodhani, Amy Zhang, and Joelle Pineau. Multi-task reinforcement learning with context-based representations. In International Conference on Machine Learning, pp. 9767 9779. PMLR, 2021.
Samantha Stahlke, Atiya Nova, and Pejman Mirza-Babaei. Artificial players in the design process: Developing an automated testing tool for game level and world design. In Proceedings of the Annual Symposium on Compuler-Humen Interaction in Play, pp. 267 280, 2020. doi: 10.1145/3410444.344422
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018
Pin Wang, Dapeng Liu, Jiayu Chen, Hanhan Li, and Ching Yao Chan. Decision making for autonomous driving via augmented adversarial inverse reinforcement learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 1036-1042. IEEE, 2021.
Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. Aduances in Newral Information Processing Systems, 30, 2017.
Runzhe Yang, Xingyuan Sun, and Karthik Narasimhan. A generalized algorithm for multi-objective rein- forcement learning and policy adaptation. Advances in neural information processing systems, 32, 2019.
Brian D Zicbart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, et al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433 1438. Chicago, IL, USA, 2008.
Appendix
A Gradient of the IRL Likelihood
The likelihood of the expert demonstrations for the IRL was given in equation 3 equation follows from
The scaled gradient in
N T
w N
log G Vw
V
N
rw (
(8) ,at - N log Z
t=
V w
Tro (T:) - loo 7
V un Vwi(((T)) V.. log Z
N
N
1=1
Tuo (T() - 7 Vn / exp (rmo (T)
Too (Ti) 2 exp (Tw Vwr (T))d
E (r) (V wT ww (T)) - ((TlW)/wTae
EE Tww (Ti)) (E (w)
(7)
where individual steps correspond to standard algebraic manipulation, or replacing empirical averages with expoctations. This gradient is used for updating the weights w of the sub-reward functions in gradient-basod optimization algorithuns by employing a surrogate loss function
((w) Er-egg(((ri wi) [n (T:)) Eysr [Tm(
(8)
If we assume that the experts behave according our parameterized policy with weights w*, we can replace #" (7) = x<<] (r]="), and it is easy to that value of the loss at this (desired) point L(w") 0.
16

---

Published in Transactions on Machine Learning Research (06/2024)
We remark that in specific pathological cases we can reach zero loss also with solutions that are not interesting from the perspective of IRL. In particular, if we have a sub-reward F7 such that rx 8') = c. is constant, and Wi = 1.0, then the loss becomes
T E
C(wr) ()
Σ nh ( at +1
(x)
rk( 0e, 0t, +1)
(r [Trc] Ey (+) [Trc]
(Eroeg(x((ww)) [Tr] Eywx [T+))
O,
if we assume equal length trajectorics. That is, we can optimize the objoctive by assigning all weight on that constant reward. This problem case easily avoided by not using constant sub-rewards, but exact characterization of conditions for which the correct IRL solution is the only global optimum of the objective remains open.
B Experimental Details
B.1 PPO hyperparameters
We used Proximal Policy Optimization (PPO) (Schulman et al. 2017) algorithm for updating the contextual policy. PPO works by rumning a policy in the environment, collecting N episodes of states, actions, rewards and action probabilities from Narenas parallel environments, and then updating the policy with the gathered episodes. Like all actor-critic algorithms, the method contains a policy и (actor) that is used to compute actions and a value estimator V (critic) that is used for evaluating rewards-to-go for a state. The loss function that PPO minimizes is given by
E min (r:(0) clip(r,()), clip0),, --, (6)1))))))) ciTE (vs))) v(s.)) c2 log (a)(s)))
E
(9)
where A, the advantage estimator, r() = xe(ee]s))
Teas (a.]s.) the ratio of action probabilities computed by
comparing action probabilitics given by network with current paramcters 0 with those computed with lold; the parameters that were used for generating the episodes ~ We compute the advantages using Generalized Advantage Estimator (GAE-)A ) method Schulman et al. (2015). where the advantage A, = 6g + (T1)66+1 + ...+ (7A)T-++1 8 is calculated using a value function estimator V. discount factor and GAE parameter À and where 6, = 7 + ( (st). Value function estimator (sx)) trained to estimate the rewards- to-go V(s)) by using a mean squared error loss, weighted by a constant e₁ (second term of the loss). The entropy term (third term) is maximized to encourage as much randomness possible. The effect of entropy is weighted by factor of 62.
We evaluate the loss in equation with mini-batches of size Nhatch- The parameters of the actor and critic networks are optimized with Adam optimizer (Kingma & Ba 2014) using learning rates Qsctor and Qcritic respectively. The optimization is run for Nepoch epochs for every N episodes collected. Optimization is halted when a total of Nootal timesteps is reached. The hyperparameter values that were used to train the cRL policy in our experiments are given Table 1
B.2 Inverse Reinforcement Learning
The IRL hyperparameters our method are provided in Table 2. The IRL part of Algorithm 2 is run for I itcrations, optimizing equation 3 with Adam (Kingma & Ba 2014) optimizer using OIRL as the lcarning
rale. The expectations in equation 3 are evaluated using batches of sizes No. and Nexpert respectively. To evaluate the first term of equation 3 we use No of most recent simulated trajectories, while for the second expectation we use N Nexpert randomly sampled expert trajectorics. We simulate Nai trajectorics per IRL iteration and update the weights w using a batch of expert and simulation trajectories for UIRL iterations.
Remember that we update the policy multiple times, when begin updating, first iteration has 0
17

---

Published in Transactions on Machine Learning Research (06/2024)
5-D Nav.
Derk
Total timesleps
(Ntotal)
2e6
5e5
Arenas
(N (arenna))
32
Episodes per update
(N)
3
Epohcs
(Nepoct)
35
20
Batch size
(Nbatch)
2048
4096
KL clip fraction
(c)
0.125
0.2
Actor loss coefficient
(ci)
0.5
0.5
Entropy coefficient
(22)
0.031
0.01
Actor learning rate
(actor)
0.005
0.003
Critic learning rate
(eritiic)
0.021
0.005
Discount factor
(7)
0.91
0.99
GAE-À
(A)
0.959
0.95
Table 1: Hyperparameters used for training the cRL
5-D Nav.
Derk
Iterations
(I)
1200
40
Learning rate
(AIRL)
0.01
0.1
Env. simulation
(Nsim)
1
32
Weight batch
(Nar)
8 (last)
32 (last)
Expert batch
(Nexpert)
32
32
Updates per iteration
(UIRL)
2
Table 2: Hyperparameters used for training the IRL
The bascline that compare against, adversarial inverse reinfrocement learning (AIRL) Fuu et al. 2018 is established adversarial method that learns by iterating between training a discriminator and training policy. Table@proviie
all of the hyperparameters used for training ATRL. We use PPO as the policy training algorithm, for which the hyperparameter names and descriptions are the same as for learning the cRL policy (Table D. We train the AIRL for NAIRL iterations, where we first simulate Naim environment trajectories with the current policy. We than sample a batch of No simulated trajectories, called the simulation batch, and Nexpert trajectorics for expert batch. Then, the discriminator updated using binary cross-entropy loss with simulation batch labelled with negative labels, and expert batch labelled with positive labels using AdamW optimizer (Loshchilov & Hutter 2019 with smoothing parameters 3 [0.5, 0.999] and learning rate of aD. After the discriminator update, the policy is updated for N Nepochs using batch size of Nbatch using the same procedure as described in Section B.1
For the m-D navigation experiment we found the recovered weights to oscillate around the correct values (see. Fig. 4 because of two things: a) We used off-policy samples to estimate the first term of equation 3 (see Table sii but No 8, mcaning one on-policy and off-policy samples from the last simulations are used) and b) because of the theoretical minimum of the IRI loss in equation 3 being 0, but the loss allowing negative values well. We clamped the IRL loss m-D navigation environment to mitigate (although not completely removing) the oscillation so that the clamped loss becomes L(w) max(0, ((w)). Clamping would have been possible also on Derk environment, but due to the parallel computation of the environment we only used on-policy samples for evaluation and opted out from clamping the IRI. loss.
B.3 Hyperparameter Selection
The values for the hyperparameters used in cRL (Table were initially randomlly searched in 2-d navigation task to get an initial guess of good values. Then for each experiment, they were manually tuned during preliminary experimentation. The values for IRL experiment (Table 2 were manually tuned by inspecting at the convergences of the losses and the weights with random seed. The values for AIRL (Table 3) were random searched using trajectories related to single weight vector.
18

---

Published in Transactions on Machine Learning Research (06/2024)
0.75
0.00
0.50
0.00
650
1.00
0.50
0.25
1.00
0.25
0.00
500
1000
Rorotoo
Figure 4: All 20 convergence plots for K goals for the IRL. part.
0.75
0.00
0.75
0.50
0.00
550
1.00
0.50
0.25
1.00
0.50
0.25
1000
1000
Roro
Figure 5: All 20 convergence plots for K goals for the IRL part.
19

---

Published in Transactions on Machine Learning Research (06/2024)
5-D Nav.
Iterations
(AIRL) (N)
10000
Episodes per update
Epohes
(Nooch))
Batch size
batch)
150
KT. clip iraction
0.2
Actor loss cocfficient
(c1
0.5
Entropy coefficient
(2)
0.01
Actor learning rate
(aacoor)
0.0003
Critic learning rate
Oteritic)
0.001
Disc. learning rate
(aD)
0.001
Discount, factor
0.99
GAE-À
A)
0.95
Env. simulation
N Naim)
8
Weight batch
Nw
40 (last)
Expert batch
NN Nexpert)
32
Disc. regularizer
(c3)
0.01
Table 3: Hyperparameters used for the AIRL baseline. The PPO hyperparanters match the ones Table
B.4
m-D Navigation
B.4.1
Goal locations
The of goals {9 in m-D navigation task were chosen so that they are maximally apart from each other on the m-dimensional 0.5-radius circle.
In particular, sampled 1000 sets K locations (a!"1) minimum pairwise distance. That is,
(91) ..... a (gk) 10000 and chose the set that had the largest
max min (|
122 i,j E (I....., K),
The reason for the sample-based approach is that placing equidistant points m-ball proves to be non-trivial problem in spaces larger than 2-dimensional.
B.4.2 Training cRL
Contextual RL policy trained in pre-training phase, and for fair comparison of the sample complexities of different methods, the pretraining needs to be accounted for. We compare the complexity in 5 dimensional navigation environment and show the training curves of cRL for K K (2,...,8) 8) goals in Figure
We see that the pretraining converges around 500K time steps. which is equivalent of approx. 50 AIRL computations, making our proposed approach faster for populations larger than this. The plot highlights K 3 goals, as the comparison was done on environment with three goals.
B.5 Derk environment
B.5.1 State
The state of the original Derk contained the following 64 features: Hitpoints, AbilityOReady, PriendStatueDistancc, PriondStatuekngle, Priend1Distance, Friend1Angle, Friend2Distance, Friend2Angle, EnemyStatuebistance, EnemyStatueAngle, EnemylDistance, EnemylAngle, Enemy2Distance, Enemy2Angle, Enemy3Distance, Enemy3Angle, HasPocus, FocusRelativeRotattion FocusPacingUs, FocusFocusingāack, FocusHittpoints, AbilitylReady, Ability2Ready, FocusDazed, FocusCrippled, HeightProntl, HeightFrout5,
HeightBack2, PositionleftRight, Positionl/pDww,, Stuck, UnusedSense31, HasTalons, HaeBloodClase,
HasCleavers, HasCripplere, HaeHeaalanGlaand, HasVampireGland, HasFrogLege, HasPistol, HasMagnum,
HasBlaster, HasParalyzingDart, HasIronBubblegun, HaaHiellsuBubbbeqqu,, HasShell, HasTronbone, FocusHaasaaonss,
20

---

Published in Transactions on Machine Learning Research (06/2024)
-50
Mean reward
-100
-150
n. n_goals
-200
250
0.00
0.25
0.50
0.75
1.00
1.25
1.50
1.75
2.00
Env steps
1e6
Figure 6: Episode returns for training contextual reinforcement learning. The returns are smoothed using 50 episode sliding mean.
FocusHteeBloodClsse, FocueHasCleavers, Focustia@Cripplere, FocusffesHealllg@@landd FocushaaeVempiieeGllnd, FocusilasProgLege, FocusHasPistol, FocusHlaMaaguu, FocusHasBlasterr, FocusHlssarallyzinggarr,, FocusttsstrooBBubbleqqu,, PecusttaaiellluuBubbblee FocusHasshelll FocustlasTrombone, UnusedExtraSense00, UmusedBxtraSense31.
The sub-reward functions w, that act as the context, as additional features that are provided for the policy. The final dimension of the input to the policy thus 66, the last two being reward weights for functions dmmg and hhal
B.5.2 Action
The action space that we considered was 5-dimensional discrete space, each action being {a,, 02 03, 04, a5,a55. a.). The definitions and options for each dimension are follows:
41 E {-0.8, 0.4, 0.0, 0.8), the length of the step that agent takes to front or back (positive for forward)
02 0.408,0.440000044,0000 the amount that the agent turns during step (positive for turning right)
43 E (0.1, 0.5, 0.7, 0.9), fraction that interpolates the movement towards the focus of the agent (1.0 ignores aj and and just moves towards the focus)
a4 (0,1,,,3), which skill to use. 0: none, 1: melee, misc(never used in our case), 3: heal.
0s 1, 2, 4, 7), where to focus. keep current focus, 1-3: own statue and team, 4-7: enemy statue and team
21

---

Published in Transactions on Machine Learning Research (06/2024)
B.5.3 Reward
The reward functions r "dmg
and heal were defined as following:
0.1 min d2, ds}. if focus enemy n melee n dy < 0.072 l 0.24
dmg
0.5 min d2, da). if focus enemy heal n 0.06 < 0.072 n ley 0.2
(10)
min(d1.dg d2, ds}. otherwise,
where first condition rewards for melee attacks infficted in sufficient proximity of an enemy and the latter penalizes for healing an enemy (without the penalty, the agent altered between healing the enemy and attacking it). The dy and Ps stand for distance and angle to the focused agent. The minimum distance to the enemies min(d1, d2, ] is used for slight encouragement for movement lowards the enemies to help early exploration. The reward for healing Thea is defined similarly, but the enemy is swapped to a teammate and there is no encouragement for reducing the distance to enemies:
1.0, if focus teammate heal n dy < 0.072 lpyl< 0.24
Theal=
5.0, if focus teammnate O melee n 0.06 < dy 0.072 n lpsl< 0.2
(11)
0.0, otherwisc.
The amount of reward were chosen so that the healing strategy and the damage reward result in similar total returns if applied consistently (one can attack more often than heal).
Since the sub-rewards relate directly to specific actions (attacking or healing) a user can take, one could here in principle attempt inferring the reward preferences also by direct inspection of the relative frequencies of these actions. That is, higher relative use of the healing action would be interpreted directly as preference for that reward. Such interpretation may, however, be severely misleading. The marginal distribution of actions is p(a|ur)= f p(a]s, w)p(s)w) (/)ds, where the distribution of states p(s[wr depends on the policy and hence the reward. Even preference of healing corresponds to high p(a = heal")s, w) for many states, the preference also heavily influences the states the user visits and consequently the overall frequency of healing action is not guaranteed to be high. For instance, healing-oriented user may try to stay further from the combat actions that strongly influences the frequency of states the attacking and healing actions are even possible.
C Barycentric formulation
To reason why the m-D navigation task identifiable only for + 1 coordinates, consider a point p R. that resides within the set of goals {g .9 9 each goal being in the same m-dimensional space, 9; R". The point is a linear combination of the goals and corresponding weights 0 < wy for each goal (usually called barycentric coordinates)
= wigl +... + WKGK,
(12)
which lends itself for linear problem where we take into account that weights sum up to 1:
(m+1)xx
1
W1
Ø11
91K
(13)
[
9m1
Jmk
Pm
which is not solvable uniquely if > m + 1.
An example of unidentifiable (non-unique) case is in Fig. 1 where a trajectory going to origin can be explained e.g. by (0.5, 0.0, 0.5, 0.0) and (0.25, 0.25, 0.25, 0.25).
22

---

Published in Transactions on Machine Learning Research (06/2024)
-55
-60
Normalized regret
0.95
Regret
-65
0.90
goals
goals
-75
8
0.80
100
200
300
400
500
100
200 300
400
500
IRL Iterations
IRL Iterations
Figure 7: Left: Raw regret, Bx-r(. (r wate (T)l as function of IRL iterations for various mum- ber of sub-rewards K. The curves are average regrets over the 20 expert trajectories used in Section 4.1 and over 100 trajectories generated from ((, ()) Right: Normalized regret (Er~a(-))) (oao
where the performance is normalized w.r.t the generating parameters, i.c. what is achievable with cRI. with the generating weights. Even though the weight recovery deteriorates for large number of goals because of non-identifiability of the solution (Figure 2 we still recover weights that lead to good policies.
D Regrets of Recovered Policies
In Section 1.1 we confirmed that we can recover the true w accurately. Here we provide an alternative perspective for validating the quality of the solution, by inspecting the regret curves for policies using the recovered weights. The main goal is to confirm that can also recover good policies with the recovered weigths, but additionally show that the IRL solution recovers useful v also in cases where the weights are non-identifiable due to the the number of goals being too high for the environment.
For a pair of generating and recovered weights (w((en), w)), we define regret as E (r (wteen) (T)] to measure the expected ground-truth return of policy using the recovered weights. Figure 7 (left) plots regrets for the same K considered in Figure 2 showing that for all choices the policies converge well. Figure 7 (right) as good policy trained with the ground truth reward, and < 1 means suboptimal behavior. The normalization is done using (Er-a(, mnieee [r)]/E,ax((,u (wten)) (anttee (7)]) -1 where we need the reciprocal bocause the rewards in our always negative.
confirms the policies are good, by plotting the regrets normalised that means that
23


Reviews:
[{"review_id": "y1yHN1DsTm", "reviewer": "Reviewer_mwio", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The idea of using cRL to exploit the similarities between reward functions and re-use implicitly the experience collected in the past to make the full IRL pipeline more sample efficient is of interest.\n- The experimental results are promising. In the first experiment, the sample efficiency compared with AIRL is visible. Moreover, the authors succeed in showing that the proposed approach is able to recover the original reward weights.\n\n**Weaknesses**\n- The first methodological concern is about the appropriateness of using cRL when the final task is IRL. While I accept the informal rationale that cRL is able to save samples thanks to the pre-trained policy, I am not sure if similar rewards (more formally, reward functions characterized by weight vectors that are close, representing the context) lead to similar optimal policies. Surely, they will lead to similar value functions, but, in IRL our goal is to discriminate whether a reward function allows learning the expert's policy (we do not really care about the value functions). Can the authors specify whether they have a more formal justification for the appropriateness of using cRL for IRL?\n\n- The experimental validation is quite limited for what concerns the comparison with state-of-the-art IRL approaches. If the ultimate goal of the paper is to make IRL more sample efficient, I believe that a comparison with IRL approaches that do not require solving the forward RL problem should be performed. For instance: \n\nRamponi, G., Likmeta, A., Metelli, A. M., Tirinzoni, A., & Restelli, M. (2020, June). Truly batch model-free inverse reinforcement learning about multiple intentions. In International conference on artificial intelligence and statistics (pp. 2359-2369). PMLR.\n\nKlein, E., Piot, B., Geist, M., & Pietquin, O. (2013). A cascaded supervised learning approach to inverse reinforcement learning. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2013, Prague, Czech Republic, September 23-27, 2013, Proceedings, Part I 13 (pp. 1-16). Springer Berlin Heidelberg.\n\n- It is not completely clear whether what is proposed in the paper is an *abstract general pipeline* or a *specific combination* of an IRL algorithm with a cRL algorithm. The authors, indeed, in Section 2.3 propose a specific maximum-likelihood formulation of the IRL problem (which is a choice of an IRL approach), and in Section 3.2.3 a specific algorithm for cRL (Woillemont et al. 2021). The authors should clarify in the presentation whether these choices are compulsory or can be replaced with other IRL and cRL algorithms (and with which modifications).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposed to combine Inverse Reinforcement Learning (IRL) with Contextual Reinforcement Learning (cRL). Starting from the idea that a large variety of IRL approaches is required to solve the RL problem for every candidate reward function, the authors propose to address this task with cRL, where the context represented by the reward weights recovered from IRL, determines the reward function. In such a way, the experience collected for learning the optimal policy with one reward function can be reused in the future. The author focuses on linearly parametrized reward function for promoting interpretability of the output of the IRL procedure. Experimental validation in an m-D navigation task and on the Derk game is provided.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to \"Strengths And Weaknesses\". Anyway my main concern is the first one about the appropriateness of the methodological approach.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "LG6gLYVxao", "reviewer": "Reviewer_1a9e", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ Good motivation. I like the application to play-testing.\n+ Nice idea to use cRL to speed up IRL. It's simple and seems to work well.\n+ Good comparison with AIRL.\n\nWeaknesses:\n- Some parts of the paper were unclear and hard to follow. \n- Only using 2 subrewards for the complex benchmark seems limiting. It would have been nice to see a more expressive reward. \n- Only 2 environments.\n- No discussion of actual regret in terms of policy performance, just a comparison of reward similarity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a highly scalable approach for performing inverse RL on a population of users. The idea is to use contextual RL to pre-train an optimal policy that works for any reward function and then use this to improve the scalability of IRL by removing the need to run RL iterations during reward inference. The approach is shown to be much more efficient than AIRL and to work well on both a navigation and battle video game environment.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "would having state and action spae change on context break everything?\n\nwhy not try selecting weighted combinations of multiple sub-rewards during training? It would be nice to see an example of this and how it compares with the current approach. It isn't clear to me that interpolation will always work if not explicitly trained on some examples of interpolation.\n\nWhen analyzing the complexity in 3.2.4, what is I?\n\nIn 4.1.2, It's not clear to me that the action space is really high-dimensional. Isn't it 2-d since actions are directions to move? I don't understand how you get 80 or what m is in this domain. what is the nuisance part of the action and why is it included?\n\nCritical: I'm confused about alpha in the Dirichlet model. Early in the paper it says alpha ->0, but then in 4.1.2 alpha = 1.0. Why the discrepancy?\n\n4.1.3 Typo \"pre-taining\" -> \"pre-training\"\n\nCritical: In Figure 2 it is unclear what the difference is between Recovered and Mean pred in the center plot.\n\nCritical: An RL performance curve on the true reward weights for the experiment results shown in Fig 2 is missing. For example, even though some demonstrations may have multiple weight vectors that describe then (especially for large K), it shouldn't matter in terms of policy performance. I would like to see some measure of regret wrt the user's true reward and the behavior obtained via the IRL+cRL approach proposed in this paper. I think a regret plot would show the full picture since sometimes you can get close to the true reward but still learn a bad policy and sometimes you don't have to learn the exact same reward to get the same policy.", "broader_impact_concerns": "NA", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TPcVVlKgPf", "reviewer": "Reviewer_HHoC", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper is relevant to some real-world IRL problems where demonstrations come from a population of individuals who may be optimizing different rewards that lie within a class of parameterized functions. In practice, when the population size is large, pretraining a contextual policy to convergence is worth the sample cost, since the policy can then be fixed and reused within IRL for each individual, without needing to train a policy from scratch each time.\n\nWeaknesses:\n1. In terms of the real-world use case of game development and understanding players' preference for different sub-rewards, it is not clear from this paper that RL and IRL are needed for this problem. One can simply look at the distribution of actions and game events.\n2. In the abstract and introduction, the authors make an implicit assumption that the individual demonstrations in a population correspond to different rewards. This doesn't have to be the case in general, so omitting a clear statement of assumption invites questions.\n3. The introduction states that \"humans naturally do not exactly behave as reward maximizing agents\". This is a strong claim without justification or reference.\n4. When training the contextual policy, the method randomly selects context $\\omega$ from a degenerate Dirichlet distribution. This choice is insufficiently motivated: is this the best thing to do if the population of demonstrations follows another distribution? How would the proposed method accommodate this scenario?\n5. The use of RL and IRL for the application settings of interest in this paper isn't fully justified. The paper motivates the setting of IRL on a population by stating the need for game developers to understand the game players' preferences for optimizing different sub-rewards. Without using IRL, one can still understand their preferences based on the distribution of game features and actions taken by players, e.g. in the Derk experiment, look at the distribution of damage and healing actions, which shows players' preference for the two reward components. That is easy to do, so it's not clear why and when one would need RL/IRL and the proposed method.\n6. Besides comparing to vanilla adversarial IRL (AIRL), the authors should compare to a modification of AIRL whereby the RL policy that is being trained for one individual is reused as the initialization of the policy for the next individual.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers inverse reinforcement learning (IRL) in the scenario in which there is a large population of expert demonstrations, and each individual demonstration is optimal for a reward function within a class of parameterized reward functions. The authors propose to pretrain a contextual policy on randomly sampled reward function parameters, and use the same contextual policy for gradient descent on an IRL loss for any individual. This contextual RL + IRL method (cRL+IRL) is more sample efficient than training a policy from scratch in the inner loop of IRL for each individual. The reward function is assumed to be a convex combination of component functions, where the weights are the parameters to be recovered by IRL. Experiments in an m-dimensional gridworld navigation task show that cRL+IRL recovers ground truth reward parameters on tasks up to 4 reward components, and is an order of magnitude more sample efficient than adversarial IRL. Experiments on a multi-agent team competition shows cRL+IRL can recover the reward parameters of two distinct populations of players.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The authors should address these points to be considered for acceptance.\n1. Support the claim that RL/IRL has benefits for the application setting of interest that can't be gained from simpler methods such as looking at distribution of events and features.\n2. Directly state the assumption that the individual demonstrations in a population correspond to different rewards.\n3. Support the claim about human behavior.\n4. Justify the choice of random sampling from a degenerate Dirichlet distribution.\n5. Compare to a modification of AIRL whereby the RL policy that is being trained for one individual is reused as the initialization of the policy for the next individual.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents a novel approach that combines Inverse Reinforcement Learning (IRL) with Contextual Reinforcement Learning (cRL) to enhance the efficiency and scalability of IRL. The authors propose pre-training a contextual policy that can adapt to various reward functions, thereby reusing past experience and improving sample efficiency. The method is tested on an m-dimensional navigation task and the Derk game, showing promising results in recovering reward weights and outperforming traditional adversarial IRL (AIRL) in terms of sample efficiency.

The innovative use of cRL in IRL is praised for its practical applicability, especially in real-world scenarios such as game development and user behavior analysis. The experiments demonstrate significant improvements in sample efficiency, supporting the method's effectiveness and versatility.

The reviewers have raised some concerns about the methodological justification of using cRL for IRL. The assumption that similar reward functions lead to similar optimal policies needs more formal support. The experimental comparisons with state-of-the-art IRL methods are considered insufficient, and the scope of the experiments is limited to two environments with not sufficiently expressive rewards. Some parts of the paper are unclear or poorly explained, and the assumptions about demonstrations and human behavior lack explicit statements and support.

The authors have properly addressed most of these issues in their rebuttals. Still, I have pointed out further issues about handling the constraint in the optimization problem and the identifiability problem.
The authors modified their paper to explain how they project the solution after each gradient update and briefly mention the limitations of their approach regarding the identifiability problem.
I would like the authors to discuss this problem more: according to the basis functions used for the reward, their optimization problem can have multiple optimal solutions. Searching for the weights that minimize the difference between the expert performance and the one of the optimal contextual policy can lead to optimal solutions (difference equal to zero) where the difference in the single reward components can be significant. The authors should highlight these problems and suggest how they could be addressed.

Furthermore, the use of contextual policies in which the context depends only on the reward function should actually be referred to as Multi-Objective Reinforcement Learning and the corresponding literature should be considered.
An example of papers that learn policy parameterized with respect to the reward function are:

Castelletti, A., Pianosi, F., & Restelli, M. (2012, June). Tree-based fitted Q-iteration for multi-objective Markov decision problems. In The 2012 International Joint Conference on Neural Networks (IJCNN) (pp. 1-8). IEEE.

Parisi, S., Pirotta, M., & Restelli, M. (2016). Multi-objective reinforcement learning through continuous Pareto manifold approximation. Journal of Artificial Intelligence Research, 57, 187-227.

Yang, R., Sun, X., & Narasimhan, K. (2019). A generalized algorithm for multi-objective reinforcement learning and policy adaptation. Advances in neural information processing systems, 32.
Decision: Accept (poster)