Stochastic Inverse Reinforcement Learning 
Authors: Ce Ju
Venue:  (2021)
Keywords: Inverse Reinforcement Learning
Stochastic Methods
MCEM

Abstract: The goal of the inverse reinforcement learning (IRL) problem is to recover the reward functions from expert demonstrations. However, the IRL problem like any ill-posed inverse problem suffers the congenital defect that the policy may be optimal for many reward functions, and expert demonstrations may be optimal for many policies. In this work, we generalize the IRL problem to a well-posed expectation optimization problem stochastic inverse reinforcement learning (SIRL) to recover the probability distribution over reward functions. We adopt the Monte Carlo expectation-maximization (MCEM) method to estimate the parameter of the probability distribution as the first solution to the SIRL problem. The solution is succinct, robust, and transferable for a learning task and can generate alternative solutions to the IRL problem. Through our formulation, it is possible to observe the intrinsic property for the IRL problem from a global viewpoint, and our approach achieves a considerable performance on the objectworld. 

Under review as a conference paper at ICLR 2021
STOCHASTIC INVERSE REINFORCEMENT LEARNING
Anonymous authors
Paper under double-blind review
ABSTRACT
The goal of the inverse reinforcement learning (IRL) problem is to recover the reward functions from expert demonstrations. However, the IRL. problem like any ill-posed inverse problem suffers the congenital defect that the policy may be optimal for many reward functions, and expert demonstrations may be optimal for many policies. In this work, we generalize the IRL problem to a well-posed expectation optimization problem stochastic inverse reinforcement learning (SIRL) to recover the probability distribution over reward functions. We adopt the Monte Carlo expectation maximization (MCEM) method to estimate the parameter of the probability distribution the first solution to the SIRL. problem. The solution is succinct, robust, and transferable for a learning task and can generate alternative solutions to the IRL problem. Through our formulation, it is possible observe the intrinsic property for the IRL problem from global viewpoint, and our approach achieves considerable performance on the objectworld.
INTRODUCTION
The IRL. problem addresses an inverse problem that a set of expert demonstrations determines a reward function over a Markov decision process (MDP) if the model dynamics are known Russell (1998); Ng et al. (2000). The recovered reward function provides a succinct, robust, and transferable definition of the learning task and completely determines the optimal policy. However, the IRL problem is ill-posed that the policy may be optimal for many reward functions and expert demonstrations may be optimal for many policies. For example, all policies are optimal for a constant reward function. In a real-world scenario, experts always act sub-optimally or inconsistently, which is another challenge.
To overcome these limitations, two classes of probabilistic approaches for the IRL, problem are proposed, i.e., Bayesian inverse reinforcement learning (BIRL. Ramachandran & Amir (2007) based on Bayesians' maximum a posteriori (MAP) estimation and maximum entropy IRL (MaxEnt) Ziebart et al. (2008); Ziebart (2010) based frequentists' maximum likelihood (MLE) estimation. BIRL solves for the distribution of reward functions without an assumption that experts behave optimally, and encode the external priori information in choice of prior distribution. However, BIRL also suffers from the practical limitation that a large number of algorithmic iterations is required for the procedure of Markov chain Monte Carlo (MCMC) in a sampling of posterior over reward functions. Advanced techniques, for example Kernel technique Michini & How (2012) and gradient method Choi & Kim (2011), are proposed to improve the efficiency and tractability of this situation.
MaxEnt employs the principle of maximum entropy to resolve the ambiguity in choosing demon- strations over policy. This class of methods, inheriting the merits from previous non-probabilisticc IRL approaches including Ng et al. (2000); Abbeel & Ng (2004); Ratliff et al. (2006); Abbeel et al. (2008); Syed & Schapire (2008); Ho al. (2016), imposes regular structures of reward functions in a combination of hand-selected features. Formally, the reward function is a linear or nonlinear combi- nation of the feature basis functions which consists of a set of real-valued functions (0x(s, a))==1 hand-selected by experts. The goal of this approach is to find the best-fitting weights of feature basis functions through the MLE approach. Wulfmeier et al. (2015) and Levine et al. (2011) 1) use deep neural networks and Gaussian processes to fit the parameters based on demonstrations respectively but still suffer from the problem that the true reward shaped by the changing environment dynamics.
Influenced by the work of Finn et al. (2016a;b), Fu et al. (2017) propose framework called adversarial IRL (AIRL) to recover robust reward functions in a changing dynamics based on adversarial learning and achieves superior results. Compared with AIRL, another adversarial method called generative

---

Under review as a conference paper at ICLR 2021
adversarial imitation learning (GAIL) Ho & Ermon (2016) seeks to directly recover the expert's policy rather than reward functions. Many follow-up methods enhance and extend GAIL for multipurpose in various application scenarios Li et al. (2017); Hausman et al. (2017); Wang et al. (2017). However, GAIL is in a lack of an explanation of expert's behavior and a portable representation for the knowledge transfer which are the merits of the class of the MaxEnt approach, because the MaxEnt approach is equipped with the "transferable" regular structures over reward functions.
In this paper, under the framework of the MaxEnt approach, propose generalized perspective of studying the IRL problem called stochastic inverse reinforcement learning (SIRL). It is formulated as an expectation optimization problem aiming to recover probability distribution over the reward function from expert demonstrations. The solution of SIRL is succinct and robust for the learning task in the meaning that it can generate more than one weight over feature basis functions which compose alternative solutions to the IRL. problem. Benefits of the class of the MaxEnt method, the solution to our generalized problem STRL is also transferable. Since of the intractable integration maxiiiiiionnni (MCEM) approach Wei
in our formulation, we employ the Monte Carlo Tanner (1990) to give the first solution to the SIRL problem a model-based environment. In general, the solutions to the IRL problem are not always best-fitting in the previous approaches because a highly nonlinear inverse problem with the limited information very likely to get trapped in secondary maximum in the recovery. Taking advantage of the Monte Carlo mechanism of global exhaustive search, our MCEM approach avoids the secondary maximum and theoretically convergent demonstrated by pieces of literature Caffo et al. (2005); Chan & Ledolter (1995). Our approach is also quickly convergent because of the preset simple geometric configuration over weight space in which we approximate it with a Gaussian Mixture Model (GMM). Hence, our approach works well in real-world scenario with a small and variability set of expert demonstrations.
In particular, the contributions of this paper are threefold:
1. We generalize the IRL problem to a well-posed expectation optimization problem SIRL.
2. We provide the first theoretically existing solution SIRL by the MCEM approach.
3. We show the effectiveness of our approach by comparing the performance of the proposed method to those of the previous algorithms on the objectworld.
PRELIMINARY
An MDP is a tuple M (S,A,T, where S is the set of states, A is the set of actions, and the transition function (a.k.a. model dynamics) T := (((++11 = 8 8t = 8, at = a) for 8, s E S and a E is the probability of being current state s, taking action a and yielding next state s'. Reward function R(s, a) is real-valued function and 7 E [0, 1) is the discount factor. A policy π S + A deterministic or stochastic, where the deterministic one is written as a = π (s), and the stochastic one is conditional distribution (a]s). Sequential decisions are recorded in series of episodes which consist of states 8, actions a, and rewards r. The goal of reinforcement learning aims to get optimal policy '² for maximizing the expected total reward, i.c.
Ne R(s,, 0r)
n == arg max
t=
Given an MDP without a reward function R, i.e., MD T). and m expert demonstrations S : !S!..... }. Each expert demonstration Ç* is sequential of state-action pairs. The goal of the IRL problem is to estimate the unknown reward function R(s, a) from expert demonstrations Ç and the reward functions compose complete MDP. The estimated complete MDP yields optimal policy that acts as closely as the expert demonstrations.
2.1 REGULAR STRUCTURE OF REWARD FUNCTIONS
In this section, we provide a formal definition of the regular (linear/nonlinear) structure of reward functions. The linear structure Ng et al. (2000); Ziebart et al. (2008); Syed & Schapire (2008); Ho
2

---

Under review as a conference paper at ICLR 2021
et al. (2016) is M linear combination of feature basis functions, written as,
R(s, Σ O4 (i(s,a).
where 0, 8 x - Rd are d-dimensional feature functions hand-selected by experts. The nonlinear structure Wulfmeier et al. (2015) is the form as follows.
R(s, a) = (ó .ar
where N is neural networks of hand-crafted reward feature basis functions {0i(s, a))
In the following section, we propose an optimization problem whose solution is a probability distribution over weights of basis functions {s, a==
2.2 PROBLEM STATEMENT
Given an MDP, : (S, 7,)) with known transition function T == P(s s/1st = 8,
a) for s, E S and a E A and a hand-crafted reward feature basis function { A stochastic regular structure the reward function assumes weights W of the reward feature functions (s). which are random variables with a reward conditional probability distribution D (W) conditional expert demonstrations (E. Parametrizing D D (WI(BB) with parameter θ, our aim is estimate the best-fitting parameter é* from the expert demonstrations Ç such that D DR (WIç_, θ*) more likely generates weights compose reward functions as the ones derived from expert demonstrations, which is called stochastic inverse reinforcement learning problem.
Suppose representative trajectory class C. satisfies that each trajectory element set O E C. is subset of expert demonstrations { with the cardinality at least e 15"), where is preset threshold and [5 is the number of expert demonstrations, written as,
C.¹ == folo c _ with 10 2 c. 1.
Integrate out unobserved weights W, and the SIRL problem is formulated to estimate parameter θ on an expectation optimization problem over the representative trajectory class as follows:
θ arg max Boece S (M(O,
where trajectory element set O assumes to be uniformly distributed for the sake of simplicity in this study, and fM is the conditional joint probability density function of trajectory element O and weights W for reward feature functions conditional on parameter θ
2.2.1
NOTE:
Problem is well-posed and typically not intractable analytically.
Trajectory element set O is usually known from the rough estimation of the statistics in expert demonstrations practice.
We introduce representative trajectory class overcome limitation the sub-optimality of expert demonstrations _ Ramachandran & Amir (2007), which is also called a lack of sampling representativeness in statistics Kruskal & Mosteller (1979), c.g., driver's demon- strations encode his own preferences in driving style but may not reflect the true rewards of an environment. With the technique of representative trajectory class, each subset of driver's demonstrations (i.e. trajectory element O) constitutes subproblem in Problem 1.
METHODOLOGY
In this section, we propose novel approach to estimate the best-fitting parameter O* in Problem 1, which is called the two-stage hierarchical method, variant of MCEM method.
3

---

Under review as a conference paper at ICLR 2021
3.1 TWO-STAGE HIERARCHICAL METHOD
The two-stage hierarchical method requires to write parameter @ in profile form e : (01, 22)
The conditional joint density [M(OO, W ()) in Equation can be written as the product of two conditional densities IM and M as follows:
fM(O, 1 O2) gM(O)W, hm(W)02))
(2)
Take the log of both sides Equation 2, and we have
log (O, WI 22) log 90 01) + log (w
(3)
We maximize the right side of Equation 3 over the profile parameter @ in the expectation- maximization (EM) update steps at the t-th iteration independently as follows,
+ = arg max E (loggM(OW e1) 6¹): 0,
(4)
0 arg max E 02
log (oggh>()11:)))))
(5)
3.1.1 INITIALIZATION
We randomly initialize profile parameter 0 (0f, O%) and sample a collection of No rewards weights (W9', WN ~ (WI,, 0%). The reward weights W compose reward R Ree in each learning task M = AS,A,T R 7) for i ,
3.1.2 FIRST STAGE
In the first stage, we aim to update parameter O in the intractable expectation of Equation 4. Specifically, take Monte Carlo method estimate model parameters 0 an empirical expectation at the t-th iteration,
El [log (O). 0*
Σ log a (0 O No
1
(6)
where reward weights at the 1-th iteration W are randomly drawn from the reward conditional prob- ability distribution DR (WIÇ,, @²) and compose a set of learning tasks MM == AS T, Ryyer 7)
with a trajectory element set Of uniformly drawn from representative trajectory class C. for i I.... No.
The parameter 0/+1 in Equation has N₂ coordinates written as 0(+1 == ((0)+)), ( For each learning task M the i-th coordinate (0)+1); ); is derived from maximization of posteriori,
(0)+))
(o)+)); == arg max log IM; (O .0).
which is a convex formulation maximized by a gradient ascent method.
In practice, move m steps uphill to the optimum in each learning task M₁ The update formula of m-step reward weights Wm is written as
T/
Wmee = W9
X V(or), log loggm; (O: (01):),
where the learning rate X; at the f-th iteration is preset. Hence, the parameter O++1 is represented as @4+1 = (Wme)) Mpm N.
4

---

Under review as a conference paper at ICLR 2021
3.1.3 SECOND STAGE
In the second stage, we aim to update parameter θ₂ in the intractable expectation of Equation Specifically, we consider the empirical expectation at the (-th iteration as follows,
N2 Σ (loogh = No 1=1
log h my (Wm 10
where hM is implicit but fitting of m--tep reward weights (Wm x J==l in generative model yields a large empirical expectation value. The reward conditional probability distribution ( O is generative model formulated Gaussian Mixture Model (GMM), i.c.
K
DR (WIS "0'5",,2**)).
Σ ak N(W Ex),
where oe 2 0 and Σk=1 0g 1, and parameter set 0 = {ors PK, Ea]kEr
estimate parameter 0% in GMM by EM approach and initialize GMM with the 1-th iteration parameter O2 with the procedure as follows:
For 1, No we have
Expectation Step: Compute responsibility Yi for m-step reward weight Wm
ay -V(mm Ja E,)
Yky ag N(Wm Pkx Ex)
Maximization Step: Compute weighted mean H4y and variance Eg by,
Nr 11=1 Yey
No
Σ Vif (Wme" -Mj) (Wmee ppg)²
M4
aj
Ei=l N5
N.
Em1 76
1=1
After EM converges, 0²+1= == faxi Mks Eg I== and profile parameter O = (0]+1 x
Finally, when the two-stage hierarchical method converges, parameter θ₂ of profile parameter θ is our desired best-fitting parameter O for DR (WIC", O*).
3.2 TERMINATION CRITERIA
In this section, we discuss the termination criteria in our algorithm. EM terminates usually when the parameters do not substantively change after enough iterations. For example, one classic termination criterion EM terminates the i-th iteration satisfying as follows,
10
01
max
< EEM:
00
+
8FM
for user-specified δÆM and EEM, where is the model parameter in EM.
However, the same termination criterion for MCEM has risk of early terminating because of the Monte Carlo error in the update step. Hence, adopt a practical method in which the following stopping criterion holds in three consecutive times,
10
0--1|
max O'| + θMCEM EMCEM:
for user-specified 5MCEM and MMCEM Booth & Hobert (1999). Other stopping criteria for MCEM refers to Caffo et al. (2005); Chan & Ledolter (1995).
5

---

Under review as a conference paper at ICLR 2021
3.3 CONVERGENCE ISSUE
The convergence issue of MCEM is more complicated than EM. In light of model-based interactive MDP R, we can always increase the sample size during each iteration of MCEM. In practice, require the Monte Carlo sample size satisfy the following inequality,
Σ No
< 00.
Additional requirement for the convergence property is compact assumption over that parameter space. A comprehensive proof refers to Chan & Ledolter (1995); Fort et al. (2003).
The pseudocode is given in Appendix.
EXPERIMENTS
We evaluate our approach classic environment objectworld introduced by Levine et (2011) which is a particularly challenging environment with a large number of irrelevant features and the highly nonlinearity of the reward functions. Note that since almost only objectworld provides tool that allows analysis and display the evolution procedure of the SIRL problem in a 2D heat map, we skip the typical invisible physics-based based control tasks for the evaluation of our approach, i.e. cartpole Barto et al. (1983), mountain car Moore (1990), MuJoCo Todorov et al. (2012), and etc.
We employ the expected value difference (EVD) proposed by Levine et al. (2011) be the metric of optimality as follows:
DC re
EVD(W) = Σ 7 R(s,, ax) π
R(st, az) x(W)),
which is a measure the difference between the expected reward earned under the optimal policy n² given by the true reward, and the policy derived from the rewards sampling from our reward conditional probability distribution D(W]çE O7), where @ the best estimation parameter in our approach.
4.1 OBJECTWORLD
The objectworld is a learning environment for the IRL problem. It is an x grid board with colored objects placed in randomly selected cells. Each colored object assigned one inner color and one outer color from C' preselected colors. Each cell on the grid board is a state, and stepping to four neighbor cells (up, down, left, right) or staying place (stay) are five actions with 30% chance of moving in random direction.
The ground truth of the reward function is defined in the following way. Suppose two primary colors of C' preselected colors are red and blue. The reward of a state is if the state is within steps of an outer red object and steps of an outer blue object, if the state is within 3 steps of an outer distractors. Continuous
red object, and 0 otherwise. The other pairs of inner and outer colors and discrete versions feature basis functions are provided. For the continuous version, o(s) is 2C-dimensional real-valued feature vector. Each dimension records the Euclidean distance from the state to objects. For example, the first and second coordinates are the distances to the nearest inner and outer red object respectively, and
through all C' colors. For the discrete version, o(s) is a (2C' (OC'-N)ddiin -d)dimensionnal binary feature vector. Each N-dimensional vector records a binary representation of the distance the nearest inner or outer color object with the d-th coordinate the corresponding continuous distance is less than d.
4.2 EVALUATION PROCEDURE AND ANALYSIS
In this section, we design three tasks to evaluate the effectiveness of our generative model reward conditional probability distribution D(W[ç ²*). For each task, the environment setting is as follows. The instance of 10 x objectoorld has 25 random objects with 2 colors and 0.9 discount factor. 20 expert demonstrations are generated according the given optimal policy for the

---

Under review as a conference paper at ICLR 2021
recovery. The length of each expert demonstration is 5-grid size trajectory length. Four algorithms for the evaluation includes MaxEnt, DeepMaxEnt, SIRL, and DSIRL, where SIRL and DSIRL are implemented as Algorithm in Appendix. The weights drawn from reward conditional probability distribution D(W(Çe, O*) ) as the coefficients of feature basis functions {0 ((8,,))=11.
In evaluation, SIRL and DSIRL start from 10 samples and double the sample size per iteration until converges for the convergence issue, refer to Section 3.2. In the first stage, the epochs of algorithm iteration are trajectory set O is preset as 0.95. In the second stage, 3-component GMM for SIRL and DSIRL is set with at most 1000 iterations before convergence. Additionally, the architecture of neural networks in DeepMaxEnt and DSIRL are implemented as 3-layer culnecconeected with the sigmoid function.
20 and the learning rates are 0.01. The parameter in representative
4.2.1 EVALUATION PLATFORM
All the methods are implemented in Python 3.5 and Theano 1.0.0 with machine learning distributed framework Ray Moritz et (2018). The experiments are conducted on machine with Intel(R) Core(TM) 17-7700 CPU @ 3.60GHz and Nvidia GeForce GTX 1070 GPU.
4.2.2 RECOVERY EXPERIMENT
In the recovery experiment, compare the true reward function, the optimal policy, and the optimal value with ones derived from expert demonstrations under the four methods. Since our approach average of all the outcomes which are prone imitate the true optimal value from expert demonstrations, we use the mean of reward conditional probability distribution for SIRL and DSIRL as comparison.
In Figure 1, the EVD of the optimal values in the last row are 48.9, 31.1, 33.7 and 11.3 for four methods respectively, and the covariances of GMM model for SIRL and DSIRL are limited up to 5.53 and 1.36 on each coordinate respectively. It yields that in a highly nonlinear inverse problem, the recovery abilities SIRL and DISRL are better than MaxEnt's 's and DeepMaxEnt's 's respectively. The reason is mainly because Monte Carlo mechanism in our approach alleviates the problem of getting stuck in local minima by allowing random exit from it.
4.2.3 ROBUSTNESS EXPERIMENT
In the robustness experiment, we evaluate the robustness of our approach that solutions generated by C θ*) are effective the IRL problem. To capture the robust solutions, design the following generative algorithm with the pseudocode in Algorithm 1.
Reward
Maxtint
DeepMoxtrtt
DSILL
Reward
policy
Optimal policy
Optimal value
Optimal va
Figure Results recovery experiment.
Figure 2: Results for the robustness experiment.

---

Under review as a conference paper at ICLR 2021
In the right generative algorithm, Frobenius norm to measure the distance be- tween weights drawn from D(WIÇ", (*)) follows,
Algorithm 1:
Generative Algorithm
Input: D(W)ç required solution set size N, and preset thresholds and 6.
Output: Solution set 9 (Wi)
| V Tr(W WT).
We also constrain that each drawn weight W ~ D(WIçE @*)) in the solution set G satisfies as follows,
while do W ~ D(W ;²);;
for any W E S do
if I - Wil d and EVD(W) < then g < W;
I Wi 8 and EVD(W) <,
end
where w represents any member in the so- lution set G. 8, are the preset thresholds the generative algorithm.
end
i i i+
end
In Figure 2, the right column figures are generated from weights in the solution set G whose EVD values are around 10.2. Note that the recovered reward function in the first row has similar but different pattern appearance. The optimal value derived from these recovered reward functions has very small EVD value with the true reward. yields the effectiveness of our robust generative model which can generate more than one solutions to the IRI. problem.
4.2.4 HYPERPARAMETER EXPERIMENT
In the hyperparameter experiment, we evaluate the effectiveness of our approach under the influence of different preset quantities and qualities of expert demonstrations. The amount of information carried in expert demonstrations composes specific learning environment, and hence it has an impact on the effectiveness of our generative model. We verify three hyperparameters' including the number of expert demonstrations in Figure 3, the trajectory length of expert demonstrations in Figure and the portion size representative trajectory class Cº Figure on the objectworld. The shadow of the line in the figures represents the standard error for each experimental trail. Notice that the EVDs for SIRL and DSIRL are both decreasing as the number and the trajectory length expert demonstrations, and the portion size in the representative trajectory class point in Figure 3 that very few expert demonstrations (less than 200) for our approach also yields a small EVDs, which manifests the merit of Monte Carlo mechanism in our approach.
increasing. A notable
Figure 3: Results under 40, 80, 160, 320, 640, 1280 and 2560 ex- pert demonstrations.
Figure 4: Results under 2, 16, 32, and 64 grid size trajectory length of expert demonstrations.
Figure 5: Results under 0.65, 0.70, 0.75, 0.80, 0.85, 0.90, and 0.95 portion size in e...
5 CONCLUSION
In this paper, we propose a generalized problem SIRL for the IRL problem to get the distribution of reward functions. The new problem is well-posed and we employ the method of MCEM to give the first succinct, robust, and transferable solution. In the experiment, we evaluate our approach on the objectworld and the experimental results confirm the effectiveness of our approach.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Pieter Abbeel and Andrew Y Ng. Apprenticeship learning via inverse reinforcement learning. In Proceedings of the twenty-first international conference on Machine learning, pp. 1. ACM, 2004.
Pieter Abbeel, Dmitri Dolgov, Andrew Y Ng, and Sebastian Thrun. Apprenticeship learning for motion planning with application to parking lot navigation. In 2008 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1083-1090. IEEE, 2008
Andrew G Barto, Richard S Sutton, and Charles W Anderson. Neuronlike adaptive elements that can solve difficult learning control problems. IEEE transactions on systems, man, and cybernetics, (5): 834 846, 1983.
James G Booth and James P Hobert. Maximizing generalized linear mixed model likelihoods with an automated monte carlo em algorithm. Journal of the Royal Statistical Society: Series B (Statistical Methodology), (1:1):265-223) 1999.
Brian S Caffo, Wolfgang Jank, and Galin Jones. maximization. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2):235-251, 2005.
Ascent-based monte carlo expectation-
KS Chan and Johannes Ledolter. Monte carlo em estimation for time series models involving counts. Journal of the American Statistical Association, 90(429):242-255) 1995.
Jaedeug Choi and Kee-Eung Kim. Map inference for bayesian inverse reinforcement learning. In Advances in Neural Information Processing Systems, pp. 1989-1997, 2011.
Chelsea Finn, Paul Christiano, Pieter Abbeel, and Sergey Levine. connection between generative adversarial networks, inverse reinforcement learning, and energy-based models. arXiv preprint arXiv:1611.03852, 2016a.
Chelsea Finn, Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pp. 49-58, 2016b.
Gersende Fort, Eric Moulines, et al. Convergence of the monte carlo expectation maximization for curved exponential families. The Annals of Statistics, (144:11200- 1259,, 2003.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse reinforce- ment learning. arXiv preprint arXiv: 11248, 2017.
Karol Hausman, Yevgen Chebotar, Stefan Schaal, Gaurav Sukhatme, and Joseph J Lim. Multi-modal imitation learning from unstructured demonstrations using generative adversarial nets. In Advances in Neural Information Processing Systems, pp. 1235-1245, 2017.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. In Advances in neural information processing systems, pp. 4565-4574 2016.
Jonathan Ho, Jayesh Gupta, and Stefano Ermon. Model-free imitation learning with policy optimiza- tion. In International Conference on Machine Learning, pp. 2760-2769, 2016.
William Kruskal and Frederick Mosteller. Representative sampling, ii:: The current statistical literature. International Statistical Review/Revue Internationale de Statistique, pp. 245-265, 1979.
Sergey Levine, Zoran Popovic, and Vladlen Koltun. Nonlinear inverse reinforcement learning with gaussian processes. In Advances in Neural Information Processing Systems, pp. 19-27, 2011.
Yunzhu Li, Jiaming Song, and Stefano Ermon. Infogail: Interpretable imitation learning from visual demonstrations. In Advances in Neural Information Processing Systems, pp. 3812-3822, 2017.
Bernard Michini and Jonathan P How. Improving the efficiency of bayesian inverse reinforcement learning. In 20/2 IEEE International Conference on Roboties and Automation, pp. 3651-3656. IEEE, 2012.
Andrew William Moore. Efficient memory- -ased learning for robot control. 1990.

---

Under review as a conference paper at ICLR 2021
Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, Melih Elibol, Zongheng Yang. William Paul, Michael Jordan, et al. Ray: A distributed framework for emerging [AI] applications. In 13th {USENIX) Symposium on Operating Systems Design and Implementation ((OSDI) 18), pp. 561-577, 2018. 2018.
Andrew Y Ng. Stuart J Russell, et al. Algorithms for inverse reinforcement learning. In Icml, volume 1, pp. 2. 2000.
Deepak Ramachandran and Eyal Amir. Bayesian inverse reinforcement learning. In ISCAI, volume pp. 2586-2591, 2007.
Nathan D Ratliff, Andrew Bagnell, and Martin A Zinkevich. Maximum margin planning. In Proceedings of the 23rd international conference on Machine learning, pp- 729-736. ACM, 2006.
Stuart Russell. Learning agents for uncertain environments. In Proceedings of the eleventh annual conference on Computational learning theory, pp. 101-103, 1998.
Umar Syed and Robert E Schapire. game-theoretic approach to apprenticeship learning. In Advances in neural information processing systems, pp. 1449-1456, 2008
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 5026-5033.
IEEE, 2012.
Ziyu Wang, Josh S Merel, Scott E Reed, Nando de Freitas, Gregory Wayne, and Nicolas Heess. Robust imitation of diverse behaviors. In Advances in Neural Information Processing Systems, pp. 5320-5329. 2017.
Greg CG Wei and Martin Tanner. A monte carlo implementation of the em algorithm and the poor man's is data augmentation algorithms. Journal of the American statistical Association, 85(411): 699-704, 1990.
Markus Wulfmeier, Peter Ondruska, and Ingmar Posner. Maximum entropy deep inverse reinforce- ment learning. arXiv preprint arXiv: / 508888 2015.
Brian D Ziebart. Modeling purposeful adaptive behavior with the principle of maximum causal entropy. 2010.
Brian D Ziebart, Andrew L Maas, Andrew Bagnell, and Anind K Dey. Maximum entropy inverse reinforcement learning. Aaai, volume 8, pp. 1433-1438. Chicago, II., USA, 2008
10

---

Under review as a conference paper at ICLR 2021
A APPENDIX
The pseudocode SIRL in Section 3 is as follows.
Algorithm 2: Stochastic Inverse Reinforcement Learning
Input: Model-based environment (S,A, 7) and expert demonstrations Ç Monte Carlo sample size No. and preset thresholds θMCEM and CMCEM- Output: Reward conditional probability distribution D (WIs". 6*))
Initialization: Randomly initialization of profile parameter O" := 0%);
while stopping criteria not satisfied (refer to Section 3.2) do
Draw N, reward weights W DD'((W(" 6%) to compose learning task M₁ with
uniformly drawn trajectory element set i;;
# First Stage: Monte Carlo estimation of weights for reward function; for M do
Evaluate V(er), log gM! (OHW) (01)))
Compute updated weight parameter
Wm + W.9 + E X V(e.), log log (Oiii ((.)),
end
Update 0 + (W Ti=1:
# Second Stage: Fit GMM with m-step reward weight (Wme' No with EM parameter initialization 0
while EM not converge do
N..W (gy,t,))
Expectation Step: Yr +
EK R
Maximization Step:
N2 Viy mm
N
H1 - 1=
aj N. 1=1
Yiji
Σ Yij
1= Vig (Wmer
No
- M1)) (Wmm)) - 1₂)
Dg +
E 76
end
Update 0²,11 and profile parameter 9 +- (0/-1, .0/t1)
end


Reviews:
[{"review_id": "wxCZwyptP2h", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n--------\nThe paper describes a method for inverse reinforcement learning---called stochastic IRL---that learns a distribution over reward functions. In that sense, the method is similar to Bayesian approaches, however, the learned distribution doesn't seem to approximate the posterior for any given prior. It is hard to summarize the algorithm because it is highly unclear what the method actually does, but I will try my best by stating my most likely hypothesis.\nThe algorithm starts with an initial set of $N_0$ parameter vectors, each corresponding to the weights for a linear reward function. Each of these weights are improved by using a fixed number of gradient steps using MaxEnt-IRL (Ziebart et al. 2010), where different (overlapping) subsets of the demonstrations are used for the different weights. Subsequently, a GMM over weights is fitted to the $N_0$ improved weights using maximum likelihood (EM). This procedure is iterated, where the weights at each iteration are drawn from the current GMM. The number of weight vectors is doubled at every iteration t, i.e., $N_{t} = 2 N_{t-1}$.\nI'm not at all confident that I got this right. For example, I solely inferred the use of MaxEnt-IRL from a sentence in the introduction (\"In this paper, under the framework of the MaxEnt approach, [...]\").\nThe algorithm is evaluated on a 10x10 gridworld and compared with MaxEnt-IRL and DeepMaxEnt-IRL. The experiments compare two versions of the algorithm called SIRL and DSIRL. The paper doesn't explain how they differ, but a neural network parameterization is mentioned for DSIRL, so I guess D stands for deep and uses DeepMaxEnt-IRL instead of MaxEnt-IRL.\n\nStrong points\n-------------\nLearning a distribution over reward functions is a laudable goal for inverse reinforcement learning as it can be a principled way to deal with the uncertainty in modeling the expert.\n\n\nConcerns\n---------\n__Clarity__\n The paper is really hard to follow, even though I'm very familiar with related work, and the algorithm and the equations seem to be quite simple. There are several reasons for this lack of clarity.\n1) Lack of details. Actually not only details, some of the most fundamental aspects of the algorithm are not mentioned anywhere. For example, the likelihood of the demonstrations given the weights, $g(\\mathcal{O}|\\mathcal{W},\\Omega)$ is nowhere defined. $\\theta$ is also not defined. \n2) Some of the equations are very confusing. For example, the second to last equations at page 4 seems to overload the subscript \"i\" which seems to index both the sampled weight as well as the gradient step, depending on the variable. Even when disentangling these different meanings the equation looks weird to me. Based on the surrounding text, I guess it should mean m gradient steps on the likelihood are performed, but the equation says different. It is also strange that the weights W are updated based on the gradient w.r.t. $\\Omega_1$ and that $\\Omega_1$ is set to the updated weights in the last equation at page 4. It seems like Omega_1 is superfluous. Also, the mixture model $\\mathcal{D}(\\mathcal{W}|\\zeta, \\Omega_2)$ seems to be conditional independent of $\\zeta$ given $\\Omega_2$. What's the difference between $\\mathcal{D}(\\mathcal{W}|\\zeta, \\Omega_2)$ and $h(\\mathcal{W}|\\Omega_2)$?\n3) Some algorithmic choices are not motivated at all (for example, doubling the samples at every iteration), others are only (insufficiently) motivated in hindsight. For example, the use of \"representative trajectory classes\" is motivated after introducing them and based on the example of different drivers with different driving styles. However, the paper never mentioned a multi-expert scenario and also doesn't try to cluster the demonstrations but only randomly removes few demonstrations; thus, it is not even clear how the trajectory classes would tackle such problem. \n4) Bad structure. While it is in general a legitimate approach to introduce the algorithm step by step, I think you should always also provide a rough sketch of the overall algorithm already early on, so that the reader has some context when you introduce the details. The way of introducing the trajectory classes is also a good example for the bad structure of the paper. Before introducing them, there is no mentioning of them, there is no motivation for them, and for all I can tell the paper doesn't even discuss the problem that they are supposed to tackle. And then in Section 2, they are introduced by stating \"Suppose that we have trajectory classes [paraphrased]\" and moving on as if nothing happened. \n5) I must say that the writing is also quite bad. Some sentence are hard to follow due to grammar mistakes, especially if the reader doesn't have enough background to infer their meaning. For example \"[...] suffer from the problem that the true reward shaped the changing environment.\" should probably mean something like \"suffer from the problem that the learned reward function is shaped---that is, it is entangled with the dynamics---and, thus, does not transfer to different environments\". As another example \"However, GAIL is in a lack of an explanation of expert’s behavior and a portable representation for the knowledge transfer which are the merits of the class of the MaxEnt approach, because the MaxEnt approach is equipped with the \"transferable\" regular structures over reward functions.\"\n\n__Soundness__ \nIt doesn't make sense to talk a lot about the soundness of the approach before clarifying what it actually does. However, it seems to me that the learned distribution is only heuristic and does not approximate the posterior for any given prior. Also, for the linear reward function, the maximum likelihood objective is convex, so I think that the different weights should even converge to the same solution. \n\n__Evaluation__\nI don't demand MuJoCo experiments and the like, and in some cases evaluations on gridworlds can be sufficient. But I must say, that the excuse for not performing continuous control problems because the reward functions can not be visualized by 2d-heatmaps lured out a smile when reading the article. If I understand the algorithm correctly, it is significantly more expensive than standard MaxEnt-IRL which already doesn't scale to such problem settings as it requires iteratively solving the reinforcement learning problem. \nThe results on the objectworld are also not convincing. The expected value difference is for 80 demonstrations larger than for 40, and for 160 demonstrations larger than for 80. Similary, the performance initially degrades when increasing trajectory lengths and only improves again when taking trajectories of length 32 or more. \nThe sentence \"A notable point in Figure 3 is that very few expert demonstrations (less than 200) for our approach also yields a small EVDs, which manifests the merit of Monte Carlo mechanism in our approach.\" is highly misleading. 200 trajectiories are not \"few\" for a 10x10 gridworld and an EVD over 20 is not small. When introducing this environment, Levine et al. (2011) achieved an EVD of around 1 based on 8 demonstrations (using 8 steps instead of 5).\nThe robustness experiment seems to subsample the weights based on the EVD (which requires knowledge of the true reward) before evaluation. \n\nQuestions\n--------------\nTo better understand the proposed algorithm, I have the following questions.\n1) How does the algorithm relate to MCEM? Is it some sort of EM within MCEM, where the first stage corresponds to the Monte-Carlo E-Step and the EM in the second stage corresponds to M-step of MCEM?\n2) How are the densities g and h defined?\n3) How does $\\Omega_1$ affect the optimziation. How does it differ from $\\mathcal{W}$?\n4) How exactly are the m-step update steps performed? Do they indeed correspond to m gradient steps on the MaxEnt objective?\n5) Section 3.3.: Should the inequality hold for $\\lim_{t \\to \\infty}$?\n6) What's the difference between DSIRL and SIRL?\n7) How do the algorithm compare in terms of computational cost?\n\nAssessment\n----------\nUnfortunately, I don't see a chance for acceptance here. Even if the algorithm was sound and sufficiently novel, the paper would need to be almost completely rewritten in order to address severe problems of the current presentation.", "extra_scores": {}, "extra_text": {}}, {"review_id": "qU2O9IxcXo", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a novel method for inverse reinforcement learning: inferring a (distribution over) reward functions from a set of expert demonstrations. Prior work has either learned a point-estimate, notably maximum entropy IRL, or used Bayesian methods to learn a probability distribution over reward functions. Maximum entropy IRL has scaled to complex environments with unknown dynamics and non-linear rewards (with methods such as AIRL), but do not learn a probability distribution. By contrast, Bayesian IRL is more theoretically principled, but has not scaled to complex environments or non-linear rewards. This paper performs maximum likelihood estimation of a parameter for a *generative model* over probability distributions, using a Monte-Carlo expectation-maximization (MCEM) method. It therefore still outputs a probability distribution like Bayesian IRL, but is able to learn non-linear rewards unlike prior Bayesian methods.\n\nStrengths:\n  - The method is novel.\n  - The motivation of the work is good: you identify important problems on IRL in the first introductory paragraph. (It would benefit from more follow-through -- which of these problems have you solved?)\n  - Reporting examples of learned reward functions (or mean of the distribution learned) as well as summary statistics like EVD gives a more thorough understanding of the method.\n\nWeaknesses:\n  - The paper was difficult to understand.\n  - The contribution seems limited. The method still seems challenging to scale (and no attempt is made to evaluate this), which seems to be it's main advantage relative to Bayesian IRL. Moreover, it is unclear if the probability distribution learned is well-calibrated, unlike Bayesian IRL.\n  - Weak baselines. DeepMaxEnt is a fairly old approach -- why not try e.g. AIRL? There's also no information as to how you trained the baselines, so it's difficult to know whether e.g. hyperparameter tuning was performed appropriately (noting that this environment is fairly different to what DeepMaxEnt was originally trained in). You should report the results of GPIRL given that this algorithm was developed for exactly this environment.\n\nI find the approach intriguing and would encourage you to continue developing it, but the submission is too preliminary to accept at this stage. In particular, I would suggest the following modifications:\n  + Significant edits to improve clarity. For example, the abstract is quite terse, especially the second sentence. I understood it since I am very familiar with this area, but most readers would be lost. I had to read section 2.2 several times to understand what you were proposing -- here (and elsewhere) you would benefit from giving the reader some intuition before diving into the math. The basic idea is relatively simple: you are performing maximum likelihood estimation on parameters $\\Theta$ that define a probability distribution over weights. Then there are some details: how you sample the data when you perform MLE (effectively from a power set of the demonstrations, thresholded by some minimum size), \n  + Clarify what the benefits of your method relative to prior work are, and then rigorously justify this (whether theoretically or empirically). For example, is having a probability distribution over rewards actually a desiderata (in which case you should evaluate if they're well-calibrated), or is it an artifact of the method? Likewise, being candid about it's limitations would help the reader evaluate whether it is appropriate for their application or what novel research directions exist to improve it.\n  + Improved evaluation. Stronger baselines as discussed in \"Weakness\" above. More environments. Perhaps report runtimes -- this would help assess scalability. I found this line uncompelling: \"Note that since almost only objectworld provides a tool that allows analysis and display the evolution procedure of the SIRL problem in a 2D heat map, we skip the typical invisible physics-based control tasks for the evaluation of our approach\". First, this is wrong: objectworld is not that unique, you could visualize reward over e.g. a gridworld or tabular MDP, and many IRL papers have done so. Second, providing this visualization does not preclude also evaluating in more complex environments -- as the AIRL paper did, for example. I strongly suspect your algorithm simply won't scale to such environments (at least without considerable hand-designed feature engineering) -- if this is true then admit it and discuss how you can address it in future work, and if it's false then show that I'm wrong with results.\n  + You should discuss limitations of your work somewhere, e.g. conclusion. For example the convergence guarantee only holds in the limit of infinite data if I understand -- there is no finite-time bound? This is common for MCMC methods which your approach is related to, so not too surprising, but it's important to make the reader aware of.\n\nSome questions I would appreciate clarification on:\n  - How is $f_M$ actually defined? In particular, what do the weights $\\mathcal{W}$ really do? A naive reading would suggest that you take a linear combination of the \"feature basis functions\" -- but I think you must do something more complex since you evaluate in an environment with a highly non-linear reward?\n  - What is Algorithm 1: Generative Algorithm actually meant to be doing? It seems one could not use this in practice, since it requires computing the EVD, which is only computable if one knows the ground-truth reward -- in which case no need for IRL. So I assume it is meant solely for evaluation -- but I am unsure what this is evaluation. If this is meant to be focusing on the most robust samples, then you should report what probability mass these samples have. Also, is \"any W'\" meant to be a universal or existential quantifier?\n\nSome points to improve clarity. At a high-level:\n  1. Paper might benefit from separate related work section. Intro is currently serving as this. But it detracts from the story. You could just summarize the IRL problem, what the key deficiency in existing work is that you're trying to solve, and then dive into describing your method and contributions.\n  2. Many parentheses should be parenthetical, e.g. “are known Russel (1998)” -> “are known (Russel, 1998)” – change \\cite to \\citep in the source.\n  3. As mentioned before, would benefit from proofreading for grammar. Examples: “problem is ill-posed that the” -> “problem is ill posed: the”, “an inverse problem that a” -> “an inverse problem where a”; “is in a lack of an explanation” -> “does not explain”; “variability set” -> “variable set”.\n\nSome specific points:\n  - Abstract, “considerable performance” – weaselword, how good is it relative to expert, to a baseline, ...?\n  - Preliminary, description of transition dynamics: “being current state s, taking action a and yielding next state s'” – does not make the conditionality clear. Perhaps “transitioning to next state s' when taking action a in state s”.\n  - Preliminary, MDP\\R: I have normally seen this include the discount factor, which is important to understand expert behavior. I think you assume this too in your algorithm? If not, flag this, and describe how you recover rewards without knowing the discount.\n  - Typo: \"experimental trail\"->\"experimental trial\".", "extra_scores": {}, "extra_text": {}}, {"review_id": "YqyHaCWzJ80", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to solve underspecified IRL problem  using an MCEM approach. They claim it is the first succint, robust and transferrable solution and have some results on a Gridworld-like environment. Assessment: The MCEM framework actually is a good fit to the IRL problem and I dont recall if this has been explicitly called out in the literature before. But to me, the connection and resulting algorithm don't seem enough it terms of innovation and usefulness. Arguably, both BIRL and MaxentIRL do essentially (sampled) EM with different parametric forms. The experiments are unacceptably small scale and inconclusive.\n\nThere were confusing parts in the exposition that I will outline below.\n\nGoing by section:\n\nIntro: A good overview of the IRL literature. \n\nsec 2: In the problem statement itself you need to say something about how the expert demonstrations are connected to R. Otherwise they seem like there is no relation at all.\n\n2.2:\nI'm trying to understand the motivation behind the definition of C^E_\\epsilon. \n\nIn the objective formulation you are setting \\Theta to maximize the likelihood of a randomly choose subset of expert trajectories. I have never seen such a thing done before and it is interesting and possibly a route to robustness, but I would have liked more explanation and discussion. In particular, the choice of a uniform measure across subsets of different cardinalities seems bad.\n\n Would it be simpler to just introduce a hidden variable for each trajectory, stating whether it is valid or not (i.e.  drawn from the true reward function and should therefore be used in likelihood computation). I agree then with the 3rd point in 2.2.1, this could be a different way to model expert sub-optimality than BIRL. \n\n3.1.2:\n    It took me a while to read this and I dont think I understand. You seem to set \\Theta_2 = W by the end. Then this seems to be very complicated and redundant. In particular, it is very confusing how the dimensionality of the parameter space can depend on the number of monte carlo samples and changes at each iteration ! Maybe I misunderstood something.\n\n3.1.3: \n  This seems like a straightforward EM derivation.\n\n3.2:\n\n What is the difference between the 2 stopping criteria? Just the fact that the 2nd has a patience of 3? if so, seems trivial.\n\n3.3:\n\n what is being formally claimed here? That given the condition on N_t, convergence will hold and proof is in one of the references?\n\n4 (Experiments):\n\n  Experiments are done on exactly 2 small instances of a variant of Gridworld. This is much smaller scale than the experiments you would expect to see in an ICLR paper (even granted that ICLR is very theory-focused). Previous papers have used IRL for real world problems like robotics, vehicle routing etc. \n\nSummation: A couple interesting starting point ideas in the theory part, but not completely fleshed out and ultimately the development of the EM approach is partly straightforward and part of it is very confusing to me (sec 3.1.2).  Experiments are very unconvincing in terms of scope.  \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "GzfXAw_M5QH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose an approach to model-based inverse reinforcement learning which estimates a Gaussian mixture model over reward-function parameters. The method uses MCEM and samples reward functions from a current estimate of the GMM, updates them via a gradient-descent based maximum likelihood approach and then updates the GMM to fit the updated parameters. The authors evaluate the approach on objectworld.\n\n* The paper is at times hard to follow and should be rewritten to be more clear. The contributions and assumptions could be stated more clearly and the paper would strongly benefit from proof-reading. It would also be helpful to disentangle the machinery of MCEM from the novel algorithmic contributions of this paper.\n* The choice of a GMM to represent the distribution of parameters is not motivated at all in the paper. Intuitively, the main benefit is to allow for k reward-function archetypes that represent the set of expert trajectories well; however, there are no examples nor any evaluation to show in which case this is beneficial.\n* DSIRL is used as an acronym for a variant of the method but is not defined in the paper as far as I can tell.\n* While the method can seemingly be applied to deep as well as linear representations, it is unclear what the chosen features and representation is in the experiments.\n* The method appears to draw a set of weights W_i as well as a corresponding set of expert trajectories O_i at random in each iteration. The motivation for the use of O_i is that it may correspond to different modes in the expert set, e.g. demonstrations by different experts; however, the assignment of weights to object sets is not consistent between iterations so it is unclear to me how this would be able to handle different experts.\n* In the beginning it is mentioned that IRL methods require knowledge of the transition model. While many methods do, modern IRL methods are model-free more often than not, so this claim is misleading.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GcR71LzajNH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Summary\n\nThe authors proposed inverse reinforcement learning (IRL) algorithm based on Monte Carlo expectation-maximization (MCEM) that maximizes the predictive distribution of trajectories given the reward distribution parameter (eq (1)). In my understanding, the knowledge of the environment dynamics is assumed. The authors tried to validate the proposed idea on objectworld (Levine et al., 2011)\n\n### Quality\nThe quality needs to be improved in the sense that a clear theoretical link between the target problem and MC-EM cannot be found in the submission. For example, the main objective (3) is optimized through (4) and (5), but the relation is unclear. There are lots of such things in the submission. \n\n### Clarity\nThe readability of the submission is poor and needs to be improved. Lots of terms are unclear to me (e.g., succinctness, robustness, transferability of rewards). At some part of derivation, I couldn’t understand the motivation. Experiment settings are unclear, and the results are not confident and seem irreproducible with given information. \n\n### Originality\nExploiting the distribution of reward is considered in Bayesian IRL. I think the probabilistic view was originated from Bayesian IRL (e.g., uniform prior on rewards may cover the idea of this work). The submission only sets MaxEntIRL as its baseline, but I think Bayes IRL should have been considered. \n\n### Significance\nThere seems to be a minor contribution \n\n### Detailed comments\n(p.1, `Abstract`) `expert demonstrations may be optimal for many policies`\n- I feel this statement is weird since we haven’t defined the optimality of expert demonstrations.\n\n(p.1, `Abstract`) `we generalize the IRL problem to a well-posed expectation optimization problem stochastic inverse reinforcement learning (SIRL) to recover the probability distribution over reward functions.`\n- SIRL tries to solve the inherent issue of IRL problem, not **generalize** IRL. Also, since Bayesian IRL also recovers the reward distribution, I couldn’t get the major advantage of the SIRL from this statement.\n\n(p.1, `Abstract`) `The solution is succinct, robust, and transferable`\n- Definitions of these expressions seem ambiguous to me. \n\n(p.1, `Abstract`) `a global viewpoint`\n- Again, ambiguous. \n\n(p.1, `Introduction`) \n- It would be better to write it in a more abstract way and separately write down the `Related Work` section.\n- References should be much clearer: LaTeX commands like `\\citet{}` and `\\citep{}` should both be used. \n\n(p.1, `Introduction`) `if the model dynamics are known`\n- Recent works on IRL such as adversarial IRL (Fu et al, 2017) didn’t require the knowledge of model dynamics.\n\n(p.1, `Introduction`) `The recovered reward function provides a succinct, robust, and transferable definition of the learning task`\n- `succinct, robust, and transferable`: Ambiguous\n\n(p.1, `Introduction`) First paragraph\n- Lots of words from `Abstract` seem to be repeated. \n\n(p.1, `Introduction`) `In a real-world scenario, experts always act sub-optimally or inconsistently, which is another challenge.`\n- The sentence seems abrupt. The terms like `sub-optimal` and `inconsistent` here are awkward. \n\n(p.1, `Introduction`) `imposes regular structures of reward functions in a combination of hand-selected features`\n- GAIL (Ho et al, 2016) doesn’t require a hand-crafted feature. \n\n(p.1, `Introduction`) `hand-selected by experts`\n- A word `experts` here seems to imply a reward designer, not an expert on target tasks. I’d rather use a different word here. \n\n(p.1, `Introduction`) `based on demonstrations respectively`\n- `respectively` seems inappropriate. \n\n(p.1, `Introduction`) `Influenced by the work of Finn et al. (2016a;b)`\n- How these references affected AIRL needs to be mentioned. \n\n(p.2, `Introduction`) `because the MaxEnt approach is equipped with the \"transferable\" regular structures over reward functions.`\n- In Ziebart et al., 2008, transferability wasn’t mentioned. \n- I believe the statement -- MaxEnt itself gives transferable reward feature -- is wrong but you should share the correct reference if this is true. \n\n(p.2, `Introduction`) `The solution of SIRL is succinct and robust for the learning task in the meaning that it can generate more than one weight over feature basis functions which compose alternative solutions to the IRL problem`\n- This explanation seems insufficient to understand the meanings of “succinctness” and “robustness”. \n\n(p.2, `Introduction`) `Benefits of the class of the MaxEnt method,`\n- Thanks to the benefits of the class of the MaxEnt method?\n\n(p.2, `Introduction`) `Since of the intractable integration in our formulation,`\n- Due to the intractable integral in our formulation? \n- I think the intractability of the mathematical derivation didn’t need to be mentioned in `Introduction`. \n\n(p.2, `Introduction`) `in a model-based environment`\n- when model dynamics is known? \n\n(p.2, `Introduction`) `In general, the solutions to the IRL problem are not always best-fitting in the previous approaches because a highly nonlinear inverse problem with the limited information is very likely to get trapped in a secondary maximum in the recovery.`\n- I couldn’t understand what the authors wanted to emphasize. \n- It seems like they intended to emphasize the problem of local optima, but I don’t know if such a problem is exactly what’s happening in IRL. \n\n(p.2, `Introduction`) `global exhaustive search`\n- What does `global` imply? Knowledge of dynamics?\n\n(p.2, `Introduction`) `theoretically convergent demonstrated by pieces of literature`\n- is theoretically convergent?\n- How the theorem in the references (Caffo et al., 2005, Chan and Ledolter, 1995)  is applicable to the proposed idea should be much clearer since this is one main advantage that the authors argue. For example, what kind of assumptions are required to acquire global optimality? What is the algorithmic assumption of MC-EM for optimality? How are those assumptions linked with IRL setting?\n\n(p.2, `Introduction`) `is also quickly convergent`\n- converges quickly?\n- How can we guarantee the convergence speed? Empirically or theoretically?\n\n(p.2, `Introduction`) `the preset simple geometric configuration over weight space in which we approximate it with a Gaussian Mixture Model (GMM)`\n- preset -> predefined?\n- approximate it -> approximate\n\n(p.2, `Introduction`) `We generalize the IRL problem`\n- It seems the objective is not a generalization. \n\n(p.2, `Preliminary`) $\\mathcal{T}:=\\mathbb{P}(s_{t+1}=s’|s_t=s, a_t=a)$\n- $\\mathcal{T}(s’|s, a):=\\mathbb{P}(s_{t+1}=s’|s_t=s, a_t=a)$\n\n(p.2, `Preliminary`)  `a sequential of state-action pairs`\n- a sequence of state-action pairs?\n\n(p.2, `Preliminary`) `The estimated complete MDP yields an optimal policy that acts as closely as the expert demonstrations.`\n- The discount factor should be considered as well. \n\n(p.3, `Regular Structure of Reward Functions`) $\\mathcal{N}$\n- I’d rather use a different letter since $\\mathcal{N}$ is used to indicate Gaussian distribution in Section `Second Stage`. \n\n(p.3, `Regular Structure of Reward Functions`) $\\{\\phi_i(s, a)\\}_{i=1}$\n- $\\{\\phi_i(s, a)\\}_{i=1}^M$?\n\n(p.3, `Problem Statement`) $\\mathrm{MDP}\\backslash R:=(\\mathcal{S}, \\mathcal{A}, \\mathcal{T}, \\gamma)$\n- The definition doesn’t match with one without the discount factor $\\gamma$ in `Preliminary`.\n\n(p.3, `Problem Statement`) $\\{\\phi_i(s)\\}_{i=1}^M$\n- $\\{\\phi_i(s, a)\\}_{i=1}^M$?\n\n(p.3, `Problem Statement`) weights $\\mathcal{W}$\n- The definition should be provided. \n- Either $\\mathcal{W}=(\\alpha_1, …, \\alpha_M)$ (for linear model) or the weights of neural network (for non-linear model)?\n\n(p.3, `Problem Statement`) `more likely generates weights to compose reward functions as the ones derived from expert demonstrations`\n- Is this only a special case of Bayesian IRL?\n\n(p.3, `Problem Statement`) `Suppose a representative trajectory class ~`\n- The explanation should be clarified. In my understanding, $\\mathcal{C}_\\epsilon^E$ is a class of sets of trajectories.\n- Why do we need to care such a class with $\\epslion$ threshold?\n\n(p.3, `Problem Statement`) Integrate out unobserved weights $\\mathcal{W}$\n- What does *unobserved* weights mean?\n- Integrate out -> Marginalizing out?\n\n(p.3, `Problem Statement`) trajectory element set $\\mathcal{O}$ assumes to be uniformly distributed for the sake of simplicity in this study\n- I don’t fully understand what’s the advantage of considering a representative trajectory class and why it is required. \n- The section `Note:` tries to explain it, but more explanation or theorems seems to be needed. How can we theoretically guarantee that using a representative trajectory class doesn’t affect our estimation? It seems to me that we cannot guarantee the optimality with this class is the same as the original optimality. \n\n (p.3, `Problem Statement`) $f_{\\mathcal{M}}$\n- How this quantity is related to reward weights is unclear to me. The relationship between weights and $f_\\mathcal{M}$ for both linear and non-linear models should be specified.\n\n(p.3, `Note:) \n- Instead of using a separate section, I’d rather put these statements in the middle of `Problem Statement` for a clearer explanation. \n\n(p.4, `Two-stage Hierarchical Method`)\n- Why do we need to use two-stage method instead of single-stage method (joint optimization over $\\Theta_1$ and $\\Theta_2$)? The advantage of two-stage methods should be briefly mentioned when it first appears for readability. \n- How does the iterative update rule (4), (5) guarantee the optimization of (3)? It’s unclear to me due to the expectation in (4) and (5). My guess is that direct optimization of RHS of (3) is not possible and (4) and (5) might be either lower or upper bound of (3) due to Jensen’s inequality. \n\n(p.4, `Initialization`) `~in each learning task`\n- Do we care about multi-task learning or multiple reward weights only? I believe the latter case.\n\n(p.6, `Experiments`) `since almost only objectworld provides a tool that allows analysis and display the evolution procedure of the SIRL problem in a 2D heat map, we skip the typical invisible physics-based control tasks for the evaluation of our approach, i.e. cartpole Barto et al. (1983), mountain car Moore (1990), MuJoCo Todorov et al. (2012), and etc.`\n- I think this makes the contribution weaker. At least a few classic control tasks should have been considered. One way of evaluating the quality of rewards is retraining the agent with acquired reward, which is already widely used in the literature. \n\n(p.6, `Objectworld`) \n- One figure for illustration will enhance readability. \n\n(p.7, `Evaluation Procedure and Analysis`) `DSIRL`\n- DSIRL abbreviates Deep SIRL but wasn’t mentioned. \n\n(p.7, `Recovery Experiments`)\n- How many runs were used? How’s the mean and confidence interval of the empirical result? \n\n(p.8, `Robustness Experiments`)\n- I couldn’t understand how the robustness of reward is related to the proposed experiments. How the robustness is defined and its relation to the experiment should be clarified.  \n\n(p.8, `Hyperparameter Experiments`)\n- How is the range of hyperparameter search for all methods? Currently, only the results for SIRL and DSIRL are given. \n\n(p.8, `Conclusion`)\n- It seems like both succinctness and transferability were not discussed in the main part of the submission. \n\n### References\n- Levine et al., 2011, “Nonlinear inverse reinforcement learning with gaussian processes“\n- Fu et al., 2017, “Learning robust rewards with adversarial inverse reinforcement learning”\n- Ho et al., 2016, “Generative adversarial imitation learning”\n- Ziebart et al., 2008, “Maximum Entropy Inverse Reinforcement Learning”\n- Caffo et al., 2005, “Ascent-based Monte Carlo expectation-maximization”\n- Chan and Ledolter, 1995, “Monte Carlo em estimation for time series models involving counts“\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper describes a method called 'stochastic' inverse reinforcement learning. It is somewhat unclear how this differs from other probabilistic approaches to IRL. In particular Bayesian approaches have been used in the past to obtain distributions over reward functions. However, SIRL tries to estimate a generative model over such distributions. All the reviewers foudn the paper suffering from lack of clarity, in particular with respect to how the model/algorithm is constructed. There are some possible technical problems with respect to claims about inferring demonstrations by different experts (cf. work on multi-task IRL). The experiments also seem to be insufficient.
Decision: Reject