Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces a formulation for the contextual inverse reinforcement learning (COIRL) problem and proposed three algorithms for solving the proposed problem. Theoretical analysis of scalability and sample complexity are conducted for cases where both the feature function and the context-to-reward mapping function are linear. Experiments were conducted in both a simulated driving domain and a medical treatment domain to compare the three proposed algorithms empirically. Empirical results for using a deep network as the contextual mapping function is also provided.

As a special case of IRL for POMDPs, the contextual IRL problem (with latent contexts) is an interesting research topic that is of interest to researchers working on generalizing IRL to a greater range of real-world applications. This paper was written with clarity and detailed descriptions of experiments. The authors presented their algorithms with thorough theoretical analysis. However, further ablation study and proofs are needed to demonstrate the proposed problem formulation and algorithms outperform existing IRL frameworks.

The authors motivate the problem of COIRL from the formulation of contextual MDP by Hallak et al. (2015). However, the referenced work of Hallak et al. (2015) (published as a preprint on arXiv) does not provide a sufficient argument/evidence for why or in what cases this particular formulation, especially when the context is observed (as assumed by this paper), is better than alternative ones, such as directly modeling the context as part of the state. When a discrete number of contexts exist (as in the simulated driving domain: average vehicle v.s. ambulance), the proposed problem can be reduced to solving a set of normal IRL problems, i.e. learning a reward function under each observed context. For continuous context variables (such as age and weight for patients in the medical treatment domain), an intuitive solution is to model them as part of the state/feature and run normal IRL algorithms. However, the authors did not analyze or show in either of their experiments how existing IRL algorithms compare with the proposed algorithms. Without further explanation, the authors claim, in the fourth paragraph of section 1, that “..Apprenticeship Learning algorithms that try to mimic the expert cannot be used..”, yet it is not trivial to understand why the proposed formulation and algorithms would outperform existing methods/baselines. 

Lastly, it would be more insightful to the readers if the authors can provide some discussion on how they would extend their algorithms to the more interesting/practical case where the context is not directly observed and analyze how latent contexts would affect the performance/complexity of their proposed algorithms.

Overall, this is a well-written paper on an exciting research topic but lacks sufficient analysis and experimental results to support the significance of the intended contributions.


Reference (from the paper): 

Hallak, A., Di Castro, D., & Mannor, S. (2015). Contextual Markov decision processes. arXiv preprint arXiv:1502.02259.


==================================


Edit after rebuttal period: 

- The updated version of the paper includes preliminary results of comparisons with alternative methods and shows that COIRL does have better scalability for modeling problems with large number of observable contexts. It would be great if the authors can show how the baseline performs in the two original experiments as well to understand the improvements gained from adopting COIRL for these problems.

- As stated before, I think COIRL is an exciting research topic on its own. While many recent works on IRL focuses on generalizing to complex problem with deep neural nets as function approximation tools, it is important to analyze the underlying problems from bottom up (in smaller domains and with assumptions that may not always hold) in order to better understand the limits of different solutions. I'd certainly love to see more follow-up works on this topic and how it can be extended to more complex problems.

- Hence, after carefully reading the authors responses and the updated version of the paper, I changed my decision to 'Weak Accept'.


review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: The authors consider the problem of inverse reinforcement learning for CMDPs in which the reward function is a function of the context. They  propose different algorithms for learning the function and evaluate their algorithms on a driving simulator and a sepsis treatment problem (based on real data from the MIMIC corpus).

I think this is a good paper, studying an interesting problem and proposing useful solutions, so it should be accepted. The paper could be improved by putting more focus on the presentation of the studied problem and its variants (also the ones the authors mention their approach is easy to generalize to) and reducing a little the focus on the algorithm. In total they propose three algorithms and give convergence guarantees for all of them. Of course this analysis is important but I found it somewhat distracting from the main flow of the paper. Maybe these guarantees could be moved into a separate section.

A few more points:
* Please comment on why you see estimating contextual transition dynamics as an orthogonal problem.
* In the last paragraph of the conclusion you talk about a safety critical application. For which applications do you think this is practical? I would assume that constantly reviewing of an AI systems' actions is very impractical. (But I agree on these aspects being important.)
* One of the main limitations of this work seems to be that the CMDP\M has to be known. Please comment on how one could expand the analysis/applications/experiments to extend to the case where the CMDP is not known.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This work focuses on the problem of 'contextual' inverse reinforcement learning, where the reward is a function of the current state of the MDP, and a set of context features, which remain constant within each episode.  The primary contribution of this work is the formulation of inverse reinforcement learning (for restricted spaces of context-dependent reward functions) as a convex optimization problem.  Based on this formulation, the paper describes several IRL algorithms based on approaches to solving convex and non-convex optimization problems, including variations of mirror descent, and evolution strategies (in principle allowing for the optimization of reward functions with arbitrary parametric representations).  The algorithms presented in this work all assume that computing an optimal policy for a specific reward function is a relatively inexpensive subroutine, which limits their applicability to domains where such planning is straightforward.  Experimental results are presented for a simple highway driving domain, as well as a simulated patient treatment domain constructed from real-world clinical data.

While the paper focuses one contextual IRL as its key contribution, the paper fails to sufficiently motivate the contextual IRL problem as a useful specialization of inverse reinforcement learning.  While there are clearly problems where a distinct 'context' can be identified, it isn't clear what advantage is gained by considering the context as a separate set of features, as opposed to treating the context as an additional set of state features which do not change within an episode.  With the latter formulation, many existing approaches to inverse reinforcement learning or imitation learning would be applicable.  The specific formulation of the reward as the product of the context and state features is somewhat more flexible than limiting the reward to linear functions of both, but it would be straightforward to apply the proposed algorithms to the case where the state and context are concatenated, and the reward is a quadratic function of state-context vector.

The experimental results are not particularly useful in evaluating the proposed algorithms, as the tasks involved are relatively simple, discrete-state MDPs (with continuous state features), and more importantly, no comparisons with existing IRL approaches are provided.  The potential scalability of evolution strategy optimization to more complex, non-convex reward representations such as deep networks is mentioned, but never empirically demonstrated.
review_title: Official Blind Review #3

Meta-review: The authors introduce a framework for inverse reinforcement learning tasks whose reward functions are dependent on context variables and provide a solution by formulating it as a convex optimization problem.  Overall, the authors agreed that the method appears to be sound.  However, after discussion there were lingering concerns about (1) in what situations this framework is useful or advantageous, (2) how it compares to existing, modern IRL algorithms that take context into account, and (3) if the theoretical and experimental results were truly useful in evaluating the algorithm.  Given that these issues were not able to be fully resolved, I recommend that this paper be rejected at this time.
Decision: Reject