Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper proposes an imitation learning algorithm that learns from state-only demonstrations and assumes partial knowledge of the transition dynamics. The demonstrated states are decomposed into "responsive" and "unresponsive" features. The imitation policy is trained in an environment where the responsive state features are simulated and controllable by the agent, and the unresponsive state features are replayed from the demonstrations. The agent is rewarded for tracking the responsive state features in the demonstrations.

Overall, I was confused by this paper. Isn't it problematic that the imitation agent is trained in an environment where unresponsive state features from the demonstrations are replayed and not affected by the agent's actions? It would be nice to expand Section 4 to explain why it makes sense to reflect the unresponsive component in the transition kernel. It would also be helpful to include some of this information in Section 1.

I am also confused by the method. What is the purpose of observing unresponsive features, if the reward function for the imitation agent is only defined in terms of the responsive features? Is it that the imitation policy is conditioned on the unresponsive features?

There are several important experimental details missing from Sections 4 and 6. What distance metric was used to define the reward function? For example, was it Euclidean distance in pixel space for the Atari games? The main experimental results in Figure 2 are difficult to interpret without knowing the how the reward function is defined in the eMDP. Could the authors either provide definitions of the reward functions of the eMDPs in the experiments, or measure performance of the imitation agents using more interpretable or standardized metrics (e.g., game score) for the chosen environments?

It is also concerning that the Google Drive links to code and supplementary materials aren't anonymized.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper provides a theoretical basis called expert-induced MDP (eMDP) for formulating imitation learning through RL when only partial information on the transition kernel is given. Examples include imitating one player in multiplayer games where the learner cannot explicitly have the transition kernel for other players. eMDP just uses the available part of transition kernel (e.g. the target player's state-action transitions) for learning, but substitute the next state from demonstrations 'as is' into the unavailable part. In this situation, using IPM (Wasserstein-like) divergence, it is easily guaranteed that eMDP is consistent with the optimal solution in the limit. However, this does not guarantee anything when the optimum is not reached. The paper's main contributions would be to provides error bounds (Lemma 5.2 in particular) for the convergence rate in terms of Lipschitz constants and the maximum error between eMDP and the demonstration set. This roughly means that when transition functions are continuous and smooth and the sample demonstration set well characterize the underlying MDPs, then eMDP is theoretically guaranteed to work well. If these factors (Lipschitz continuity and the max deviation of state distributions conditioned on the given demonstration set) are expected to be small, eMDP can nicely perform learning with partial knowledge. Some demonstrations confirm eMDP works when simple BC fails in multiplayer games.

Overall, the paper is worth accepting. The proposed eMDP would be a technically simple solution, but providing a solid theoretical basis for that solution to these commonly seen situations of imitation learning with partial knowledge would be quite beneficial. The paper is well written with intuitive examples and nice overview summaries for the theoretically complicated parts. Though the implications of the experimental part are not clear, the paper fairly states this point and also provides a video for visual assessment with displaying ghost imitations. 

One minor question: For the experimental part, BC baselines for multiplayer games just imitates the target player's behavior (player1), completely ignoring the states/actions of player2 (ignoring the states of bricks and balls for breakout-v0)? 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: * Paper summary.
The paper considers an IL problem where partial knowledge about the transition probability of the MDP is available. To use of this knowledge, the paper proposes an expert induced MDP (eMDP) model where the unknown part of transition probability is modeled as-is from demonstrations. Based on eMDP, the paper proposes a reward function based on an integral probability metric between a state distribution of expert demonstrations under the target MDP and a state distribution of the agent under the eMDP. Using this reward function, the paper proposes an IL method that maximizes this reward function by RL. The main theoretical result of the paper is that the error between value function of the target MDP and that of the eMDP can be upper-bounded. Empirical comparisons against behavior cloning on discrete control tasks show that the proposed method performs better. 

* Rating.  
The main contribution of the paper is the eMDP model, which enables utilizing prior knowledge about the target MDP for IL. While this idea is interesting, eMDP is too restrictive and its practical usefulness is unclear. Moreover, there are other issues that should be addressed such as clarity and experiments. I vote for weak rejection. 

* Major comments:
- Limited practicality due to an assumption on the unresponsive transition kernel.
In eMDP, the unresponsive transition kernel is modeled from demonstrations by directive using the observed next states (in demonstrations) as a next state of the agent. This modeling implicitly assumes that the agent cannot influence the unresponsive part of the state space. This is too restrictive, since actions of the agent usually influence other parts of the state space such as opponents and objects. For instance, in pong, actions of the agent indeed influence trajectories of the ball. Due to this restrictive assumption, I think the practicality of eMDP is too limited.

Also, it is unclear what happens when the transition of unresponsive state space is stochastic. In this case, the unresponsive transitions of eMDP would be incorrect since eMDP assumes deterministic transitions as-is from demonstrations. Though I am not sure about this. 

- The equality in Eq. (3) should be an upper-bound. The IPM is defined by the absolute difference between summations (expectations), but the right-hand side of Eq. (3) is summations (expectations) of the absolute difference. These two are not equal, and the right-hand side should be an upper-bound. 

- The paper is difficult to follow. There are many skipping contents in the paper. Especially, in the description of eMDP and its solution, where the paper describes the optimal solution of eMDP (Section 2) before describing the eMDP model (Section 4). Also, it is unclear from my first reading pass what is the actual IL procedure. Including a pseudo-code would help. 

- The state-value function’s error bound ignores a policy. The proof utilizes the state distribution P(s’|s). However, this distribution should depend on a policy. It is unclear to me what is the policy function used in this error bound. Does this error-bound hold for any policy or only for the optimal policy? In the case that it only holds for the optimal policy, this result still does not provide useful guarantees when the optimal policy is not learned, similarly to the result in Proposition 2.1.

- Empirical evaluation lacks strong baseline methods. The paper compares the proposed method against behavior cloning, which is known to perform poorly under limited data. The paper requires stronger baseline methods such as those mentioned in Section 3. Also, it is strange that the vertical axis in Figure 2 represents the reward function of eMDP, which is an artificial quantity in Eq. (5). The results should present the reward function of the target MDP, which is the ground-truth reward of the task.

* Besides the major comments, I have few minor comments and questions:
- Does the reward function in Eq. (5) correspond to the Wasserstein distance for any metric d_s? If it is not, then the value function’s error-bound for the Wasserstein distance is not applicable to this reward function. 
- What is the reward function r(s) in the value function’s error-bound that we can control the Lipschitz constant? We do not know the ground-truth reward r(s,a) so it cannot be controlled. The reward r(s,s’) of eMDP is given by the metric d_s of the state space so again we cannot control it. 
- What is the metric d_s used in the experiments? 
- How do you perform BC without expert actions in the experiments? 
- Confusing notations. E.g., the function r is used for r(s,a) (MDP’s reward), r(s, s’) (eMDP’s reward), and r(s) (reward in the error bound’s proof); these r’s have different meaning and should be denoted differently.
- Typos. E.g., “Lemma 2.1” in Section 5 should be “Proposition 2.1”, “Lemma A.1” should be “Proposition 2.1”. "tildeF". etc.

*** After authors' response.
I read the other reviews and the response. My major concern was the impracticality of eMDP. While I can see that the eMDP model provides advantages over BC, I still think that it is impractical since it assumes the agent cannot influence the unresponsive part. Moreover, extracting next-unresponsive states as-is from demonstrations is problematic with stochastic transitions, since actual next-unresponsive states can be different from next-unresponsive states in demonstrations even when the sames state and actions are provided. Given that the major concern remains, I still vote for weak rejection. 

review_title: Official Blind Review #3

Meta-review: The paper introduces the concept of an Expert Induced MDP (eMDP) to address imitation learning settings where environment dynamics are part known / part unknown. Based on the formulation a model-based imitation learning approach is derived and the authors obtain theoretical guarantees. Empirical validation focuses on comparison to behavior cloning. Reviewers raised concerns about the size of the contribution. For example, it is unclear to what degree the assumptions made here would hold in practical settings.
Decision: Reject