Venue: ICLR 2018 conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: To speed up RL algorithms, the authors propose a simple method based on utilizing expert demonstrations. The proposed method consists in explicitly learning a prediction function that maps each time-step into a state. This function is learned from expert demonstrations. The cost of visiting a state is then defined as the distance between that state and the predicted state according to the learned function. This reward is then used in standard RL algorithms to learn to stick close to the expert's demonstrations. An on-loop variante of this method consists of learning a function that maps each state into a next state according to the expert, instead of the off-loop function that maps time-steps into states.
While the experiments clearly show the advantage of this method, this is hardly surprising or novel. The concept of encoding the demonstration explicitly in the form of a reward has been around for over a decade. This is the most basic form of teaching by demonstration. Previous works had used other models for generalizing demonstrations (GMMs, GPs, Kernel methods, neural nets etc..). This paper uses a three layered fully connected auto-encoder (which is not that deep of a model, btw) for the same purpose. The idea of using this model as a reward instead of directly cloning the demonstrations is pretty straightforward. 

Other comments:
- Most IRL methods would work just fine by defining rewards on states only and ignoring actions all together. If you know the transition function, you can choose actions that lead to highly rewarding states, so you don't need to know the expert's executed actions.
- "We assume that maximizing likelihood of next step prediction in equation 1 will be globally optimized in RL". Could you elaborate more on this assumption? Your model finds rewards based on local state features, where a greedy (one-step planning) policy would reproduce the expert's demonstrations (if the system is deterministic). It does not compare the global performance of the expert to alternative policies (as is typically done in IRL).
- Related to the previous point: a reward function that makes every step of the expert optimal may not be always exist. The expert may choose to go to terrible states with the hope of getting to a highly rewarding state in the future. Therefore, the objective functions set in this paper may not be the right ones, unless your state description contains features related to future states so that you can incorporate future rewards in the current state (like in the reacher task, where a single image contains all the information about the problem). What you need is actually features that can capture the value function (like in DQN) and not just the immediate reward (as is done in IRL methods). 
- What if in two different trajectories, the expert chooses opposite actions for the same state appearing in both trajectories? For example, there are two shortest paths to a goal, one starts with going left and another starts with going right. If you try to generate a state that minimizes the sum of distances to the two states (left and right ones), then you may choose to remain in the middle, which is suboptimal. You wouldn't have this issue with regular IRL techniques, because you can explain both behaviors with future rewards instead of trying to explain every action of the expert using only local state description. 
review_title: The paper presents a method for speeding RL algorithms by learning a reward function from expert demonstrations. The learned reward function explicitly penalizes deviations from the demonstrations. Experiments in simulated environments show the benefits of this method, but the concept is not original.

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper uses inverse reinforcement learning to infer additional shaping rewards from demonstrated expert trajectories.  The key distinction from many previous works in this area is that the expert’s actions are assumed to not be available, and the inferred reward on a transition is assumed to be a function of the previous and subsequent state.  The expert trajectories are first used to train either a generative model or an LSTM on next state prediction. The inferred reward for a newly experienced transition is then defined from the negative error between the predicted and actual next state.  The method is tested on several reacher tasks (low dimensional continuous control), as well as on two video games (Super Mario Bros and Flappy Bird).  The results are positive, though they are often below the performance of behavioral cloning (which only trains from the expert data but also uses the expert’s actions).  The proposed methods perform competitively with hand-designed dense shaping rewards for each task.

The main weakness of the proposed approach is that the addition of extra rewards from the expert trajectories seems to skew the system’s asymptotic behavior away from the objective provided by the actual environment reward.  One way to address this would be to use the expert trajectories to infer not only a reward function, but also an initial state value function (trained on the expert trajectories with the inferred reward).  This initial value function could be added to the learned value function and would not limit asymptotic performance (unlike the addition of inferred rewards as proposed here).  This connection between reward shaping and initial Q values was described by Wiewirora in 2003 (“Potential-based Shaping and Q-Value Initialization are Equivalent”).  

I am also uncertain of the robustness of the proposed approach when the learning agent goes beyond the distribution of states provided by the expert (where the inferred reward model has support).  Will the inferred reward function in these situations go towards zero?  Will the inferred reward skew the learning algorithm to a worse policy?  How does one automatically balance the reward scale provided by the environment with the the reward scaling provided by psi, or is this also assumed to be manually crafted for each domain?  These questions make me uncertain of the utility of the proposed method.

review_title: This paper provides an empirically effective method for inverse reinforcement learning when given thousands of expert state trajectories without having access to the expert’s actions, but it is unclear if the method still performs well beyond distribution covered by the expert trajectories.

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose to solve the inverse reinforcement learning problem of inferring the reward function from observations of a behaving agent, i.e. trajectories, albeit without observing state-action pairs as is common in IRL but only with the state sequences. This is an interesting problem setting. But, apparently, this is not the problem the authors actually solve, according to eq. 1-5. Particularly eq. 1 is rather peculiar. The main idea of RL in MDPs is that agents do not maximize immediate rewards but instead long term rewards. I am not sure how this greedy action should result in maximizing the total discounted reward along a trajectory. 
Equation 3 seems to be a cost function penalizing differences between predicted and observed states. As such, it implements a sort of policy imitation, but that is quite different from the notion of reward in RL and IRL. Similarly, equation 4 penalizes differences between predicted and observed state transitions. 
Essentially, the current manuscript does not learn the reward function of an MDP in the RL setting, but it learns some sort of a shaping reward function to do policy imitation, i.e. copy the behavior of the demonstrator as closely as possible. This is not learning the underlying reward function. So, in my view, the manuscript does a nice job at policy fitting, but this is not reward estimation. The manuscript has to be rewritten that way. 
One could also argue that the manuscript would profit from a better theoretical analysis of the IRL problem, say:
C. A. Rothkopf, C. Dimitrakakis. Preference elicitation and inverse reinforcement learning. ECML 2011
Overall the manuscript leverages on deep learning’s power of function approximation and the simulation results are nice, but in terms of the soundness of the underlying RL and IRL theory there is some work to do.
review_title: Nice simulations, but theoretically incomplete.

Meta-review: The paper presents a method for learning from expert state trajectories using a similarity metric in a learned feature space. The approach uses only the states, not the actions of the expert. The reviewers were variously dissatisfied with the novelty, the theoretical presentation, and the robustness of the approach. Though it empirically works better than the baselines (without expert demos) this is not surprising, especially since thousands of expert demonstrations were used. This would have been more impressive with fewer demonstrations, or more novelty in the method, or more evidence of robustness when the agent's state is far from the demonstrations.
Decision: Reject