Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes to extend the determinist policy gradient algorithm to learn from demonstrations. The method is combined with a type of density estimation of the expert to avoid noisy policy updates. It is tested on Mujoco tasks with expert demonstrations generated with a pre-trained network. 

I found the paper a bit hard to read. My interpretation is that the main original contribution of the paper (besides changing a stochastic policy for a deterministic one) is to integrate an automatic estimate of the density of the expert (probability of a state to be visited by the expert policy) so that the policy is not updated by gradient coming from transitions that are unlikely to be generated by the expert policy. 

I do think that this part is interesting and I would have liked this trick to be used with other imitation methods. Indeed, the deterministic policy is certainly helpful but it is tested in a deterministic continuous control task. So I'm not sure about how it generalizes to other tasks. Also, the expert demonstration are generated by the pre-trained network so the distribution of the expert is indeed the distribution of the optimal policy. So I'm not sure the experiments tell a lot. But if the density estimation could be combined with other methods and tested on other tasks, I think this could be a good paper.  
review_title: Hard to read

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper considers the problem of model-free imitation learning. The problem is formulated in the framework of generative adversarial imitation learning (GAIL), wherein we alternate between optimizing reward parameters and learner policy's parameters. The reward parameters are optimized so that the margin between the cost of the learner's policy and the expert's policy is maximized. The learner's policy is optimized (using any model-free RL method) so that the same cost margin is minimized. Previous formulation of GAIL uses a stochastic behavior policy and the RIENFORCE-like algorithms. The authors of this paper propose to use a deterministic policy instead, and apply the deterministic policy gradient DPG (Silver et al., 2014) for optimizing the behavior policy. 
The authors also briefly discuss the problem of the little overlap between the teacher's covered state space and the learner's. A state screening function (SSF) method is proposed to drive the learner to remain in areas of the state space that have been covered by the teacher. Although, a more detailed discussion and a clearer explanation is needed to clarify what SSF is actually doing, based on the provided formulation.
Except from a few typos here and there, the paper is overall well-written. The proposed idea seems new. However, the reviewer finds the main contribution rather incremental in its nature. Replacing a stochastic policy with a deterministic one does not change much the original GAIL algorithm, since the adoption of stochastic policies is often used just to have differentiable parameterized policies, and if the action space is continuous, then there is not much need for it (except for exploration, which is done here through re-initializations anyway). My guess is that if someone would use the GAIL algorithm for real problems (e.g, robotic task), they would significantly reduce the stochasticity of the behavior policy, which would make it virtually similar in term of data efficiency to the proposed method.
Pros:
- A new GAIL formulation for saving on interaction data. 
Cons:
- Incremental improvement over GAIL
- Experiments only on simulated toy problems 
- No theoretical guarantees for the state screening function (SSF) method
review_title: This paper proposes an extension of the generative adversarial imitation learning (GAIL) algorithm by replacing the stochastic policy of the learner with a deterministic one. Simulation results with MuJoCo physics simulator show that this simple trick reduces the amount of needed data by an order of magnitude.

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper lists 5 previous very recent papers that combine IRL, adversarial learning, and stochastic policies. The goal of this paper is to do the same thing but with deterministic policies as a way of decreasing the sample complexity. The approach is related to that used in the deterministic policy gradient work. Imitation learning results on the standard control problems appear very encouraging.

Detailed comments:

"s with environment" -> "s with the environment"?

"that IL algorithm" -> "that IL algorithms".

"e to the real-world environments" -> "e to real-world environments".

" two folds" -> " two fold".

"adopting deterministic policy" -> "adopting a deterministic policy".

"those appeared on the expert’s demonstrations" -> "those appearing in the expert’s demonstrations".

"t tens of times less interactions" -> "t tens of times fewer interactions".

Ok, I can't flag all of the examples of disfluency. The examples above come from just the abstract. The text of the paper seems even less well edited. I'd highly recommend getting some help proof reading the work.

"Thus, the noisy policy updates could frequently be performed in IL and make the learner’s policy poor. From this observation, we assume that preventing the noisy policy updates with states that are not typical of those appeared on the expert’s demonstrations benefits to the imitation.": The justification for filtering is pretty weak. What is the statistical basis for doing so? Is it a form of a standard variance reduction approach? Is it a novel variance reduction approach? If so, is it more generally applicable?

Unfortunately, the text in Figure 1 is too small. The smallest font size you should use is that of a footnote in the text. As such, it is very difficult to assess the results.

As best I can tell, the empirical results seem impressive and interesting.

review_title: Combines IRL, adversarial training, and ideas from deterministic policy gradients. Paper is hard to read. MuJoCo results are good.

Meta-review: All of the reviewers found some aspects of the formulation and experiments interesting, but they found the paper hard to read and understand. Some of the components of the technique such as the state screening function (SSF) seem ad-hoc and heuristic without much justification. Please improve the exposition and remove the unnecessary component of the technique, or come up with better justifications.
Decision: Reject