Venue: ICLR 2018 conference

Reviews:

final_score: rating: null
review_content: The paper proposes to employ an algorithm from Haarnoja 2017 and Schulman 2017 for reinforcement learning where some of the data comes from potentially adversarial user demonstrations.
The paper does not seem to be able to make up its mind whether the approach is novel or not. "We propose a unified LfD approach" but then all the algorithmic details are "as shown by Haarnoja / Schulman 2017". It gets a bit messy with arXiv papers (in this case they don't seem to have appeared yet in a peer-reviewed venue) but the authors are treating these two papers as published, so I'll also treat this paper as an extension of these. Which means that this paper boils down to "just" taking an RL approach from Haarnoja / Schulman 2017 that works well with data that is truly off-policy and employing it for learning from (imperfect) demonstrations. Hence we have essentially an experimental paper.
The approach does reasonably well, but I'd like to have seen more extensive discussions and an analysis WHY it performs better (or in some experiments actually worse) compared to the baselines.

Minor comments
===============
Eq (1): \operatorname{argmax} https://en.wikibooks.org/wiki/LaTeX/Advanced_Mathematics
Eq (4): I found this very strange, the text reads like this result is from 2010 while the entropy formulation above is claimed to be from 2017...
Sect. 3: "Equation 8" -> "Equation (8)" \eqref{}
Sect. 3.1: So what's new compared to Haarnoja / Schulman 2017? 
Sect. 3: you talk about normalization but only explain in Sect 3.2 (and even there the discussion should be improved)  which part of the update corresponds to this normalization
Sect. 5.2: "method(Mnih" => "method (Mnih"
review_title: So what's new?

final_score: rating: 5: Marginally below acceptance threshold
review_content: SUMMARY:

The motivation for this work is to have an RL algorithm that can use imperfect demonstrations to accelerate learning. The paper proposes an actor-critic algorithm, called Normalized Actor-Critic (NAC), based on the entropy-regularized formulation of RL, which is defined by adding the entropy of the policy as an additional term in the reward function.
Entropy-regularized formulation leads to nice relationships between the value function and the policy, and has been explored recently by many, including [Ziebart, 2010], [Schulman, 2017], [Nachum, 2017], and [Haarnoja, 2017].
The paper benefits from such a relationship and derives an actor-critic algorithm. Specifically, the paper only parametrizes the Q function, and computes the policy gradient using the relation between the policy and Q function (Appendix A.1).

Through a set of experiments, the paper shows the effectiveness of the method.


EVALUATION:

I think exploring and understanding entropy-regularized RL algorithm is important. It is also important to be able to benefit from off-policy data. I also find the empirical results encouraging. But I have some concerns about this paper:

- The derivations of the paper are unclear.
- The relation with other recent work in entropy-regularized RL should be expanded.
- The work is less about benefiting from demonstration data and more about using off-policy data.
- The algorithm that performs well is not the one that was actually derived.

* Unclear derivations:
The derivations of Appendix A.1 is unclear. It makes it difficult to verify the derivations.

To begin with, what is the loss function of which (9) and (10) are its gradients?

To be more specific, the choices of \hat{Q} in (15) and \hat{V} in (19) are not clear.  For example, just after (18) it is said that “\hat{Q} could be obtained through bootstrapping by R + gamma V_Q”. But if it is the case, shouldn’t we have a gradient of Q in (15) too? (or show that it can be ignored?)

It appears that \hat{Q} and \hat{V} are parameterized independently from Q (which is a function of theta). Later in the paper they are estimated using a target network, but this is not specified in the derivations.

The main problem boils down to the fact that the paper does not start from a loss function and compute all the gradients in a systematic way. Instead it starts from gradient terms, each of which seems to be from different papers, and then simplifies them. For example, the policy gradient in (8), which is further decomposed in Appendix A.1 as (15) and (16) and simplified, appears to be Eq. (50) of [Schulman et al., 2017] (https://arxiv.org/abs/1704.06440). In that paper we have Q_pi instead of \hat{Q} though.

I suggest that the authors start from a loss function and clearly derive all necessary steps.


* Unclear relation with other papers:
What part of the derivations of this work are novel? Currently the novelty is not obvious.
For example, having the gradient of both Q and V, as in (9), has been stated by [Haarnoja et al., 2017] (very similar formulation is developed in Appendix B of https://arxiv.org/abs/1702.08165).
An algorithm that can work with off-policy data has also been developed by [Nachum, 2017] (in the form of a Bellman residual minimization algorithm, as opposed to this work which essentially uses a Fitted Q-Iteration algorithm as the critic).

I think the paper could do a better job differentiating from those other papers.


* The claim that this paper is about learning from demonstration is a bit questionable. The paper essentially introduces a method to use off-policy data, which is of course important, but does not cover the important scenario where we only have access to (state,action) pairs given by an expert. Here it appears from the description of Algorithm 1 that the transitions in the demonstration data have the same semantic as the interaction data, i.e., (s,a,r,s’).
This makes it different from the work by [Kim et al., 2013], [Piot et al., 2014], and [Chemali et al., 2015], which do not require such a restriction on the demonstration data.


* The paper mentions that to formalize the method as a policy gradient one, importance sampling should be used (the paragraph after (12)), but the performance of such a formulation is bad, as depicted in Figure 2. As a result, Algorithm 1 does not use importance sampling.
This basically suggests that by ignoring the fact that the data is collected off-policy, and treating it as an on-policy data, the agent might perform better. This is an interesting phenomenon and deservers further study, as currently doing the “wrong” things is better than doing the “right” thing. I think a good paper should investigate this fact more.
review_title: Unclear derivations; Novelty is not obvious

final_score: rating: 5: Marginally below acceptance threshold
review_content: Thanks for all the explanations on my review and the other comments. While I can now clearly see the contributions of the paper, the minimal revisions in the paper do not make the contributions clear yet (in my opinion that should already be clear after having read the introduction). The new section "intuitive analysis" is very nice.

*******************************

My problem with this paper that all the theoretical contributions / the new approach refer to 2 arXiv papers, what's then left is an application of that approach to learning form imperfect demonstrations.

Quality
======
The approach seems sound but the paper does not provide many details on the underlying approach. The application to learning from (partially adversarial) demonstrations is a cool idea but effectively is a very straightforward application based on the insight that the approach can handle truly off-policy samples. The experiments are OK but I would have liked a more thorough analysis.

Clarity
=====
The paper reads well, but it is not really clear what the claimed contribution is.

Originality
=========
The application seems original.

Significance
==========
Having an RL approach that can benefit from truly off-policy samples is highly relevant.

Pros and Cons
============
+ good results
+ interesting idea of using the algorithm for RLfD
- weak experiments for an application paper
- not clear what's new
review_title: New approach or new application?

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper proposes a method to learn a control policy from both interactions with an environment and demonstrations. The method is inspired by the recent work on max entropy reinforcement learning and links between Q-learning and policy gradient methods. Especially the work builds upon the recent work by Haarnoja et al (2017) and Schulman et al (2017) (both unpublished Arxiv papers). 

I'm also not sur to see much differences with the previous work by Haarnoja et al and Schulman et al. It uses demonstrations to learn in an off-policy manner as in these papers. Also, the fact that the importance sampling ration is always cut at 1 (or not used at all) is inherited from these papers too. 

The authors say they compare to DQfD but the last version of this method makes use of prioritized replay so as to avoid reusing too much the expert transitions and overfit (L2 regularization is also used). It seems this has not been implemented for comparison and that overfitting may come from this method missing. 

I'm also uncomfortable with the way most of the expert data are generated for experiments. Using data generated by a pre-trained network is usually not representative of what will happen in real life. Also, corrupting actions with noise in the replay buffer is not simulating correctly what would happen in reality. Indeed, a single error in some given state will often generate totally different trajectories and not affect a single transition. So imperfect demonstration have very typical distributions. I acknowledge that some real human demonstrations are used but there is not much about them and the experiment is very shortly described. 
review_title: Similar to previous work - Experimental result not so convincing

Meta-review: I appreciate the experimental results, which includes a comparison against several baselines, however, I echo some of the concerns raised by the reviewers that the formulation is unclear and hard to follow. Moreover, the novelty over [Nachum, 2017] and [Haarnoja, 2017] seems small. Especially because [Nachum, 2017] also used expert trajectories to improve the performance in their experiments.

Detailed comment:
The use of log-sum-exp state values is only valid for the optimal policy, so it is not clear how an on-policy state value is replaced with the log-sum-exp state value. Also, because the equations that you derive characterize the optimal policy, I am not sure if you need importance correction at all.
Decision: Invite to Workshop Track