Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Hi,

Could you please clarify the following:
  1. It sounds like alpha is kept fixed in experiments and results are reported for the "exploratory" policy (softmax with temperature alpha) rather than the greedy one. Is that correct and if yes, why?
  2. In both the grid world and Atari, is the critic estimate of Q(s, a) obtained by summing the (discounted) observed rewards for up to t_max timesteps after taking action a in state s, plus the (discounted) estimated V(last observed state)? (= as in A3C)
  3. For the Q-learning step are you freezing the target network and clipping the error as in the Nature DQN paper? If not, why?

Thanks!
review_title: A few more questions

final_score: rating: null
review_content: No questions
review_title: No questions

final_score: rating: null
review_content: Hi,

In 3.2 you consider the case where "the measure in the expectation (is) independent of theta". However this is not the case in practice since both the state and action distributions depend on theta. Could you please comment on how this affects the proposed interpretation? I believe it would be important to explain it in the paper.

Thanks!
review_title: Independence assumption

final_score: rating: null
review_content: Hi,

I'm having some trouble following the reasoning to reach eq. 4 from eq. 3. Equation 3 is valid "for any c", and yet it leads you to a specific value for c, which does not make sense to me. More precisely, the specific point causing this problem seems to be "the gradient of the policy with respect to the parameters is the indicator function": I believe this is incorrect since the parameters are not all free, they are linked together by the constraint sum_a pi(s, a) = 1. For instance if we have two actions then there is a single parameter, and the gradient of the policy with respect to this parameter is either 1 or -1, but not zero.

Please let me know if I'm missing something here, thanks!
review_title: From eq. 3 to eq. 4

final_score: rating: null
review_content: When initially reading the paper I assumed you just forgot to mention in section 2 that the distribution d^pi used a discounted weighted of states, for the policy gradient theorem from Sutton et al (1999) to hold (see above eq. 2 in that paper), however it seems like no such discounting is being applied when actually doing the updates in practice. Obviously in the tabular case it does not matter, but could it be a problem with DQNs? (which value of gamma is being used by the way?)
review_title: Discounted weighting of states?

final_score: rating: 7: Good paper, accept
review_content: Nice paper, exploring the connection between value-based methods and policy gradients, formalizing the relation between the softmax-like policy induced by the Q-values and a regularized form of PG.  

Presentation: 
Although that seems to be the flow in the first part of the paper, I think it could be cast as a extension/ generalization of the dueling Q-network – for me that would be a more intuitive exposition of the new algorithm and findings. 

Small concern in general case derivation: 
Section 3.2: Eq. (7) the expectation (s,a) is wrt to \pi, which is a function of \theta -- that dependency seems to be ignored, although it is key to the PG update derivation. If these policies(the sampling policy for the expectation and \pi) are close enough it's usually okay -- but except for particular cases (trust-region methods & co), that's generally not true. Thus, you might end up solving a very different problem than the one you actually care solving.

Results:
A comparison with the dueling architecture could be added as that would be the closest method (it would be nice to see if and in which game you get an improvement)

Overall: strong paper, good theoretical insights. 
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: This paper shows how policy gradient and Q-Learning may be combined together, improving learning as demonstrated in particular in the Atari Learning Environment. The core idea is to note that entropy-regularized policy gradient leads to a Boltzmann policy based on Q values, thus linking pi & Q together and allowing both policy gradient and Q-Learning updates to be applied.

I think this is a very interesting paper, not just for its results and the proposed algorithm (dubbed PGQ), but mostly because of the links it draws between several techniques, which I found quite insightful.

That being said, I also believe it could have done a better job at clearly exposing these links: I found it somewhat difficult to follow, and it took me a while to wrap my head around it, even though the underlying concepts are not that complex. In particular:
- The notation \tilde{Q}^pi is introduced in a way that is not very clear, as "an estimate of the Q-values" while eq. 5 is an exact equality (no estimation)
- It is not clear to me what section 3.2 is bringing exactly, I wonder if it could just be removed to expand some other sections with more explanations.
- The links to dueling networks (Wang et al, 2016) are in my opinion not explicit and detailed enough (in 3.3 & 4.1): as far as I can tell the proposed architecture ends up being very similar to such networks and thus it would be worth telling more about it (also in experiments my understanding is that the "variant of asynchronous deep Q-learning" being used is essentially such a dueling network, but it is not clearly stated). I also believe it should be mentioned that PGQ can also be seen as combining Q-Learning with n-step expected Sarsa using a dueling network: this kind of example helps better understand the links between methods
- Overall I wish section 3.3 was clearer, as it draws some very interesting links, but it is hard to see where this is all going when reading the paper for the first time. One confusing point is w.r.t. to the relationship with section 3.2, that assumes a critic outputting Q values while in 3.3 the critic outputs V. The "mu" distribution also comes somewhat out of nowhere.

I hope the authors can try and improve the readability of the paper in a final version, as well as clarify the points raised in pre-review questions (in particular related to experimental details, the derivation of eq. 4, and the issue of the discounted distribution of states).

Minor remarks:
- The function r(s, a) used in the Bellman equation in section 2 is not formally defined. It looks a bit weird because the expectation is on s' and b' but r(s, a) does not depend on them (so either it should be moved out of the expectation, or the expectation should also be over the reward, depending on how r is defined)
- The definition of the Boltzmann policy at end of 2.1 is a bit confusing since there is a sum over "a" of a quantity that does not depend (clearly) on "a"
- I believe 4.3 is for the tabular case but this is not clearly stated
- Any idea why in Fig. 1 the 3 algorithms do not all converge to the same policy? In such a simple toy setting I would expect it to be the case.

Typos:
- "we refer to the classic text Sutton & Barto (1998)" => missing "by"?
- "Online policy gradient typically require an estimate of the action-values function" => requires & value
- "the agent generates experience from interacting the environment" => with the environment
- in eq. 12 (first line) there is a comma to remove near the end, just before the dlog pi
- "allowing us the spread the influence of a reward" => to spread
- "in the off-policy case tabular case" => remove the first case
- "The green line is Q-learning where at the step an update is performed" => at each step
- In Fig. 2 it says A2C instead of A3C

NB: I did not have time to carefully read Appendix A
review_title: Interesting links between policy-based and value-based methods

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This is a very nicely written paper which unifies some value-based and policy-based (regularized policy gradient) methods, by pointing out connections between the value function and policy which have not been established before. The theoretical results are new and insightful, and will likely be useful in the RL field much beyond the specific algorithm being proposed in the paper. This being said, the paper does exploit the theory to produce a unified version of Q-learning and policy gradient, which proves to work on par or better than the state-of-art algorithms on the Atari suite. The empirical section is very well explained in terms of what optimization were done.
One minor comment I had was related to the stationary distribution used for a policy - there are subtleties here between using a discounted vs non-discounted distribution which are not crucial in the tabular case, but will need to be addressed in the long run in the function approximation case. This being said, there is no major problem for the current version of the paper. 
Overall, the paper is definitely worthy of acceptance, and will likely influence a broad swath of RL, as it opens the door to further theoretical results as well as algorithm development.
review_title: An excellent paper pointing out connections between existing algorithms, which also leads to new algorithms with excellent results

Meta-review: Reviewers agree that this a high-quality and interesting paper exploring important connections between widely used RL algorithms. It has the potential to be an impactful paper, with the most positive comment noting that it "will likely influence a broad swath of RL". 
 
 Pros:
 - The main concepts of the paper came through clearly, and all reviewers felt the idea was interesting and novel.
 - The empirical part of the paper was convincing and "empirical section is very well explained" 
 
 Cons:
 - There and some concerns about the writing and notation. None of these were too critical though, and the authors responded 
 - Reviewers asked for some more comparisons with alternative formulations.
Decision: Accept (Poster)