Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This work first establishes the connection of maximizing the lower bound of accumulated reward and supervised learning on near-optimal policies. Then it proposes a general framework for policy learning: during the exploration stage, the agent will collect near-optimal trajectories while in the exploitation stage, the agent will perform supervised learning on the collected data. Under this framework, the author argues that the ranking loss could outperform the state-of-the-art on the Atari benchmark.

The overall idea is intuitive yet interesting, and the empirical result is quite impressive. 

Some questions which I think the paper could discuss more:
- In the paper, the near-optimal policy is defined with an absolute threshold, which could be task/environment-specific. I am wondering whether the author tried to set the `near-optimal policy` as a relative value (in the current replay buffer). Then the hyper-parameter could be shared.
- I think some empirical comparisons of the gradient variance (i.e., for Corollary 2) will be more demonstrative, although I could imagine that the near-optimal trajectories will have smaller variance. 
- The choice of C could be tricky in the method as the whole algorithm is highly depending on it. How does the author choose C? If C is tuned for each environment, I am not sure whether it is a fair comparison with C51/Rainbow/IQN. 
- More algorithm training details on the experimental setting (like the hyper-parameters) are needed. 

======
From eq (7) -> eq (8): Does the Taylor expansion near $\lambda_{ji}=0$ make sense?

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper presents a new view on policy gradient methods from the perspective of ranking. The end goal in policy learning is to achieve the right ranking of actions at a state (in the case when deterministic policies are optimal), and the paper proposes a method of doing this inspired from the work on learning to rank. They further argue that in the case with stochastic optimal policies, REINFORCE with softmax policies is rank wise optimal, which is not surprising, but at the same time interesting as well. The other main part of the paper is casting off-policy RL as supervised learning similar to work on self-imitation learning and reward weighted regression methods. This section is presented differently from the past analyses of self-imitation methods and requires the existence of UNOP, which seems like a strong assumption. They then instantiate the framework with GPI based exploration, and show that it achieves better performance than IQN and Rainbow on a subset of atari games.

I am leaning towards a weak reject for this paper, although I am happy to revise my score based on the rebuttal. While the paper is interesting with regards to the ranking perspective, I am not fully convinced about the novelty of the reduction of off-policy learning to supervised learning. This appears already in past works (which the paper cites in the appendix) and the assumption of the existence of UNOP seems strong. I find using a Q-learning agent for exploration a bit complicated and perhaps unnecessary. Also, the paper currently lacks intuition about the effectiveness of their policy gradient approach on top of the data collected from an DQN-based agent. Since reward shaping is done at the trajectory level, why would we expect the supervised regression step to do better than the best trajectory in the data? Also, can the same Q-learning based method perform better if one controls for the number of gradient updates? How would the other methods such as self-imitation or reward weighted regression instead of their proposed supervised learning approach perform on top of data collected from a policy iteration based exploration policy? But the results seem quite promising. 

I would also appreciate some more clarity in terms of writing and presentation.  While this is done in the appendix, I would suggest making references with regards to the supervised learning section (Sec 5) more explicit and refining some of the text in the paper, although this is a minor point.
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes to reparameterize the policy using a form of ranking, and updating the policy gradient update accordingly. This on it's own isn't particularly useful, however the authors use this insight to attempt to convert the RL problem into a supervised learning problem. I think this is an interesting paper and approach, however some issues remain.

High level:

The reward threshold appears to require a-priori knowledge of the correct scale to set it at, and the maximum possible reward, which is unfortunate. Also I'm not sure it would make sense in a sparse reward environment, or an environment with small reward leading up to a single large reward, since the trajectories that get small reward might be discarded erroneously. Maybe there should be an annealing schedule for this parameter? This deserves some discussion at least.

The experimental results are very good, however you are using off-policy replay data and therefore have the issue that using the same data many times can (often) improve performance for any off-policy algorithm. This needs to be controlled and investigated very closely, however I don't even see the batch size being used listed anywhere. The authors need to really demonstrate that the benefit is coming from their algorithm and *not* just the re-use of replay data that the baselines don't get to see as much of.

More minor:

Firstly, there is very bad writing in places, .e.g, abstract: "The state-of-the-art uses action value function to derive policy". Things like this appear in many places.

Q-learning, Q-values sometimes Q is upper case sometimes lower case.

Relative action values are never defined, but if they are what I think they are then they are usually referred to as 'advantage values', and are generally given the notation of A(s,a).

Does the policy in (2) sum to 1?

Assumption 1 is written very confusingly. I would state it as something like:

"Assumption 1. For a state s and action i the set of events {E^i_j}_{j \neq i}, where E^i_j corresponds to action ai being ranked higher than action aj, are conditionally independent, given a MDP and a stationary policy."

Assumption 1: What is is conditioned on?

Thm 1: the statement of the theorem has an equality, but the proof has a clear 'approximate equality', this is very misleading since the statement is not actually true!

I don't follow the proof of corollary 5, especially the sentence explaining (65) -> (66), which doesn't parse. Again the proof is actually approximate but the statement is given using an equality, which is false.

The appendix is very messy and several parts of it just consist of series of equations with no supporting explanation.

Refs:
The original DQN paper (Mnih et al) should be cited after you mention DQN.

I was surprised not to see references to 
ACER: https://arxiv.org/abs/1611.01224 and 
PGQ: https://arxiv.org/abs/1611.01626 when discussing the off-policy / RL + SL work.

Further, in your discussion about entropy regularized RL "However, the discrepancy between
the entropy-regularized objective and original long-term reward exists due to the entropy term." - this has been corrected by a recent work: https://arxiv.org/abs/1807.09647. Probably worth mentioning it here too.
review_title: Official Blind Review #2

Meta-review: The paper introduces a novel and effective approach to policy optimization.  The overall contribution is sufficient to merit acceptance.  Nevertheless, the authors should improve the presentation and experimental evaluation in line with the reviewer criticisms.  The criticisms of AnonReviewer2 in particular should not be neglected.  Regarding the theory, I agree with AnonReviewer3 that the UNOP assumption is too limiting.  The paper would be much stronger if this assumption could be significantly weakened, or better justified.
Decision: Accept (Poster)