Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: 
# Summary
The paper shows that good old reward weighted regression (RWR) with value-function baseline is still state-of-the-art algorithm.

# Decision
The paper is well-written and provides many evaluations. The contribution should be articulated more carefully, though, taking into account that most algorithmic ideas are present in prior work (https://openreview.net/forum?id=H1gdF34FvS&noteId=Bkxi11nsdr). Perhaps, the experience replay part is somewhat novel. It seems emphasizing more that the aim is to show that simple methods are competitive rather than focusing on novelty could be a good idea.

Provided that the authors incorporate the feedback of the other reviewers and update the paper accordingly, it will make a good contribution.

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper proposes an off-policy reinforcement learning method in which the model parameters have been updated using the regression style loss function. Specifically, this method uses two regression update steps: one update value function and another one update policy using weighted regression. To compare the proposed method with others [main comprasion], 6 MuJoCo tasks are used for continuous control and LunarLander-v2 for discrete space. 

-- Even though this paper has done a good job in terms of running different experiments, the selection of some of the benchmarks seems arbitrary. For example, for discrete action space, this paper uses LunarLander which is rarely used in any papers so it makes very difficult to draw a conclusion based on these results. Common 49 Atari-2600 games should have been used for comparison. The same thing about experiments in section 5.3 is true too as those tasks are not that well-known. 

-- The proposed method doesn't outperform previous off-policy methods on Mujoco task (Table 1). Since the main claim of this paper is a new off-policy method, outperforming the previous off-policy methods is a fair game. The current results are not convincing enough.

-- There are significant overlaps between this paper, "Fitted Q-iteration by Advantage Weighted Regression", "Model-Free Preference-Based Reinforcement Learning ", and "Reinforcement learning by reward-weighted regression for operational space control" which makes the contribution of this paper very incremental.

-- The authors used only 5 seeds to run Mujoco experiments. Given the sensitivity of Mujoco for different starting points, the experiments should have been run at least with 10 different seeds.

Questions:
1) Shouldn't be an importance sampling ratio between \pi and \mu in the equations? starting from eq.5. 
2) Does the algorithm optimize the respect to $w$ as well? (eq. 15) if yes, why it is not mentioned in algorithm 1? Plus, since $d(s)$ is uniform dist. (at least this is assumed for implementation), eq. 14,15,and 16 (wherever there is d(s)), those can be simplified, e.g. \hat{V} = \sum(w_i V_i), wouldn't be better just introduced simplified version rather than current ones? (referring to the only equation above section 3.3)
3) Is this the same code used to report results in this paper? if yes, I didn't see any seed assignment in the code?! and what is "action_std" in the code?

There are a couple of recent works in merging on-policy with off-policy updates which you might want to cite them. 


review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: [Note: I wrote this review after John Schulman's first comment, before any reply, and before Gehrard Neumann's comment]

The authors propose an actor-critic algorithm based mostly on regression. Being off-policy, the algorithm can learn from multiple policies. It can also be applied to continuous as well as to discrete actions, and it can be trained in a batch RL setting. They compare it to a set of state-of-the-art algorithms in standard openAI gym continuous action benchmarks and show competitive performance despite a much simpler implementation of the algorithm.

I basically subscribe to John Schulman's comment below, both about empirical results and about citing Self-Imitation Learning, but I have a stronger point against the paper, which is insufficient positionning with respect to the relevant literature.

The paper does not cite or discuss the one below, though it looks VERY close:

@inproceedings{neumann2009fitted,
  title={Fitted Q-iteration by advantage weighted regression},
  author={Neumann, Gerhard and Peters, Jan R},
  booktitle={Advances in neural information processing systems},
  pages={1177--1184},
  year={2009}
}

This paper also starts from RWR and performs weighted regression based on the advantage rather than on the return. So to me it is exactly the same idea, and it is mandatory that the authors clearly establish what is the novelty of their work with respect to this previous paper.

Less importantly, the authors may also want to have a look at :

@article{zimmer2019exploiting,
  title={Exploiting the sign of the advantage function to learn deterministic policies in continuous domains},
  author={Zimmer, Matthieu and Weng, Paul},
  journal={arXiv preprint arXiv:1906.04556},
  year={2019}
}

which also uses ideas along the same line.

To me, a good way to improve the novelty of this work would be to perform a detailed empirical study of the inner mechanisms of the algorithm on very simple benchmarks where the value function and policy could be visualized. In particular, how stable is the estimation of the value function? This is known to be an issue, as most algorithms avoid approximating it and prefer estimating the Q-function.

review_title: Official Blind Review #3

Meta-review: This paper caused a lot of discussions before and after the rebuttal. The concerns are related to the novelty of this paper, which seems to be relatively limited. Since we do not have a champion among positive reviewers, and the overall score is not high enough, I cannot recommend its acceptance at this stage.

Decision: Reject