Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper present a novel approach to reinforcement learn from batched data that come from very different sources. To achieve this, they propose to learn a prior model and then constrain the RL policy to a trust-region that does not deviate from the domain where the current policy is close to. 

The paper is clearly written.

Strength: the paper is very novel and tries to solve a very challenging problem. The success of this approach shows the potential that collecting large-scale data without distinguishment is possible. With that, the data efficiency will be much improved! The experiment itself shows superior performance than other methods. This method also works for real robots.

Weakness: There is no algorithm box so it can be hard to fully reproduce. The paper mentions stableness for the method however the analysis on this aspect is limited.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper proposes a novel off-policy reinforcement learning algorithm based on a learned prior. Although it is just an extension of existing works, it does outperform the-state-of-the-art methods and can be used in real-world robots.The technique details are well introduced and analysed. The experimental results and comparison are sufficient. 

However, there are also some details require further explanation. For example, In Equation (10), the Langrange multiplier should be constrained to non-negative. But the authors have not taken it into consideration in the optimization process.
Besides, there are also some typesetting problems in the paper, such as in Fig. 7.

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: This paper proposes a method for “offline RL” (a.k.a “batch RL”), i.e. reinforcement learning from a given static dataset, with no option to perform on-policy data collection. Contrary to prior work (Fujimoto et al, Kumar et al, Agarwal et al) in this area which focuses on making Q-learning robust in the offline RL setting, the authors in this paper instead propose making the policy update step robust to the batch RL setting. They achieve this by constraining the policy that is being learned to stay close to a learned “prior policy” (using KL divergence as a distance metric). The authors provide two different ways of learning this prior policy  - the first one is a simple behavior model policy (which involves fitting a model using maximum likelihood on the entire dataset - effectively performing behavior cloning on the entire dataset), while the second is a more sophisticated “advantage behavior model”, which fits a model only on the “good” data (i.e. environment transitions with a positive advantage, where the advantage is estimated using the current policy). The authors extend the MPO and SVG algorithms to work with this constrained update step. 

The authors provide experiments on the standard DM control suite tasks and some robotic tasks (both simulated and real world). For all experiments, the authors collect data by first running MPO in the standard RL setting (i.e. online data collection), and use the replay buffer from these successful RL runs to relearn a policy. The proposed method performs comparable to other methods in some settings (like all the experiments in the 2K episodes setting in Figure 2), and performs slightly better in some other settings (like Hopper and Quadruped in the 10K episodes setting). The gains are more significant in the non-standard robotics domains (Stack, Stack/Leave). The real world robotics results are not compared to any baseline (perhaps due to evaluation being expensive / time-consuming). However, I think the experimental results, especially Figures 7 and 8, point to a potential flaw in the proposed algorithm. In Figure 7 in the appendix, the learned behavior model (ABM) gets about the same performance as the proposed method (ABM+MPO) in pretty much all the four tasks. In one of the tasks (hopper), ABM+MPO does slightly better than ABM alone, but in this case plane MPO is better than ABM + MPO, indicating that this might just be a task on which MPO does well (for some unexplained reason). Similarly, in the robotics results in Figure 8, ABM alone does comparable to ABM+MPO in five of the seven tasks. 

These experimental results imply to me that most of the performance actually comes from the learned prior (ABM), and whether performing MPO on top of it leads to any improvement or not is hard to predict^[1]. The ABM model itself is somewhat novel, and similar ideas under review at ICLR this year have shown that this alone is a useful algorithm for offline RL, see https://openreview.net/forum?id=H1gdF34FvS and https://openreview.net/forum?id=BJlnmgrFvS. However, the ABM model is not emphasized much in the paper. It is also unclear to my as to why the ABM/BM results were placed in the Appendix for Figure 3, while similar results for experiments in Figure 1 were provided using dotted lines.  

For the paper to be accepted for publication, I think it needs to make a stronger argument (experimentally, at least) about the proposed algorithm being superior to ABM. If this is not really the case, then I think it would require a substantial rewrite to emphasize that ABM is where most of the model performance comes from (as seen in concurrent work listed above). 

[1]: A slight caveat here: in the proposed Algorithm 1, the ABM model is learned online along with the policy, but I believe it could be learned without the policy - for example, by using MC returns as in https://openreview.net/forum?id=H1gdF34FvS. 
----------------------

Edit: The author response has convinced me to bump my rating to a weak accept. 
review_title: Official Blind Review #1

Meta-review: The authors present a novel stable RL algorithm for the batch off-policy setting, through the use of a learned prior.  Initially, reviewers had significant concerns about (1) reproducibility, (2) technical details, including the non-negativity of the lagrange multiplier, (3) a lack of separation between performance contributions of ABM and MPO, (4) baseline comparisons.  The authors satisfactorily clarified points (1)-(3) and the simulated baseline comparisons for (4) seem reasonable in light of how long the real robot experiments took, as reported by the authors.  Futhermore, the reviewers all agree on the contribution of the core ideas.  Thus, I recommend this paper for acceptance.
Decision: Accept (Poster)