Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper studies shorting coming of existing off-policy methods when it comes to prior data and fine-tuning and shows that those existing methods can't effectively utilize previously collected data with online updates. To address this problem, they propose to constraint policy updates with respect to behavioral policy. Their proposed method is built mainly on the top of AWR [1]. 

- There is a significant similarity between the proposed method and AWR [1]. The main difference that I can see is AWR doesn't have a fine-tuning step but this paper does. Can authors list their differences with AWR? 

- CRR [3] is very similar to the proposed method in this paper as well, what are the differences? ( I disagree with the authors that CRR is a concurrent work with this paper)

-  This paper claims that AWAC can utilize various types of prior data without any changes to their method and it is agnostic to the type of behavioral policy (e.g. random vs. expert). I don't see how this is the case. For example, if data were collected by a random policy, this method constrains the current policy to be a uniform one.

- Experiments don't make a case for this method, i.e. AWAC is only better than others in relocate-binary-v0 and barely better in Walker2d-v2.  ABM almost gets the same performance.

- Fine-Tuning with random policy data is not convincing either. I'd suggest running some experiments with D4RL [5] dataset like *-medium-expert, *-medium-replay, and *-random to build a better case for your proposed method.

- One reason that existing RL methods can't fully utilize previously collected data with online updates is the distribution shift between the policy (a.k.a behavioral policy) that collected the data and the learned policy [2, 3]. Can the authors comment on this and discuss how the proposed method addresses the distribution shift problem that likely the root cause of this problem? A section in the paper about this will help to improve this paper.  
 
- Related work section needs significant improvement. It only lists various papers without any useful discussion.  

Even though I found this paper interesting, I'm still not convinced about the contribution of this paper. Given similarity and overlap with previous works [1,3], more experiments could be helpful to see how this method is better than others and why one should select this method vs. others.

Minor comments:

It seems there are various problems with your bibtex, for example, "Behavior Regularized Offline Reinforcement Learning" wasn't published in iclr and author names are mixed with other information like in  "A Generalized Path Integral Control Approach to Reinforcement Learning" "usc" in author names and "Web Services" appeared with author names "P3O: Policy-on Policy-off Policy Optimization."  

[1] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning. sep 2019. 

[2] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems. Technical report, 2020

[3 Rasool Fakoor, Pratik Chaudhari, and Alexander J Smola. P3O: Policy-on Policy-off Policy Optimization. In Conference on Uncertainty in Artificial Intelligence (UAI), 2019. 

[4] Ziyu Wang, Alexander Novikov, Konrad Zołna, Jost Tobias Springenberg, Scott Reed, Bobak Shahriari, Noah Siegel, Josh Merel, Caglar Gulcehre, Nicolas Heess, and Nando De Freitas. Critic Regularized Regression. 2020.

[5] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning, 2020
review_title: interesting paper but some concerns 

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

In this paper, the authors intend to accelerate on-line reinforcement learning with off-line datasets. To achieve this goal, they propose an algorithm called advantage weighted actor-critic (AWAC), which uses an implicit constraint to reduce accumulated bootstrapping error when doing off-line training and reduce the conservation when doing on-line fine-tuning. The experiments show that the proposed method can learn difficult, high-dimensional, sparse reward dexterous manipulation problems from human demonstrations and off-policy data.

Pros:
1. The organization is clear and related work is sufficient. This paper introduces the off-line training and on-line fine-tuning problem and its challenge by gradually summarize the existing studies, and with sufficient investigation of existing work, it is easy for readers to dive into the problem
2. The proposed method is easy to implement. Thanks to the implicit formulation of the constraint, the proposed AWAC algorithm can sample directly from off-line datasets without fitting a parametric model. This strategy is especially advantageous when the dimensionality is high. Besides, since this algorithm does not need the parametric model any more, it is more friendly for users who are not familiar with the off-line dataset.
3. The details of experimental settings are provided, and thus there should be no issues with the repeatability of the experiments. 

Cons:
1. It is doubtful for the contribution on online fine-tuning. The constraint is designed for off-line training because off-line training has the problem of bootstrapping error accumulation. There is no judgment or proof that online training also suffers from the bootstrapping problem. How can the authors claim their contribution on on-line learning with the constraint?
2. The technical depth is not high. It seems the only contribution of this paper is to combine the constraint with an unparameterized strategy. But I would like to say that this is not a big problem, as many influential reinforcement learning algorithms are straightforward but have very significant improvement, e.g., DDQN and PPO.
3. The writing needs polish. E.g., in Equation (4), it should be 'max' instead of 'argmax', and there exists non-English spelling in Section 3.1.
review_title: A fairly good idea with not so high technical depth

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper describes an approach to use offline data to accelerate the
online learning process of reinforcement learning.
The proposed approach, called AWAC, uses dynamic programming to train
a critic and supervised learning to trained a constrained actor.
The idea is to use a static dataset of experience tuples for
pre-training and use some online interactions to learn the optimal
policy for the current task.
For the policy improvement step they optimize the policy to maximize
the estimated Q-value function but constrained to remain close to the
actions observed in the data, using the Kulback-Leibler divergence.
This constrain is incorporated in the optimization and solve using the
Lagrangian. 
The proposed approach is very similar to AWR, however, instead of
estimating the value function of the behavior policy with a Monte
Carlo approach, they estimate the Q function of the current policy via
bootstrapping. 
AWAC is tested in several MuJoCo simulator problems, three shown in
the main paper (in-hand rotation of a pen, opening a door by
unlatching the handle, and picking up a sphere and relocating it in a
target location), and others shown in the appendix.

Positive:
- Use previously collected data for online learning is certainly
relevant for reinforcement learning is domains such as robotics. The
proposed approach shows that a slight change in how to evaluate the
Q-function to estimate the advantage function can produce a
significant difference

Negative:
- The paper follows very closely the description of AWR with only
a slight variation in how to evaluate the advantage function. 
- It is not clear how close the static dataset has to be to a "good"
policy and how it affects the learning process.

review_title: An incremental improvement on combining offline data with online learning

final_score: rating: 3: Clear rejection
review_content: This paper studies challenges of offline RL with online fine-tuning and proposes an off-policy actor-critic method to address these challenges. The proposed method uses a supervised learning style to update the model parameters and avoids the behavior model estimation.  Empirical results show that the proposed method provides rapid learning with prior demonstration data and online experience.

I have a major concern about the novelty of this paper. The major component of the proposed AWAC method, updating the model parameter using supervised learning, is exactly the same as AWR (Peng et. al., 2019). One minor difference seems that AWAC uses an off-policy policy evaluation, but this contribution is very marginal. 

This paper provides an analysis on challenges of combining offline RL with online improvement, which motivates this paper. However, most of the discussed challenges are quite well-known. For example, one major challenge is to estimate the behavior model in offline data. This paper does not discuss techniques/methods that address this challenge, like DualDICE (Nachum et al., 2019) and CQL (Kumaret al., 2020). A comparison to these methods is highly recommended. In addition, this paper may also include additional model-based offline RL methods, like MOPO.

review_title: Lack of novelty

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper presents an improved actor-critic algorithm for learning from offline data while requiring minimum interactions during online learning. The approach implicitly updates a low-variance policy without the need to estimate the sampling policy. The empirical study shows that AWAC leads to a faster convergence rate compared to the competitors.

Overall, the paper provides an incremental contribution over the existing works for off-policy RL problems. However, the proposed algorithm demonstrates some practical benefits in simulated scenarios and is analyzes from different perspectives. Hence, I vote for accepting, given the following concerns are addressed in the rebuttal period.

Pros:
- Interesting adaption of algorithms to effectively use the offline data in online RL scenarios which is independent from the sampling policy
- Ensuring the policy to stay close to the observed data without directly estimating the sampling policy
- Nicely motivated with helpful examples
- Clarifies differences with the related work

Comments:
- The presentation of the paper could be improved. There are some inconsistencies in the text. At the beginning the approach is introduced to be based on dynamic programming and then turns into temporal difference learning, which is misleading. In addition, offline pre-training and online fine-tuning is not quite accurate as the algorithm 1 is using the mix of data in one phase which does not involve pre-training, or is it actually in two phases?

- The paper seems to be an incremental improvement/combination of several recent works mentioned in the paper.

- The experiments are all on simulated environments. It would be interesting to see how the approach performs on real-world applications, and with some other evaluation metric than convergence rate. 

- Demonstration data is not always as complete as mentioned in the paper. Expert demonstrations that are used for imitation learning, normally lack the reward values and in the form of (s,a) pairs only. 


Minor:
-Typo in section 3.2: is policy is pre-trained
- Plot 4 in Fig. 2 not addressed in section 3.3
review_title: Incremental but practically interesting approach

Meta-review: This paper proposed a new method improving online reinforcement learning using offline datasets. Three reviewers suggested (borderline) acceptance and two did rejection. The main concerns of reviewers are (a) limited/incremental novelty (from all reviewers) and (b) limited experiments (from three reviewers). AC also agrees that the authors' response for novelty beyond the prior works, e.g., AWR (and CRR), is not convincing enough (although their goals/settings are different). AC also thinks that more discussion, analysis and results when offline datasets are poor (e.g., far from experts) are necessary to meet the high standard of ICLR (the authors provided some, but AC thinks it is not convincing enough). Hence, AC recommend rejection. 
Decision: Reject