Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors attempted to address the challenge posed by distribution shift to fine tuning policies learned through offline RL. The proposed solution includes two parts: (1) a method called “balanced replay” that uses a density approximation of how similar the old offline samples are to the new online samples and accordingly determines how offline and online samples in the replay buffer should be mixed during fine tuning; and (2) an ensemble of “pessimistic” Q functions (trained with conservative deep Q learning) to further strengthen the stability of the fine tuning.
strengths_and_weaknesses: Strengths

i.	The case for the need to address distribution shift in fine tuning policies learned through offline RL is well motivated: Section 3 and Figure 1 are well structured and well presented. Kudos to the authors!

ii.	Balanced replay is a relatively novel approach.

iii.	The ensemble of pessimistic Q functions is empirically effective.

Weaknesses

i.	The relationship between the two parts of the solution is unclear. The pessimistic ensemble seems to be an extra add-on. It seems to be an afterthought for better performance  rather than an integrated part of an architecturally cohesive solution.

ii.	Discussion of the ensemble, about why and how it works, stopped at vague allusions to “resolution” without detail. (Technical Issue v below).

iii.	The treatment of BCQ as a baseline seems quite unfair. See Technical Issues (vi) below for more detail.

iv.	The effectiveness of the method in the real world is doubtful. See Technical Issues (ix and x) below for more detail.

summary_of_recommendation: a.	There is an interesting idea that is worth discussing by the RL community: “balanced replay”. Also, the analysis of distribution shift through a careful study of one concrete case is quite helpful and illuminating.

b.	The relationship between the pessimistic ensemble and distribution shift is not clearly established. From the experiments, it seems that the ensemble is mainly contributing towards stabilizing training. Overall, the ensemble part seems to be a separate proposal.

c.	The effectiveness of the approach for real-world robotics is doubtful. See below in "Issues".

d.	There are multiple technical issues that need to be fixed, addressed or discussed. See below in “Issues”.

issues: Technical issues

i.	Section 3 and Figure 1: Can the discussion and presentation here be used to motivate the necessity of the use of ensemble in the solution resolution? How should the discussion and Figure 1 (or experiments therein) be changed?

ii.	Should we also consider offline RL approaches (e.g. model-based) that learn the structure more explicitly before fine-tuning? How would that affect the distribution shift situation? Should we also consider model-based baselines? (No need to actually include such baselines. Just theoretically speaking.)

iii.	How does our analysis (e.g. in Section 3) inform about data collection & selection for the offline phase itself? Should we also compare with approaches that do data collection differently for the offline phase to improve fine-tuning? Note that the injection of uniform random policy rollouts (Line 291) seems to be an example. Should we also consider this in our theoretical treatment at least?

iv.	Line 155: In the design space here, other than d^on / d^off used in the paper, d^on / d^(on union off) also seems a sensible choice that might have the added benefit of trending towards 1 asymptotically.

v.	“Ensemble methods” (paragraph starting on Line 207): it is not clear that “obtain a high-resolution pessimism” is mutually exclusive. I.e. it is not clear “instead” is warranted.

vi.	Line 236-237: Is it justified to apply the same update rules as offline training when using BCQ as a baseline? Why not transfer/fine-tune using e.g. SAC? That seems to be a much more fair comparison. This is so especially in light of Figure 3 in which BCQ is barely learning during online fine-tuning. But BCQ was not designed for fine tuning. To do justice to BCQ, the comparison should presumably be about the quality of policy initialization.

vii.	Line 268: “Interpreting Q(s, a) as the confidence value for classifying real and fake transitions” -- How is this justifiable at all? Are we making a category mistake here? Action value and confidence about real vs. fake are radically different kinds of things.

viii.	Figure 5: “eight runs” and everywhere else is 4 runs. Why do we need to go up to eight runs here? What’s the reason behind this shift from the practice elsewhere in the paper.

ix.	Figure 13 and 14: the main expected effect or phenomenon -- stable and efficient fine-tuning -- seems highly unstable. All methods see big drops during the first 100 episodes. This raises questions about whether the method is going to be truly effective when used with physical robots. Of course, an alternative conclusion here could be that the method is good but that these environments are inadequate for testing the new architecture. It seems that adequate discussion of these curves is missing.

x.	Line 291: “we replace a subset of the original dataset with uniform random policy rollouts.” Is this justified? In real world applications, data are more likely from medium-level policies, less likely from uniform random policies. In fact, it could be the case that in the real world it’s much harder to obtain data with uniform random policy rollouts. That this sort of uniform random rollouts are needed at all seems worrying, especially if offline RL is supposed to use real world data.

Writing issues

i.	Line 83-84 “dataset of random halfcheetah transitions”. The notion of a “random dataset”  should be more appropriately introduced. More detail was provided later around Line 123. But it is more desirable if the notion is explained before it is used.

ii.	Line 131-133: It may be more helpful to say right there and then positively and explicitly that that’s what the balanced replay approach is intended to do. That wrap up Section 3 more adequately.

iii.	Figure 2: Diagram is good. But it is misleading to have labels of the parts of the diagram in shaded boxes. Consider remove the boxes around “Pessimistic Online Training”, “Environment Interaction” and “Balanced Replay”.

iv.	Line 196: the acronym AWAC should be introduced here rather than later on Line 233.

v.	Figure 1 and Figure 5: It will be helpful and make the figures more self-contained if the environments used in the experiments are explicitly identified in the caption.

vi.	Figure 13: should “CQL objective” in the caption be “SAC objective”?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This main contribution of this paper is a balanced replay scheme to balance the offline and online data. It uses an offline policy evaluation method to compute the importance sampling ratio between offline samples and online samples, which is further used to prioritize experience replay of offline samples. The motivation is clear and experimental results are promising. Overall, it's a good paper. 
strengths_and_weaknesses: Strengths:
  (1) The idea of Leveraging offline samples which relate to the current policy to boost online policy evaluation and improvement is reasonable since offline dataset may consist of samples with different styles and quality, some of which may harm the currently learned policy, let alone the policy improvement.
  (2) Extensive experimental results demonstrate the effectiveness of the proposed balanced replay scheme.


Weaknesses:
(1) Though the authors claim that their method can prevent the policy from being destroyed by online OOD samples, we can still observe a severe performance drop at the early period of online interaction in most cases, why?
(2) While pessimism protects the policy from severe bootstrapping error,  it may also prevent the online policy from efficiently exploring the unknown environment (in fact, there have been many works using pessimism for online exploration). In other words, the online policy may be too conservative for online policy improvement. I think it's necessary to provide such ablation studies telling if pessimism is helpful for the ultimate online policy performance
summary_of_recommendation: Overall, this is a good paper. But If the authors could provide with more experiments to clarify the two weaknesses, I will raise my score.
issues: As mentioned in "Strengths and Weaknesses".
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper identifies state-action distribution shift as the major problem in offline-to-online RL, and proposes two mechanisms, balanced replay buffer and pessimistic Q-ensemble to solve it. Through well-organized experiments, the paper shows that the balance experience replay can effectively balance the usage of online data and offline data and the pessimistic Q-ensemble is able to mitigate distributions shift.  The proposed method obtains good empirical results on D4RL Mujoco locomotion tasks and vision-based robotic manipulation tasks.
strengths_and_weaknesses: Strengths: 
1. The motivation of this paper is strong and the proposed method effectively solves the identified state-action distribution shift problem. 
2. The empirical results are good. It outperforms previous methods in most cases. 
3. The paper is well written and easy to follow. 
4. The experiments are well-designed and show the benefits of the proposed methods.

Weaknesses: 
1. In section 3.3, the author compares the choice of CQL and FQE and shows that the CQL is more suitable for offline-to-online RL. But according to the CQL paper, for many offline datasets, directly using non-pessimistic Q-function suffers from distribution shift and can’t learn well during offline training. Thus, I think it will be good to compare CQL with another offline RL method, e.g. BCQ etc. 
2. In section 4.1, the training details are not very clear for me. For example, the author uses P, Q to represent d^{off} and d^{on}, thus the density ratio w(s,a):=d^{on}(s,a)/d^{off}(s,a) = Q/P, but in the last sentence of page 4, the author uses ”estimate the density ratio dP/dQ with a parametric model w_{phi}(x)”, which is inconsistent.

summary_of_recommendation: My recommendation is "Weak Accept".

This paper identifies state-action distribution shift as the major problem in offline-to-online RL, and proposes two mechanisms, balanced replay buffer and pessimistic Q-ensemble to solve it. The motivation of this paper is strong and the proposed method effectively solves the identified state-action distribution shift problem. The proposed method obtains good empirical results on D4RL Mujoco locomotion tasks and vision-based robotic manipulation tasks. But the description of the training details can be more clear. Thus, my recommendation is "Weak Accept".
issues: 1. In section 3.3, the author compares the choice of CQL and FQE and shows that the CQL is more suitable for offline-to-online RL. But according to the CQL paper, for many offline datasets, directly using non-pessimistic Q-function suffers from distribution shift and can’t learn well during offline training. Thus if possible, I think it will be good to compare CQL with another offline RL method, e.g. BCQ etc. 

2. In section 4.1, the training details are not very clear for me. For example, the author uses P, Q to represent d^{off} and d^{on}, thus the density ratio w(s,a):=d^{on}(s,a)/d^{off}(s,a) = Q/P, but in the last sentence of page 4, the author uses ”estimate the density ratio dP/dQ with a parametric model w_{phi}(x)”, which is inconsistent.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents a method for off-line to on-line reinforcement learning. First, given a dataset of behavior in a task, an ensemble of "pessimistic" Q-functions and policies is trained off-line where the Q-function is learned conservatively to avoid over-estimation of Q-values in out of distribution state-action samples. Subsequently, on-line fine-tuning is done using a Balanced Experience Replay. This replay buffer consists of old samples from the off-line learning phase and new samples collected during on-line learning. Transitions are sampled based on their "online-ness", a value proportional to the density ratio between on-line samples and off-line samples. The further traning progresses and moves away from the initial data, the fewer off-line samples are chosen to update the policy.
The method is evaluated on simulated locomotion tasks from open-ai gym, as well as vision based manipulation tasks. It performs especially well in terms of learning speed if the intial policy is random.
strengths_and_weaknesses: Strenghts:
- The overall method seems very intuitive and well motivated
- The efficacy of each component is shown via ablation studies

Weaknesses:
- Especially on halfcheetah, the asymptotic performance seems to stay well below what SAC achieves alone (12k vs 15k). An intuition why this is the case would be interesting.
summary_of_recommendation: First of all, I am not an expert in off-line RL, so I'm not familiar with all the work that is out there on the topic. Most of the parts like CQL, ensembles, and prioritzed replay based on likelihoods is not entirely new but the parts are modified nicely to fit the off-line to on-line RL scheme and put together well and results look promising. Therefore, I would recommend accept but put my confidence score rather low.
issues: - I don't fully get the definition of the behavior policy in line 69. It looks like it is the number of times an action a_0 is taken in state s_0 divided by how often state s_0 occurs in the replay buffer. But, given states and actions are continuous, how often will this be different from 1?
- Why is the performance of half cheetah worse, if the policy is initially trained with the medium dataset?
- Why does the performance using the medium-expert dataset on hopper and walker2d not reach the performance of AWAC and partially BCQ-ft?
- Is there a difference in the entropy of the policy after off-line training with a random, medium or expert data set? For example does the policy not explore enough if trained on an expert data set?
- Why are the manipulation tasks only compared to [10]?
review_title: null

Meta-review: The reviewers generally appreciate the strengths of the paper, with a well-motivated and mostly novel approach to offline RL. I am glad the authors were active during the rebuttal period and were able to address many reviewer concerns. I note that Reviewer Jt9W still has one concern about the method; namely, I advise the authors to add a paragraph or two conceding and elaborating on potential shortcomings of their proposed approach, including the potential instability issues.
Decision: Accept (Poster)