Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 3: Clear rejection
review_content: Summary
-------

Offline RL allows agents to be trained on static, offline datasets. If
the agent has already undergone offline training however, distribution
shift makes further online training difficult. The authors propose to
address this problem by keeping separate offline and online replay
buffers, and sampling different proportions of each. In addition, they
propose to ensemble a distillation policy. The proposed method is
evaluated on logged MuJoCo datasets answering whether BRED improves over
other fine-tuning methods and whether balancing or replay contribute
more to BRED's success.

Decision
--------

BRED is an interesting approach, but the experiments do not demonstrate
that it is better than the baseline. As such, my preliminary rating of
this paper is rejection. I appreciate that the sampling scheme proposed
is simple and well motivated to address the problem of distribution
shift. It is not clear whether the sampling scheme actually addresses
the problem of distribution shift however, as it gives the agent access
to additional experience. The distillation scheme also does not seem to
address distribution shift directly and the experiments do not
demonstrate that it helps performance outside of the Walker2d-random
task. Moreover, it is not clear whether the baselines are also
ensembles, which would make the experiment results less compelling.

Originality
-----------

The contributions are: a sampling scheme combining offline and online
samples, as well as a distillation scheme. The sampling scheme is
incremental and simple, but seems like a novel approach. Distillation on
the other hand, is well-explored in RL (Czarnecki et al. 2019) and does
not seem to contribute much to the problem addressed in the paper. Taken
together, the paper has limited novelty.

Quality and Clarity
-------------------

The paper is easy-to-follow and well-written. However, I have some
issues with the title. The title suggests that your aim is to address
distribution shift in online RL. However, distribution shift is a
problem due to offline training. Of course, the problem you are
addressing is further online training after offline training. Perhaps
the title should be reworded to make this more clear.

Strengths
---------

-   There is a clear exposition of the problem addressed, namely
    distribution shift. This is well motivated by the t-SNE
    visualization. Section 3 in particular highlights the correlation
    between distribution shift and the performance of offline RL
    algorithms.
-   BRED, the proposed method, is a straightforward solution via
    balancing offline and online replay buffers. In addition, the
    schedule for updating the proportion sampled of each replay buffer
    is well argued and also simple.
-   The results are clearly motivated by questions in the beginning of
    the section. Sensible baselines are used to compare the proposed
    method (BRED), and these are discussed and reasoned in detail.
    Furthermore, there is a good coverage of tasks (3 MuJoCo) and
    difficulties (random, medium, medium-replay, medium-expert).

Weaknesses
----------

-   While fine-tuning an offline RL agent can pose challenges, similar
    challenges can arise in supervised learning. For example, Ash and
    Adams (2019) note that in different regimes, warm-starting neural
    network training can hamper generalization. Overall, I feel that the
    correlation between distribution shift and online performance after
    offline training is not shown to be causal, but correlative.

-   It is unclear why distillation is included, as it does not directly
    address the problem of distribution shift in offline RL, outside of
    the intuition that it mitigates bad $Q$ estimates. This is not
    enough motivation however, and the experiment results (Figure 5
    and 6) does not show any significant benefit of ensemble
    distillation over independent ensembles (besides Walker2d-random).
    Furthermore, this may not be a fair comparison to other baseline
    methods which do not seem to be ensembles.

-   Despite a very well-motivated experiment section, the experimental
    conclusions are far too strong. The results do not back this up,
    when there is high overlap in the confidence intervals for many
    Figures (in each section). Furthermore, there are unexplained
    peculiarities in the results, such as very big drops in performance
    or the fact that the agent trained with the offline dataset is
    substantially better than the medium one.

Detailed Comments
-----------------

-   "pointed out that offline RL algorithms… are not amenable to
    fine-tuning, due to the difficulty of modeling the
    dataset-generating policy in the online setup."

    This is not clear to me, how does fine-tuning connect to modelling
    the dataset-generating policy? Furthermore, how does the
    dataset-generating policy (I assume this is the behavior policy)
    connect to online RL?

-   Figure 1b: the agent is trained on the offline dataset and then that
    dataset is thrown away for the online agent, whereas the agent in
    red gets to keep using it. This does show the difficulty of
    fine-tuning but does not provide conclusive evidence that
    distributional shift is the cause. Fine-tuning can be difficult even
    in supervised learning (Ash and Adams 2019).

-   "On the other hand, when using online samples exclusively, the agent
    is exposed to unseen samples only, for which Q function does not
    provide a reliable value estimate. This may lead to bootstrapping
    error, and hence a dip in performance as seen in Figure 1b."

    Can't this be shown experimentally in an environment to provide
    conclusive evidence for distribution shift being the cause of the
    dip in performance?

-   Figure 3: It seems that halfcheetah-random results in a better
    policy than halfcheetah-medium. This does not make sense to me. In
    addition, there is a strange dip in performance for CQL-ft in Figure
    3g. Many of the figures have overlapping confidence intervals
    (although the shaded region is only the standard deviation, the
    corresponding confidence interval would be approximately the
    standard deviation. Anyway these should be confidence intervals
    rather than standard deviation). While BRED does perform
    statistically significantly better in some tasks, I don't think the
    comparison is fair due to ensembling used only for BRED.

-   Figure 4: Going from figure 4a to 4b, balanced replay is shown to be
    more beneficial for a dataset generated by a random behavior policy.
    This seems to suggest that the main benefit of BRED is artificial
    exploration. Counter-intuitvely, this trend is reversed in Figure
    4c. Moreover, the online-only curve is very different in Figure 4c
    than 4a and 4b, and to my understanding there should not be any
    difference. Due to the noisiness of the curves, its hard to make any
    straightforward conclusion from these results.

-   Figure 5 and 6: Again, there is no statistical difference in medium
    and medium-replay between independent ensembles and a distillation
    ensemble.

Minor Comments
--------------

-   Equation 3: missing brackets on the left side.


Ash, Jordan T., and Ryan P. Adams. 2019. “On Warm-Starting Neural Network Training.” *arXiv:1910.08475*.
<http://arxiv.org/abs/1910.08475v2>.

Czarnecki, Wojciech Marian, Razvan Pascanu, Simon Osindero, Siddhant M. Jayakumar, Grzegorz Swirszcz, and Max Jaderberg. 2019. “Distilling Policy Distillation.” *arXiv:1902.02186*.
<http://arxiv.org/abs/1902.02186v1>.
review_title: Inconclusive empirical evidence for BRED's performance and whether it directly addresses distribution shift

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary: 
This paper proposes to deal with distribution shift problem between online and offline samples when the agent trained by offline data is fine-tuned with online interactions. Two mechanisms are introduced: (1) using two replay buffers for offline and online data respectively, and training the agent with data sampled from these two buffers with a certain ratio (the ratio changes in a way that more online data is used in later epochs); (2) learning an ensemble of independent agents in the offline phase, and distilling them into a mean policy to overcome bootstrapping error. Empirical results demonstrate that the proposed method perform well during fine-tuning when there is distribution shift.
 

Strengths:
1. The paper is overall well-written and easy to follow.
2. The problem studied by this paper is well-motivated, and the proposed methods are simple yet effective, making intuitive sense.
3. The experiment section shows that the proposed method (BRED) outperforms baselines on 3 mujoco tasks with multiple types of offline data. Further ablation study verifies the effectiveness of the two proposed mechanisms.

 
Weakness: 
1. Although empirical results are good, my main concern is that the novelty of this paper is limited. The main contribution of this paper is combining CQL with two tricks: balanced replay and ensemble distillation. As already discussed in the related work section, these two tricks are commonly used in the literature, even though the concrete setting varies.
2. The specific way of combining the offline buffer and the online buffer (online fraction grows linearly in timestep) could be effective in some cases, but might not always be the best choice. I am curious whether the authors have tried other ways. And especially, can the agent chooses the ratio adaptively based on the degree of distribution shift and the learning performance?
3. The definition of distribution shift in this paper is descriptive and informal. It will be nice to provide deeper analysis or insights w.r.t. the distribution shift problem. For example, is there a way to rigorously characterize the distribution shift? Is there a measurement for how large the gap is? This will be relevant to addressing point 2 above. 
4. Although policy ensemble can make the learned Q-function more accurate and stable, it is not very clear to me why it tackles distribution shift. Even the distilled policy has a more accurate estimation of the Q values, the estimation is still w.r.t. the offline data distribution. It still suffers from the bootstrapping error when using out-of-distribution samples. 

 
Minor comments: 
1. Figure 1 is a good visualization. However, the figure is not very informative since details are not provided (e.g., what is contained in a sample, in what sense / which dimension the distribution is different). 
2. In the experiment, I am wondering how an “ensemble-only” method would work and compare to baselines, i.e., use multiple pertained CQL agents, and fine-tune the distilled policy. The “Effects of ensemble distillation” paragraph shows a comparison between the proposed ensemble distillation and the ensemble of independent policies. However, the result does not reflect how ensemble compares with non-ensemble.
3. The paper focuses on the distribution shift problem in offline RL. However, there could be multiple interpretations of distribution shift, e.g., state distribution changes due to the change of behavior policy, or dynamics distribution changes due to the slight change of the environment, etc. I think this paper is mainly dealing with the state distribution change case. But it will be better if the authors can make it explicit.

review_title: The idea is good, but more work is needed.

final_score: rating: 4: Ok but not good enough - rejection
review_content: ##### Summary
The paper proposes a fine-tuning method for an offline RL algorithm, CQL. The method incorporates a balanced replay scheme for both online and offline samples, and an ensemble distillation to stabilize policy learning. The proposed method is evaluated on benchmark environments. 

The paper is clearly written and easy to read. However, the problem setting in the paper is not well-motivated and the goal is unclear. The paper makes some claims without supporting them (details come later). Moreover, some important experiments details are missing which makes it hard to assess the empirical results. I think all of these issues needs to be addressed before publication.  Therefore, I recommend to reject the paper.

##### Supporting arguments and clarification questions
First of all, I think it is not motivated why online fine-tuning procedure is necessary (see first paragraph in the introduction). If online simulation/interaction with the real environment is practical, why do we need to train an offline RL agent first and perform online fine-tuning (given that the paper claims that fine-tuning is challenging)? If online simulation is not practical, how could we perform online fine-tuning? 

It is also unclear to me what is the goal we want the proposed method to achieve  (e.g., what are the evaluation metrics)? Do we want the fine-tunning method achieve the best sample efficiency compared to purely online algorithm, or achieve the best asymptotic performance compared to other fine-tuning methods? Without a clear goal, it is hard to assess the significance or soundness of the proposed method.  

In section 3, what is the reason for fine-tuning a CQL agent by SAC updates? Have you tried fine-tuning by DQN updates or its variant? I think SAC is learning a value function with an extra entropy term and CQL is learning the true value function (with some regularization during the optimization procedure), so these algorithm are essentially learning two different targets. It makes sense that we would see instability during fine-tuning since they are optimizing towards different targets. Therefore, I don’t think the clam “this instability occurs due to the shift between offline and online data distribution” is supported in the experiment. It might be due to the fact that we are optimizing different objectives. 

In the experiment, how were the hyper-parameters selected? The hyper-parameters should be selected based on the offline dataset, not the fine-tuning performance. I think the paper should also include a baseline, which is the performance achieved the offline RL agents (i.e., horizontal lines in the learning curve). In Figure 3, it seems like BRED is much more stable compared to other methods in Hopper medium, so I wonder if the reason is that BRED already learns a good policy with offline dataset and it does not change much during the fine-tuning procedures. 

In section 5, the paper provides a nice discussion on the related works. However, the paper claims that “this method (Nair et al., 2020) relies on regression, hence the learned policy seldom surpasses the best data-generating policy”. Can you elaborate more on this?

review_title: Nice paper but some issues need to be addressed

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper considers the problem of policy learning in Markov Decision Process (MDP) from the combination of online and offline samples. The offline samples are generated by a behavior policy in the same MDP model, i.e., the behavior agent and the learning agent share the same state-action space. The learning procedure goes as follows. One first trains a MDP policy from the offline data; the online samples are then used to fine-tune the learned policy.

The authors propose a simple yet effective approach. First, the authors keep separate offline and online replay buffers, and carefully balance the number of samples from each buffer during updates. Then,  multiple actor-critic offline RL policies are trained, and a single policy is distilled from these policies using ensemble methods. Experiment results show that the proposed method consistently outperforms state-of-art algorithms.

This paper is clearly written and well organized. I am not sure about the novelty of the proposed method, since it seems to follow the line of carefully reweight online and offline samples. However, the experimental results show a significant improvement over existing methods.

Question for the authors:
1. In practice, what is a good heuristic for selecting the initial fraction p0 of online samples? How sensitive is the learned policy w.r.t. the initial fraction p0?
review_title: An interesting work

Meta-review: This paper addresses an distribution shift and biased Q-values that happens when offline agents are finetuned in an online manner. The final revision of the paper is very well written and easy to understand. The proposed method in the paper is interesting, and aiming to address an important issue in RL. The proposed method involves a combination of two well-known methods in RL to tackle the distribution shift issue, the paper first suggests to use a balanced replay mechanism a replay for online experiences and another one for the offline. The second improvement is coming from the ensemble distillation.

It seems like in the light of the reviews, the authors have improved manuscript. However, I would like to recommend the paper for rejection. I would like the authors to do further experiments on the individual components of the algorithms, for example what if we run all the experiments only with BR or only using ED how would the performance change. How much improvement is coming from each one of those individual components? As it stands, it is not clear to me right now, and the proposed solution looks a bit complicated and hacky. 

The balanced replay mechanism is very similar to the replay approaches that are used for learning from demos methods like R2D3 [1] and DQfD. Also the ensemble distillation approach is very akin to RAND [2] and distillation approaches that are used in lifelong learning algorithms. It is not clear, why it is that important for offline RL. It should potentially improve online RL as well, perhaps some experiments on online RL would be interesting.

Nevertheless, I think the paper is very interesting and attempting to address a very important problem in RL. I would recommend the authors to resubmit the paper to a different venue after doing some small changes on it.

[1] Gulcehre, C., Le Paine, T., Shahriari, B., Denil, M., Hoffman, M., Soyer, H., ... & Barth-Maron, G. (2019, September). Making Efficient Use of Demonstrations to Solve Hard Exploration Problems. In International Conference on Learning Representations.

[2] Burda, Y., Edwards, H., Storkey, A., & Klimov, O. (2018). Exploration by random network distillation. arXiv preprint arXiv:1810.12894.




Decision: Reject