Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an update rule to adaptively change the regularizer term (BC here) weight during online learning for more stable fine tuning to improve over an expert.
main_review: Pros:

** This paper considers an important problem setting for RL.

** Paper is well written and is easy to follow.

Cons:

** The proposed solution is heuristic and is similar to other methods that use annealing to tune the regularizer weight. And similar to other heuristics in this space, this method also has this limitation that it requires pre knowledge about the final performance of the task. I would like to see a more principled solution that at-least does not require such knowledge. 

** Evaluations are done on simple low dimensional tasks. Given the limited technical contribution, i would like to see a study on more complicated tasks with for example visual inputs where representation learning introduce additional challenges for finetuning.

** The other claimed contribution on using ensemble of Q functions is orthogonal to the problem under study as it has shown before that ensemble of Q-functions helps to stabilize learning. 
summary_of_the_review: This paper considers an important topic for RL. However the proposed solution is heuristic and evaluations are not convincing given the simplicity of the tasks. Overall i think this paper does not offer enough contribution in its current form.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new offline RL with online fine-tuning method. The authors first pretrain the policy using recent offline RL method TD3+BC with offline data and then collect on-policy data to further improve the pretrained policy. To prevent the policy from either degrading performance or failing to improve at the fine-tuning stage, the authors propose an automatic scheme to adjust the $\alpha$ term that controls the BC loss in TD3+BC to ensure that the policy does not continue to contain itself to the behavior policy if there's room for improvement and also is able to stick to the previous policy if the performance is already near optimal. The authors conduct evalutions in D4RL mujoco environments and show that the approach is able to outperform prior methods in halfcheetah.
main_review: I think the paper is clearly written and easy to understand. The proposed algorithm is also novel in the sense that it gives an adaptive scheme of selecting the amount of conservatism during online training depending on the current policy performance. The empirical results also clearly show that the method can prevent the policy from degrading in medium-expert datasets where we don't need to change the current policy much, aligning with the initial motivation of the method.

I also have a few concerns. First, while the results of the method generally align with the intuition that the policy should use large alpha on narrow datasets with good data and use large alpha on diverse datasets with low-quality data, the performance of the method on hopper and walker2d is a bit underwhelming. The method seems to be the same as hopper and also worse than Balanced Replay on three of the walker datasets. While the authors mentioned that the less satisfying results on walker2d are caused by the predefined target score in D4RL, I wonder if that is potentially a limitation of the method since it relies on the human-defined target score. Also, how would the method do if we change the target score to be higher (e.g. R_max * T)?

Moreover, I think the paper lacks some theoretical understanding of the method. The intuition of the method makes perfect sense, but I would like to see how this adaptive scheme can give use some theoretical insights on policy improvement guarantees. Prior works [1] have studied this and I wonder how the proposed method can be connected to that.

Finally, beyond the mujoco tasks, I think it would be good to evaluate the method on more realistic tasks such as the dexterous manipulation tasks used in the AWAC paper. 

Minor comment: it would be great to have an ablation study on the REDQ ensemble since that seems to be a bit disjoint to the theme of the paper.

[1] Xie, Tengyang, et al. "Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning." arXiv preprint arXiv:2106.04895 (2021).
summary_of_the_review: Given my comments in the above section, I think the paper could be improved if the authors can show the results of the method with a more realistic target score (e.g. R_max * T), provide the theoretical justifications, and evaluate the method on more realistic domains. I would vote for a weak reject given the current status.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper studies the fine-tuning problem from offline to online RL. While the naive approach to fine-tune offline policy suffers from a sudden distributional shift by online samples and too much behavior constraint in offline algorithms. The proposed method leverage (1) the adaptive coefficient tuning in TD3+BC loss, (2) randomized ensembles of Q-functions (proposed by Chen et al. 2021), and (3) down-sampling of offline data. The experiments seem to show better or competitive results to other approaches.
main_review: ## Strengths
- This paper is well-written and easy to follow.
- The focus of this paper, fine-tuning in RL seems to be a significant topic towards real-world applications, overcoming the bottleneck of current offline RL research.
- The empirical comparisons seem to show better or competitive performance to AWAC, Balanced Replay, TD3_ft, and REDQ.

## Weaknesses
- The proposed method REDQ+AdaptiveBC seems to require a lot of hyper-parameter tuning. First, $\alpha_{offline}=0.4$ is a very small value compared to original TD3+BC, which uses $\alpha_{offline}=2.5$ after a grid search from \{1.0, 2.0, 2.5, 3.0, 4.0\}. This implies that $\alpha_{offline}=0.4$ (and 100 for walker2d-random) might be a result of extensive hyper-parameter tuning. In addition, I guess $K_p$, $K_d$ (and even target return) must be well-balanced. I'm not sure how they are determined.
- Also, REDQ+AdaptiveBC seems incremental on top of TD3+BC, combining the established Q-ensemble method (Chen et al. 2021) and carefully-designed coefficient tuning, and resembles Balanced Replay (Lee et al. 2021). I summarized the problems in fine-tuning and the remedies for them on the table. Both Balanced Replay and REDQ+AdaptiveBC overcome the data distribution shift issues using the ensembles of Q-functions. For imbalance issues of the offline-online replay buffer, Balanced Replay utilizes  Prioritized Experience Replay, and REDQ +AdaptiveBC downsamples the offline data, both of which aim to reduce the ratio of offline data and encourage to leverage the online samples when online fine-turning. During the online training phase, behavior regularization for stable offline training can prevent the improvement of the policy. Balanced Replay, proposed on top of CQL, removes the CQL loss and train value function and policy as online SAC, and REDQ+AdaptiveBC gradually reduce the ratio of BC loss. I think the novelty and contribution of this paper are not enough.

|Problem|Balanced Replay (Lee et al. 2021) | REDQ+AdaptiveBC  |
| ---- | ---- | ---- |
|  **Distribution Shift**  |  Pessimistic Q-Ensemble  |  REDQ (Chen et al. 2021)  |
|  **Imbalanced Replay Buffer** |  PER  |  Downsampling  |
|  **Behavior Regularization**  | CQL-SAC switching |  Adaptive $\alpha_{online}$  |

- I think the comparison might be insufficient, since Balanced Replay, proposed on top of CQL, is based on TD3+BC in this paper. I don't think the comparison to CQL-based Balanced Replay isn't fair. In addition, while adaptive BC heavily depends on TD3+BC algorithms, Balanced Replay seems to work on both CQL and TD3+BC. I also think the authors can add TD3+BC baseline (keeping $\alpha_{offline}$ during online training) and +REDQ/-REDQ ablations.

### Minor Comments
- I'm not sure how can I interpret the results of Figure 4. In my understanding, downsampling of replay buffer is introduced to overcome the instability of the beginning of the online training due to the imbalanced replay buffer (as discussed in Lee et al. 2021), but the caption says "Downsampling does not hurt the training". Doesn't downsampling improve the performance?
- You can consider adding Ghasemipour et al. (2021), that proposes the ensemble-based offline RL algorithms, to the related work.
- Prior works of offline-to-online fine-tuning (AWAC, Balanced Replay) work on robotic environments. Comparison on these environments, not only current MuJoCo locomotion, might be helpful.


### Reference
Fujimoto and Gu. A Minimalist Approach to Offline Reinforcement Learning.  ArXiv preprint arXiv:2106.06860 (2021).

Lee et al. Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble. Conference on Robot Learning (2021).

Chen et al. Randomized Ensembled Double Q-Learning: Learning Fast Without a Model. International Conference on Learning Representations (2021).

Ghasemipour et al. EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL. International Conference on Machine Learning (2021).

summary_of_the_review: As I described in Main Review, this paper seems to require extensive hyper-parameter search (proper $\alpha_{offline}$ and well-balanced $K_p$, $K_d$), and to propose the incremental algorithms on top of TD3+BC (and resemble Lee et al. 2021), combining Q-ensemble method proposed by Chen et al. (2021), downsampling for imbalanced replay buffer, and adaptive $\alpha_{online}$ tuning. Due to the lack of novelty and significant contributions to the ICLR (or RL) community, I vote for rejection.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors combine REDQ and TD3+BC and propose an adaptive mechanism to update $\alpha$ during the online fine-tuning. The proposed method decreases $\alpha$ for further improvement and increases  $\alpha$  to avoid performance collapse.
main_review: The paper is well organized and the motivation is really clear. Offline-to-online learning considered in this paper is an important problem. I agree that adaptively updating the behavior policy constraint is the key point in online fine-tuning. In the experiments, the authors compare the two SOTA offline-to-online methods AWAC and Balanced Replay, and ablation studies show the effectiveness of adaptive $\alpha$.

My main concern is the novelty. REDQ and TD3+BC are prior work, and it seems that the main contribution in this paper is the adaptive mechanism of $\alpha$. Although this paper tries to solve an important problem, the contribution of the proposed method is limited.

The update rule of $\alpha$ is heuristic. Although it is well-motivated, more theoretical analysis is expected to demonstrate how the adaptive  $\alpha$ achieves the balance between performance improvement and performance dropping. And the selection of hyperparameters KP and KD is arbitrary. More empirical results are needed to show how the two hyperparameters affect the update of $\alpha$​. In Figure 3, AdaptiveBC does not significantly outperform ManualBC. Although ManualBC uses grid search, the search space is not too large. 

In Figure 2, if no expert data is involved, REDQ (scratch) could achieve a good performance in 250000 timesteps, which shows pure online learning is enough to deal with this setting. In that case, it is important to consider the sample efficiency and deployment efficiency[1] in offline-to-online learning.

[1]https://arxiv.org/abs/2006.03647
summary_of_the_review: The paper aims to solve an important problem: adaptive behavior policy constraint in offline-to-online learning. However, the proposed method is heuristic, and more theoretical analysis is expected.
review_title: null

Meta-review: The paper proposes an approach that allows online finetuning of an offline RL policy by adaptively changing a BC regularization term.

Even after discussions with the authors, the reviewers had several concerns. First, the paper seems to be limited in novelty as the "REDQ+AdaptiveBC seems incremental on top of TD3+BC". Second, there were concerns that the adaptive regularization term was insufficient as a contribution given its heuristic nature.

Given the consensus among reviewers of this paper, I recommend rejecting this paper.
Decision: Reject