Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel alignment step for actor-critic RL approaches from offline to online settings. The paper relies on the observations that the Q functions OOD are much more untrustworthy than the entropy regularized policies learned from the offline data. They propose to disregard Q-values learned offline to combat overestimation under distribution shift and rather propose a procedure to rebuild these Q functions from the policy and a state-only value estimate during the online fine-tuning phase. They demonstrate little to no drop in performance when offline to online transfer happens on a range of controlled simulated tasks. 
strength_and_weaknesses: Strengths

* The approach to disregard learned Q values during the alignment phase and aligning them into the warmstart policy and state-only function is an interesting idea for transfer from offline to offline settings.
* The paper does some good ablations (eg Figure 1 and Figure 4) where they first justify the alignment procedure and later verify that BC regularization alone does not guarantee offline -> online stability, both of which are very valuable observations.
* The approach does not require any of the offline data during online phase, which can be suitable for applications that are covered by data privacy considerations such as healthcare.
* The results are strong and consistent over the environments taken into consideration while all other baselines fail to transfer well in some cases or the other.


Areas to improve/ Clarifications and Questions

* Should we also consider offline RL approaches such as the model-based ones which learn the dynamics structure before the fine-tuning phase for a fair comparison to how they respond to OOD actions? Running full-scale experiments might be a costly exercise, but I still feel additional experiments or some discussion around this aspect can advise which approaches to embrace for offline-online RL.
* Transferring Q values to policies might sometimes swing the pendulum to the other side from over-optimistic estimates to pessimistic estimates in the policy, which may prevent exploration in the online fine-tuning case. Prior works (eg [2]) have alluded to this phenomenon. It might be worth investigating whether such a phenomenon affects the model from learning more optimal policies in certain environments where it underperforms other baselines.
* (please correct me if i misunderstand) The alignment procedure works only when the Q function estimates the values of the behavior policy, but I am not sure whether this transfer is possible for methods that do perform multi-step DP on the value functions such as Implicit Q Learning [1]. It might be worth having a discussion about this specific aspect in the paper and see how this transfer might work for those value functions.  
* The paper uses the last checkpoint during offline training as the initialization for online training. This might not always be the best offline trained model as it does not account for overfitting and thus, exacerbating the offline -> online OOD effect even more. Since this is a study in controlled environments, policy selection could still be done by evaluating in the environment and picking the best possible policy to warmstart. This can expose whether overfitting might be a strong reason for the models to suffer in terms of generalizing to the online environment.  
* Minor : The paper could have benefitted from demonstrating transfer from offline to online in more complicated real-life environments plagued by distribution shift, such as robotics where sample collection is costly. This would have made the impact of results even stronger.
* Minor:  For baselines in transfer such as SAC -> SAC, while distribution shift is a problem, an interesting ablation would be to constrain the policies from diverging too much during the online training at each training step such as adding KL penalties for policy divergence, for instance. This would further explain whether we need this alignment step as compared to controlling policy divergence in the online phase from the bootstrapped policy.

Minor comments

Figure 2 might be useful to point users to table 1 to understand the legends better.

Page 2 - Our approach does not “reply” on should have rather been rely on

Exposition in figure 1 is not clear. Kindly amend the description to better demonstrate how the alignment is benefitting over the baseline.

Figure 4 can be made more self-contained as ablation 1 and ablation 2 are not really clear anyone trying to parse the figures.

References
​​1. Implicit Q Learning – https://arxiv.org/pdf/2110.06169.pdf

2. https://arxiv.org/abs/2002.12174


clarity,_quality,_novelty_and_reproducibility: * The paper is well-written barring a few interpretation difficulties illustrated in the earlier sections.

* There are two key original ideas in the work -- disregarding Q values and aligning them with policies learned offline, reconstructing Q values from the policy and state-only values during the online fine-tuning phase.

* The authors have also shared fair amount of details about their experiment settings and code, thus encouraging reproducibility of the experiments which were run in environments that are extensively studied in literature. 
summary_of_the_review: While the results are not significantly better than a strong competitive baseline CQL + BR, the ideas of aligning Q values from offline to online phase and reconstructing Q values for online finetuning are fairly interesting. The experiments have been run on limited  environments and not in large domains where the performance shift arising out of the distribution shift can be very evident. The paper has done a fair job of running ablations and demonstrating which components of the alignment step are important for the transfer. While there are certain clarifications highlighted earlier, in the interest of the idea, I am recommending to marginally accept the paper and hope that the authors do address the comments in subsequent iterations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a new approach for improving the transition between offline and online training of policies via RL. In actor-critic algorithms, the source of errors in the online phase comes from extrapolation errors in the pretrained Q-function. In the online phase, when the pretrained policy gets into states that are not in the offline dataset, the Q-function can overestimate the values of poor actions and then the policy optimization in turn deteriorates the policy. The proposal in this paper is to fix this by throwing away the trained Q-function and effectively creating a new one that is "aligned" with the policy in the sense that the Q-values are fit to the log-policy. This is also somewhat theoretically justified by the common interpretation of policies being Boltzmann distributions with the Q-function as the energy.
strength_and_weaknesses: Strengths:
- A nice and simple idea.
- Attacks an important and well motivated.
- The first third of the paper is quite clearly written.

Weaknesses:
- Clarity could be improved in parts (e.g. beta-clipping, discounts).
- Figure 1 could be made more rigorous.
- Experiments are thorough but the plots are hard to read.
- Benefit of the approach is not clear empirically.
  * Improvements are only clear in the expert offline data regime.
- The additional implementational details in A.2 and Eq. 8 seem like a confounder. Are these used for all baselines as well?
clarity,_quality,_novelty_and_reproducibility: Clarity:
- Well-written text.
- Mathematical expressions are quite clear but provide some details that are superfluous, e.g.:
  * You could safely remove all occurrences of the environment discount d.
  * The detail of clipped double Q and Z is somewhat unnecessary and distracting. Could probably be stated and the exact update given in the appendix.
- Figure 4 has ablations that only make sense when looking at Appendix E, which isn't ideal.

Quality:
- Experiments in the paper are quite thorough.
- I like Figure 1 in principle, but it's unclear whether the top and bottom row are for a single sample state each, in which case it becomes anecdotal evidence. Could the authors find a way to produce a similar plot averaged over all states in/out of the offline dataset?
- From Figures 2 and 3, it's difficult to tell whether the method is helping much except for the datasets which include expert data.
- The tricks in Appendix A.2 seem quite significant and are hopefully used in the baselines as well.

Novelty: I haven't seen such an alignment step used to correct the Q-value extrapolation issue in offline pretraining. It's a neat idea.

Reproducibility: Given that the offline datasets and the authors' code is opensourced, I'm adequately satisfied regarding the reproducibility of this work.
summary_of_the_review: Overall I think this paper introduces a neat and simple idea for offline-to-online RL. It's an area of great significance for real world RL applications. Unfortunately, the exposition is slightly lacking, particularly empirically. It seems the benefits of the method are really only significant when the dataset has a lot of expert trajectories. Given the alignment nature of the idea, I worry that this is a built-in weakness [1] and I'm not yet convinced by the experimental results.

I was an emergency reviewer so I will drop my confidence score to reflect that.

[1] The authors even state: "the Q-function is generally problematic for out-of-distribution actions while the policy learned offline is assumed trustworthy."
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a way to transfer (or align ) a trained method from offline learning to online learning. This method has three steps: (modified) offline learning, actor-critic alignment, and online training. In  modified  offline learning, they use a modified version of TD3-BC where they use soft-actor-critic instead  of TD3 and add behavior cloning term. During the alignment step, they fit the "baseline", and finally during online-training, they changed the Q function with their "baseline". To show the effectiveness of their method, they use a subset of D4RL benchmark (i.e HalfCheetah-*, Hopper-*, Walker2d-*) and they get interesting results.
strength_and_weaknesses: Strength:

- Utilizing a trained model in the offline-rl and fine-tuned during online learning is an interesting direction to consider.
- Some of the results are promising. 


Weaknesses:

- Although authors claim their method "outperforms or matches the current SOTAs", I disagree with this statement. Take table 2 and Figure 2 as examples. In table 2, CQL + BR has pretty much the same performance as SAC + ACA ( proposed method). Although standard  deviation is not reported in these experiments, it is hard to say if these results are significant. In addition, if we look at Figure 2, green lines are only better in 5 to 6 cases out of 15 cases. Considering the strong performance of [1], why should someone select this method over [1]? What is the advantage of this method vs [1]? discarding Q-function by itself is not an advantage until it shows up in the results.

- This method has lots of moving parts and it would be potentially hard to make if this method applies to new environments.  Consider training settings for each step of this method.

- It is hard to follow the writing and understand the motivation for some choices. For instance, $Z$ is a function of only states but during the final step of this algorithm, it will be used to initialize R(s, a). Since R(s,a) is a function of both s and a, why does this initialization make sense? Also, it is not clear the motivation behind  $Z$. Can you explain?  It seems to me $Z$ behaves like a value function normalized by policy. But it is hard to tell. Also, can you explain how you derive equation 5 and motivation beyond that? Finally what happens if we ignore the first two steps of your method and just initialize R(s,a) randomly? How does performance change? 

- In my view, the issue of distribution shift is overlooked in this paper and there are no good cases for that in this work. Examples of distribution shift can be change of state distribution during online training, changing reward, changing transition functions, etc. 

- Why were only these environments selected? Are there reasons for not running experiments using Adroit, Maze environment form D4RL benchmarks? 

Other comments:
- Compounding error is usually the problem in model-based RL. Can you expand and discuss how it is related to offline RL?

- Extrapolation error is mainly the source of the issues in offline RL. However, this paper implies that it is a distribution shift. Since not all extrapolations are caused by distribution shift and vice versa. Can you explain how these two are related?


[1] Seunghyun Lee, Younggyo Seo, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offline-to-online reinforcement learning via balanced replay and pessimistic q-ensemble.
clarity,_quality,_novelty_and_reproducibility: The paper is not clearly written and not everything motivated very well. 
summary_of_the_review: Offline to online alignment/transfer in RL is an important and interesting area and this paper looks at the right problem. However, the current paper has many issues and I don't think it meets ICLR acceptance bar. In addition, the paper doesn't experiment in which distribution shift really happens, this is important as this paper motivates the issue of offline to online transfer. Finally, it requires significant revision as it is hard to follow.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel offline-to-online RL method.
The proposed method introduces an offline critic re-training phase to explicitly align the critic with the offline-trained policy.
In the critic re-training phase, a refined critic parameterization deducted from SAC policy objective prevents value overestimation.
Experimental evaluation shows its comparative fine-tuning sample efficiency compared to SOTA offline-to-online RL methods.
strength_and_weaknesses: ### Strength
* The proposed method can achieve comparable online convergence with SOTA baselines without its access to the offline data or importance sampling strategy.

### Weakness
1. Unclear parts in experiment results.
    - Some of the CQL offline training curves in Figure 2 seem to be not converged yet. (HC medium-expert, HC expert, and H expert, following the notation in table 2). It could have affected the initial stability and also the final score after a fixed amount of online training, which makes the quantitative comparison in Table 2 unreliable.
    - In Figure 1 result, SAC+ML is not even aligned with the in-distribution sample which is considered to be from the bias introduced by ML term. (or possibly from not converged Q which is also problematic.) Thus, this qualitative result is showing the misalignment not only by distributional shift but also by biased objective. Consequently, the given result is not enough to illustrate the existence of the misalignment problem of prior offline RL methods. 

2. Ambiguous term - misalignment
    - Despite the "actor-critic misalignment" being the key problem that this paper aims to solve, it is neither formally defined nor kindly explained.
    - Moreover, as described in weakness-1, the qualitative result is not clearly showing whether the described problem really appears in other methods.

3. IQL reports better benchmark results than AWAC even for online fine-tuning, thus more proper baseline for AWR-based offline RL methods. I think this comparison is also important to show whether AWR policy objective is also affected by the actor-critic misalignment. But the current bad results of AWAC seem to be from offline RL performance rather than the actor-critic mismatch.
clarity,_quality,_novelty_and_reproducibility: ### Clarity, Quality
Most of the part is clearly written and understandable, but there are a few unclear parts (described in the prior section).

### Novelty
The proposed method is novel.

### Reproducibility
Implementation code and a complete set of hyper-parameters are provided for reproducing.
summary_of_the_review: The proposed method and empirical support are quite convincing.
ACA can replace the SOTA offline-to-online RL method even without access to offline data.
However, as described in the weaknesses, some important points related to the motivation of the idea are not clearly shown.
Also, the quantitative comparison possibly includes unfair comparisons for several tasks.
Thus, I vote for rejecting this paper until those concerns are resolved.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new offline-to-online RL method that proposes to use SAC as the backbone RL algorithm and directly aligns the policy with the critic function at the end of the offline training phase and use the newly aligned critic as an initialization of the critic function during online fine-tuning. The authors show that the proposed approach can achieve reasonable performance on D4RL-v2 tasks.
strength_and_weaknesses: Strength:

1. The paper tackles offline-to-online finetuning, which is an important problem to study in RL.

2. The empirical results suggest that proposed method is able prevent dip at the start of the fine-tuning, which is promising.

Weaknesses:

1. I'm confused by the fact that the authors need to do explicit alignment at the end of offline training as the added BC regularization should be able to handle distributional shift of Q-values on out-of-distribution actions. It is unclearly why the authors need to do this additional step. It appears to me that this step is rather ad-hoc.

2. The authors also didn't compare to more recent offline-to-online methods such as IQL. It is important to add such a comparison to show the benefits of the method. It also appears that the improvement over other approaches is rather marginal, which is not conclusive enough to show the benefits of the method.
clarity,_quality,_novelty_and_reproducibility: I think the paper is not clear enough to show the necessity of algorithm design choice. The originality of the method is also a bit limited as it highly builds upon TD3+BC.
summary_of_the_review: Based on the comments above, I would vote for a weak reject.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: ### Summary 
Offline RL can significantly reduce the amount of online interactions by leveraging abundant offline data. However, the benefits of pretraining with Offline RL diminishes with the state-action distribution shift which can cause bootstrapping errors. This paper proposes an approach that addresses the problem of distribution shift that happens when transitioning into online RL from offline RL. The paper shows empirical results on three environments from the datasets D4RL-v2, including HalfCheetah, Hopper, and Walker2d.

In the strengths and the weaknesses, I will be  summarising the points reviewers have made during the rebuttal period:

### Strengths

- Studying an important problem with an interesting approach.
- Strong and convincing experimental results.

### Weaknesses
- Clarity could be improved in parts (e.g. beta-clipping, discounts) but also there are some terms that are not clearly explained such as 'actor-critic misalignment'.
- Experiments are thorough but the plots are hard to read. More ablations are needed on possibly smaller/synthetic tasks where different parts of the algorithm can be controlled more carefully.
- It is hard to follow the writing and understand the motivation for some choices.

### Decision

This paper attacks a vital problem to get RL into real-world problems. The proposed approach is interesting, and the scores of this paper were borderline. The authors have done an excellent job during the rebuttal presenting new results to address the concerns raised by the reviewers. However, the paper still has many fixes to be done before acceptance. The writing of the paper requires some improvements, for example, there are unclear statements like "marked state-action distribution" and "actor-critic misalignment." I recommend the authors be more rigorous in explaining these concepts carefully. I think this paper would benefit from further ablations, possibly in simpler and more toyish environments. Most experiments focus on the datasets coming from experts or near-experts. Ablations on the quality of the data and the effect on the performance of the online finetuning would be useful. All those changes require significant revision of the original paper. I recommend the authors make the required revision and strongly encourage a future submission to another ML venue.



justification_for_why_not_higher_score: Noted in the decision part of the meta-review. The paper requires a major revision before acceptance.
Decision: Reject