Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a scheme of policy expansion (PEX) to bridge offline and online RL. Instead of directly fine-tuning the policy learned by offline dataset during online interaction, PEX freezes the learned policy and expands the policy set with a newly added policy. This new policy is optimized during the online learning phase. Then, a single policy is derived from the policy set, where action candidates are sampled by each policy and then the final action is selected by a resampling distribution. In the D4RL benchmark, PEX overall outperforms all of the baseline algorithms.

strength_and_weaknesses: [Strengths]
1. This work provides a simple yet effective scheme for offline-to-online RL. The proposed method is easy to implement.
2. In the experiments, PEX significantly outperforms baseline algorithms.

[Weaknesses]
1. In the experiments, comparisons with existing methods for offline-to-online RL are missing, e.g. [1,2].
2. To claim that PEX is a general framework, rather than an IQL-specific algorithm, it would be great to demonstrate the effectiveness of diverse combinations of PEX and existing offline RL algorithm backbones. For example, it would be great to see the performance of CQL+PEX or FisherBRC+PEX and whether PEX can consistently outperforms the baselines even for those different backbones.
3. The proposed method is designed heuristically, without theoretical justification.

[Questions]
1. In the PEX framework, the offline training phase is to learn a Q function and a policy, where the policy should be frozen while the value should be fine-tuned (i.e. transfer Q). I am wondering why value-finetuning is beneficial but policy-finetuning is detrimental to offline-to-online performance. It would be great to provide a more detailed explanation about this.
2. In Figure 2, why is the performance of PEX in halfcheetah-random better than in halfcheetah-medium-replay even though medium-replay dataset would contain more high-reward experiences?
3. It would be great to see the result using Q-transfer only without the policy expansion, i.e. initialize the Q-function with the pretrained one, and run an online RL algorithm using the Q initialization.
4. How sensitive is the algorithm with respect to the hyperparameter alpha?


[1] Nair et al., AWAC: Accelerating Online Reinforcement Learning with Offline Datasets, 2020

[2] Lee et al., Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble, 2021


clarity,_quality,_novelty_and_reproducibility: Overall, the paper is well-written and easy to follow. The idea of policy expansion and its combination with adaptive policy composition seems novel, but it is a rather heuristically designed method, rather than a principled one. 
summary_of_the_review: The idea of policy expansion is interesting, but it lacks theoretical justification. For the experiments, it would be great to make a comparison with more baselines (see Weaknesses part) and to see the generality of the proposed PEX framework by conducting experiments with different offline RL backbones.

== post-rebutal
Thank you very much for your detailed responses. The added experiments are also greatly appreciated. I raised my score accordingly.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a new offline-to-online finetuning approach that proposes to maintain both the policy learned offline and the new online policy during online-finetuning. The authors first train the offline policy and Q-function, select the policy from a categorical distribution with the learned Q-values during online data collection, and optimize the online policy using data sampled from both the offline dataset and the newly collected online samples. The offline policy is kept frozen during the finetuning step. The authors show that doing so can alleviate the distributional shift and over-conservatism issues when switching from offline training to online training with a single policy. Empirical results suggest that the method can outperform prior method IQL and its variants on the D4RL benchmark.
strength_and_weaknesses: Strength:

1. The idea of using a policy set to reduce the distributional shift issue and conservatism when switching from offline to online training is neat.
2. The empirical results are pretty impressive. the ablation studies are also quite thorough, clearly suggesting the benefits of different components of the algorithm.

Weaknesses:

1. I think the authors should compare the method to more offline-to-online works such as Lee et al., 2021, Yang et al,. 2021, Lu et al., 2022, Zheng et al., 2022 and etc. I think it is important to see if the method can do better than approaches that are not like IQL.
2. The story of using a policy set seems a bit disconnected to the main point of the paper. The main idea of the paper is essentially to train a different online policy and maintain the offline policy for exploration whereas the policy set idea is more than this. I think the authors should make the writing more clear and to the point rather than making too many less relevant statements.
3. The authors don't provide much theoretical insight of the method.
4. The error bars of the experiments are quite high. I wonder if the authors should run more random seeds / do hypothesis testing to make sure the results are statistically significant.
clarity,_quality,_novelty_and_reproducibility: The paper is overall easy to follow but makes many less relevant claims such as expanding policy set. The originality of the work is good. The reproducibility is unclear as the code is not provided.
summary_of_the_review: Based on my comments above, I vote for a weak accept of the paper.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers the offline-to-online RL setting and proposes policy expansion (PEX) by constructing the stochastic online policy: a mixture of the offline policy (trained using the offline data) and the online policy (trained with the online rollouts). In the experiments, the paper shows that PEX has some performance gain over some very recent methods that consider a similar setting. 
strength_and_weaknesses: # Strength 
* The idea of policy expansion, i.e., instead of using the offline policy as a warm start, one can use the offline policy as a checkpoint and combine it with the online policy (which could warm start from the offline training) and still query the checkpoint online, seems a general framework. Also as the paper already points out, the PEX framework could work agnostically to the particular online and offline RL methods. 
* The experiments study is on the general D4RL benchmark, and the paper shows that PEX is competitive in most environments and outperforms some recent baselines such as JSRL and BT.
* The ablation study gives us more insights into some specific design choices such as the benefit of transferring the offline Q-function and why we should freeze the offline policy. 

# Weakness
* The work seems to overlook a few more previous works in this online RL with offline dataset setup, such as [1,2,3,4]. 
* [5] is also very relevant to this paper's setting, and seems even more relevant than the current baselines. I wonder why [5] is not considered as in the experiment section. Also, I believe the paper should contain more explanation on why the current baselines are significant enough (I might be wrong but the baselines seem both from unpublished papers?)
* From the technical side, there are a few points that confuse me:
 1. when does $P_W$ make sense? First there is minor ambiguity in the algorithm: when it says that sampling $a \sim \tilde \pi$ according to (6), in the definition of $P_w$, $\textrm{Q}$ seems arbitrary although we can assume in the context of the algorithm it may refer to $Q_{\phi}$. However, $P_W$ is only a valid weight if $Q_{\phi}$ is accurate, otherwise, it seems just some random heuristic to select which policy to execute. But if $Q_{\phi}$ is accurate, then we should be done because an accurate $Q_{\phi}$ should implies a near optimal policy and thus we don't need to rollout and collect data anymore. Adding on this, there is no explanation of Fig.9, which records the usage of the two policies: it seems like the coefficient stays almost the same during the entire training, does this mean the $Q_{\phi}$ is already accurate at the beginning so minor updates of $Q_\phi \implies$ no changes in the amount of policy switching?

2. Another issue is why we should ever rollout the offline policy $\pi_\beta$. The ablation shows that it is beneficial to include the offline dataset into the online replay buffer, which makes perfect sense. However, the action take by $\pi_\beta$ is just a reflection of the offline data distribution, because the pessimism/penalty applied in the offline RL algorithms. Thus why should we rollout with $\pi_\beta$ because it does not seem to help with exploration/increasing data diversity?

* Finally, the stochastic policy constructed with both the offline policy/expert policy and the online policy seems relevant to the DAgger algorithm [6], with the different being that DAgger is using imitation learning loss while PEX is using RL loss. Nevertheless, it seems fair to include a discussion about such stochastic policy approaches. 

[1] Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep Q-learning from demonstrations.

[2] Stephane Ross and J Andrew Bagnell. Agnostic system identification for model-based reinforcement learning.

[3] Ashvin Nair, Bob McGrew, Marcin Andrychowicz, Wojciech Zaremba, and Pieter Abbeel. Overcoming exploration in reinforcement learning with demonstrations.

[4] Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nicolas Heess, Thomas Rotho ̈rl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstrations for deep reinforcement learning on robotics problems with sparse rewards.

[5] Ashvin Nair, Murtaza Dalal, Abhishek Gupta, and Sergey Levine. Accelerating online reinforcement learning with offline datasets.

[6] Ross, Stéphane, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning.
clarity,_quality,_novelty_and_reproducibility: The overall clarity of the paper seems good, except some notation could be more clear in some specific context (such as the definition of $\textrm{Q}$). 

Regarding reproducibility, the link to the code in the paper does not seem to work for me.
summary_of_the_review: Overall, there are several flaws in the current version of the paper: 1. insufficient literature review; 2. baselines not strong enough (at least no argument to show that the baselines are strong enough); 3. some technical part includes heuristic that may seem rather random. Thus I think the current version of the paper does not deliver enough contribution to the community and thus would recommend a reject at the current stage. 

===========================
raised my score after the rebuttal
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper addresses how online RL can benefit from an existing dataset and/or a policy created by offline RL. The suggested method proposes that both the offline policy (fixed) and the online policy (to be trained) provide action candidates, which are then evaluated using a Q-function (to be learned online). The evaluations are converted into selection probabilities and the action to be executed is drawn accordingly. The method is being tested on several D4RL benchmark tasks.
strength_and_weaknesses: **Strengths**
- The paper deals with a problem that is relevant for practical applications
- The method is very interesting. I find it surprising that it works.

**Weaknesses**
- The, in my opinion, central aspect of how actions are selected is presented somewhat scarcely.  I am not sure that the method can be reliably re-implemented using the explanations provided.

**Further suggestions for improvement**
- I think the calculus of how the selection probabilities are calculated from the Q-values of the candidate actions should be explicitly presented.
- The temperature parameter $\alpha$ probably has central influence on the functioning of the method. It would be interesting to see this influence through experiments.
- Also the entropy of the two policies may have a significant influence. It would be interesting to see this influence through experiments.
- Some sentences could be worded more concisely. For example, sentence „After learning the offline policy, we use it as one candidate policy in a policy set, and further learn another policy that will be responsible for further learning as an expansion to the policy set" seems a bit awkward and thus difficult to understand.\
I stumbled over „limiting its applicability to many practical scenarios“, wouldn't „limiting its applicability in many practical scenarios“ or „which limits its applicability in many practical scenarios“ be better?
- Why do you write "e.g." in italics while "et al." is not italic? This seems inconsistent to me.

„to a a number“\
„onehot“ -> „one-hot“\
„the policy set set as“\
„q-learning“ -> „Q-learning“\
„Should i run“ -> „Should I run“

clarity,_quality,_novelty_and_reproducibility: **Clarity**
should be improved as stated above

**Quality**
Good

**Novelty**
Good

**Reproducibility**
should be improved as stated above

summary_of_the_review: A surprisingly simple approach to solve an important problem. The description of the approach should be more explicit in parts and thus more reproducible. The language should be improved in parts, errors should be fixed.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper addresses the problem of jump-starting reinforcement learning with the dataset collected a-priori, known as offline-to-online reinforcement learning. The main idea is to model the policy as a mixture model with two components, where the first component is the policy pre-trained by an offline RL algorithm and the second component is the target policy being learned online. The pre-trained policy is kept fixed, while the target policy is updated by online interaction data. The mixture weight is determined by the softmax distribution of Q-values. This remarkably simple idea is shown to perform very well on standard benchmark tasks. 

However, there are certainly weaknesses in a number of areas: 
- The algorithm designs are not very intuitive here and there (1) why freeze the offline policy rather than fine-tune it? (2) why use online Q-function to evaluate the quality of offline action? i.e. use offline Q-function to evaluate offline actions and online Q-function for online actions? 
- Readability: (1) the policy set contains just two policies, so why just call it a "policy pair" or something similar? I guess the authors initially designed the method to progressively increase the set over training epochs? In this sense, the terminology "policy expansion" sounds too much, and perhaps "dual policy framework" would sound better?  (2) A.4 contains only one study but the section title and main text says it contains result"s". 




justification_for_why_not_higher_score: The weaknesses listed above:
- The algorithm designs are not very intuitive here and there (1) why freeze the offline policy rather than fine-tune it? (2) why use online Q-function to evaluate the quality of offline action? i.e. use offline Q-function to evaluate offline actions and online Q-function for online actions? 
- Readability: (1) the policy set contains just two policies, so why just call it a "policy pair" or something similar? (2) A.4 contains only one study but the section title and main text says it contains result"s". 
justification_for_why_not_lower_score: - The remarkably simple idea that works very well against baselines
Decision: Accept: poster