Fine-Tuning Offline Policies With Optimistic Action Selection
Authors: Max Sobol Mark
Ali Ghadirzadeh
Xi Chen
Chelsea Finn
Venue: Submitted to ICLR 2023 (2023)

Abstract: Offline reinforcement learning algorithms can train performant policies for hard tasks using previously-collected datasets. However, the quality of the offline dataset often limits the levels of performance possible. We consider the problem of improving offline policies through online fine-tuning. Offline RL requires a pessimistic training objective to mitigate distributional shift between the trained policy and the offline behavior policy, which will make the trained policy averse to picking novel actions. In contrast, online RL requires exploration, or optimism. Thus, fine-tuning online policies with the offline training objective is not ideal. Additionally, loosening the fine-tuning objective to allow for more exploration can potentially destroy the behaviors learned in the offline phase because of the sudden and significant change in the optimization objective. To mitigate this challenge, we propose a method to facilitate exploration during online fine-tuning that maintains the same training objective throughout both offline and online phases, while encouraging exploration. We accomplish this by changing the action-selection method to be more optimistic with respect to the Q-function. By choosing to take actions in the environment with higher expected Q-values, our method is able to explore and improve behaviors more efficiently, obtaining 56% more returns on average than the alternative approaches on several locomotion, navigation, and manipulation tasks.

Under review as a conference paper at ICLR 2023
FINE- TUNING OFFLINE POLICIES WITH OPTIMISTIC ACTION SELECTION
Anonymous authors Paper under double-blind review
ABSTRACT
Offline reinforcement learning algorithms can train performant policies for hard tasks using previously-collettee datasets. However, the quality of the offline dataset often limits the levels of performance possible. We consider the prob- lem of improving offline policies through online fine-tuning. Offline RI. requires pessimistic training objective to mitigate distributional shift between the trained policy and the offline behavior policy, which will make the trained policy averse picking novel actions. In contrast, online RL requires exploration, optimism. Thus, fine-tuning online policies with the offline training objective is not ideal. Additionally, loosening the fine-tuning objective to allow for more exploration potentially destroy the behaviors learned in the offline phase because of the sudden and significant change in the optimization objective. To mitigate this challenge, we propose method to facilitate exploration during online fine-tuning that main- tains the same training objective throughout both offline and online phases, while encouraging exploration. We accomplish this by changing the method to be more optimistic with respect the Q-function. By choosing to take actions in the environment with higher expected Q-values, our method is able to explore and improve behaviors more efficiently, obtaining 56% more returns on average than the alternative approaches on several locomotion, navigation, and manipulation tasks.
INTRODUCTION
Offline reinforcement learning (RL. algorithms show strong performance on different tasks even when the available data contains sub-optimal behaviors such random exploration or another RI. agent's behavior. However, the quality of the offline data can limit the performance of an offline RL agent, and in these cases, further online fine-tuning can help improve the policy using additional environment interactions. A key challenge to online fine-tuning is that the training mechanisms designed for the offline setting requires conservative objective to mitigate distributional shift between the learned policy and the behavior policy. However, during the online fine-tuning phase, the conservative objective causes the agent to keep doing the same behaviors, preventing it from exploring as needed to further improve the policy. In this paper, we aim to mitigate these challenges and develop more effective method for offline-to-online RL.
not effective improving the policy online; offline learning
This challenge of misaligned priorities between the offline and online training phases could in prin- ciple be tackled by switching training objectives when the online phase starts. For example, a policy trained with an offline RL algorithm such as CQL (Kumar et al., (020) could be fine-tuned using an off-policy online algorithm designed for efficient exploration, such as SAC (Haarnoja et al.) 2018). However, significantly changing the training objective between the offline and online phases can cause instability in the performance, potentially resulting in the destruction of behaviors learned during the offline phase (Nair et al. (2020). can also exacerbate distributional shift between the offline data and the learned policy (Lee et al. 2022) since the more exploratory training objective will make the policy deviate from the offline behavior policy even more, resulting in unsatisfactory fine-tuning performance.
The problem of online fine-tuning from offline training presents an impasse: on the one hand, online fine-tuning requires efficient exploration, which is not feasible using the conservative offline training

---

Under review as a conference paper at ICLR 2023
objective; on the other hand, changing the objective might destroy the behaviors learned by the offline training. A key insight of our method is that we collect optimistic data without changing the training objective. To collect such exploratory data, we aim to the knowledge embedded in the Q-function to direct exploration, i.e., selecting actions that are estimated to be better than the ones given by the policy. Concretely, during online fine-tuning, estimated Q-values, thus providing exploratory data that can be used to improve the policy, while using the same offline objective with this new data.
execute actions that have high
The key contribution of this paper is an exploration technique that allows for stable and efficient offlie--o-online fine-tuning. Our approach is simple, as it only modifies the action-selection pro- cess, and can in principle be implemented on top of any offline learning algorithm that doesn't 't explicitly penalize the Q-values of distrii-dittitttiion state-action pairs. We show results built on top of LAPO (Chen et al.) (022) and IQL. (Kostrikov et al., 2021) for several locomotion, navigation, and manipulation tasks, and find that by using this aetion-selection mechanism agents get an average of 56% more returns than the next best prior method.
2 RELATED WORK
Exploration mechanisms: One simple technique for exploration is to apply noise or randomness
to the agent's s behavior (Mnih et al.] 2015; Lillicrap et al., 2015: Schulman et al.] 2015). Other
approaches explore in more targeted way through novelty seeking (Houthooft et al. 2016 Belle-
mare et al. 2016: Pathak et al., 2017 Ostrovski et al. 2017 Fu et al. 2017a; Burda et al. 20 a)
and entropy regularization (Pong et al. 2019 Eysenbach et al. 2018 Florensa et al. 2017 Gre-
gor et al. 2016 These prior exploration methods are focused on online RL and are ill-suited for offline-to-one onllie fine-tuning on their own, as exploration can negatively impact offline training. Our exploration mechanism is based on modifying the action-selection mechanism to choose actions more optimistically using a Q-function pre-trained with offline RL.
Offline to online fine-tuning: The problem of leveraging offline data during online RL has been widely studied especially when the offline data corresponds to demonstrations. Prior work has proposed a variety of approaches for this setting. ranging from online inverse RL and imitation learning (Lu et al., 2022, Ziebart et al. 2008 [Finn et al., 2016 Fu al., 2017b] Ho Ermon 2016 Kostrikov et al. 2018) to using the demonstrations to initialize the policy or replay buffer (Peters& Schaal, 2008 Vecerik et al. [017 Hester et a 2018 Rajeswaran et al., 2018 Zhu al,
2018b; Kober & Peters 2009). We consider more general case where the offline data may contain low or mixed quality data. The simplest approach for the general case of offline to online RL is to fine-tune online using the same offline objective (Kumar jet al. 2020; Kostrikov et al. 2021; Nair et al.] 2020: Lyu et al.) 2022); however, this approach may be too conservative during the online phase, resulting in too little exploration and plateauing performance. Instead of maintaining the same offline training objective during the online phase, Wu al.) (2022) propose gradually make the training objective less conservative. As we empirically show, this approach does not significantly improve exploration for the considered baselines. Lee al. (2022) instead propose target the over-conservatism of using an offline training objective during the online phase with prioritized replay buffer that prefers more on- on-policy samples. This approach could in principle be combined with our method for enhanced exploration, but we leave this direction for future work. Unlike these past works, we develop a simple and sample-efficient fine-tuning approach that effectively trade-offs conservatism in the offline phase and optimism in the online phase without sacrificing stability.
201 8aj Gupta et al.) 2019 Zhu et al.
Offline meta reinforcement learning: with efficient online adaptation using an offline dataset. However, the meta-RL domain assumes access to a multi-task offline dataset, and trains an agent to quickly adapt to new tasks (Mitchell et al. 2021 Dorfman et al.) 2021 Pong al. 2022). As opposed to offline meta RL, our work (and offline-to to-nee-to-oninee fine-tuning in general) does not assume access to multi-task data, and deals with adaptation and improvement in the context of the same offline task.
Offline meta RL related to online fine-tuning, since it deals
2

---

Under review as a conference paper at ICLR 2023
Offline Phase
Online Phase
Policy
Ensiroment
(00
Policy
Q-Funetion
Q-F e-Function
Offinne Detaset
Polley
Replay Buffer (Initialzeed with offine dataoct)
Function
Figure 1: The online fine-tuning setting has two phases: an offline phase, in which no interaction with the environment is possible, and all learning has to be done using the offline dataset; and an online phase, which the agent has perform well from the beginning. and keep improving with access to the environment. Our method O3F trains policy and a critic ensemble in the offline phase, and then selected. As illustrated on the right, actions selected optimistically based the estimated scores given by the scoring function from equation 2 For stability, the same training objective as the offline phase is used in the online phase, while incorporating the online experiences into the replay buffer.
online phase, fine-tunes the policy and critic ensemble while modifying how actions
PRELIMINARIES
We address policy learning in Markov decision process (MDP), defined by the tuple (S, p, r), where S and A denote the state and action spaces. The unknown state transition probability distri- bution p S S A [0, oo) represents the probability density of the next state s' E S given the current state S and action E A. The reward function assigns a sparse reward to every state-action transitions r A R.
3.1 OFFLINE REINFORCEMENT LEARNING
In the offline RL setting, an agent only has access to dataset of experiences D = {( s'x that are collected prior to the RL training using an unknown behavior policy ng. Off Off-policy RL approaches can leverage the offline dataset to train critic network (Q-function) Q (5, a) with parameters 0, which estimates the long-term discounted reward attained by executing action a at state and following the policy thereafter. The critic network can then be used to update the policy in an actor-critic training scheme. The critic network can be trained using the following temporal difference learning objective:
((0) "(s...""=
max Qg(s' (s',a'))=aa))))))))) Qe(s,
(1)
where i denotes the delayed parameters of the Q-function. Notice that is a potentially out-of- distribution action, (s's', a') might give an incorrect value. The max operator evaluated over many notentially-incorre values is then likely to update the Q-function with incorrect information, resulting in sub-optimal policies. To alleviate this problem, prior offline RL methods propose two general solutions: (1) assigning low Q-values to the out-of-distribution actions by training a Q- function with a pessimistic objective (Kumar et al.) 2020 Lyu al.) 2022), and (2) training Q-function using equation [] while constraining the policy only produce actions contained in the offline dataset D (Kostrikov al.] 202T] [Cen et al. 2022). The action-selection mechanism we present in this work relies sampling the Q-function for slightly pairs and obtaining non- pessimistic value estimates. Our method is therefore only applicable to the second class of offline RL algorithms that constrain the actor, and not the critic.
state-action
3.2 OFFLINE-TO-ONLING FINE- TUNNNG
As shown in Figure having trained a policy and Q-function with offline data, the goal of online fine-tuning further improve the behavior of the learned policy by allowing the agent interact
3

---

Under review as a conference paper at ICLR 2023
[a] Q-function bias
(b) Normalized evaluation return
(c) Critie loss
Neenaltod eetenn
0.6
00
Chitee Ioee
20- Q-imetto
0.4
10
0-
100000
10000
Online contectuncel stopa
Online cavtrotunenl stopa
Online castronanen stoze
SAC (from CQL)
CQL
Figure 2: Average Q-function bias, performance, and critic loss for SAC initialized from CQL and trained with the aallcheetta-meddiun dataset in its replay buffer. The beginning of the online fine-tuning training destroys the performance CQL learned offline, and training with the optimistic SAC objective makes the Q-function greatly over-estimate state-action values.
with the environment. In every episode of the online fine-tuning phase, new samples are collected by unrolling the latest policy in the environment and are added to the replay buffer. Then, the policy and the Q-function are updated by the actor-critic learning algorithm which is employed for the online fine-tuning phase. Offline-to-onlinn fine-tuning presents two closely related challenges: (1) conservative offline learning objectives limit agents' exploration power, which is vital for efficient online learning, and (2) changing training objectives when switching from the offline phase to the online phase can cause significant drop performance, due distributional shift and the sudden change in objective. A suitable method for online fine-tuning should handle the transition from offline to online while improving the policy performance by carefully exploring the environment.
OBJECTIVE SWITCHING BETWEEN OFFLINE AND ONLINE PHASES
The pessimistic objectives in offline RL algorithms prevents proper exploration in online settings and limits the agent from efficiently improving its performance. As shown in Figure Bb online training with CQL does not notably improve the learning performance even though the agent is no longer bounded by the offline setting. This limited performance in the online setting can be explained by the restricted exploration capability of the agent. CQL avoids querying out-of-distribution actions, hence it stabilizes the offline training by explicitly assigning low Q-values to actions outside the support of the offline dataset. As a side-effect, CQL would have difficulties in efficiently exploring the environment, and therefore it demonstrates poor performance at improving the policy in online settings.
As discussed earlier, naive solution to this problem would be to pre-train constrained policy with offline RL, and then modify the objective at the fine-tuning phase by lifting the constraints making them less conservative. However, our empirical studies show that this approach do not perform well in practice. Figure[[(b) shows the results of using the soft actor-critic (SAC) algorithm for online fine-tuning of a policy trained with CQL. SAC leverages maximum entropy objective to encourage exploration which is helpful to improve the performance at the online fine-tuning phase. However, as shown by the figure, fine-tuning the CQL policy with SAC leads to immediate drop in performance, which takes about 15,000 online environment steps to recover. We hypothesize that this drop in performance stems from the sudden change in the Q-function objective (from CQL to SAC). As shown in figure cc. the critic loss spikes once the objectives changes. Additionally, as shown in figure updating the objective results in 40x over-estimation of the Q-values for least several thousands of steps after switching the objective. This renders the pre-trained Q-function sub- optimal and in need of more updates with online data. Updating the policy using this sub-optimal Q-functoon destroy the behaviors that were learned during the offline training phase.
We may encounter similar problem even for a minor modification of the objective. Figure illustrates the online fine-tuning performance for CQL, AWAC. and IQL on the HalfCheetah envi- ronment with medium offline dataset for three settings: (1) no change in the objective, (2) changing the syper-parameters the beginning of the online fine-tuning phase to make the objective less
4

---

Under review as a conference paper at ICLR 2023
(a) No training objective change
(b) Sharp change in conservative weight
(e) Slow change in conservative weighht
Nornalized petunn
0.88
0.8
0.7
11.7
0.7
0.6-
0.8
0.8
300000
300000
500000
Ouline environment steps
Ouline enviromment steps
Ouline environment steps
CQL AWAC
IQL
Figure 3: Comparison of results for online fine-tuning on halfcheetah-mediiu--Vy for CQL, AWAC, and IQL for settings: (1) fine-tuning with the same offiine training objective, (2) immediately changing the training objective to be less conservative the online phase, and (3) linearly changing the hyper-parameters to make the training objective less conservative. For CQL we reduce the Lagrange threshold from to 0.5, for AWAC we reduce the Lagrange multiplier from 1 to 0.1, and for IQL. we increase the expectile from 0.7 to 0.85.
conservative, and (3) linearly updating the values of the hyper-parameters over time so as to make the change smooth as possible. The syper-parametts Lagrange multiplier for AWAC, and the expectile value for IQL. As shown by the figure, abruptly changing the CQL objective to be less conservative negatively affect the learning performance while smoothly updating the objective yields a similar level of performance as training with a fixed ob- jective. IQL shows slightly better performance when the objective updated abruptly, and AWAC training does not seem to be affected by the change in the apper-paraeete. To conclude, our em- pirical results show that making the objective less conservative during the online phase does not significantly improve the online fine-tuning performance.
the Lagrange threshold for CQL, the
OPTIMISTIC ACTION SELECTION MECHANISM
To explore efficiently without changing the offline training objective, take advantage of the knowledge that the Q-function might contain about behaviors proximal to the ones the current pol- icy produces. Provided that the Q-function has been trained so not explicitly penalize out- of-distribution actions (as, for instance, CQL would), we can expect the Q-function to sometimes assign reasonable values to slightly dust---distiibution state-action pairs. In this section we describe the offline to online optimistic fine-tuning (O3F) algorithm, an action-seles sellctinn process that samples multiple action-candidan candidates, and queries the Q-function decide which action explore.
5.1 OPTIMISM VTA ACTION RANKING
We leverage the knowledge embedded in the Q-function choose actions that better than the ones given by the policy. However, only slightly out-of-distribution samples from the Q-function be trusted since value estimates for state-action pairs not supported by the training data can have arbitrarily-incores
values. This motivates the design of our action-selection mechanism, as it only queries the Q-function for actions close to the ones given by the pre-trainee tre-trained policy. As demonstrated by Algorithm [ O3F requires an offline dataset, policy and an ensemble of critic networks pre- trained with the offline data, and the offline update rule for actor-critic training that is used for offline pre-training. We initialize a replay buffer with the offline dataset. In every step of online fine-tuning, 03F randomly samples N candidate actions from a normal distribution, e; ~ (a,, o²), where the mean a is the action given by the policy, actions are then ranked according to some scoring function f(s, a) + R, and the candidate with the highest score gets executed in the environment. The new online experience is then incorporated into the replay buffer. After every step, O3F updates the actor and the critic using a batch of data from the replay buffer and the provided offline update rule.
(s)s), and a is small constant value. The candidate
5

---

Under review as a conference paper at ICLR 2023
5.2 UNCERTAINTY OPTIMISM SCORING FUNCTION
We consider two design choices for the scoring function [(s, a): (1) the estimated Q-value, i.e., [(8,a) a)
Algorithm 1 O3F
1 Σj= (Qo, (s, a)]. and (2) an upper m confidence bound of the estimated Q- values which refer to the uncer- tainty optimism scoring function. In both cases, O3F only queries actions with trusted values, i.e., actions close the policy output. This is partieu- larly important for the uncertainty op- timism scoring function, as the uncer- tainty of the value estimates farther away from the policy output can be ar- bitrarily high, which results in favoring -uutr--distribution actions.
Require: Offline algorithm Update( (0.),0), (00,2) D) Require: Offline policy ma. Critic ensemble (Qo, (8. a)) Require: Offline dataset D
1: for step t in . do
2:
a ~ TTa(a)s))
3: Sample n action-candidates e; ~ N Naa d²)
4: Get best candidate ranking by expected Q-value.
at = arg max; 1 -)= Q0, (5):
5:
81+1 ~ p(8-11 at)
6:
D D ((5, Oe, r(st, at), (1+1111)
7:
Φ, (0.) + Update [0:)(0, D)
8: end for
We model the uncertainty of the value- estimates by using an ensemble of independently-trained Q-functions. Let (s,a) with parameters {0,}. We define Q (s, a) = 1 E (Q, and Qua(s, a) =
RJ be the ensemble of Q-functions
m EE Qo, (s, (s,a) mmcan(s. a)l2. The proposed scoring function is then the upper confidence bound given by (Chen et al. 2017 Ghadirzadeh et al. 2016):
f(s.a) = Q (s, a) w uu a),
where i is a syper-parameter balancing the optimistic level of the scoring function. For two action candidates with similar estimated values, the upper confidence bound selects the action with higher prospects to encourage exploration.
5.3 IMPLEMENTATION DETAILS FOR O3F
We evaluate our method O3F applied on top of two offline RL algorithms, LAPO (Chen et al. 2022)
and IQL (Kostrikov et al. 202T). For both algorithms, the offline training phases are carried out in a similar fashion to the original works. For LAPO, we find that training the critic in manner similar to Chen et al. (202T) improves sample-efficiency. In particular, we train an ensemble of Q-functions, and use random subset of the Q-functions to generate the training target. This modification lets us update the critic 20 times per environment step during the online fine-tuning phase. This style of critic-traiiigg made IQL unstable, and thus we do not include it in the final method for IQL. Instead, IQL's critic is trained the same way as the original paper, with the distinction that we train Q-functions instead of to be consistent with the LAPO-based implementation.
Our method introduces 3 seper-parameters: N, the number of standard deviation of the normal distribution; and w. the optimism weight for scoring function. We found that N = 100 works well on every considered task. On locomotion tasks, we found a = 0.2 to yield the best performance, and every other task uses a 0.05. We hypothesize that because of the dense reward function of the locomotion tasks, the Q-function can potentially be estimated more globally which enables O3F to evaluate a wider range of actions, and not only the ones located close to the policy output. On section 6.3 we compare the performances of w = 1 and w = 0, and find that the uncertainty optimism scoring function improves performance for the halfcheetah-mediim-- task, decreases performance on kitchen-mixed mixed-v0, and maintains performance diverse-vl. To avoid unnecessary complexity, the rest of the experiments use w = 0.
to generate; a,, the
antmaze-large-
EXPERIMENTAL EVALUATION
The goal of our experiments is to, first and foremost, evaluate how O3F compares prior methods. Moreover, we aim to study whether O3F is compatible with multiple base offline RL methods and to ablate the role of different design choices within O3F.

---

Under review as a conference paper at ICLR 2023
halicheetah-random-ss
walker22-randoom-22
hopper-random-v22
0.2
0.5
0.5
0.0
0.0
0.0
halfcheetah-mes m-deraam-mmdimm2
wa walker2d
0.75
0.50
antmmaze www.zaaa---ii. diverse-v1
antimaze-
diverse-vl larees--e
kitchen-complete-vi
kitchen-partial-ct
kitchen-mixed-vvl
0.5
0.5
0.0
500000
500000
Online enviromment stops
03F (LAPO)
AWAC
CQL
IQL
Figure 4: Normalized scores on D4RL tasks. Our method implemented top of LAPO outperforms prior methods on the challenging Antmaze and Kitchen tasks, and either outperforms or performs comparatively prior methods on locomotion tasks with random and medium datasets.
6.1 EXPERIMENTAL SET-UP Up
Tasks. We evaluate O3F on a variety of locomotion, manipulation, and navigation tasks from the D4RL benehmark Fu et al. (2020). We consider twelve total problems: six in simple environments and six in more complex environments. For the simple tasks, we use the D4RL Gym-MuJoCo lo- comotion tasks consisting of the HalfCheetah, Hopper, and Walker2D domains. Since we want to evaluate the ability to improve with online experience, we focus the random and medium offline datasets, which have the lowest offline performance (as opposed to, e.g., using the expert dataset, which leaves little room for further improvement). The random dataset consists of experiences collected by a random policy, while the medium dataset uses experiences collected by a mediocre policy trained with early stopping. Since the locomotion tasks relatively simple and straightfor- ward, we also evaluate on the ant-maze and Franka kitchen environments, which are considerably more difficult. The ant-maze domain requires both learning walk, as well as reaching the goal in maze. We consider the three diverse ant-maze datasets, which involve mazes of increasing difficulty. The Franka kitchen domain involves controlling a simulated Franka robot kitchen environment solving sequence of tasks. We consider all three Franka kitchen datasets in the D4RL benchmark: "complete", "mixed", and "partial".
Comparisons. We evaluate the performance of O3F implemented on top of both LAPO (Chen al. 2022) and IQI. (Kostrikov et al.) 202T) in comparison to the prior work on offline-to-online fine-tuning tuning. In particular, we consider three strong prior methods baselines: (1) CQL (Kumar ct al. 2020), (2) IQL (Kostrikov et al. 202I) and AWAC (Nair et al., 2020). CQL and IQL are used in offline-to-onling fine-tuning settings by adding new experiences the replay buffer and training with the same offline procedure provided all offline and online data. AWAC (Nair et al. 2020 is specifically designed for the online fine-tuning setting.
For CQL and AWAC, we use d3rlpy (Takuma Seno 202T). an offline RL library that provides imple-
mentations of multiple offline RL algorithms, and allows for online fine-tuning. For IQI., we use the

---

Under review as a conference paper at ICLR 2023
antmoopslopeed
0.9
0.3
11.-
Nanndiied
0.7
0.4
0.5
0.0
0.0
300000
DAKAAAI
Online envtrnment stepe
Online coviroemenn steps
Ouline curtrommnnt steps
(LAPO)
((QU))
Randomn Noise
Figure 5: Comparison of O3F applied top of both LAPO and IQL to baseline of the base algorithm with action noise during online data collection. We find that O3F is compatible with both LAPO and IQI., and outperforms LAPO and IQL with random noise in two out of three domains while matching the performance in the third domain.
original paper's implementation. For all three prior methods, we use the original hyper-parameteen including the policy and the critic function network architecture. Every experiment starts with train- ing a model only using the offline dataset, like the original work. Then, we initialize replay buffer with the offline dataset, and proceed alternate between taking step in the environment and adding the experience to the replay buffer, and training the agent with the original offline procedure.
6.2
MAIN RESULTS
Figure shows the comparison of our method (O3F on top of LAPO) Comparisons of O3F with LAPO and IQL are made in section 6.3 to or better than all of the prior methods in all the tasks except for hopper-random-v2, in which AWAC obtains the best performance. O3F generally does not see a drop in performance after the start of the online fine-tuning phase, which shows that acting optimistically after the offline training phase can improve performance without sacrificing stability. We find that CQL and AWAC are unable to solve the Franka kitchen and AntMaze tasks, with the exception of AWAC on the simplest AntMaze setting (umaze), while O3F on top of LAPO reliably outperforms IQL on these tasks. For the simpler locomotion tasks, O3F outperforms all the other methods for saffiheetaa-randdoms walker2d-random- and halfcheetah-mediinee on walker2d-medium--22 and hopper--edium--t and underperforms AWAC for hopper-randon-vv. On average, O3F performs 56% better than the next best method (IQL).
to the three prior methods. O3F performs comparably
performs comparatively to the best prior method
6.3 FURTHER COMPARISONS AND ABLATIONS
Action candidate ranking: We compare O3F with the simple baseline adding random noise to the actions. O3F top of both LAPO and IQL. implements the action-selection mechanism de- scribed in Section 5. and the implementation details in Section (53 In contrast, we compare to the original approaches of LAPO and IQL with noise added the actions during online data collection, i.c. with noise e ~ (O, o") (where is the same as in O3F). Results for this experiment can be seen in Figure S First, find that O3F is compatible with both LAPO and IQL, matching or outperforming the performance of the base algorithm. For LAPO, our approach improves perfor- mance on halfcheetah- performance for kitchen-minn sitchen-mixed-vO, where performance was already close to saturation (we find that even with further fine-tuning-) For IQL.,
by 58%, on antmaze-large largii-vrer---eeee--e-e by 28%, and maintains the
LAPO does not surpass 0.75 returns on O3F improves performance by 14% halfchertah-mediim--2 and by 39% on kitehen-mixed-vo, while maintaining performance on antmaze-large- -i
Uncertainty optimism scoring function: We compare two designs for the scoring function in our method: (1) scoring action candidates by estimating their value using the Q-function ensemble (this corresponds to (s, a) from Eq. 2 when
0); and (2) the uncertainty optimism scoring function introduced in section 5.2 (fo (s, a) when w = 1). We show results for environments (halfcheetah- and kitchen-mixed-v0) Figure 6 The uncertainty opti-
medium-v2, antmazz-large-divern
8

---

Under review as a conference paper at ICLR 2023
lite Ricebreemiooddrt
0.-
0.3
0.7
0.4
300000
Online carticamest steps
Ouline eustrommeni stepe
Online cartronment steps
OSF (ur
1)
O3F (w
Figure 6: Evaluation of two different scoring functions for action candidates. w = 1 corresponds to the uncertainty optimism scoring function, which prefers actions with higher q-value uncertainty. w = doesn't it take uncertainty into account. find that the uncertainty optimism scoring function performs best in the Franka kitchen domain, but is less advantageous in the other two domains.
mism scoring function outperformed the baseline scoring function on the kitchen environment, but environment. On the antmaze- lar.aaee--rrg-dveee---ee
underperformed in the nalfchectah-mmdiume there weren't 't significant differences in performance between the two scoring functions. To avoid unnecessary complexity, we focus our experiments on the baseline scoring function (f (s, where 0).
CONCLUSION
presented a simple yet efficient offline-to-online fine-tuning strategy to further improve pre- trained offline policy. The goal of online fine-tuning is improve the agent's performance by properly exploring the environment without forgetting the behaviors already learned by the offline training. However, most offline RI. algorithms exploring state-action pairs not supported by the offline data. Therefore, is challenging to fine-tune the learned policy acquire new behaviors. We empirically showed that fine-tuning with the same objective does not result in satisfactory fine-tuning performance, and making the objective less con- servative negatively affects the behaviors already learned by the offline training. We presented O3F, an offline to online optimistic fine-tuning algorithm which facilitates exploration without changing the objective. O3F improves exploration by using an action- selection method which favors opti- mistic actions with respect to Q-function pre-trained with offline data. We empirically showed that fine-tuning with O3F can on average achieve 56% more returns compared to IQL,, AWAC and CQL. baselines several locomotion, navigation and manipulation tasks.
pessimistic in that they prohibit the agent from
Limitations: There are two important limitations for the proposed algorithm: (1) it is only ap- plicable to offline RL methods that do not explicitly penalize the Q-values of outtof-distribution state-action pairs, and (2) it can only refine the behaviors already learned the offline phase, with- out the capability of discovering new behaviors. We argue that the second limitation is generally applicable to fine-tuning tuning methods that are built on top of pessimistic offline RL algorithms. We will study this limitation in details a part of our future work.
REFERENCES
Marc Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos. Unifying count-based exploration and intrinsic motivation. Advances neural information pro- cessing systems, 29, 2016.
Yuri Burda, Harri Edwards, Deepak Pathak, Amos Storkey, Trevor Darrell, and Alexei A Efros Large-scale study of curiosity-driven learning. In International Conference Learning Repre- sentations, 2018a.
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. In International Conference Learning Representations, 2018b.

---

Under review as a conference paper at ICLR 2023
Richard Y Chen, Szymon Sidor, Pieter Abbeel, and John Schulman. Ucb exploration via q- ensembles. arXiv preprint (04.::/706.0102,, 2017.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang. Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent-variable advantage-weighted policy optimization for offline rl. arXiv preprint
2022.
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2101.0500 10009992. 2021.
Ron Dorfman, Idan Shenfeld, and Aviv Tamar. Offline meta reinforcement -iiliiiiid challenges and effective data collection strategies. Advances in Neural Information Processing Systems, 34:4607-4618, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Divensity all you need: Learning skills without reward function. In International Conference on Learning Representa- fions, 2018.
Chelsea Finn. Sergey Levine, and Pieter Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning, pp. 49-58. PMLR, 2016.
Carlos Florensa, Yan Duan, and Pieter Abbeel. Stochastic neural networks for hierarchical rein- forcement learning. arXiv preprint arXiv: 7770000012, 2017.
Justin Fu, John Co-Reyes, and Sergey Levine. Ex2: Exploration with exemplar models for deep reinforcement learning. Advances in neural information processing systems, 30, 2017a.
Justin Fu, Katie Luo, and Sergey Levine. Learning robust rewards with adversarial inverse rein- forcement learning, arXiv preprint arXiv:1710.1122: 2017b.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072/9 2020.
Ali Ghadirzadeh, Judith Bútepage, Atsuto Maki, Danica Kragic, and Mårten Björkman. A sensori- motor reinforcement learning framework for physical human-robot interaction. In 2016 IEEF/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 2682-2688. IEEE, 2016.
Karol Gregor, Danilo Jimenez Rezende, and Daan Wierstra. Variational intrinsic control. arXiv preprint arXiv:161 1.07507, 2016.
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay Pol- icy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning. Techni- cal Report arXiv: arXivv1910.1m 1991..1996,,11111111111111111119 arXiv, October 2019. URL URL.:cppp///axl..orr//ab///////
11956 1rXiv:1910.11999 [cs, stat] type: article.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning. pp. 1861-1870. PMLR, 2018.
Todd Hester, Matej Vecerik, Olivier Pietquin, Mare Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Ian Osband, et al. Deep q-larning from demonstrations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural
information processing systems, 29, 2016.
Rein Houthooft, Xi Chen, Yan Duan, John Schulman, Filip De Turck, and Pieter Abbeel. Vime: Variational information maximizing exploration. Advances neural information processing sys- tems, 29, 2016.
Jens Kober and Jan Peters. Learning motor primitives for robotics. In 2009 IEEE International Conference on Robotics and Automation, pp. 2112-2118, May 2009. doï: 10. 1099...0......00 5152577. ISSN: 1050-4729.
10

---

Under review as a conference paper at ICLR 2023
Iy Kostrikov, Kumar Krishna Agrawal, Debidatta Dwibedi, Sergey Levine, and Jonathan Tomp- son. Discriminator-actor-riiii Addressing sample inefficiency and reward bias in adversarial imitation learning. arXiv preprint arXiv:1809.02922. 2018.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:21 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179 19:1179-1191, 2020.
Seunghyun Lee, Younggyo Sco, Kimin Lee, Pieter Abbeel, and Jinwoo Shin. Offine-to-online reinforcement learning via balanced replay and pessimistic q-ensemble. In Conference on Robot Learning. pp. 1702-1712, PMLR, 2022.
Timothy Lillicrap. Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint ar arXiv:1509.0299 00009, 2015.
Yao Lu, Karol Hausman, Yevgen Chebotar, Mengyuan Yan, Eric Jang, Alexander Herzog, Ted Xiao, Alex Irpan, Mohi Khansari, Dmitry Kalashnikov, al. Aw-opt: Learning robotic skills with imitation andreinforcement at scale. In Conference on Robot Learning, pp. 2022.
PMLR,
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly Conservative Q-Learning for Offline Reinforcement Learning. Technical Report arXiv:2206.00 04745, arXiv, June 2022. URL http:
04745 HtXiv:2206.04744 [cs] type: article.
Eric Mitchell, Rafael Rafailov, Xue Bin Peng. Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In International Conference on Machine Learning, pp. 7780-7791. PMLR, 2021.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):529-333 2015.
Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online rein- forcement learning with offline datasets. arXiv preprint arXiv:2006.09:: 2020.
Georg Ostrovski, Marc G Bellemare, Aäron Oord, and Rémi Munos. Count-based exploration with neural density models. In International conference on machine learning, pp. 2721-2730. PMLR, 2017.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference machine learning, pp. 2778-2787 PMLR, 2017.
Jan Peters and Stefan Schaal. Reinforcement learning of motor skills with policy gradients. Neural Networks, 21(4):682-697, May 2008. ISSN 08936080, doi: 10.10166 URL https:
www.hhghh.......e........ elsewierreeeeee.eeee.ee.....e... com wwtp:///ttieeeeee/////3333333330
Vitchyr H Pong, Murtaza Dalal, Steven Lin, Ashvin Nair, Shikhar Bahl, and Sergey Levine. Skew- fit: State-covering self-supervised reinforcement learning. arXiv preprint arXiv:1903.032: 2019.
Vitchyr H Pong. Ashvin V Nair, Laura M Smith, Catherine Huang, and Sergey Levine. Offline meta- reinforcement learning with online self-supervision. In International Conference on Machine Learning. pp. 17811-17829. PMLR, 2022.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning Complex Dexterous Manipulation with Deep Reinforce- ment Learning and Demonstrations, Technical Report arXiv: 7t....00...99. arXiv, June 2018. URL
http://arxiv. wwtp://wwwww//w/////////// 10087 arXiv: 10087 [cs] type: article.

---

Under review as a conference paper at ICLR 2023
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning. pp. 1889-1889 1897. PMLR, 2015.
Michita Imai Takuma Seno. d3rlpy: An offline deep reinforcement library. In NeurIPS 2021 Offline Reinforcement Learning Workshop, December 2021.
Mel Vecerik, Todd Hester, Jonathan Scholz, Fumin Wang, Olivier Pietquin, Bilal Piot, Nico- las Heess, Thomas Rothörl, Thomas Lampe, and Martin Riedmiller. Leveraging demonstra- tions for deep reinforcement learning robotics problems with sparse rewards. arXiv preprint VrXiv:1707.080
2017.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported Policy Opti- mization for Offline Reinforcement Learning. Technical Report Siwi::2222.0623399 arXiv, May 2022. URL /tt/ttt://arrxxw.or///bbb http:///wwwo///a/////..eeeee 02222.06233 arXiv:2202.06239
[cs] type: article.
Henry Zhu, Abhishek Gupta, Aravind Rajeswaran, Sergey Levine, and Vikash Kumar. Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Cost. Technical Report arXiv:1810.06045. arXiv, October 201 8a. URL
06045 1rXiv:181000660 [cs] type: article.
Yuke Zhu, Ziyu Wang, Josh Merel, Andrei Rusu, Tom Erez, Serkan Cabi, Saran Tunyasuvunakool, János Kramár, Raia Hadsell, Nando de Freitas, and Nicolas Heess. Reinforcement and Imitation Learning for Diverse Visuomotor Skills. Technical Report 1rXiv:1802.09555 arXiv, May 2018b.
URL http:://txiii../t/////// org awwaaasss.1. 09564 arXiv:
[es] type: article.
Brian Ziebart, Andrew L Maas, J Andrew Bagnell, Anind K Dey, al. Maximum entropy inverse reinforcement learning. In Aaai, volume 8, pp. 1433-1436 Chicago, II. USA, 2008.
12


Reviews:
[{"review_id": "rCYpm6LaXOA", "reviewer": "Reviewer_ggWX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes offline to online optimistic finetuning (O3F), an algorithm for offline to online RL. The algorithm maintains the same offline learning objective during online learning, but uses a different action distribution that selects actions with potentially higher Q values. This is achieved by simple adding gaussian noise to the mean action output by the policy network and then selecting the action with highest Q value. Experiments are run on a variety of datasets from the D4RL suite, showing better performance than some baselines. ", "summary_of_the_review": "Overall I think this paper addresses an interesting problem and proposes a potentially clever algorithm. However, as implemented, I do not have confidence in the quality of the experimental results and rate the paper as a reject. If the authors provide improved baselines and comparison of hyperparameter tuning, I would consider raising my score. ", "strength_and_weaknesses": "\n### Strengths\n\n1. The proposed algorithm is simple and elegant. Moreover, the motivation is reasonable, saying that we can get better performance by encouraging better exploration without forgetting the information learned during offline training. \n\n2. Experiments are presented on a variety of tasks from the D4RL suite, that seem to show consistent gains.\n\n### Weaknesses\n\n1. The most serious issue with the paper is the improper choice of baselines. First and foremost it is especially strange that the main experiment does not include LAPO as a baseline. Since the main claim of the paper is that the O3F action selection mechanism is the important innovation, this should be isolated. This means that either the main result should use LAPO as a baseline, or examine O3F on top of IQL. Second, neither of the most closely related works, Wu et al 2022 and Lee et al. 2022 that are cited in the introduction, are used as baselines in the experiments (excluding the comparison in Figure 3 on one task with a different base learner than the one used later on). \n\n2. Some relevant ablations are missing. Figure 5 is a good start on an ablation showing that the action selection is better than adding random noise to the actions. The other relevant comparisons that should be added are to (a) sample actions directly from the distribution output by the learned policy, and (b) choose deterministic actions from the mean of the output of the learned policy. These ablations would give a more fair comparison to what happens without the exploration that O3F claims to provide.  \n\n3. There are no details about hyperparameter tuning and it seems that there was possibly substantially more hyperparameter tuning for O3F than for the baselines. Specifically, from the paper there seem to be several hyperparamters that are potentially tuned for the method differently than baselines including the size of the Q ensemble, number of Q updates, number of randomly sampled actions, and the variance of the gaussian noise.\n\n4. There are no experiments verifying that the improved performance is indeed due to improved exploration as is frequently claimed in the paper. Such an experiment by e.g. measuring the amount of coverage during online training with and without the O3F action selection mechanism would help to better connect the motivation with the results.  \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ft0ekvwbrP", "reviewer": "Reviewer_NH1h", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes O3F, which is an action selection algorithm when doing an offline-to-online fine tuning. Since value estimates and policies we had was trained under a conservative objective (in many offline Rl algorithms), it may be too pessimistic in online fine-tuning, and may not improve much. To alleviate this issue, this paper proposes to select actions according to the upper confidence bound of Q functions.", "summary_of_the_review": "Overall, the problem setting and the motivation part were interesting, but the proposed algorithm O3F is not an algorithm that properly addresses the issues presented. The experiments are not also enough to show the improvement of O3F. In these sense, I believe that the contribution of this paper is limited and lean toward rejection.", "strength_and_weaknesses": "Strength:\n- interesting problem setting\n- a simple practical method that works\n\nWeakness:\n- interesting motivation but proposed algorithm is not aligned to it:\n\nThis paper starts with the motivation that the critics and the policies learned by pessimistic offline RL algoritms will not in general be improved well in the online fine-tuning case. The main example the paper presents is CQL, which has an explicit regularization term that makes Q conservative outside the data region, having lower value than the true Q value. It would have been the source of Q-function bias in Figure 2-(a), because suddenly removing the regularization will make all the Q estimates outside the data support invalid. In Figure 3 the paper shows all CQL, AWAC, IQL struggles when applied to online to offline fine-tuning and the objective suddenly changes to become non-pessimistic.\n\nHowever, the proposed algorithm O3F does not deal with such critic objective change, and all the interesting motivation suggested above is actually not addressed in the algorithm. O3F also mainly deals with LAPO algorithm, which does not seem to have an explicit regularization that gives a pessimism. Does it even have a similar problem that CQL suffers as mentioned above? I assume not.\n\n- Lack of originality of O3F:\n\nThe proposed algorithm O3F is now simply an action selection by first sampling actions from policy and ranking them according to upper confidence bounds. Selecting actions by ranking them according to Q values is a well known practical method used in BCQ/BEAR. Selecting actions by upper confidene bound is also a well known method. The suggested O3F is combination of those two, and no theoretical properties of such algorithm is desribed in the paper.\n\n- Experiment that does not clearly show the improvement of O3F:\n\nSince O3F is simply an action selection mechanism in online fine-tuning phase, I think the authors should have given more focus on a same algorithm with different action selections. In the result of figure 4, LAPO seems to simply outperform all the other algorithms from the start, and it is hard to see the improvement made by O3F. In figure 5 and 6, the difference between action selection algorithms are shown, but it is only shown on 3 domains. Although I agree on that the proposed algorithm will lead to performance improvement, the experiments do not support that well.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "cZvcB8-2DO", "reviewer": "Reviewer_b1rp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the problem of finetuning policies that were previously trained on offline datasets, a challenging problem since offline pre-training often leverages pessimistic objectives, which would hurt online fine-tuning in searching for new behaviors. To approach this, it proposes O3F, a method that works on top of some classes of offline RL algorithms by sampling actions based on a gaussian distribution and ranks them according to a critic ensemble and their disagreement. The work reports results on several continuous control benchmarks, showing better performance among other offline RL algorithms, and a few ablations regarding the action selection method and the ranking function.", "summary_of_the_review": "Given that the proposed contribution is mostly empirical, the experimental methodology does not provide enough evidence to clearly support the introduced method, as detailed in the weaknesses. Furthermore, the method presents critical limitations and does not seem to satisfy one of its main motivations (policy improvement due to exploration in the online fine-tuning phase). I do not believe that the paper is ready for acceptance, although I am open to changing my review in case of further experiments and some rewriting to better describe the challenges addressed by the proposed work.", "strength_and_weaknesses": "Strengths:\n\n- The online fine-tuning of offline pre-trained policies is a very important topic for RL nowadays, which makes the proposed work well-motivated;\n- The proposed solution is simple and seems to be effective. This makes it suitable for practitioners and real-world problems that rely on the same assumptions;\n\nMajor Concerns:\n- The proposed method relies on strong assumptions that are not discussed in the paper. First, the gaussian parametrization relies on a continuous action space. Secondly - and most important - it relies on a continuity property of the Q-functions (trained on the behavior policy data) for “closer” actions (i.e., numerically similar actions would have similar outcomes for the same state). This should be reasonable for some environments, but it is hard to evaluate the generality of this assumption without any evidence.\n\n- In the first paragraph, the work states as motivation the fact that fine-tuning with conservative objectives will cause the agent to keep doing the same behaviors, hurting the exploration needed for policy improvement. Nevertheless, one of the limitations of the proposed method (stated in Section 7) is that it only refines already learned behaviors and does not discover new ones. From this perspective, O3F does not seem to satisfy one of its main motivations. It is important to clearly state the challenges mitigated by the proposed method.\n\n- Figure 2 aims to provide empirical evidence for the proposed problem. However, there are some questionable methodological points. First, the results do not seem to show results with confidence intervals among different seeds, so it is not possible to assess statistical significance. Secondly, it is not clear why the baseline method is CQL, especially given that it is not even in the class of offline RL methods leveraged by O3F. Why not consider LAPO/IQL? Finally, it would also be helpful to show the results from standard, online SAC from scratch, as a sanity check to validate that the offline training is indeed necessary for this task.\n\n- Figure 3 aims to empirically refute the hypothesis that making the objective less conservative improves online fine-tuning. In this case, it is crucial to report the results from Wu et. al. [1], since it provides empirical evidence in the opposite direction. Perhaps the proposed heuristics in Figure 3 are not good enough to prove this hypothesis. A strong argument needs to be grounded on state-of-the-art results.\n\n- From Figure 4, it is difficult to credit the final good performance to the online finetuning or to the offline training, because the offline training methods differ. This causes different starting performances during online fine-tuning. If the idea here is to just compare the final best performance, it would be better to have a table with asymptotic performance. Furthermore, it is necessary to consider an online method (SAC) as a sanity baseline, to make sure the offline pretraining is indeed necessary for such environments. Also, you should also consider LAPO-only and O3F on top of IQL, to better understand the impact of the proposed method.\n\n- Figure 5 brings the most critical result, as it ablates the proposed method. Therefore, it is necessary to show the results in all environments (not only 3) to give enough empirical support. It also needs to show the performance of LAPO-only and IQL-only, which are the offline-only baselines.\n\n- Figure 6: analogous to the previous figure, it would be interesting to provide results from all environments. The current report shows mixed results, and it is not clear if the uncertainty is indeed necessary. Given that most of the reported results do not account for the uncertainty optimism scoring function, the argument towards “optimistic” action selection is weak.\n\nFurther Suggestions/Minor Concerns:\n\n- There is no study on the gaussian noise variance. It would be interesting to have any notion about how sensible is this parameter, especially because online fine-tuning often requires sample efficiency and hyperparameter tuning is prohibitively expensive.\n\n\nReferences \n\n[1] Wu el. al. Supported Policy Optimization for Offline Reinforcement Learning. NeurIPS, 2022.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7hhfyGbNXwF", "reviewer": "Reviewer_jHot", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper focuses on how to use offline data to accelerate online learning. The authors consider this problem from an exploration perspective. The intuition is that offline RL algorithms typically estimate action values conservatively to avoid overestimation errors due to OOD actions. Such conservative estimates potentially hurt exploration when transferring to an online learning setting. As a result, the authors propose to estimate the variance of action value estimates (which is done by using an ensemble of action value networks). The agent can take actions according to Q + variance term, similar to a UCB-style algorithm. The paper emphasizes that simple modifications of offline RL algorithms cannot do well when transferring to an online RL setting. Hence there is a strong motivation to use the proposed method. Experiments on frequently used offline RL testing beds are conducted to verify their effectiveness. \n", "summary_of_the_review": "The paper studies an interesting topic and proposes an intuitive and simple idea to achieve its goal. It lacks some important empirical results. I am willing to adjust the score based on the author's response. ", "strength_and_weaknesses": "Strength: \n\n1. The topic is interesting and potentially useful in practical scenarios. \n\n2. The proposed method is simple but seems to be new, and it makes intuitive sense. Specifically, the exploration angle to accelerate online RL seems to be new. \n\n3. The presentation is reasonably clear. \n\nWeaknesses: \n\nThe paper is mostly empirical, which is fine. But I want to see more empirical verifications, which I consider critical to make the proposed method persuasive. \n\nThe central goal is to use offline data to accelerate online RL, and the acceleration is done from the exploration perspective. \n\nAs a result, I think it is important to show a baseline without using offline data at all but using an ensemble for exploration. This is critical to validate the utility of offline data. \n\nTo verify such an ensemble approach is needed for exploration, it is also important to have a baseline using a naive & computationally more efficient exploration (say, do CEM only without using the score function computed by the ensemble) method.  \n\nI am willing to adjust my score based on the author’s response and other reviews. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject