Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration

## Abstract

Abstract Studying how to fine-tune offline reinforcement learning (RL) pre-trained policy is profoundly significant for enhancing the sample efficiency of RL algorithms. However, directly fine-tuning pre-trained policies often results in sub-optimal performance. This is primarily due to the distribution shift between offline pre-training and online fine-tuning stages. Specifically, the distribution shift limits the acquisition of effective online samples, ultimately impacting the online fine-tuning performance. In order to narrow down the distribution shift between offline and online stages, we proposed Q conditioned state entropy (QCSE) as intrinsic reward. Specifically, QCSE maximizes the state entropy of all samples individually, considering their respective Q values. This approach encourages exploration of low-frequency samples while penalizing high-frequency ones, and implicitly achieves State Marginal Matching (SMM), thereby ensuring optimal performance, solving the asymptotic sub-optimality of constraint-based approaches. Additionally, QCSE can seamlessly integrate into various RL algorithms, enhancing online fine-tuning performance. To validate our claim, we conduct extensive experiments, and observe significant improvements with QCSE ( about 13 % for CQL and 8 % for Cal-QL). Furthermore, we extended experimental tests to other algorithms, affirming the generality of QCSE.

## 1 Introduction

Figure: Figure 1: Demonstration of QCSE.
Refer to caption: x1.png

Offline RL holds a unique advantage over online RL, as it can be exclusively trained using pre-existing static offline RL datasets, eliminating the needs for interactions with the environment to acquire new online samples (Levine et al., ).
However, offline RL encounters specific limitations, including the challenges of learning sub-optimal performance, and the risks of overestimating out-of-distribution (OOD) state actions (Kumar et al., ), and limited exploration, particularly when the offline dataset is sub-optimal and fails to provide sufficient coverage (Lei et al., ). Therefore, we need to further fine-tune the offline pre-trained policy in online setting to address the aforementioned limitations (Fujimoto and Gu, ; Kostrikov et al., ; Wu et al., ; Mark et al., ).

Drawing the inspiration from modern machine learning, where pre-training is succeeded by online fine-tuning on downstream tasks (Brown et al., ; Touvron et al., ), it seems plausible to further enhance the performance of offline pre-trained policy through the process of online fine-tuning. However, offline algorithms often confine the offline pre-trained policy’s likelihood within offline support, resulting in unstable fine-turning performance due to distribution shifts between offline pre-training and online fine-tuning. To address these limitations, previous works attempt to introduce additional constrains (Kostrikov et al., ; Zheng et al., ; Zhao et al., ; Wu et al., ) during the online fine-turning stage. However, these algorithms still suffer from a demonstrated performance decline (as shown by in Figure 1) or asymptotic sub-optimality during the initial online fine-tuning stage (Nakamoto et al., ; Wu et al., ; Li et al., ). In addition to the predominant regularization-based approaches,  introduced a method aimed at aligning the value estimation during offline and online stages, thereby ensuring standard online fine-turning.

From a general perspective, all of these approaches can mitigate or resolve the negative impact caused by distribution shifts between offline and online stages. Specifically, constraint-based approaches maximize action prediction within the action support and then gradually loosen the constraints, thus alleviating policy collapse caused by distribution shifts. Regarding the value-aligned approach, ensuring standard online fine-tuning involves aligning the value estimation during both offline pre-training and online fine-tuning stages, which mitigate the distribution shift in value estimation between offline and online stages, consequently ensuring stable and asymptotic policy fine-tuning performance. Therefore, such insight opens a question that: Can we focus on exploring effective samples to alter the distribution shift between offline and online stages to guarantee the online fine-tuning performance?

To answer this question, we showcase the necessity of state entropy maximization in online fine-tuning stage: 1) Exploring diverse online samples help to diminish the negative impact of conservative pre-training. Specifically, it’s feasible to gather diverse samples to mitigate the conservatism of offline pre-trained policy (Luo et al., ) and further collect effective online samples to improve the fine-tuning performance. 2) Effective exploration is the crucial factor in enabling the asymptotically optimality. In particular, if the exploration can ensure that the marginal state distribution matches the target (expert) density $\textit{i}.\textit{e}.$ State Marginal Matching (SMM) (Lee et al., ), it can further guarantee that the empirical policy approaches the optimal policy.

Therefore, it’s intuitively plausible to reduce the distribution shift between the offline and online learning stages by gathering diverse samples, while approximately reaching the optimal performance via theoretically optimal exploration, such as State Marginal Matching (SMM). Based on this insight, we introduced Q conditioned State Entropy (QCSE), which involves separately estimating the state entropy conditioned on the Q estimates of each sample. By maximizing their average, we implicitly achieve SMM, aiding the empirical policy approaching the optimal policy. The advantage of QCSE lies in that it can approximately realize SMM, and thus naturally solve the asymptotic sub-optimality of constraint-based approaches. Meanwhile, QCSE inherently benefits from the exploratory advantages of VCSE (Kim et al., ). Specifically, Q-conditioned state entropy maximization can alleviate biased exploration towards low-value state regions as the state distribution becomes more uniform. Additionally, QCSE resolves the biased exploration of VCSE by considering differences in transitions that were previously overlooked, thus improving the online training stability. To summarize, the contribution of our research can be summerized as follows:

- •
We will imply that state entropy maximization can implicitly realize SMM in online stage, and further contribute to approaching the optimal performance.
- •
Based on the theoretical advantage of SMM, we proposed Q conditioned state entropy exploration (QCSE) that can implicitly realize SMM during online fine-tuning stage.
- •
Compared with VCSE, QCSE takes transition information into consideration, thereby protecting transitions from being disrupted by indiscriminate entropy maximization.

## 2 Related Work

#### Offline RL.

The notorious challenges in offline RL pertains to the mitigation of out-of-distribution (OOD) issues, which are a consequence of the distributional shift between the behavior policy and the training policy (Fujimoto et al., ). To effectively address this issue, 1) conservative policy-based model-free methods adopt the following approaches: Adding policy regularization (Fujimoto et al., ; Kumar et al., ; Wu et al., ; Liu et al., ), or implicit policy constraints (Peng et al., ; Siegel et al., ; Zhou et al., ; Chen et al., ; Wu et al., ; Liu et al., ). 2) And, conservative critic-based model-free methods penalize the value estimation of OOD state-actions via conducting pessimistic Q function (Kumar et al., ) or uncertainty estimation (An et al., ; Bai et al., ; Rezaeifar et al., ; Wu et al., ) or implicitly regularizing the bellman equation (Kumar et al., ). In terms of the model-base offline RL, it similarly train agent with distribution regularization (Hishinuma and Senda, ), uncertainty estimation (Yu et al., ; Kidambi et al., ; Lu et al., ), and value conservation (Yu et al., ). In our research, due to the remarkable sampling efficiency and outstanding performance of model-free algorithms in both offline and online RL settings, and we prove that QCSE satisfy the guarantee of Soft-Q optimization (theorem 4.1), thus we select Conservative Q-Learning (CQL) and Calibrated Q-Learning (Cal-QL) as our primary baseline methods. Additionally, to conduct a thorough assessment of the effectiveness of our proposed approaches, we have also expanded our evaluation to encompass a diverse set of other model-free algorithms, including Soft-Actor-Critic (SAC) (Haarnoja et al., ), Implicit Q-learning (IQL) (Kostrikov et al., ), TD3+BC (Fujimoto and Gu, ), and AWAC (Nair et al., ).

#### Offline-to-Online RL.

Previous researches have demonstrated that offline RL methods offer the potential to expedite online training, a process that involves incorporating offline datasets into online replay buffers (Nair et al., ; Vecerik et al., ; Todd Hester and et al., ) or initializing the pre-trained policy to conduct online fine-tuning (Kostrikov et al., ; Beeson and Montana, ). However, there exhibits worse performance when directly fine-tuning the offline pre-trained policy (Nakamoto et al., ; Lee et al., ), and such an issue can be solved by adapting a balanced replay scheme aggregated with pessimistic pre-training (Lee et al., ), or pre-training with pessimistic Q function and fine-tuning with exploratory methods (Wu et al., ; Mark et al., ; Nakamoto et al., ). In particular, our approach QCSE differs from these methods in that it enhances online fine-tuning solely by augmenting online exploration. (More related works have been added to Appendix G.1)

#### Online Exploration.

Recent advances in the studies of exploration can obviously improve online RL sample efficiency, among that, remarkable researches include injecting noise into state actions(Lillicrap et al., ) or designing intrinsic reward by counting visitation or errors from predictive models (Badia et al., ; Sekar et al., ; Whitney et al., ; Burda et al., ). In particular, the approaches most related to our study are to utilize state entropy as an intrinsic reward (Kim et al., ; Seo et al., ).

## 3 Preliminary

#### Reinforcement Learning (RL).

We formulate RL as Markov Decision Process (MDP) tuple i.e.$\mathcal{M}=(\mathcal{S},\mathcal{A},r,d_{\mathcal{M}},p_{0},\gamma)$. Specifically, $p_{0}$ denotes the distribution of initial state (observation), $\mathbf{s}_{0}\sim p_{0}$ denotes initial observation, $\mathcal{S}$ denotes the observation space, $\mathcal{A}$ denotes the actions space, $r(\mathbf{s},\mathbf{a}):\mathcal{S}\times\mathcal{A}\mapsto\mathbb{R}$ denotes the reward function, $d_{\mathcal{M}}(\mathbf{s}^{\prime}|\mathbf{s},\mathbf{a}):\mathcal{S}\times
\mathcal{A}\rightarrow\mathcal{S}$ denotes the transition function (dynamics), and $\gamma\in[0,1]$ denotes discount factor.
The goal of RL is to obtain the optimal policy $\pi^{*}:\mathcal{S}\mapsto\mathcal{A}$ to maximize the accumulated discounted return $\textit{i}.\textit{e}.$ $\pi^{*}:=\arg\max_{\pi}\mathbb{E}_{\tau\sim\pi}[R(\tau)]$, where $\mathbb{E}_{\tau\sim\pi}[R(\tau)]=\mathbb{E}[\sum_{t=0}^{t=T}\gamma^{t}r(
\mathbf{s}_{t},\mathbf{a}_{t})]$, and $\tau=\big{\{}\mathbf{s}_{0},\mathbf{a}_{0},r_{0},\cdots,\mathbf{s}_{T},\mathbf
{a}_{T},r_{T}|\mathbf{s}_{0}\sim p_{0},\mathbf{a}_{t}\sim\pi(\cdot|\mathbf{s}_
{t}),\mathbf{s}_{t+1}\sim d_{\mathcal{M}}(\cdot|\mathbf{s}_{t},\mathbf{a}_{t})
\big{\}}$ is the rollout trajectory.
Generally, RL has to estimate a Q function $\textit{i}.\textit{e}.$ $Q(\mathbf{s},\mathbf{a})=\mathbb{E}_{\tau\sim\pi}[\sum_{t=0}^{T}\gamma^{t}r(
\mathbf{s}_{t},\mathbf{a}_{t})|\mathbf{s}_{0}=\mathbf{s},\mathbf{a}_{0}=
\mathbf{a}]$, and a value function by $V(\mathbf{s}):=\mathbb{E}_{\mathbf{a}\sim\mathcal{A}}[Q(\mathbf{s},\mathbf{a})]$ to assistant in updating policy $\pi$. In this research, we mainly focus on improving model-free algorithms to conduct offline-to-online RL setting.

#### Model-free Offline RL.

Typically, model-free RL algorithms alternately optimize policy with Q-network i.e. $\pi:=\arg\max_{\pi}\mathbb{E}_{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim\pi(
\cdot|\mathbf{s})}[Q(\mathbf{s},\mathbf{a})]$,
and conduct policy evaluation by the Bellman equation iteration i.e. $Q:=\arg\min_{Q}\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})\sim
\mathcal{D}}[(Q(\mathbf{s},\mathbf{a})-\mathcal{B}_{\mathcal{M}}Q(\mathbf{s},
\mathbf{a}))^{2}]$, where $\mathcal{B}_{\mathcal{M}}Q(\mathbf{s},\mathbf{a}):=r(\mathbf{s},\mathbf{a})+
\gamma\cdot Q(\mathbf{s}^{\prime},\pi(\cdot|\mathbf{s}^{\prime}))$.
In particular, model-free offline RL aims to learn from the static RL datasets $\mathcal{D}$ collected by behavior policy $\pi_{\beta}$ without access to the environment to collect new trajectories, thus suffers from the OOD state actions and sub-optimal performance, therefore, it’s necessary to further fine-tune the pre-trained policy online to
further alleviate the limitations of offline pre-trained policy.

#### Drawbacks of previous offline-to-online algorithms.

However, directly fine-tuning the offline pretrained policy may encounter distribution shift issues, potentially leading to policy collapse. Despite that constraint-based approaches can facilitate stable online fine-tuning, they may suffer from sub-asymptotic optimality. Different from the majority of offline-to-online approaches, we propose Q conditioned state entropy exploration (QCSE), which alleviates distribution shift issue by implicitly realizing SMM through Q conditioned state entropy maximization.

## 4 Q conditioned state entropy maximization (QCSE)

Previously, suggests that error or count based exploration approaches (Pathak et al., ; Burda et al., ) are insufficient to converge to a singular exploratory policy, and propose SMM to realize converging to singular exploratory policy. Building on the theoretical advantages of SMM, we aim to enhance online exploration through state entropy maximization, thereby improving online fine-tuning performance.
{definition}[Marginal State distribution] Given the rollout trajectory $\tau\sim\pi$, we define the marginal state distribution of $\pi$ as $\rho_{\pi}(\mathbf{s})=\mathbb{E}_{\mathbf{s}_{0}\sim p_{0},\mathbf{a}_{t}\sim
\pi(\cdot|\mathbf{s}_{t}),\mathbf{s}_{t+1}\sim d_{\mathcal{M}}(\cdot|\mathbf{s
}_{t},\mathbf{a}_{t})}[\frac{1}{N}\sum_{t=1}^{T}\mathbbm{1}(\mathbf{s}_{t}=
\mathbf{s})].$
{definition}[State Marginal Matching]
Given the target (optimal) state density $p^{*}(\mathbf{s})$ and the offline initialized marginal state distribution $\rho_{\pi}(s)$. We define State Marginal Matching (SMM) as optimizing policy to minimize $D_{\rm KL}[\rho_{\pi}(\mathbf{s})||p^{*}(\mathbf{s})]$, $\textit{i}.\textit{e}.$ $\pi:=\arg\min_{\pi}D_{\rm KL}[\rho_{\pi}(\mathbf{s})||p^{*}(\mathbf{s})]$, where $D_{\rm KL}$ denotes Kullback-Leibler (KL) divergence (^1^11For the given distributions $p(x)$ and $p(x)$ on the domain $X$. KL divergence denotes $D_{\rm KL}[p(x)||q(x)]=\mathbb{E}_{x\sim X}[p(x)\log\frac{p(x)}{q(x)}]$.).
{definition}[Critic Conditioned State Entropy] Given empirical policy $\pi\in\Pi$, its critic network $Q(\mathbf{s},\mathbf{a}):\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$, and given state density of current empirical policy: $\rho_{\pi}(\mathbf{s})$, where $\int_{\mathbf{s}\in\mathcal{D}}\rho_{\pi}(\mathbf{s})=1$. We define the critic conditioned entropy as $\mathcal{H}_{\pi}(\mathbf{s}|Q)=\mathbb{E}_{\mathbf{s}\sim\rho_{\pi}}[-\log p(
\mathbf{s}|Q(\mathbf{s},\pi(\cdot|\mathbf{s})))]$.
To explain why state entropy maximization helps to address the challenges of offline-to-online RL, we first define the basic concepts and notations, specifically, we define the marginal state distribution as Definition 4, and the process of the marginal state distribution $\rho_{\pi}(\mathbf{s})$ approaching target density $p^{*}(\mathbf{s})$ $\textit{i}.\textit{e}.$ state marginal matching (SMM) as Definition 1. Subsequently, we illustrate how maximizing state entropy can approximate SMM during the online fine-tuning stage, thereby facilitating the acquisition of the optimal policy.

#### State entropy maximization implicitly realize SMM during the online fine-tuning stage.

To elaborate on the relationship between state entropy maximization and SMM, we derive the process of maximizing the state entropy of the empirical policy $\pi$ $\textit{i}.\textit{e}.$ $\max\mathbb{E}_{\mathbf{s}\sim\rho_{\pi}}[\mathcal{H}_{\pi}[\mathbf{s}]]:=\max
\mathbb{E}_{\mathbf{s}\sim\rho_{\pi}}[-\log p(\mathbf{s})]$, $s.t.\max\mathbb{E}_{\tau\sim\pi}[R(\tau)]$, and arrive at the following expression:

$$ $\displaystyle\max\limits_{\rho_{\pi}\atop{s.t.\max\mathbb{E}_{\tau\sim\pi}[R( \tau)]}}\mathbb{E}_{\mathbf{s}\sim\rho_{\pi}}[\mathcal{H}_{\pi}[\mathbf{s}]] \leq\max\limits_{\rho_{\pi}\atop{s.t.\max\mathbb{E}_{\tau\sim\pi}[R(\tau)]}} \underbrace{\int_{\mathbf{s}\sim\mathcal{S}_{1}}-\rho_{\pi}(\mathbf{s})\log \rho_{\pi}(\mathbf{s})}_{J_{\rm term1}}+\underbrace{\int_{\mathbf{s}\sim \mathcal{S}_{2}}-p^{*}(\mathbf{s})\log p^{*}(\mathbf{s})}_{J_{\rm term2}},$ (1) $$

where $p^{*}(\mathbf{s})$ denotes the target density, $\rho_{\pi}$ denotes the marginal state distribution initialized by the offline dataset, as defined in Definition 4. $\mathcal{S}_{1}$ denotes domain where $\rho_{\pi}(\mathbf{s})>p^{*}(\mathbf{s})|_{\mathbf{s}\sim\mathcal{S}_{1}}$, and $\mathcal{S}_{2}$ denotes domain where $\rho_{\pi}(\mathbf{s})\leq p^{*}(\mathbf{s})|_{\mathbf{s}\sim\mathcal{S}_{2}}$. proof of Equation 1 see Appendix B.1.

Since $p^{*}(\mathbf{s})$ remains invariant during the training process, maximizing $J_{\rm term2}$ is equivalent to narrowing down the domain $\mathcal{S}_{2}$. Meanwhile, maximizing $J_{\rm term1}$ is equivalent to penalizing $\rho_{\pi}(\mathbf{s})|_{\mathbf{s}\sim\mathcal{S}_{1}}$. Consequently, it penalizes exploration in $\mathcal{S}_{2}$ while guiding $\rho_{\pi}$ towards $p^{*}$ during exploration in $\mathcal{S}_{1}$. Both $J_{\rm term_{1}}$ and $J_{\rm term_{2}}$ narrow the gap between $\rho_{\pi}(\mathbf{s})$ and $p^{*}(\mathbf{s})$. Therefore, state entropy maximization approximately solves the distribution shift issue and facilitates the approximation of SMM during online fine-tuning stage. Furthermore, it encourages the empirical policy approaching the optimal policy. Based on this analysis, we introduce QCSE that computes Q conditioned entropy as intrinsic reward to narrow down the distribution shift between offline and online stages.

#### Implementation of QCSE.

The mathematical formulation of QCSE, as shown in Equation 2, involves calculating the Q conditioned state entropy as intrinsic reward to encourage the agent to explore the environment uniformly. In particular, we use $\rm Tanh$ to limit the magnitude of the $r^{\rm QCSE}$, preventing it from overwhelming the task reward and thereby reducing biases during critic updating.

$$ $\small\begin{split}\small r^{\mathrm{mod}}(\mathbf{s},\mathbf{a})=\lambda\cdot \mathrm{Tanh}(\underbrace{\mathcal{H}(\mathbf{s}|\mathrm{min}(Q_{\phi_{1}}( \mathbf{s},\mathbf{a}),Q_{\phi_{2}}(\mathbf{s},\mathbf{a})))}_{r^{\rm QCSE}})+ r(\mathbf{s},\mathbf{a})\bigg{|}_{(\mathbf{s},\mathbf{a})\sim\mathcal{D}_{ \mathrm{online}}},\end{split}$ (2) $$

where $\phi_{1}$ and $\phi_{2}$ are separately the params of double Q Networks. However, we cannot directly obtain $\rho_{\pi}(\mathbf{s})$, therefore, we cannot directly calculate state entropy. In order to approximate $\rho_{\pi}(\mathbf{s})$, we refer to (Kim et al., ), and use the Kraskov-Stögbauer-Grassberger (KSG) estimator (Kraskov et al., ) to approximate state entropy as the intrinsic reward, $\textit{i}.\textit{e}.$ Equation 3.

$$ $\small\begin{split}\small r^{\rm QCSE}(\mathbf{s},\mathbf{a})&=\frac{1}{d_{s}} \phi(n_{\hat{Q}_{\phi}}(i)+1)+\log 2\cdot\max(||\mathbf{s}_{i}-\mathbf{s}_{i}^ {knn}||,||\hat{Q}(\mathbf{s},\mathbf{a})-\hat{Q}(\mathbf{s},\mathbf{a})^{knn}| |)\bigg{|}_{(\mathbf{s},\mathbf{a})\sim\mathcal{D}_{\rm online}},\end{split}$ (3) $$

where $\hat{Q}(\mathbf{s},\mathbf{a})=\min(Q_{\phi_{1}}(\mathbf{s},\mathbf{a}),Q_{
\phi_{2}}(\mathbf{s},\mathbf{a}))$, and $\mathbf{s}^{knn}_{i}$ is the $n_{\mathbf{s}}(i)$-th nearest neighbor of $\mathbf{s}_{i}$ in the space $\mathcal{S}$ spanned by $\mathbf{s}_{i}$. Specifically, $n_{\mathbf{s}}(i)$ represents the number of neighborhoods around $\mathbf{s}_{i}$, where the distances to $\mathbf{s}_{i}$ are less than $\frac{\epsilon(\mathbf{s}_{i})}{2}$. And $\epsilon(\mathbf{s}_{i})$ represents twice the distance from $\mathbf{s}_{i}$. $n_{\hat{Q}}(i)$ and $\hat{Q}^{knn}_{i}$ can be similar defined by replacing $\mathbf{s}$ by $\hat{Q}$. For more detail information, please refer to page 4, line 8 to 9 of (Kim et al., )).

### 4.1 Advantages of QCSE

#### Monotonic of QCSE.

Despite the concise and simple mathematical form of QCSE, QCSE has advantage lies in guaranteeing the monotonic Soft-Q optimization (supported by theorem 4.1), thus can be seamlessly utilized in conjugation with Soft-Q based RL algorithms including CQL-SAC (Kumar et al., ) etc.
{theorem}[Converged QCSE Soft Policy is Optimal]
Repetitive using lemma B.1 and lemma 7 to any $\pi\in\Pi$ leads to convergence towards a policy $\pi^{*}$. And it can be proved that $Q^{\pi^{*}}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\geq Q^{\pi}\left(\mathbf
{s}_{t},\mathbf{a}_{t}\right)$ for all policies $\pi\in\Pi$ and all state-action pairs $\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\in\mathcal{S}\times\mathcal{A}$, provided that $|\mathcal{A}|<\infty$. Proof see Appendix B.1.

Figure: Figure 2: Q condition vs. V condition. In this experiment, we selected AWAC as the base algorithm and compared using V network and Q network to calculate the intrinsic reward’s condition. The experimental results indicate that using the Q-network to compute the condition leads to overall better performance for AWAC. (Nair et al., ) points out that AWAC demonstrates poor online fine-tuning performance.
Refer to caption: x2.png

#### Q condition protects transitions from being disrupted by entropy maximization.

Another advantage of QCSE is that it utilizes Q instead of V as the intrinsic reward’s condition. This approach incorporates transition information, thereby preserving transitions that disrupted by indiscriminate state entropy maximization. For instance, assuming that there exists two transitions $T_{1}=(\mathbf{s},\mathbf{a}_{1},\mathbf{s}_{1})$ and $T_{2}=(\mathbf{s},\mathbf{a}_{2},\mathbf{s}_{2})$. Since $T_{1}$ and $T_{2}$ have the same current observation $\mathbf{s}$, they are related to the same value conditioned intrinsic reward $-\log(\mathbf{s}|V(\mathbf{s}))$, therefore, low-value transitions still receive relatively high intrinsic rewards, subsequently biasing the Q estimation, and further impacting decision-making. However, by computing intrinsic rewards conditioned on $Q(\mathbf{s},\mathbf{a})$ and maximizing state entropy with distinct Q values separately, QCSE can further mitigate the biased exploration issues left unresolved by VCSE. To validate our claims, we chose AWAC as baseline, and separately utilize both Q-network and V-network to compute the intrinsic reward’s condition, conducting tests on tasks sourced from Gym-Mujoco domain. As shown in Figure 2, using Q to compute condition has better performance compared to using V.

## 5 Practical implementation

We follow the standard offline-to-online RL process to test QCSE. Specifically, we first pre-train the policy with the selected algorithm on a specific offline dataset. Then, we further fine-tune the pre-trained policy online using QCSE. Finally, we test using the policy fine-tuned online. In terms of the real implementation, QCSE augments the reward of online dataset by calculating the Q conditional state entropy (via Equation 3) which is highly compatible with Q-ensemble or double-Q RL algorithms. For algorithms that do not employ Q-ensemble or double Q, it is still possible to use QCSE; however, they may not benefit from the advantages associated with Q-ensemble, as clarified in the following section. When it comes to the hyper-parameters of QCSE, setting $\lambda$ in Equation 2 to 1 is generally sufficient to improve the performance of various baselines on most tasks. However, it is important to note that QCSE’s effectiveness is influenced by the number of k-nearest neighbor (knn) clusters, as we have demonstrated in our ablation study. Additionally, for parameters unrelated to QCSE, such as those of other algorithms used in conjunction with QCSE, it is not necessary to adjust the original parameters of these algorithms (see more details including hyper-parameters, model architecture, compiting recourese, e.g. in Appendix D.6). In the following section we will conduct experimental evaluation to validate the effectiveness of QCSE.

## 6 Evaluation

Our experimental evaluation aims to achieve the following primary objectives: 1) Investigating the effectiveness of QCSE utilizing soft-Q based algorithms in facilitating offline-to-online RL and assessing its performance. 2) Assessing the viability of integrating QCSE with another model-free algorithms to enhance their sample efficiency. 3) Conducting diverse experiments to showcase the performance differences or relationships between QCSE and various exploration methods, including SE (Seo et al., ), VCSE (Kim et al., ), RND (Burda et al., ), and SAC (Haarnoja et al., ). 4) Conducting a series of ablation experiments to verify the effectiveness of QCSE. We first introduce our tasks and baselines.

#### Task and Datasets.

We experiment with 12 tasks from mujoco (Brockman et al., ) and Antmaze in D4RL (Fu et al., ). The selected tasks cover various aspects of RL challenges, including reward delay and high-dimensional continuous control. Specifically: 1) In the Antmaze tasks, the goal is to control a quadruped robot to reach the final goal. Notably, this agent does not receive an immediate reward for its current decision but instead only receives a reward of +1 upon successfully reaching the goal or terminating. This setup presents a form of reward delay, making these tasks adapt to evaluate the long horizontal decision-making capability of algorithms. 2) In Gym-locomotion tasks, the goal is to enhance the agent’s localmotion capabilities, presenting a contrast to the Antmaze domain where Gym-Mujoco tasks feature high-dimensional decision-making spaces. Also, the agent in Gym-Mujoco has the potential to obtain rewards in real time.

Figure: Figure 3: Online fine-tuning curve on selected tasks. We tested QCSE by comparing Cal-QL-QCSE, CQL-QCSE to Cal-QL, CQL on selected tasks in the Gym-Mujoco and Antmaze domains, and then reported the average return curves of multi-time evaluation. As shown in this Figure, QCSE can improve Cal-QL and CQL’s offline fine-tuning sample efficiency and achieves better performance than baseline (CQL and Cal-QL without QCSE) over all selected tasks.
Refer to caption: x7.png

#### Baselines for Comparison.

For convenience, we name any algorithm Alg paired with QCSE as Alg-QCSE. Now we introduce our baselines. We primarily compare CQL-QCSE and Cal-QL-QCSE to CQL (Kumar et al., ) and Cal-QL (Nakamoto et al., ). We also verify that QCSE can be broadly plugged into various model-free algorithms including SAC (Haarnoja et al., ), IQL (Kostrikov et al., ), TD3+BC (Fujimoto and Gu, ), and AWAC (Nair et al., ), thus improving their online fine-tuning performance. In particular, Cal-QL is the recent state-of-the-art (SOTA) offline-to-online RL algorithm that has been adequately compared to multiple offline-to-online methods (O3F (Mark et al., ), ODT (Zheng et al., ), and mentioned baselines), and demonstrated obvious advantages.

**Table 1: Normalized score after online fine-tuning. We report the online fine-tuned normalized return. QCSE obviously improves the performance of CQL and Cal-QL. In particular, CQL-QCSE (mean score of 92.5) is the best out of the 12 selected baselines. Notably, part of Antmaze’s baseline results are quoted from existing studies. Among them, AWAC’s results are quoted from (Kostrikov et al., ) and CQL’s results are quoted from (Nakamoto et al., ). Addtionally we have report the**
| Offline-to-online Tasks | IQL | AWAC | TD3+BC | CQL | CQL+QCSE | Cal-QL | Cal-QL+QCSE |
| --- | --- | --- | --- | --- | --- | --- | --- |
| antmaze-large-diverse | 59 | 00 | 00 | 89.2$\pm$3.2 | 89.8$\pm$3.2 | 86.3$\pm$0.2 | 94.5$\pm$1.7 |
| antmaze-large-play | 51 | 00 | 00 | 91.7$\pm$3.8 | 92.6$\pm$ 1.3 | 83.3$\pm$9.0 | 95.0$\pm$1.1 |
| antmaze-medium-diverse | 92 | 00 | 00 | 89.6$\pm$0.3 | 98.9$\pm$0.2 | 96.8$\pm$1.0 | 99.6$\pm$0.1 |
| antmaze-medium-play | 94 | 00 | 00 | 97.7$\pm$0.2 | 99.4$\pm$0.4 | 95.8$\pm$0.9 | 98.9$\pm$0.6 |
| Total (Antmaze) | 296 | 00 | 00 | 368.2 | 380.7 | 352.2 | 388.0 |
| halfcheetah-medium | 57 | 67 | 49 | 69.9$\pm$1.0 | 87.9$\pm$2.3 | 45.6$\pm$0.0 | 46.9$\pm$0.0 |
| walker2d-meidum | 93 | 91 | 82 | 123.1$\pm$4.0 | 130.0$\pm$0.0 | 80.3$\pm$0.4 | 90.0$\pm$3.6 |
| hopper-medium | 67 | 101 | 55 | 56.4$\pm$0.4 | 62.4$\pm$ 1.3 | 55.8$\pm$0.7 | 61.7$\pm$2.6 |
| ant-medium | 113 | 121 | 43 | 123.8$\pm$1.5 | 136.9$\pm$1.6 | 96.4$\pm$0.3 | 104.2$\pm$3.0 |
| halfcheetah-medium-replay | 54 | 44 | 49 | 42.0$\pm$1.9 | 55.6$\pm$0.5 | 32.6$\pm$0.6 | 32.9$\pm$1.7 |
| walker2d-medium-replay | 90 | 73 | 90 | 98.1$\pm$5.7 | 112.7$\pm$1.5 | 27.2$\pm$8.7 | 47.7$\pm$6.4 |
| hopper-medium-replay | 91 | 56 | 88 | 17.6$\pm$11.1 | 27.1$\pm$15.8 | 13.8$\pm$2.1 | 7.7$\pm$4.1 |
| ant-medium-replay | 123 | 127 | 127 | 84.7$\pm$3.6 | 116.6$\pm$3 | 83.1$\pm$0.7 | 73.1$\pm$8.3 |
| Total (Gym Mujoco) | 688 | 680 | 583 | 615.6 | 729.2 | 434.8 | 464.2 |
| Avg (Antmaze&Gym Mujoco) | 82.2 | 56.7 | 48.6 | 82.0 | 92.5 | 65.6 | 71.0 |

### 6.1 Main results

#### Can QCSE improve offline-to-online RL?

It is clear that QCSE enhances the online fine-tuning performance of both CQL (improve 13$\%$) and Cal-QL (improve 8$\%$). This is evidenced by the stable and progressed fine-tuning curves shown in Figure 3 and the fine-tuned results shown in Table 1.
Specifically, CQL and Cal-QL combined with QCSE exhibit fewer gaps in the training curves, indicating that QCSE helps to reduce the domain shift between the offline and online stages, thereby benefiting the online fine-tuning process. Additionally, CQL-QCSE outperforms both CQL and Cal-QL on nearly all selected tasks, supporting our hypothesis that enhancing the agent’s exploration capabilities by state entropy maximization can help guarantee the asymptotic optimality of the online fine-tuned policy via implicitly achieving State Marginal Matching (SMM). Furthermore, when examining tasks with larger distribution shifts, such as medium-replay and medium, CQL-QCSE demonstrates better asymptotic optimality than CQL, reinforcing this claim. (In Table 1, we present the average training results from the last 20 to 100 steps across multiple runs for Antmaze and gym-medium. For gym-medium-replay, we report the average of the maximum values obtained from multiple runs.)

Figure: Figure 4: Performance of Alg-QCSE. We test QCSE with AWAC, TD3+BC, and IQL on selected Gym-Mujoco tasks, QCSE can obviously improve the performance of these algorithms on selected Gym-Mujoco tasks, showing QCSE’s versatility.
Refer to caption: x20.png

#### Can QCSE be plugged into other model-free algorithms?

To address the second question, we conducted comparative experiments, assessing the performance of our QCSE across various model-free algorithms such as TD3+BC, AWAC, IQL, and SAC. Notably, our QCSE, functioning as a plug-and-play reward augmentation algorithm, eliminates the need for additional modifications ($\textit{i}.\textit{e}.$ we seamlessly integrate QCSE to modify the reward during training with those algorithms). Illustrated in Figure 4, the incorporation of QCSE leads to performance improvements across almost all algorithms during online fine-tuning. This demonstrates the versatile applicability of QCSE to a wide range of RL algorithms, extending beyond the scope of CQL or Cal-QL.

#### Is QCSE more effective than previous exploration methods in the offline-to-online setting?

As shown in Figure 2, QCSE outperforms VCSE. We further conduct a comprehensive comparison between QCSE and other exploration approaches, including SE and RND. Additionally, we explore the impact on performance when combining SAC with QCSE. As depicted in Figure 5 (a), the combination of SAC with QCSE achieves enhanced performance on the selected Gym-Mujoco tasks. Such improvement further validates Theorem 4.1, supporting that QCSE ensures monotonic soft Q optimization. Meanwhile, as shown in Figure 5 (b), algorithms combined with QCSE exhibit superior performance and greater stability compared to other exploration methods, including SE and RND, across all selected tasks.

Figure: Figure 5: Performance comparison for variety of exploration Methods. (a) Online fine-tuning performance difference between SAC and SAC-QCSE. (b) Online fine-tuning performance difference between various exploration methods with IQL and AWAC.
Refer to caption: x25.png

#### Can QCSE surpass previous efficient offline-to-online algorithms?

In order to more intuitively demonstrate the effectiveness of QCSE, we replaced QCSE with a series of past efficient offline-to-online algorithms and conducted comparisons. As shown in Figure 6, we select CQL as the base algorithm and aggregate it with QCSE, APL (Zheng et al., ), PEX (Zhang et al., ), SUNG (Guo et al., ) and BR (Lee et al., ) to test on tasks of Antmaze and Gym-Mujoco (medium, medium-replay) domains, and CQL-QCSE archives the best performance (85.6) over all selected baselines, which demonstrating QCSE’s competitive performance.

### 6.2 Ablations

Figure: Figure 6: Performance Comparison. CQL+efficient offline-to-online approaches.
Refer to caption: x29.png

#### Effect of Hyperparameter.

We primarily investigate the effect of the hyperparameter in Equation 13. Specifically, the Q conditioned state entropy is approximated using the KSG estimator, where the number of state clusters serves as a crucial hyperparameter. As shown in Figure 7, the performance can indeed be influenced by the number of state clusters, and a trade-off exists among the sizes of these state clusters. For instance, the optimal cluster settings of walker2d and hopper are saturated around 20 and 10, respectively. In contrast, a task like amaze-large-diverse requires a larger number of clusters (about 25). We consider the main reason is that different tasks require varying degrees of exploration, and thus need different cluster settings.

Figure: Figure 7: We evaluate the performance when varying the number of state clusters. We assess QCSE by configuring different sizes of k-nearest neighbor (knn) clusters and subsequently observe the impact of these parameter settings on online fine-tuning, and it can be observed that the choice of knn cluster settings exerts a notable influence on QCSE’s performance.
Refer to caption: x30.png

#### Scaled Ablations.

We utilize AWAC to further validate the effectiveness of QCSE. Specifically, we examine the impact of utilizing an offline pre-trained Q-network versus a randomly initialized Q-network to compute intrinsic rewards, as shown in Figure 8 (a). Offline pre-trained Q as condition performs better than randomly initialized Q as condition. On the other hand, in Figure 8 (b), we illustrate the progression of state entropy throughout the training process. Specifically, we can observe that the state entropy of AWAC-QCSE eventually exceeds AWAC baseline, indicating the influence of QCSE on state entropy.

Figure: Figure 8: (a) Ablation experiments to validate the impact of pre-trained Q network. (b) Quantitative results on the agent’s state entropy.
Refer to caption: x34.png

## 7 Conclusions

We propose a generalized offline-to-online framework called QCSE. On the theoretical aspect, we demonstrate that QCSE can implicitly realize SMM. Meanwhile, we showcase QCSE can guarantee the monotonicity of soft-Q optimization. On the experimental aspect, QCSE leads to improvements for both CQL and Cal-QL, validating our theoretical claims. We also extend the tests of QCSE to other model-free algorithms, and experimental results showed that QCSE performs better when combined with other model-free algorithms, demonstrating its generality.

#### Limitations and future work.

Although we propose that maximizing state entropy can approximate State Marginal Matching (SMM) to ensure fine-tuned performance, we do not provide an explicit approach to address distribution shift issues. In the future, we aim to enhance QCSE to explicitly achieve SMM and assess whether this leads to improved fine-tuned performance.

## Appendix A Ethical Claim

Despite the potential of offline RL to learn from the static datasets without the necessity to access the online environment, the offline method does not guarantee the optimal policy. Therefore, online fine-tuning is essential for policy improvement. In this study, we propose a novel and versatile reward augmentation framework, named Q conditioned state entropy maximization (QCSE) which can be seamlessly plugged into various model-free algorithms. We believe our approach is constructive and will enhance the sample efficiency of offline-to-online RL. Additionally, given that QCSE is an integrated algorithm, we also believe it can broadly and readily benefit existing algorithms.

## Appendix B Theoretical Analysis

In this section, we provide the supplementary mathematical analysis for QCSE.

### B.1 State entropy maximization and state marginal matching

In this section, we will discuss why maximizing state entropy contributes to achieving state marginal matching.

$$ $\begin{split}\max\limits_{\rho_{\pi}(\mathbf{s})\atop{s.t.\max\mathbb{E}_{\tau \sim\pi}[R(\tau)]}}\mathcal{H}[\mathbf{s}]&=\max\limits_{\rho_{\pi}(\mathbf{s} )\atop{s.t.\max\mathbb{E}_{\tau\sim\pi}[R(\tau)]}}\int_{\mathbf{s}\sim\mathcal {S}}-\rho_{\pi}(\mathbf{s})\log\rho_{\pi}(\mathbf{s})\\ &=\max\limits_{\rho_{\pi}(\mathbf{s})\atop{s.t.\max\mathbb{E}_{\tau\sim\pi}[R( \tau)]}}\int_{\mathbf{s}\sim\mathcal{S}_{1}}-\rho_{\pi}(\mathbf{s})\log\rho_{ \pi}(\mathbf{s})+\int_{\mathbf{s}\sim\mathcal{S}_{2}}-\rho_{\pi}(\mathbf{s}) \log\rho_{\pi}(\mathbf{s})\\ &\leq\max\limits_{\rho_{\pi}(\mathbf{s})\atop{s.t.\max\mathbb{E}_{\tau\sim\pi} [R(\tau)]}}\underbrace{\int_{\mathbf{s}\sim\mathcal{S}_{1}}-\rho_{\pi}(\mathbf {s})\log\rho_{\pi}(\mathbf{s})}_{\rm term1}+\underbrace{\int_{\mathbf{s}\sim \mathcal{S}_{2}}-p^{*}(\mathbf{s})\log p^{*}(\mathbf{s})}_{\rm term2},\\ \end{split}$ (4) $$

where $p^{*}(\mathbf{s})$ denotes the target density, $\rho_{\pi(\mathbf{s})}$ denotes the marginal state distribution initialized by the offline dataset, as defined in Definition 4. $\mathcal{S}_{1}$ denotes domain where $\rho_{\pi}(\mathbf{s})>p^{*}(\mathbf{s})|_{\mathbf{s}\sim\mathcal{S}_{1}}$, and $\mathcal{S}_{2}$ denotes domain where $\rho_{\pi}(\mathbf{s})\leq p^{*}(\mathbf{s})|_{\mathbf{s}\sim\mathcal{S}_{2}}$.
In this section, we examine the mathematical viability of the QCSE framework, focusing on two key aspects: 1) Guarantee of Soft policy optimization 2) Prevention of OOD state actions.

We first introduce the modified soft Q Bellman backup operator, denoted as Equation 5,

$$ $\mathcal{T}_{QCSE}^{\pi}Q\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\triangleq r \left(\mathbf{s}_{t},\mathbf{a}_{t}\right)+r^{QCSE}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)+\gamma\mathbb{E}_{\mathbf{s}_{t+1}\sim p}\left[V\left( \mathbf{s}_{t+1}\right)\right]$ (5) $$

In this equation, the term $V\left(\mathbf{s}_{t}\right)=\mathbb{E}_{\mathbf{a}_{t}\sim\pi}\left[Q\left(
\mathbf{s}_{t},\mathbf{a}_{t}\right)-\log\pi\left(\mathbf{a}_{t}\mid\mathbf{s}
_{t}\right)\right]$ is defined.

[Soft Policy Evaluation with QCSE.]
Given the modified soft bellman backup operator $\mathcal{T}_{QCSE}^{\pi}$ in Equation 5, along with a mapping $Q^{0}:\mathcal{S}\times\mathcal{A}\rightarrow\mathbb{R}$ where $|\mathcal{A}|<\infty$. We define an iterative sequence as $Q^{k+1}=\mathcal{T}^{\pi}Q^{k}$. It can be shown that when index $k$ tends towards infinity, the sequence $Q^{k}$ converges to a soft Q-value of $\pi$.

proof.
Let us define the QCSE reward as follows

$$ $r^{\pi}_{QCSE}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\triangleq r\left( \mathbf{s}_{t},\mathbf{a}_{t}\right)+\lambda\operatorname{Tanh}\left(\mathcal{ H}\left(\mathbf{s}_{t}\mid\min\left(Q_{\phi_{1}}(\mathbf{s}_{t},\mathbf{a}_{t} ),Q_{\phi_{2}}(\mathbf{s}_{t},\mathbf{a}_{t})\right)\right)\right)+\mathbb{E}_ {\mathbf{s}_{t+1}\sim p}\left[\mathcal{H}\left(\pi\left(\cdot\mid\mathbf{s}_{t +1}\right)\right)\right]$ (6) $$

and rewrite the update rule as

$$ $Q\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\leftarrow r^{\pi}_{QCSE}\left( \mathbf{s}_{t},\mathbf{a}_{t}\right)+\gamma\mathbb{E}_{\mathbf{s}_{t+1}\sim p, \mathbf{a}_{t+1}\sim\pi}\left[Q\left(\mathbf{s}_{t+1},\mathbf{a}_{t+1}\right) \right].$ (7) $$

Then we can apply mathematical analysis of convergence for policy evaluation as outlined in (Sutton and Barto, ) to prove the result. It is essential to note that the assumption $|\mathcal{A}|<\infty$ is necessary to ensure the boundedness of the QCSE reward.”
{lemma}[Soft Policy Improvement with QCSE]
Let $\pi_{\text{old}}\in\Pi$, and let $\pi_{\text{new}}$ be the solution to the minimization problem defined as:

$$ $\pi_{\mathrm{new}}=\arg\min_{\pi^{\prime}\in\Pi}\mathrm{D}_{\mathrm{KL}}\left( \pi^{\prime}\left(\cdot\mid\mathbf{s}_{t}\right)\|\frac{\exp\left(Q^{\pi_{ \mathrm{old}}}\left(\mathbf{s}_{t},\cdot\right)\right)}{Z^{\pi_{\mathrm{old}}} \left(\mathbf{s}_{t}\right)}\right).$ (8) $$

Then, it follows that $Q^{\pi_{\text{new }}}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\geq Q^{\pi_{
\text{old }}}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)$ for all $\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\in\mathcal{S}\times\mathcal{A}$ provided that $|\mathcal{A}|<\infty$.

proof.
Starting from Equation 9, which has been established in the work by (Haarnoja et al., ), as:

$$ $\mathbb{E}_{\mathbf{a}_{t}\sim\pi_{\text{new }}}\left[Q^{\pi_{\text{old }}} \left(\mathbf{s}_{t},\mathbf{a}_{t}\right)-\log\pi_{\text{new }}\left(\mathbf{ a}_{t}\mid\mathbf{s}_{t}\right)\right]\geq V^{\pi_{\text{old }}}\left(\mathbf{ s}_{t}\right),$ (9) $$

we proceed to consider the soft Bellman equation, which can be expressed as:

$$ $\displaystyle Q^{\pi_{\text{old }}}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)$ $\displaystyle=r\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)+r_{QCSE}\left( \mathbf{s}_{t},\mathbf{a}_{t}\right)+\gamma\mathbb{E}_{\mathbf{s}_{t+1}\sim p} \left[V^{\pi_{\text{old }}}\left(\mathbf{s}_{t+1}\right)\right]$ (10) $\displaystyle\leq r\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)+r_{QCSE}\left( \mathbf{s}_{t},\mathbf{a}_{t}\right)+\gamma\mathbb{E}_{\mathbf{s}_{t+1}\sim p} \left[\mathbb{E}_{\mathbf{a}_{t+1}\sim\pi_{\text{new }}}\left[Q^{\pi_{\text{ old }}}\left(\mathbf{s}_{t+1},\mathbf{a}_{t+1}\right)-\log\pi_{\text{new }} \left(\mathbf{a}_{t+1}\mid\mathbf{s}_{t+1}\right)\right]\right]$ $\displaystyle\vdots$ $\displaystyle\leq Q^{\pi_{\text{new }}}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)$ $$

Here, we have iteratively expanded $Q^{\pi_{\text{old}}}$ on the right-hand side by applying both the soft Bellman equation and the inequality from Equation 9.

[Converged QCSE Soft Policy is Optimal]
Repetitive using Lemma 1 and Lemma 2 to any $\pi\in\Pi$ leads to convergence towards a policy $\pi^{*}$. And it can be proved that $Q^{\pi^{*}}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\geq Q^{\pi}\left(\mathbf
{s}_{t},\mathbf{a}_{t}\right)$ for all policies $\pi\in\Pi$ and all state-action pairs $\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\in\mathcal{S}\times\mathcal{A}$, provided that $|\mathcal{A}|<\infty$.

proof.

Let $\pi_{i}$ represent the policy at iteration $i$. According to Lemma 2, the sequence $Q^{\pi_{i}}$ exhibits a monotonic increase. Given that rewards and entropy and thus $Q$ are bounded from above for policies within the set $\Pi$, the sequence converges to a certain policy $\pi^{*}$. It is essential to demonstrate that $\pi^{*}$ is indeed an optimal policy. Utilizing a similar iterative argument as employed in the proof of Lemma 2, we can establish that $Q^{\pi^{*}}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)>Q^{\pi}\left(\mathbf{s}_
{t},\mathbf{a}_{t}\right)$ holds for all $\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)\in\mathcal{S}\times\mathcal{A}$. In other words, the soft value associated with any other policy in $\Pi$ is lower than that of the converged policy. Consequently, $\pi^{*}$ is confirmed as the optimal policy within the set $\Pi$.

[Conservative Soft Q values with QCSE]
By employing a double Q network, we ensure that in each iteration, the Q-value from the single Q network, denoted as $Q^{\pi_{i}}_{\text{single Q}}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)$, is greater than or equal to the Q-value obtained from the double Q network, represented as $Q^{\pi_{i}}_{\text{double Q}}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)$, for all $(\mathbf{s}_{t},\mathbf{a}_{t})\in\mathcal{S}\times\mathcal{A}$, where the action space is finite.

proof. Let’s begin by defining $\hat{Q}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)=\min\left(Q_{\phi_{1}}\left(
\mathbf{s}_{t},\mathbf{a}_{t}\right),Q_{\phi_{2}}\left(\mathbf{s}_{t},\mathbf{
a}_{t}\right)\right).$ We then proceed to examine the difference between the augmented rewards in the context of QCSE for the single Q and double Q networks:

$$ $\displaystyle r_{QCSE}(\mathbf{s}_{t},\mathbf{a}_{t}|\hat{Q}\left(\mathbf{s}_{ t},\mathbf{a}_{t}\right))-r_{QCSE}(\mathbf{s}_{t},\mathbf{a}_{t}|Q\left( \mathbf{s}_{t},\mathbf{a}_{t}\right))$ (11) $\displaystyle=$ $\displaystyle\sum_{i=0}^{N}\log 2\max(||s_{i}-s_{i}^{knn}||,||\hat{Q}\left( \mathbf{s}_{t},\mathbf{a}_{t}\right)-\hat{Q}^{knn}\left(\mathbf{s}_{t},\mathbf {a}_{t}\right)||)-$ $\displaystyle\sum_{i=0}^{N}\log 2\max(||s_{i}-s_{i}^{knn}||,||Q\left(\mathbf{s }_{t},\mathbf{a}_{t}\right)-Q^{knn}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)||)$ $\displaystyle=$ $\displaystyle\log\frac{\prod_{i=0}^{N}\max(||s_{i}-s_{i}^{knn}||,||\hat{Q} \left(\mathbf{s}_{t},\mathbf{a}_{t}\right)-\hat{Q}^{knn}\left(\mathbf{s}_{t}, \mathbf{a}_{t}\right)||)}{\prod_{i=0}^{N}\max(||s_{i}-s_{i}^{knn}||,||Q\left( \mathbf{s}_{t},\mathbf{a}_{t}\right)-Q^{knn}\left(\mathbf{s}_{t},\mathbf{a}_{t }\right))||}$ $\displaystyle\approx$ $\displaystyle\log\frac{\prod_{i=0}^{N}\max(||s_{i}-s_{i}^{knn}||,\mathcal{H}( \hat{Q}))}{\prod_{i=0}^{N}\max(||s_{i}-s_{i}^{knn}||,\mathcal{H}(Q)||)}$ $\displaystyle\leq$ $\displaystyle\log\frac{\prod_{i=0}^{N}\max(||s_{i}-s_{i}^{knn}||,\mathcal{H}(Q ))}{\prod_{i=0}^{N}\max(||s_{i}-s_{i}^{knn}||,\mathcal{H}(Q))}=0$ $$

Consequently, we establish that $r_{QCSE}(\mathbf{s}_{t},\mathbf{a}_{t}|\hat{Q}(\mathbf{s}_{t},\mathbf{a}_{t}))
\leq r_{QCSE}(\mathbf{s}_{t},\mathbf{a}_{t}|Q(\mathbf{s}_{t},\mathbf{a}_{t}))$. Now we consider the modified soft Bellman equation

$$ $\displaystyle Q^{\pi_{i}}_{\rm double\,Q}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)$ (12) $\displaystyle=$ $\displaystyle r\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)+r_{QCSE}(\mathbf{s}_ {t},\mathbf{a}_{t}|\hat{Q}(\mathbf{s}_{t},\mathbf{a}_{t}))+\gamma\cdot\mathbb{ E}_{\mathbf{s}_{t+1}\sim p}[\hat{V}(\mathbf{s}_{t+1})]$ $\displaystyle=$ $\displaystyle r\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)+r_{QCSE}(\mathbf{s}_ {t},\mathbf{a}_{t}|\hat{Q}(\mathbf{s}_{t},\mathbf{a}_{t}))+\gamma\cdot\mathbb{ E}_{\mathbf{s}_{t+1}\sim p,\mathbf{a}_{t+1}\sim\pi}\left[\hat{Q}\left(\mathbf{ s}_{t+1},\mathbf{a}_{t+1}\right)-\log\pi\left(\mathbf{a}_{t+1}\mid\mathbf{s}_{ t+1}\right)\right]$ $\displaystyle\vdots$ $\displaystyle=$ $\displaystyle r\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)+r_{QCSE}(\mathbf{s}_ {t},\mathbf{a}_{t}|\hat{Q}(\mathbf{s}_{t},\mathbf{a}_{t}))+\gamma\cdot\mathbb{ E}_{\mathbf{s}_{t+1}\sim p,\mathbf{a}_{t+1}\sim\pi}[r^{mod}(\mathbf{s}_{t+1}, \mathbf{a}_{t+1}|\hat{Q}(\mathbf{s}_{t+1},\mathbf{a}_{t+1}))]\cdots+$ $\displaystyle\gamma^{n}\cdot\mathbb{E}_{\mathbf{s}_{t+n}\sim p,\mathbf{a}_{t+n }\sim\pi}[r^{mod}(\mathbf{s}_{t+n},\mathbf{a}_{t+n}|\hat{Q}(\mathbf{s}_{t+n}, \mathbf{a}_{t+n}))]+\cdots+\text{entropy terms}$ $\displaystyle\leq$ $\displaystyle r\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)+r_{QCSE}(\mathbf{s}_ {t},\mathbf{a}_{t}|Q(\mathbf{s}_{t},\mathbf{a}_{t}))+\gamma\cdot\mathbb{E}_{ \mathbf{s}_{t+1}\sim p,\mathbf{a}_{t+1}\sim\pi}[r^{mod}(\mathbf{s}_{t+1}, \mathbf{a}_{t+1}|Q(\mathbf{s}_{t+1},\mathbf{a}_{t+1}))]\cdots+$ $\displaystyle\gamma^{n}\cdot\mathbb{E}_{\mathbf{s}_{t+n}\sim p,\mathbf{a}_{t+n }\sim\pi}[r^{mod}(\mathbf{s}_{t+n},\mathbf{a}_{t+n}|Q(\mathbf{s}_{t+n},\mathbf {a}_{t+n}))]+\cdots+\text{entropy terms}$ $\displaystyle=$ $\displaystyle Q^{\pi_{i}}_{\rm single\,Q}\left(\mathbf{s}_{t},\mathbf{a}_{t}\right)$ $$

where we have repeatedly expanded $\hat{Q}$ in terms of QCSE rewards to obtain the final inequality $Q_{\text{single}\mathrm{Q}}^{\pi_{i}}\geq Q_{\text{double }\mathrm{Q}}^{\pi_{i
}}.$

## Appendix C Experimental Setup

In this section, we introduce the benchmarks and dataset we utilized, specifically, we mainly utilize Gym-Mujoco and antmaze to test our algorithm.

### C.1 Gym-Mujoco

Our benchmars from Gym-Mujoco domain mainly includes halfcheetah, ant, hopper and walker2d, and concrete information of these benchmarks can be referred to table 2. In paticular, the action and observation space of these locomotion benchmarks are continuous and any decision making will receive an immediate reward.

**Table 2: Introduction of D4RL tasks (Gym-Mujoco).**
| Environment | Task Name | Samples | Observation Dim | Action Dim |
| --- | --- | --- | --- | --- |
| halfcheetah | medium | $10^{6}$ | 6 | 17 |
| walker2d | medium | $10^{6}$ | 6 | 17 |
| hopper | medium | $10^{6}$ | 3 | 11 |
| ant | medium | $10^{6}$ | 8 | 111 |
| halfcheetah | medium-replay | 2.02$\times 10^{5}$ | 6 | 17 |
| walker2d | medium-replay | 3.02$\times 10^{5}$ | 6 | 17 |
| hopper | medium-replay | 4.02$\times 10^{5}$ | 3 | 11 |
| ant | medium-replay | 3.02$\times 10^{5}$ | 8 | 111 |

### C.2 Antmaze

Our benchmars from antmaze mainly includes antmaze-large-diverse, antmaze-medium-diverse, antmaze-large-play and antmaze-medium-play, concrete information of our benchmarks can be referred to table 3.

**Table 3: Introduction of D4RL tasks (Antmaze).**
| Environment | Task Name | Samples | Observation Dim | Action Dim |
| --- | --- | --- | --- | --- |
| antmaze | large-diverse | $10^{6}$ | 29 | 8 |
| antmaze | large-play | $10^{6}$ | 29 | 8 |
| antmaze | medium-diverse | $10^{6}$ | 29 | 8 |
| antmaze | medium-play | $10^{6}$ | 29 | 8 |

## Appendix D Implantation Details

### D.1 Offline-to-Online implantation

The workflow of our method is similar to the most of offline-to-online algorithms that we firstly pre-train on offline datasets, followed by online fine-tuning (Interacting with online environment to collect online dataset and followed by fine-tuning on offline and online datasets).

### D.2 Evaluation Details

Our evaluation method can be refered to (Fu et al., ). That is for each evaluation, we freeze the parameter of trained model, and then conducting evaluation 10$\sim$50 times and then computing the normalized score via $\rm\frac{score_{\rm evaluation}-score_{\rm expert}}{score_{\rm expert}-score_{
\rm random}}$, and then averaging these normalized evaluation scores.

### D.3 QCSE implantation

In QCSE framework, we modify our reward as :

$$ $\displaystyle r^{\mathrm{mod}}(\mathbf{s},\mathbf{a})=\lambda\cdot\underbrace{ \mathrm{Tanh}(\mathcal{H}(\mathbf{s}|\mathrm{min}(Q_{\phi_{1}}(\mathbf{s}, \mathbf{a}),Q_{\phi_{2}}(\mathbf{s},\mathbf{a}))))}_{r^{\rm mod}}+r(\mathbf{s} ,\mathbf{a}),\,\,\,\,(\mathbf{s},\mathbf{a})\sim\mathcal{D}_{\mathrm{online}}$ (13) $$

To calculate the intrinsic reward $r^{\rm mod}$ for the online replay buffer $D_{\rm online}$, we use the KSG estimator, as defined in Equation 14, to estimate the conditional state density of the empirical dataset $D_{\rm online}$

$$ $r^{\rm mod}(\mathbf{s},\mathbf{a})=\frac{1}{d_{s}}\phi(n_{v}(i)+1)+\log 2\cdot \max(||\mathbf{s}_{i}-\mathbf{s}_{i}^{knn}||,||\hat{Q}(\mathbf{s},\mathbf{a})- \hat{Q}(\mathbf{s},\mathbf{a})^{knn}||),(\mathbf{s},\mathbf{a})\sim\mathcal{D} _{\rm online}.$ (14) $$

Given that the majority of our selected baselines are implemented using the double Q($\{Q_{\phi_{1}},Q_{\phi_{2}}\}$), the offline pre-trained double Q can be readily utilized for the computation of intrinsic rewards, and we found that the performance of QCSE is sutured when $\lambda$ is set to 1.
We also provide a (Variance Auto Encoder) VAE implantation (Equation 15) of QCSE, this realization is computing efficiency, but require extraly training a VAE model, due to Equation 3 won’t require training thus we mainly test Equation 3.

$$ $\small\begin{split}r^{\rm mod}(\mathbf{s},\mathbf{a})=-\log p_{\hat{\phi}}(s| \hat{Q}(\mathbf{s},\mathbf{a}))=-\log\mathbb{E}_{z\sim q_{\phi}(z|\mathbf{s}, \hat{Q}(\mathbf{s},\mathbf{a}))}[\frac{p_{\hat{\phi}}(\mathbf{s}|\hat{Q}( \mathbf{s},\mathbf{a}))}{q_{\phi}(z|\mathbf{s},\hat{Q}(\mathbf{s},\mathbf{a})) }],(\mathbf{s},\mathbf{a})\sim\mathcal{D}_{\rm online}.\end{split}$ (15) $$

We will test and compare the performance difference and computing efficiency between Equation 15 and Equation 14 in the future.

### D.4 Codebase

Our implementation is based on Cal-QL:[https://github.com/nakamotoo/Cal-QL](https://github.com/nakamotoo/Cal-QL), VCSE:[https://sites.google.com/view/rl-vcse](https://sites.google.com/view/rl-vcse). Additionally, we have included our source code in the supplementary material for reference. Readers can refer to our pseudocode (see Algorithm 1) for a comprehensive understanding of the implementation details.
$\hat{Q}$ see (^2^22where $\phi_{1}$ and $\phi_{2}$ are the params of double $Q$ Networks and $\hat{Q}(\mathbf{s},\mathbf{a})=\min(Q_{\phi_{1}}(\mathbf{s},\mathbf{a}),Q_{ \phi_{2}}(\mathbf{s},\mathbf{a}))$, and $x^{knn}_{i}$ is the $n_{x}(i)$-th nearest neighbor of $x_{i}$.). Part of our source code will be released at: .

Figure: Algorithm 1 Training QCSE

$$ $\mathcal{L}(Q)={\mathbb{E}}_{(\mathbf{s},\mathbf{a})\sim\mathcal{D}}[(Q( \mathbf{s},\mathbf{a})-\mathcal{B}_{\mathcal{M}}^{\pi}Q(\mathbf{s},\mathbf{a}) )^{2}]+{\mathrm{E}}_{\mathbf{s}\sim\mathcal{D},\mathbf{a}\sim\pi}[\mathrm{max} (Q(\mathbf{s},\mathbf{a}),V^{\mu}(\mathbf{s}))]-{\mathrm{E}}_{(\mathbf{s}, \mathbf{a})\sim\mathcal{D}}[Q(\mathbf{s},\mathbf{a})].$ (16) $$

$$ $\mathcal{L}(Q)=\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})\sim \mathcal{D}}[(Q(\mathbf{s},\mathbf{a})-\mathcal{B}_{\mathcal{M}}^{\pi}Q( \mathbf{s},\mathbf{a}))^{2}]\\ +\mathbb{E}_{(\mathbf{s},\mathbf{a},\mathbf{s}^{\prime})\sim\mathcal{D}}[-Q( \mathbf{s},\mathbf{a})+Q(\mathbf{s}^{\prime},\pi(\mathbf{s}^{\prime}))],$ (17) $$

$$ $\mathcal{J}(\pi_{\theta})={\mathrm{E}}_{\mathbf{s}\sim\mathcal{D}}[-Q(\mathbf{ s},\pi_{\theta}(\mathbf{s}))+\alpha\log(\pi_{\theta}(\mathbf{s}))].$ (18) $$

### D.5 Computing Resources

Our experiments were run on a computer cluster with 4×32GB RAM, AMD EPYC 7742 64-Core CPU, and NVIDIA-A100 GPU, Linux. Most of our code base (The implantation of Cal-QL, CQL, TD3+BC, SAC) are based on JAX (^3^33[https://github.com/google/jax.git](https://github.com/google/jax.git)), part of our implantation (IQL, AWAC) are based on Pytorch(^4^44[https://pytorch.org/](https://pytorch.org/)) (We use different deep learning frameworks mainly to preliminary validate that our algorithm can work in various of deep learning frameworks).

### D.6 Our Hyper-parameter

#### Hyper-parameter of QCSE.

The K-nearest neighbors (knn) for QCSE are configured as follows: [0, 10, 15, 25, 50, 85, 100, 110], and the parameter $\lambda$ in Equation 13 is set to 1.

#### Hyper-parameter of Baselines

In the context of these algorithms, we conducted tests related to AWAC and IQL using the repository available at [https://github.com/tinkoff-ai/CORL](https://github.com/tinkoff-ai/CORL), while tests related to Cal-QL and CQL were performed using the repository accessible at [https://github.com/nakamotoo/Cal-QL](https://github.com/nakamotoo/Cal-QL).
The following five tables present fundamental but critical hyperparameter settings for five baseline algorithms.

**Table 4: Hyper-parameters of AWAC.**
| Hyperparameter | Value |
| --- | --- |
| 0ffline pre-train iterations | $1e^{6}$ |
| 0nline fine-tuning iterations | $1e^{6}$ |
| Buffer size | 20000000 |
| Batch size | 256 |
| learning rate | $3e^{-4}$ |
| $\gamma$ | 0.99 |
| awac $\tau$ | 5e-3 |
| awac $\lambda$ | 1.0 |
| Actor Architecture | 4$\times$ Layers MLP (hidden dim 256) |
| Critic Architecture | 4$\times$ Layers MLP (hidden dim 256) |

**Table 5: Hyper-parameters of IQL.**
| Hyperparameter | Value |
| --- | --- |
| 0ffline pre-train iterations | $1e^{6}$ |
| 0nline fine-tuning iterations | $1e^{6}$ |
| Batch size | 256 |
| learning rate of $\pi$ | $3e^{-4}$ |
| learning rate of V | $3e^{-4}$ |
| learning rate of Q | $3e^{-4}$ |
| $\gamma$ | 0.99 |
| IQL $\tau$ | 0.7 $\#\,\,\textit{Coefficient for asymmetric loss}$ |
| $\beta$ (Inverse Temperature) | 3.0$\#\,\,\textit{small beta $\rightarrow$ BC, big beta $\rightarrow$ maximizing Q}$ |
| Actor Architecture | 4$\times$ Layers MLP (hidden dim 256) |
| Critic Architecture | 4$\times$ Layers MLP (hidden dim 256) |

**Table 6: Hyper-parameters of TD3+BC.**
| Hyperparameter | Value |
| --- | --- |
| 0ffline pre-train iterations | $1e^{6}$ |
| 0nline fine-tuning iterations | $1e^{6}$ |
| learning rate of $\pi$ | $1e^{-4}$ |
| learning rate of Q | $3e^{-4}$ |
| $\gamma$ | 0.99 |
| Batch size | 256 |
| TD3 alpha | 2.5 |
| Actor Architecture | 4$\times$ Layers MLP (hidden dim 256) |
| Critic Architecture | 4$\times$ Layers MLP (hidden dim 256) |

**Table 7: Hyper-parameters of Cal-QL. We only provide the basic setting, for more detail setting, please directly refer to [https://nakamotoo.github.io/projects/Cal-QL](https://nakamotoo.github.io/projects/Cal-QL)**
| Hyperparameter | Value |
| --- | --- |
| 0ffline pre-train iterations | $1e^{6}$ |
| 0nline fine-tuning iterations | $1e^{6}$ |
| learning rate of $\pi$ | $1e^{-4}$ |
| learning rate of Q | $3e^{-4}$ |
| $\gamma$ | 0.99 |
| Batch size | 256 |
| Actor Architecture | 4$\times$ Layers MLP (hidden dim 256) |
| Critic Architecture | 4$\times$ Layers MLP (hidden dim 256) |

**Table 8: Hyper-parameters of CQL. CQL uses Cal-QL’s code-base, and we only need to remove Cal-QL’s calibration loss when deploying CQL.**
| Hyperparameter | Value |
| --- | --- |
| 0ffline pre-train iterations | $1e^{6}$ |
| 0nline fine-tuning iterations | $1e^{6}$ |
| learning rate of $\pi$ | $1e^{-4}$ |
| learning rate of Q | $3e^{-4}$ |
| $\gamma$ | 0.99 |
| Batch size | 256 |
| Actor Architecture | 4$\times$ Layers MLP (hidden dim 256) |
| Critic Architecture | 4$\times$ Layers MLP (hidden dim 256) |

## Appendix E Appended Experimental Results

In Table 9, we compare a series of different efficient offline-to-online methods, including APL, PEX, and BR. Specifically, we tested these methods on the ant-maze domain and the medium and medium-replay tasks in the Gym-Mujoco environment. We found that QCSE shows the best overall performance, indicating that QCSE, when paired with CQL, can achieve superior results.

**Table 9: Comparison of various efficient offline-to-online methods. Part of experimental results are quoted from (Guo et al., ).**
| Task | CQL+APL | CQL+PEX | CQL+BR | CQL+SUNG | CQL+QCSE |
| --- | --- | --- | --- | --- | --- |
| antmaze-large-diverse | 0 | 0 | 0.1 | 44.1 | 89.8 |
| antmaze-large-play | 0 | 0 | 0 | 52.7 | 92.6 |
| antmaze-medium-diverse | 36.8 | 0.3 | 13.6 | 85.6 | 98.9 |
| antmaze-medium-play | 22.8 | 0.3 | 22.2 | 86.3 | 99.4 |
| halfcheetah-medium | 44.7 | 43.5 | 56.7 | 79.7 | 87.9 |
| walker2d-meidum | 75.3 | 34.0 | 81.7 | 86.0 | 130.0 |
| hopper-medium | 102.7 | 46.3 | 97.7 | 104.1 | 62.4 |
| halfcheetah-medium-replay | 78.6 | 45.5 | 64.9 | 75.6 | 55.4 |
| walker2d-medium-replay | 103.2 | 40.1 | 88.5 | 108.2 | 112.7 |
| hopper-medium-replay | 97.4 | 66.5 | 78.8 | 101.9 | 27.1 |
| Average Fine-tuned | 56.2 | 27.6 | 50.4 | 82.4 | 85.6 |

## Appendix F Extended Experiments

### F.1 Trend of State Entropy Changing

#### State Entropy as Intrinsic Reward.

If the state density $\rho(\mathbf{s})$ is unknown, we can instead using non-parametric entropy estimator to approximate the state entropy (Seo et al., ). Specifically, given N i.i.d. samples $\{\mathbf{s}_{i}\}$, the k-nearest neighbors (knn) entropy estimator can be defined as(^5^55$d_{s}$ is the dimension of state and $\Gamma$ is the gamma function, $\hat{n}_{\hat{\pi}}\propto 3.14$.):

$$ $\hat{H}_{N}^{k}(S)=\frac{1}{N}\sum_{i=1}^{N}\log\frac{N\cdot||\mathbf{s}_{i}-s _{i}^{knn}||_{2}^{d_{\mathbf{s}}}\cdot\hat{n}_{\hat{\pi}}^{\frac{d_{\mathbf{s} }}{2}}}{k\cdot\Gamma(\frac{d_{\mathbf{s}}}{2}+1)}\propto\frac{1}{N}\sum_{i=1}^ {N}\log||\mathbf{s}_{i}-\mathbf{s}_{i}^{knn}||.$ (19) $$

#### Visualization of State Entropy Changing.

In this experiment, for each training step, we select the buffer and randomly sample 5000 instances to approximate the entropy using Equation 10. and then plot the trend of approximated state entropy. For the majority of the tasks, the state entropy of AWAC-QCSE was either progressively greater than or consistently exceeded that of AWAC-base. This indicates that QCSE effectively enhances the agent’s exploratory tendencies, enabling them cover much more observation region.

Figure: Figure 9: The Changing of Approximated Entropy along with increasing training steps. We found that the approximated state entropy in the buffer collected by AWAC using QCSE was greater in the later stages of online finetuning.
Refer to caption: x39.png

### F.2 pre-trained Q vs. Random Q

#### Pre-trained Q condition versus un-pre trained Q condition.

To validate the statement in our main paper that intrinsic reward computation is influenced by the initialization of $Q$, we conducted experiments comparing the effects of pre-trained initialized $Q$ and
from-scratch(^6^66We use from-scratch $Q$ to compute intrinsic reward, while continuing to utilize the offline-initialized $Q$ for conducting online fine-tuning.) trained $Q$ during intrinsic reward calculation. Our findings indicate that intrinsic rewards based on offline-initialized $Q$ generally outperform those derived from a from-scratch trained $Q$ across most tasks.

Figure: Figure 10: Offline Pre-trained Q condition vs. Randomly initialized Q condition. In the majority of our selected Gym-Mujoco tasks, the use of offline-initialized intrinsic reward conditions yielded better performance and higher sample efficiency. To provide clarity, AWAC-base means AWAC algorithm without any modification, AWAC-QCSE signifies AWAC with QCSE augmentation, and AWAC-QCSE (scratch) denotes AWAC with QCSE where the computation of reward conditions satisfying note 6
Refer to caption: x44.png

### F.3 Aggregated metric with QCSE

To demonstrate the significant improve brought by QCSE we adapt the method proposed by , QCSE can significantly improve the performance of CQL and Cal-QL.

Figure: Figure 11: Aggregate metrics with QCSE (SERA). We refer to (Agarwal et al., ) to analyze QCSE’s performance. Specifically, higher median, IQM, and mean scores are better, QCSE can significantly improve the performance of CQL and Cal-QL.
Refer to caption: x49.png

## Appendix G Extended Related Work

### G.1 Offline-to-online RL

In this section, we systematically introduce recent developments in offline-to-online learning and summarize the corresponding methods, the first perspective involves adopting a conservative policy optimization during online fine-tuning, typically achieved through the incorporation of policy constraints. Specifically, there are three main approaches within this category. The first approach constrains the predictions of the fine-tuning policy within the scope of offline support during online fine-tuning (Wu et al., ). While this method contributes to achieving stable online fine-tuning performance, it tends to lead to overly conservative policy learning, and the accuracy of the estimation of offline support also influences the effectiveness of online fine-tuning. The second approach utilizes an offline dataset to constrain policy learning (Nair et al., ; Kostrikov et al., ; Xiao et al., ; Mark et al., ). However, the effectiveness of fine-tuning cannot be guaranteed if the dataset quality is poor. This method is sensitive to the quality of the dataset. The third approach employs pre-trained policies to constrain online fine-tuning, but this paradigm is influenced by the quality of the pre-trained policy (Zhang et al., ; Yu and Zhang, ). The second perspective involves adopting a conservative approach during offline training, specifically using pessimistic constraints to learn Q to avoid OOD (Out-of-Distribution) issues. Research in this category primarily includes: Learning a conservative Q during offline pre-training and employing an appropriate experience replay method during online learning or using Q ensemble during offline pre-training to avoid OOD issues (Lee et al., ; Lyu et al., ; Hong et al., ). However, as this approach introduces conservative constraints during critic updates, the value estimates between offline and online are not aligned, leading to a decrease in performance during early online fine-tuning. Therefore, Cal-QL introduces a calibrated conservative term to ensure standard online fine-tuning (Nakamoto et al., ). Addtionally, there are also some other methods, such that ODT (Zheng et al., ) combined sequence modeling with Goal conditioned RL to conduct offline-to-online RL.

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Could you please provide more clarity on how the hyper-parameter settings, particularly the choice of  λ and the number of k-nearest neighbor (knn) clusters, affect the results? I would appreciate a more detailed explanation of their impact on the performance of QCSE, as the current description is a bit unclear to me.
soundness: 3
strengths: This paper presents a novel approach to offline-to-online reinforcement learning, demonstrating strong originality and significance. The authors provide clear and thorough explanations of their proposed method, making the paper highly readable and understandable.
summary: This paper proposes a new reinforcement learning method called Q-conditioned State Entropy Maximization (QCSE) which aims to improve the performance of offline-to-online RL process. The authors prove that QCSE can achieve State Marginal Matching (SMM), an exploration strategy theoretically ensuring optimal performance. Experiments show that QCSE significantly enhances the performance of existing model-free algorithms like CQL and Cal-QL, with an average improvement of about 13% and 8%. Additionally, QCSE exhibits general applicability to other model-free algorithms such as SAC, IQL, and TD3+BC.
weaknesses: On the novelty factor, contributions are not very significant. 

The paper would benefit from a more thorough analysis and discussion of the limitations and potential challenges associated with the proposed framework.

The experimental evaluation should include comparisons with a broader range of state-of-the-art methods to offer a more comprehensive assessment of QCSE's performance.

The theoretical analysis is insufficient and needs strengthening.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 2
questions: The same as Weaknesses.
soundness: 3
strengths: 1. The idea proposed in this paper has a degree of novelty, as it connects the offline-to-online learning problem with exploration mechanisms, leading to a new algorithm. This represents a relatively fresh perspective.

2. The experiments in this paper are quite comprehensive, evaluating the algorithm's effectiveness, modularity, and advancement from multiple perspectives. Additionally, the experiments cover a wide range of tasks, providing robust empirical support for the proposed method.
summary: This paper explores strategies for optimizing the fine-tuning of pre-trained policies in offline reinforcement learning (RL) to enhance sample efficiency. Traditional fine-tuning methods often fall short due to a distribution mismatch between offline pre-training and subsequent online fine-tuning, which restricts the quality of online sample acquisition and hampers performance. To mitigate this distribution shift, the authors introduce Q-conditioned state entropy (QCSE) as an intrinsic reward mechanism. QCSE maximizes the entropy of individual states based on their respective Q-values, promoting exploration of underrepresented samples while disincentivizing overrepresented ones. This approach implicitly aligns with State Marginal Matching (SMM), resolving the asymptotic limitations often seen in constraint-based techniques. Moreover, QCSE integrates seamlessly with various RL algorithms, leading to significant improvements in online fine-tuning. Experimental results show approximately 13% performance gains for CQL and 8% for Cal-QL, with additional tests confirming QCSE’s versatility across different algorithms.
weaknesses: My main concerns about this paper lie in the method description section, where I feel that several key points are not fully explained by the authors.

1. Regarding the interpretation of Equation (1), if the goal is to maximize the right-hand side $J_2$, since the optimal policy’s corresponding $p^*$ is fixed, it’s clear that the scope of $S_2$ should be expanded, rather than "narrowing down the domain $S_2$" as the authors suggest. Meanwhile, to maximize $J_1$, it’s evident that the range of $S_1$ should be reduced, i.e., $p(s)$ over $S_1$ should be decreased to approach $p^*(s)$. Therefore, the overall approach seems to be "increasing exploration in $S_2$ while penalizing exploration in $S_1$." However, this is contrary to the authors’ description in lines 203–209. Could you clarify your reasoning behind the claim of "narrowing down the domain $S_2$" and explain how this aligns with maximizing $J_2$. Additionally, you could  revisit the description in lines 203-209 to ensure consistency with the mathematical formulation.

2. In the "Implementation of QCSE" section, I did not see how the practical algorithm for QCSE was derived from Equation (1). Specifically, what do the $S_1$ and $S_2$ in Equation (1) correspond to here? What role does maximizing the Critic Conditioned State Entropy $H(s|Q)$ in Equation (2) play, and how is it related to Equation (1)? Particularly, if there are indeed issues with the interpretation of Equation (1) mentioned above, does that mean Equation (2) would not hold either? The authors are suggested to provide a step-by-step derivation showing how the practical algorithm in Equation (2) follows from the theoretical formulation in Equation (1). Specifically, could you clarify on how $S_1$ and $S_2$ from Equation (1) are represented in the implementation, and how maximizing $H(s|Q)$ relates to the objectives in Equation (1)?

3. The origin of Equation (3) is also unclear. The authors simply refer to the literature, but they do not explain how (3) is derived from (2), which is not obvious. Therefore, the authors have an obligation to provide the derivation process from (2) to (3).

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 2
questions: - I was not able to understand the math presented in the paper, as discussed in the weaknesses section. Then, why do we need to consider the so-called critic conditioned state entropy? I agree with the idea that we need exploration for offline-online fine-tuning, and state marginal matching can be one of exploration methods. But for that, we can simply put reward for entropy maximization; how QCSE and VCSE differ from it?
- In the experiments, QCSE adopted algorithms does improve from the baselines, but their performance is still highly correlated to the baselines' performance (e.g., walker2d -medium-replay). Why is there a gap between the theory and the experiment results? Following the papers' arguments, the proposed algorithm does state marginal matching, and shouldn't it lead to an optimal policies?
soundness: 1
strengths: - Strong empirical performance.
summary: The paper proposes a new method that estimates Q-conditioned state entropy, to improve offline-to-online RL. Typically, if we train offline agent online, it results in sub-optimal online performance due to limited exploration of effective samples. QCSE introduces an intrinsic reward based on the entropy of states conditioned on Q-values, achieving State Marginal Matching. This approach outperforms previous offline-to-online methods.
weaknesses: - There are a number of notations and theoretical arguments that I was not able to fully understand (or wrong):
  - In equation (1), it is said that the state entropy maximization is: 
 $ \max E_{ \mathbf{s} \sim \rho_\pi } \left[ \mathcal{H}_\pi [ \mathbf{s} ] \right] $ 
s.t. 
 $ \pi := \arg \max _ \pi E _ { \tau \sim \pi } [ R( \tau ) ] $ . If $\pi$ is maxmimizing reward as written in constraints, what are we maxmizing in the objective?
  - On line 194, it is written: $\max E _ {\mathbf{s} \sim \rho_\pi}[-\log p(\mathbf{s})]$; is $p(x)$ $\rho_\pi(s)$?
  - The proof of equation (1) seems to be wrong. in equation (4), it is argued that $\rho_\pi \le p^*$ in $\mathcal{S}_2$, but in that case, the inequality should be in the opposite direction. 
  - From the first place, how $ \max E_{ \mathbf{s} \sim \rho_\pi } \left[ \mathcal{H}_\pi [ \mathbf{s} ] \right] $ can be equivalent to KL minimization between $ \rho _ \pi $ and $ p ^ * $? the former converges to maxent distribution (uniform-like), whereas the latter converges to $ p ^ * $.

- While the paper defines critic conditioned state entropy and use it as its key concept, it is very hard to understand intuitively what it is, and the paper does not explain it well. 
  - according to the definition, it seems like, critic conditioned state entropy averages $-\log p(s|Q(s, \pi(\cdot|s)))$ over $\rho _ \pi$. What is $p(s|Q(s, \pi(\cdot|s))$ indeed? How can a given term depend on the conditioned term?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: * A key baseline named ENOTO [1] is missing. In ENOTO, the authors also incorporate exploration techniques into their methods, which should be discussed and compared. Besides, ENOTO could also be combined with different offline RL algorithms. It seems from the curves in Fig.5 and Fig.6 in ENOTO's paper and the Fig.3 of the submitted paper that the proposed method QCSE  is inferior to ENOTO. So what's the advantage of QCSE compared with ENOTO?
*I'm curious whether performance drop could happen when QCSE is applied on medium-expert and expert datasets? These two kinds of datasets are quite nightmare for most offline-to-online RL algorithms. Could QCSE handle them well?


[1] ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles. IJCAI 2024.
soundness: 2
strengths: *  By promoting exploration based on Q-conditioned state entropy, the method encourages the agent to explore states that are low-frequency yet potentially valuable. This approach goes beyond traditional exploration strategies, providing a fresh perspective on tackling distribution mismatch.
* The framework’s compatibility with various offline RL algorithms, including CQL and Cal-QL, showcases its versatility. QCSE’s intrinsic reward structure can be integrated into different RL algorithms, making it adaptable for a range of applications beyond the specific benchmarks tested, such as robotics or recommendation systems where offline-to-online RL is critical.
summary: The paper introduces Q-Conditioned State Entropy Exploration (QCSE), an innovative offline-to-online reinforcement learning (RL) framework aimed at enhancing online fine-tuning of policies trained on offline data. The paper introduces Q-Conditioned State Entropy Exploration (QCSE), an innovative offline-to-online reinforcement learning (RL) framework aimed at enhancing online fine-tuning of policies trained on offline data. The paper introduces Q-Conditioned State Entropy Exploration (QCSE), an innovative offline-to-online reinforcement learning (RL) framework aimed at enhancing online fine-tuning of policies trained on offline data. The paper validates QCSE on several benchmarks, including CQL and Cal-QL algorithms, and demonstrates its effectiveness across various offline-to-online tasks. QCSE shows performance improvements, especially in environments with larger distribution shifts, and offers compatibility with multiple RL algorithms, making it versatile for real-world application
weaknesses: * It seems the online fine-tuning is not efficient enough as several baselines could achieve better performance within 250000 online fine-tuning steps such as BR and ENOTO. And these baselines are missing.
* QCSE’s reliance on entropy maximization and Q-conditioning likely introduces sensitivity to hyperparameter choices, such as size of k-nearest neighbor.

[1] Offline-to-online re- inforcement learning via balanced replay and pessimistic q-ensemble. CoRL 2022.

[2] ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles. IJCAI 2024.

Meta-review: The paper proposes a novel intrinsic motivation quantity for exploration in offline-to-online RL. While this is an important topic, and the paper is mostly well written and motivated, some concerns were raised regarding the mathematical clarity, and indeed correctness, of the proposed approach. Other concerns included the discussion of novelty and limitation, as well as the strength of the empirical evaluation, in particular with regard to a broader range of baselines.

Decision: Reject