Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation
Authors: Xiao Huang
Xu Liu
Enze Zhang
Tong Yu
Shuai Li
Venue: ICML 2025 poster (2025)
Keywords: offline-to-online reinforcement learning
data augmentation
diffusion models
Primary area: reinforcement_learning

Abstract: Offline-to-online Reinforcement Learning (O2O RL) aims to perform online fine-tuning on an offline pre-trained policy to minimize costly online interactions. Existing work used offline datasets to generate data that conform to the online data distribution for data augmentation. However, generated data still exhibits a gap with the online data, limiting overall performance. To address this, we propose a new data augmentation approach, Classifier-Free Diffusion Generation (CFDG). Without introducing additional classifier training overhead, CFDG leverages classifier-free guidance diffusion to significantly enhance the generation quality of offline and online data with different distributions. Additionally, it employs a reweighting method to enable more generated data to align with the online data, enhancing performance while maintaining the agent's stability.  Experimental results show that CFDG outperforms replaying the two data types or using a standard diffusion model to generate new data. Our method is versatile and can be integrated with existing offline-to-online RL algorithms. By implementing CFDG to popular methods IQL, PEX and APL, we achieve a notable 15\% average improvement in empirical performance on the D4RL benchmark such as MuJoCo and AntMaze.

Offine-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation
Xiao Huang *1
Xu Liu" Enze Zhang' Tong Yu? Shuai Li
Abstract
agents learn from static dataset pre-collected by arbitrary policies. Current research in offline RL focuses primarily addressing the challenge of distribution mismatch or out- of-distribution (OOD) actions through the implementation of a pessimistic update scheme (Kumar et al., 2020) or in combination with imitation learning (Kumar et al., 2019). However, when dealing with fixed and suboptimal dataset, it becomes exceedingly challenging for offline RL attain the optimal policy (Kidambi et al., 2020).
Offline-to-onling Reinforcement Learning (O20 RL) aims to perform online fine-tuning on an of- fline pre-trained policy to minimize costly online interactions. Existing work uses offline datasets to generate data that conform to the online data distribution for data augmentation. However, gen- crated data still exhibits a gap with the online data, limiting overall performance. To address this issue, we propose new data augmentation approach, Classifier-Fres Diffusion Generation (CFDG). Without introducing additional classifier training overhead, CFDG leverages classifier fassi guidance diffusion to significantly enhance the generation quality of offline and online data with different distributions. Additionally, employs reweighting method enable more generated data align with the online data, enhancing per- formance while maintaining the agent's 's stabil- ity. Experimental results show that CFDG out- performs replaying the two data types or using a standard diffusion model to generate new data. Our method is versatile and can integrated with existing offline-to-online RL algorithms. By im- plementing CFDG popular methods IQL, PEX and APL, we achieve notable 15% average im- provement in empirical performance on the D4RL. benchmark such as MuJoCo and AntMaze.
Some recent work addresses the above issues by employing an offline-to-online setting. Such methods (Lee et al., 2022; Nair et al., 2020) focus on pre-training a policy using the offline dataset and fine-tuning the policy through further online interactions. 020 RL aims to fine-tune a pre-trained policy using limited online interactions to achieve the op- timal policy. To utilize the offline dataset, some studies directly replayed samples in the online phase (Lee et al., 2022), leading to performance improvements. Nevertheless, this approach neglects the distribution shift issue, as the data distribution in the offline dataset may differ from that induced by the current policy. To address the distribution shift problem, existing work (Zheng et al., 2023) suggests that their different characteristics require separate updat- ing strategies for online and offline data, respectively. In particular, offline data can deter agents from prematurely converging to suboptimal policies due to the diversity of available data, while online data can contribute to train- ing stability and accelerate convergence (Nair al., 2020; Thrun & Littman, 2000).
In addition to fully utilizing offline and online data, there are also works that leverage generative models for data augmentation to provide diverse samples for the agent's s training. Prior work has considered upsampling online data with VAEs or GANs (Huang et al., 2017; Imre, 2021). To fully leverage offline data, Energy-guided Diffusion Sam- pling (EDIS) (Liu et al., 2024) generates data aligned with the online policy based on the offline dataset to address the issue of distribution shift. In addition to using dif- fusion model to generate data, it formulates three distinct energy functions to guide the diffusion sampling process, ensuring alignment with the online policy. Intuitively, on- line data more closely aligned with the current online policy. The limitation of previous works lies in using of-
1. Introduction
Traditionally, Reinforcement Learning (RL) (Haarnoja et al., 2018) is considered paradigm for online learning, where agents learn from online interactions with the environment. Due to costly online interactions in some real-worke appli- cations, offline RL (Levine et al., 2020) is proposed where
Equal contribution 'Shanghai Jiao Tong University. Shanghai, China Adobe Research, California, United States. Correspon- dence to: Shuai <shuaili8 @ sjtu.edu.en>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Offlin-to-Online Reinforcement Learning with Classifter-Free Diffusion Generation
fline data for data augmentation, resulting in generated data that still exhibits significant discrepancies from the online policy, Although it achieves good results, adding multiple new models introduces additional training overhead and time costs, and the entire guidance process becomes quite complex. Consequently, a fundamental question arises: Can we use a simpler approach to guide the model in generating higher-quality data?
T(a 8) that maximizes the expected cumulative return E Ei= 2i=o r'r44)].
2.1. Offline Reinforcement Learning
In Offline RL, the agent can only access static dataset D collected by behavior policy T s). Offline RL approaches can take advantage of the offline dataset to train critic network (Q-function) (5, a) with parameters 6, which estimates the long-term discounted reward achieved by executing action a in state s and following the policy T. The critic network can be trained using the following temporal difference (TD) learning objective:
To address this issue, revisit the relationship between offline data and online data and conduct distributional anal- ysis for both types of data and the generated data. Given the completely distinct characteristics of offline and online data, we perform data augmentation for both simultaneously. To simplify the model, we utilize a classifier-free guidance dif- fusion model for data augmentation. This approach serves two purposes: first, it treats offline and online data as two labeled categories, enabling simultaneous sampling of both types with single training process, thereby reducing time costs; second, it avoids using an additional pre-trained clas- sifier, allowing data augmentation to adapt to varying data distributions in different RL tasks. After data generation, we use a reweighting method to prioritize data that is more aligned with the online policy for agent training, aiming to improve performance. Experimental results show that our CFDG method not only enhances the quality of gener- ated samples compared to existing model- methods but also significantly improves the performance of O20 RL. Our contributions are summarized below:
(o = E(s.a..dd [rr 1Qa (8', (b)) - Qo(s, a)) "))
(1)
where @ denotes the target value network for stabilizing the learning process. Since (s's') is potentially out of the dis- tribution, Qo could give an incorrect value, resulting in sub- optimal policies. To mitigate the well-known extrapolation error in value networks for OOD actions (Fujimoto et al., 2019; Kumar et al., 2020), offline RL methods typically constrain the policy to perform actions close to the dataset through policy constraint (Fujimoto al., 2019; Fujimoto & Gu, 2021; Kumar et al., 2019), value regularization (Ku- mar et al., 2020; An et al., 2021), etc. One representative offline RL method is CQL (Kumar et al., 2020). CQL adds a conservative regularizer R()) to prevent overestimation in the Q- valuue for OOD actions by minimizing the Q-values under the policy æ. The training objective of CQL given by
We analyzed the distributions of offline data, online data, and the generated data in 020 RI, and found that the current method still generates data that is not sufficiently aligned with the online policy.
By treating offline data and online data as two distinct labels, we introduce classifier- free guidance to direct the diffusion model in generating both types of data. We perform reweighting on the generated data to make it more aligned with the online policy.
1 min X (EewD,aw [Q0(s, B...- (Qo(s, a))
- + (o)),
Conservative regularizer R(())
(2)
We conducted experiments on the Locomotion and AntMaze tasks, demonstrating that our data augmen- tation method significantly improves multiple 020 RL algorithms, including IQL, PEX, and APL. Fur- thermore, our approach outperforms the existing data augmentation method EDIS.
where À balances the standard policy improvement loss and conservative regularization.
2.2. Offline-to-online Reinforcement Learning
Building upon the concepts of offline RI., offline-to-online RL aims at enhancing performance by fine-tuning pre- trained offline policy, which contains two phases: (i) offline pre-training. where offline datasets are used to pre-train the policy, and (ii) online fine-tuning, where online interactions are used to refine the pre-trained policy.
2. Preliminaries
We represent the environment as a Markov decision pro- cess (MDP) defined by a tuple AS,A, P, R, po. 7), where S is the state space. A is the action space, P(s'] s, a) is the transition distribution, Po is the initial state distribution, R(s, a) is the reward function and 7 E (0, 1) is the dis- count factor. The objective of RL agent to find policy
Currently, in 020 RL algorithms, there are two paradigms for utilizing offline data and online data. An approach is to set the ratio of online data to offline data at 1:11, ensuring each batch contains an equal split of online and offline data,
2

---

Offline-to-nlinn Reinforcement Learning with Classifter-Free Diffusion Generation
such as PEX (Zhang al., 2023) and Cal-QL (Nakamoto et al., 2024). Another approach utilizes online-offline replay buffer (OORB) in APL (Zheng et al., 2023) and SUNG (Guo et al., 2023), with each batch having a prob- ability of containing online data and a probability of containing offline data. As mentioned in Equation (2), is a trade-off coefficient and decides whether we use the regularizer. When data is sampled from the online buffer, À is set to 0, otherwise 1. Formally, this strategy can be explained below:
the denoising neural network to model the conditional dif- fusion score, i.e., c0 the unconditional score eg (Z t) is jointly estimated by ran- domly dropping the text prompt with a certain probability at each training iteration. Then the gradients for the classifier Pe (c/(r) can be estimated as:
-z V log p(xxlc) while
p
Vz, log p(c Vz, log po Vz, po(s))
1 (col c,t) t)).
O:
{
0 if (s. a) online buffer otherwise.
A x=
(3)
Then the corresponding diffusion score can be derived as:
2.3. Diffusion Models
(0(x2.c,t) = ca( (
co( (6)
Diffusion models (Ho et al., 2020) are class of generative models inspired by non-equilibrium thermodynamics that learn to iteratively reverse a forward noising process and generate samples from noise. Diffusion models define a probability distribution (x0) for the observed data zo by marginalizing over the latent variables ,.....,,TT. where po(2)) := pe(xo:7) )dx17. A forward diffusion chain gradually adds noise the data zo q(00)) in T steps with pre-defined variance schedule 3,, expressed as
where w is set as a global scalar parameter to control the guidance degree of the condition.
3. Classifier-free Diffusion Generation
In order to perform data augmentation for different types of data, in this section, we first visualize the data distribu- tions of offline and online data in O20 RI. and analyze the data generated by EDIS (Liu et al., 2024). To ensure the generated data aligns better with online data, we chose to use classifier-fres guidance direct the diffusion model in generating both online and offline data. This approach eliminates the need for pre-trained classifier before sam- pling, reducing training costs and making it more adaptable to different RL. tasks than other guidance methods.
q(T1TT (o) = IT= ( 11-1); g( (1-1)) = Nax v1 B,TT).
(4)
A reverse diffusion chain, constructed as (= == Nar: 0, 1) Po [x,), is then optimized by maximizing the evidence lower bound defined as E. In Pa Pa( q(2ir [ (Blei al., 2017). After training, sam- pling from the diffusion model involves sampling Fr ~ p(IT) and running the reverse diffusion chain go from
3.1. Data Distribution Analysis
To effectively perform data augmentation in 020 RL, it is essential to conduct a detailed analysis of the distributions of both offline and online data. We conducted experiments on EDIS and simultaneously analyzed the data generated by it. We employed t-SNE (Van der Maaten & Hinton, 2008), dimensionality reduction technique designed to visualize high--dmeesiionl data by mapping it into a low-dimensional space while preserving local structures. This method was used to analyze three types of data: the offline dataset, the online data collected during the fine-tuning process, and the generated data from the diffusion model.
t= T to == 0.
2.4. Classifier-free Guidance
In practical scenarios, growing demand exists to condi- tion the generation a label For example, diffusion models can generate images consistent with input prompts in image synthesis. To address this requirement, classifier guidance (Dhariwal & Nichol, 2021) ) incorporates an auxil- iary classifier Ps (c(=:) to guide the sampling in each reverse denoising step, thereby increasing the likelihood of c given Tr. While this method has demonstrated some performance improvements, training robust classifier for all reverse steps, particularly for the highly noisy input at the initial step, poses a significant challenge and incurs additional training costs.
As shown in the left panel of Figure 1, the offline data is more evenly distributed, while the online data is more dis- persed. The generated data primarily learns from the offline data and is adjusted towards the current online policy using energy guidance. As a result, the generated data retains most of the characteristics of the offline data while also exhibiting some similarities to the online data. This is why EDIS opts
To avoid training& a separate classifier model, classifier-fres guidance (Ho& & Salimans, 2021) takes e as another input of
3

---

Colline-to-Onling Reinforcement Learning with Classifter-Free Diffusion Generation
score estimator 6ø(2)) together with a conditional model po(z|c) parameterized through cg(z,, c). We use sin- gle neural network to parameterize both models, where for the unconditional model we can simply input null token 2 for the class identifier c. when predicting the score, i.e. Eo (zx) = Eg(2,, c = Ø). We jointly train the uncondi- tional and conditional models simply by randomly setting c to the unconditional class identifier Ø with some probability Puncond= set hyperparameter. We then perform sampling using the following linear combination of conditional and unconditional score estimates:
Dur
Figure 1: The t-SNE visualization of offline, online, and generated data. The left plot shows generated data from EDIS, and the right plot shows data from CFDG (Ours).
to replace the offline data with the generated data during the agent training process, using the generated data in con- junction with the online data. Although EDIS allows offline data to align with online data to some extent, it overlooks the fact that online data itself can also be augmented, Given the limited number of online data samples, augmenting on- line data can better adapt to the current policy and provide more samples to strengthen the policy's performance. In the Section 4.2, we also found that direct use of online data for generation yields better results than EDIS.
Eo(2i, c) = (1 w wee
(7)
where w is parameter that controls the degree of the clas- sifier guidance. Eσ is constructed from score estimates that are non-conservative vector fields due to the use of uncon- strained neural networks, so there in general cannot exist a scalar potential such as a classifier log likelihood for which Eg is the classifier-uuin guided score.
Algorithm Classifier-fres guidance sampling in 020 RL. Our additions are highlighted in blue.
Based on the above analysis, directly augmenting online data is more effective than augmenting offline data. How- ever, we also aim to maximize the utilization of both online and offline data, addressing the limited quantity of online data while ensuring that both types of data are better aligned with the current policy. Then, whether augment both types of data naturally led us to consider using conditional diffusion. To better adapt both types of data to the current policy changes, we use classifier guidance to steer the dif- fusion model in generating new data that aligns with the changing data distribution.
Input: Offline phase loss function [ LT
Colline )
online phase loss function {Comline LT ). classifier- guidance diffusion model M.
Initialize: o. online buffer Dons offline buffer Dotts offline synthetic buffer san online synthetic buffer Don_yy
while in offline training phase do
Offine policy training using batches from the offline replay buffer Delr
0 0+
XQ a offline (0), 1 - o A V offline (+)
end while
3.2. Classifier-free Guidance Sampling
while in online training phase do for each environment step do
Classifier-free guidance (Ho & Salimans, 2021) does not require an additional classifier but instead trains the model using both conditional and unconditional inputs based on labels. For the classifier guidance method (Dhariwal & Nichol, 2021), the classifier is trained using noise-corrupted data produced by the forward process of the conditional diffusion model. Consequently, training an additional clas- sifier can be difficult, especially when a significant amount of noise is added to the clean data. Classifier-frr free guidance can solve the above issues and eliminate the need for a pre- trained classifier before sampling, reducing training costs and making it more adaptable to different tasks than the classifier guidance method.
D.m + Dan U {(x s())
end for
if step meets M update frequency then
Update the conditional diffusion model M with sam- ples from Dan and D.n
Generate offline samples from M and add them to DotLsn
Generate online samples from M and add them to Donsyn
end if
for each gradient step do
Sample data from Dum U DanU Dotlayn Densyy
0 + 0 - XQ V L
online (0), o + o - dx V Vo (Ø)
We built on the original code implementation of classifier- free guidance, integrating it into the Elucidated Diffusion Model (Karras et al., 2022). We chose to train an uncon- ditional diffusion model po (z) parameterized through a
end for end while
In the 020 RL setting, we use the classifier-free diffusion
4

---

Offline-to-nliin Reinforcement Learning with Classifter-Free Diffusion Generation
model to augment both online data and offline data during the online phase. Algorithm 1 describes the classifier-free guidance sampling process in detail. To reduce the high time cost of constructing the diffusion model and generating data, we set an update frequency to perform data augmentation using the diffusion model at regular intervals. During the online training phase, we update our diffusion model M using samples from the offline buffer Doff and online buffer Don- Then the diffusion model M can sample synthetic data and add them to the synthetic buffer Doffaayn and Don_ayn- After data generation, we can sample batches from the four buffers mentioned. To make the most of different types of data, we need to design specific data utilization methods tailored to different 020 RL. algorithms.
exploration for the agent. In the right plot of Figure 1, we also show the data distribution generated by our CFDG method. By reweighting and increasing the weight of online synthetic data, we enable the agent to perform sufficient exploration. Compared to EDIS, our data aligns better with the current online policy.
4. Experiments
In this section, we show the efficiency of our CFDG method through empirical validation. Section 4.1 commence by showcasing its excellent performance on the D4RL bench- mark (Fu et al., 2020) and also shows generalizability and statistical improvements on baselines like IQL (Kostrikov et al., 2021), PEX (Zhang et al., 2023) and APL. (Zheng et al., 2023). Section 4.2 compares our method, CFDG, with other model-based approaches such as SynthER (Lu et al., 2024) and EDIS (Liu et al., 2024), highlighting the superiority of our conditional diffusion model over other diffusion models. In Section 4.3, we perform an ablation study to examine two key components of our method, gen- erating online data and generating offline data using CFDG. Both are shown effectively improve the performance of the algorithm.
3.3. Data Reweighting
020 RL algorithms usually combine the offline data and online data with a ratio during the training process. Since the addition of synthetic data, another hyperparamette. the synthetic data ratio is also required in our framework. In our data augmentation method, synthetic data sampled from Doff syn and Dan syn- To better align with the online policy, we adjust the sampling ratio and reweight the data, increasing the weight of the online synthetic data.
We first treat the online synthetic data and offline synthetic data as a whole and define how they should be used in relation to the offline data and online data. According to the two data usage methods mentioned in Section 2.2, we design two new data usage methods for synthetic data. For the first method, which combines 50% online data and 50% offline data, we use a simple method that concatenates synthetic data with sampled online data and offline data. The synthetic data ratio r represents the proportion of synthetic data in each batch. Each batch is concatenation of online data, offline data, and synthetic data.
4.1. Offline-t-onnline RL Experiments
Datasets Our method is mainly validated on two D4RL (Fu et al., 2020) benchmarks: Locomotion and AntMaze, which are used by IQL (Kostrikov et al., 2021) and PEX (Zhang et al., 2023). Locomotion includes di- verse environmental datasets collected by varying quality policies. We assess algorithms on hopper, halfcheetah, and walker2d environment datasets, each with four quality lev- els. AntMaze tasks involve guiding an ant-like robot in mazes of three sizes (umaze, medium, large), each with two different goal location datasets. We focus on the two larger size mazes (medium, large), as opposed to using the umaze, which leaves little room for further improvement). The evaluation environments are listed in Table I's first col- umn. Additional experiments on the Adroit benchmark are shown in Appendix A.
For the second method for utilizing data, online data and offline data are sampled from OORB following Bernoulli distribution. With a probability p. data is sampled from the online buffer, and with probability - p, they are sampled from the offline buffer. Besides, when data is sampled from the online buffer, À is set O, otherwise 1, Equation 3 shows. Since we add synthetic data into the framework, we use simple method where synthetic data can be seen as part of online data or offline data and be used for training as the online data or offline data does. The synthetic data ratio r is also used to represent the proportion of synthetic data in each batch. Each batch is a concatenation of online data and synthetic data or offline data and synthetic data.
Baselines We consider the following baselines: (i) IQL (Kostrikov et al., 2021) learns a value net- work to match the expectile of the critic network to address -utt---diittibttinn problems. (ii) PEX (Zhang et al., 2023) freezes the pre-training policy and introduces policy expansion to enhance exploration. (iii) APL (Zheng et al., 2023) leverages the distinct advantages of offline and online data for adaptive constraints. These three are SOTA 020 RL algorithms and cover two paradigms of data utilization. achieving excellent performance on the D4RL benchmark. According to Section 2.2, IQL and PEX, they follow the
For the ratio of the two types of synthetic data, since online synthetic data aligns better with the current online policy, we choose to increase the weight of online synthetic data. This ensures that the entire synthetic data set supports better
5

---

Oflline-to-nliine Reinforcement Learning with Classifier-Free Diffusion Generation
Table 1: Enhanced performance achieved by CFDG after online fine-tuning on the Loco- motion and AntMaze tasks. We evaluate the normalized scores of standard base algorithms (including IQL. (Kostrikov et al., 2021), PEX (Zhang et al., 2023) and API. (Zheng al., 2023), denoted as "Base") in comparison to the base algorithms augmented with CFDG (referred as "Ours"). All results are assessed across random seeds. The superior scores are highlighted in blue.
IQL
PEX
APL
Dataset'
Base
Ours
Base
Ours
Base
Ours
halfchcctah-r-v2
5316
65 +
78+2
81 7
93:88
103 #03
halfcheetah-mr--
5440
65 +
6813
83 3
76:440
96:22
halfcchetah-mm-v2
69112
75 11
78.15
87 1
77:139
86 28
halfchectah-me--2
95
93
90
-3
96+:3
98 +
hopper-r-v2
16 13
1011
810
8
+
301:40
hopper-mr-v2
66+333
86 + 30
66+225
83 20
88+
100 + 15
hopper-m-v2
93:#
97 7
91::30
100 7
103 2
99:111
hopper-me-v2
68-288
103 + 17
74:22
94 - 19
104#10
112
walker2d-r-v2
15.18
18 1 16
181110
6$ 37
12:111
27 T 42
walker2d-mr-v2
81=17
108 2
101
112
70+:35
109 12
walker2d-m-v2
8817
96 4
10118
108 5
92:126
102 1 21
walker2d-me-v2
113+0
118
116
111+44
111+1
120 + 10
Locomotion total
810
933
890
1024
972
1081
antmaze-mp-v2
82 13
76+55
82:113
88
antmaze-md-v2
82:110
86 5
90:114
98 5
antmaze-lp-v2
48+11
52 + 18
54+19
56 + 21
antmaze-ld-v2
38::16
52 22
38:116
42
AntMaze total
250
266
264
284
r: random, mr: medium-replay, m: medium, me: medium-expert, mp: medium-play. md: medium- diverse, Ip: large-play, ld: large-diverse.
first paradigm, while APL follows the second, allowing us to test the generality of our algorithm.
the baseline algorithm on over 10 datasets in Locomotion tasks and datasets in AntMaze tasks. After using CFDG for data augmentation, APL's performance improved by 11 %, while both IQL and PEX achieved 15% performance improvement.
We use the original papers' implementation for all three baselines. Every experiment starts with training model only using the offline dataset, the same as in the original work. We note that, since APL did not conduct experiments on the AntMaze dataset, we also did not perform experi- ments on it.
4.2. Comparisons between CFDG and Model-based Methods
In addition demonstrating the effectiveness of incorpo- rating CFDG into standard 020 RL algorithms, also aim to prove that CFDG outperforms current SOTA data augmentation methods. We compare CFDG with two model- based methods, SynthER (Lu et al., 2024) and EDIS (Liu et al., 2024). In 020 RL, SynthER directly uses a diffu- sion model to augment online data, while EDIS employs an energy-guided diffusion model generate new data from offline data. The key difference between CFDG and these two methods is that CFDG separates offline data and online data into two distinct labels and uses conditional diffusion model for generation. By accounting for the differences in the distributions of these two types of data and their dis- tinct roles in 020 RL, performing data augmentation for each separately can significantly enhance the algorithm's performance. The base algorithm is IQL., and the results shown in Figure 2 and Figure
Settings For IQL and PEX, we perform IM update steps for offline pre-trainne pre-training and then 1M environment steps for online fine-tuning. For API., we perform 1M pre-training steps and M.IM fine-tuning steps ensure consistency with the original paper. For our data augmentation method, the synthetic buffer size is set to IM. The update frequency of M is 10K in APL and 100K IQL and PEX. The generated data ratio r is set to 1/3. Therefore, the percentage of online data, offline data and generated data is 1 In the generated data, the ratio of generated online data to The above configurations
generated offline data is 8 keep the same across all tasks, datasets and methods. Some detailed CFDG parameters are included in Appendix A.2.
As shown in Table 1, the integration of CFDG outperforms all baselines. Using a diffusion model with classifier-fe free guidance to generate offline data and online data can surpass
6

---

Oflline-toOnliine Reinforcement Learning with Classifter-Free Diffusion Generation
SynthER
EDIS
CFDG
Ha Hallchhetahh-meiium--xeeer
Halfcheetah-mediuurrrplaa
Halfcheetah--medumm
Halfcheetah-random
90
80
80
73
60
70
60
20
20
0
200
400
600
800
1000
200
400
600
500
1000
200
400
600
800
1000
0
200
400
600
800
1000
Hopper-medium-exper
Sopper-medium-replay
Hopper-medium
Hopper-random
Average Return
120
120
30
100
100
80
20
60
40
40
20
20
2.0
600
800
1000
200
400
600
800
1000
200
4.00
600
800
1000
0
200
4000
600
800
1000
Walker2ö-mediiu-eexee
Waikke2do-eedium-replayy
Walker22dmmedium
Walker2d-random
120
100
100
100
60
40
20
Environment Steps (x10²)
Figure 2: Learning curves of base algorithm augmented with model-basee methods SynthER, EDIS and CFDG. Results averaged over 5 random seeds.
Aggregated Performance
Locomotion tasks, shown in Figure 3.
SynihhER
verify that the data generated by our method has higher quality and is better aligned with the current on- line policy, we analyzed the Jensen-Shannon (JS) diver- gence (Menéndez et al., 1997) between the data generated by EDIS and CFDG and the online data. We also compared the JS divergence between offline data and online data, as shown in Table 2. Our CFDG method generates data that is more closely aligned with the online data, which can be used for agent training, leading to improved performance.
Average
200
Environment Steps (x10P)
Figure 3: Aggregated learning curves of normalized return over seeds on 12 Locomotion tasks.
Table 2:
Divergence comparisons for generated data from (lower is better). Each result is the average score over random seeds.
Based on the experimental results, SynthER which using online data for data augmentation outperforms EDIS, which performs data augmentation based on offline data. aligns with the intuition that online data more aligned with the current policy. However, using CFDG to augment both online and offline data simultaneously further improves per- formance. This is particularly evident the halfcheetah environment, where CFDG achieves 15% performance improvement over previous model-basee methods across
Generated EDIS
Generated
Divergence
Offline
CFDG
State
0.69:F0.01
0.56+00.22
0.50
Action
0.32++0.03
0.31+0.02
0.27-0.02
Transition
0.6910..3
0.6410.04
0.61
Compared to other model-based methods, CFDG can gen- erate higher-quality data samples. summary, performing data augmentation on both online and offline data allows

---

Offlineto-Online Reinforcement Learning with Classifter-Free Diffusion Generation
the agent to train on more comprehensive samples, leading to better performance.
Recent studies place a growing emphasis on adaptability, PEX (Zhang et al., 2023) freezes the pre-trained policy and initializes random policy to enhance exploration. From data-centric perspective, APL (Zheng et al., 2023) and SUNG (Guo et al., 2023) impose constraints exclusively on data from offline datasets and data with high uncertainty, respectively. Our method also focuses on data utilization, us- ing data augmentation to expand the available data, thereby enabling more comprehensive learning and improving the agent's performance.
4.3. Ablation Studies
The two main differences between the CFDG method and ex- isting model-based approaches are: (i) It uses classifier-fres free guidance to steer the diffusion model in generating new data. (ii) It performs data augmentation on both offline data and online data. Therefore, we need to demonstrate that both components effectively enhance performance. We conduct an ablation study on the Locomotion tasks to compare the full CFDG method against its variants: one without classifier guidance and another with classifier guidance but without offline data augmentation. We evaluate performance across three environments: halfcheetah, hopper, and walker2d, av- eraging the performance across four tasks.
Diffusion Models in RL Pearce et al. (Pearce et al., 2022) propose using diffusion model to better imitate human behaviors due to their expressiveness and stability. Dif- fuser (Janner et al., 2022) applies a diffusion model as a trajectory generator, where the full trajectory of state-action pairs forms a single sample for the diffusion model. Ad- ditionally, a separate return model trained to predict the cumulative rewards of each trajectory sample, and its guid- ance is incorporated into the reverse sampling stage. This approach is similar to Decision Transformer (Chen et al., 2021), which also learns a trajectory generator through GPT- 2 with the help of the true trajectory returns. However, when used in online settings, sequence models can no longer pre- dict actions from states autoregressively since the states are an outcome of the environment. Additionally, diffusion models are also used for data augmentation in RL. Syn- thER (Lu et al., 2024) directly uses a diffusion model to augment online data. EDIS (Liu et al., 2024) employs an energy-guided diffusion model to generate new data from offline data. Our method also uses diffusion models for data augmentation. However, considering the unique character- istics of the O20 RL settings and the differences between offline and online data, we utilize classifier- free guidance to generate new data.
Table 3: Ablation results of CFDG on Locomotion tasks. All results are averaged over the four datasets and are as- sessed across 5 random seeds.
CFDG w/o guidance
CFDG w/o offline DA
Dataset
CFDG
Halfcheetah
80.65+2.43
81.22+1.78
84.44+2.115
Hopper
65.75+9..1
6..50+7..35 81.59-14.83
68.22++4.74
Walker2d
75.9916.12
93.65.16.00
Average
74.13
76.77
82.10
As shown in Table 3, compared to standard diffusion, using classifier guidance improves the quality of data generation and enhances the performance of the agent after data aug- mentation. Building on classifier guidance, applying data augmentation to both offline and online data further im- proves the agent's performance. This highlights that both classifier guidance and data augmentation for both types of data are essential components of our CFDG method, con- tributing significantly to performance improvement.
6. Conclusion
5. Related Work
In this paper, we analyze the distributions of offline and on- line data in the O20 RL setting and improve upon existing data augmentation methods by addressing their limitations. We use a diffusion model with classifier-free guidance to simultancously augment both types of data. In the online phase. we input offline data and online data as two distinct labels into the diffusion model. With a single round of train- ing, we can sample both types of data. Our method, CFDG, is simple and can be easily integrated with existing 020 RL algorithms, significantly boosting their performance while surpassing other data augmentation methods. Although our method has achieved superior performance, there is still room for future work. We can explore fine-tuning large pre-trdined models and leveraging their generalization capa- bilities to synthesize new pixel- environment data in RL.
Offline-to-online Reinforcement Learning Offline-to- online RL aims to improve suboptimal offline policies through online fine-tuning. Prior works usually improve the agent by adding a regularizer to mitigate the distribution shift problem or leverage both offline data and online data in online fine-tuning. IQL (Kostrikov et al., 2021) incorporates a weighted behavioral cloning step to enhance online policy improvement and applicable in both online and offline-to- online scenarios. OFF2ON (Lee et al., 2022) employs a bal- anced replay scheme to address the distribution shift issue. It uses offline data by only selecting ne-olcn- samples. However, practical scenarios may involve agents pretrained by various offline RL algorithms, highlighting the necessity for developing generic offline-to-online RI. framework.
8

---

Offlineto-Online Reinforcement Learning with Classifter-Free Diffusion Generation
Acknowledgement
S. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
The corresponding author Shuai Li is sponsored by CCF- Tencent Open Research Fund.
Fujimoto, S. and Gu, S. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20 20145, 2021.
Impact Statement
Fujimoto, S., Meger, D., and Precup. D. Off-policy deep reinforeement learning without exploration. In Interna- tional conference machine learning, pp. 2052-2062. PMLR, 2019.
This work presents novel approach to data augmentation in Offline-to-Online Reinforcement Learning (020 RL). By improving the aligmment between generated data and online policy distributions, our method enhances training stability and overall agent performance while reducing com- putational overhead. The broader impact of this research includes advancing the efficiency of RL algorithms, po- tentially accelerating adoption in real-world applications such as robotics, autonomous systems, and decision-making frameworks that require fine-tuning with limited online in- teractions.
Guo, S., Sun, Y., Hu, J., Huang, S., Chen, H., Piao, H., Sun, L., and Chang, Y. A simple unified uncertainty-guidd reinforcement learning.
framework for arXiv preprint wrXiv:2306.07544 2023.
Haarnoja, T., Zhou, A., Abbeel, P., and Levine, S. Soft actor-critic: Off-policy maximum entropy deep reinforce- ment learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
From an ethical perspective, our method does not inherently introduce bias but relies the quality and representative- ness of offline datasets. Future work should ensure diverse and unbiased data selection to mitigate unintended conse- quences. Additionally, while our approach optimizes RI. performance, care must be taken in high-stakes applications to prevent over-reliance on synthetic data without proper validation.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. NeurlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Ho, J., Jain, A., and Abbeel, Denoising diffusion proba- bilistic models. Advances in neural information process- ing systems, 33:6840-6851, 2020.
Overall, this research contributes the broader field of machine learning by improving sample efficiency in rein- forcement learning while maintaining ethical considerations in data-driven decision-making.
Huang, V., Ley, T., Vlachou Baathou-Konchlaki... M., and Hu, W. Enhanced experience replay generation for efficient re- inforcement learning. arXiv preprint 1rXiv:1705.0824 2017.
References
Imre, B. An investigation of generative replay in deep reinforcement learning. B.S. thesis, University of Twente, 2021.
An, G., Moon, S., Kim, J.-H., and Song, H. O. Uncertainty- based offline reinforcement learning with diversified q- ensemble. Advances neural information processing systems, 34:7436-7447, 2021.
Janner, M., Du, Y.. Tenenbaum, J., and Levine, S. Planning with diffusion for flexible behavior synthesis. In Interna- tional Conference Machine Learning, pp. 9902-9915. PMLR, 2022.
Blei, D. M., Kucukelbir, A., and McAuliffe, J. D. Varia- tional inference: A review for statisticians. Journal of the American statistical Association, 112(518):859-88) 877, 2017.
Karras, T., Aittala, M., Aila, T., and Laine, S. Elucidating the design space of diffusion- based generative models, Advances in Neural Information Processing Systems, 35: 26565-26577, 2022.
Chen, L., Lu, K., Rajeswaran, A., Lee, K., Grover, A., Laskin, M., Abbeel, P., Srinivas, A., and Mordatch, I. De- cision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:1 5084-5594,, 2021.
Kidambi, R., Rajeswaran, A., Netrapalli, P., and Joachims, T. Morel: Model-based offline reinforcement learning. Advances in newral information processing systems, 33: 21810-21823, 2020.
Dhariwal, P. and Nichol, A. Diffusson models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 8794, 2021.
Kostrikov, I., Fergus, R., Tompson, J., and Nachum, O. Of- fline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp- 5774-5783, PMLR, 2021.
Fu, J., Kumar, A., Nachum, O., Tucker, G., and Levine,

---

Offline-to-nline Reinforcement Learning with Classifter-Free Diffusion Generation
Kumar, A., Fu, J., Soh, M., Tucker, G., and Levine, S. Stabilizing off off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Zheng, H., Luo, X., Wei, P., Song, X., Li, D., and Jiang. J. Adaptive policy learning for offline-toooninne reinforce- ment learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1 1137--11800 2023.
Kumar, A., Zhou, A., Tucker, G., and Levine, S. Con- servative q-learning for offline reinforcement learning- Advances in Neural Information Processing Systems, 33; 1179-1191, 2020.
Lee, S., Seo, Y., Lee, K., Abbeel, P., and Shin, J. Offline- to-online reinforcement learning via balanced replay and pessimistic q g-enseblle. In Conference on Robot Learn- ing, pp. 1702-1712. PMLR, 2022.
Levine, S., Kumar, A., Tucker, G., and Fu, J. Offline rein- forcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint XXiv:2005.01643 2020.
Liu, X.-H., Liu, T.-S., Jiang, S., Chen, R., Zhang, Z., Chen, X., and Yu, Y. Energy-guided diffusion sampling for offline-to-online reinforcement learning. In Proceedings of the 41st International Conference on Machine Learn- ing, pp. 31541-111111 2024.
Lu, C., Ball, P., Teh, W., and Parker-Holder, J. Syn- thetic experience replay, Advances Neural Information Processing Systems, 36, 2024.
Menéndez, M. L., Pardo, J., Pardo, L., and Pardo, M. The jensen-shannon divergence. Journal of the Franklin Insti- tute, 334(():307-318 1997.
Nair, A., Gupta, A., Dalal, M., and Levine, S. Awac: Accel- erating online reinforcement learning with offline datasets. arXiv preprint XiXXix:20060933550 2020.
Nakamoto, M., Zhai, S., Singh, A., Sobol Mark, M., Ma, Y., Finn, C., Kumar, A., and Levine, S. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Ad- vances in Neural Information Processing Systems, 36, 2024.
Pearce, T., Rashid, T., Kanervisto, A., Bignell, D., Sun, M., Georgescu, R., Macua, S. V., Tan, S. Z. Momennejad, I., Hofmann, K., et al. Imitating human behaviour with diffu- sion models. In Deep Reinforcement Learning Workshop NeurlPS 2022, 2022.
Thrun, S. and Littman, M. L.. Reinforcement learning: an introduction. AI Magazine, 2131):1031133, 2000.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Zhang, H., Xu, W., and Yu, H. Policy expansion for bridging cifline-to-onling reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023.
10

---

Offline-toOnlinn Reinforcement Learning with Classifter-Free Diffusion Generation
A. Appendix
A.1. Adroit Results
To further validate the effectiveness and generalization our method, we conduct additional experiments on the Adroit cen-human-vl, and door-human-vl), suite of challenging and realistic continuous control tasks designed test fine motor skills in a human-like hand. Our base algorithm in the Adroit setting is IQL, as EDIS is also evaluated top of IQL.. As shown in Table 4, proposed method, CFDG, significantly outperforms both the base model and the prior state-of-the-art approach EDIS across all three tasks.
benchmark relocate-human-t
Table 4: Comparisons between CFDG and EDIS on Adroit tasks. Each result the average score over 5 random seeds.
Dataset
Base
EDIS
CFDG
relocat-humaan-e 0,4+0.,4
0.2
1.5
pen-human-vl 72.2-164.0
10.4
59,4
door-human-vl
4.5
6.2.13.2
31,7:16.5
Average
25.7
26.6
43.3
Our results show that CFDG achieves over 50% improvement over baselines and the model-based EDIS approach on Adroit tasks. These results demonstrate that CFDG is not only effective in standard benchmark settings but also robust more realistic and difficult environments, further supporting its applicability to real-world scenarios.
A.2. Details and Hyperparameters of CFDG
We use the PyTorch implementation IQL and PEX from https://github.com/// haaia//hhoooomiaaaaaaaaaaaa//// and the implementation of APL from https///iidub.com/ha099999 com/zhan0903//PPL and primarily followed the authors' recommended parameters. The hyperparameters used in our CFDG module are detailed in Table 5:
Table 5: Hyperparameters and their values in CFDG.
Hyperparameter
Value
Denoising Network
Residual MLP
Denoising Network Depth
layers
Denoising Steps
128 steps
Denoising Network Learning Rate
3 10
Denoising Network Hidden Dimension
1024 units
Denoising Network Batch Size
256 samples
Denoising Network Activation Function
ReLU
Denoising Network Optimizer
Adam
Learning Rate Schedulc
Cosine Annealing
Training Epochs Training Interval Environment Step
100K epochs
10K steps (APL), 100K steps (IQL PEX)
The formulation of diffusion we use in our paper the Elucidated Diffusion Model (Karras al., 2022). We parametrize the denoising network Dg an MLP with skip connections from the previous layer. The base size of the network uses width of 1024 and depth of 6. We a batch size of 256 for all tasks, For the diffusion sampling process, we the stochastic SDE sampler (Karras et al., 2022) with the default hyperparameters used for the ImageNet. We use higher number of diffusion timesteps at 128 for improved sample fidelity. We use the implementation at diffusion-pytorch.
A.3. Computational Resources
We train CFDG integrated with base algorithms on an NVIDIA RTX 2080Ti, with approximately 23 hours required for 10K fine-tuning on MuJoCo Locomotion tasks in APL, while 16 hours for 100K fine-tuning in IQL & PEX. The detailed computational consumption is shown Table 6. As pointed out (Karras et al., 2022), the sampling time is faster than prior diffusion designs, which is much shorter compared with training. The introduction of the diffusion model does indeed entail an inevitable increase in computational and time costs. However, this tradeoff between improved performance and

---

Offline-to-nllin Reinforcement Learning with Classifter-Free Diffusion Generation
higher computational cost is common consideration in diffusion model research. In our future work, aim further refine and optimize the extra costs.
Table 6: Computational consumption of different algorithms.
Algorithm
Online phase training time
Maximal GPU memory
APL
20h
2G
APL-CEDG
23h
3G
PEX
14h
2G
PEX-CFDG
3G
IQL
12h
2G
JQL-CFDG
15h
3G
12


Reviews:
[{"review_id": "KjyPAYgqfu", "reviewer": "Reviewer_JMww", "summary": "This paper introduces Classifier-Free Diffusion Generation (CFDG), a model-based data augmentation method for offline-to-online RL. The key idea is to train a diffusion-based data generation model with classifier-free guidance to differentiate between online and offline data. The generated data is then used to augment real data during online RL. In the experiments, CFDG was integrated with several RL methods, and was also compared to other data augmentation approaches, demonstrating performance improvements", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.\n\n**[W1] Integration with Other Online RL Algorithms**\n\nAlthough our primary focus is offline-to-online RL, our approach can be easily integrated with other online RL algorithms that leverage offline data. The adaptive Policy Learning (APL) algorithm [1], for instance, uses different update strategies for online and offline data in the online phase—employing online RL updates for online data. If APL is not pre-trained offline, it can be considered as an online RL algorithm utilizing offline data.\n\nIn our experiments with locomotion tasks (halfcheetah, hopper, and walker2d) using APL, we observed that omitting offline pretraining led to a decline in performance. However, our CFDG method still performed well even in the absence of pretraining, demonstrating its robustness. \n\n| Dataset     |    Base     |    CFDG     | Base w/o offline pretrain | CFDG w/o offline pretrain |\n| ----------- | :---------: | :---------: | :-----------------------: | :-----------------------: |\n| Halfcheetah | $85.5±22.5$ | $95.8±9.8$  |        $72.0±24.4$        |        $92.5±8.3$         |\n| Hopper      | $86.5±16.0$ | $85.3±16.8$ |        $83.8±21.1$        |        $84.2±17.3$        |\n| Walker2d    | $71.3±18.3$ | $89.5±21.3$ |        $70.3±25.2$        |        $79.8±19.3$        |\n| **Average** |   $81.1$    |   $90.2$    |          $75.4$           |          $85.5$           |\n\nAll results are averaged over the four datasets and are assessed across 5 random seeds.\n\n**[W2] Guidance Weight in Classifier-Free Guidance**\n\nIn our classifier-free guidance approach, the guidance weight is set to 1. We have conducted an ablation study where we varied the guidance weight from 0 to 5 (in increments of 0.1). The results indicated that the performance is largely unaffected by this parameter, except when the guidance weight is set to 0. In this case, the diffusion model becomes unconditional, leading to a noticeable drop in performance.\n\n**[W3] Use of Diffusion Model in Offline Pretraining Phase**\n\nWe also experimented with using the diffusion model for data augmentation during the offline pretraining phase. While this approach improved the performance immediately after offline pretraining, the benefits were not carried over once the online phase began.\n\n| Dataset     |          CFDG           | CFDG w/ DA in offline phase |\n| ----------- | :---------------------: | :-------------------------: |\n| Halfcheetah | $48.2 \\rightarrow 74.5$ |   $51.1 \\rightarrow 73.8$   |\n| Hopper      | $37.7 \\rightarrow 74.0$ |   $65.3 \\rightarrow 75.1$   |\n| Walker2d    | $52.1 \\rightarrow 85.1$ |   $59.7 \\rightarrow 84.3$   |\n\nThe base algorithm of our experiment is IQL. All results are averaged over the four datasets and are assessed across 5 random seeds. As shown in our results ($\\text{A} \\rightarrow \\text{B}$ where $\\text{A}$ represents the offline pretraining score and $\\text{B}$ represents the online fine-tuning score), using the diffusion model for data augmentation in the offline phase did not result in sustained improvements in the online phase.\n\nThis can be attributed to the fact that the performance bottleneck in offline-to-online RL is mainly determined by the quality of online data. Our CFDG method addresses this issue by using the diffusion process to learn the distributional differences between offline and online data, ultimately producing data that aligns more closely with the online policy. This alignment is critical for improving the upper performance limit of the agent.\n\n[1] Adaptive policy learning for offline-to-online reinforcement learning.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No theoretical components were presented.", "experimental_designs_or_analyses": "Yes", "relation_to_broader_scientific_literature": "The paper will contribute to the growing use of generative models (specifically diffusion models) within RL.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Strengths:\n\n- The idea of leveraging classifier-free guidance for data generation is simple but clear.\n- The method has the flexibility to be combined with various RL methods.\n- The presented results are compelling, as they show improvements over several RL methods as well as other data augmentation techniques.\n\nWeaknesses / Questions:\n\nThere are several weaknesses and parts that are unclear to me:\n\n- In Section 4.1, while the method was combined with several offline-to-online RL methods, in principle, it can also be integrated with other online RL algorithms that leverage offline data. Would it be possible to evaluate whether the proposed approach can improve SOTA sample-efficient RL algorithms, such as RLPD [1]?\n\n- While the method uses classifier-free guidance, what value is used for the guidance weight when generating offline/online data? Is the performance sensitive to this value? It would also be interesting to provide further ablation studies on this parameter, as it directly influences how online or offline the generated data is.\n\n- During the offline pre-training phase, why is the diffusion model not trained and used for augmentation, but instead only introduced after the online phase begins? How would the performance be affected if the diffusion model were also pre-trained and used from the beginning of the offline phase?\n\n[1] Ball et al., Efficient Online Reinforcement Learning with Offline Data, 2023", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See the Weaknesses section", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lXMihCwM4H", "reviewer": "Reviewer_dppj", "summary": "This paper introduces CFDG, a framework that applies data augmentation to both offline and online datasets in offline-to-online algorithms.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable review! If you have any additional comments, please feel free to share them—we would be happy to address any questions or clarifications you may have.", "claims_and_evidence": "The paper conducts extensive experiments on the D4RL dataset to evaluate the effectiveness of the proposed framework.", "methods_and_evaluation_criteria": "Yes, the evaluation criteria adopted are a common practice in this area.", "theoretical_claims": "No major theoretical claims are presented in the paper.", "experimental_designs_or_analyses": "Yes, I have reviewed the experimental designs used to evaluate the effectiveness of the proposed framework on the D4RL benchmarks. Additionally, the ablation studies make sense.", "relation_to_broader_scientific_literature": "This paper falls within the area of offline-to-online RL algorithms and introduces a data augmentation approach to enrich existing offline and online datasets.", "essential_references_not_discussed": "Current references are appropriate.", "other_strengths_and_weaknesses": "No further comments.", "other_comments_or_suggestions": "No further comments.", "questions_for_authors": "No further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vqyv99WAOd", "reviewer": "Reviewer_3ebJ", "summary": "Offline-to-online Reinforcement Learning (O2O RL) aims to perform online fine-tuning on an offline pre-trained policy to minimize costly online interactions. To this end, existing work used offline datasets to augment online data. However, a distribution gap exists between the generated data and the online data, limiting overall performance. Hence, the authors propose a new data augmentation approach, Classifier-Free Diffusion Generation (CFDG). By leveraging classifier-free generation developed in diffusion models, CFDG enhances the\ngeneration quality of offline and online data. It also employs a reweighting method to better align generated data with the online data. Experiments validate these claims in the widely used D4RL benchmark.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.\n\n**[W1] Technical Novelty and Insights**\n\nWhile our method is simple and intuitive, its effectiveness lies in its ability to improve offline-to-online RL performance across various tasks. By leveraging classifier-free guidance, our approach enables better alignment between generated data and the online policy, leading to substantial performance gains.\n\n**[W2] Additional Experiments on Challenging Offline RL Benchmarks**\n\nTo further validate our method, we conducted additional experiments on **Adroit tasks**, which are widely recognized as challenging benchmarks for offline RL due to their high-dimensional action space and complex dexterous manipulation requirements.\n\nWe compared CFDG against EDIS [1], a state-of-the-art model-based approach, as well as standard baselines. The results demonstrate that CFDG achieves over 50% improvement over both baselines and EDIS on Adroit tasks, further confirming its effectiveness in robotic manipulation scenarios. \n\n| Dataset           |   Base    |   EDIS    |   CFDG    |\n| ----------------- | :-------: | :-------: | :-------: |\n| relocate-human-v1 |  0.4±0.4  |  0.2±0.6  |  1.5±1.8  |\n| pen-human-v1      | 72.2±64.0 | 73.4±10.3 | 96.8±59.4 |\n| door-human-v1     |  4.5±7.9  | 6.20±3.2  | 31.7±6.5  |\n| **Average**       |   25.7    |   26.6    |   43.3    |\n\n[1] Energy-guided diffusion sampling for offline-to-online reinforcement learning.", "claims_and_evidence": "Experimental results support the author's claims but experiments in more challenging environments, e.g., manipulation tasks, are required to further evaluate the effectiveness of this method.", "methods_and_evaluation_criteria": "The proposed method is simple and intuitive: it treats offline and online data as two labeled categories, enabling simultaneous sampling of both types with a single diffusion training process; it avoids using an additional pre-trained classifier, allowing flexible data augmentation to adapt to varying data distributions in different RL tasks.", "theoretical_claims": "no need to check the correctness of theoretical claims.", "experimental_designs_or_analyses": "Experimental results in more challenging environments, e.g., robotic manipulation tasks, are required to further evaluate the effectiveness of this method.", "relation_to_broader_scientific_literature": "The proposed method is closely related to the realm of generative modelling and directly adopts the mature techniques that are developed by classifier-free diffusion models to address the offline and online data generation in O2O RL.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Although the proposed method is simple and intuitive, its technical novelty and insights are limited. Moreover, I expect that more experimental results can be conducted in more challenging offline RL benchmarks.", "other_comments_or_suggestions": "no", "questions_for_authors": "no", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u3bExsc9pI", "reviewer": "Reviewer_MxTS", "summary": "The paper proposes CFDG, a novel method of generative data augmentation for offline-to-online RL. The paper points out that offline and online data have different characteristics that are important for high performance. To this end, CFDG employs a conditional diffusion model where the condition is a binary value: offline / online. Experiment results validate that it achieves competitive performance in various tasks.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive review and valuable suggestions! Below, we provide a detailed response to your questions and comments. If any of our responses fail to sufficiently address your concerns, please inform us, and we will promptly follow up.\n\n**[W1] Training Separate Diffusion Models for Offline and Online Data**\n\nWe conducted experiments on locomotion tasks (halfcheetah, hopper, and walker2d) where we trained separate diffusion models for offline and online data. The results show that our CFDG approach outperforms training two independent diffusion models.\n\n| Dataset     | Independent |    CFDG    |\n| ----------- | :---------: | :--------: |\n| Halfcheetah | 81.02±1.84  | 84.44±2.15 |\n| Hopper      | 66.23±8.32  | 68.22±4.74 |\n| Walker2d    | 75.89±7.13  | 93.65±6.00 |\n\nAll results are averaged over the four datasets and are assessed across 5 random seeds. The key reason for this improvement is that classifier-free guidance enables the generation of data that is better aligned with the online policy. As demonstrated in Section 4.2 of our paper, conditioning the diffusion model on both offline and online data labels allows for more effective guidance, reducing redundancy between newly generated online data and existing offline data. This ultimately enhances the quality of online data, leading to better performance.\n\nWhile the computational cost of training two separate models is not prohibitive in terms of GPU resources, it doubles the training time of diffusion model, making it less efficient without yielding better results.\n\n**[W2] Experiments on More Realistic Tasks**\n\nTo further validate our approach, we conducted additional experiments on Adroit tasks, which are considered more challenging due to their high-dimensional action space and dexterous control requirements. The specific results are shown here.\n\n| Dataset           |   Base    |   EDIS    |   CFDG    |\n| ----------------- | :-------: | :-------: | :-------: |\n| relocate-human-v1 |  0.4±0.4  |  0.2±0.6  |  1.5±1.8  |\n| pen-human-v1      | 72.2±64.0 | 73.4±10.3 | 96.8±59.4 |\n| door-human-v1     |  4.5±7.9  | 6.20±3.2  | 31.7±6.5  |\n| **Average**       |   25.7    |   26.6    |   43.3    |\n\nOur results show that CFDG achieves over 50% improvement over baselines and the model-based EDIS [1] approach on Adroit tasks, demonstrating its effectiveness beyond locomotion and AntMaze tasks.\n\n[1] Energy-guided diffusion sampling for offline-to-online reinforcement learning.", "claims_and_evidence": "The main claim is straightforward: we need to use a conditional diffusion model to augment the dataset for the offline-to-online RL problem. The claim is well supported by the data distribution analysis section 3.1.", "methods_and_evaluation_criteria": "The authors follow the conventional experiment settings of offline-to-online RL.", "theoretical_claims": "There is no theoretical claim.", "experimental_designs_or_analyses": "The authors follow the conventional experiment settings of offline-to-online RL.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N\\A", "other_strengths_and_weaknesses": "Please see above", "other_comments_or_suggestions": "There are several comments and suggestions listed below.\n\n1. As the key motivation is that there are some different characteristics between the offline and online datasets, why don’t we train separate diffusion models to augment each of them? As there are only two classes, it seems that the cost of training two models is not expensive. I recommend that the authors compare the results with the independent diffusion model.\n2. Experiments are conducted on Lomocotion and Antmaze tasks. I recommend that the authors conduct experiments on more realistic tasks such as Adroit.", "questions_for_authors": "Please see above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Offline-to-Online Reinforcement Learning with Classifier-Free Diffusion Generation

Paper-Summary: This paper introduces Classifier-Free Diffusion Generation (CFDG), a data augmentation framework for offline-to-online reinforcement learning that enhances the fine-tuning of offline pre-trained policies with limited online interactions. The key idea is to use a classifier-free guided diffusion model to generate synthetic samples conditioned on both offline and online data, thus bridging the distribution shift between them. Unlike prior methods that rely on separate classifiers or energy-guided sampling, CFDG leverages a single diffusion model trained with label-free conditioning to simultaneously synthesize both types of data, followed by a reweighting strategy to prioritize samples aligned with the current policy. The method integrates with existing offline-to-online  RL algorithms, yielding consistent improvements  across multiple D4RL benchmarks. 

Review and feedback: We received 4 expert reviews with the scores 2, 3, 3, 3 and the average score is 2.75 

The reviewers are positive about many aspects of the paper. The proposed method is conceptually simple, adapting a well-established generative modeling technique (classifier-free diffusion) to the offline-to-online RL setting in a practical way. The empirical results show consistent improvements across multiple tasks in the Locomotion and AntMaze domains, supported by ablation studies and distribution shift analyses. The paper is well-written and clearly structured, with clean experimental protocols and reproducible implementation details provided in the appendix.

At the same time, reviewers have pointed out many weaknesses. While the application of classifier-free guidance to offline-to-online RL is novel, reviewers noted that the methodological contribution is modest, as it largely repurposes known components (e.g., conditional diffusion, sample reweighting) without significant new theoretical insights or algorithmic innovation. No theoretical analysis is provided regarding why classifier-free guidance is preferable in this setting, or under what conditions the approach would generalize or fail. Reviewers have also asked for additional experiments and comparisons with some recent baselines.  

I recommend the authors to address these concern while preparing the resubmission.
Decision: Accept (poster)