SlowGuess/NeurIPS_2023_OCR huggingface dataset:
generates samples that have significantly lower dynamics MSE than explicit augmentations, even for datapoints that are far away from the training data. This implies that a high level of generalization has been achieved by the SYNTHER model, resulting in the ability to generate novel, diverse, yet dynamically accurate data that can be used by policies to improve performance.
4.1.2 Scaling Network Size
A further benefit we observe from SYNTHER on the TD3+BC algorithm is that upsampled data can enable scaling of the policy and value networks leading to improved performance. As is typical for RL algorithms, TD3+BC uses a small value and policy network with two hidden layers, and width of 256, and a batch size of 256. We consider increasing the size of both networks to be three hidden layers and width 512 (approximately  $6 \times$  more parameters), and the batch size to 1024 to better make use of the upsampled data in Table 3.
We observe a large overall improvement of  $11.7\%$  for the locomotion datasets when using a larger network with synthetic data (Larger Network + SYNTHER). Notably, when using the original data (Larger Network + Original Data), the larger network performs the same as the baseline. This suggests that the bottleneck in the algorithm lies in the representation capability of the neural network and synthetic samples from SYNTHER enables effective training of the larger network. This could alleviate the data requirements for scaling laws in reinforcement learning [1, 28]. However, for the IQL and EDAC algorithms, we did not observe an improvement by increasing the network size which suggests that the bottleneck there lies in the data or algorithm rather than the architecture.
4.2 Online Evaluation
Next, we show that SYNTHER can effectively upsample an online agent's continually collected experiences. In this section, we follow the sample-efficient RL literature [12, 16] and consider 3 environments from the DeepMind Control Suite (DMC, Tunyasuvunakool et al. [71]) (cheetah-run, quadruped-walk, and reacher-hard) and 3 environments the OpenAI Gym Suite [10] (walker2d, halfcheetah, and hopper). As in Chen et al. [12], D'Oro et al. [16], we choose the base algorithm to be Soft Actor-Critic (SAC, Haarnoja et al. [24]), a popular off-policy entropy-regularized algorithm, and benchmark against a SOTA sample-efficient variant of itself, 'Randomized Ensembled Double Q-Learning' (REDQ, Chen et al. [12]). REDQ uses an ensemble of 10 Q-functions and computes target values across a randomized subset of them during training. By default, SAC uses an update-to-data ratio of 1 (1 update for each transition collected); the modifications to SAC in REDQ enable this to be raised to 20. Our method, 'SAC (SYNTHER)', augments the training data by generating 1M new samples for every 10K real samples collected and samples them with a ratio  $r = 0.5$ . We then match REDQ and train with a UTD ratio of 20. We evaluate our algorithms over 200K online steps for the DMC environments and 100K for OpenAI Gym.
Figure 6: SYNTHER greatly improves the sample efficiency of online RL algorithms by enabling an agent to train on upsampled data. This allows an agent to use an increased update-to-data ratio (UTD=20 compared to 1 for regular SAC) without any algorithmic changes. We show the mean and standard deviation of the online return over 6 seeds. DeepMind Control Suite environments are shown in the top row, and OpenAI Gym environments are shown in the bottom.
In Figure 6, we see that SAC (SYNTHER) matches or outperforms REDQ on the majority of the environments with particularly strong results on the quadruped-walk and halfcheetah-v2 environments. This is particularly notable as D'Oro et al. [16] found that UTD=20 on average decreased performance for SAC compared with the default value of 1, attributable to issues with overestimation and overfitting [12, 48]. We aggregate the final performance on the environments in Figure 1b, normalizing the DMC returns following Lu et al. [50] and OpenAI returns as in D4RL. Moreover, due to the fast speed of training our diffusion models and fewer Q-networks, our approach is in fact faster than REDQ based on wall-clock time, whilst also requiring fewer algorithmic design choices, such as large ensembles and randomsubseting. Full details on run-time are given in Appendix D.2.
4.3 Scaling to Pixel-Based Observations
Finally, we show that we can readily scale SYNTHER to pixel-based environments by generating data in the latent space of a CNN encoder. We consider the V-D4RL [50] benchmarking suite, a set of standardized pixel-based offline datasets, and focus on the 'cheetah-run' and 'walker-walk' environments. We use the associated  $\mathrm{DrQ + BC}$  [50] and BC algorithms. Whilst the original image observations are of size  $84\times 84\times 3$ , we note that the CNN encoder in both algorithms generates features that are 50 dimensional [77]. Therefore, given a frozen encoder pre-trained on the same dataset, we can retain the fast training and sampling speed of our proprioceptive models but now in pixel space. We present full details in Appendix F.
Analogously to the proprioceptive offline evaluation in Section 4.1, we upsample 5M latent transitions for each dataset and present downstream performance in Table 4. Since the V-D4RL datasets are smaller than the D4RL equivalents with a base size of 100K, we would expect synthetic data to be beneficial. Indeed, we observe a statistically significant increase in performance of  $+9.5\%$  and  $+6.8\%$  on  $\mathrm{DrQ + BC}$  and BC respectively; with particularly strong highlighted results on the medium and expert datasets. We believe this serves as compelling evidence of the scalability of SYNTHER to high-dimensional observation spaces and leave generating data in the original image space, or extending this approach to the online setting for future work.
Whilst generative training data has been explored in reinforcement learning; in general, synthetic data has not previously performed as well as real data on standard RL benchmarks.
Generative Training Data. Imre [34], Ludjen [51] considered using VAEs and GANs to generate synthetic data for online reinforcement learning. However, we note that both works failed to match the original performance on simple environments such as CartPole—this is likely due to the use of a
Table 4: We scale SYNTHER to high dimensional pixel-based environments by generating data in the latent space of a CNN encoder pre-trained on the same offline data. Our approach is composable with algorithms that train with data augmentation and leads to a  $+9.5\%$  and  $+6.8\%$  overall gain on  $\mathrm{DrQ + BC}$  and BC respectively. We show the mean and standard deviation of the final performance averaged over 6 seeds.
Environment	DrQ+BC [50]	BC [50]
Original	SynthER	Original	SynthER
walker-walk	mixed	28.7±6.9	32.3±7.6	16.5±4.3	12.3±3.6
medium	46.8±2.3	44.0±2.9	40.9±3.1	40.3±3.0
medexp	86.4±5.6	83.4±6.3	47.7±3.9	45.2±4.5
expert	68.4±7.5	83.6±7.5	91.5±3.9	92.0±4.2
cheetah-run	mixed	44.8±3.6	43.8±2.7	25.0±3.6	27.9±3.4
medium	53.0±3.0	56.0±1.2	51.6±1.4	52.2±1.2
medexp	50.6±8.2	56.9±8.1	57.5±6.3	69.9±9.5
expert	34.5±8.3	52.3±7.0	67.4±6.8	85.4±3.1
Average	51.7±5.7	56.5±5.4 (+9.5%)	49.8±4.2	53.2±4.1 (+6.8%)
weaker class of generative models which we explored in Section 3.1. Huang et al. [32] considered using GAN samples to pre-train an RL policy, observing a modest improvement in sample efficiency for CartPole. Chen et al. [13], Yu et al. [79] consider augmenting the image observations of robotic control data using a guided diffusion model whilst maintaining the same action. This differs from our approach which models the entire transition and can synthesize novel action and reward labels.
Outside of reinforcement learning, Azizi et al. [7], He et al. [26], Sariyildiz et al. [60] consider generative training data for image classification and pre-training. They also find that synthetic data improves performance for data-scarce settings which are especially prevalent in reinforcement learning. Sehwag et al. [64] consider generative training data to improve adversarial robustness in image classification. In continual learning, "generative replay" [65] has been considered to compress examples from past tasks to prevent forgetting.
Generative Modeling in RL. Prior work in diffusion modeling for offline RL has largely sought to supplant traditional reinforcement learning with "upside-down RL" [62]. Diffuser [36] models long sequences of transitions or full episodes and can bias the whole trajectory with guidance towards high reward or a particular goal. It then takes the first action and re-plans by receding horizon control. Decision Diffuser [4] similarly operates at the sequence level but instead uses conditional guidance on rewards and goals. Du et al. [17] present a similar trajectory-based algorithm for visual data. In contrast, SYNTHER operates at the transition level and seeks to be readily compatible with existing reinforcement learning algorithms. Pearce et al. [55] consider a diffusion-based approach to behavioral cloning, whereby a state-conditional diffusion model may be used to sample actions that imitate prior data. Azar et al. [6], Li et al. [47] provide theoretical sample complexity bounds for model-based reinforcement learning given access to a generative model.
Model-Based Reinforcement Learning. We note the parallels between our work and model-based reinforcement learning [35, 49, 78]; which tends to generate synthetic samples by rolling out using forward dynamics models. Two key differences of this approach to our method are: SYNTHER synthesizes new experiences without the need to start from a real state and the generated experiences are distributed exactly according to the data, rather than subject to compounding errors due to modeling inaccuracy. Furthermore, SYNTHER is an orthogonal approach which could in fact be combined with forward dynamics models by generating initial states using diffusion, which could lead to increased diversity.
(a) Offline TD3+BC Larger Networks (Full results in Table 3)
Figure 7: RLiable [3] analysis allowing us to aggregate results across environments and show the probability of improvement for SYNTHER across our empirical evaluation.
(b) Online DMC Evaluation (Full results in Figure 6)
(c) Offline V-D4RL Evaluation (Full results in Table 4)
In this paper, we proposed SYNTHER, a powerful and general method for upsampling agent experiences in any reinforcement learning algorithm using experience replay. We integrated SYNTHER with ease on six distinct algorithms across proprioceptive and pixel-based environments, each fine-tuned for its own use case, with no algorithmic modification. Our results show the potential of synthetic training data when combined with modern diffusion models. In offline reinforcement learning, SYNTHER allows training from extremely small datasets, scaling up policy and value networks, and high levels of data compression. In online reinforcement learning, the additional data allows agents to use much higher update-to-data ratios leading to increased sample efficiency.
We have demonstrated that SYNTHER is a scalable approach and believe that extending it to more settings would unlock extremely exciting new capabilities for RL agents. SYNTHER could readily be extended to  $n$ -step formulations of experience replay by simply expanding the input space of the diffusion model. Furthermore, whilst we demonstrated an effective method to generate synthetic data in latent space for pixel-based settings, exciting future work could involve generating transitions in the original image space. In particular, one could consider fine-tuning large pre-trained foundation models [59] and leveraging their generalization capability to synthesize novel views and configurations of a pixel-based environment. Finally, by using guidance for diffusion models [29], the generated synthetic data could be biased towards certain modes, resulting in transferable and composable sampling strategies for RL algorithms.
Cong Lu is funded by the Engineering and Physical Sciences Research Council (EPSRC). Philip Ball is funded through the Willowgrove Studentship. The authors would like to thank the anonymous Reincarnating Reinforcement Learning Workshop at ICLR 2023 and NeurIPS 2023 reviewers for positive and constructive feedback which helped to improve the paper. We would also like to thank Shimon Whiteson, Jakob Foerster, Tim Roektaschel and Ondrej Bajgar for reviewing earlier versions of this work.
[1] Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Lokst-Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuysl, Sarah York, Alexander Zacherl, and Lei Zhang. Human-timescale adaptation in an open-ended task space, 2023. URL https://arxiv.org/abs/2301.07608.
[2] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning, 2020.
[3] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 29304-29320. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/f514cec81cb148559cf475e7426eed5e-Paper.pdf.
[4] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657, 2022.
[5] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-based offline reinforcement learning with diversified q-ensemble. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 7436-7447. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/3d3d286a8d153a4a58156d0e02d8570c-Paper.pdf.
[79] Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Brian Ichter, Karol Hausman, and Fei Xia. Scaling robot learning with semantically imagined experience, 2023. URL https://arxiv.org/abs/2302.11550.
A Data Modeling
In this section, we provide further details for our data modeling. Our diffusion model generates full environment transitions i.e., a concatenation of states, actions, rewards, next states, and terminals where they are present. For the purposes of modeling, we normalize each continuous dimension (non-terminal) to have 0 mean and 1 std. We visualize the marginal distributions over the state, action, and reward dimensions on the standard halfcheetah medium-replay dataset in Figure 8 and observe that the synthetic samples accurately match the high-level statistics of the original dataset.
We note the difficulties of appropriately modeling the terminal variable which is a binary variable compared to the rest of the dimensions which are continuous for the environments we investigate. This is particularly challenging for "expert" datasets where early termination is rare. For example, walker2d-expert only has  $\approx 0.0001\%$  terminals. In practice, we find it sufficient to leave the terminals un-normalized and round them to 0 or 1 by thresholding the continuous diffusion samples in the middle at 0.5. A cleaner treatment of this variable could be achieved by leveraging work on diffusion with categorical variables [31].
Figure 8: Histograms of the empirical marginal distribution of samples from SYNTHER in blue on the halfcheetah medium-replay dataset against the original data in orange. Dashed lines indicate the mean  $\pm$  one standard deviation in the original dataset. SYNTHER faithfully reproduces the high-level statistics of the dataset.
A.1 Data Compression
An immediate advantage of sampling data from a generative model is compression. In Table 5, we compare the memory requirements of SYNTHER and the original data by the number of 32-bit floating point numbers used by each for some sample D4RL [21] datasets. For the original data, this simply scales linearly with the size of the dataset. On other hand, SYNTHER amortizes this in the number of parameters in the denoising network, resulting in a high level of dataset compression, at the cost of sampling speed. This property was also noted in the continual learning literature with generative models summarizing previous tasks [65]. As we discuss in Appendix B.3, sampling is fast with 100K transitions taking around 90 seconds to generate.
Table 5: SYNTHER provides high levels of dataset compression without sacrificing downstream performance in offline reinforcement learning. Statistics shown are for the standard D4RL MuJoCo walker2d datasets which has a transition dimension of 42, and the residual denoiser used for evaluation on these environments in Section 4.1. Figures are given to 1 decimal place.
Dataset	# FP32s in Original Dataset	# Diffusion Parameters	Compression
mixed	12.6M	1.9×
medium	42M	6.5M	6.5×
medium-expert	84M	12.9×
B.1 TVAE and CTGAN
In Section 3.1, we compared SYNTHER to the VAE and GAN baselines, TVAE and CTGAN. As these algorithms have not been used for reinforcement learning data before, we performed a hyperparameter search [42] across the following spaces:
Table 6: Hyperparameter search space for TVAE. We highlight the default choice in bold.
Parameter	Search Space
no. layers	{1,2,3,4}
width	{64,128,256,512}
batch size	{250,500,1000}
embedding dim	{32,64,128,256}
loss factor	{0.02,0.2,2,20}
Table 7: Hyperparameter search space for CTGAN. We highlight the default choice in **bold**.
Parameter	Search Space
no. layers	{1,2,3,4}
width	{64,128,256,512}
batch size	{250,500,1000}
embedding dim	{32,64,128,256}
discriminator steps	{1,2}
These ranges are similar to those listed in Tables 10 and 11 of Kotelnikov et al. [42]. We used 30 trials along with the default.
B.2 Denoising Network
The formulation of diffusion we use in our paper is the Elucidated Diffusion Model (EDM, Karras et al. [38]). We parametrize the denoising network  $D_{\theta}$  as an MLP with skip connections from the previous layer as in Tolstikhin et al. [70]. Thus each layer has the form given in Equation (3).
$$
x _ {L + 1} = \operatorname {l i n e a r} \left(\operatorname {a c t i v a t i o n} \left(x _ {L}\right)\right) + x _ {L} \tag {3}
$$
The hyperparameters are listed in Table 8. The noise level of the diffusion process is encoded by a Random Fourier Feature [57] embedding. The base size of the network uses a width of 1024 and
depth of 6 and thus has  $\approx 6\mathrm{M}$  parameters. We adjust the batch size for training based on dataset size. For online training and offline datasets with fewer than 1 million samples (medium-replay datasets) we use a batch size of 256, and 1024 otherwise.
For the following offline datasets, we observe more performant samples by increasing the width up to 2048: halfcheetah medium-expert, hopper medium, and hopper medium-expert. This raises the network parameters to  $\approx 25\mathrm{M}$ , which remains fewer parameters than the original data as in Table 5. We provide ablations on the depth and type of network used in Table 10.
Table 8: Default Residual MLP Denoiser Hyperparameters.
Parameter	Value(s)
no. layers	6
width	1024
batch size	{ 256 for online and medium-replay, 1024 otherwise }
RFF dimension	16
activation	relu
optimizer	Adam
learning rate	3 × 10-4
learning rate schedule	cosine annealing
model training steps	100K
B.3 Elucidated Diffusion Model
For the diffusion sampling process, we use the stochastic SDE sampler of Karras et al. [38] with the default hyperparameters used for the ImageNet, given in Table 9. We use a higher number of diffusion timesteps at 128 for improved sample fidelity. We use the implementation at https://github.com/lucidrains/denoising-diffusion-pytorch which is released under an Apache license.
Table 9: Default ImageNet-64 EDM Hyperparameters.
Parameter	Value
no. diffusion steps	128
σmin	0.002
σmax	80
Schurn	80
Stmin	0.05
Stmax	50
Snoise	1.003
The diffusion model is fast to train, taking approximately 17 minutes for 100K training steps on a standard V100 GPU. It takes approximately 90 seconds to generate 100K samples with 128 diffusion timesteps.
C SYNTHER Ablations
We consider ablations on the number of generated samples and type of denoiser used for our offline evaluation in Section 4.1.
C.1 Size of Upsampled Dataset
In our main offline evaluation in Section 4.1, we upsample each dataset (which has an original size of between  $100\mathrm{K}$  to  $2\mathrm{M}$ ) to  $5\mathrm{M}$ . We investigate this choice for the walker medium-replay dataset in Figure 9 and choose 10 levels log-uniformly from the range [50K, 5M]. Similarly to He et al. [26], we find that performance gains with synthetic data eventually saturate and that  $5\mathrm{M}$  is a reasonable heuristic for all our offline datasets.
Figure 9: Ablations on the number of samples generated by SYNTHER for the offline walker medium-replay dataset. We choose 10 levels log-uniformly from the range [50K, 5M]. We find that performance eventually saturates at around 5M samples.
C.2 Network Ablations
We ablate the hyperparameters of the denoising network, comparing 3 settings of depth from  $\{2,4,6\}$  and analyze the importance of skip connections. The remaining hyperparameters follow Appendix B.2. We choose the hopper medium-expert dataset as it is a large dataset of 2M. As we can see in Table 10, we see a positive benefit from the increased depth and skip connections which leads to our final choice in Table 8.
Table 10: Ablations on the denoiser network used for SYNTHER on the hopper medium-expert dataset. We observe that greater depth and residual connections are beneficial for downstream offline RL performance. We show the mean and standard deviation of the final performance averaged over 4 seeds.
Network	Depth	Eval. Return
MLP	2	86.8±18.7
4	89.9±17.9
6	100.4± 6.9
Residual MLP	2	78.5±11.3
4	99.3±14.7
6	101.1±10.5
D RL Implementation
For the algorithms in the offline RL evaluation in Section 4.1, we use the 'Clean Offline Reinforcement Learning' (CORL, Tarasov et al. [68]) codebase. We take the final performance they report for the baseline offline evaluation. Their code can be found at https://github.com/tinkoff-ai/CORL and is released under an Apache license.
For the online evaluation, we consider Soft Actor-Critic (SAC, Haarnoja et al. [24]) and use the implementation from the REDQ [12] codebase. This may be found at https://github.com/watchernyu/REDQ and is released under an MIT license. We use the 'dmcgym' wrapper for the DeepMind Control Suite [71]. This may be found at https://github.com/ikostrikov/dmcgym and is released under an MIT license.
D.1 Data Augmentation Hyperparameters
For the data augmentation schemes we visualize in Figure 1a, we define:
1. Additive Noise [45]: adding  $\epsilon \sim \mathcal{N}(0, 0.1)$  to  $s_t$  and  $s_{t+1}$ .
2. Multiplicative Noise [45]: multiplying  $s_t$  and  $s_{t + 1}$  by single number  $\epsilon \sim \mathrm{Unif}([0.8,1.2])$ .
3. Dynamics Noise [8]: multiplying the next state delta  $s_{t + 1} - s_t$  by  $\epsilon \sim \mathrm{Unif}([0.5, 1.5])$  so that  $s_{t + 1} = s_t + \epsilon \cdot (s_{t + 1} - s_t)$ .
D.2 Online Running Times
Our online implementation in Section 4.2 uses the default training hyperparameters in Appendix B.2 to train the diffusion model every 10K online steps, and generates 1M transitions each time. On the 200K DMC experiments, 'SAC (SynthER)' takes  $\approx 21.1$  hours compared to  $\approx 22.7$  hours with REDQ on a V100 GPU. We can further break down the running times of 'SAC (SynthER)' as follows:
- Diffusion training: 4.3 hours
- Diffusion sampling: 5 hours
- RL training: 11.8 hours