SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Cong Lu*, Philip J. Ball*, Yee Whye Teh, Jack Parker-Holder
A key theme in the past decade has been that when large neural networks and large datasets combine they can produce remarkable results. In deep reinforcement learning (RL), this paradigm is commonly made possible through experience replay, whereby a dataset of past experiences is used to train a policy or value function. However, unlike in supervised or self-supervised learning, an RL agent has to collect its own data, which is often limited. Thus, it is challenging to reap the benefits of deep learning, and even small neural networks can overfit at the start of training. In this work, we leverage the tremendous recent progress in generative modeling and propose Synthetic Experience Replay (SYNTHER), a diffusion-based approach to flexibly upsample an agent's collected experience. We show that SYNTHER is an effective method for training RL agents across offline and online settings, in both proprioceptive and pixel-based environments. In offline settings, we observe drastic improvements when upsampling small offline datasets and see that additional synthetic data also allows us to effectively train larger networks. Furthermore, SYNTHER enables online agents to train with a much higher update-to-data ratio than before, leading to a significant increase in sample efficiency, without any algorithmic changes. We believe that synthetic training data could open the door to realizing the full potential of deep learning for replay-based RL algorithms from limited data. Finally, we open-source our code at https://github.com/conglu1997/SynthER.
(a) IQL [41] on a reduced  $15\%$  subset of walker2d medium-replay [21].
(b) SAC [24] on 6 DeepMind Control Suite [71] and OpenAI Gym [10] environments.
Figure 1: Upsampling data using SYNTHER greatly outperforms explicit data augmentation schemes for small offline datasets and data-efficient algorithms in online RL without any algorithmic changes. Moreover, synthetic data from SYNTHER may readily be added to any algorithm utilizing experience replay. Full results in Section 4.
In the past decade, the combination of large datasets [14, 63] and ever deeper neural networks [15, 25, 43, 73] has led to a series of more generally capable models [11, 56, 58]. In reinforcement learning (RL, Sutton and Barto [67]), agents typically learn online from their own experience. Thus,
*Equal contribution. Correspondence to cong.liu@stats.ox.ac.uk and ball@robots.ox.ac.uk.
Figure 2: SYNTHER allows any RL agent using experience replay to arbitrarily upsample, or increase the quantity of, their experiences (in grey) and train on synthetic data (in blue). We evaluate our approach up to a factor of  $100 \times$  more data in Section 4.2, across both proprioceptive and pixel-based environments. By leveraging this increased data, agents can learn effectively from smaller datasets and achieve higher sample efficiency. Details of the upsampling process are given in Figure 3.
to leverage sufficiently rich datasets, RL agents typically make use of experience replay [19, 53], where training takes place on a dataset of recent experiences. However, this experience is typically limited, unless an agent is distributed over many workers which requires both high computational cost and sufficiently fast simulation [18, 37].
Another approach for leveraging broad datasets for training RL policies is offline RL [2, 46], whereby behaviors may be distilled from previously collected data either via behavior cloning [61], off-policy learning [22, 44] or model-based methods [39, 49, 78]. Offline data can also significantly bootstrap online learning [9, 27, 75]; however, it is a challenge to apply these methods when there is a mismatch between offline data and online environment. Thus, many of the successes rely on toy domains with transfer from specific behaviors in a simple low-dimensional proprioceptive environment.
Whilst strong results have been observed in re-using prior data in RL, appropriate data for particular behaviors may simply not exist and thus this approach falls short in generality. We consider an alternative approach—rather than passively reusing data, we leverage tremendous progress in generative modeling to generate a large quantity of new, synthetic data. While prior work has considered upsampling online RL data with VAEs or GANs [32, 34, 51], we propose making use of diffusion generative models [30, 38, 66], which unlocks significant new capabilities.
Our approach, which we call Synthetic Experience Replay, or SYNTHER, is conceptually simple, whereby given a limited initial dataset, we can arbitrarily upsample the data for an agent to use as if it was real experience. Therefore, in this paper, we seek to answer a simple question: Can the latest generative models replace or augment traditional datasets in reinforcement learning? To answer this, we consider the following settings: offline RL where we entirely replace the original data with data produced by a generative model, and online RL where we upsample experiences to broaden the training data available to the agent. In both cases, SYNTHER leads to drastic improvements, obtaining performance comparable to that of agents trained with substantially more real data. Furthermore, in certain offline settings, synthetic data enables effective training of larger policy and value networks, resulting in higher performance by alleviating the representational bottleneck. Finally, we show that SYNTHER scales to pixel-based environments by generating data in latent space. We thus believe this paper presents sufficient evidence that our approach could enable entirely new, efficient, and scalable training strategies for RL agents. To summarize, the contributions of this paper are:
- We propose SYNTHER in Section 3, a diffusion-based approach that allows one to generate synthetic experiences and thus arbitrarily upsample data for any reinforcement learning algorithm utilizing experience replay.
- We validate the synthetic data generated by SYNTHER in offline settings across proprioceptive and pixel-based environments in Section 4.1 and Section 4.3, presenting the first generative approach to show parity with real data on the standard D4RL and V-D4RL offline datasets with a wide variety of algorithms. Furthermore, we observe considerable improvements from upsampling for small offline datasets and scaling up network sizes.
- We show how SYNTHER can arbitrarily upsample an online agent's training data in Section 4.2 by continually training the diffusion model. This allows us to significantly increase an agent's update-to-data (UTD) ratio matching the efficiency of specially designed data-efficient algorithms without any algorithmic changes.
Reverse Denoising Process
Forward Diffusion Process
Figure 3: SYNTHER generates synthetic samples using a diffusion model which we visualize on the proprioceptive walker2d environment. On the top row, we render the state component of the transition tuple on a subset of samples; and on the bottom row, we visualize a t-SNE [72] projection of 100,000 samples. The denoising process creates cohesive and plausible transitions whilst also remaining diverse, as seen by the multiple clusters that form at the end of the process in the bottom row.
We model the environment as a Markov Decision Process (MDP, Sutton and Barto [67]), defined as a tuple  $M = (\mathcal{S}, \mathcal{A}, P, R, \rho_0, \gamma)$ , where  $\mathcal{S}$  and  $\mathcal{A}$  denote the state and action spaces respectively,  $P(s'|s, a)$  the transition dynamics,  $R(s, a)$  the reward function,  $\rho_0$  the initial state distribution, and  $\gamma \in (0, 1)$  the discount factor. The goal in reinforcement learning is to optimize a policy  $\pi(a|s)$  that maximizes the expected discounted return  $\mathbb{E}_{\pi,P,\rho_0}[\sum_{t=0}^{\infty} \gamma^t R(s_t, a_t)]$ .
2.2 Offline Reinforcement Learning
In offline RL [46], the policy is not deployed in the environment until test time. Instead, the algorithm only has access to a static dataset  $\mathcal{D}_{\mathrm{env}} = \{(s_t, a_t, r_t, s_{t+1})\}_{t=1}^T$ , collected by one or more behavioral policies  $\pi_b$ . We refer to the distribution from which  $\mathcal{D}_{\mathrm{env}}$  was sampled as the behavioral distribution [78]. In some of the environments we consider, the environment may be finite horizon or have early termination. In that case, the transition tuple also contains a terminal flag  $d_t$  where  $d_t = 1$  indicates the episode ended early at timestep  $t$  and  $d_t = 0$  otherwise.
Diffusion models [30, 66] are a class of generative models inspired by non-equilibrium thermodynamics that learn to iteratively reverse a forward noising process and generate samples from noise. Given a data distribution  $p(\mathbf{x})$  with standard deviation  $\sigma_{\mathrm{data}}$ , we consider noised distributions  $p(\mathbf{x};\sigma)$  obtained by adding i.i.d. Gaussian noise of standard deviation  $\sigma$  to the base distribution. The forward noising process is defined by a sequence of noised distributions following a fixed noise schedule  $\sigma_0 = \sigma_{\mathrm{max}} > \sigma_1 > \dots >\sigma_N = 0$ . When  $\sigma_{\mathrm{max}}\gg \sigma_{\mathrm{data}}$ , the final noised distribution  $p(\mathbf{x};\sigma_{\mathrm{max}})$  is essentially indistinguishable from random noise.
Karras et al. [38] consider a probability-flow ODE with the corresponding continuous noise schedule  $\sigma(t)$  that maintains the desired distribution as  $\mathbf{x}$  evolves through time given by Equation (1).
$$
\mathrm {d} \mathbf {x} = - \dot {\sigma} (t) \sigma (t) \nabla_ {\mathbf {x}} \log p (\mathbf {x}; \sigma (t)) \mathrm {d} t \tag {1}
$$
where the dot indicates a time derivative and  $\nabla_{\mathbf{x}}\log p(\mathbf{x};\sigma (t))$  is the score function [33], which points towards the data at a given noise level. Infinitesimal forward or backward steps of this ODE either nudge a sample away or towards the data. Karras et al. [38] consider training a denoiser  $D_{\theta}(\mathbf{x};\sigma)$  on an L2 denoising objective:
$$
\min  _ {\theta} \mathbb {E} _ {\mathbf {x} \sim p, \sigma , \epsilon \sim \mathcal {N} (0, \sigma^ {2} I)} \| D _ {\theta} (\mathbf {x} + \epsilon ; \sigma) - \mathbf {x} \| _ {2} ^ {2} \tag {2}
$$
and then use the connection between score-matching and denoising [74] to obtain  $\nabla_{\mathbf{x}}\log p(\mathbf{x};\sigma) = (D_{\theta}(\mathbf{x};\sigma) - \mathbf{x}) / \sigma^{2}$ . We may then apply an ODE (or SDE as a generalization of Equation (1)) solver to reverse the forward process. In this paper, we train our diffusion models to approximate the online or offline behavioral distribution.
Algorithm 1 SYNTHER for online replay-based algorithms. Our additions are highlighted in blue.
1: Input: real data ratio  $r \in [0,1]$   
2: Initialize:  $\mathcal{D}_{\mathrm{real}} = \emptyset$  real replay buffer,  $\pi$  agent,  $\mathcal{D}_{\mathrm{synthetic}} = \emptyset$  synthetic replay buffer,  $M$  diffusion model  
3: for  $t = 1,\dots,T$  do  
4: Collect data with  $\pi$  in the environment and add them to  $\mathcal{D}_{\mathrm{real}}$   
5: Update diffusion model  $M$  with samples from  $\mathcal{D}_{\mathrm{real}}$   
6: Generate samples from  $M$  and add them to  $\mathcal{D}_{\mathrm{synthetic}}$   
7: Train  $\pi$  on samples from  $\mathcal{D}_{\mathrm{real}} \cup \mathcal{D}_{\mathrm{synthetic}}$  mixed with ratio  $r$   
8: end for
3 Synthetic Experience Replay
In this section, we introduce Synthetic Experience Replay (SYNTHER), our approach to upsampling an agent's experience using diffusion. We begin by describing the simpler process used for offline RL and then how that may be adapted to the online setting by continually training the diffusion model.
3.1 Offline SYNTHER
For offline reinforcement learning, we take the data distribution of the diffusion model  $p(\mathbf{x})$  to simply be the offline behavioral distribution. In the proprioceptive environments we consider, the full transition is low-dimensional compared with typical pixel-based diffusion. Therefore, the network architecture is an important design choice; and similarly to Pearce et al. [55] we find it important to use a residual MLP denoising [70] network. Furthermore, the choice of the Karras et al. [38] sampler allows us to use a low number of diffusion steps ( $n = 128$ ) resulting in high sampling speed. Full details for both are provided in Appendix B. We visualize the denoising process on a representative D4RL [21] offline dataset, in Figure 3. We further validate our diffusion model on the D4RL datasets in Figure 8 in Appendix A by showing that the synthetic data closely matches the original data when comparing the marginal distribution over each dimension. In Section 4.3, we show the same model may be used for pixel-based environments by generating data in a low-dimensional latent space.
Next, we conduct a quantitative analysis and show that the quality of the samples from the diffusion model is significantly better than with prior generative models such as VAEs [40] and GANs [23]. We consider the state-of-the-art Tabular VAE (TVAE) and Conditional Tabular GAN (CTGAN) models proposed by Xu et al. [76], and tune them on the D4RL halfcheetah medium-replay dataset. Full hyperparameters are given in Appendix B.1. As proposed in Patki et al. [54], we compare the following two high-level statistics: (1) Marginal: Mean Kolmogorov-Smirnov [52] statistic, measuring the maximum distance between empirical cumulative distribution functions, for each dimension of the synthetic and real data; and (2) Correlation: Mean Correlation Similarity, measuring the difference in pairwise Pearson rank correlations [20] between the synthetic and real data.
We also assess downstream offline RL performance using the synthetic data with two state-of-the-art offline RL algorithms, TD3+BC [22] and IQL [41], in Table 1. The full evaluation protocol is described in Section 4.1. The diffusion model is far more faithful to the original data than prior generative models which leads to substantially higher returns on both algorithms. Thus, we hypothesize a large part of the failure of prior methods [34, 51] is due to a weaker generative model.
3.2 Online SYNTHER
SYNTHER may be used to upsample an online agent's experiences by continually training the diffusion model on new experiences. We provide pseudocode for how to incorporate SYNTHER
Table 1: SYNTHER is better at capturing both the high-level statistics of the dataset (halfcheetah medium-replay) than prior generative models and also leads to far higher downstream performance. Metrics (left) computed from 100K samples from each model, offline RL performance (right) computed using 5M samples from each model. We show the mean and standard deviation of the final performance averaged over 8 seeds.
Model	Metrics	Eval. Return
Marginal	Correlation	TD3+BC	IQL
Diffusion (Ours)	0.989	0.998	45.9±0.9	46.6±0.2
VAE [76]	0.942	0.979	27.1±2.1	15.2±2.2
GAN [76]	0.959	0.981	24.3±1.9	15.9±2.4
Table 2: A comprehensive evaluation of SYNTHER on a wide variety of proprioceptive D4RL [21] datasets and selection of state-of-the-art offline RL algorithms. We show that synthetic data from SYNTHER faithfully reproduces the original performance, which allows us to completely eschew the original training data. We show the mean and standard deviation of the final performance averaged over 8 seeds. Highlighted figures show at least parity over each group (algorithm and environment class) of results.
Environment	Behavioral Policy	TD3+BC [22]	IQL [41]	EDAC [5]
Original	SYNTHER	Original	SYNTHER	Original	SYNTHER
halfcheetah-v2	random	11.3±0.8	12.2±1.1	15.2±1.2	17.2±3.4	-
mixed	44.8±0.7	45.9±0.9	43.5±0.4	46.6±0.2	62.1±1.3	63.0±1.3
medium	48.1±0.2	49.9±1.2	48.3±0.1	49.6±0.3	67.7±1.2	65.1±1.3
medexp	90.8±7.0	87.2±11.1	94.6±0.2	93.3±2.6	104.8±0.7	94.1±10.1
walker2d-v2	random	0.6±0.3	2.3±1.9	4.1±0.8	4.2±0.3	-
mixed	85.6±4.6	90.5±4.3	82.6±8.0	83.3±5.9	87.1±3.2	89.8±1.5
medium	82.7±5.5	84.8±1.4	84.0±5.4	84.7±5.5	93.4±1.6	93.4±2.4
medexp	110.0±0.4	110.2±0.5	111.7±0.6	111.4±0.7	114.8±0.9	114.7±1.2
hopper-v2	random	8.6±0.3	14.6±9.4	7.2±0.2	7.7±0.1	-
mixed	64.4±24.8	53.4±15.5	84.6±13.5	103.2±0.4	99.7±0.9	101.4±0.8
medium	60.4±4.0	63.4±4.2	62.8±6.0	72.0±4.5	101.7±0.3	102.4±0.5
medexp	101.1±10.5	105.4±9.7	106.2±6.1	90.8±17.9	105.2±11.6	109.7±0.2
locomotion average	59.0±4.9	60.0±5.1	62.1±3.5	63.7±3.5	92.9±2.4	92.6±2.1
maze2d-v1	umaze	29.4±14.2	37.6±14.4	37.7±2.0	41.0±0.7	95.3±7.4	99.1±18.6
medium	59.5±41.9	65.2±36.1	35.5±1.0	35.1±2.6	57.0±4.0	66.4±10.9
large	97.1±29.3	92.5±38.5	49.6±22.0	60.8±5.3	95.6±26.5	143.3±21.7
maze average	62.0±28.2	65.1±29.7	40.9±8.3	45.6±2.9	82.6±12.6	102.9±17.1
into any online replay-based RL agent in Algorithm 1 and visualize this in Figure 2. Concretely, a diffusion model is periodically updated on the real transitions and then used to populate a second synthetic buffer. The agent may then be trained on a mixture of real and synthetic data sampled with ratio  $r$ . For the results in Section 4.2, we simply set  $r = 0.5$  following Ball et al. [9]. The synthetic replay buffer may also be configured with a finite capacity to prevent overly stale data.
We evaluate SYNTHER across a wide variety of offline and online settings. First, we validate our approach on offline RL, where we entirely replace the original data, and further show large benefits from upsampling small offline datasets. Next, we show that SYNTHER leads to large improvements in sample efficiency in online RL, exceeding specially designed data-efficient approaches. Furthermore, we show that SYNTHER scales to pixel-based environments by generating data in latent space. Finally, we perform a meta-analysis over our empirical evaluation using the RLiable [3] framework in Figure 7.
4.1 Offline Evaluation
We first verify that synthetic samples from SYNTHER faithfully model the underlying distribution from the canonical offline D4RL [21] datasets. To do this, we evaluate SYNTHER in combination with 3 widely-used SOTA offline RL algorithms: TD3+BC (Fujimoto and Gu [22], explicit policy regularization), IQL (Kostrikov et al. [41], expectile regression), and EDAC (An et al. [5], uncertainty-based regularization) on an extensive selection of D4RL datasets. We consider the MuJoCo [69] locomotion (halfcheetah, walker2d, and hopper) and maze2d environments. In these experiments, all datasets share the same training hyperparameters in Appendix B, with some larger datasets using a wider network. For each dataset, we upsample the original dataset to 5M samples; we justify this choice in Appendix C.1. We show the final performance in Table 2.
Our results show that we achieve at least parity for all groups of environments and algorithms as highlighted in the table, regardless of the precise details of each algorithm. We note significant improvements to maze2d environments, which are close to the 'best' performance as reported in CORL [68] (i.e., the best iteration during offline training) rather than the final performance. We hypothesize this improvement is largely due to increased data from SYNTHER, which leads to less overfitting and increased stability. For the locomotion datasets, we largely reproduce the original results, which we attribute to the fact that most D4RL datasets are at least 1M in size and are already sufficiently large. However, as detailed in Table 5 in Appendix A.1, SYNTHER allows the effective size of the dataset to be compressed significantly, up to  $12.9 \times$  on some datasets. Finally, we present results on the AntMaze environment in Appendix E.1, and experiments showing that the synthetic and real data are compatible with each other in Appendix E.2.
Figure 4: SYNTHER is a powerful method for upsampling reduced variants of the walker2d datasets and vastly improves on competitive explicit data augmentation approaches for both the TD3+BC (top) and IQL (bottom) algorithms. The subsampling levels are scaled proportionally to the original size of each dataset. We show the mean and standard deviation of the final performance averaged over 8 seeds.
4.1.1 Upsampling for Small Datasets
We investigate the benefit of SYNTHER for small offline datasets and compare it to canonical 'explicit' data augmentation approaches [8, 45]. Concretely, we wish to understand whether SYNTHER generalizes and generates synthetic samples that improve policy learning compared with explicitly augmenting the data with hand-designed inductive biases. We focus on the walker2d (medium, medium-replay/mixed, medium-expert) datasets in D4RL and uniformly subsample each at the transition level. We subsample each dataset proportional to the original dataset size so that the subsampled datasets approximately range from 20K to 200K samples. As in Section 4.1, we then use SYNTHER to upsample each dataset to 5M transitions. Our denoising network uses the same hyperparameters as for the original evaluation in Section 4.1.
In Figure 4, we can see that for all datasets, SYNTHER leads to a significant gain in performance and vastly improves on explicit data augmentation approaches. For explicit data augmentation, we select the overall most effective augmentation scheme from Laskin et al. [45] (adding Gaussian noise of the form  $\epsilon \sim \mathcal{N}(0,0.1)$ ). Notably, with SYNTHER we can achieve close to the original levels of performance on the walker2d-medium-expert datasets starting from only  $3\%$  of the original data. In Figure 1a, we methodically compare across both additive and multiplicative versions of RAD, as well as dynamics augmentation [8] on the  $15\%$  reduced walker medium-replay dataset.
Why is SYNTHER better than explicit augmentation? To provide intuition into the efficacy of SYNTHER over canonical explicit augmentation approaches, we compare the data generated by SYNTHER to that generated by the best-performing data augmentation approach in Figure 1a, namely additive noise. We wish to evaluate two properties: 1) How diverse is the data? 2) How accurate is the data for the purposes of learning policies? To measure diversity, we measure the minimum L2 distance of each datapoint from the dataset, which allows us to see how far the upsampled data is from the original data. To measure the validity of the data, we follow Lu et al. [49] and measure the MSE between the reward and next state proposed by SYNTHER with the true next state and reward defined by the simulator. We plot both these values in a joint scatter plot to compare how they vary with respect to each other. For this, we compare specifically on the reduced  $15\%$
subset of walker2d medium-replay as in Figure 1a. As we see in Figure 5, SYNTHER generates a significantly wider marginal distribution over the distance from the dataset, and generally produces samples that are further away from the dataset than explicit augmentations. Remarkably, however, we see that these samples are far more consistent with the true environment dynamics. Thus, SYNTHER
Figure 5: Comparing L2 distance from training data and dynamics accuracy under SYNTHER and augmentations.
Table 3: SYNTHER enables effective training of larger policy and value networks for TD3+BC [22] leading to a  $11.7\%$  gain on the offline MuJoCo locomotion datasets. In comparison, simply increasing the network size with the original data does not improve performance. We show the mean and standard deviation of the final performance averaged over 8 seeds.
Environment	Behavioral Policy	Baseline	Larger Network
Original Data	SYNTHER
halfcheetah-v2	random	11.3±0.8	11.0±0.7	12.8±0.8
mixed	44.8±0.7	44.8±1.1	48.0±0.5
medium	48.1±0.2	50.2±1.8	53.3±0.4
medexp	90.8±7.0	95.5±5.4	100.1±2.7
walker2d-v2	random	0.6±0.3	2.6±2.1	4.3±1.7
mixed	85.6±4.6	76.4±9.9	93.6±3.6
medium	82.7±5.5	84.5±1.7	87.2±1.2
medexp	110.0±0.4	110.3±0.5	110.2±0.3
hopper-v2	random	8.6±0.3	10.3±5.6	19.5±11.2
mixed	64.4±24.8	62.4±21.6	86.8±12.8
medium	60.4±4.0	61.9±5.9	65.1±4.7
medexp	101.1±10.5	104.6±9.4	109.7±4.1
locomotion average	59.0±4.9	59.5±5.5	65.9±3.7