Title: Synthetic Experience Replay
Venue: NeurIPS 2023
Accepted: yes
Keywords: Reinforcement Learning

Diffusion Models

Synthetic Data

Sample-Efficient RL
Abstract: A key theme in the past decade has been that when large neural networks and large datasets combine they can produce remarkable results. In deep reinforcement learning (RL), this paradigm is commonly made possible through experience replay, whereby a dataset of past experiences is used to train a policy or value function. However, unlike in supervised or self-supervised learning, an RL agent has to collect its own data, which is often limited. Thus, it is challenging to reap the benefits of deep learning, and even small neural networks can overfit at the start of training. In this work, we leverage the tremendous recent progress in generative modeling and propose Synthetic Experience Replay (SynthER), a diffusion-based approach to flexibly upsample an agent's collected experience. We show that SynthER is an effective method for training RL agents across offline and online settings, in both proprioceptive and pixel-based environments. In offline settings, we observe drastic improvements when upsampling small offline datasets and see that additional synthetic data also allows us to effectively train larger networks. Furthermore, SynthER enables online agents to train with a much higher update-to-data ratio than before, leading to a significant increase in sample efficiency, without any algorithmic changes. We believe that synthetic training data could open the door to realizing the full potential of deep learning for replay-based RL algorithms from limited data. Finally, we open-source our code at https://github.com/conglu1997/SynthER.

Synthetic Experience Replay

Abstract

A key theme in the past decade has been that when large neural networks and large datasets combine they can produce remarkable results. In deep reinforcement learning (RL), this paradigm is commonly made possible through experience replay , whereby a dataset of past experiences is used to train a policy or value function. However, unlike in supervised or self-supervised learning, an RL agent has to collect its own data, which is often limited. Thus, it is challenging to reap the benefits of deep learning, and even small neural networks can overfit at the start of training. In this work, we leverage the tremendous recent progress in generative modeling and propose Synthetic Experience Replay (SYNTHER), a diffusion-based approach to flexibly upsample an agent's collected experience. We show that SYNTHER is an effective method for training RL agents across offline and online settings, in both proprioceptive and pixel-based environments. In offline settings, we observe drastic improvements when upsampling small offline datasets and see that additional synthetic data also allows us to effectively train larger networks. Furthermore, SYNTHER enables online agents to train with a much higher update-to-data ratio than before, leading to a significant increase in sample efficiency, without any algorithmic changes . We believe that synthetic training data could open the door to realizing the full potential of deep learning for replay-based RL algorithms from limited data. Finally, we open-source our code at https://github.com/conglu1997/SynthER .

0

20

40

60

80

None

SynthER

(a) IQL [41] on a reduced 15% subset of

walker2d medium-replay [21].

0

20

40

60

80

100

Normalized Return

SAC

REDQ

SAC (SynthER)

Data-Efficient Online RL

(b) SAC [24] on 6 DeepMind Control Suite [71] and OpenAI Gym [10] environments.

1 Introduction

In the past decade, the combination of large datasets [14, 63] and ever deeper neural networks [15, 25, 43, 73] has led to a series of more generally capable models [11, 56, 58]. In reinforcement learning (RL, Sutton and Barto [67]), agents typically learn online from their own experience. Thus,

Figure 2: SYNTHER allows any RL agent using experience replay to arbitrarily upsample, or increase the quantity of, their experiences (in grey ) and train on synthetic data (in blue ). We evaluate our ap- proach up to a factor of 100 × more data in Section 4.2, across both pro- prioceptive and pixel-based envi- ronments. By leveraging this in- creased data, agents can learn ef- fectively from smaller datasets and achieve higher sample efficiency. Details of the upsampling process are given in Figure 3.

SynthER

UPSAMPLE

Train agent on

Rolloutdata

upsampled data

(s, a, r, s')

Agentinteracts with

theenvironment

to leverage sufficiently rich datasets, RL agents typically make use of experience replay [19, 53], where training takes place on a dataset of recent experiences. However, this experience is typically limited, unless an agent is distributed over many workers which requires both high computational cost and sufficiently fast simulation [18, 37].

Another approach for leveraging broad datasets for training RL policies is offline RL [2, 46], whereby behaviors may be distilled from previously collected data either via behavior cloning [61], off-policy learning [22, 44] or model-based methods [39, 49, 78]. Offline data can also significantly bootstrap online learning [9, 27, 75]; however, it is a challenge to apply these methods when there is a mismatch between offline data and online environment. Thus, many of the successes rely on toy domains with transfer from specific behaviors in a simple low-dimensional proprioceptive environment.

Whilst strong results have been observed in re-using prior data in RL, appropriate data for particular behaviors may simply not exist and thus this approach falls short in generality. We consider an alternative approach-rather than passively reusing data, we leverage tremendous progress in generative modeling to generate a large quantity of new, synthetic data. While prior work has considered upsampling online RL data with VAEs or GANs [32, 34, 51], we propose making use of diffusion generative models [30, 38, 66], which unlocks significant new capabilities.

Our approach, which we call Synthetic Experience Replay , or SYNTHER, is conceptually simple, whereby given a limited initial dataset, we can arbitrarily upsample the data for an agent to use as if it was real experience. Therefore, in this paper, we seek to answer a simple question: Can the latest generative models replace or augment traditional datasets in reinforcement learning? To answer this, we consider the following settings: offline RL where we entirely replace the original data with data produced by a generative model, and online RL where we upsample experiences to broaden the training data available to the agent. In both cases, SYNTHER leads to drastic improvements, obtaining performance comparable to that of agents trained with substantially more real data. Furthermore, in certain offline settings, synthetic data enables effective training of larger policy and value networks, resulting in higher performance by alleviating the representational bottleneck. Finally, we show that SYNTHER scales to pixel-based environments by generating data in latent space . We thus believe this paper presents sufficient evidence that our approach could enable entirely new, efficient, and scalable training strategies for RL agents. To summarize, the contributions of this paper are:

We propose SYNTHER in Section 3, a diffusion-based approach that allows one to generate synthetic experiences and thus arbitrarily upsample data for any reinforcement learning algorithm utilizing experience replay.

We validate the synthetic data generated by SYNTHER in offline settings across proprioceptive and pixel-based environments in Section 4.1 and Section 4.3, presenting the first generative approach to show parity with real data on the standard D4RL and V-D4RL offline datasets with a wide variety of algorithms. Furthermore, we observe considerable improvements from upsampling for small offline datasets and scaling up network sizes.

We show how SYNTHER can arbitrarily upsample an online agent's training data in Section 4.2 by continually training the diffusion model. This allows us to significantly increase an agent's update-to-data (UTD) ratio matching the efficiency of specially designed dataefficient algorithms without any algorithmic changes .

Reverse Denoising Process

Noise

Data

5

<

F

Forward Diffusion Process

2 Background

2.1 Reinforcement Learning

We model the environment as a Markov Decision Process (MDP, Sutton and Barto [67]), defined as a tuple M = ( S , A , P, R, ρ 0 , γ ) , where S and A denote the state and action spaces respectively, P ( s ′ | s, a ) the transition dynamics, R ( s, a ) the reward function, ρ 0 the initial state distribution, and γ ∈ (0 , 1) the discount factor. The goal in reinforcement learning is to optimize a policy π ( a | s ) that maximizes the expected discounted return E π,P,ρ 0 [ ∑ ∞ t =0 γ t R ( s t , a t )] .

2.2 Offline Reinforcement Learning

In offline RL [46], the policy is not deployed in the environment until test time. Instead, the algorithm only has access to a static dataset D env = { ( s t , a t , r t , s t +1 ) } T t =1 , collected by one or more behavioral policies π b . We refer to the distribution from which D env was sampled as the behavioral distribution [78]. In some of the environments we consider, the environment may be finite horizon or have early termination. In that case, the transition tuple also contains a terminal flag d t where d t = 1 indicates the episode ended early at timestep t and d t = 0 otherwise.

2.3 Diffusion Models

Diffusion models [30, 66] are a class of generative models inspired by non-equilibrium thermodynamics that learn to iteratively reverse a forward noising process and generate samples from noise. Given a data distribution p ( x ) with standard deviation σ data, we consider noised distributions p ( x ; σ ) obtained by adding i.i.d. Gaussian noise of standard deviation σ to the base distribution. The forward noising process is defined by a sequence of noised distributions following a fixed noise schedule σ 0 = σ max > σ 1 > · · · > σ N = 0 . When σ max ≫ σ data, the final noised distribution p ( x ; σ max ) is essentially indistinguishable from random noise.

Karras et al. [38] consider a probability-flow ODE with the corresponding continuous noise schedule σ ( t ) that maintains the desired distribution as x evolves through time given by Equation (1).

where the dot indicates a time derivative and ∇ x log p ( x ; σ ( t )) is the score function [33], which points towards the data at a given noise level. Infinitesimal forward or backward steps of this ODE either nudge a sample away or towards the data. Karras et al. [38] consider training a denoiser D θ ( x ; σ ) on an L2 denoising objective:

and then use the connection between score-matching and denoising [74] to obtain ∇ x log p ( x ; σ ) = ( D θ ( x ; σ ) -x ) /σ 2 . We may then apply an ODE (or SDE as a generalization of Equation (1)) solver to reverse the forward process. In this paper, we train our diffusion models to approximate the online or offline behavioral distribution.

Algorithm 1 SYNTHER for online replay-based algorithms. Our additions are highlighted in blue .

1: Input: real data ratio r ∈ [0 , 1]

2: Initialize: D real = ∅ real replay buffer, π agent, D synthetic = ∅ synthetic replay buffer, M diffusion model

3: for t = 1 , . . . , T do

4: Collect data with π in the environment and add them to D real

5: Update diffusion model M with samples from D real

6: Generate samples from M and add them to D synthetic

7: Train π on samples from D real ∪ D synthetic mixed with ratio r

8: end for

3 Synthetic Experience Replay

In this section, we introduce Synthetic Experience Replay (SYNTHER), our approach to upsampling an agent's experience using diffusion. We begin by describing the simpler process used for offline RL and then how that may be adapted to the online setting by continually training the diffusion model.

3.1 Offline SYNTHER

For offline reinforcement learning, we take the data distribution of the diffusion model p ( x ) to simply be the offline behavioral distribution. In the proprioceptive environments we consider, the full transition is low-dimensional compared with typical pixel-based diffusion. Therefore, the network architecture is an important design choice; and similarly to Pearce et al. [55] we find it important to use a residual MLP denoising [70] network. Furthermore, the choice of the Karras et al. [38] sampler allows us to use a low number of diffusion steps ( n = 128 ) resulting in high sampling speed. Full details for both are provided in Appendix B. We visualize the denoising process on a representative D4RL [21] offline dataset, in Figure 3. We further validate our diffusion model on the D4RL datasets in Figure 8 in Appendix A by showing that the synthetic data closely matches the original data when comparing the marginal distribution over each dimension. In Section 4.3, we show the same model may be used for pixel-based environments by generating data in a low-dimensional latent space.

Next, we conduct a quantitative analysis and show that the quality of the samples from the diffusion model is significantly better than with prior generative models such as VAEs [40] and GANs [23]. We consider the state-of-the-art Tabular VAE (TVAE) and Conditional Tabular GAN (CTGAN) models proposed by Xu et al. [76], and tune them on the D4RL halfcheetah medium-replay dataset. Full hyperparameters are given in Appendix B.1. As proposed in Patki et al. [54], we compare the following two high-level statistics: (1) Marginal: Mean Kolmogorov-Smirnov [52] statistic, measuring the maximum distance between empirical cumulative distribution functions, for each dimension of the synthetic and real data; and (2) Correlation: Mean Correlation Similarity, measuring the difference in pairwise Pearson rank correlations [20] between the synthetic and real data.

We also assess downstream offline RL performance using the synthetic data with two state-of-theart offline RL algorithms, TD3+BC [22] and IQL [41], in Table 1. The full evaluation protocol is described in Section 4.1. The diffusion model is far more faithful to the original data than prior generative models which leads to substantially higher returns on both algorithms. Thus, we hypothesize a large part of the failure of prior methods [34, 51] is due to a weaker generative model.

3.2 Online SYNTHER

SYNTHER may be used to upsample an online agent's experiences by continually training the diffusion model on new experiences. We provide pseudocode for how to incorporate SYNTHER

into any online replay-based RL agent in Algorithm 1 and visualize this in Figure 2. Concretely, a diffusion model is periodically updated on the real transitions and then used to populate a second synthetic buffer. The agent may then be trained on a mixture of real and synthetic data sampled with ratio r . For the results in Section 4.2, we simply set r = 0 . 5 following Ball et al. [9]. The synthetic replay buffer may also be configured with a finite capacity to prevent overly stale data.

4 Empirical Evaluation

We evaluate SYNTHER across a wide variety of offline and online settings. First, we validate our approach on offline RL, where we entirely replace the original data, and further show large benefits from upsampling small offline datasets. Next, we show that SYNTHER leads to large improvements in sample efficiency in online RL, exceeding specially designed data-efficient approaches. Furthermore, we show that SYNTHER scales to pixel-based environments by generating data in latent space. Finally, we perform a meta-analysis over our empirical evaluation using the RLiable [3] framework in Figure 7.

4.1 Offline Evaluation

We first verify that synthetic samples from SYNTHER faithfully model the underlying distribution from the canonical offline D4RL [21] datasets. To do this, we evaluate SYNTHER in combination with 3 widely-used SOTA offline RL algorithms: TD3+BC (Fujimoto and Gu [22], explicit policy regularization), IQL (Kostrikov et al. [41], expectile regression), and EDAC (An et al. [5], uncertaintybased regularization) on an extensive selection of D4RL datasets. We consider the MuJoCo [69] locomotion (halfcheetah, walker2d, and hopper) and maze2d environments. In these experiments, all datasets share the same training hyperparameters in Appendix B, with some larger datasets using a wider network. For each dataset, we upsample the original dataset to 5M samples ; we justify this choice in Appendix C.1. We show the final performance in Table 2.

Our results show that we achieve at least parity for all groups of environments and algorithms as highlighted in the table, regardless of the precise details of each algorithm . We note significant improvements to maze2d environments, which are close to the 'best' performance as reported in CORL [68] (i.e., the best iteration during offline training) rather than the final performance. We hypothesize this improvement is largely due to increased data from SYNTHER, which leads to less overfitting and increased stability. For the locomotion datasets, we largely reproduce the original results, which we attribute to the fact that most D4RL datasets are at least 1M in size and are already sufficiently large. However, as detailed in Table 5 in Appendix A.1, SYNTHER allows the effective size of the dataset to be compressed significantly, up to 12 . 9 × on some datasets. Finally, we present results on the AntMaze environment in Appendix E.1, and experiments showing that the synthetic and real data are compatible with each other in Appendix E.2.

0.1

0.2

0.3

0.4

0.5

0

20

40

60

80

medium-replay

0.05

0.10

0.15

0.20

0

20

40

60

80

100

medium

0.02

0.04

0.06

0.08

0.10

0

50

100

medium-expert

0.1

0.2

0.3

0.4

0.5

Subsample Level

0

20

40

60

80

100

IQL Return

0.05

0.10

0.15

0.20

Subsample Level

20

40

60

80

0.02

0.04

0.06

0.08

0.10

Subsample Level

0

25

50

75

100

SynthER

RAD

Original

4.1.1 Upsampling for Small Datasets

We investigate the benefit of SYNTHER for small offline datasets and compare it to canonical 'explicit' data augmentation approaches [8, 45]. Concretely, we wish to understand whether SYNTHER generalizes and generates synthetic samples that improve policy learning compared with explicitly augmenting the data with hand-designed inductive biases. We focus on the walker2d (medium, medium-replay/mixed, medium-expert) datasets in D4RL and uniformly subsample each at the transition level. We subsample each dataset proportional to the original dataset size so that the subsampled datasets approximately range from 20K to 200K samples. As in Section 4.1, we then use SYNTHER to upsample each dataset to 5M transitions. Our denoising network uses the same hyperparameters as for the original evaluation in Section 4.1.

In Figure 4, we can see that for all datasets, SYNTHER leads to a significant gain in performance and vastly improves on explicit data augmentation approaches. For explicit data augmentation, we select the overall most effective augmentation scheme from Laskin et al. [45] (adding Gaussian noise of the form ϵ ∼ N (0 , 0 . 1) ). Notably, with SYNTHER we can achieve close to the original levels of performance on the walker2d-medium-expert datasets starting from only 3% of the original data . In Figure 1a, we methodically compare across both additive and multiplicative versions of RAD, as well as dynamics augmentation [8] on the 15% reduced walker medium-replay dataset.

Why is SYNTHER better than explicit augmentation? To provide intuition into the efficacy of SYNTHER over canonical explicit augmentation approaches, we compare the data generated by SYNTHER to that generated by the best-performing data augmentation approach in Figure 1a, namely additive noise. We wish to evaluate two properties: 1) How diverse is the data? 2) How accurate is the data for the purposes of learning policies? To measure diversity, we measure the minimum L2 distance of each datapoint from the dataset, which allows us to see how far the upsampled data is from the original data. To measure the validity of the data, we follow Lu et al. [49] and measure the MSE between the reward and next state proposed by SYNTHER with the true next state and reward defined by the simulator. We plot both these values in a joint scatter plot to compare how they vary with respect to each other. For this, we compare specifically on the reduced 15%

0.0

2.5

5.0

7.5

L2 Distance from Dataset

0

10

20

30

40

Dynamics MSE

SynthER

Augmented Data

subset of walker2d medium-replay as in Figure 1a. As we see in Figure 5, SYNTHER generates a significantly wider marginal distribution over the distance from the dataset, and generally produces samples that are further away from the dataset than explicit augmentations. Remarkably, however, we see that these samples are far more consistent with the true environment dynamics. Thus, SYNTHER

generates samples that have significantly lower dynamics MSE than explicit augmentations, even for datapoints that are far away from the training data. This implies that a high level of generalization has been achieved by the SYNTHER model, resulting in the ability to generate novel, diverse, yet dynamically accurate data that can be used by policies to improve performance.

4.1.2 Scaling Network Size

A further benefit we observe from SYNTHER on the TD3+BC algorithm is that upsampled data can enable scaling of the policy and value networks leading to improved performance. As is typical for RL algorithms, TD3+BC uses a small value and policy network with two hidden layers, and width of 256, and a batch size of 256. We consider increasing the size of both networks to be three hidden layers and width 512 (approximately 6 × more parameters), and the batch size to 1024 to better make use of the upsampled data in Table 3.

We observe a large overall improvement of 11.7% for the locomotion datasets when using a larger network with synthetic data (Larger Network + SYNTHER). Notably, when using the original data (Larger Network + Original Data), the larger network performs the same as the baseline. This suggests that the bottleneck in the algorithm lies in the representation capability of the neural network and synthetic samples from SYNTHER enables effective training of the larger network . This could alleviate the data requirements for scaling laws in reinforcement learning [1, 28]. However, for the IQL and EDAC algorithms, we did not observe an improvement by increasing the network size which suggests that the bottleneck there lies in the data or algorithm rather than the architecture.

4.2 Online Evaluation

Next, we show that SYNTHER can effectively upsample an online agent's continually collected experiences. In this section, we follow the sample-efficient RL literature [12, 16] and consider 3 environments from the DeepMind Control Suite (DMC, Tunyasuvunakool et al. [71]) (cheetah-run, quadruped-walk, and reacher-hard) and 3 environments the OpenAI Gym Suite [10] (walker2d, halfcheetah, and hopper). As in Chen et al. [12], D'Oro et al. [16], we choose the base algorithm to be Soft Actor-Critic (SAC, Haarnoja et al. [24]), a popular off-policy entropy-regularized algorithm, and benchmark against a SOTA sample-efficient variant of itself, 'Randomized Ensembled Double QLearning' (REDQ, Chen et al. [12]). REDQ uses an ensemble of 10 Q-functions and computes target values across a randomized subset of them during training. By default, SAC uses an update-to-data ratio of 1 (1 update for each transition collected); the modifications to SAC in REDQ enable this to be raised to 20. Our method, 'SAC (SYNTHER)', augments the training data by generating 1M new samples for every 10K real samples collected and samples them with a ratio r = 0 . 5 . We then match REDQ and train with a UTD ratio of 20. We evaluate our algorithms over 200K online steps for the DMCenvironments and 100K for OpenAI Gym.

0

50

100

150

200

0

200

400

600

800

cheetah-run-v0

0

50

100

150

200

0

200

400

600

800

1000

quadruped-walk-v0

0

50

100

150

200

0

200

400

600

800

1000

reacher-hard-v0

0

20

40

60

80

100

Timesteps (K)

0

1000

2000

3000

4000

5000

walker2d-v2

0

20

40

60

80

100

Timesteps (K)

0

2000

4000

6000

8000

10000

halfcheetah-v2

0

20

40

60

80

100

Timesteps (K)

0

1000

2000

3000

4000

hopper-v2

SAC (SynthER)

REDQ

SAC

In Figure 6, we see that SAC (SYNTHER) matches or outperforms REDQ on the majority of the environments with particularly strong results on the quadruped-walk and halfcheetah-v2 environments. This is particularly notable as D'Oro et al. [16] found that UTD=20 on average decreased performance for SAC compared with the default value of 1, attributable to issues with overestimation and overfitting [12, 48]. We aggregate the final performance on the environments in Figure 1b, normalizing the DMC returns following Lu et al. [50] and OpenAI returns as in D4RL. Moreover, due to the fast speed of training our diffusion models and fewer Q-networks, our approach is in fact faster than REDQ based on wall-clock time, whilst also requiring fewer algorithmic design choices, such as large ensembles and random subsetting. Full details on run-time are given in Appendix D.2.

4.3 Scaling to Pixel-Based Observations

Finally, we show that we can readily scale SYNTHER to pixel-based environments by generating data in the latent space of a CNN encoder. We consider the V-D4RL [50] benchmarking suite, a set of standardized pixel-based offline datasets, and focus on the 'cheetah-run' and 'walker-walk' environments. We use the associated DrQ+BC [50] and BC algorithms. Whilst the original image observations are of size 84 × 84 × 3 , we note that the CNN encoder in both algorithms generates features that are 50 dimensional [77]. Therefore, given a frozen encoder pre-trained on the same dataset, we can retain the fast training and sampling speed of our proprioceptive models but now in pixel space. We present full details in Appendix F.

Analogously to the proprioceptive offline evaluation in Section 4.1, we upsample 5M latent transitions for each dataset and present downstream performance in Table 4. Since the V-D4RL datasets are smaller than the D4RL equivalents with a base size of 100K, we would expect synthetic data to be beneficial. Indeed, we observe a statistically significant increase in performance of +9.5% and +6.8% on DrQ+BC and BC respectively; with particularly strong highlighted results on the medium and expert datasets. We believe this serves as compelling evidence of the scalability of SYNTHER to high-dimensional observation spaces and leave generating data in the original image space, or extending this approach to the online setting for future work.

5 Related Work

Whilst generative training data has been explored in reinforcement learning; in general, synthetic data has not previously performed as well as real data on standard RL benchmarks.

Generative Training Data. Imre [34], Ludjen [51] considered using VAEs and GANs to generate synthetic data for online reinforcement learning. However, we note that both works failed to match the original performance on simple environments such as CartPole-this is likely due to the use of a

weaker class of generative models which we explored in Section 3.1. Huang et al. [32] considered using GAN samples to pre-train an RL policy, observing a modest improvement in sample efficiency for CartPole. Chen et al. [13], Yu et al. [79] consider augmenting the image observations of robotic control data using a guided diffusion model whilst maintaining the same action. This differs from our approach which models the entire transition and can synthesize novel action and reward labels .

Outside of reinforcement learning, Azizi et al. [7], He et al. [26], Sariyildiz et al. [60] consider generative training data for image classification and pre-training. They also find that synthetic data improves performance for data-scarce settings which are especially prevalent in reinforcement learning. Sehwag et al. [64] consider generative training data to improve adversarial robustness in image classification. In continual learning, 'generative replay' [65] has been considered to compress examples from past tasks to prevent forgetting.

Generative Modeling in RL. Prior work in diffusion modeling for offline RL has largely sought to supplant traditional reinforcement learning with 'upside-down RL' [62]. Diffuser [36] models long sequences of transitions or full episodes and can bias the whole trajectory with guidance towards high reward or a particular goal. It then takes the first action and re-plans by receding horizon control. Decision Diffuser [4] similarly operates at the sequence level but instead uses conditional guidance on rewards and goals. Du et al. [17] present a similar trajectory-based algorithm for visual data. In contrast, SYNTHER operates at the transition level and seeks to be readily compatible with existing reinforcement learning algorithms. Pearce et al. [55] consider a diffusion-based approach to behavioral cloning, whereby a state-conditional diffusion model may be used to sample actions that imitate prior data. Azar et al. [6], Li et al. [47] provide theoretical sample complexity bounds for model-based reinforcement learning given access to a generative model.

Model-Based Reinforcement Learning. We note the parallels between our work and model-based reinforcement learning [35, 49, 78]; which tends to generate synthetic samples by rolling out using forward dynamics models. Two key differences of this approach to our method are: SYNTHER synthesizes new experiences without the need to start from a real state and the generated experiences are distributed exactly according to the data, rather than subject to compounding errors due to modeling inaccuracy. Furthermore, SYNTHER is an orthogonal approach which could in fact be combined with forward dynamics models by generating initial states using diffusion, which could lead to increased diversity.

0.6

0.8

P(X > Y)

Baseline

Baseline

0.6

0.7

0.8

0.9

P(X > Y)

SAC (SynthER)

SAC (SynthER)

REDQ

SAC

0.60

0.65

0.70

0.75

P(X > Y)

BC (SynthER)

DrQ+BC (SynthER)

BC

DrQ+BC

(a) Offline TD3+BC Larger Networks (Full results in Table 3)

(b) Online DMC Evaluation (Full results in Figure 6)

(c) Offline V-D4RL Evaluation (Full results in Table 4)

6 Conclusion

In this paper, we proposed SYNTHER, a powerful and general method for upsampling agent experiences in any reinforcement learning algorithm using experience replay. We integrated SYNTHER with ease on six distinct algorithms across proprioceptive and pixel-based environments , each fine-tuned for its own use case, with no algorithmic modification . Our results show the potential of synthetic training data when combined with modern diffusion models. In offline reinforcement learning, SYNTHER allows training from extremely small datasets, scaling up policy and value networks, and high levels of data compression. In online reinforcement learning, the additional data allows agents to use much higher update-to-data ratios leading to increased sample efficiency.

We have demonstrated that SYNTHER is a scalable approach and believe that extending it to more settings would unlock extremely exciting new capabilities for RL agents. SYNTHER could readily be extended to n -step formulations of experience replay by simply expanding the input space of the diffusion model. Furthermore, whilst we demonstrated an effective method to generate synthetic data in latent space for pixel-based settings, exciting future work could involve generating transitions in the original image space. In particular, one could consider fine-tuning large pre-trained foundation models [59] and leveraging their generalization capability to synthesize novel views and configurations of a pixel-based environment. Finally, by using guidance for diffusion models [29], the generated synthetic data could be biased towards certain modes, resulting in transferable and composable sampling strategies for RL algorithms.

Acknowledgments

Cong Lu is funded by the Engineering and Physical Sciences Research Council (EPSRC). Philip Ball is funded through the Willowgrove Studentship. The authors would like to thank the anonymous Reincarnating Reinforcement Learning Workshop at ICLR 2023 and NeurIPS 2023 reviewers for positive and constructive feedback which helped to improve the paper. We would also like to thank Shimon Whiteson, Jakob Foerster, Tim Rocktäschel and Ondrej Bajgar for reviewing earlier versions of this work.

References

Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani, Avishkar Bhoopchand, Nathalie Bradley-Schmieg, Michael Chang, Natalie Clay, Adrian Collister, Vibhavari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria LoksThompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei Zhang. Human-timescale adaptation in an open-ended task space, 2023. URL https://arxiv.org/abs/2301.07608 .

Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International Conference on Machine Learning , 2020.

Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning at the edge of the statistical precipice. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 29304-29320. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper_files/paper/2021/file/ f514cec81cb148559cf475e7426eed5e-Paper.pdf .

Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657 , 2022.

Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertaintybased offline reinforcement learning with diversified q-ensemble. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , volume 34, pages 7436-7447. Curran Associates, Inc., 2021. URL https://proceedings.neurips.cc/paper/2021/file/ 3d3d286a8d153a4a58156d0e02d8570c-Paper.pdf .

Mohammad Gheshlaghi Azar, Rémi Munos, and Bert Kappen. On the sample complexity of reinforcement learning with a generative model. arXiv preprint arXiv:1206.6461 , 2012.

Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification, 2023.

Philip J Ball, Cong Lu, Jack Parker-Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning , volume 139 of Proceedings of Machine Learning Research , pages 619-629. PMLR, 18-24 Jul 2021. URL https://proceedings.mlr.press/v139/ball21a.html .

Philip J. Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data, 2023. URL https://arxiv.org/abs/2302.02948 .

Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Xinyue Chen, Che Wang, Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without a model. In International Conference on Learning Representations , 2021. URL https://openreview.net/forum?id=AY8zfZm0tDd .

Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation, 2023. URL https://arxiv.org/abs/2302. 06671 .

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Marc G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=OpC-9aBBVJe .

Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation, 2023. URL https://arxiv.org/abs/2302.00111 .

Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures. In Proceedings of the 35th International Conference on Machine Learning , 2018.

William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In Proceedings of the 37th International Conference on Machine Learning , 2020.

E. C. Fieller, H. O. Hartley, and E. S. Pearson. Tests for rank correlation coefficients. i. Biometrika , 44(3/4):470-481, 1957. ISSN 00063444. URL http://www.jstor.org/ stable/2332878 .

Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.

Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems , 2021.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems , volume 27. Curran Associates, Inc., 2014. URL https://proceedings. neurips.cc/paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf .

Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Offpolicy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research , pages 1861-1870. PMLR, 10-15 Jul 2018. URL https://proceedings.mlr.press/v80/haarnoja18b.html .

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR , abs/1512.03385, 2015.

Ruifei He, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition?, 2022. URL https://arxiv.org/abs/2210.07574 .

Todd Hester, Matej Vecerik, Olivier Pietquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Arnold, Ian Osband, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations, 2017. URL https://arxiv.org/abs/1704.03732 .

Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning, 2023. URL https://arxiv.org/abs/2301.13442 .

Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NeurIPS 2021 Workshop on Deep Generative Models and Downstream Applications , 2021. URL https://openreview. net/forum?id=qw8AKxfYbI .

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 6840-6851. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ 4c5bcfec8584af0d967f1ab10179ca4b-Paper.pdf .

Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=6nbpPqUCIi7 .

Vincent Huang, Tobias Ley, Martha Vlachou-Konchylaki, and Wenfeng Hu. Enhanced experience replay generation for efficient reinforcement learning, 2017. URL https://arxiv.org/ abs/1705.08245 .

Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(24):695-709, 2005. URL http://jmlr.org/papers/v6/ hyvarinen05a.html .

Baris Imre. An investigation of generative replay in deep reinforcement learning, January 2021. URL http://essay.utwente.nl/85772/ .

Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems , 2019.

Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning , 2022.

Steven Kapturowski, Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent experience replay in distributed reinforcement learning. In International Conference on Learning Representations , 2019.

Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=k7FuTOWMOc7 .

Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 21810-21823. Curran Associates, Inc., 2020. URL https://proceedings.neurips. cc/paper/2020/file/f7efa4f864ae9b88d43527f4b14f750f-Paper.pdf .

Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings , 2014.

Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations , 2022. URL https: //openreview.net/forum?id=68n2s9ZJWF8 .

Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Modelling tabular data with diffusion models, 2022.

Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25 , 2012.

Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 11791191. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/ 2020/file/0d2b2061826a5df3221116a5085a6052-Paper.pdf .

Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 19884-19895. Curran Associates, Inc., 2020. URL https://proceedings.neurips. cc/paper/2020/file/e615c82aba461681ade82da2da38004a-Paper.pdf .

Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.

Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the sample size barrier in model-based reinforcement learning with a generative model. Advances in neural information processing systems , 33:12861-12872, 2020.

Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating statistical overfitting. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=14-kr46GvP-.

Cong Lu, Philip Ball, Jack Parker-Holder, Michael Osborne, and Stephen J. Roberts. Revisiting design choices in offline model based reinforcement learning. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=zz9hXVhf40 .

Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations, 2022. URL https://arxiv.org/abs/2206.04779 .

A.C.P.P. Ludjen. Generative replay in deep reinforcement learning, June 2021. URL http: //essay.utwente.nl/87315/ .

Frank J Massey Jr. The kolmogorov-smirnov test for goodness of fit. Journal of the American statistical Association , 46(253):68-78, 1951.

Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature , 518:529-33, 02 2015. doi: 10.1038/nature14236.

N. Patki, R. Wedge, and K. Veeramachaneni. The synthetic data vault. In 2016 IEEE International Conference on Data Science and Advanced Analytics (DSAA) , pages 399-410, Oct 2016. doi: 10.1109/DSAA.2016.49.

Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=Pv1GPQzRrC8 .

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog , 1(8):9, 2019.

Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems , volume 20. Curran Associates, Inc., 2007. URL https://proceedings.neurips. cc/paper/2007/file/013a006f03dbc5392effeb8f18fda755-Paper.pdf .

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models, 2021.

Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake it till you make it: Learning transferable representations from synthetic imagenet clones. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Stefan Schaal. Learning from demonstration. In M.C. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems , volume 9. MIT Press, 1996. URL https://proceedings.neurips.cc/paper/1996/file/ 68d13cf26c4b4f4f932e3eff990093ba-Paper.pdf .

Juergen Schmidhuber. Reinforcement learning upside down: Don't predict rewards-just map them to actions. arXiv preprint arXiv:1912.02875 , 2019.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022. URL https://openreview.net/forum? id=M3Y74vmsMcY .

Vikash Sehwag, Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=WVX0NNVBBkV .

Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep generative replay. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017. URL https://proceedings.neurips.cc/paper/2017/file/ 0efbe98067c6c73dba1250d2beaa81f9-Paper.pdf .

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning , volume 37 of Proceedings of Machine Learning Research , pages 2256-2265, Lille, France, 07-09 Jul 2015. PMLR. URL https://proceedings.mlr.press/v37/sohl-dickstein15.html .

Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction . The MIT Press, second edition, 2018. URL http://incompleteideas.net/book/the-book-2nd. html .

Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as a 'Launchpad' , 2022. URL https://openreview.net/forum? id=SyAS49bBcv .

Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems , pages 5026-5033. IEEE, 2012. doi: 10.1109/IROS.2012.6386109.

Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-mixer: An all-MLP architecture for vision. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=EI2KOXKdnP .

Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts , 6:100022, 2020. ISSN 2665-9638. doi: https:// doi.org/10.1016/j.simpa.2020.100022. URL https://www.sciencedirect.com/science/ article/pii/S2665963820300099 .

Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research , 9(11), 2008.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation , 23(7):1661-1674, 2011. doi: 10.1162/NECO_a_00142.

Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning, 2022. URL https://arxiv.org/abs/2211.04974 .

Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante, and Kalyan Veeramachaneni. Modeling tabular data using conditional gan. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/ file/254ed7d2de3b23ab10936522dd547b78-Paper.pdf .

Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. In International Conference on Learning Representations , 2022. URL https://openreview.net/forum?id=_SJ-_yyes8 .

Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems , volume 33, pages 14129-14142. Curran Associates, Inc., 2020. URL https://proceedings.neurips.cc/paper/2020/file/ a322852ce0df73e204b7e67cbbef0d0a-Paper.pdf .

Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Brian Ichter, Karol Hausman, and Fei Xia. Scaling robot learning with semantically imagined experience, 2023. URL https://arxiv.org/abs/ 2302.11550 .

Supplementary Material

A Data Modeling

In this section, we provide further details for our data modeling. Our diffusion model generates full environment transitions i.e., a concatenation of states, actions, rewards, next states, and terminals where they are present. For the purposes of modeling, we normalize each continuous dimension (non-terminal) to have 0 mean and 1 std. We visualize the marginal distributions over the state, action, and reward dimensions on the standard halfcheetah medium-replay dataset in Figure 8 and observe that the synthetic samples accurately match the high-level statistics of the original dataset.

We note the difficulties of appropriately modeling the terminal variable which is a binary variable compared to the rest of the dimensions which are continuous for the environments we investigate. This is particularly challenging for 'expert' datasets where early termination is rare. For example, walker2d-expert only has ≈ 0 . 0001% terminals. In practice, we find it sufficient to leave the terminals un-normalized and round them to 0 or 1 by thresholding the continuous diffusion samples in the middle at 0.5. A cleaner treatment of this variable could be achieved by leveraging work on diffusion with categorical variables [31].

State0

State1

State2

State3

State 4

Real

Real

Real

Real

Real

SynthER

SynthER

SynthER

SynthER

SynthER

0.50

0.25

0.50

10

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

0.5

0.0

0.5

1.0

State5

State6

State7

State8

State9

Real

Real

Real

Real

Real

SynthER

SynthER

SynthER

SynthER

SynthER

1.0

0.5

State 10

0.0

0.5

1.0

0.5

State 11

0.0

1.0

0.5

State 12

0.0

0.5

2.5

0.0

State13

2.5

5.0

7.5

-2

State 14

0

Real

Real

Real

Real

Real

SynthER

SynthER

SynthER

SynthER

SynthER

-5

State 15

-30

20

-10

State 16

0

10

20

-20

Action0

20

20

-10

Action 1

0

10

20

-20

Action2

20

Real

Real

Real

Real

Real

SynthER

SynthER

SynthER

SynthER

SynthER

-20

20

-20

10

0

10

20

1.0

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

Action3

Action4

Action5

Reward

Real

Real

Real

Real

SynthER

SynthER

SynthER

SynthER

1.0

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

1.0

0.5

0.0

0.5

1.0

A.1 Data Compression

An immediate advantage of sampling data from a generative model is compression. In Table 5, we compare the memory requirements of SYNTHER and the original data by the number of 32-bit floating point numbers used by each for some sample D4RL [21] datasets. For the original data, this simply scales linearly with the size of the dataset. On other hand, SYNTHER amortizes this in the number of parameters in the denoising network, resulting in a high level of dataset compression, at the cost of sampling speed. This property was also noted in the continual learning literature with generative models summarizing previous tasks [65]. As we discuss in Appendix B.3, sampling is fast with 100K transitions taking around 90 seconds to generate.

B Hyperparameters

B.1 TVAE and CTGAN

In Section 3.1, we compared SYNTHER to the VAE and GAN baselines, TVAE and CTGAN. As these algorithms have not been used for reinforcement learning data before, we performed a hyperparameter search [42] across the following spaces:

These ranges are similar to those listed in Tables 10 and 11 of Kotelnikov et al. [42]. We used 30 trials along with the default.

B.2 Denoising Network

The formulation of diffusion we use in our paper is the Elucidated Diffusion Model (EDM, Karras et al. [38]). We parametrize the denoising network D θ as an MLP with skip connections from the previous layer as in Tolstikhin et al. [70]. Thus each layer has the form given in Equation (3).

The hyperparameters are listed in Table 8. The noise level of the diffusion process is encoded by a Random Fourier Feature [57] embedding. The base size of the network uses a width of 1024 and

depth of 6 and thus has ≈ 6 Mparameters. We adjust the batch size for training based on dataset size. For online training and offline datasets with fewer than 1 million samples (medium-replay datasets) we use a batch size of 256, and 1024 otherwise.

For the following offline datasets, we observe more performant samples by increasing the width up to 2048: halfcheetah medium-expert, hopper medium, and hopper medium-expert. This raises the network parameters to ≈ 25 M, which remains fewer parameters than the original data as in Table 5. We provide ablations on the depth and type of network used in Table 10.

B.3 Elucidated Diffusion Model

For the diffusion sampling process, we use the stochastic SDE sampler of Karras et al. [38] with the default hyperparameters used for the ImageNet, given in Table 9. We use a higher number of diffusion timesteps at 128 for improved sample fidelity. We use the implementation at https:// github.com/lucidrains/denoising-diffusion-pytorch which is released under an Apache license.

The diffusion model is fast to train, taking approximately 17 minutes for 100K training steps on a standard V100 GPU. It takes approximately 90 seconds to generate 100K samples with 128 diffusion timesteps.

C SYNTHER Ablations

We consider ablations on the number of generated samples and type of denoiser used for our offline evaluation in Section 4.1.

C.1 Size of Upsampled Dataset

In our main offline evaluation in Section 4.1, we upsample each dataset (which has an original size of between 100K to 2M) to 5M. We investigate this choice for the walker medium-replay dataset in Figure 9 and choose 10 levels log-uniformly from the range [50K, 5M]. Similarly to He et al. [26], we find that performance gains with synthetic data eventually saturate and that 5M is a reasonable heuristic for all our offline datasets.

10

5

10

6

Number of Samples (log-scale)

0

20

40

60

80

100

Normalized Return

Size of Upsampled Dataset Ablations

IQL (SynthER)

TD3+BC (SynthER)

C.2 Network Ablations

We ablate the hyperparameters of the denoising network, comparing 3 settings of depth from { 2 , 4 , 6 } and analyze the importance of skip connections. The remaining hyperparameters follow Appendix B.2. We choose the hopper medium-expert dataset as it is a large dataset of 2M. As we can see in Table 10, we see a positive benefit from the increased depth and skip connections which leads to our final choice in Table 8.

D RL Implementation

For the algorithms in the offline RL evaluation in Section 4.1, we use the 'Clean Offline Reinforcement Learning' (CORL, Tarasov et al. [68]) codebase. We take the final performance they report for the baseline offline evaluation. Their code can be found at https://github.com/tinkoff-ai/CORL and is released under an Apache license.

For the online evaluation, we consider Soft Actor-Critic (SAC, Haarnoja et al. [24]) and use the implementation from the REDQ [12] codebase. This may be found at https://github.com/ watchernyu/REDQ and is released under an MIT license. We use the 'dmcgym' wrapper for the DeepMind Control Suite [71]. This may be found at https://github.com/ikostrikov/dmcgym and is released under an MIT license.

D.1 Data Augmentation Hyperparameters

For the data augmentation schemes we visualize in Figure 1a, we define:

Additive Noise [45]: adding ϵ ∼ N (0 , 0 . 1) to s t and s t +1 .

Multiplicative Noise [45]: multiplying s t and s t +1 by single number ϵ ∼ Unif ([0 . 8 , 1 . 2]) .

Dynamics Noise [8]: multiplying the next state delta s t +1 -s t by ϵ ∼ Unif ([0 . 5 , 1 . 5]) so that s t +1 = s t + ϵ · ( s t +1 -s t ) .

D.2 Online Running Times

Our online implementation in Section 4.2 uses the default training hyperparameters in Appendix B.2 to train the diffusion model every 10K online steps, and generates 1M transitions each time. On the 200K DMC experiments, 'SAC (SynthER)' takes ≈ 21 . 1 hours compared to ≈ 22 . 7 hours with REDQ on a V100 GPU. We can further break down the running times of 'SAC (SynthER)' as follows:

Diffusion training: 4.3 hours

Diffusion sampling: 5 hours

RL training: 11.8 hours

Therefore, the majority of training time is from reinforcement learning with an update-to-data ratio (UTD) of 20. We expect the diffusion training may be heavily sped-up with early stopping, and leave this to future work. The default SAC algorithm with UTD=1 takes ≈ 2 hours.

E Further Offline Results

In this section, we include additional supplementary offline experiments to those presented in Section 4.1.

E.1 AntMaze Data Generation

We further verify that SYNTHER can generate synthetic data for more complex environments such as AntMaze [21]. This environment replaces the 2D ball from Maze2D with the more complex 8-DoF 'Ant' quadruped robot, and features: non-Markovian policies, sparse rewards, and multitask data. In Table 11, we see that SYNTHER improves the TD3+BC algorithm where it trains (on the 'umaze' dataset) and achieves parity otherwise.

E.2 Offline Data Mixtures

In Section 4.1, we considered exclusively training on synthetic data. We present results in Table 12 with a 50-50 mix of real and synthetic data to confirm that the two are compatible with each other, similar to Section 4.2. We do so by including as many synthetic samples as there are real data. As we stated before, we do not expect an increase in performance here due to the fact that most D4RL datasets are at least 1M in size and are already sufficiently large.

F Latent Data Generation with V-D4RL

We provide full details for the experiments in Section 4.3 that scale SYNTHER to pixel-based environments by generating data in latent space for the DrQ+BC [50] and BC algorithms. Concretely, for the DrQ+BC algorithm, we consider parametric networks for the shared CNN encoder, policy, and Q -functions, f ξ , π ϕ , and Q θ respectively. We also use a random shifts image augmentation, aug. Therefore, the Q -value for a state s and action a is given by Q θ ( f ξ ( aug ( s )) , a ) . The policy is similarly conditioned on an encoding of an augmented image observation.

The policy and Q -functions both consist of an initial 'trunk' which further reduces the dimensionality of the CNN encoding to d feature = 50 , followed by fully connected layers. We represent this as π ϕ = π fc ϕ ◦ π trunk ϕ and Q θ = Q fc θ ◦ Q trunk θ . This allows us to reduce a pixel-based transition to a low-dimensional latent version. Consider a pixel-based transition ( s, a, r, s ′ ) where s, s ′ ∈ R 84 × 84 × 3 . Let h = f ξ ( aug ( s )) and h ′ = f ξ ( aug ( s ′ )) . The latent transition we generate is:

This has dimension 4 · d feature + | a | +1 and includes specific supervised features for both the actor and the critic; we analyze this choice in Appendix F.1. For example, on the 'cheetah-run' environment considered in V-D4RL, since | a | = 6 , the overall dimension is 207 which is suitable for our residual MLP denoising networks using the same hyperparameters in Table 8. This allows us to retain the fast training and sampling speed from the proprioceptive setting but now in pixel space.

To obtain a frozen encoder f ξ and trunks π trunk ϕ , Q trunk θ , we simply train in two stages. The first stage trains the original algorithm on the original data. The second stage then retrains only the fully-connected portions of the actor and critic, π fc ϕ and Q fc θ , with synthetic data. Thus, our approach could also be viewed as fine-tuning the heads of the networks. The procedure for the BC algorithm works the same but without the critic.

We use the official V-D4RL [50] codebase for the data and algorithms in this evaluation. Their code can be found at https://github.com/conglu1997/v-d4rl and is released under an MIT license.

F.1 Ablations On Representation

We analyze the choice of low-dimensional latent representation we use in the previous section, in particular, using specific supervised features for both the actor and critic. We compare this against using actor-only or critic-only features for both the actor and critic, which corresponds to a choice of π trunk ϕ = Q trunk θ , in Table 13. We note that both perform worse with an especially large drop-off for the critic-only features. This may suggest that non-specific options for compressing the image into low-dimensional latents, for example, using auto-encoders [40], could be even less suitable for this task.

Peer reviews:
comments: SUMMARY:
Authors propose to use diffusion models for generating new data based on online interactions or offline dataset. Generating new samples allows online and offline algorithms to perform better. Method can be useed with any offline method and any online method which utilize replay buffers.

STRENGTHS:
Useful idea which works. Both offline and online RL settings  are covered as well as both visual and non-visual tasks. Good ablations and large-scale experiments.

WEAKNESSES:
I  would recommend to validate the approach using Antmaze D4RL datasets as this domain is much more challenging than locomotion and maze2d.

QUESTIONS:
Maybe I missed it, but how would  performance change in offline setting if we used original data with the synthetic one? Does it decrease performance? I don't understand the motivation to throw away original data otherwise except demonstrating that diffusion produces good data which could be an ablation.

LIMITATIONS:
I'm don't knowvery much about how complicated it is to train a diffusion model to solve an arbitary task so making it work seems like a potential bottleneck. Probably, it is faster and easier to run an additional hyperparameter search for the RL algorithm than make diffusion work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents Synthetic Experience Replay, a reinforcement learning algorithm employing a generative model based on diffusion to enrich the training dataset of a learning agent. The method is adapted to offline and online RL, and compared to traditional augmentation strategies (e.g., the addition of random noise) in continuous control tasks, from proprioceptive and visual inputs, while applied to different policy optimization algorithms. The paper also presents detailed analyses on the approach, including results on the quality of the generated data and on using larger neural networks.\n\nSTRENGTHS:\n**Originality**: the general direction of using synthetic data for training machine learning algorithms has been vastly explored in the past, with some applications to reinforcement learning. The approach presented in the paper is, thus, not outstandingly novel. However, one of the main points in the paper is that the main bottleneck in previous approaches leveraging synthetic data in the context of RL has been the quality of the generative model. To the best of my knowledge, this is the first use of diffusion models for synthetic data generation. \n\n**Quality**: the quality of the work is reasonably good. The experiments are well-chosen and the experimental designs are generally good. \n\n**Clarity**: the writing quality is generally very high. The paper is well-written and easy to follow in most parts, with a clear presentation of the experimental results. \n\n**Significance**: I believe demonstrating the effectiveness of powerful generative models in generating synthetic data for reinforcement learning algorithms to use is important for the community, given the simplicity and generality of the approach.\n\nWEAKNESSES:\n**Major Concerns**\n- My main concern is on the number of seeds employed for the experiments in the paper. I understand the authors might be subject to computational constraints, but I find 4 repetitions only to be generally not enough to fully trust an individual experiment. I would suggest the authors to run more seeds per experiment and, if possible, use the methods proposed in \"Deep Reinforcement Learning at the Edge of the Statistical Precipice\" (Agarwal at al., 2021) for aggregation across multiple tasks.\n- In Section 3.1, the paper says that Tabular VAE and Conditional Tabular GAN have been evaluated using the default hyperparameters proposed in the original paper that applies these generative models to tabular data. However, no dataset from the benchmark employed in the original paper contains robotics interactions similar to the one from D4RL, and this thus begs to question of whether better hyperparameters for such different datasets might exist. I believe the paper would be improved by evaluating those approaches after a careful search for optimal hyperparameters.\n\n**Minor Concerns**\n- Some potentially important references are missing from the paper. In particular, I'm thinking of the recent paper on using synthetic experiences (meta-learned) \"Should Models Be Accurate?\" (Saleh et al., 2022), and of some recent papers applying, albeit with other goals, diffusion models to reinforcement learning such as \"Planning with Diffusion for Flexible Behavior Synthesis\" (Janner et al., 2022) and \"Is Conditional Generative Modeling all you need for Decision-Making? (Ajay et al., 2022).\n- The clarity of the paper would benefit from a more explicit description of how the diffusion model is employed for generating the data. One can guess it is naively applied to generating each dimension of each state, action or reward, but being explicit is better than being implicit in this case.\n\nQUESTIONS:\nI ask the author to address the concerns highlighted above in my review.\n\nLIMITATIONS:\nThe computational limitations of the approach are stated in the paper, saying that the approach is computationally better than REDQ. However, the comparison is executed at a replay ratio of 20; it would be desirable to understand what is the actual computational cost of the new data generation and training of the diffusion model even at lower replay ratios.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Synthetic Experience Replay (SYNTHER), a novel diffusion-based approach to upsample an agent's collected experience in deep reinforcement learning (RL). The authors demonstrate that SYNTHER is effective for training RL agents in both offline and online settings, and can improve performance for small offline datasets and larger networks. The paper presents results in both state-based and pixel-based environments and provides evidence that synthetic training data can enable more sample-efficient and scalable training for RL agents.\n\nSTRENGTHS:\n1. The paper introduces a novel approach, SYNTHER, which employ generative diffusion model to upsample an agent's collected experience. This method can potentially resolves the challenge of data scarcity in RL. Generally, this method is easy to understand and effective, and may have a broad impact on sample-efficient RL algorithms.\n2. The authors provide thorough experiments and evaluations in offline and online settings, demonstrating the effectiveness of SYNTHER in improving performance and sample efficiency. The authors also demonstrated the superiority of the diffusion-based model compared to other generative models such as GAN and VAE.\n3. The paper is well-structured, with clear introduction, background and methodology. The author also provide open-source code for reproducibility.\n\nWEAKNESSES:\n1. The performance improvement of SYNTHER seems to be very limited under the setting of small networks.\n2. The paper didn't demonstrate the advantages of using SYNTHER on existing RL algorithms in both online and pixel based scenarios, nor did it study the impact on model-based RL algorithms in online settings.\n\nQUESTIONS:\n1. Can you provide the experimental results of SYNTHER with a model-based RL method without BC?\n2. Can you provide the experimental results of SYNTHER in online and pixel-based setting?\n3. SYNTHER can only generate 1-step transition, so n-step bootstrap or GAE cannot be used in imagined data. Did the baselines use n-step bootstrap or GAE value estimation?\n\nLIMITATIONS:\nSYNTHER have limited results in pixel-based and online settings. This paper has no negative social impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-06"}

comments: SUMMARY:
Building on the recent success of generative models, the authors propose Synthetic Experience Replay to improve how experience or data is used within reinforcement learning algorithms. While data is normally collected through interaction with an environment, the authors suggest that the experience replay can be augmented with additional data now artificially generated from a diffusion model. The paper considers both the online and offline settings of RL, compares with previous tabular data augmentation methods, and measures performance on pixel-based environments.

STRENGTHS:
+ [Quality] The authors conduct an exhaustive set of experiments on standard online and offline RL benchmarks.
+ [Clarity] The paper is well-motivated, organized, and clearly written. Related works are carefully documented and contributions are clear.
+ [Significance] The results are quite compelling, especially in the case of upsampling from small datasets. Considering that no algorithmic changes are necessary, many works stand to benefit from this simple, yet effective strategy.

WEAKNESSES:
+ [Originality] Augmenting data in the replay buffer with synthetic transitions is not new. For example, previous works have attempted to learn a dynamics model during the course of standard online RL training to fill up the replay buffer with fake transitions. Most of the novelty in this work comes from 1) using a diffusion model to generate data and 2) demonstrating experimentally that this kind of synthetic data can be useful in a variety of settings.

QUESTIONS:
-

LIMITATIONS:
-
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06