Synthetic Experience Replay
Authors: Cong Lu
Philip J. Ball
Yee Whye Teh
Jack Parker-Holder
Venue: NeurIPS 2023 poster (2023)
Keywords: Reinforcement Learning
Diffusion Models
Synthetic Data
Sample-Efficient RL

Abstract: A key theme in the past decade has been that when large neural networks and large datasets combine they can produce remarkable results. In deep reinforcement learning (RL), this paradigm is commonly made possible through experience replay, whereby a dataset of past experiences is used to train a policy or value function. However, unlike in supervised or self-supervised learning, an RL agent has to collect its own data, which is often limited. Thus, it is challenging to reap the benefits of deep learning, and even small neural networks can overfit at the start of training. In this work, we leverage the tremendous recent progress in generative modeling and propose Synthetic Experience Replay (SynthER), a diffusion-based approach to flexibly upsample an agent's collected experience. We show that SynthER is an effective method for training RL agents across offline and online settings, in both proprioceptive and pixel-based environments. In offline settings, we observe drastic improvements when upsampling small offline datasets and see that additional synthetic data also allows us to effectively train larger networks. Furthermore, SynthER enables online agents to train with a much higher update-to-data ratio than before, leading to a significant increase in sample efficiency, without any algorithmic changes. We believe that synthetic training data could open the door to realizing the full potential of deep learning for replay-based RL algorithms from limited data. Finally, we open-source our code at https://github.com/conglu1997/SynthER.

Synthetic Experience Replay
Cong Lu", Philip J. Ball", Yee Whye Teh, Jack Parker-Holder University Oxford
Abstract
A key theme the past decade has been that when large neural networks and large datasets combine they can produce remarkable results. In deep reinforcement learning (RL), this paradigm is commonly made possible through experience replay, whereby dataset of past experiences is used to train policy value function. However, unlike in supervised or self-supervised learning, an RL agent has collect its own data, which is often limited. Thus, it is challenging to reap the benefits of deep learning, and even small neural networks can overfit at the start of training. In this work, we leverage the tremendous recent progress in generative modeling and propose Synthetic Experience Replay (SYNTHER), a diffusion-based approach to flexibly upsample agent's collected experience. We show that SYNTHER is an effective method for training RL agents across offline and online settings, in both proprioceptive and pixel-based environments. In offline settings, we observe drastic improvements when upsampling small offline datasets and see that additional synthetic data also allows us effectively train larger networks. Furthermore, SYNTHER enables online agents to train with a much higher update-to-det ratio than before, leading to significant increase sample efficiency, without any algorithmnic changes. We believe that synthetic training data could open the door to realizing the full potential of deep learning for replay-based RL algorithms from limited data. Finally, we open-source our code at https: //g1 thub. hwom//ongl1199///5500 SyythER
Data Augmentation Methods
Data-Efficient Online RL
SynthER
(SymthER)
Additive Naise
Dynamics Noise
REDQ
Multiplicative Noise
SAC
Normalized Retur
Normalized Return
(a) IQL 41 on a reduced 15% subset of walker2d medium-replay (21).
(b) SAC 24 on 6 DeepMind Control Suite 71 and OpenAl Gym [TO environments.
Upsampling data using SYNTHER greatly outperforms explicit data augmentation schemes for small offiine datascts and data-cfficien algorithms in online RL without any algorithmic changes. Morcover, synthetic data from SYNTHER may readily be added any algorithm utilizing experience replay. Full results in Scction
Figure 1:
Introduction
In the past decade, the combination of large datasets [4 63 and ever deeper neural networks
15 25| 43 [3] has led to a series of more generally capable models [1]66]]]]]] [58] In reinforcement learning (RL, Sutton and Barto (67]), agents typically learn online from their experience. Thus,
Equual contribution. Correspondence cong. wwwssttttt.... ac. uk and ballerobots.c ax..c.uk
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

SynthER
Figure 2: SYNTHER allows any RL agent using experience replay to arbitrarily upsample, or increase the quantity of, their experiences (in grey) and train on synthetic data (in blue). We evaluate our ap- proach up to a factor 100x more data in Section[4.2 across both pro- prioceptive and pixel-based envi- ronments. By leveraging this in- creased data, agents can learn ef- fectively from smaller datasets and achieve higher sample efficiency. Details of the upsampling process given in Figure 3
UPSAMPLE
Train agent on upsampled data
Rollout data (s, a, r,s
Agent interacts with the environment
to leverage sufficiently rich datasets, RL agents typically make use of experience replay where training takes place dataset of recent experiences. However, this experience is typically limited, unless an agent distributed over many workers which requires both high computational cost and sufficiently fast simulation [8 37
19
Another approach for leveraging broad datasets for training RL policies is offline RL 2 46 whereby
behaviors may be distilled from previously collected data either via behavior cloning [6]. off- off-policy
learning [22 44 or model- based methods (39 49 78.. Offline data can also significantly bootstrap
online learning 99 27 75 ; however, it is a challenge apply these methods when there is mismatch between offline data and online environment. Thus, many of the successes rely on toy domains with transfer from specific behaviors in a simple -in-dimenioona proprioceptive environment.
Whilst strong results have been observed in re-using prior data in RL, appropriate data for particular behaviors may simply not exist and thus this approach falls short in generality. We consider an alternative approach-rathes than passively reusing data, we leverage tremendous progress in genera- tive modeling to generate a large quantity of new, synthetic data. While prior work has considered upsampling online RL data with VAEs GANs [32 34 [1]], we propose making use of diffusion generative models 30 38 66], which unlocks significant new capabilities.
Our approach, which we call Synthetic Experience Replay, SYNTHER, is conceptually simple, whereby given a limited initial dataset, we can arbitrarily upsample the data for an agent to use as if it was real experience. Therefore, in this paper, seek to answer simple question: Can the latest generative models replace or augment (raditional datasets in reinforcement learning? To answer this, consider the following settings: offline RL where entirely replace the original data with data produced by a generative model, and online RL where we upsample experiences to broaden the training data available to the agent. In both cases, SYNTHER leads to drastic improvements, obtaining performance comparable to that of agents trained with substantially more real data. Furthermore, in certain offline settings. synthetic data enables effective training of larger policy and value networks, resulting in higher performance by alleviating the representational bottleneck. Finally, we show that SYNTHER scales to pixel-based environments by generating data latent space. We thus believe this paper presents sufficient evidence that our approach could enable entirely new, efficient, and scalable training strategies for RL agents. To summarize, the contributions of this paper are:
We propose SYNTHER in Section 3 a diffusion based approach that allows one to generate synthetic experiences and thus arbitrarily upsample data for any reinforcement learning algorithm utilizing experience replay.
We validate the synthetic data generated by SYNTHER in offline settings across proprio- ceptive and baxel-based environments in Section I Section 4.3, presenting the first generative approach to show parity with real data on the standard D4RL and V-D4RL offline datasets with a wide variety of algorithms. Furthermore, we observe considerable improvements from upsampling for small offline datasets and scaling up network sizes.
We show how SYNTHER can arbitrarily upsample an online agent's training data in Sec- tion 4.2 by continually training the diffusion model. This allows us to significantly increase an agent's update-to-datt (UTD) ratio matching the efficiency of specially designed data- efficient algorithms without any algorithmic changes.

---

Reverse Denoising Process
6
Noise
Data
Forward Diffusion Process
Figure 3: SYNTHER generates synthetic samples using diffusion model which we visualize on the proprio- ceptive walker2d environment. On the top row, we render the state component of the transition tuple on a subset of samples; and on the bottom row, we visualize t-SNE [72 projection 100,000 samples. The denoising process creates cohesive and plausible transitions whilst also remaining diverse, as seen by the multiple clusters that form at the end of the process in the bottom row.
2 Background
2.1 Reinforcement Learning
We model the environment as Markov Decision Process (MDP, Sutton and Barto (67] defined as tuple M = AS,A, R, 7o, where S and A denote the state and action spaces respectively, P(s'Is,) a) the transition dynamies, R(s, a) the reward function, Po the initial state distribution, and (0, 1) the discount factor. The goal in reinforcement learning is to optimize a policy π (als) that maximizes the expected discounted return Ex EE ? ac)].
2.2 Offfine Reinforcement Learning
In offline RL [46. the policy is not deployed in the environment until test time. Instead, the more
algorithm only has access static dataset Denv { (1, Pt, $(+1)) collected by behavioral policies T&- We refer to the distribution from which Dew was sampled as the behavioral distribution (78). In some of the environments we consider, the environment may be finite horizon have early termination. In that case, the transition tuple also contains a terminal flag de where de = 1 indicates the episode ended early at timestep and de 0 otherwise.
2.3 Diffusion Models
Diffusion models 330 666 are a class of generative models inspired by non-equilibrium thermody- namics that learn to iteratively reverse forward noising process and generate samples from noise. Given data distribution p(x) with standard deviation data we consider noised distributions p(x; o) obtained by adding i.i.d. Gaussian noise of standard deviation to the base distribution. The forward noising process is defined by sequence noised distributions following a fixed noise schedule do max 01
o 0. When max danaa the final noised distribution p(x; d is essentially indistinguishable from random noise.
Karras et al. [88 consider a probability- flow ODE with the corresponding continuous noise schedule (t) that maintains the desired distribution as x evolves through time given by Equation
dx - àà (L) logp(x; (())dt
where the dot indicates a time derivative and Vx p(x; (()) is the score function points towards the data at a given noise level. Infinitesimal forward or backward steps of this ODE either nudge sample away towards the data. Karras et al. (38] consider training denoiser Do(x; o) on an L2 denoising objective:
33 which
min Ex0p... I| [x +
x||
and then use the connection between score-matching and denoising 74 to obtain Vx loggp[ o) = (Do(x; rr x)/ We may then apply an ODE (or SDE as a generalization of Equation (1) solver to reverse the forward process. this paper, we train our diffusion models to approximate the online or offline behavioral distribution.
3

---

Algorithm SYNTHER for online replay-based algorithms. Our additions are highlighted in blue.
1: Input: real data ratio E [0. 1]
2: Initialize: Deal 0 real replay buffer, т agent, D Dysthetic synthetic replay buffer, M diffusion model
3: fort =w.==,..... ,,..., T
do
4: Collect data with in the environment and add them Dreai
Update diffusion model with samples from Deal
6: Generate samples from M and add them to Dyyntess
7: Train π samples from Deaa U syathetic mixed with ratio
8: end for
3 Synthetic Experience Replay
In this section, we introduce Synthetic Experience Replay (SYNTHER), our approach to upsampling an agent's experience using diffusion. We begin by describing the simpler process used for offline RL and then how that may be adapted to the online setting by continually training the diffusion model.
3.1 Offline SYNTHER
For offline reinforcement learning, we take the data distribution of the diffusion model p(x) to simply be the offline behavioral distribution. In the proprioceptive environments we consider, the full transition is low-dimensional compared with typical pixel-based diffusion. Therefore, the network architecture an important design choice; and similarly to Pearce et al. [55] we find it important to residual MLP denoising [70] network. Furthermore, the choice of the Karras et al. [38] sampler allows us to use a low number of diffusion steps (n = 128) resulting in high sampling speed. Full details for both are provided in Appendix B We visualize the denoising process
representative D4RL [21] offline dataset, in Figure We further validate our diffusion model on the D4RL datasets in Figure in Appendix A showing that the synthetic data closely matches the original data when comparing the marginal distribution over each dimension. In Section 4.3, we show the same model may be used for pixel-based environments by generating data in low-dimensional latent space.
Next, we conduct a quantitative analysis and show that the quality of the samples from the diffusion model is significantly better than with prior generative models such VAEs 440 and GANs (23). We consider the state-of-the-art Tabular VAE (TVAE) and Conditional Tabular GAN (CTGAN) models proposed by Xu et al. [76). and tune them on the D4RL halfcheetah medium-replay dataset. Full hyperparameters are given in Appendix B.I. As proposed in Patki et al. (54). we compare the following two high-level statistics: (1) Marginal: Mean Soetimooooo--mmmiine
(52) statistic, measuring the maximum distance between empirical cumulative distribution functions, for each dimension of the synthetic and real data; and (2) Correlation: Mean Correlation Similarity, measuring the difference in pairwise Pearson rank correlations 20 between the synthetic and real data.
We also assess downstream offline RL performance using the synthetic data with two state-of-the art offline RL algorithms, TD3+BC [22] and IQL [41]. in Table [ The full evaluation protocol is described in Section 4. The diffusion model is far more faithful to the original data than prior generative models which leads to substantially higher returns on both algorithms. Thus, we hypothesize large part of the failure of prior methods 33 51 is due weaker generative model.
3.2 Online SYNTHER
SYNTHER may be used upsample an online agent's experiences by continually training the diffusion model on new experiences. We provide pseudocode for how to incorporate SYNTHER
Table 1: SYNTHER is better at capturing both the high-level statistics of the dataset (halscheetah medium- replay) than prior generative models and also leads to far higher downstream performance. Metrics (left) computed from 100K samples from each model, offline RL performance (right) computed using 5M samples from each model. We show the mean and standard deviation of the final performance averaged over seeds.
Metrics
Eval. Return
Model
Marginal
Correlation
TD3+BC
IQL
Diffusion (Ours)
0,989
0.998
45.9-10.9
46.6:10.2
VAF 7
0.942
0.979
27.1+2.1
15.2422.2
GAN 76
0.959
0.981
24.3++1.9
15.9+2.4

---

Table 2: A comprchensive evaluation of SYNTHER wide variety of proprioceptive D4RL 21 datasets selection of state-of the-art art offline RL algorithms. show that synthetic data from SYNTHER faithfully reproduces the original performance. which allows us to completely eschew the original training data. We show the mean and standard deviation of the final performance averaged over 8 seeds. Highlighted figures show at least parity over each group (algorithm and environment class) of results.
Behavioral
TD3+BC (22
IQL
EDAC
Environment
Policy
Original
SYNTHER
Original
SYNTHER
Original
SYNTHER
random
I1.300
12.2=11
15.211.2
17.243.4
halfchectahh-v2
mixed
44,8=0.7
45,9=0.9
43.5+004
46.6+0.2
62.1+13
63.0+1.3
medium
48.11002
49.9
48.3+0.11
49.610.3
67.7+1.2
65.111.3
medexp
908+77.0
87.2=11.1
94.6+=0.2
93.3+2.6
104.8+0.7
94,1+10.1
random
0.620..
2.3=19
4.120.8
4,220.3
walker2d-v2
mixed
85.644..
90.5+43
82.6+800
83.345.9
87.1+3.2
1.5
medium
82.7.55.5
84.8
84.015.4
84.7.55.5
93.4116
93.4
medexp
110.0=044
110.2=055
111,7=0.
111.
114.8+0.9
114.7=1.2
rundom
8.6+0.3
14.6=9.4
7.22002
7.740.1
mixed
64.44:24.8
53.4
84.6+13.5
103.24:0.4
99.7++0.9
101 0.
hopper-v2
medium
60.41:4.0
63.41:22
62.816.0
72.01:.5
101.7410.3
402.410.5
medexp
101.1=10,5
105.4=9.7
106.2=6.1
90,8+179
105.2+11.6
109.7+0.2
locomotion average
59.0::4.9
60.0=5.1
62.1
63.7+3,55
92,942,4
92.6:12.11
umuze
29.4=14.2
37.6=1444
31.7=2.0
41.0=0.7
95.3477
18.1+18.6
maze2d-vl
medium
59.5:141.9
65.2433.1
35.5
$7.0
66.4410099
large
97.1429..3
92.5=38.5
49.612220
60.815,3
95.61126.5
143.3+21.7
maze average
62.0 02.01.28.2
65.1 29.7
40.918.3
45.61229
82.6112.66
102.9 17.1
into any online replay-based RL agent in Algorithm I and visualize this in Figure 2 Concretely, diffusion model is periodically updated on the real transitions and then used to populate a second synthetic buffer. The agent may then be trained on mixture of real and synthetic data sampled with ratio r. For the results in Section 4.2, we simply set 0.5 following Ball al. (9. The synthetic replay buffer may also be configured with a finite capacity to prevent overly stale data.
4 Empirical Evaluation
We evaluate SYNTHER across a wide variety of offline and online settings. First, we validate our approach offline RL, where we entirely replace the original data, and further show large benefits from upsampling small offline datasets. Next, we show that SYNTHER leads to large improvements in sample efficiency in online RI., exceeding specially designed data-efficient approaches. Furthermore, we show that S SYNTHER scales to pixel-based environments by generating data in latent space. Finally, we perform meta-analysis over our empirical evaluation using the RLiable 3 framework in Figure 7
4.1 Offline Evaluation
We first verify that synthetic samples from SYNTHER faithfully model the underlying distribution from the canonical offline D4RL (21) datasets. To do this, we evaluate S SYNTHER in combination
with 3 widely-used SOTA offline RL algorithms: TD3+BC (Fujimoto and Gu [22 explicit policy regularization), IQL (Kostrikov et al. [1], expectile regression), and EDAC (An et al. [5], uncertainty- based regularization) on an extensive selection of D4RL datasets. We consider the MuJoCo [69] locomotion (halfcheetah, walker2d, and hopper) and maze2d environments. In these experiments, all datasets share the same training hyperparameters in Appendix E, with some larger datasets using wider network. For each dataset, we upsample the original dataset to 5M samples; we justify this choice in Appendix C. We show the final performance in Table 2
Our results show that we achieve at least parity for all groups of environments and algorithms as highlighted in the table, regardless of the precise details of each algorithm. We note significant improvements to maze2d environments, which are close to the 'best' performance as reported in CORL 68 (i.e., the best iteration during offline training) rather than the final performance. We hypothesize this improvement is largely due to increased data from SYNTHER, which leads to less overfitting and increased stability. For the locomotion datasets, we largely reproduce the original results, which we attribute to the fact that most D4RL datasets are at least 1M in size and are already sufficiently large. However, as detailed in Table i Appendix[A.1 size of the dataset to be compressed significantly, up to 12.9x on some datasets. Finally, we present results on the AntMaze environment in Appendix E.T and experiments showing that the synthetic and real data are compatible with each other in Appendix E.2
SYNTHER allows the effective
5

---

medium-replay
medium
medium-eeppert
100
80
80
80
23
20
0.3
0.05
0.20
80
IQL Return
40
0.1
022
03
54
0.5
0.05
0.10
0.15
0:20
0:22
0.04
0.00
0.08
0.10
Subsample Level
Subeample Level
Subsample Level
SynthER
Original
Figure 4: SYNTHER is a powerful method for upsampling reduced variants of the walker2d datasets and vastly improves competitive explicit data augmentation approaches for both the TD3+BC (top) and IQL. (bottom) algorithms. The subsampling levels scaled proportionally the original size of each dataset. We show the mean and standard deviation the final performance averaged over seeds.
4.1.1 Upsampling for Small Datasets
We investigate the benefit of SYNTHER for small offline datasets and compare it to canonical 'explicit' data augmentation approaches g
Concretely, we wish to understand whether SYNTHER generalizes and generates synthetic samples that improve policy learning compared with explicitly augmenting the data with hand-designed inductive biases. We focus on the walker2d (medium, medium-replly/miixed medium-expert) datasets in D4RL and uniformly subsample each at the transition level. We subsample each dataset proportional to the original dataset size so that the subsampled datasets approximately range from 20K to 200K samples. As in Section 4. we then use SYNTHER to upsample each dataset to 5M transitions. Our denoising network uses the same hyperparameters as for the original evaluation in Section 4.1
In Figure 4 we can see that for all datasets, SYNTHER leads to a significant gain performance and vastly improves on explicit data augmentation approaches. For explicit data augmentation, we select the overall most effective augmentation scheme from Laskin et al. 45 (adding Gaussian noise of the form (0, 0.1)). Notably, with SYNTHER datasets starting from only 3% of the original data.
achieve close to the original levels of
performance on the liikerdd-meiium-eeee In Figure Ta we methodically compare across both additive and multiplicative versions of RAD, as well as dynamics augmentation [8] on the 15% reduced walker medium-replay dataset.
Why is SYNTHER better than explicit augmentation? To provide intuition into the efficacy of S SYNTHER over canonical explicit augmentation approaches, we compare the data gen- erated by SYNTHER to that generated by the best-performing data augmentation approach in Figure Ta namely additive noise. We wish to evaluate two properties: 1) How diverse is the data? 2) How accurate is the data for the purposes of learn- ing policies? To measure diversity, we measure the minimum 1.2 distance of each datapoint from the dataset, which allows us to see how far the upsampled data is from the original data. To measure the validity of the data, we follow Lu al. [49] and measure the MSE between the reward and next state proposed by SYNTHER with the true next state and reward defined by the simulator. We plot both these values in joint scatter plot to compare how they vary with respect each other. For this, we compare specifically on the reduced 15% subset of walker2d medium-replay as Figure Ia As we see Figure SYNTHER generates significantly wider marginal distribution over the distance from the dataset, and generally produces samples that are further away from the dataset than explicit augmentations. Remarkably, however, we that these samples are far more consistent with the true environment dynamics. Thus, SYNTHER
40
SynthER
Augimented Data
Dynamics MSE
20
10
L2 Distance from Dataset
Figure 5: Comparing L2 distance from
training data and dynamics accuracy der SYNTHER and augmentations.

---

Table SYNTHER enables effective training of larger policy and value networks for TD3+BC [22] leading to a 11.7% % gain on the offline MuloCo locomotion datasets. In comparison, simply increasing the network size with the original data does not improve performance. We show the mean and standard deviation of the final performance averaged over seeds.
Behavioral
T Larger Network
Environment
Baseline
Policy
Original Data
SYNTHER
random
11.3
11.010.7
1..8.1.8
mixed
4..
4..8+1.11
48.0++0.5
halfcheetah-v2
medium
1.2
50.2+
53.3++0.4
medexp
90.8+7.0
95.5+5.4
100.1+2.7
random
0.6
2.6+2.11
4.3+
mixed
85.6+4.6
76,4+999
93.6++3.6
walker2d-v2
medium
82.7=5.5
8.,5+117
87.2+1.2
medexp
110.0:10.4
110.3.00.5
110.2.10.3
random
8.6
10.3
19.5
mixed
64.4
62,4+21.6
86.8+
hopper-v2
medium
60.4:14.0
61.9:1599
65.1:14.7
medexp
101.1 + 10.5
104.6449.4
109.7-1-4.1
locomotion average
59.0:t499
59,5
3..7
generates samples that have significantly lower dynamics MSE than explicit augmentations, even for datapoints that far away from the training data. This implies that high level of generalization has been achieved by the SYNTHER model, resulting in the ability to generate novel, diverse, yet dynamically accurate data that can be used by policies to improve performance.
4.1.2 Scaling Network Size
further benefit we observe from SYNTHER on the TD3+BC algorithm that upsampled data can enable scaling of the policy and value networks leading to improved performance. As is typical for RL algoritmms, TD3+BC uses a small value and policy network with two hidden layers, and width of 256, and batch size of 256. We consider increasing the size of both networks to be three hidden layers and width 512 (approximately 6x more parameters), and the batch size to 1024 to better make use of the upsampled data in Table
We observe a large overall improvement of 11.7% for the locomotion datasets when using larger network with synthetic data (Larger Network + SYNTHER). Notably, when using the original data (Larger Network + Original Data), the larger network performs the same as the baseline. This suggests that the bottleneck in the algorithm lies in the representation capability of the neural network and synthetic samples from SYNTHER enables effective training of the larger network. This could alleviate the data requirements for scaling laws in reinforcement learning [8). However, for the IQL and EDAC algorithms, we did not observe an improvement by increasing the network size which suggests that the bottleneck there lies in the data algorithm rather than the architecture.
4.2 Online Evaluation
Next, show that SYNTHER can effectively upsample online agent's continually collected experiences. In this section, we follow the sample-efficient RL literature [12 [6] and consider
environments from the DeepMind Control Suite (DMC, Tunyasuvunakool al. (7T) (cheetah-run, quadruped-walk, and reacher-hard) and 3 environments the OpenAl Gym Suite 10] (walker2d, halfcheetah, and hopper). As in Chen al. [12]. D'Oro et [16]. we choose the base algorithm to be Soft Actor-Critic (SAC. Haarnoja et al. (24)). a popular off-policy entropy-regularize algorithm, and benchmark against a SOTA sample-efficient variant of itself, 'Randomized Ensembled Double Q- Learning' (REDQ, Chen et al. [12 D. REDQ uses an ensemble of 10 Q-functions and computes target values across a randomized subset of them during training. By default, SAC uses an update- toee-to-dtzz ratio of (1 update for each transition collected); the modifications to SAC in REDQ enable this to be raised to 20. Our method, 'SAC (SYNTHER)', augments the training data by generating 1M new samples for every 10K real samples collected and samples them with ratio = 0.5. We then match REDQ and train with UTD ratio 20. We evaluate our algorithms over 200K online steps for the DMC environments and 100K for OpenAl Gym.

---

cheetah-run-v0
quadruped-walk--0
reacher-hard-v0
Return (DMC)
1000
1002
833
800
000
805
422
400
200
200
0
100
1:00
200
0
walker2d-v2
halfcheetah-v2
hopper-v2
Eval Retum (OpenAl)
8000
10003
4002
4000
8030
2002
3000
2002
2000
1908
1002
Timestops (K)
Timestops (K)
Timastaps (K)
SAC (SynthER)
REDQ
SAC
Figure 6: SYNTHER greatly improves the sample efficiency of online RL algorithms by enabling an agent to train on upsampled data. This allows an agent to use an increased update-to-data ratio (UTD=20 compared to for regular SAC) without any algorithmic changes. We show the mean and standard deviation of the online return over 6 seeds. DeepMind Control Suite environments shown in the top row, and OpenAl Gym environments are shown in the bottom.
In Figure we see that SAC (SYNTHER) matches or outperforms REDQ on the majority of the environments with particularly strong results on the quadruped- walk and halfcheetah--// environ- ments. This is particularly notable as Orro et al. [16] found that UTD=20 on average decreased performance for SAC compared with the default value of 1, attributable to issues with overestimation and overfitting 12 48). We aggregate the final performance on the environments Figure [b] normalizing the DMC returns following Lu al. (50] and OpenAl returns in D4RL. Moreover, due to the fast speed of training our diffusion models and fewer Q-networks, our approach is in fact faster than REDQ based on wall-clock time, whilst also requiring fewer algorithmic design choices, such as large ensembles and random subsetting. Full details on run-time are given in Appendix D.2
4.3 Scaling to Fixel-Based Observations
Finally, we show that we can readily scale SYNTHER to pixel-based environments by generating data in the latent space of CNN encoder. We consider the V-D4RL (50) benchmarking suite, set standardized pixel-based offline datasets, and focus on the 'chectah-run' and 'walker walkerrwwakk' environments. We use the associated DrQ+BC [50] and BC algorithms. Whilst the original image observations are of size 84 84 x 3, we note that the CNN encoder in both algorithms generates features that are 50 dimensional 777. Therefore, given frozen encoder pre-trained on the same dataset, we can retain the fast training and sampling speed of our proprioceptive models but now in pixel space. We present full details in Appendix
Analogously to the proprioceptive offline evaluation Section 4.1 we upsample 5M latent transitions for each dataset and present downstream performance in Table 4 Since the V-D4RL datasets are smaller than the D4RL equivalents with base size of 100K, we would expect synthetic data to be beneficial. Indeed, we observe a statistically significant increase in performance of +9.5 % and +6.8% on DrQ+BC and BC respectively: with particularly strong highlighted results on the medium and expert datasets. We believe this serves compelling evidence of the scalability of SYNTHER to high-dimensional observation spaces and leave generating data in the original image space, or extending this approach to the online setting for future work.
Related Work
Whilst generative training data has been explored in reinforcement learning; in general, synthetic data has not previously performed as well as real data on standard RL benchmarks.
Generative Training Data. Imre 34 Ludjen [1 considered using VAEs and GANs to generate
synthetic data for online reinforcement learning. However, we note that both works failed match the original performance simple environments such as CartPole-the is likely due to the use of

---

Table We scale SYNTHER to high dimensional pixel-based environments by generating data the latent space of a CNN encoder pre-trained on the same offline data. Our approach is composable with algorithms that train with data augmentation and leads to +9.5% % and +6.8% overall gain on DrQ+BC and BC respectively.
We show the mean and standard deviation of the final performance averaged over 6 seeds.
DrQ+BC [50)
BC [0)]
Environment
Original
SynthER
Original
SynthER
mixed
28.7:6.9
32.317.66
16.514.3
12.313.6
medium
46.8+:2.3
44.0+2299
40.9+3.1
40.3+3.0
walker-walk
medexp
86.44:5.6
83.4+6.33
47.7+3.9
45.2++4.5
expert
68.4+7.5
83.6+7.5
91.5+3.9
92.0+4.2
mixed
44.8+3.66
43.8+2.7
25.0+:3.6
27.9+3.4
medium
53.0#3.0
56.0+1.2
51.6+1.4
5.22+1.2
cheetah-run
medexp
50.6+88.2
56.9+8.1
57.5+6.3
69.9
expert
34.5.18.3
52.3.1..0
67.4.6.8
85.413.1
Average
51.7=5.7
56.5#5.44 (49.5%
49,8
53.2=4.1 (+6.8%)
weaker class of generative models which we explored in Section 3.1 Huang et
32 considered
using GAN samples to pre-train RL policy, observing modest improvement in sample efficiency for CartPole. Chen et al. [13,, Yu et al. [79] consider augmenting the image observations of robotic control data using guided diffusion model whilst maintaining the same action. This differs from our
approach which models the entire transition and can synthesize novel action and reward labels.
Outside of reinforcement learning, Azizi et al. 77 He et al. [26), Sariyildiz et al. [60] consider generative training data for image classification and pre-training. They also find that synthetic data improves performance for data-scarce settings which are especially prevalent in reinforcement learning. Sehwag al. (64] consider generative training data improve adversarial robustness in image classification. In continual learning. "generative replay" [65] has been considered to compress examples from past tasks to prevent forgetting.
Generative Modeling in RL. Prior work in diffusion modeling for offline RL has largely sought to supplant traditional reinforcement learning with "upside-down RL," (62] Diffuser (36) models long sequences of transitions or full episodes and can bias the whole trajectory with guidance towards high reward or a particular goal. It then takes the first action and re-plans by receding horizon control. Decision Diffuser 4 similarly operates at the sequence level but instead uses conditional guidance on rewards and goals. Du al. [17 present similar trajectory-baase data. In contrast, S SYNTHER operates at the transition level and seeks to be readily compatible with existing reinforcement learning algorithms. Pearce et al. [55] consider a diffusion-based approach to behavioral cloning, whereby state coneiicondiional diffusion model may be used to sample actions that imitate prior data. Azar et al. (6. Li al. [47 provide theoretical sample complexity bounds for model-based reinforcement learning given access to a generative model.
algorithm for visual
Model-Based Reinforcement Learning. We note the parallels between our work and model-based reinforcement learning (35] 49 78): which tends to generate synthetic samples by rolling out using forward dynamics models. Two key differences of this approach to our method are: SYNTHER synthesizes new experiences without the need to start from real state and the generated experiences are distributed exactly according to the data, rather than subject to compounding errors due to modeling inaccuracy. Furthermore, SYNTHER is an orthogonal approach which could in fact be combined with forward dynamics models by generating initial states using diffusion, which could lead to increased diversity.
Algenthes X
Akyyctivv
Algorthee X
AlgorthreX X
Larger Network X0vitern
Saadine
BAG S5yinertty
DiOveo Benieero
Larerr Netvor (eo Sydtar)
Sceedine
BAC (3ye6Rt)
REDO
D0 (yyeer))
SC
1)
10
(a) Offline TD3+BC Larger Networks (Full results in Table
(b) Online DMC Evaluation (Full
(c) Offline V-D4RL Evaluation
results in Figure 6)
(Full results in Table 4
Figure 7: RLiable 3 analysis allowing us aggregate results across environments and show the probability of improvement for SYNTHER across our empirical evaluation.

---

6 Conclusion
In this paper, we proposed SYNTHER, powerful and general method for upsampling agent experi- ences in any reinforcement learning algorithm using experience replay. We integrated SYNTHER with ease on six distinct algorithms across proprioceptive and pixel-based environments, each fine-tuned for its own use case, with no algorithmic modification. Our results show the potential of synthetic training data when combined with modern diffusion models. offline reinforcement learning, SYNTHER allows training from extremely small datasets, scaling up policy and value networks, and high levels of data compression. In online reinforcement learning, the additional data allows agents to use much higher update-to-data ratios leading to increased sample efficiency.
We have demonstrated that SYNTHER scalable approach and believe that extending more settings would unlock extremely exciting new capabilities for RL agents. SYNTHER could readily be extended to n-step formulations of experience replay by simply expanding the input space of the diffusion model. Furthermore, whilst we demonstrated an effective method to generate synthetic data in latent space for pixel-based settings, exciting future work could involve generating transitions in the original image space. In particular, one could consider fine-tuning large pre- tre-trained foundation models [59] and leveraging their generalization capability to synthesize novel views and configurations of a pixel-based environment. Finally, by using guidance for diffusion models [29]. the generated synthetic data could be biased towards certain modes, resulting in transferable and composable sampling strategies for RL algorithms.
Acknowledgments
Cong Lu funded by the Engineering and Physical Sciences Research Council (EPSRC). Philip Ball is funded through the Willowgrove Studentship. The authors would like to thank the anonymous Reincarnating Reinforcement Learning Workshop at ICLR 2023 and NeurIPS 2023 reviewers for positive and constructive feedback which helped to improve the paper. We would also like to thank Shimon Whiteson, Jakob Foerster, Tim Rocktäschel and Ondrej Bajgar for reviewing earlier versions of this work.
References
[1] Adaptive Agent Team, Jakob Bauer, Kate Baumli, Satinder Baveja, Feryal Behbahani. Avishkar Bhoopchand, Nathalie Bradley-Schmies, Michael Chang, Natalie Clay, Adrian Collister, Vib- havari Dasagi, Lucy Gonzalez, Karol Gregor, Edward Hughes, Sheleem Kashem, Maria Loks- Thompson, Hannah Openshaw, Jack Parker-Holder, Shreya Pathak, Nicolas Perez-Nieves, Nemanja Rakicevic, Tim Rocktäschel, Yannick Schroecker, Jakub Sygnowski, Karl Tuyls, Sarah York, Alexander Zacherl, and Lei Zhang. task space, 2023. URL /tttps://arxiv wrg. abs////s//// 2aa.:///30..0000 07608
adaptation in an open-ended
[2] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective offline reinforcement learning. In International Conference on Machine Learning, 2020.
[3] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Mare Belle- mare. Deep reinforcement learning at the edge of the statistical precipice. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 99304-29320. Curran Associates, Inc., 2021. URL http:/ttttt::/ppccediinaa neurips.e..pr.re cc/ wwt.cpccppppppppc.ee paper_ fwwes/llppppeeee http//ppapr///////eeee E552ecc31cc146550044416677666663
[4] Anurag Ajay, Yilun Du Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision decision-making?? arXiv preprint arXiv:2211.1565: 2022.
[5] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified qeensemble. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 7436-7447. Curran As- sociates, Inc., 2021. URL https:/proocetinggss wwwpeediiinnnneuuiisseee cc/ petprr/p..cc/ppppp222////eee /atprr//22///ille/ Scc22488l55444555500022555000 Paper, pdf
10

---

[6] Mohammad Gheshlaghi Azar, Rémi Munos, and Bert Kappen. On the sample complexity of reinforcement learning with a generative model. arXiv preprint 1XXi::2006.6400 2012.
[7] Shekoofeh Azizi, Simon Kornblith, Chitwan Saharia, Mohammad Norouzi, and David J. Fleet. Synthetic data from diffusion models improves imagenet classification, 2023.
[8] Philip J Ball, Cong Lu, Jack facilitate zero-shot dynamics generalization from a single offline environment. In Marina Meila and Tong Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 619-629, PMIR,, 18-24 Jul 2021. URL
and Stephen Roberts. Augmented world models
mlr ptres//139// balllia.... html
[9] Philip J. Ball, Laura Smith, Iya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data, 2023. URI.https://aaxiv http:////aaxvvvv..aa//.. wrgg/bs////// 02948
[10] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang. and Wojciech Zaremba. Openai gym, 2016.
[11] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[12] Xinyue Chen, Che Wang. Zijian Zhou, and Keith W. Ross. Randomized ensembled double q-learning: Learning fast without model. In International Conference on Learning Represen- tations, 2021. URL /ttps:://///o wwwp:/eee/ootmmmemooommmmm 2d
[13] Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation, 2023. URL 06671
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[15] Jacob Devlin, Ming- Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: deep bidirectional transformers for language understanding. arXiv preprint arXiv: arXiv:1810.00 2018.
Pre-training of
[16] Pierluca D'Oro, Max Schwarzer, Evgenii Nikishin, Pierre-Luc Bacon, Mare G Bellemare, and Aaron Courville. Sample-efficient reinforcement learning by breaking the replay ratio barrier. In The Eleventh International Conference Learning Representations, 2023. URL
wiw.orummmidp-----ee 9BBBJJe
[17] Yilun Du, Mengjiao Yang, Bo Dai, Hanjun Dai, Ofir Nachum, Joshua B. Tenenbaum, Dale Schuurmans, and Pieter Abbeel. Learning universal policies via text-guided video generation,
2023. http:tss:: //atxxs org/abs.////eeoooooooo.oooeoe 22rg/222/3330..... 2302 000111
[18] Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: Scalable distributed Re with importance weighted actonrr architectures. In Proceedings of the 35th International Conference Machine Learning, 2018.
[19] William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney. Revisiting fundamentals of experience replay. In Proceedings of the 37th International Conference on Machine Learning. 2020.
[20] E. C. Fieller, H. O. Hartley, and E. S. Pearson. Tests for rank correlation coefficients. Biometrika, 44(34))470-481, 1957. ISSN 00063444. URL http:://www jstor org/ Stable/233280
[21] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.

---

[22] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021.
[23] lan Goodfellow, Jean Pouget-Abadie Mehdi Mirza, Bing Xu, David Warde-Farte Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Z. Ghahramani, M. Welling,
C. Cortes, N. Lawrence, and K.Q. Weinberger, editors, Advances in Neural Information Pro- cessing Systems, volume 27. Curran Associates, Inc., 2014. URL wtw.tttsss//pceeeeeeeee http:///00/0//e0a000222200600660
neur
pdf
[24] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In Jennifer Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 1861-1870. PMLR, URL httpt::ssspppeeeeeeeeeeeee warhnngssrr..... www.m.l...r.//e.mmmmmmmmmmm press/ w00 hht.aaaaanooaaaaaaa .tmll
10-15 Jul 2018.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CoRR, abs/15122.m (bs//1512.0338,, 2015.
[26] Ruifei Hc, Shuyang Sun, Xin Yu, Chuhui Xue, Wenqing Zhang, Philip Torr, Song Bai, and Xiaojuan Qi. Is synthetic data from generative models ready for image recognition?, 2022. URL https: //axxiv...vvv org/ abs/ http:/2210.0...
[27] Todd Hester, Matej Vecerik, Olivier Pictquin, Marc Lanctot, Tom Schaul, Bilal Piot, Dan Horgan, John Quan, Andrew Sendonaris, Gabriel Dulac-Aroid, Ian Osband, John Agapiou, Joel Z. Leibo, and Audrunas Gruslys. Deep q-learning from demonstrations, 2017. URL
https://arxie org/ aataabbs/11000 03732
[28] Jacob Hilton, Jie Tang, and John Schulman. Scaling laws for single-agent reinforcement learning, 2023. URL
http://awwvv..rraaaa//./////.... 13442
[29] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrlPS 202/ Workshop on Deep Generative Models and Downstream Applications, 2021. BoLlhtttp://ppereviii
[30] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic mod- els. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 6840-6851. Curran Associates, Inc., 2020. URL ltttps://proceddii: weaediinnnnnnnuupiinee www.:neeurrrcpppaaeeee pea.rpps.cc/ppppp///0/////1 pdf
[31] Emiel Hoogeboom, Didrik Nielsen, Priyank Jaini, Patrick Forré, and Max Welling. Argmax flows and multinomial diffusion: Learning categorical distributions. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URI htt.ttttts//ooperevieee http://oooomiiieeinpppoooooo.o
[32] Vincent Huang, Tobias Ley, Martha Mlachou-Konchylaki, and Wenfeng Hu. Enhanced experi- ence replay generation for efficient reinforcement learning, 2017. URL httthttt////axiiit org/ 011/17555002245
[33] Aapo Hyvärinen. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(24):695 2005. ptsttp://jmlr httph//r.rrrgppppeeeeeeeee
hyvarinen05a.1 html
[34] Baris Imre. An investigation of generative replay in deep reinforcement learning, January 2021.
URL httt:://essayy: wtw.esayyyuwwteeee utwente.ee... n1/85772/
[35] Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. In Advances in Neural Information Processing Systems, 2019.
[36] Michael Janner, Yilun Du, Joshua Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. In International Conference on Machine Learning, 2022.
12

---

[37] Steven Kapturowski. Georg Ostrovski, Will Dabney, John Quan, and Remi Munos. Recurrent experience replay in distributed reinforcement leaming. In International Conference on Learning Representations, 2019.
[38] Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL
https://openreviit www.:/ee/ffoooeeee httt://orummmddk//uttoot
[39] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33. pages 21810-21 Curran Associates, Inc., 2020. URL.httaps://proceeddng neurips 2020/ 45
pdf
[40] Diederik P. Kingma and Max Welling. Auto-Encoding Variational Bayes. In 2nd International Conference on Learning Representations, ICLR 2014, Banff, AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014.
[41] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. In International Conference on Learning Representations, 2022. URL https
/openreview //openeevie..ne//frrrmmea 1ett/foruuniaa--8a2222ZZz
[42] Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, and Artem Babenko. Tabddpm: Mod- elling tabular data with diffusion models, 2022,
[43] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. In F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems 25, 2012.
[44] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1 1179- 1191. Curran Associates, Inc., 2020. URL httpps: inguuuurppdnng...nrrppp.....ppp nwwrips......... cc/paper/ 2020/Eiee00222211226555888 2aw.1318666555111111111166666660
pdf
[45] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 19884-19895. Curran Associates, Inc., 2020. httphtttss//pprrceeedi neurips .
[46] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems, 2020.
[47] Gen Li, Yuting Wei, Yuejie Chi, Yuantao Gu, and Yuxin Chen. Breaking the sample size barrier in model-based reinforcement learning with generative model. Advances in neural information processing systems, 33:
2020.
[48] Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating statistical overfitting. International Conference on Learning Representations, 2023. URL
wtt.://onnntttooootddddddd 1r4kkPP
[49] Cong Lu, Philip Ball, Jack Parker-Holder Michael Osborne, and Stephen J. Roberts. Revisiting design choices in offline model based reinforcement learning. International Conference on Learning Representations, 2022. URL http:////ppppeeeeeeeeeee/rrrrrrr
[50] Cong Lu, Philip Ball, Tim G. J. Rudner, Jack Parker-Holder Michael A. Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations, 2022. URL pattts://arrxe wwwrxii..or/abbe. htrgg/bb/22000 04779
[51] A.C.P.P. Ludjen. Generative replay in deep reinforcement learning, June 2021. URL. Utt.tttp! //essay utwente n1/87315/
13

---

[52] Frank J Massey Jr. The kolmogorov-smirnov test for goodness of fit. Journal of the American statistical Association, 46(253):68-78 78, 1951.
[53] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei Rusu, Joel Veness, Marc Bellemare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charles Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-33 -3,9993330000 02 2015. doi: 10.1038/nature/ 14236.
[54] N. Patki, R. Wedge, and K. Veeramachaneni. The synthetic data vault. In 2016 IEEE Interna- tional Conference on Data Science and Advanced Analytics (DSAA), pages 399-410, Oct 2016. doi: 10.1109/DSAA.2016.44
[55] Tim Pearce, Tabish Rashid, Anssi Kanervisto, Dave Bignell, Mingfei Sun, Raluca Georgescu, Sergio Valcarcel Macua, Shan Zheng Tan, Ida Momennejad, Katja Hofmann, and Sam Devlin. Imitating human behaviour with diffusion models. In International Conference on Learning Representations, 2023. URL htttp:://opeereeaiiiit http:://pppppoeeeeeeeeeerrrrraap
[56] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
[57] Ali Rahimi and Benjamin Recht. Random features for large-scale kernel machines. In J. Platt, D. Koller, Y. Singer, and S. Roweis, editors, Advances in Neural Information Processing Systems, volume 20. Curran Associates, Inc., 2007. wtt.:/eeeeee111ffadaaaaaa Paper pdff
neurips
[58] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXiv:2004006/25, 2022.
[59] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models, 2021.
[60] Mert Bulent Sariyildiz, Karteek Alahari, Diane Larlus, and Yannis Kalantidis. Fake till you make it: Learning transferable representations from synthetic imagenet clones. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[61] Stefan Schaal. Learning from demonstration. In M.C. Mozer, M. Jordan, and T. Petsche, editors, Advances in Neural Information Processing Systems, volume 9. MIT Press, 1996. URL
neurips...pppp. cc/ httppppee//9///1111/ pdf
[62] Juergen Schmidhuber. Reinforcement lcarning upside down: Don't predict rewards-juss map them to actions. arXiv preprint arXiv:1912.02287 2019.
[63] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kacz- marczyk. and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next gen- eration image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL btttp:://operre///// httt:::://ppmrrrieeeeetffforr net/forum? M3774
[64] Vikash Sehwag. Saeed Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust learning meets generative models: Can proxy distributions improve adversarial robustness? In International Conference on Learning Representations, 2022. URL https://openrevies.
[65] Hanul Shin, Jung Kwon Lee, Jaehong Kim, and Jiwon Kim. Continual learning with deep gener- ative replay. In Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neurul Information Processing Systems, volume 30. Cur- ran Associates, Inc., 2017. U /tatttts:/pproeeeeee http://///oooo....////e negriss.eurips.. wttpsecccpaccee wctp./pppppp//1///eeell Beefee8866rrccr3aaaa12222b22aaa
Paper pdf
14

---

[66] Jascha Sohl-DDckstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsu- pervised learning using nonequilibrium thermodynamics. In Francis Bach and David Blei. editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 2256-2265) Lille, France, 07-09 Jul 2015. PMLR. hRtpttps::/// httpttpps:/ppcceecinnnn///eeee/e mlr ptwsss/sw..rrreepeeepeeeeerereee - cksteen11.. html
[67] Richard S. Sutton and Andrew G. Barto. Reinforcement Learning: An Introduction. The MIT Press, second edition, 2018. URL httt:://haccompeetedeeessssss htw.://eee.eeee/eooooeeoeeo www.:eeet//b/eeeee teeeee/thebbooo book- 2nd html
[68] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offline RL Workshop: Offline RL as "Launchpad", 2022. URL https://opeereviieee htt.pperrriieee.eefffoorrm net/forum?
id=SyAS49BBcV
[69] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033. IEEE, 2012. doi: 10.1 20111099
[70] Iya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Unterthiner, Jessica Yung. Andreas Peter Steiner, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. MLP-mixer: An all-MLP architecture for vision. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. UtL.htttss://ppenrevv/ www.:nnoooounnddd-Ieeee..
[71] Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap. Nicolas Heess, and Yuval Tassa. dm control: Software and tasks for continuous control. Software Impacts, 6:100022, 2020. ISSN 2665-9638, doi: https://
1000.0 https://s://w www.:weeeeenncdreeec.c.....
con/sciience/
article/pii http://1///26666682000099999
[72] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
[73] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[74] Pascal Vincent. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):6661-1674, 2011. doi: 10.1162/NECO, a 00142.
[75] Andrew Wagenmaker and Aldo Pacchiano. Leveraging offline data in online reinforcement learning, 2022. UtL.nttt::://rriivoorr///b//
04974
[76] Lei Xu, Maria Skoularidou, Alfredo Cuesta-Infante and Kalyan Veeramachaneni Modeling tabular data using conditional gan. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buu Buc. E. Fox, and R. Gareett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL httpss://c/////eeeelll www.eeiisdnnnnn..in.......o
pdf
[77] Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. International Conference on Learning Representations, 2022. URL/atttss:/pppenrrriii
net weee/fformmid=___10 _yyes8
[78] Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 14129-14142. Curran As- sociates, Inc., 2020. URL. htttps:/prrccediings stturrppp..//ppprr///////ie 23225522eedd17722244777666666b62 Paper., .df
15

---

[79] Tianhe Yu, Ted Xiao, Austin Stone. Jonathan Tompson, Anthony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Brian lchter, Karol Hausman, and Fei Xia. Scaling robot learning with semantically imagined experience, 2023. URI.ttpss: //arxvvv..vv///a///aa org abs/ 2302. 11550
16

---

Supplementary Material
A Data Modeling
In this section, we provide further details for our data modeling. Our diffusion model generates full environment transitions i.c., a concatenation of states, actions, rewards, next states, and terminals where they are present. For the purposes of modeling, we normalize each continuous dimension (non-terminal) to have 0 mean and std. We visualize the marginal distributions over the state, action, and reward dimensions on the standard halfcheetah medium-replay replaay dataset in Figure & and observe that the synthetic samples accurately match the high-level statistics of the original dataset.
We note the difficulties of appropriately modeling the terminal variable which is a binary variable compared to the rest of the dimensions which are continuous for the environments we investigate. This is particularly challenging for "expert" datasets where early termination is rare. For example, walker2d-exper only has Rs 0.0001% terminals. In practice, we find it sufficient to leave the terminals un-normalized and round them to 0 or I by thresholding the continuous diffusion samples in the middle at 0.5. A cleaner treatment of this variable could be achieved by leveraging work on diffusion with categorical variables [31.
State
State 1
State
State
State 4
Fool
0.1
State 5
State 6
State
State
State
Seudion
-45
20
60 State
State
State 11
State 13
State 14
Synthnn
Semtic
lyntho
Sprthio
-20
-20
Store
Action
Action
mu
to m.
-10
30
-0.
-3.0
8.0
-1.0
-05
13
Holl Sprtnch
62
-2.4
Figure 8: Histograms of the empirical marginal distribution of samples from SYNTHER in blue on the halfcheetah medium-repple dataset against the original data in orange. Dashed lines indicate the mean + one standard deviation in the original dataset. SYNTHER faithfully reproduces the high-level statistics of the dataset.
17

---

A.1 Data Compression
An immediate advantage of sampling data from a generative model is compression. Table 5 we compare the memory requirements of SYNTHER and the original data by the number of 32- bit floating point numbers used by each for some sample D4RL [21] datasets. For the original data, this simply scales linearly with the size of the dataset. On other hand, SYNTHER amortizes this in the number of parameters in the denoising network, resulting in high level of dataset compression, at the cost of sampling speed. This property was also noted in the continual learning literature with generative models summarizing previous tasks [65]. As we discuss in Appendix [3.3 sampling is fast with 100K transitions taking around 90 seconds to generate.
Table 5: SYNTHER provides high levels of dataset compression without sacrificing downstream performance in offline reinforcement learning. Statisties shown are for the standard D4RL MuJoCo walker2d datasets which a transition dimension of 42, and the residual denoiser used for evaluation on these environments in Section 4.1 Figures are given to decimal place.
# FP32s in Original Dataset
# Diffusion Parameters
Dataset
Compression
mixed
12.6M
1.9x
medium medium-expert
42M
6.5M
6.5x
84M
12.9x
B
Hyperparameters
B.1 TVAE and CTGAN
In Section 3.1
we compared SYNTHER to the VAE and GAN baselines, TVAE and CTGAN. As these algorithms have not been used for reinforcement learning data before, performed hyperparameter
search 42 across the following spaces:
Table 6: Hyperparameter search space for TVAE. We highlight the default choice in bold.
Parameter
Search Space
no. layers
[1,2, 3,4]
width batch size
64, 128, 256, 512 250, 500, 1000 } 32, 128. 256
embedding dim
loss factor
{ 0.02, 0.2, 2, 20)
Table 7: Hyperparameter search space for CTGAN, We highlight the default choice in bold.
Parameter
Search Space
no. layers width batch size
1, 2, 3, 4 64. 128, 256. 512 250, 500, 1000 | [ 32, 64, 128, 256 ( 1, (1,2)
embedding dim
discriminator steps
These ranges are similar to those listed in Tables 10 and 11 of Kotelnikov et al. trials along with the default.
42
We used 30
B.2 Denoising Network
The formulation of diffusion we use in our paper is the Elucidated Diffusion Model (EDM, Karras et al. (88). We parametrize the denoising network D, MLP with skip connections from the previous layer as in Tolstikhin et al. [70] Thus each layer has the form given in Equation 3
L1+1 innear(cttivationn2 (IL)) IL
(3)
The hyperparameters are listed Table The noise level of the diffusion process encoded by Random Fourier Feature [57] embedding. The base size of the network uses width of 1024 and
18

---

depth of and thus has ~ 6M parameters. We adjust the batch size for training based on dataset size. For online training and offline datasets with fewer than million samples (medium-repp) -eplly datasets) we use batch size of 256, and 1024 otherwise.
For the following offline datasets, observe more performant samples by increasing the width up to 2048: halfchectah medium-expert, hopper medium, and hopper medium-exped This raises the network parameters to 25M, which remains fewer parameters than the original data as Table We provide ablations on the depth and type of network used in Table 10
Table 8: Default Residual MLP Denoiser Hyperparameters.
Parameter
Value(s)
no. layers width batch size
6
1024
{ 256 for online and medium-replay, 1024 otherwise
RFF dimension
16
activation
relu
optimizer
Adam
learning rate learning rate schedule model training steps
10-4
cosine annealing
100K
B.3 Elucidated Diffusion Model
For the diffusion sampling process, we use the stochastic SDE sampler of Karras et al. [38 with the default hyperparameters used for the ImageNet, given in Table We use higher number of diffusion timesteps at 128 for improved sample fidelity. We use the implementation at lttpps: github. cow wecllucidrr http:/ccdrraansdnoooiinnnng www.oiiinnndiffssioo. -ifouuson-pyyorrcc pytorch which is released under Apache license.
Table 9: Default ImageNct-64 EDM Hyperparameters.
Parameter
Value
diffusion steps
128
mminn
0.002
m7max
Satur
80
Smin
0.05
Suna
50
Spuise
1.003
The diffusion model is fast train, taking approximately 17 minutes for 100K training steps on standard V100 GPU. It takes approximately 90 seconds to generate 100K samples with 128 diffusion timesteps.
C SYNTHER Ablations
We consider ablations on the number of generated samples and type of denoiser used for our offline evaluation in Section 4.1
C.1 Size of Upsampled Dataset
we upsample each dataset (which has an original size of between 100K to 2M) to 5M. We investigate this choice for the walker medium-replay dataset in
In our main offline evaluation in Section 4.1
Figure 9 and choose 10 levels log-uniformly from the range [50K, 5M]. Similarly to He et al. [26]. we find that performance gains with synthetic data eventually saturate and that SM is a reasonable heuristic for all our offline datasets.
19

---

Size of Upsampled Dataset Ablations
100
Normalized Return
80
20
IQL (SynthER)
TD3+BC (SynthER)
10"
10"
Number of Samples (log-scale)
Figure 9: Ablations on the number of samples generated by SYNTHER for the offline walker medium medium-replay dataset. We choose 10 levels log- log-uniformly from the range [50K, 5MJ. We find that performance eventually saturates around 5M samples.
C.2 Network Ablations
We ablate the hyperparameters of the denoising network, comparing 3 settings of depth from {2, 4. and analyze the importance skip connections. The remaining hyperparameters follow Appendix [B. We choose the hopper medium-expert dataset as it is large dataset of 2MM As we can see in Table 10 we see a positive benefit from the increased depth and skip connections which leads to our final choice in Table
Table 10: Ablations on the denoiser network used for SYNTHER on the hopper medium-expert dataset. We
observe that greater depth and residual connections are beneficial for downstream offline RL performance. We show the mean and standard deviation of the final performance averaged seeds.
Network
Depth
Eval. Return
16.7
MLP
4
89.9+17.9
6
100.4 6.9
11.5
Residual MLP
4
99.3
6
101.1=10.5
D RL Implementation
For the algorithms in the offline RL evaluation in Section 4.1 Learning' (CORL, Tarasov et al. [68) codebase. We take the final performance they report for the baseline offline evaluation. Their code can be found at and is released under an Apache license.
we use the 'Clean Offline Reinforcement
com/tinkoff ai/CORL
For the online evaluation, consider Soft Actor-Critic (SAC, Haarnoja al. (24) and the implementation from the REDQ [12] codebase. This may be found at https: https://github thub. com/ watchernyu/Raan and released under an MIT license. We use the 'dmegym' wrapper for the DeepMind Control Suite [71]. This may be found at httttss//ggttuut com/ wwtlikkkssooooo/oo kestrrikoo/dmgyym dmogym and is released under an MIT license.
D.1 Data Augmentation Hyperparameters
For the data augmentation schemes we visualize in Figure La we define:
1. Additive Noise 45 adding (N 0.1) st and 8t+1-
2. Multiplicative Noise
[45 multiplying s; and 8 by single number Unif([0.8, 1.2)).
20

---

3. Dynamics Noise 8: multiplying the next state delta St+1 St by é Unif([0.5, 1.5)) so that 81+1 St ($t+1 S,).
D.2 Online Running Times
Our online implementation in Section 4.2 uses the default training hyperparameters in Appendix B.2 to train the diffusion model every 10K online steps, and generates 1M transitions each time. On the 200K DMC experiments, 'SAC (SynthER) takes x 21.1 hours compared to ~ 22.7 hours with REDQ on a V100 GPU. can further break down the running times of 'SAC (SynthER) as follows:
Diffusion training: 4.3 hours
Diffusion sampling: hours
RL training: 11.8 hours
Therefore, the majority training time is from reinforcement learning with an update-to-data ratio (UTD) of 20. We expect the diffusion training may be heavily sped-up with early stopping, and leave this to future work. The default SAC algorithm with UTD=1 takes ss 2 hours.
E Further Offline Results
In this section, we include additional supplementary offline experiments to those presented in Section 4.1
E.1 AntMaze Data Generation
We further verify that SYNTHER can generate synthetic data for more complex environments such as AntMaze [21]. This environment replaces the 2D ball from Maze2D with the more complex 8-DoF "Ant" quadruped robot, and features: non-Markovian policies, sparse rewards, and multitask data. In Table TT see that SYNTHER improves the TD3+BC algorithm where it trains (on the 'umaze' dataset) and achieves parity otherwise.
Table 11:
We show synthetic data from SYNTHER achieves at least parity for more complex offline environments like AntMaze-Var v2, evaluated with the TD3+BC algorithm. We show the mean and standard deviation of the final performance averaged seeds.
TD3+BC (22
Environment
Original
SynthER
umaze
70.8+39.2
88.9-4.4
AntMaze
medium-plan play large-play
0.3.10.4
0.5
0.0.1..0
0.0-1..0
E.2
Offline Data Mixtures
we considered exclusively training on synthetic data. We present results in Table 12 with 50-50 mix of real and synthetic data confirm that the are compatible with each other, similar to Section 4.2 We do so by including as many synthetic samples as there are real data. As we stated before, we not expect an increase performance here due to the fact that most D4RL datasets are at least 1M in size and are already sufficiently large.
In Section 1.1
Table 12:
We verify that the synthetic data from SYNTHER can be mixed with the real data for offline evaluation. The 50-50 mix achieves parity with the original data, same as the synthetic data. We show the mean and standard deviation of the final performance averaged over seeds.
TD3+BC (2
IQL 41
Environment
Original 59.0=4.9
SYNTHER
50-50
Original
SYNTHER
50-50
locomotion average
60.0
59.2+3.7
62.1+3.5
63.7
62.7+5.1
21

---

F Latent Data Generation with V-D4RL
We provide full details for the experiments in Section 4.3 that scale SYNTHER to pixel-based environments by generating data in latent space for the DrQ+BC [50] and BC algorithms. Concretely, for the DrQ+BC algorithm, we consider parametric networks for the shared CNN encoder, policy, and (Q-fuccions, fç. Te. and Qo respectively. We also use random shifts image augmentation, aug. Therefore, Q-value for a state s and action a is given by (augg())(())) a)). The policy is similarly conditioned on an encoding of an augmented image observation.
The policy and Q-functions both consist of an initial 'trunk' which further reduces the dimensionality of the CNN encoding to dieature = 50. followed by fully connected layers. We represent this as To o o and Qa Q Q This allows us to reduce a pixel-based transition low-dimensional latent version. Consider a pixel-basee based transition (s, a, s') where s, E 884x84x3 Let h fg (s))())) and h' = (aug( s))). The latent transition generate is:
(1r trnk (h), Quink a, r, mljink (M), Q (h")
This has dimension 4- dieatuue + |a| + 1 and includes specific supervised features for both the actor and the critic; we analyze this choice in Appendix [.I For example, on the 'cheetah-run' environment considered in V-D4RL, since |a] = 6, the overall dimension is 207 which is suitable for our residual MLP denoising networks using the same hyperparameters in Table This allows us to retain the fast training and sampling speed from the proprioceptive setting but now in pixel space.
To obtain a frozen encoder f& and trunks T Quink, we simply train in two stages. The first stage trains the original algorithm on the original data. The second stage then retrains only the fully-connected portions of the actor and critic, n and Q with synthetic data. Thus, our approach could also be viewed as fine-tume to-innunng the heads of the networks. The procedure for the BC algorithm works the same but without the critic.
We use the official V-D4RI. (50) codebase for the data and algorithms in this evaluation. Their code can be found at https: ///ithtub com/ com//onglulggn hwtgglll199//vevo v97 and is released under an MIT license.
F1 Ablations On Representation
We analyze the choice of low-dimensional latent representation we use in the previous section, in particular, using specific supervised features for both the actor and critic. We compare this against using actor-only or critic-only features for both the actor and critic, which corresponds to a choice of T = Q in Table 13, We note that both perform worse with an especially large drop-off for the onl features. This may suggest that non-specific options for compressing the image into low www-dimesiionl latents, for example, using auto-encoders [40, could be even less suitable for this task.
Table 13: Ablations the latent representation used for SYNTHER the V-D4RL cheetah expert dataset. We observe that separate specific supervised features are essential for downstream perfor- mance with particularly large decrease if only used critic features for the actor and critic. We show the mean and standard deviation of the final performance averaged over seeds.
Latent Representation
Eval. Return
Actor and Critic (Ours)
52.347.0
Actor Only
43.5
Critic Only
16.0+2.8
22


Reviews:
[{"review_id": "GW2NkjJcHF", "reviewer": "Reviewer_jt3y", "summary": "Building on the recent success of generative models, the authors propose Synthetic Experience Replay to improve how experience or data is used within reinforcement learning algorithms. While data is normally collected through interaction with an environment, the authors suggest that the experience replay can be augmented with additional data now artificially generated from a diffusion model. The paper considers both the online and offline settings of RL, compares with previous tabular data augmentation methods, and measures performance on pixel-based environments.", "questions": "-", "limitations": "-", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ [Quality] The authors conduct an exhaustive set of experiments on standard online and offline RL benchmarks.\n+ [Clarity] The paper is well-motivated, organized, and clearly written. Related works are carefully documented and contributions are clear.\n+ [Significance] The results are quite compelling, especially in the case of upsampling from small datasets. Considering that no algorithmic changes are necessary, many works stand to benefit from this simple, yet effective strategy.", "weaknesses": "+ [Originality] Augmenting data in the replay buffer with synthetic transitions is not new. For example, previous works have attempted to learn a dynamics model during the course of standard online RL training to fill up the replay buffer with fake transitions. Most of the novelty in this work comes from 1) using a diffusion model to generate data and 2) demonstrating experimentally that this kind of synthetic data can be useful in a variety of settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time to read our paper and provide useful comments. We now address their individual concerns.\n\n**Novelty**\n\nThank you for the question - we appreciate the reviewer’s concerns. While we agree that other methods have been proposed to generate synthetic data, the strength of our approach is in it’s simplicity. Indeed, several highly influential recent RL papers, published at similar venues, have shown that simple ideas can be both intuitive and highly effective [1,2,3].\n\nOn this point, we note that not only do we demonstrate the capacity for diffusion models to faithfully model the distribution of a replay buffer (unlike prior classes of generative models), but we also observe its utility in augmenting and expanding effective buffer size, allowing us to address issues with overfitting that can arise in RL [4,5]. This results in significant improvement over existing approaches, despite surprisingly relying on minimal underlying algorithmic modification (i.e., just train our diffusion model on the replay buffer). To speak to this point about superlative results, we observe a statistically significant improvement of SynthER over prior methods in the new experiments we ran through the RLiable framework (see Figure 2 in the supplementary PDF in the general response).\n\n[1] A Minimalist Approach to Offline Reinforcement Learning, Fujimoto and Gu, NeurIPS 2021.\n\n[2] Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels, Kostrikov et al., ICLR 2021.\n\n[3] Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learning, Yarats et al., ICLR 2022.\n\n[4] Randomized Ensembled Double Q-Learning: Learning Fast Without a Model, Chen et al., ICLR 2021.\n\n[5] Efficient Deep Reinforcement Learning Requires Regulating Overfitting, Li et al., ICLR 2023.\n\nWe once again thank the reviewer for their constructive and interesting comments, which have improved the clarity of our paper, and kindly ask them to raise their score if they believe we have addressed their concerns. If issues still remain, then we would be more than happy to discuss these.\n"}, {"review_id": "y36F6knjnt", "reviewer": "Reviewer_2GdG", "summary": "The paper proposes Synthetic Experience Replay (SYNTHER), a novel diffusion-based approach to upsample an agent's collected experience in deep reinforcement learning (RL). The authors demonstrate that SYNTHER is effective for training RL agents in both offline and online settings, and can improve performance for small offline datasets and larger networks. The paper presents results in both state-based and pixel-based environments and provides evidence that synthetic training data can enable more sample-efficient and scalable training for RL agents.\n", "questions": "1. Can you provide the experimental results of SYNTHER with a model-based RL method without BC?\n2. Can you provide the experimental results of SYNTHER in online and pixel-based setting?\n3. SYNTHER can only generate 1-step transition, so n-step bootstrap or GAE cannot be used in imagined data. Did the baselines use n-step bootstrap or GAE value estimation?", "limitations": "SYNTHER have limited results in pixel-based and online settings. This paper has no negative social impacts.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper introduces a novel approach, SYNTHER, which employ generative diffusion model to upsample an agent's collected experience. This method can potentially resolves the challenge of data scarcity in RL. Generally, this method is easy to understand and effective, and may have a broad impact on sample-efficient RL algorithms.\n2. The authors provide thorough experiments and evaluations in offline and online settings, demonstrating the effectiveness of SYNTHER in improving performance and sample efficiency. The authors also demonstrated the superiority of the diffusion-based model compared to other generative models such as GAN and VAE.\n3. The paper is well-structured, with clear introduction, background and methodology. The author also provide open-source code for reproducibility.", "weaknesses": "1. The performance improvement of SYNTHER seems to be very limited under the setting of small networks.\n2. The paper didn't demonstrate the advantages of using SYNTHER on existing RL algorithms in both online and pixel based scenarios, nor did it study the impact on model-based RL algorithms in online settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time to read our paper and provide useful comments. We now address their individual concerns.\n\n**The role of the offline experiments/small networks**\n\nWe thank the reviewer for this point; we note that the D4RL datasets are by construction relatively large and generated under stochastic policies and starting points, therefore we’d expect the benefit from SynthER to be limited in this setting as the dataset is already relatively diverse. We also note small networks more or less can solve for these environments in the default dataset setting. Finally, we view the full offline experiments as more validation that SynthER can scale to larger datasets while still faithfully capture their underlying distribution so as not to harm behavior learning.\n\n**Model-based**\n\nWe see model-based as being relatively orthogonal to our work. While SynthER could of course be deployed alongside a model-based method, it’s unclear how best to use the diffusion data, as model-based methods already produce their own “synthetic” data through imagined rollouts; do we run these two processes independently or somehow combine them? One immediate direction could be to improve the quality of a world model using SynthER, but we note that the key challenge in RL lies more with issues of over-fitting in the TD-learning loss function, and regularization is required to carefully handle this [1,2]; in this work we regularize using diffusion-based data augmentation. We would expect to see limited benefits on augmenting the already ‘stable’ supervised signal in world model learning, and moreover would be disinclined to add additional complexity and training time to the already demanding model-based RL framework.\n\n[1] Stabilizing Off-Policy Deep Reinforcement Learning from Pixels, Cetin et al., ICML 2022.\n\n[2] Efficient Deep Reinforcement Learning Requires Regulating Overfitting, Li et al., ICLR 2023.\n\n**Online pixel-based**\n\nThis is an interesting research direction, which we believe is out of scope for the purposes of this paper. A key issue is addressing multi-modality to generating high-dimensional images alongside low-dimensional actions and rewards. Whilst we present some results in the paper combining the proprioceptive architecture with a fixed latent representation, we believe this is not optimal in general for pixel-based applications, particularly as learned latent representations are likely to change over the course of training.\n\nWe tried some experiments using intermediate representations from Convolutional U-Nets to jointly diffuse actions/rewards alongside images, but initial results were not promising, particularly for modeling the low-dimensional action/reward distribution. This suggests a more complex approach is required. Given that multimodal diffusion is a significant on-going body of unresolved research, we therefore leave pixel-based experiments to future work.\n\nWe once again thank the reviewer for their constructive and interesting comments, which have improved the clarity of our paper, and kindly ask them to raise their score if they believe we have addressed their concerns. If issues still remain, then we would be more than happy to discuss these.\n"}, {"review_id": "XyiDPFmPLg", "reviewer": "Reviewer_MwZd", "summary": "The paper presents Synthetic Experience Replay, a reinforcement learning algorithm employing a generative model based on diffusion to enrich the training dataset of a learning agent. The method is adapted to offline and online RL, and compared to traditional augmentation strategies (e.g., the addition of random noise) in continuous control tasks, from proprioceptive and visual inputs, while applied to different policy optimization algorithms. The paper also presents detailed analyses on the approach, including results on the quality of the generated data and on using larger neural networks.", "questions": "I ask the author to address the concerns highlighted above in my review.", "limitations": "The computational limitations of the approach are stated in the paper, saying that the approach is computationally better than REDQ. However, the comparison is executed at a replay ratio of 20; it would be desirable to understand what is the actual computational cost of the new data generation and training of the diffusion model even at lower replay ratios.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "**Originality**: the general direction of using synthetic data for training machine learning algorithms has been vastly explored in the past, with some applications to reinforcement learning. The approach presented in the paper is, thus, not outstandingly novel. However, one of the main points in the paper is that the main bottleneck in previous approaches leveraging synthetic data in the context of RL has been the quality of the generative model. To the best of my knowledge, this is the first use of diffusion models for synthetic data generation. \n\n**Quality**: the quality of the work is reasonably good. The experiments are well-chosen and the experimental designs are generally good. \n\n**Clarity**: the writing quality is generally very high. The paper is well-written and easy to follow in most parts, with a clear presentation of the experimental results. \n\n**Significance**: I believe demonstrating the effectiveness of powerful generative models in generating synthetic data for reinforcement learning algorithms to use is important for the community, given the simplicity and generality of the approach.", "weaknesses": "**Major Concerns**\n- My main concern is on the number of seeds employed for the experiments in the paper. I understand the authors might be subject to computational constraints, but I find 4 repetitions only to be generally not enough to fully trust an individual experiment. I would suggest the authors to run more seeds per experiment and, if possible, use the methods proposed in \"Deep Reinforcement Learning at the Edge of the Statistical Precipice\" (Agarwal at al., 2021) for aggregation across multiple tasks.\n- In Section 3.1, the paper says that Tabular VAE and Conditional Tabular GAN have been evaluated using the default hyperparameters proposed in the original paper that applies these generative models to tabular data. However, no dataset from the benchmark employed in the original paper contains robotics interactions similar to the one from D4RL, and this thus begs to question of whether better hyperparameters for such different datasets might exist. I believe the paper would be improved by evaluating those approaches after a careful search for optimal hyperparameters.\n\n**Minor Concerns**\n- Some potentially important references are missing from the paper. In particular, I'm thinking of the recent paper on using synthetic experiences (meta-learned) \"Should Models Be Accurate?\" (Saleh et al., 2022), and of some recent papers applying, albeit with other goals, diffusion models to reinforcement learning such as \"Planning with Diffusion for Flexible Behavior Synthesis\" (Janner et al., 2022) and \"Is Conditional Generative Modeling all you need for Decision-Making? (Ajay et al., 2022).\n- The clarity of the paper would benefit from a more explicit description of how the diffusion model is employed for generating the data. One can guess it is naively applied to generating each dimension of each state, action or reward, but being explicit is better than being implicit in this case.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time to read our paper and provide useful comments. We now address their individual concerns.\n\n**More seeds**\n\nThank you for pointing out this oversight on our behalf. We have now increased the seeds used for all the experiments in our paper to at least 6 and show as many as possible in Figure 1 and Table 1 of the supplementary PDF. We have now also performed a meta-analysis over our empirical evaluation using the RLiable framework in Figure 2 of the supplementary PDF. As the reviewer can see, our existing findings still stand, and in fact with the additional seeds as suggested, we achieve statistically significant results over prior methods.\n\n**Tuning of other generative models**\n\nThank you for pointing out this oversight on our behalf. We have now tuned the TVAE and CTGAN models following the method of [1]. We used a similar extensive search space for the models as listed in Table 10 and Table 11 of the Appendix of [1] and used 30 trials each for the halfcheetah-mixed dataset. The results are presented in Table 2 of our supplementary PDF.\n\nWith extensive tuning, we are able to improve the performance of TVAE and CTGAN on the halfcheetah-mixed dataset. However, this still remains roughly half of what is possible with the diffusion model. We will update the final version of the paper with these results.\n\n[1] TabDDPM: Modelling Tabular Data with Diffusion Models. Akim Kotelnikov, Dmitry Baranchuk, Ivan Rubachev, Artem Babenko.\n\n**How was the data generated?**\n\nThe reviewer is absolutely correct in assuming we generate all states/actions/rewards/next states at once (e.g. joint diffusion). We discuss this in Appendix A but will include this point more explicitly in the camera ready version.\n\n**Breakdown of online training time**\n\nThank you for the question, in Appendix D.1 we list the running time of the “SAC (SynthER)” runs as 21.1 hours. We can further break this down into\nDiffusion training: 4.3 hours\nDiffusion sampling: 5 hours\nRL training: 11.8 hours\nTherefore, the majority of training time is from reinforcement learning with an update-to-data ratio (UTD) of 20. We will include this breakdown in the camera ready.\n\n**Citations**\n\nWe appreciate these pointers to references, in fact, we already cite Ajay et al. (2022) and Janner et al. (2022) in our paper (please see citation 3 and 35). However, we appreciate the pointer to Saleh et al. (2022) and will aim to discuss the paper in the camera ready.\n\nWe once again thank the reviewer for their constructive comments, which have improved the clarity and thoroughness of our paper, and kindly ask them to raise their score if they believe we have addressed their concerns. If issues still remain, then we would be more than happy to discuss these.\n"}, {"review_id": "KRsK5Kw9fR", "reviewer": "Reviewer_oBNH", "summary": "Authors propose to use diffusion models for generating new data based on online interactions or offline dataset. Generating new samples allows online and offline algorithms to perform better. Method can be useed with any offline method and any online method which utilize replay buffers. ", "questions": "Maybe I missed it, but how would  performance change in offline setting if we used original data with the synthetic one? Does it decrease performance? I don't understand the motivation to throw away original data otherwise except demonstrating that diffusion produces good data which could be an ablation.", "limitations": "I'm don't knowvery much about how complicated it is to train a diffusion model to solve an arbitary task so making it work seems like a potential bottleneck. Probably, it is faster and easier to run an additional hyperparameter search for the RL algorithm than make diffusion work.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Useful idea which works. Both offline and online RL settings  are covered as well as both visual and non-visual tasks. Good ablations and large-scale experiments. ", "weaknesses": "I  would recommend to validate the approach using Antmaze D4RL datasets as this domain is much more challenging than locomotion and maze2d.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their time to read our paper and provide useful comments. We now address their individual concerns.\n\n**Antmaze experiments**\n\nWe thank the reviewer for this comment. We would like to point out that the larger dataset offline experiments are more for validation (e.g., ensuring that diffusion models can faithfully model the distribution such that it doesn’t harm performance). As we noted in our response to reviewer 2GdG, we note that the D4RL datasets are by construction relatively large and generated under stochastic policies and starting points, therefore we’d expect the benefit from SynthER to be limited in this setting as the dataset is already relatively diverse.\n\nNevertheless, we show results for AntMaze using the TD3+BC algorithm in Table 3 of the supplementary PDF. We verify that diffusion can indeed also match the original data and even improve on the umaze dataset. We hope this provides additional confidence in our approach.\n\n**Mixture experiments**\n\nThe reviewer is right, the motivation to throw away the offline data is simply to make the point that the diffusion generated samples are high enough quality to faithfully model the original distribution, in particular when that dataset is large and diverse. However, the reviewer raises an interesting point regarding the mixture of offline and diffused data. In Table 4 of our supplementary PDF, we show that similar to the online experiments, there is no problem with mixing real and synthetic offline data in a 50-50 mix. As we discussed in the paper, the performance with the real, synthetic and mixed data are close to each other.\n\n**Ease of training diffusion**\n\nThank you for the question, we find that it is actually the exact opposite. The diffusion model uses one single set of hyperparameters for almost all experiments and requires no additional tuning. This is in stark contrast to typical RL hyperparameter sweeps. Furthermore, we unlock performance that is likely not possible even with simply sweeping different hyperparameters on the base RL algorithms.\n\nWe once again thank the reviewer for their constructive comments, which have improved the quality of our experiments and clarity of our paper, and kindly ask them to raise their score if they believe we have addressed their concerns. If issues still remain, then we would be more than happy to discuss these.\n"}]

Author rebuttal:
We would like to thank the reviewers for their helpful and valuable feedback. We were pleased that the reviewers found our paper well-written, believed that our idea was widely applicable, and agreed that our experiments were thorough and of a high standard.

We noticed that each reviewer had fairly unique concerns, but observed a shared theme regarding the need for additional experimental validation. We are excited to share improvements to our experiments in our supplementary PDF and will discuss overall improvements in the general response. Any remaining individual concerns will be addressed in the specific reviewer responses.

**More seeds**

Thank you to reviewer MwZd for raising this, we have now increased the number of seeds used in all the experiments to a minimum of 6 each. We show as many as possible in Figure 1 and Table 1 of the supplementary PDF. We see no significant change in results, which should increase confidence in the robustness and reproducibility of our experiments. 

**RLiable analysis**

As additional validation, we evaluated SynthER under the RLiable framework in Figure 2 of the supplementary PDF, where we observe statistical significance in SynthER’s improvements over the baselines.

Finally, we thank all the reviewers for their constructive and interesting comments, which have improved the clarity and experimental rigor of our paper. We take this opportunity to therefore kindly ask them to raise their score if they believe we have addressed their concerns; if issues still remain, then we would be more than happy to discuss these in the coming days.


Meta-review: This paper proposes to use diffusion models to generate synthetic data for replay-based RL algorithms. The authors show that the proposed method is effective in both offline and online RL settings.

The authors addressed all the concerns of the reviewers and reviewers increased their scores and confidence based on the rebuttal. I recommend an acceptance. I strongly recommend the authors to update the final version of the paper with more seeds as shown in the rebuttal pdf.
Decision: Accept (poster)