Prioritized Generative Replay
Authors: Renhao Wang
Kevin Frans
Pieter Abbeel
Sergey Levine
Alexei A Efros
Venue: ICLR 2025 Oral (2025)
Keywords: online learning
model-based reinforcement learning
generative modeling
synthetic data
continual learning
Primary area: reinforcement learning
TL;DR: We construct a conditional generative model of an agent's online memory, allowing us to replay high-priority data at large quantities to accelerate training of online RL agents.

Abstract: Sample-efficient online reinforcement learning often uses replay buffers to store experience for reuse when updating the value function. 
However, uniform replay is inefficient, since certain classes of transitions can be more relevant to learning. While prioritization of more useful samples is helpful, this strategy can also lead to overfitting, as useful samples are likely to be more rare. In this work, we instead propose a prioritized, parametric version of an agent's memory, using generative models to capture online experience. This paradigm enables (1) densification of past experience, with new generations that benefit from the generative model's generalization capacity and (2) guidance via a family of "relevance functions" that push these generations towards more useful parts of an agent's acquired history. We show this recipe can be instantiated using conditional diffusion models and simple relevance functions such as curiosity- or value-based metrics. Our approach consistently improves performance and sample efficiency in both state- and pixel-based domains. We expose the mechanisms underlying these gains, showing how guidance promotes diversity in our generated transitions and reduces overfitting. We also showcase how our approach can train policies with even higher update-to-data ratios than before, opening up avenues to better scale online RL agents. Project page available at: https://pgenreplay.github.io

Published as a conference paper at ICLR 2025
PRIORITIZED GENERATIVE REPLAY
Renhao Wang, Kevin Frans, Pieter Abbeel, Sergey Levine, and Alexei A, Efros Department of Electrical Engineering and Computer Science University of California, Berkeley
ABSTRACT
Sample-efficient online reinforcement learning often uses replay buffers to store experience for reuse when updating the value function. However, uniform replay is inefficient, since certain classes of transitions can be more relevant to learning.
While prioritization of more useful samples is helpful, this strategy can also lead to overfitting, useful samples are likely to be more rare. In this work, we instead propose a prioritized, parametric version of an agent's s memory, using generative models capture online experience. This paradigm enables (1) densification of past experience, with new generations that benefit from the generative model's s generalization capacity and (2) guidance via family of "relevance functions" that push these generations towards more useful parts of an agent's 's acquired history. We show this recipe can be instantiated using conditional diffusion models and simple relevance functions such as curiosity- or value-based metrics. Our approach consistently improves performance and sample efficiency in both state- and pixel- based domains. We expose the mechanisms underlying these gains, showing how guidance promotes diversity in our generated transitions and reduces overfitting. We also showcase how our approach can train policies with even higher update-to- data ratios than before, opening up avenues to better scale online RL agents."
INTRODUCTION
A central problem in online reinforcement learning (RL) involves extracting signal from a continuous stream of experience. Not only does the non-i.i.d. form of this data induce learning instabilities, but agents may lose out near-term experience that is not immediately useful, but becomes important much later. The standard solution to these problems is to use a replay buffer as a form of memory (Lin, 1992; Mnih et al., 2015). By storing dataset of transitions to enable batch-wise sampling, the algorithm can decorrelate online observations and revisit past experience later in training.
However, the idea that an agent's memory must identically reproduce past transitions, and at uniform frequency, is limiting. In general, the distribution of states an agent visits is different from the optimal distribution of states the agent should train on. Certain classes of transitions are to learning, i.e. data at critical decision boundaries or data that the agent has seen less frequently. Locating such long-tailed data is tricky, yet clearly important for efficient learning. The challenge is thus to design a scalable memory system that replays relevant data, and at large quantities.
relevant
In this work, we propose a simple, plug-and-play formulation of agent's online memory that can be constructed using generative model. We train a conditional generative model in an and-to-end manner to faithfully capture online transitions from the agent. This paradigm grants two key benefits, enabling 1) the densification of past experience, allowing us to create new training data that goes beyond the data distribution observed online, and 2) guidance via a family of "relevance functions" FF that push these generations towards more useful parts of agent's acquired experience.
An ideal relevance function should be easy to compute, and naturally identify the most relevant experience. Intuitively, we should generate transitions at the "frontier" of the agent's experience. These are regions transition space that our generative model can accurately capture from the agent's 's memory, but policy has not yet completely mastered. We investigate series of possible functions, concluding that intrinsic curiosity can successfully approximate this distribution.
Prooject page available at: https//ngenreplayygihubbii

---

Published as a conference paper at ICLR 2025
High Relevance Synthetie Transitions
Relevance Condition
Physical Memory
Generative Memory
World
Agent
Mixed Relevance Real Transitions
We model an agent's online memory using a conditional diffusion model. By conditioning on measures of data relevance, we can generate samples more useful for policy learning-
Figure 1:
Our main contribution this framework for scalable, guidable generative replay, which we term "Prioritized Generative Replay" (PGR). We instantiate this framework by making use of strong diffusion model architectures. Experiments on both state-based and pixel-based RL tasks show PGR is consistently more sample-efficient than both model-free free RL algorithms and generative approaches that do not use any guidance. In fact, by densifying the more relevant transitions, PGR able to succeed in cases where unconditional generation struggles significantly. Moreover, we empirically demonstrate that PGR goes beyond simple prioritized experience replay; particular, we show that conditioning curiosity leads to more diverse and more learning-releww relevant generations. Finally, show how PGR improves with larger policy networks, and continues learning reliably with higher synthetic-to-real data ratios, setting up promising recipe for data-efficient scaling.
2 RELATED WORK
Model-based RL. Model-based reinforcement learning involves interacting with predictive model of the environment, sometimes referred to as world model (Ha & Schmidhuber, 2018), to learn a policy (Sutton, 1991). Two classes of approaches dominate: planning with the learned world model directly (Hafner et al., 2019b; Schrittwieser et al., 2020; Ye et al., 2021; Chua et al., 2018; Ebert et al., 2018), or optimizing a policy by unrolling trajectories "in the imagination" of the world model (Janner et al., 2019; 2020; Hafner et al., 2019a; Oh et al., 2017; Feinberg al., 2018). This latter approach is most relevant to our problem setting. One distinction is we do not backprop or plan actions through model directly, but rather wholly synthesize additional, high-relevance transitions. Closely related PolyGRAD (Rigter et al., 2023) which generates entire on-policy trajectories. However, by generating independent aff-policy transitions rather than attempting to approximate trajectories from the current policy, we avoid the issue of compounding errors in our model (Gu et al., 2016), and also enable easier plug-and-play with popular existing online RL methods.
Prioritized replay. Agents often benefit significantly more by learning from important experi- ences (Moore & Atkeson, 1993; Andre et al., 1997). One of the most well-known strategies which leverages this insight is prioritized experience replay (PER), which uses temporal difference (TD) error as a priority criterion determine the relative importance between transitions (Schaul et al., 2015). Since then, number of works have proposed many different criteria to determine transition importance, including state entropy (Ramicic & Bonarini, 2017). learnability (Sujit et al., 2023), or some combination of value/reward and TD-error (Cao et al., 2019; Gao al., 2021). ). To alleviate the computational cost associated with evaluating the priority of all experiences in the replay buffer, different flavors of algorithmic improvements (Kapturowski et al., 2018; Schaul et al., 2015) or approximate parametric models of prior experience (Shin et al., 2017; Novati & Koumoutsakos, 2019) have been proposed. Our proposal to model the replay buffer parametric generative model is
2

---

Published as a conference paper at ICLR 2025
closer to this second class of works. However, distinct from these methods which seek to relabel existing data, our method uses priority to generate entirely new data for more generalizable learning.
RL from synthetic data. The use of generative training data has long history in reinforcement learning (Shin et al., 2017; Raghavan et al., 2019; Imre, 2021; Hafner et al., 2019a). But only with the recent advent of powerful diffusion models have such methods achieved parity with methods trained on real data (Janner et al., 2022; Ajay et al., 2022; Zhu et al., 2023). The use of diffusion for RL was first introduced by Diffuser (Janner al., 2022). The authors propose an offline diffusion model for generating trajectories of states and actions, and directly generate plans which reach goal states or achieve high rewards. Decision Diffuser (Ajay et al., 2022) generates state-only trajectories, and employs an inverse kinematics model to generate corresponding actions. More recently, works like Ding et (2024); He et al. (2024) leverage diffusion models to augment datasets for offline RI., improving training stability. Most similar to our approach is SYNTHER (Lu et al., 2024), which augments the replay buffer online and can be seen as an unguided form of our framework. In contrast to these previous works, we posit that the strength of synthetic data models is that they can be guided towards novel transitions that are off-policy yet more relevant for learning.
BACKGROUND
This section offers a primer on online reinforcement learning and diffusion models. Here, details on conditional generation via guidance are especially important to method exposition in Section 4.
Reinforcement learning. We model the environment as a fully-observable, infinite-horizon Markov Decision Process (MDP) (Sutton & Barto, 2018) defined by the tuple M = (S. AS,A,P, R, Po: 7). Here, S,A denote the state and action spaces, respectively. P(s') s, 3. for s, s, e and a E A describes the transition dynamics, which are generally not known. R(s, a) is reward function, Po is the initial distribution over states šg, and y is the discount function. In online RL, a policy ग S + A interacts with the environment M and observes tuples (s, a, (i.e. transitions), which are stored in replay buffer D. The action- value, or Q, function is given by:
Q=(saa)
Σ r (R(ss, ae so = s, = =0) 1-1
(1)
The Q-function describes the expected return after taking action a in state 8, under the policy m. Then our goal is to learn the optimal policy x² such that x" (a [1) argnax, Q=(,,a).
Conditional diffusion models. Diffusion models are a powerful class of generative models which employ iterative denoising procedure for generation (Sohl-Dickstein et al., 2015; Ho et al., 2020). Diffusion first involves a forward process q(x"=1 x²) iteratively adding Gaussian noise to x. starting from some initial data point n². A reverse process po p= x²") then transforms random Gaussian noise into a sample from the original distribution. In particular, we learn a neural network 6g which predicts the amount of noise NN( I) injected for some particular forward step In.
For additional controllability, diffusion models naturally enable conditioning on some signal simply by formulating the forward and reverse processes q(=:"1 x". .)) and po(==-1 >=", y), respectively. Classifier-free guidance (CFG) is a common post-training technique which further promotes sample fidelity to the condition y in exchange for more complete mode coverage (Ho & Salimans, 2022). To facilitate CFG at sampling- during training, we optimize eg with the following objective:
(2) where P Puncond is the probability of dropping condition y in favor of null condition 2. During sampling, we take a convex combination of the conditional and unconditional predictions, i.e. co (x" .m. (1- w) co(x", n, n, where is a hyperparameter called the guidance scale.
(...D,,-V(O.)...----- 1,N). Remneall Ico (2", n. (1-p) (1--p)-y+p-0-0 -
4 PRIORITIZED GENERATIVE REPLAY
In this section, we introduce Prioritized Generative Replay (PGR). At its core, PGR involves parametric generative replay buffer that can be guided by a variety of relevance criteria. We first provide intuition and motivation for such a framework, and concretize how it can be instantiated. Next, we compare and contrast between various instantiations of relevance functions. Most interestingly, we will empirically show that good default choice for sample-efficient learning is intrinsic curiosity.
3

---

Published as a conference paper at ICLR 2025
Performance Finger-uun-Harrt-a
B
1000
Curiosity-PGR (ours)
1// / WMyptelllle
800
SynthER (baseline)
Average Average Return
REDQ
600
400
200
100
200
300
Environment Steps
PGR Generations (ours)
SynthER Generations
PGR improves performance by densifying subspaces data where transitions more relevant for learning reside. We project 10K generations for both our PGR and the unconditional baseline SYNTHER to the same tSNE plot. A: At epoch 1. the distribution of data generated by PGR and SYNTHER similar. B: At the inflection point of performance near epoch 130, PGR generates distinct sub-portion of the data space from SYNTHER (i.e. red and blue dots are largely separate.)
Figure 2:
C: At the end of learning, PGR still densely covers distinct subspace of the SYNTHER transitions.
4.1 MOTIVATION
Consider agent with policy æ interacting online with some environment M. The agent observes and stores transitions (s, s',r) in some finite replay buffer D. Two main issues arise:
1. D must be sufficiently large and diverse to prevent overfitting (hard to guarantee online)
2. Transitions more relevant for updating might be rare (and thus heavily undersampled)
To address the first problem, we can densify the replay distribution (D(T) by learning generative world model using the buffer. This effectively trades in our finitely-sized, non-parametric replay buffer for an infinitely-sized, parametric one. By leveraging the generalization capabilities of, e.g., diffusion models, we can interpolate the replay distribution more impoverished regions of data.
However. uniformly sampling from this parametric buffer, in expectation, implies simply replaying past transitions at the same frequency at which they were observed. more challenging environments, there may only a small fraction of data which is most relevant to updating the current policy л. Thus, address the second problem and enable sample-efficient learning, we need some mechanism to not only densify (T()), but actually guide it towards the more immediately useful set of transitions.
Our key insight is to frame this problem via the lens of conditional generation. Specifically, we seek to model (D c), for some condition By choosing the right c, we "marginalize out" parts of the unconditional distribution po (7) which are less relevant under e (see Fig. 2.) More concretely, we propose to learn a relevance function F(T) e jointly with the policy Intuitively, this relevance function measures the "priority" с of т. Overall, we can achieve both more complete as well as more relevant coverage of pD((). Clearly, the choice of F in framework is critical. We now explore number instantiations for F, and perform an analysis on their strengths and weaknesses.
4.2 RELEVANCE FUNCTIONS
We begin by outlining two desiderata for relevance functions. First, given online setting, these functions should incur minimal computation cost. This removes conditioning on strong priors which demand extensive pretraining (e.g. prior works in offline RL (Du & Narasimhan, 2019; Schwarzer et al., 2021)). Second, we want guidance signal that will not overfit easily, thereby collapsing generation to stale or low-quality transitions even as the agent's ''s experience evolves over time.
Return. Our first proposal is one of the most natural: we consider a relevance function based on episodic return. Concretely, we use the value estimate of the learned Q-function and current policy æ:
F(s,a,8,,)) (s))
4

---

Published as a conference paper at ICLR 2025
Algorithm 1 Overview of our outer loop- + inner loop framework.
Input: synthetic data ratio re [0. 1], conditional guidance scale w Initialize: Dreat Ø real replay buffer, x agent, Deyn 2 synthetic replay buffer, C generative model, "relevance function" JF
1: while forever do
>> perpetual outer loop
2: Collect transitions Treal with π in the environment and add to Dreal
3: Update FF using Dest and Eqs. (3) to (5)
4: for T....Tddo
periodic inner loop
Sample Traal from Dreal and optimize G (T) F(r)) using Eq. (2) Conditionally generate Tayn from G with guidance scale w and add to Dayn
6:
7: Train и on samples from Dreat U Dann mixed with ratio
8: end for
9:
end while
sensibly pushes generations to be more on- on-policy, since т by construction seeks out high Q-estimate states. Also, many online RL algorithms already learn a Q-function, and we readily satisfy the first condition (Mnih et al., 2015). However, the second condition is not adequately addressed by this choice of F. particular, the diversity of high-return transitions might in practice be quite low, making overfitting to the conditional generations more likely.
Temporal difference (TD) error. Another possible choice for our relevance function is TD-error, first proposed for replay prioritization by Schaul et al. (2015). Our relevance function in this case is given by the difference between the current Q-value estimate and the bootstrapped next-step estimate:
F(s, a, 5 r) = = + Qunree arrgnax Q(5 ((d),a)) ((1,a))-Q(((..)) (s),a),
One immediate shortcoming is that in practice we have no guarantees on the Qeestimates for rarer, out-of-distribution transitions. In fact, overly greedy prioritization of high TD-error transitions can lead to low-quality, myopic Q-estimates (Schaul et al., 2015).
Curiosity. A glaring problem with both returd and TD error-based relevance functions is their reliance on high-quality Q-functions. Estimation eπrors thus lead F providing poor condi- tioning signal. Moreover, online RL agents tend to overfit Q- Q-functions to early experience (Nikishin et al., 2022), which will in turn lead to rapidly overfitted F under these two choices.
Naturally then, we might consider reducing overfitting via some relevance function which promotes generation diversity. As prior work has shown, effective way to decrease overfitting to early, noisy signal in online RL is to leverage diverse experience (Zhang et al., 2018). To achieve this diversity, we model F after exploration objectives which promote engaging with "higher-novelty" transitions that are more rarely seen (Strehl & Littman, 2008). Moreover, by learning a separate function entirely, we decorrelate our relevance function from the Q-function, making overfitting less likely.
We thus turn to prior work on intrinsic motivation (Schmidhuber, 1991; Oudeyer & Kaplan, 2007) operationalize these insights. Concretely, we take inspiration from the intrinsic curiosity mod- ule (Pathak et al., 2017) to parameterize F. Given a feature encoder h, we learn a forward dynamics model g which models the environment transition function P(s' s, s,a), in the latent space of h. Then F is given by the error of this forward dynamics model:
" F(s,a.8'.)) = (()(((s))))n(()))))
(5)
4.3 PGR FRAMEWORK SUMMARY
Finally, we provide concrete overview of our framework in Algorithm 1. In the outer loop, the agent interacts with the environment, receiving a stream of real data and building a replay buffer D Dreal- as in regular online RL. In the event that we are using curiosity-based relevance function, we also perform an appropriate gradient update for F using samples from Dreat, via the loss function given by Eq. (5). Then periodically in the inner loop, and generatively densify these transitions to obtain our synthetic replay buffer Dsyn- Concretely, we take G to be conditional diffusion model. To leverage the conditioning signal given by F.
leam conditional generative model G' of Dreat
5

---

Published as a conference paper at ICLR 2025
DMC-100k (Online)
Pixel-DMC-1008 (Online)
Quadruped- Walk
Cheetah-
Reacher- Hurd
Finger-Tune Tum- Hard*
Walker-
Cheetah-
Environment
Run
Walk
Run
MHPО
505.91 252.55
450.47 132.09
777.24 98.59
631.19 98.77
DREAMER-V3
389.63
168.47
362.01 a 30.69
807.58 a 156.38
745.27 =: 90.30
353.40 a 114.12
298. a 86.37
SAC
178.31
36.85
346.61 a 61.94
654.23
211.84
591.11 = 41.44
REDQ
496.75
151.00
606.86 99.77
733.54 + 79.66
520.53 114.88
REDQ + CURIOSITY
687.14 + 93.12
682.64 a 52.89
725.70 87.78
777.66 = 116,96
DRQ-V2
514.11 81.42
489.30 + 69.26
SYNTHER
727.01 86.66
729.35
49.59
838.60
131.15
554.01 220.77
468.53
28.65
465.09 a 28.27
PGR (Rewurd)
510,39 121.11 11
660.87
87,54
715.43 a 97.56
540.85 73.29
PGR (Retur)
737.62 +: 20.13
779.42 A 30.00
893.65 a 55.71
805.42 = 92.07
PGR (TD Emror)
802.18 116.52
704.17 96.49
917.61 37.32
839, 26 49.90
PGR (Curiosity)
927.98 25.18
817.36 35.93
915.21 48.24
885.98 67.29
570.99 41.44 529.70 27.76
Table 1: Average returns on state and pixel-based DMC after 100K environment steps (5 seeds, std. dev.
err.). * is a harder environment with sparser rewards, and so we present results over 300K timesteps.
Walker28-12
HallCheesah-v2
Hopper-v2
REDQ SYNTHER
PGR
3781 34 912.44
8612.49
407 53
3007.83
511.57
Model Size (x10f parmss))
9,86
7.12
7.39 (+3.7%)
DREAMER-V3
4104.67
349.74
7126.84
539.22
3083,41
138.90
Genemtiin VRAM (GB)
4.31
(+54.7%)
SAC
879.98 + 217.52
5065.61
467.75
2033.39
= 793.96
Train Time, hours (Difiission
1.57
1.61 (+2.5%)
REDO
3819.17
906.34
6330
85
431.47
5275.66
171..0
Genemtion Time, hours
0.62
0.63 (-1.6%)
4829.33
191.16
8165.35
1594.24
3395.21
117.50
Train Time, hours (RL)
5.69
1.98
2.11 (+6.5%)
PGR (Cariocity)
5682.33 370.04
9234.61
658.77
4101.79
244.05
Train Time, hours (Total)
5.69
4.17
4.35 (+4.3%%
Table 2: Results on state-based OpenAl gym tasks. We report average return after 100K environment steps. Results are over 3 seeds, with 1 std. dev. err.
Table 3: Model runtime and latency. PGR incurs <5% additional training time compared to SynthER. Generation also fits easily on modern GPUs (<12GB).
CFG and a "prompting" strategy inspired by Peebles et al. (2022). We choose some ratio & of the transitions in the real replay buffer Preal with the highest values for F(s, a, s', s',r). and sample their conditioning values randomly to pass to G. Implementation-winn -ise, we keep both Dreat and D at IM transitions, and randomly sample synthetic and real data mixed according to some ratio r to train our policy Here, any off- offlcolicy RL algorithm can be used to learn For fair comparison prior art, we instantiate our framework with both SAC (Haarnoja et al., 2018) () and REDQ (Chen et al., 2021).
5 EXPERIMENTS
In this section, we answer three main questions. (1) First, to what extent does our PGR improve performance and sample efficiency in online RL settings? (2) Second, what are the underlying mechanisms by which PGR brings about performance gains? (3) Third, what kind of scaling behavior does PGR exhibit, if indeed conditionally-generatee synthetic data under PGR is better?
Environment and tasks. Our results span range of state-based and pixel-based tasks in the Deep.Mnnd Control Suite (DMC) (Tunyasuvunakool et al., 2020) and OpenAl Gym (Brockman, 2016) environments. In particular, our benchmark follows exactly the online RL evaluation suite of prior work in generative RL by Lu et al. (2024), facilitating direct comparison. In all tasks, we allow 100K environment interactions, a standard choice in online RL (Li et al., 2023; Kostrikov al., 2020).
Model details, training and evaluation. For state-based tasks. in addition to the SAC and REDQ model-free baselines. we also include two strong model-based online RL baselines in MBPO (Janner et al., 2019) and DREAMER-VE v3 (Hafner et al., 2023). We also compare against SYNTHER (Lu et al., 2024), a recent work which learns an unconditional generative replay model, allowing SAC to be trained with an apdate--o-data (UTD) ratio of 20. To facilitate conditional sampling, during training we randomly discard the sealar given by our relevance function F with probability 0.25.
For pixel-based tasks, our policy is based on DRQ-V2 (Yarats et al., 2021) as in Lu et al. (2022). To maintain the same approach and architecture for our generative model, we follow Lu et al. (2024); Esser et al. (2021) and generate data in the latent space of the policy's 's CNN visual encoder. That is, given a visual encoder fe. and a transition (s, a, s', .) for pixel observations s, e 3xxhx of height h and width w, we learn to (conditionally) generate transitions (fo(s), a, fo(s)),r).
In the interest of fair comparison, our diffusion and policy architectures mirror SYNTHER exactly. Thus, training FLOPs, parameter count and generation time presented in Table 3 are all directly comparable to SYNTHER. Our only additional parameters lie in a lightweight curiosity head, which is updated for only 5% of all policy gradient steps. PGR thus incurs minimal additional overhead.

---

Published as a conference paper at ICLR 2025
Cheetah-Run
Qundruped-Wall
Cheetal-Run
Quadruped- Walk
2002
Avenue Rotum
6000
520
620
600
400
420
200
200
200
230
0
40
60
100
80
100
90
20
43
50
100
20
40
80
200
Envirenment Steps
Environment Steps
Environment Steps
Environment Steps
Curioodty-PGR (ours)
PER Curiosity
TD-Frree-PGG PGR
(ours)
TD-Emor
PER
REDO
CuriosityPPGR (ours)
SywhhRR
SynthER Curiosity
REDO
REDQ w/ Curiesity
(a) Prioritized Experience Replay (PER)
(b) Exploration Bonus
Figure 3: Comparison to baselines that use (a) prioritized experience replay (PER) and (b)
exploration reward bonuses. REDQ using PER, with priority determined by curiosity Eq. (5) or
TD-error Eq. (4), still underperform their PGR counterparts. PGR also remains superior after directly adding exploration bonus in the form of curiosity to either SYNTHER or REDQ.
1000
Cheetah-Run
1000
Wail
Walker-Wail
600
800
Avenge Return
600
600
400
400
200
200
200
0
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Reacher-Hand
Finger- Turn-Haadd
Cheetah- Run
1000
2000
600
800
800
Average Retuin
400
400
200
200
0
20
60
00
100
100
200
300
20
40
60
80
100
Environment Steps
Enviromment Steps
Environment Steps
Curiostty-PGi (ours)
MeGR (ours)
Unconditonal (SyuhER)
REDQ
SAC
DXQ-22
(a) State- Based
(b) Pixel-Based
Figure 4: Sample efficiency on DMC (a) state-based and (b) pixel-based tasks. We show mean and standard deviation over five seeds. Curiosity-PCri consistently demonstrates the best sample efficiency. SYNTHER, which uses unconditional generation to augment replay, underperforms model- free algorithms like SAC and REDQ on harder sparse-reward tasks, like finger-turn thnerrturn-har..
5.1 RESULTS
As shown in Table I and Table 2, all variants of PGR successfully solve the tasks, with curiosity guidance consistently outperforming both model-free (SAC, REDQ and DRQ-V2), and model-base -ased (MBPO and DREAMER-V -3) algorithms, as well as unconditional generation (SYNTHER). We also investigate alternative measures of curiosity such as random network distillation (Burda et al., 2018) and episodic curiosity (Savinov et al., 2018) in stochastic environments, with results in Appendix A.
Comparison to prioritized experience replay (PER). We also compare PGR to PER baselines that use different measures of priority. In particular, we train REDQ with a prioritized replay buffer, using the classic TD-error (Schaul et al., 2015), compared to our PGR approach using Eq. (4) as relevance, and the latter to using Eq. (5). As shown in Fig. 3a, PGR demonstrates superior performance in both cases. This emphasizes the importance of densifying the replay distribution with generations, and not simply reweighting past experience.
priority function based on Eq. (5). The former is
Comparison to exploration bonuses. We examine how baselines improve when given bonus in the form of an intrinsic curiosity reward (c.f. Eq. (5)). In Fig. 3b, we see that curiosity- continues to outperform SYNTHER REDQ when either is augmented this way. This suggests PGR goes
7

---

Published as a conference paper at ICLR 2025
Walker2d-v2
HulfChectab-v2
Hopper-v2
SYNTHER
4.228
0.814
0.178
PGR
6.683 (+58.19%)
0.699 (-14.1%)
0.171 (-3.9%)
Walker2ddv2
HallCheetah-w2
Hopper-v2
500
SynthoR
Sarthes
Syethes
Curlectyy POR
Cur CurlechyyPCo
Curlediyy
Number of Samples
400
400
400
300
200
200
200
100
100
100
30
10
10
10P
10a
10 10-2
10-h 10
10a
10
10-4
10-3
10-2 10
100
Dytamiis MSE (log)
Dynamics MSE (log)
Dynamics MSE (log)
Figure 5: PGR does not outperform baselines due to improved generation quality. We compute mean-squared error (MSE) of dynamics over 10K generated transitions for SYNTHER and curiosity- PGR across 3 OpenAl gym environments. Top: Average MSE. Bottom: Histograms of MSE values.
beyond just improving exploration. We provide additional evidence for this claim in Appendix B. We propose our gains are a result of generating higher novelty transitions from the replay buffer, with higher diversity. This additional diversity thereby reduces overfitting of the Q-function synthetic transitions. We provide further evidence for this idea in Section 5.2.
5.2 SAMPLE EFFICIENCY
In this section, show PGR achieves its strong performance in sample-efficient manner, and reveal the mechanisms underlying this sample efficiency. As seen in Fig. 4a, for tasks with state-based inputs, SYNTHER often attains its best performance around 100K environment steps. In contrast, cUriosity-PGR is able to match this performance in both the chectab-run and quadruped- -wakk tasks after only ~00 steps. Especially noteworthy is the finger-turn hard task, where SYNTHER actually underperforms compared to the vanilla model-free REDQ baseline, while our curiosity-PGR continues outperforming. Guidance is particularly important in sparse reward tasks, where decision-critica data is also naturally sparser. These observations hold for pixel-based tasks. As we see in Fig. 4b, while SYNTIER is eventually overtaken by DRQ-V2 in both environments, our curiosity-PGR continues to consistently improve over DRQ-V2.
Is the solution to simply replace unconditional with conditional generation? Conditional genera- tion is well-known improve sample quality in the generative modeling literature (Ho & Salimans, 2022; Chen et al., 2023). Thus, one sensible assumption is that PGR exhibits stronger performance simply because our generated samples are better in quality. We show this is not the case.
Specifically, we borrow the methodology of Lu et al. (2024) and measure faithfulness of generated transitions to environment dynamics. Given generated transition (s. as.a,'', s', rr), we roll out the action a given the current state 8 in the environment simulator to obtain the ground truth next state and reward. We then measure the mean-squared error (MSE) on these two values, comparing against generated next state s' and reward r, respectively. This analysis is performed at epoch 50 (halfway through online policy learning), over 10K generated transitions, and across different environments.
As seen in Fig. 5, SYNTHER and PGR are highly similar in terms of generation quality. Thus, our motivations for using conditional generation have nothing to do with traditional arguments in generative modeling surrounding enhanced generation quality. Generations under PGR are better because what matters is not simply quality, but generating the right classes/kinds of transitions. More generally, our core contribution is to elegantly cast widely-used prioritized replay in online RL. through this lens of conditional generation.
Moreover, naive choices for conditioning fail entirely. For example, training on larger quantity of higher reward transitions should intuitively yield high-reward policy. However, as we show in Fig. 4 and Table 1, naively conditioning on high reward (Reward-PGR) actually results in worse performance than any of the other PGR variants we propose, and in fact does worse than unconditional generation (SYNTHER). This is because greedily densifying high reward transitions does not actually provide the policy with any data on how to navigate to those high reward environment regions. Thus, for obtaining strong performance, it is important to be thoughtful in choice of F.
8

---

Published as a conference paper at ICLR 2025
10K Timesteps
30K Timesteps
0.20
0.20
Quadruped-War Walk
Probability Density
0.4
0.15
CariowtyyPnn
0.15
REDQ SynthER Curiosity-PGR
0.10
0.10
0.3
0.05
6.05
Dormant Ratio
0.2
0.00
0.00
0.05
0.50
0.15
0.20
0.00
0.0
0.5
50K
Timesteps
100K Timesteps
0.20
6.00
Probability Density
6.15
0.1
0.10
0.0
0
20
40
60
80
100
Environment Steps
2.0 Curiosity Score
Curiosity Score
(a) Dormant Ratio
(b) Curiosity Score Over Training
Figure 6: Curiosity-PGa reduces overfitting and improves diversity of replay data. (a) Dormant ratio (Xu et al., 2023) (DR) of policy networks for different approaches. DR is consistently lower for PGR, indicating a minimally overfitting policy. (b) Curiosity F-values throughout training for the unconditional baseline and PGR. We show that the distribution of states accessed by the curiosity- PGR policy is significantly shifted towards higher novelty environment transitions.
What is the mechanism underlying our performance gains? We now validate our argument in Section 4.2 for conditioning on relevance functions which mitigate overfitting. We quantify the "dormant ratio" " (DR) (Sokar et al., 2023) over the course of learning on the quadruped wwak task. DR is the fraction of inactive neurons in the policy network (i.e. activations below some threshold). Prior work has shown this metric effectively quantifies overfitting in value-based RI., where higher DR correlates with policies that execute unmeaningful actions (Sokar et al., 2023; Xu al., 2023).
As we see in Fig. 6a, the REDQ baseline exhibits high and increasing DR over training, indicating aggressive overfitting. This reflects the underperformance of REDQ on quadruped-walk, Crucially, our curiosity-PGR displays a low and stable DR, which increases only marginally (after task performance has saturated, as seen in Fig. 4.) Moreover, our DR remains consistently below that of the uncondi- tional SYNTHER baseline. This further suggests our PGR generates higher-relevance transitions with more diversity, and better addresses the issue of overfitting Qufunctions the synthetic data.
How does curiosity lead to PGR's reduction in overtitting? To show that conditioning on curiosity contributes to mitigating overfitting, we characterize the signal we obtain from IF over time. Specifi- cally, we examine the curiosity-
variant on the quadruped- -wak task, measuring the distribution of F(s, a, 8', r) using Eq. (5) over 10K real transitions. We perform this evaluation every 10K timesteps, which is the frequency at which we retrain our generative model. For comparison, we also evaluate this trained F on 10K real transitions encountered by the unconditional SYNTHER baseline.
As we see in Fig. 6b, as early 10K timesteps, immediately after the policy has been trained on the first batch of synthetic data, the distribution of curiosity values is more left-skewed for the conditional model. This distribution becomes increasingly long-tailed over training, suggesting growing diversity in the observed states as training progresses. This is contrasted by the increased biasedness of the unconditional distribution towards low-curiosity transitions. Thus, the agent trained on synthetic data from PGR is learning to move towards environment states that are more "novel" over time, improving diversity in both the real replay buffer Dreat as well as synthetic replay buffer Dayn- As further confirmation, we see that 100K timesteps, the environment is relatively well-explored, and curiosity values from F diminish, which correlates with the task reward saturation that we observe. We conclude that this is ultimately the mechanism underlying the improved sample efficiency of PGR.
5.3 SCALING PROPERTIES
Finally, we perform a series of experiments examining the scaling behavior of our PGR, in comparison to the unconditional SYNTHER baseline. For baselines, we reduce the capacity of the diffusion model in both PGR and SYNTHER such that the resultant policies attain the same performance as the model-free REDO baseline on the quadruped- walk task in DMC-100K (c.f. dashed lines in Fig. 7). We

---

Published as a conference paper at ICLR 2025
-- Basellie
Larger
---f= 0.5
r= 0.75
-r= 0.885
Baseline
UTD 40
1000
1000
800
800
800
Average Retim
600
600
600
400
400
400
200
200
200
0
40
60
0
40
20
40
60
80
100
Environment Steps
Environment Steps
Environment Steps
Curiesity- (ours)
Unconditional Baseline (SynthER))
REDO
(a) Larger Policy Network
(b) Higher Synthetic Data Ratio (c) Combined, with higher UTD
Figure 7: Scaling behavior on DMC-100K quadruped-walk, PGR can effectively combine (a)
larger networks with (b) higher ratios of synthetic data, (c) allowing us to train with a much higher UTD of 40. In comparison, the unconditional SYNTHER does not scale as well, and combining (a) and (b) actually underperforms using either independently. Runs shown are averaged over 3 seeds.
then introduce sequence of experiments - first increasing the size of the policy network, holding synthetic data constant, then increasing both the amount of synthetic data as well as how aggressively we rely this data for training. Results show that PGR leverages conditionally generated synthetic data in a more scalable fashion than SYNTHER is able to use its unconditionally generated data.
Network size. We employ the experimental protocol of SYNTHER (Lu et al., 2024): for both networks, we increase the number of hidden layers from 2 to 3, and their widths from 256 to 512. This results in ~6x more parameters, so also increase batch size from 256 to 1024 to maintain per- per-parameter throughput. We see in Fig. 7a that this improves both PGR and SYNTHER, commensurately with REDQ trained with real data. This affirms that the synthetic data is a reasonable stand-in for real data.
Amount of generative data. Next, we analyze the behavior of PGR and SYNTHER when varying the fraction of generative data used per batch. This better answers the full extent to which synthetic data can replace real data. Recall our baseline models use a batch size of 256 and a synthetic data ratio r of 0.5 (i.e. every batch has 128 real and 128 generated transitions). We now double the batch size to 512 and then to 1024, each time scaling r to 0.75 and 0.875, respectively. This keeps the number of real transitions per-batch fixed at 128, increasing only the number of generated transitions used.
We hypothesize that our generated transitions conditioned on curiosity are more effective for learning than SYNTHER'S 's unconditional synthetic transitions. Indeed, we observe Fig. 7b that at r r= 0.75, PGR continues to enjoy sample-efficient learning, whereas SYNTHER fails to improve as significantly. Surprisingly, we find that both PGR and SYNTIER fail catastrophically when the synthetic data ratio is pushed to 0.875. While useful as a mechanism to augment past online experience, further work is needed to completely supplant real environment interactions with synthetic ones.
Merging insights. Finally, we combine the above two insights, and show that this allows to push the UTD ratio of PGR to new heights. For both SYNTHER and PGR, we again use larger MLPs to parameterize the policies, and synthetic data ratio of 0.75 and batch size of 512. Finally, we double the UTD from 20 to 40, and the size of the synthetic data buffer Dayn. from 1M to 2M transitions. The idea here is to preserve average diversity of the sampled synthetic transitions. We show in Fig. 7c PGR clearly leverages both components - larger networks and more generative data- in complementary fashion to scale more effectively without additional real interactions. In contrast, SYNTHER actually degrades in performance compared with using either component independently.
6 CONCLUSION
In this work, we propose Prioritized Generative Replay (PGR): a parametric, prioritized formulation of an online agent's 's memory based on an end-to-end learned conditional generative model. PGR con- ditions on a relevance function F to guide generation towards more caaming-informatiin transitions, improving sample efficiency in both state- and pixel-based tasks. We show that it is not conditional generation itself, but rather conditioning on the correct J that is critical. We identify curiosity as good default choice for F, and decisively answer why: curiosity-PGR improves the diversity of generative replay, and reduces overfitting to synthetic data. PGR also showcases a promising new formula for scalable training with synthetic data, opening up new directions in generative RI.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like thank Qiyang Li for discussion on experimental design for Section 5.2. The authors would also like to thank Qianqian Wang, Yifei Zhou, Yossi Gandelsman and Alex Pan for helpful edits of prior drafts. Amy and Chung Min Kim also provided comments on Fig. RW is supported in part by the NSERC PGS-D Fellowship (no. 587282), the Toyota Research Institute and ONR MURI. KF supported in part by an National Science Foundation Fellowship for KF, under grant No. DGE 2146752. Any opinions, findings, and conclusions or recommendations expressed in this material are those the author(s) and do not necessarily reflect the views of the NSF. PA holds concurrent appointments describes work performed at UC Berkeley and is not associated with Amazon.
Professor at UC Berkeley and as an Amazon Scholar. This paper
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is con- ditional generative modeling all you need for decision- arXiv preprint arXiv:2211.15657, 2022. 3
David Andre, Nir Friedman, and Ronald Parr. Generalized prioritized sweeping. Advances in neural information processing systems, 10, 1997. 2
Mare Bellemare, Sriram Srinivasan, Georg Ostrovski, Tom Schaul, David Saxton, and Remi Munos.
Unifying count-based exploration and intrinsic motivation. Advances in neural information processing systems, 29, 2016. 15
Brockman. Openai gym. arXiv preprint arXiv:1606.01540 2016. 6
Yuri Burda, Harrison Edwards, Amos Storkey, and Oleg Klimov. Exploration by random network distillation. arXiv preprint arXiv:1810.12885 2018. 7, 15
Xi Cao, Huaiyu Wan, Youfang Lin, and Sheng Han. High-value prioritized experience replay for off-policy reinforcement learning. In 2019 JEEE 3/st International Conference on Tools with Artificial Intelligence (ICTA)), pp. 1510-1514. IEEE, 2019. 2
Ting Chen, Ruixiang ZHANG, and Geoffrey Hinton. Analog bits: Generating discrete data using diffusion models with celf-conditionnng The Eleventh International Conference on Learning Representations, 2023, URL http:///poorrviiiwooo/o/mmmddddd
Xinyue Chen, Che Wang, Zijian Zhou, and Keith Ross. Randomized ensembled double q-learning: Learning fast without a model. arXiv preprint arXiv:2/01.05982, 20221 6
Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine Deep reinforcement learning in handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018. 2
Zihan Ding, Amy Zhang, Yuandong Tian, and Qinqing Zheng. Diffusion world model. arXiv preprint arX5v:2402.03570, 2024. 3
Yilun Du and Karthic Narasimhan. Task-agnostic dynamics priors for deep reinforcement learning.
In International Conference on Machine Learning, pp. 1696-1705. PMLR, 2019. 4
Frederik Ebert, Chelsea Finn, Sudeep Dasari, Annie Xie, Alex Lee, and Sergey Levine, Visual foresight: Model- based deep reinforcement learning for vision-based robotic control. arXiv preprint arXiv: 18/2. 000668, 2018. 2
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVE conference on computer vision and pultern recognition, pp. 12873-12883, 2021. 6
Vladimir Feinberg, Alvin Wan, Ion Stoica, Michael Jordan, Joseph E Gonzalez, and Sergey Levine. Model-based value estimation for efficient model-free reinforcement learning. arXiv preprint arXiv: 1803.000000000 2018. 2
11

---

Published as a conference paper at ICLR 2025
Meire Fortumato, Mohammad Gheshlaghi Azar, Bilal Piot, Jacob Menick, Matteo Hessel, Ian Osband, Alex Graves, Volodymyr Mnih, Remi Munos, Demis Hassabis, Olivier Pietquin, Charles Blundell, and Shane Legg. Noisy networks for exploration. In International Conference Learning http://ppeeeeeeeeeeeefooonnn////
Representations, 2018. URL
17
Jiashan Gao, Xiaohui Li, Weihui Liu, and Jingchao Zhao. Prioritized experience replay method based on experience reward. In 2021 International Conference on Machine Learning and Intelligent Systems Engineering (MLISE), pp. 214-219. IEEE, 2021. 2
Shixiang Gu, Timothy Lillicrap, Ilya Sutskever, and Sergey Levine. Continuous deep q-learning with moseel-based acceleration. In International conference on machine learning. pp. PMLR. 2016. 2
David Ha and Jürgen Schmidhuber. World models. arXiv preprini arXiv: 2018. 2
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pp. 1861-1870. PMLR, 2018.
Danijar Hafner, Timothy Lillicrap, Jimmy Ba, and Mohammad Norouzi. Dream to control: Learning behaviors by latent imagination. arXiv preprint arXiv: 2019a. 2,
Danijar Hafner, Timothy Lillicrap, Ian Fischer, Ruben Villegas, David Ha, Honglak Lee, and James Davidson. Learning latent dynamics for planning from pixels. In International conference on machine learning. pp. 2555 2565. PMLR, 2019b. 2
Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse domains 2023. 6, 18
through world models. arXiv preprint
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang. Weinan Zhang, Dong Wang, Bin Zhao, and Xue- long Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances neural information processing systems. 36, 2024. 3
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arX1v:2207.12598.
2022. 3, 8
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-688 6851, 2020. 3
Baris Imre. An investigation of generative replay in deep reinforcement learning. B.S. thesis, University of Twente, 2021.
Michael Janner, Justin Fu, Marvin Zhang, and Sergey Levine. When to trust your model: Model-based policy optimization. Advances in neural information processing systems, 32, 2019. 2, 6
Michael Janner, Igor Mordatch, and Sergey Levine. gamma- camma-models: Generative temporal difference learning for infinite-horizon prediction. Advances in Neural Information Processing Systems, 33: 1724-1735, 2020. 2
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999) 2022. 3
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney, Recurrent experience replay in distributed reinforcement learning. In International conference on learning representations, 2018. 2
Ilya Kostrikov, Denis Yarats, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. arXiv preprint arXiv:2004.13649 2020. 6
Qiyang Li, Aviral Kumar, Ilya Kostrikov, and Sergey Levine. Efficient deep reinforcement learning requires regulating overfitting, arXiv preprint arXiv:2304.10466, 2023,
Long-Ji Lin. Self-improving reactive agents based on reinforcement learning, planning and teaching. Machine learning, 8:293-321, 1992.
12

---

Published as a conference paper at ICLR 2025
Cong Lu, Philip J Ball. Tim GJ Rudner, Jack Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint arXiv:2206. 20V.::2200004779, 2022. 6
Michael A Osborne, and Yee Whye Teh.
Cong Lu, Philip Ball, Yee Whye Teh, and Jack Harker-Holder Synthetic experience replay. Advances in Neural Information Processing Systems, 36, 2024. 3, 6, 8, 10
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 518(7540):529555 533,, 2015. 1,5
Andrew W Moore and Christopher G Atkeson. Prioritized sweeping: Reinforcement learning with less data and less time. Machine learning, 13: 13:103-130 1993. 2
Evgenii Nikishin, Max Schwarzer, Pierluca D' D'Oro, Pue Bacon, and Aaron Courville. The primacy bias in deep reinforcement learning. In International conference on machine learning, pp. 16828-16847. PMLR, 2022.
Guido Novati and Petros Koumoutsakos. Remember and forget for experience replay. In International Conference on Machine Learning, pp.
PMLR, 2019. 2
Junhyuk Oh, Satinder Singh, and Honglak Lee. Value prediction network. Advances neural information processing systems, 30, 2017. 2
Ian Osband, Charles Blundell, Alexander Pritzel, and Benjamin Van Roy. Deep exploration via bootstrapped dqn. Advances in neural information processing systems, 29, 2016. 17, 18
Pierre-Yves Oudeyer and Frederic Kaplan. What is intrinsic motivation? a typology of computational approaches. Frontiers in neurorobotics, 1:108, 2007.
Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In International conference on machine learning, pp. 2078-277777 PMLR, 2017. 5, 15, 17
William Peebles, Ilija Radosavovic, Tim Brooks, Alexei A Efros, and Jitendra Malik. Learning to learn with generative models of neural network checkpoints. arXiv preprint arXiv:2209.1289) 2022. 6
Aswin Raghavan, Jesse Hostetler, and Sek Chai. Generative memory for lifelong reinforcement learning. arXiv preprint 1rXiv:1902.08344 2019.
Mirza Ramicic and Andrea Bonarini. Entropy-basee prioritized sampling in deep q-learning. In 2017 2nd international conference image, vision and computing (ICIVC), pp. 1068-1072 IEEE, 2017. 2
Mare Rigter, Jun Yamada, and Ingmar Posner. World models via policy-guided trajectory diffusion. arXiv preprint arXiv:23/2.085.. 2023. 2
Nikolay Savinov, Anton Raichuk, Raphaël Marinier, Damien Vincent, Mare Pollefeys, Timothy Lilli- crap, and Sylvain Gelly. Episodic curiosity through reachability, arXiv preprint arXiv:1810.02274. 2018. 7, 16, 17
Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay. arXiv preprint xrXiv:1511.05952, 2015. 2, 5,7
Jürgen Schmidhuber. A possibility for implementing curiosity and boredom in model-building neural controllers. Proceedings of the First International Conference Simulation of Adaptive Behavior, 1991. 5
Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert, Karen Simonyan, Laurent Sifre, Simon Schmitt, Arthur Guez, Edward Lockhart, Demis Hassabis, Thore Graepel, et al. Mastering atari, go, chess and shogi by planning with a learned model. Nature, 588(7839):604-66 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347, 2017. 16
13

---

Published as a conference paper at ICLR 2025
Max Schwarzer, Nitarshan Rajkumar, Michael Noukhovitch, Ankesh Anand, Laurent Charlin, R De- von Hjelm, Philip Bachman, and Aaron C Courville. Pretraining representations for data-efficient reinforcement learning. Advances in Neural Information Processing Systems, 34: 34:12666-1269,, 2021. 4
Hanul Shin, Jung Kwon Lee, Jachong Kim, and Jiwon Kim. Continual learning with deep generative replay. Advances in neural information processing systems, 30, 2017. 2,
Pranav Shyam, Wojeiech Jaškowski, and Faustino Gomez. Model-basse based active exploration. In International conference machine learning, pp. 5779-5788, PMLR, 2019. 18
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning, pp. 2256-2265. PMLR, 2015. 3
Ghada Sokar, Rishabh Agarwal, Pablo Samuel Castro, and Utku Evci. The dormant neuron phe- nomenon in deep reinforcement learning. In International Conference on Machine Learning, pp. 32145-32166 PMLR, 2023.
Alexander I. Strehl and Michael Littman. An analysis of model-based interval estimation for markov decision processes. Journal of Computer and System Sciences, 7408):1309-1331, 2008. 5, 15
Shivakanth Sujit, Somjit Nath, Pedro Braga, and Samira Ebrahimi Kahou. Prioritizing samples in reinforcement learning with reducible loss. Advances in Neural Information Processing Systems, 36:23237-23258, 2023. 2
Richard S Sutton. Dyna, an integrated architecture for learning, planning, and reacting, ACM Sigart Bulletin, 164):160 1991.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Saran Tunyasuvunakool, Alistair Muldal, Yotam Doron, Siqi Liu, Steven Bohez, Josh Merel, Tom Erez, Timothy Lillicrap, Nicolas Heess, and Yuval Tassa. dm_control: Software and tasks for continuous control. Software Impacts, 6: 6:100022, 2020.
Guowei Xu, Ruijic Zheng, Yongyuan Liang, Xiyao Wang, Zhecheng Yuan, Tianying Ji, Yu Luo, Xiaoyu Liu, Jiaxin Yuan, Pu Hua, et Drm: Mastering visual reinforcement learning through dormant ratio minimization. arXiv preprint arXiv:23100 19668, 2023.
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. arXiv preprint arXiv:2107.0964 2021. 6
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering atari games with limited data. Advances in neural information processing systems, 34:25476-25488 2021.
Chiyuan Zhang, Oriol Vinyals, Remi Munos, and Samy Bengio. study on overfitting in deep reinforcement learning. arXiv preprint 18Xiv:1804.0663 2018.
Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: survey, arXiv preprint arXiv:2311.01223. 2023. 3
14

---

Published as a conference paper at ICLR 2025
A ADDITIONAL RELEVANCE FUNCTIONS AND THE NOISY-TV PROBLEM
We show that PGR can instantiated using a variety of other relevance functions F, solidifying the claim that PGR is framework level contribution where can flexibly condition on wide range of F. In particular, we examine other functions which describe curiosity-based metrics that are more robust than what is offered by the prediction error- based intrinsic curiosity of Pathak et al. (2017).
A.1 CONDITIONING ON OTHER RELEVANCE FUNCTIONS
We begin by expanding the pixel-based DMC-100K benchmark in Table to include two additional results: conditioning error obtained via random network distillation (Burda et al., 201 8), and on pseudo-counts obtained via a context tree switching switching density model (Bellemare et al., 2016).
Random Network Distillation (RND). RND involves randomly initializing two identical neural networks to embed observations: fixed target f network and trainable predictor network fo (Burda al. 2018). The parameters of f are updated via minimizing the MSE with respect to the target network outputs. This optimization error then becomes exactly the relevance function we adopt.
Specifically, we parameterize the neural networks three-layer CNNs, with bottleneck latent dimension 64 and feature output dimension 512. The CNNs are followed by a two-layer MLP projection, also of dimension 512. The relevance function can then be described as:
F(s,a,s',r) = (
(6)
Other training and model details, as well as the environment details specific to pixel-based tasks, follow our description in Section 5.
Context-Trre Switching (CTS) Density Model. Pseudo-counts estimate the novelty of a given state via the frequency of visits to this state (Bellemare al., 2016). Following Theorem I of Bellemare et al. (2016), and the prior work of Strehl & Littman (2008), we set our relevance function as:
F(s,a,
+0.01)
where the learned pseudo-count (s,a) has closed form per Equation in Bellemare et al. (2016). Specifically, Ñ depends only learned density model p over observations in state-action space. Following prior work, we implement p(s, a) as a context tree switching (CTS) density model (Bellemare et al., 2016). We resize the visual observations to 42 42 pixels, and use context bins. As with our RND variant, remaining training, task and model details follow our description in Section 5.
Conclusion: As observed in Table 4, PGR can effectively condition on other relevance functions to bring about similar benefits on our dixel-based online DMC- 100K benchmark. However, one reasonable remaining question is how PGR might fare in more complex environments. Of particular interest are environments rife with stochastic transitions. which first motivated works like Burda et al. (2018) 8) and Bellemare et al. (2016).
Walker Walk
Cheetah-Run
DRQ-V2 SYNTHER
514.11 + 81.42
489.30 + 69.26
468.53 + 28.65
465.09 + 28.27
PGR (RND)
602.10 + 43.44
512.58 + 23.81
PGR (CTS)
540.78 + 88.27
508.17 + 74.03
PGR (CURIOSITY)
570.99 + 41.44
529.70 + 27.76
Table 4: Average returns on pixel- based DMC after 100K environment steps (5 seeds, 1 std. dev. err.). We now include results using relevance functions based on RND and pseudo-counts.
15

---

Published as a conference paper at ICLR 2025
115
98
124
97 B 121
Figure 8: Random RGB observations from the randomized DMLab environment. Stochastic noise is added to the lower right quadrant, i.i.d across pixels and timesteps.
A.2 ADDRESSING THE NOISY-TV PROBLEM
We turn environment where literal noisy TV is present. In particular, examine the randomized DMLab environments presented in Section S6 of Savinov et al. (2018). These environments consist of procedurally generated minigrid 3D mazes, with 9 discrete actions available at a repeat frequency of 4. Observations are first-person 84 x 84 RGB images of the agent's current view within the maze. Reward sparse (+10 for reaching the goal. at all other times), and the agent is required to reach the goal as many times as possible within 1800 environment steps. We consider two variants of this task: "Sparse" the default environment just described, and "Very Sparse" is harder version where same-room initializations that falsely give immediate reward are removed.
Importantly, stochasticity observations injected by replacing the lower right 42 42 pixels of the agent's 's view with noise sampled uniformly from [0, 255], independently for each pixel. A visualization of this input is available Fig.
We reproduce many of the same baselines as in Savinov et al. (2018), and also introduce our own. Namely, base agent trained using a vanilla PPO implementation (Schulman al., 2017). We also introduce additional variants which augment the extrinsic reward with intrinsic curiosity (either ICM or RND), as well episodic curiosity (ECO) introduced by Savinov et al. (2018). Intrinsic bonus scales, learning rates and other hyperparameters follow Table S3 of Savinov et al. (2018).
We also adapt PGR to the DMLab environment. For fair comparison, we swap out the model-free REDQ backbone with PPO, but preserve the residual MLP demoising diffusion model described in Section 5. We use relevance functions based on ICM (cff. Eq. (5)) and RND (c.f. Eq. (6)).
Combining PGR with ECO. Crucially, also show that PGR can condition derived from ECO, with no hyperparameter tuning. ECO operates by characterizing novelty via reachability that is, particular state is novel if and only if it cannot be reached within some actions of observations in a memory buffer M. To estimate reachability, ECO embeds observations using an embedding network E' and compares them other embeddings within M using comparator C, trained via logistic regression loss. The output of C is close to I if the probability of two observations being "close" each other is high, and otherwise. Given these elements, we define our relevance function via Equation 3 in Savinov et al. (2018):
relevance function
a FL F(+))) 4s, e M
(8)
Sparse
Very Sparse
PPO
7.3 2.7
4.1 + 2.4
PPO + CURIOSITY
5.6 1.8
2.8 + 1.0
PPO + RND
8.2
4.0 + 1.1
PPO + ECO
16.3
12.9 + 1.9
PGR (CURIOSITY)
9.3
5.7 + 2.2
PGR (RND)
11.2 1.0
8.0 + 1.8
PGR (ECO)
21.9 + 2.6
18.7 2.1
Table 5: Average returns randomized DMLab after 10M env. steps (10 seeds, 1 std. dev. err..
16

---

Published as a conference paper at ICLR 2025
We directly use the recommended hyperparameters in Savinov et al. (2018), setting a 0.03, 3 0.5, [M] = =00, and F percentile-99 The embedder network E is a ResNet-18 architecture with output dimension 512, followed by a four-layer MLP, also with feature and output dimensions of 512.
Conclusion: As observed in Table 5, PGR can flexibly condition on episodic curiosity to obtain strong performance in environments with highly stochastic transitions. Crucially, the generative replay PGR variants consistently outperform their model-free PPO counterparts which directly add exploration bonuses.
B COMBINING EXPLORATION BONUSES AND PGR
Our core contributions, framework for prioritized generative replay and exposition into why one particular instantiation works, are orthogonal to exploration. However, in Appendix A and also Section 5.1 of the main text, show that PGR always outperforms exploration bonus variants. This raises an interesting question: how might exploration bonuses be combined with PGR?
Quadruped- Walk
Cheetal-Run
Quadruped- -aall
Cheetah-Run
REDQ SYNTHER
496.75 151.00
606.86
99.77
NOISYNETS
688.29 65,55
770.14 70.53
727.01 = 86.66
729.35 49.59
BOOT-DQN
721.49 39.82
754.56 67.38
REDQ CURIOSITY
687.14 93.12
682.64
52.89
PGR (NOISYNETS)
939.32
36.74
893.67 43.47
SYNTHER CURIOSITY
803.87 41.52
743.39
47.60
PGR (BOOT-DON)
903.29 40.50
912.65 47.22
(CURIOSITY)
927.98 25.18
817.36 35.93
(CURIOSITY)
927.98 25.18 817.36 35.93
Table 6: Explicit Exploration Bonuses. Results on state-based DMC-100K seeds, 1 std, dev. emr.).
Table 7: Implicit Exploration Bonuses. Results on state-based DMC-100K (3 seeds, std. dev. emr.).
B. EXPLICIT EXPLORATION BONUSES.
We reiterate our results in Section 5.1 which demonstrate that PGR outperforms baselines augmented with exploration bonus. Specifically, examine the model-free baseline (REDQ), well as the unconditional generative replay baseline (SYNTIEER), when either is given an intrinsic curiosity bonus, learned via the same intrinsic curiosity module (c.f. Eq. (5) and Pathak et al. (2017)). We follow the hyperparameters of Pathak et al. (2017) and set the intrinsic reward weight to 0.1. Results in Table show that PGR continues to enjoy healthy lead over such explicit exploration bonuses.
B.2 IMPLICIT EXPLORATION BONUSES.
Next, we study how exploration bonuses can be involved more implicitly. In particular, we consider modifying the model class of the underlying policy to better facilitate exploration. We choose two well-established methods along this line of work noisy networks (Fortunato et al., 2018) and bootstrapped Q-value exploration (Osband et al., 2016).
Noisy Networks. As baseline, we replace the Q networks used in our REDQ model-free baseline with noisy Q networks from Fortunato et al. (2018). That is, each linear layer is transformed from:
y wir b,
for output y, and learnable weights and biases and b, respectively, to a "noisier" version:
y
>x+ + p² e.
(10)
Here, the noise parameters and a are learnable, whereas e are sampled noise random variables. Initialization of μ. a and e all follow the recommendations Section 3.2 Fortunato et al. (2018) exactly.
Bootstrapped V-luusss. We also compare to the work of Osband et al. (2016), where uncertainty is injected via a bootstrapping mechanism across multiple Q-networks. In particular, suppose we have an ensemble of K Q-networks, Given an episode, we select a single Q-value network to act through, and sample binary mask of length K according some distribution M. This mask describes the subset of Q-networks which are updated using this transition. This promotes exploration by preserving temporally consistent estimate of uncertainty independently for each of Qı,..., Q
17

---

Published as a conference paper at ICLR 2025
This setup fits naturally with the Q-ensemble used in our REDQ baseline and can thus be readily integrated. We follow Osband et al. (2016) and simply use an all-one mask (i.e. masking distribution M of Bernoulli())), which corresponds to basic ensemble of all Q heads.
PGR Variants. To combine PGR with NOISYNETS, we only replace linear layers used in the policy (i.e. the Q-networks involved in the REDQ baseline.) That is, for the sake of minimal comparison, the embedding layers within the intrinsic curiosity- based relevance function are not modified. To combine PGR with BOOT-DQN, we make similar choice.
Taken together, Table 6 and Table demonstrate that PGR is empirically orthogonal to, and more useful than, simple exploration bonuses. In particular, explicit exploration bonuses do not allow baselines to outperform PGR, even when PGR is using the cxact same bonuses to parameterize a relevance function. But implicit exploration bonuses can be readily integrated with the PGR framework, and may indeed bring complementary benefits, as we observe in the cheetah-run environment of Table 7. However, neither DMC- 100K nor OpenAl Gym environments are well-suited to investigate exploration problems, and hence avoid making any claims in the main text regarding exploration. We leave this intersection open as an interesting area for future work.
Conclusion:
C MODEL-BASER RL BASELINES WITH NOISY DYNAMICS
In this section, we show that PGR is substantially different from model-based RL algorithms which generate trajectories through a learned dynamics model.
Our central argument is that under imperfect or noisy observations, PGR is more robust than methods which generate trajectories or transitions directly via a Iearned dynamics model. To verify this claim, evaluate PGR against two such model-based baselines: MAX (Shyam et al., 2019) and DREAMER-VS (Hafner et al., 2023). Crucially, we noise the observations within the training data to each model's learned dynamics components. For PGR, we add isotropic Gaussian noise to the current state and the next state s' of 20% of the transitions within each batch to the ICM. For MAX, we similarly modify 20% of the transitions within the exploration data for training the model ensemble. For fair comparison, we also swap out the SAC backbone in MAX in favor of the more performant REDQ policy. Finally, for DREAMER- v3, we pollute states in transitions given to the recurrent state-space world model with the exact same noise and at the exact same rate as for PGR.
Practically, we found much easier to convert MAX to the DMC benchmark, rather than convert DREAMER-V3 to the OpenAl Gymnasium benchmark that MAX originally presented results on. We choose a subset of the DMC tasks which correspond most closely with their OpenAl Gym counterparts, arriving at the cheetah-run, walker-walk and hopper-hop tasks presented in Table 8. Following our setup details Section 5, we again allow for 100K online environment interactions.
Cheetah-Run
Walker-Walk
Hopper-Hop
MAX
644.79 + 63.90
509.63 48.76
48.30 + 16.56
Original
DREA CREAMER-Y. .V3
362.01 # 30.69
627.79 41.53 865.33
86.31 # 21.21
PGR (CURIOSITY)
817.36 + 35.93
75.39
94.45 + 12.07
MAX
363.26 + 58.86
421.06
33.77
17.07 + 9.82
Noised
DREA MER-V
199.74 34.60
386.13
59.71
35.42 + 18.03
PGR (CURIOSITY)
697.67 # 29.94
734.02 48.57
64.45 + 10.86
Table 8: Average returns on original and noisy state-based DMC-100K (5 seeds, std. dev. err.). We deliberately noise the actions seen during training by the dynamics models for various algorithms. PGR demonstrates more robustness to imperfect dynamics models.
As we observe from Table 8, PGR significantly outperforms both model-based algorithms which plan/generate trajectories directly through their learned dynamics. While such algorithms deeply couple the learning of environment dynamics and the policy, PGR offers an alternative and weaker coupling. In particular, PGR uses the ICM to learn transition probabilities p(s'ls, a) (i.e. environment dynamics). The prediction error e from the ICM then in turn serves as a conditioning signal to a diffusion model which learns p(s, a, s', r e). Thus, even if the ICM is poorly trained due to imperfect input states, the diffusion model will still generate transition tuples (s, a, s' which adhere well to the true environment dynamics.
18

---

Published as a conference paper at ICLR 2025
Conclusion: PGR offers distinct design choice from model-based RL algorithms which plan directly through a generative model. We empirically validate that this distinct design choice can be advantageous when dynamics predictions imperfect due to noisy observations.
D MISCELLANEOUS QUANTITATIVE EVALUATION
For comparison, Table 9 presents quantitative readouts of the PER baselines initially shown in Fig. 3a.
We also address questions regarding how PGR might be compatible with different generative model classes. In particular, we replace the underlying diffusion model in SYNTHER and PGR with an unconditional and conditional VAE, respectively. We design our VAE to approximate the capacity of the diffusion model in both SYNTHER and PGR (~6.8 million parameters). Specifically, use a residual net encoder and decoder with 4 and 8 layers, respectively. Each layer has bottleneck dimension 1 128, with latent dimension of 32, resulting in 2.6M encoder and 5.3M decoder parameters.
Results on three tasks (state-based DMC-100K) are shown in Table 9. We conclude that a) indeed conditional generation continues to outperform unconditional generation, but b) overall performance much lower than either SYNTHER or PGR, and arguably within variance of the REDQ baseline. We believe this performance gap well justifies the focused use of diffusion models.
Quadruped- Walk
Cheetah-Run
Reacher-Hard
REDQ
496.75 151.00
606.86 99.77
733.54 + 79.66 838.60 = 131.15
SYNTHER
727.01 = 86.66
729.35 = 49.59
PER (TD-ERROR)
694.02 = 99.17
685.23 63.76
810.37 = 89.22
PER (CURIOSITY)
726.93 = 71.59
627.68 = 55.50
763.21 + 52.29
UNCOND. VAE
384.38 # 154.30
549.36
190.79
700.65 + 161.18
COND. VAE
501.99 = 79.88
668.49 76.81
792.85 93.73
PGR (Curiosity)
927.98 + 25.18
817.36 + 35.93
915.21 +: 48.24
Table Further results on DMC-100K. We include prioritized experience replay baselines from Figure 2a of the main text for easier comparison. We also show PGR with different generative model classes, such as variational autoencoders (VAEs).
Finally, we show how chose the frequency of the inner loop in Algorithm
In Fig. 9, we plot the performance of PGR on the hopper-stand environment as a function of the frequency of the inner loop. We show that performance increases modestly the more frequently the inner loop is performed. This is intuitive - grounding the diffusion model more regularly in the real transitions allows it to better generate on-policy transitions, improving the stability of policy learning.
To tradeoff effectively between diminishing returns and training time, we use heuristic elbow method, arriving at regenerating the diffusion buffer once every 10K iterations as "optimal."
Hopper
1000
900
800
Avenge RAtzS
30000
20003
30000
40000
50000
Stepd)
Figure 9: PGR improves with increased frequency of the generative inner loop in Algorithm 1. To tradeoff efficiency and performance, we select the inner loop to run periodically every 10K iterations.
19


Reviews:
[{"review_id": "h0XUtHBAKU", "reviewer": "Reviewer_1k48", "summary": "The work proposes a form of sample based experience replays that leverages a generative model to provide and augment samples drawn from the replay buffer. To avoid overfitting, a set of guidance functions are used to steer the generative process toward diverse and useful samples. The generative replay mechanism is a diffusion model that is conditioned on some auxiliary information. The authors propose a few different versions of this conditioning such as intrinsic curiosity, TD error, or Q values. The idea is that using these scores, the generative model can be steered towards generating high quality samples. Given such a replay mechanism, this work evaluates model free and model-based RL agents trained via this generative replay on gym and dmc.The results show improvement on both pixel based and state based tasks. There are also ablations with larger policy networks and higher generative data rations, which show further improvements.\n\n-------------------------------------------------\nI thank the authors for a substantive rebuttal that addressed my and (as far as I can tell) other concerns. I therefore raise my score to an 8.", "questions": "I'll rephrase my above concerns as questions. \n\n1. How is this method novel with respect to prior work that uses intrinsic rewards on rollouts from a learned dynamics model? It seems like a very similar approach to acquiring data that scores well under a given guidance function F, where F can be ICM or another intrinsic reward. \n\n2. How does this method handle noisy-tvs?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work proposes a scalable method for training model-free or model-based agents in a variety of domains. I believe the formulation is simple enough to be integrated into and improve other approaches. \n\n* I also found the presentation clear and easy to read. \n\n* I found the scaling experiments to be very compelling, I'm a little concerned about the general thrust of driving up the syn-real data ratio as high as possible, since we do need to ground the generations in real experience. But I still think insights here are valuable.", "weaknesses": "I have two points of contention with this work. \n1. From a paradigm perspective, I don't understand how this is different from prior work in model-based RL that apples intrinsic rewards to a learned dynamics model [1] or world-model [2]. These methods also utilize a generative model as a copy of the environment, then train the agent in simulation to acquire interesting data (under the intrinsic reward). It seems that this method does the same, except that instances, rather than full trajectories are generated. I do see how this is different than just applying an intrinsic bonus during training, since here the synthetic data has a chance to be more diverse. \n\n2. I thank the authors for providing numerous experiments, but I am not at all convinced that this method is robust to the choice of guiding function F. ICM is known to be susceptible to the noisy TV problem, where difficult-to-model environmental factors score arbitrarily high under ICM. The chosen tasks are too simple perceptually to see this problem. This in and of itself is not a problem, but it means that we need to search for another F that works for our task which is hard in practice. In the meantime, there are other intrinsic rewards that do not suffer from this pathology [3]. \n\n\n\n[1] Shyam, Pranav, Wojciech Jaśkowski, and Faustino Gomez. \"Model-based active exploration.\" International conference on machine learning. PMLR, 2019.\n\n[2] Hafner, Danijar, et al. \"Dream to control: Learning behaviors by latent imagination.\" arXiv preprint arXiv:1912.01603 (2019).\n\n[3] Savinov, Nikolay, et al. \"Episodic curiosity through reachability.\" arXiv preprint arXiv:1810.02274 (2018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iTXRe82jKN", "reviewer": "Reviewer_Yyu8", "summary": "This paper proposes a conditional diffusion model as a synthetic replay buffer to combat early overfitting in online reinforcement learning and to diversify experiences for the learning model. This is achieved with a relevance function that selects samples that are rare but important for learning based on the temporal difference error, the value of a learned Q-function and a modified intrinsic curiosity module.", "questions": "What exploration method does the agent use?\nCould the exploration method be improved instead of the sample generation to improve diversity of samples?\nWould a combination of both a better exploration and this method be the optimal and a possible solution?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "One of the strength of this paper are the clear and concise language as well as good structured presentation of the proposed method.\nIt is quite logical to improve on the already existing prioritized experience replay method and implement it in the generative domain. The method is explained well and should be quite easily reproducable.\nOverall the research could be a valuable contribution to the reinforcement learning community.", "weaknesses": "A topic i feel like missed somewhat are the different ways to approach generative replay such as mentions of other generative models (e.g. variational auto encoders, gaussian mixture models) and why they were not used.\nOne thing i found rather off putting and this is very nitpicky is that the Tables 1, 2 and 3 are a bit crammed and slightly off from each other.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "knTLqMoytb", "reviewer": "Reviewer_ir9Q", "summary": "The paper proposes to use conditional diffusion models to improve the experience replay for an RL learning agent. The method proposed improves performance by improving the diversity of the samples in the experience replay buffer and reducing overfitting.", "questions": "1. Is the method compatible with different kinds of exploration bonuses? If so, how do you think they would compare?\n2. How do you think the method would do when simply having diverse samples does not imply usefulness? An example is the noisy tv problem.\n3. How sensitive is the algo towards the frequency of the inner loop in Algo 1?\n4. Can multiple relevance functions be combined?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and provides a clear explanation of their method.\n2. The research problem addressed in the paper is well laid out and is an important one to improve the performance of RL methods.", "weaknesses": "1. While the method shows improved performance, it is a bit simple as it combines existing elements in diffusion models and RL to propose the solution.\n2. It would be useful to compare the effect of different kinds of exploration bonuses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BB1f0dVd4V", "reviewer": "Reviewer_8yrp", "summary": "This paper introduces a framework called Prioritized Generative Replay (PGR), a novel approach to enhance sample efficiency in online reinforcement learning (RL). Traditionally, replay buffers store experienced transitions and replay them uniformly or with prioritization based on metrics like TD-error. However, the authors point out that uniform replay can be inefficient, and prioritization can lead to overfitting. PGR addresses these issues by using a conditional generative model to create a parametric replay buffer. \n\nThe paper claims that this allows for two key advantages:\n1) Densification: The generative model can create new, plausible transitions beyond those directly experienced, enriching the training data, especially in sparsely explored regions.\n2) Guidance: By conditioning the generative model on \"relevance functions,\" the generated transitions can be steered towards areas more critical for learning, such as states with high novelty or uncertainty.\n\nThe authors also explore various relevance functions, including return, TD-error, and curiosity. They find that curiosity, based on the prediction error of a learned dynamics model, performs best. This is attributed to its ability to promote diversity in the generated transitions, thus reducing overfitting. They also show that their approach consistently improves performance and sample efficiency in both state- and pixel based domains.", "questions": "1) How robust is PGR to errors in the learned dynamics model? Are there ways to mitigate the impact of inaccurate dynamics predictions on the curiosity-based relevance function?\n2) Could PGR be extended to offline RL settings? If so, what modifications would be necessary?\n3) How does PGR's performance compare against PER baselines which use approximate parametric models of prior experience?\n4) Are there any other relevance functions thats been tried out? As thats core to the working of PGR.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1) PGR offers a fresh perspective on replay buffers by combining generative modeling with guided replay. Framing this problem as a conditional generation problem with diffusion models is novel.\n2) Diffusion model typically uses one single set of HPs requires no additional tuning I'd assume. This works well for PGR\n3) Empirical results on various benchmarks demonstrate that PGR consistently outperforms existing model-free and model-based RL algorithms, as well as a generative replay baseline without guidance. Also has been shown to work in both state-based and pixel-based environments. \n4) PGR is shown to scale well with larger policy networks and higher synthetic-to-real data ratios (important ablation that I wanted to see), potentially enabling more data-efficient training of large-scale RL agents. Really important result for scaling to many real use cases.\n5) The authors also provide insights into why PGR works, particularly highlighting the role of curiosity in promoting diversity and reducing overfitting.", "weaknesses": "1) The curiosity-based relevance function relies on a learned dynamics model, which might be challenging to train accurately in complex environments.\n2) Increasing Synthetic Data ratio does not benefit PGR and the unconditional baseline (SynthER) equally. PGR scales better at r=0.75 than SYNTHER but neither benefits from 0.875. We would think the trend would be consistent? whats the intution behind this? Also this figure 7 could be improved with the variation in r being shown\n3) (Minor) writing issues throughout the paper with some missing words etc. Please re-read the paper and make the necessary changes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)