Synthetic Data is Sufficient for Zero-Shot Visual Generalization from Offline Data
Authors: Ahmet H. Güzel
Ilija Bogunovic
Jack Parker-Holder
Venue: Accepted by TMLR ()

Abstract: Offline reinforcement learning (RL) offers a promising framework for training agents using pre-collected datasets without the need for further environment interaction. However, policies trained on offline data often struggle to generalise due to limited exposure to diverse states.The complexity of visual data introduces additional challenges such as noise, distractions, and spurious correlations, which can misguide the policy and increase the risk of overfitting if the training data is not sufficiently diverse. Indeed, this makes it challenging
to leverage vision-based offline data in training robust agents that can generalize to unseen environments. To solve this problem, we propose a simple approach—generating additional synthetic training data. We propose a two-step process, first augmenting the originally collected offline data to improve zero-shot generalization by introducing diversity, then using a diffusion model to generate additional data in latent space. We test our method across both continuous action spaces (Visual D4RL) and discrete action spaces (Procgen), demonstrating that it significantly improves generalization without requiring any algorithmic changes to existing model-free offline RL methods. We show that our method not only increases the diversity of the training data but also significantly reduces the generalization gap at test time while maintaining computational efficiency. We believe this approach could fuel additional progress in generating synthetic data to train more general agents in the future.

Published in Transactions on Machine Learning Research (07/2025)
Synthetic Data is Sufficient for Zero-Shot Visual General- ization from Offline Data
Ahmet H. Güzel
ahmel..uzel. www.puww....... 23 uk
University College London Al Centre
Ilija Bogunovic
i. Wucll. uk
University College London Al Centre
Jack Porder
i warker-holderrucl.cco uk
University College London AI Centre
Reviewed on OpenReview: https: // openreview. net/ forun? id= gFmSFa4080
Abstract
Offline reinforcement learning (RL) offers a promising framework for training agents us- ing pre-collected datasets without the need for further environment interaction. However, policies trained on offline data often struggle generalise due to limited exposure to di- verse Itates.The complexity of visual data introduces additional challenges such noise, distractions, and spurious correlations, which misguide the policy and increase the risk of overfitting if the training data is not sufficiently diverse. Indeed, this makes it challenging to leverage vision-based offline data in training robust agents that can generalize to unseen environments. To solve this problem, we propose a simple approach approach-generatinnn -generating additional synthetic training data. We propose a two-step process, first. augmenting the originally col- lected offline data to improve zero-shot generalization by introducing diversity, then using a diffuusion model to generate additional data in latent space. We test our method across both continuous action spaces (Visual D4RL) and discrete action spaces (Procgen), demonstrat- ing that it significantly improves generalization without requiring any algorithmic changes to existing model-free offline RL methods. We show that our method not only increases the diversity of the training data but also significantly reduces the generalization gap al test time while maintaining computational efficiency. We believe this approach could fuel additional progress generating synthetic data to train more general agents in the future.
1 Introduction
Offline reinforcement learning (RL) offers compelling approach for training agents using pre-collected datasots without additional environment interaction (Levine et al.] (020)) This paradigm is particularly valuable in domains like healthcare (Liu et al. 2020), robotics (Singla et al. 202I), and autonomous driving [Kiran al. 2021),, where real-time data collection can be costly risky. However, generalizing policies trained on high-dimeusional visual inputs in offline RL remains a significant challenge, and has received relatively little attention in the research community. Agents may learn irrelevant correlations between visual features and actions, reducing their ability to perform well in new settings (Song el. al. 2019 Raileanu & Fergus 2021 Additionally, offline RL policies tend to exhibit risk-averse behavior, avoiding novel actions in unfamiliar states, which further hampers generalization (Modiratta et al. 2024) To tackle these challenges, we propose a two-step method that combines data augmentation with diffusion model-based upsampling improve generalization offline RT. While both data augmentation (Laskin al. 2020 Yarats et al. 2021a by Raileanu et al. 2021) and the use of diffuusin models for replay buffer upsampling (Lu et al.) 2023bb have been explored independently in the online RL domain, contribution lies in their integration and adaptation for the unique challenges of offline RI to achieve greater diversity and more robust generalization.

---

Published in Transactions on Machine Learning Research (07/2023)
First, we apply data augmentation techniques to the offline dataset, introducing variability that helps reduce overfitting. Then, we use a diffusion model to upsample the augmented dataset in the latent space, generating additional synthetic data points that capture unseen transitions. This approach broadens the distribution of experience replay data without incurring significant compulational overhead, allowing policies to generalize more effectively to new environments. Our results demonstrate that our method significantly reduces the generalization gap across various difficulty levels in two recent visual offinne RI. benchmarks, highlighting the effectiveness of combining augmentation and diffusion-based upsampling.
Train
Test
Baseline
V-D4RL
1.0
V-D4RL
0.8
Upsampled
Upsampled-
Augmented
Augmented
V-D4RL
Normalized
0.0
(walker-walk)
Divergence
0.00
0.05 0.10 0.15
Generalization Performance
Train
Test
Procgen
Procgen
1.0
Baseline
Upsampled
0.8
Upsampled
Augmented
Augmented
Ours
Procgen (coinrun)
Normalized 0.0
0.0
0.1
0.3
0.4
Divergence
Generalization Performance
(a)
(b)
(c)
Figure 1: V-D4RL (continuous) and Procgen (discrete) benchmarks illustrate the generalization challenge in offline RL by showcasing visual differences between training and testing environments (a). Jensen-Shanna divergence heatmaps demonstrate how well each method aligns training and testing distributions, with our two-stage approach outperforming the data upsampling, and angmentation methods; darker colors indicato higher divergence (b). The performance of each method in unseen environments, normalized to the baseline, consistently shows our approach performing best with reduced variability across runs (c). For detailed analysis, refer to Section 5 for details.
To summarize, our contributions
We introduce a practical method that integrates data augmentation and diflusion-based upsampling to improve generalization in offline RL from visual inputs, without requiring modifications to existing model-free offline RI. algorithms.
We show our approach expands data diversity without increasing computational costs, improving zero-shol generalization across both continuous and discrete control tasks.
the first to propose practical, scalable method that addresses generalization in both continuous (V-D4RL) and discrete (Procgen) control tasks within offline RL.
To the best of our knowledge,
2 Background
2.1 Offline Reinforcement Learning from Visual Observations
Reinforcement learning (RL) typically involves an agent interacting with an environment modeled as a Markov Decision Process (MDP) Sutton & Barto 2018), where the objective to optimize the expected cumulative return ((x) )) = ExPP,po Ei=
0,),, where J(T) represents the expected return a policy H, E,, is the expectation over the policy n, the environment dynamics P(s'|s, a), and the initial state distribution Pa- The term ? E [0, 1) is the discount factor, controlling how much future rewards are valued, and R(s a,) is the reward function at time step depending the state st and action ar. The goal is to find policy that maximizes this cumulative discounted return time.

---

Published in Transactions on Machine Learning Research (07/2025)
However, in offline RL, the agent must learn from a static dataset D = {(or. T4 o) without any interaction with the environment during training [Levine et al. 20200. This dataset observations collected by onc or more behavior policies. When dealing with visual observations (high-dimensional inputs), addi- tional challenges arise. Unlike proprioceplive observations in standard RL, visual inputs are prone to noise and spurious correlations (Lu al.) 2023a), making offline RL particularly vulnerable to overfitting. Small environmental changes (e.g. lighting background) can cause significant shifts in data distribution. With- out interaction to correct for these shifts, agents struggle to generalize from visual observations (Raileanu & Fergus p.21 Given these challenges, the core problem is: How can improve the generalization perfor- mance of model-free offine RL methods from visual observations and ensure the robust deployment of agents in unseen environments for both continuous and discrete action spaces!
2.2 Diffusion Models
Diffusion models generate data by reversing noise-adding process, starting from noise and gradually de- noising recover the original data distribution (Ho et al., 2020; Rombach et al. 2022). Noise removal is guided by learned denoising model Do(a; trained using an L2 objective:
min 0
=11
(1)
This allows the model to estimate the data distribution at different noise levels. Additional details, including the use of ODEs or SDEs for the reverse process, can be found Karras et al. (2022). Diffusion models have shown superior performance in generating diverse synthetic datasets compared Generative Adver- sarial Networks (GAN) and Variational Autoencoders (VAEs), which makes them particularly effective for
improving generalization in reinforcement learning Lu et al. (2023b). This is why chose diffusion models, as their ability to generate diverse data makes them ideal for our approach.
3 Method
3.1 Overview of the Proposed Method
To address the generalization challenges of off-line RL from visual observations, we present simple, practical approach that combines data augmentation and diffusion model-based upsampling.
1. Data Augmentation to Increase Initial Dataset Diversity: We apply specific data augmen- tation techniques to offline datasets to increase the diversity of the initial dataset Dy- This step aims to introduce variability and reduce overfitting to spurious correlations in visual inputs.
Upsampling with Diffusion Modcls: We employ diffusion model to upsample (SynthER (Lu) et al. 2023bl) the augmented dataset Do, generating additional synthetic samples in the latent space. This further increases dataset diversity and helps the policy generalize better to unseen environments.
By integrating data angmentation with diffusion model-based upsampling, our method effectively covers wider range of potential scenarios without significantly increasing computational overhead.
3.2 Step 1: Data Augmentation for Initial Dataset Diversity Enchancement
To construct an initial dataset Do that captures key environment dynamics, we apply a set of data augmen- tation techniques to improve robustness to variations in visual inputs. Specifically, we focus on rotation, color jätfering, color culout, and background image overlay, which were empirically found improve general- ization. These augmentations introduce variations that prevent the agent from learning spurious correlations in the visual inputs. However, believe that data augmentation alone may not fully capture the diversity of real-world scenarios, particularly in unseen environments, making it necessary to complement this with
3

---

Published in Transactions on Machine Learning Research (07/2023)
synthetic data generation through diffusion models, which has proven to improve diversity even more effec- tively than augmentation techniques (Lu et al. 2023b Full details for this part are given in supplementary matcrial.
3.3 Step 2: Latent Space Upsampling with Diffusion Models
We first train an encoder-based model-free visual offline RL algorithm on the augmented dataset Do. using the selected image augmentation techniques described in Section ??. For the V-D4RL benchmark, we use the DrQ+BC algorithm (Tuu al. 2023a), while CQL is employed for the Procgen benchmark (Kumar eet al. ).20 In both cases, the networks consist of CNN encoder fe, policy network Th, and Q-function networks Qo. This initial training enables the model to learn robust representations from diverse visual inputs, tailored to the specific requirements of each environment.
After the initial training, we extract latent space parameters from the augmented dataset by passing the augmented observations through the trained encoder Se and the linear head layers, which sit between the encoder and the MLPs the policy and Q-function networks. For each transition (s, a, r,,) Do, the following computations are made:
h fe (Auggment)s)))
h' fe (euumment((s)
(2)
2x 1 Jin (h),
2/4 T (h') n
(3)
=Q Q (h),
=QQ Q lin (h')
(4)
where Tht and Q!in denote the linear head layers of the policy and Q-function networks.
We combine the latent representations to construct the latent transitions using concatenation, which produces feature vectors with dimensions equal the sum of the individual vectors' dimensions:
2τ =Q:
=4=4 =
(5)
resulting in the latent dataset Diateut = {(2,,,,00 a, 2.), which is used to train the diffusion model Main. Following La et al. (2023b) and Karras al. (2022), the diffusion model generates synthetic latent transitions (zd, aa, Td, 2/),, producing the upsampled dataset Daift
Finally, combine the original and upsampled datasets to create expanded dataset:
Dups = Diatent U Dait
(6)
The encoder f and the linear head layers of the policy and Q-functions are frozen during fine- fine-tuning, allowing the training to focus refining the MLP layers of the policy and value networks using the diverse data provided by Dups- This ensures stable representations while improving the model's ability to generalize to unseen environments with minimal computational overhead. Our empirical approach, which combines data augmentation with synthetic data generation through upsampling in the latent space, significantly increases the diversity of the data set, as demonstrated in our results (Section 5). Figure 2 illustrates the architecture of our method, built on the DrQ-BC model.
4 Experimental Setup
4.1 Environments and Datasets
We evaluated our method on two challenging offline RL benchmarks that test generalization capabilities in different domains:
Visual D4RL (V-DARL) [Lu et al. 2023a): This benchmark a visual input version of the D4RL benchmark Fu al., 2021) and focuses on continuous control tasks with visual input. It features varying levels of visual distractions (easy, medium, hard) and is designed to assess generalization in continuous action spaces.

---

Published in Transactions on Machine Learning Research (07/2025)
S, S'
Cribic Lineer
Latent Space
20.2/a
(s, a, s))
S, s'
Trained
Offline Datasel
Z
Actor Linear
Upsampling
Critic MLP
a
r-r
Augmentation
J(Q))
Forward Pess
Dan
Diffusion Model Upeample
Actor MLP
Critic Network Treining
Diatant
Actor Network Training
Jm
Figure 2: Illustration of our method in the V-D4RL benchmark using the DrQ+BC network. Green arrows indicate data augmentation, while blue and orange arrows represent the training flows for the actor and critic networks, respectively. The red components highlight the diffusion model upsampling process, which generales additional latent space transitions to increase the dataset diversity.
Offline Procgen (Mediratta al.) 2024)): This is an offinn version of Procgen benchmark Cobbe et al. (2020) procedurally generated games that targets discrete control tasks. It tests zero-shot generalization to entirely unseen levels.
These offline RL benchmarks were chosen to comprehensively evaluate our method's performance diverse environments, including both continuous and discrete action spaces, well as its ability to gener- alize in the presence of visual distractions and to completely novel scenarios. Figure 3 illustrates sample observations from the V-D4RL and Procgen datasets, highlighting the visual diversity and complexity across training and testing environments.
V-D4RL
Procgen
Train
Test
Original
Easy
Medium
Mard
chesteh-rur
coinrun
waiker-walk
ninja
humanold--walk
jumper
Figure 3: Sample screenshots from the V-D4RL (left) and Procgen (right) datasets, showing the training environments and testing environments.
For our experiments, we generated different datasct variants to ovaluate the offectiveness of our method. We used a Baseline dataset without augmentation or upsampling, which was originally provided by both benchmarks; an Upsampled dataset where the original dataset was increased in size using diffusion model- based upsampling, without any augmentation; an Augmented dataset where data augmentation techniques were applied without changing the dataset size; and Ours dataset that combined both augmentation and upsampling. For more details on the experimental setup, please refer to the supplementary material.

---

Published in Transactions on Machine Learning Research (07/2025)
4.2 Implementation Details
algorithm, which extends DrQ-v2 (Yarats et al. 2021a), using BC (as in TD3BC, Fujimoto & Gu (2021)) For Procgen we used Conservative Q-Learning (CQL) (Kumar et al. 2020). The hyperparameters, network architectures, and other implomentation details follow the standard settings provided the original benchmark papers. For completeness, we provide all hyperparameters and network architecture details in supplementary material.
For VD4RL we used the DrQ+BC [Lu et al. 2023a
4.3
Evaluation Metrics
4.3.1
Generalization Performance Metric
To quantify our model's ability to generalize to unseen environments with visual distractions, we adopt a generalization performance metric inspired by the normalization approach commonly used in reinforcement learning (RL) studies, such as the Procgen benchmark Cobbe et al. (2020) which defines the normalized return (Rnorm) as:
R Rain
Rnorm
Rmax Rmin
where R is the agent's performance, Rmin represents the lowest possible score (e.g., the baseline's test performance), and Rmax corresponds the highest possible score. Inspired by this formulation, we define our Generalization Performance as the proportion of the baseline's generalization gap that method closes. Specifically, we calculate it by subtracting the baseline's mean test return (Buet) from our method's mean test return (Tiest,, then dividing by the difference between the baseline's mean training return (Btrain) and Brest- Generalization performance value of 1 indicates that our method completely eliminates the gap, providing a clear benchmark for comparison.
4.3.2 Latent Space Distribution Analysis
To gain deeper insights into the impact of our method on learned representations, we performed a latent space distribution analysis using the Jensen-Shannon (JS) divergence (Menéndez et al., 1997]. For each dataset variant, we extracted latent space representations by passing both training and testing observations (collected during the evaluation steps, as detailed in supplementary material through the trained encoder ft and the actor-critic networks. Let Atrai and Mest represent the sets of latent representations for training and testing observations, respectively. For each environment, we estimated probability distributions over the latent space dimensions using kernel densily estimation (KDE) to non-parametrrcally capture the distributions of Atrain and Mest. We then computed the JS divergence between the training and testing distributions for each dimension, resulting in a divergence vector d = [d1, ..,....dd dn), where n represents the dimensionality of the latent space. The mean JS divergence d across all dimensions was used to summarize how closely the training and testing distributions aligned, with lower divergence indicating better generalization. To facilitate comparison across different environments, each environment, ensuring consistency in scale. A closer match between training and test distributions suggests improved generalization performance by our method.
normalized the mean JS divergence values within
5
Experimental Results and Discussion
5.1 V-D4RL Benchmark Results
As seen in Figure f models trained on the Baseline and Upsampled datasets showed minimal improvement in the generalization gap across all environments (cheetah-rwn, walker-walk, and humanoid- walk)) suggest- ing that upsampling alone does not significantly improve generalization. The baseline consistently shows generalization gap of 0 because the results normalized over it, and thus its results are not explicitly shown. In contrast, the Augmented model demonstrated substantial reduction in the generalization gap,

---

Published in Transactions on Machine Learning Research (07/2023)
cheetah-run
walker-walk
humanoid-war
Upsampled
Augmented
Ours
0.0
0.1
0.3
0.00
0.05
0.10
0.15
0.00
0.03
0.06
0.09
Generalization Performance
Generalization Performance
Generalization Performance
(a)
cheetah-run
walker-walk
humanoid-walk
Baseline
0.8
Upsampled
0.6
Augmented
Ours
Easy Medium Hard
Easy Medium Js Divergence (b)
Easy Medium Hard Js Divergence
0.0
Js Divergence
Figure 4: (a) Generalization performance averaged across all difficulty levels (easy, medium, hard) for dif- ferent environments. (b) Normalized JS divergence values for each environment which normalized relative to that enviromment. Lower values (lighter colors) indicate closer alignment between the distributions of training and test data, suggesting better generalization.
highlighting the importance of data augmentation in improving robustness against unseen visual perturba- tions. The best results were achieved with the Ours dataset, where combining data augmentation with upsampling further improved generalization across all environments. The JS Divergence analysis similarly showed that the Ours dataset achieved the lowest divergence, indicating a closer aligument between the training and testing distributions the latent space. This further supports that combining augmentation with upsampling leads more consistent latent representation and, consequently, better generalization. These findings align with the generalization gap and return values in Table
Table 1: Performance evaluation the V-D4RL benchmark across different datasets and environments, trained using the DrQ+BC algorithm. All return values are based on the mean over five random seeds.
Environment
Method
Original
Easy
Medium
Hard
Test Mean
Baseline
250.1 = 10.9
4.2 1.3
3.1 = 0.6
3.8
3.53 + 0.9
Upsampled
315.2 1 20.1
4.6 1 0.7
3.5 1 0.3
4.1
4.06 1 0.7
cheetaht-run
Augmented
350.5 1 15.5
81.2 8.1
60.7 6.2
41.3
61.1 5.9
Ours
360.2 + 10.1
86.1
71.2
54.4
70.6 + 5.2
Bascline
570.2 1 6.7
2..4
31.6 3.3
29.9
32.3 1 0.9
Upsampled
665.2 +
32.4
30.3
28.9 1.8
30.5
walker-woalk
Augmented
799.5 + 10.5
131.5 + 10.1
75.1 5.2
5..5
85.7 - 5.6
Ours
845.9 +
141.1 + 12.1
92.3 2.3
6.67
99.7
Baseline
15.4 + 2.1
1.3 0.2
1.0 - 0.3
1.1
1.1 0.2
Upsampled
20.4 2.3
0.3
1.3 0.4
1.1
1.2 0.3
hamanoidd-walk
Augmented
23.4
1.6
1.5 0.3
1.2
1.4 0.3
Ours
28.5 1.4
0.2
2.3 0.1
1.7
2.2 0.2
The results indicate that data augmentation improves generalization amidst visual distractions. Our ap- proach achieves notable generalization performance in zero-shot testing circumstances.

---

Could incorporating a small, strategically chosen subset of data that closely aligns with the evaluation dis- tribution-ulle combined with augmentation and upsampling techniques -ofeeees--fffrrrrroe offer viable strategy to improve Published in Transactions on Machine Learning Research (07/2023)
5.1.1 Leveraging Fixed Distracting Data for Improved Generalization
Building on our findings from the V-D4RL benchmark, where our two-stage approach of augmentation and upsampling significantly improved generalization, we further tested the robustness of our method. Although previous research (Lu et al. 2023ab indicated that training with fixed distracting crafted distractions oflered minimal benefits for generalization, we hypothesized that our method could effectively leverage even a small portion of such data. To test this, we incorporated 5% of the fixed distract- ing data into our training dataset, combining it with 95% of the original baseline data to create a composite dataset for baseline. We then applicd our augmentation and upsampled technique, we refers to this as Ours Fixed Dala Distruction (FDD). For more details on the experimental selup, please refer to the supplementary material.
hand-
cheetah-run
cheetah-run
1.0
Baseline
Upsampled
0.8
Augmented
Augmented
Upsampled
Ours
0.4
Ours
Ours (FDD)
0.2
Ours (FDD)
0.0
0.1
0.2
0.3
Easy
Medium
Hard
Generalization Performance
Js Divergence
(a)
(b)
Figure 5: (a) Generalization performance averaged over all difficulty levels, and (b) comparison of normalized JS divergence values for cheetah-run expert dataset.
As illustrated in Figure including just 5% of the fixed distracting data improved generalization performance across all test environments, further narrowing the generalization gap. The latent space distributions also aligned more closely with the test data, as evidenced by reduced JS divergence values. Notably, we saw significant. improvements, especially on medium and hard difficulty levels, when this fixed distracting data was combined with our augmentation and upsampling techniques. (TTble 2
Table 2: Performance the cheetah-run expert dataset with and without incorporating 5% fixed distracting data (FDD). Results are based on the mean of five random seeds.
Environment
Method
Original
Easy
Medium
Hard
Test Mean
Baseline
250.1 + 10.9
4.2 1.3
3.1 0.6
3.3 0.8
3.53 0.9
Upsampled
315.2 + 20.1
4.6 + 0.7
3.5 0.3
4.1 0.9
4.06 0.7
cheetah-run
Augmented
350.5 + 15.5
8.1
60.7
41.3 3.4
61.1 5.9
Ours
360.2 + 10.1
86.1 7.1
71.2
54.4 2.4
70.6 + 5.2
Ours (FDD)
267.9 7.2
103.4 5.3
1 5.0
59.8 3.4
82.8
Environment
Method
(Test Mean) / Train
Train
(Test Mean)
Baseline
0.01
246.6
Upsampled
0.01
311.1
chectah-run
Augmented
0.17
289.4
Ours
0.20
289.6
Ours (FDD)
0.31
185.1
Our approach effectively utilizes this data to add diversity and improve robustness to unseon distractions, leading an intriguing question:
generalization in few-shot learning scenarios?
8

---

Published in Transactions on Machine Learning Research (07/2023)
5.2 Results on Procgen Benchmark
Our results on the Offtine Procgen Benchmark further validate the generalization capabilities of our method, demonstrating its effectiveness not only in continuous control environments like V-D4RL but also in discrete control tasks. Figure B illustrates the generalization performance across three different environments across datasets.
Coinrun
Ninja
Jumper
Upsampled-
Augmented
Ours
0.00 0.18
0.36 0.54 0.72
0.00
0.12
0.24
0.36
0.00
0.05
0.10
0.15
Generalization Performance
Generalization Performance
Generalization Performance
(a)
Coinrun
Ninja
Jumper
1.0
1.0
1.0
Baseline
Baseline
Baseline
0.8
0.8
0.8
Upsampled
Upsampled
Upsampled
0.6
0.6
0.6
Augmented
0.4
Augmented
0.4
Augmented
0.4
0.2
0.2
0.2
Ours
Ours
Ours
Divergence
0.0
Divergence
0.0
JS Divergence
0.0
(b)
Figure 6: Performance evaluation on the Proegen benchmark across three different games, trained using the CQL algorithm. (a) Generalization performance for three games Procgen. (b) Comparison normalized JS divergence values.
Table 3: Performance of the CQL algorithm on the Procgen Benchmark, based on the mean over five random seeds.
Environment
Method
Train Return
Test Return
Baseline
8.51 1 0.27
7.17 + 0.27
Upsampled
8.96 1
7.37 0.37
Coinran
Augmented
8.65 + 0.31
7.50 + 0.54
Ours
8.74 0.34
8.02 0.44
Baseline
5.94 + 0.23
4.41 + 0.21
Upsampled
6.18 + 0.39
4.52 + 0.41
Ninja
Augmented
5.83 + 0.31
4.75 +: 0.35
Ours
6.05 + 0.27
4.82 0.27
Baseline
7.62 1. 0.19
4.23 0.2.4
Upsampled
7.94 + 0.32
4.36 + 0.28
Jumper
Augmented
7.55 + 0.28
4.51 0.19
Ours
7.35 + 0.24
4.72
Environment
Method
Test.
Train
Train
Test
Baseline
0.72
2.09
Upsampled
0.70
2.28
Proegen Averaged
Augmented
0.76
1.76
Ours
0.79
1.53

---

Published in Transactions on Machine Learning Research (07/2025)
As shown in Table applying data augmentation alone to the baseline dataset led to marginal improvements. However, our propased improvements across all three games. These results underscore the effectivenoss of our solution, confirming its ability to improve generalization performance in discrete control lasks like Procgen, in line with the gains observed in the V-DARL continuous control benchmark. Thus, our method demonstrates its effectiveness not only in continuous control tasks but also in discrete control environments. While the improvements in Procgen are consistent with the trends observed in V-D4RL, they validate the generalization capabilities of our solution across broader range of offline RL challenges. Tt is important to note that Mediratta et al. (Mediratta et al., 2024) provided an offline Procgen dataset collected from only 200 levels, limiting our ability to compare the effects of increasing the number of levels on generalization performance relative to our method. Despite this limitation, our results demonstrate that without expanding the dataset's 's level diversity, our approach significantly amplifies generalization performance. This suggests that our method effectively compensates for the limited number of levels through data augmentation and upsampling alone.
augmented and upsampled sinsett-yelled the most significant
6 Related Work
Generalization in RL has been extensively studied, primarily in the context of online RL. A substantial body of work has focused on training agents to generalize across novel transition dynamics and reward
functions (Rajeswaran et al., 2018 Machado et al. 2017 Packer et 2019 Cobbe et al. 2020 Kirk et al.
2023
Justesen et al.] 2018
Nichol et al. 2018 Küttler et al.] 2020;
Bengio et al. 2020 Bertran et al.
2020:
Dunion et al. 2023 Almuzairee et al.) 2024). RL euvironments such as Procgen (Cobbe et al. 2020 and the NetHack Learning
Ghosh et a 2021 Lyu et al. 2024 Fhrenberg et al., 2022; Lyle et al. 2022;
Environment [uumar et al., 2020) have been specifically developed to assess generalization in online RL. However, these studies largely focus on interactive settings where agents can gather new data during training, leaving generalization in offline RL. relatively unexplored.
Visual offline RL introduces additional challenges, particularly when using large-scale datasets. Dalasets like Atari, StarCraft, and MineRL contain millions of samples but require significant computational resources, limiting their accessibility to many rescarchers (Agarwal et al., 2020 Vinyals et al., 2017 Fan et al.
2022
In contrast, benchmarks such as V-D4RL and offline Proegen (Lu et al.) 2023a Mediratta et al. 2024)
offer more accessible alternatives, with 100,000 and 1 million samples per environment, respectively, while still supporting meaningful evaluation of generalization in continuous and discrete control tasks. Both benchmarks highlight the generalization challenges in offline RL, especially with model-free methods. Our
work ex extends these efforts by evaluating generalization across diverse, procedurally generated environments in both continuous and discrete control tasks.
Data augmentation in RL has been widely successful in improving generalization in online RL mothods (aarats al. 2021b a Raileanu et al. [002 Laskin et al. 2020: Ma al. 2024 Corrado et al. 2024 Pitis et al. 2020 Sinha et al.) 2021 Lee et al. 2024), but its application in offline RL remains underexplored. Our work leverages augmentation to address generalization in offline settings, demonstrating its potential for visual tasks. Unlike methods such as DrAC (Raileanu et al.) 2021) and SVEA (Hansen et al. 2021), which involve algorithmic changes in online RL settings, our approach focuses on non-algorithmic enhancements using simple visual augmentations, as domonstrated by (Laskin ct al. 2020) combined with diffusion- basod upsampling. This combination improves data diversity and generalization, providing a scalable, practical solution for offline RL settings.
Diffusion models have emerged promising tool for improving RL solutions, though they have primarily been used as policies or planners rather than as data synthesizers (Zhu et al. 2024 Jackson et al.] 2024). Recent works, such as ROSIE' (Yu et al.) 2023) and GenAug (Chen et al, 2023). have employed diffusion models for synthetic data generation to improve generalization. However, these methods operate in online robotic learning contexts, relying on continuous interaction with the environment. In contrast, our approach applies diffusion-based upsampling offline RL, where additional interaction with the environment is not possible. Inspired by SynthER (Lu et al. 2023b), our method improves data diversity through upsampling, making it the first diffusion model-based data synthesis aimed at solving the generalization problem in model-free offline RL with visual inputs, across both continuous and discrete control tasks.
10

---

Published in Transactions on Machine Learning Research (07/2023)
7 Conclusion
We presented a practical two-step approach that improves generalization in offline reinforcement learning from visual inputs. By combining targeted data augmenlation with diffusion model-based synthetic data generation in the latent space, our approach increases training data diversity without significant computa- tional overhead, allowing model-free offline RL algorithms to better handle risk-averse behavior in unscen enviromments. Our experiments on the V-DARL benchmark (continuous control) and Procgen benchmark (discrete control) demonstrate that our approach consistently reduces the generalization gap and improves performance in unseen environments. Additionally, our method effectively leverages small amounts of hand- crafted, fixed distracting data to further improve generalization, suggesting potential applications for few-shot learning in offline RT. These benchmarks challenge the development of better offline RL algorithms for vi- sual observations, and to our knowledge, we are the first to apply this approach across both continuous and discrete action spaces. By broadening the data distribution in both pixel and latent spaces, we provide a scalable two-step solution to the generalization challenges in offiine RL.
While our method shows significant improvements, there certain limitations. Although working in the latent space keeps computational overhead relatively low, extending this approach to the pixel space would introduce significantly higher costs, especially in environments with high-resolution visual inputs. Addition- ally, our method required extensive experimentation to identify the best settings for data augmentation and diffusion model parameters, which may limit its immediate applicability to more complex enviromments like robotics or autonomous driving. We opted for model-free algorithms in this work due to their sampling efficiency and lower computational load. However, these algorithms may face limitations in handling more complex tasks, particularly those requiring long-horizon planning. Future work will focus scaling this ap- proach to such environments and exploring its integration with various RL algorithms, including model-based ones, to improve generalization in offline RL settings.
References
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline rein- forcement learning, 2020. URL httpss://axiv org. wwwp:/ogg//a..eee. ab//1907.01555
Abdulaziz Almuzairee, Nicklas Hansen, and Henrik I. Christensen. A recipe for unbounded data augmenta- tion in visual reinforcement learning. 2024. URL https://arxiv. brrg/abs//un 2405./245.... 1400.174166....
Emmanuel Bengio, Joelle Pineau, and Doina Precup. Interference and generalization in temporal difference learning. 2020. URL https://arxiv. org. http:/bbs/0000...
Martin Bertran, Natalia Martinez, Mariano Phielipp, and Guillermo Sapiro. Instance based generalization in reinforcement learning. 2020. URL
01089
Zoey Chen, Sho Kiami, Abhishek Gupta, and Vikash Kumar. Genaug: Retargeting behaviors to unseen situations via generative augmentation, 2023. CRL httpp:::/arriv.or//aas////////
Karl Cobbe, Christopher Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation to benchmark reinforcement learning. 2020. URL
Nicholas E. Corrado, Yuxiao Qu, John U. Balis, Adam Labiosa, and Josiah P. Hanna. Guided data aug- mentation for offline reinforcement learning and imitation lcarning, 2024. URL
18247
Mhairi Dunion, Trevor Mclnroe, Kevin Sebastian Luck, Josiah Hanna, and Stefano V. Albrecht. Conditional mutual information for disentangled representations in reinforcement learning. Tn Conference on Neural Information Processing Systems, 2023.
Andy Ehrenberg, Robert Kirk, Minqi Jiang, Edward Grefenstette, and Tim Rocktäschel. A study of off- policy learning in environments with procedural content generation. In ICLR Workshop Agent Learning in Open- Open-Endedte Endedness, 2022. URL http:///pppaeeeeeeeeeeeeerrrreee
11

---

Published in Transactions on Machine Learning Research (07/2025)
Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De- An Huang, Yuke Zhu, and Anima Anandkumar. Minedojo: Building open-ended embodied agents with internet-scale knowlodge, 2022. URL btcts://aaxxvv.org/ab///// org/abs/22066 08853
Justin Fu, Aviral Kumar, Ofr Nachum, George Tucker, and Sergey Levine. D4rl: Datasots for doep data- driven reinforcement learning. 2021. URL https://arxiv..rgg abs ab/.ab//..2022
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. 2021.
Dibya Ghosh, Jad Rahme, Aviral Kumar, Amy Zhang, Ryan P Adams, and Sergey Levine. Why generaliza- tion in rl is difficult: Epistemic pomdps and implicit partial obscrvability. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advonces in Newral Information Processing Sys- tems, volume 34, pp. 23502 25515. Curran Associates, Inc., 2021. URL htt/htpps///proeeddngg neurips.. httppp/.....r.eeee//t//t///////t
pdf
Nicklas Hansen, Hao Su, and Xiaolong Wang. Stabilizing deep q-learning with convnets and vision trans- formers under data augmentation. 2021. URL httpp://aaxxv..org/////////.....
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising dillusion probabilistic models, 2020. URL https
//arxiv.or wwwxii/.ogg/eee httg//bb//006...../// .12000 11239
Matthew Thomas Jackson, Michael Tryfan Matthews, Cong Lu, Benjamin Ellis, Shimon Whiteson, and 06356
Jakob Focrster. Policy-guided diffusion, 2024. URL
Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Scbastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. 2018.
URL https://arxiv..o org/ www.////........ 1872.10729
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. 2022.
B Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A. Al Sallab, Senthil Yogamani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey, 2021. URL https: 7/arxiv.org abs/2002.024044
Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of zero-shot generalisation in deep reinforcement learning. Journal of Artificial Intelligence Research, 76, 2023. ISSN 1076-9757. doi:
10.1613/jair...4111 URL http://dt::tt doi org 10 1613//aar....11177
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforce- ment learning. 2020. URL http://aaxi...rrrraa//000...00 04779
Heinrich Küttler, Nantas Nardelli, Alexander H. Millor, Roberta Raileanu, Marco Selvatici, Edward Grefen- stelte, and Tim Rocktäschel. The nethack learning environment. 2020. URL httpps///axxv, http://///xxvvo.o/aassss..
2006.13760
Michael Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Picter Abbecl, and Aravind Srinivas. Reinforcement learning with augmented data. 2020.
Jaewoo Lee, Sujin Yun, Tacyoung Yun, and Jinkyoo Park. Gta: Generative trajectory augmentation with guidance for offline reinforcement learning, 2024. URL atttp://aaxxiv.org/bbbss. 16907
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. 2020.
Siqi Liu, Kay Choong See, Kee Yuan Ngiam, Leo Anthony Celi, Xingzhi Sun, and Mengling Feng, Reinforce- ment learning for clinical decision support in critical care: Comprehensive review. J Med Internet Res, 22 :7)177, Jul 2020. ISSN 1438-8871. doi: 10.2196/18477 CRLpstts:: ://www jmir 00tg/22000///1114777 /at
12

---

Published in Transactions on Machine Learning Research (07/2025)
Cong Lu, Philip J. Ball, Tim G. J. Rudner, Jack Parker-Holder, Michael A. Osborne, and Yee Whye Tch. Challenges and opportunities offiie reinforcement. learning from visual observations. 2023a.
Cong Lu, Philip J. Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. 2023b.
Clare Lyle, Mark Rowland, Will Dabney, Marta Kwiatkowska, and Yarin Gal. Learning dynamics and generalization in reinforcement learning. 2022. URL
02126
Jiafei Lyu, Le Wan, Xiu Li, and Zongging Tu. Understanding what affects generalization gap in visual reinforcement learning: Theory and empirical evidence. eeideecce 2024.
Guozheng Ma, Zhen Wang, Zhecheng Yuan, Xueqian Wang, Bo Yuan, and Dacheng Tao. comprehensive survey of data augmentation in visual reinforcement learning, 2024. URL https://arxiv org/abs/2210. 04561
Marlos C. Machado, Mare G. Bellemare, Erik Talvitie, Joel Veness, Malthew Hausknecht, and Michael Bowling. Revisiting the arcade learning environment: Evaluation protocols and open problems for general agents. 2017. URL
06009
Ishita Mediratta, Qingfei You, Minqi Jiang, and Roberta Raileamu. The generalization gap in offline rein- forcement learning. 2024.
M.L. Menéndez, J.A. Pardo, Pardo, and M.C. Pardo. The jensen-shannon divergence. Journal of the Franklin Institule, 334(2):307 318, 1997. ISSN 0016-0032. doi: hutp:///bbiooo////00000000000000 00063-4. URL
Alex Nichol, Vicki Pfau, Christopher Hesse, Oleg Klimov, and John Schulman. Gotta learn fast: A new benchmark for generalization in rl. 2018. URL
Charles Packer, Katelyn Gao, Jernej Kos, Philipp Krähenbühl, Vladlen Koltun, and Dawn Song. Assessing generalization in deep reinforcement learning. 2019. URL
12282
Silviu Pitis, Elliot Creagor, and Animesh Garg. Countorfactual data augmentation using locally factored dynamics, 2020. URL httpps::/aaaavv.orraaaaa///////a 028863
Roberta Raileanu and Rob Fergus. Decoupling value and policy for generalization in reinforcement learning. 2021. URL
Roberta Raileanu, Max Goldstein, Denis Yarats, Ilya Kostrikov, and Rob Fergus. Automatic data augmenta- tion for generalization in deep reinforcement learning. 2021. URL /tttps:///axiiiii wtw//rr//////////oooe 12862,
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, Giulia Vezzani, John Schulman, Emanuel Todorov, and Sergey Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. 2018. URL https://taaaat http://aaxxvvv.o//a//eees/ 10087
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-
resolution image synthesis with latent diffusion models, 2022. https : https///gggnngaccecooooo cot.://aaacccc/o/oo////occoaaa/
Adish Singla, Anna N. Rafferty Goran Radanovic, and Neil T. Heffernan. Reinforcement learning for oducation: Opportunitics and challenges. 2021. URL https: wwwpaaiii..oggaa////a//aa htggabs//107..ee
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4r1: Surprisingly simple self-supervision for offline reinforcement learning, 2021. URL
Xingyou Song, Yiding Jiang, Stephen Tu, Yilun Du, and Behnam Neyshabur. Observational overfitting in reinforcement learning. 2019. URL
Richard S. Sutton and Andrew G. Barto. Reinforceement Learning: An Introduction. The MIT Press, second edition, 2018. URL http:///ccmmpeeaaccaaaeeeeeeeeea
html
13

---

Published in Transactions on Machine Learning Research (07/2023)
Oriol Vinyals, Timo Ewalds, Sergey Bartunov, Petko Georgiev, Alexander Sasha Vezhnevets, Michelle Yeo, Alireza Makhzani, Heinrich Küttler, John Agapiou, Julian Schrittwieser, John Quan, Stephen Gaffney, Stig Petersen, Karen Simonyan, Tom Schaul, Hado van Hassolt, David Silver, Timothy Lillicap,, Kevin Calderone, Paul Keet, Authony Brunasso, David Lawrence, Anders Ekermo, Jacob Repp, and Rodney Tsing. Starcraft ii: A new challenge for reinforcement learning, 2017. URL https://arxis waaaxiii..oggaaaa abs/ 1708.04788
Denis Yarats, Rob Fergus, Alessandro Lazaric, and Lerrel Pinto. Mastering visual continuous control: Improved data-augmented reinforcement learning. 2021a.
Denis Yarats, Πlya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning represenlations, 2021b.
Tianhe Yu, Ted Xiao, Austin Stone, Jonathan Tompson, Authony Brohan, Su Wang, Jaspiar Singh, Clayton Tan, Dee M, Jodilyn Peralta, Brian Ichter, Karol Hausman, and Fei Xia. Scaling robot learning with semantically imagined experience, 2023. URL https://arxiw.orrrt www.ggg///////o//eo/ooooeeeeeee/ abs/////////2// 11550
Zhengbang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Haoquan Guo, Tingting Chen, and Weinan Zhang. Diffusion models for reinforcement learning: A survey, 2024. URL https: htttp:://aaatt httppp://axxvv.rrg abs hts/2311.0122
14


Reviews:
[{"review_id": "EjIgQ8Bas2", "reviewer": "Reviewer_XXdz", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper reads very nicely. In general, the flow of ideas and is clear, goals are clear, etc. \n2. I appreciate that the paper directly shows the proposed method reduces the generalization gap with JS divergence. I often see papers that say \"we resolve problem X, which then improves performance\" but only show performance improvements -- not that the method actually resolves problem X. \n\n**Weaknesses**\n1. The paper states that this method has limited computational overhead, but experiments do not quantify this overhead.\n1. It's somewhat unclear to me why the experiments do not compare against other diffusion-based data augmentation methods like GTA [1]. I think the core goal is to show that augmentation followed by upsampling outperforms augmentation and upsampling individually, in which case it's fine to simply use one diffusion strategy. In any case, the authors should clarify why they chose these specific baselines and why they are sufficient.\n\n\n---\n# References\n\n\n1. Lee at. al. GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning. Neurips 2024. https://arxiv.org/abs/2405.16907", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes synthetic data generation method to increase data diversity and improve generalization in offline reinforcement learning (RL) from visual inputs. The method first applies standard visual data augmentation techniques on the offline dataset and then uses diffusion-based latent space data generation to expand the dataset. Experiments demonstrate that this two-step approach significantly reduces the generalization gap in both continuous and discrete-action benchmarks when combined with standard offline learning algorithms.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "To preface: My first two requested changes focus on additional references for the related work. These are \"soft\" requests; I don't expect the authors to cite all of them, and the authors can decide whether to include them or not. I think the non-visual augmentation works focusing on offline RL [1,3,6] and the diffusion-based augmentation strategy GTA [9] are most relevant.\n\n1. **Prior works in data augmentation for non-visual tasks.** I think it's understandable to focus on visual data augmentation in the related work, though I will point out that there is a sizable literature on non-visual data augmentation methods, some of which have been used for offline RL.\n    * Many data augmentation methods leverage symmetries and/or invariances in a task's dynamics to generate dynamically accurate data [1-5]. \n    * Others methods add noise to states/actions [6,7].\n    * Pitis et. al [8] use a dynamics-model to generate augmented data.\n    * [1] and [6] focus on offline learning, and [3] focuses on both offline and online learning. The others focus only on online learning.\n\n1. **Discussion on data diversity and data quality for offline learning.** Most offline data augmentation methods focus on improving dataset diversity, but it would be worth acknowledging that diversity is only part of the story. The *quality* of the augmented data is also important, and I think it may be worth mentioning even briefly.  \n    * Corrado et. al [1] introduce a framework for generating *expert-quality* augmented data that outperforms frameworks that focus on generating diverse augmented data. They study both offline RL and imitation learning. \n    * GTA [9] uses a diffusion model to generate diverse *high-return* trajectories.\n    * MoCoDA [8] gives the user control over the distribution of augmented data generated so they can generate **task-relevant** data. MoCoDA outperforms its predecessor CoDA which only focuses on generated diverse augmented data.\n    * I remember seeing a visual augmentation paper that focused on generating task-relevant augmentations, but I cannot locate the paper. I will add it if I find it.\n\n1. Figure captions should state what error bars and shaded regions represent (e.g. +/- standard deviation)\n1. I think it's clearer to write \"Augmented Upsampled\" or \"Augmented Upsampled (Ours)\" rather than \"Ours\" in tables and figures.\n\n---\n\n# References\n\n\n1. Corrado et. al. Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning. RLC 2024. https://arxiv.org/abs/2310.18247\n\n2. Corrado & Hanna. Understanding when Dynamics-Invariant Data Augmentations Benefit Model-free Reinforcement Learning Updates. ICLR 2024. https://arxiv.org/abs/2310.17786\n\n3. Pitis et. al. “Counterfactual Data Augmentation using Locally Factored Dynamics.” NeurIPS 2020. https://arxiv.org/abs/2007.02863\n\n4. Abdolhosseini et. al. “On Learning Symmetric Locomotion.” ACM SIGGRAPH 2019. https://www.cs.ubc.ca/~van/papers/2019-MIG-symmetry/index.html\n\n5. Adrychowicz et. al. \"Hindsight Experience Replay.\" NeurIPS 2017. https://arxiv.org/abs/1707.01495\n\n6. Sinha et. al. S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning in Robotics. CoRL . https://arxiv.org/abs/2103.06326\n\n7. Qiao et. al. Efficient differentiable simulation of articulated bodies. ICML 2021. https://arxiv.org/abs/2109.07719\n    * This paper focuses on differentiable simulation, but they use a data augmentation method they call \"sample enhancement\"\n\n8. Pitis et. al. MoCoDA: Model-based Counterfactual Data Augmentation. NeurIPS 2022. https://arxiv.org/abs/2210.11287\n\n\n9.  Lee at. al. GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning. Neurips 2024. https://arxiv.org/abs/2405.16907", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IYdFHrBmPB", "reviewer": "Reviewer_qbQm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\nThe authors present results that support their claims. Furthermore, the proposed approach is straightforward to apply without modifying any RL algorithms. \n\nWeaknesses\nThe authors want to improve generalizability but never define what generalization means. \n\nThe authors mention the ‘unique challenges of offline RL’ compared to online RL. However, they never explicitly mention these challenges since the proposed methods (individually) have already been applied to online RL. Especially since data augmentation techniques and data upsampling techniques are commonly applied in machine learning in general, the novelty of the manuscript mainly lies in the combination of the approaches and their application to offline RL. \n\nDid you do any ablations on the types of data augmentations? Did some help more than others? You claim that they “were empirically found to improve generalization.” Who determined this? And why did you choose these? \n\nIn 5.1.1, do all datasets include the FDD, or is it the same as before? I’m asking because suddenly, the upsampled performs much better than the augmented (Figure 5a) compared to Figure 4 a. A similar switch can be observed in Figure 5b, and suddenly the results for all the other approaches have slightly different colors. This should use the same scale. If the datasets are the same (which they appear to be since the values in the table are the same), then most of Figure 5 and Table 2 repeat the majority of the results from Figure 4 and Table 1 with the exception that there may be a potential switch between data entries. The bottom part of Table 2 also provides no additional information. \n\nOther Comments\nRisk-averse behavior can also be regarded as a feature rather than a bug of offline RL since safety is a concern for real-world applications. \n\nSince you change the dataset size, this will also result in additional computational costs for the entire dataset for training. Did you observe that your approach led to faster convergence? And did you experiment with different dataset sizes? \n\nWhy did you choose DrQ+BC and CQL and not any other algorithms? \n\nFor V-D4RL the test set distribution seems to be closest to the Easy setting (according to the JS divergence in Figure 4) or the Medium setting (according to Table 1). Is this the case? \nThe authors raise an interesting question at the end of section 5.1.1 but don’t address it any further. Especially how do you know exactly what the test distribution will be?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose data augmentation and upsampling approaches to artificially increase dataset sizes for offline RL. The authors' proposed approach only focuses on the dataset, so any offline RL algorithm can make use of this technique. The authors test the approach on two benchmarks using continuous and discrete action spaces, respectively.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Page 2, 1. row: No space before Our\n\nPage 2, first row in 2.1: ‘learning interacting’ - should probably be interacting only\n\nPage 2, end of first paragraph in 2.1: ‘discounted return’\n\nPage 2, last sentence: ‘datasetf observations’ - typo + the verb seems to be incomplete\n\nPage 3, second row: No space before Unlike\n\nPage 3, second row in 2.2: ‘Noiseoval’ - probably noise removal\n\nPage 4, 3.3 title: Why not call this: Step 2: Latent Space …\n\nEquation 5: Seems counterintuitive to use a ‘+’ here (I assume this stands for concatenation?)\n\nFigure 2: The yellow entry in the legend should be ‘Critic Network Training’ instead of twice Actor Network Training\n\nPage 5: Missing a period at the end of section 4.1\n\nThe generalization performance after equation 7 should be written as an equation as well. This will be much easier to follow. \n\nWhy is 5.1.1 a single sub-sub section?\n\nPage 6, second row: space between ‘generalization’ and comma. \n\nPage 6, second to last row of first paragraph: ‘we called Ours Fixed ..’ - should rather be something like ‘we refer to this as…’", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ftPf4J5etU", "reviewer": "Reviewer_JrFF", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The proposed data augmentation method does not require modifications to existing offline RL algorithms, making it easy to integrate into any visual offline RL framework to enhance generalisation ability.\n\n2. The combination of standard data augmentation techniques and latent space generation using a diffusion model is a new and effective contribution to the community, offering a robust method for synthetic data generation.\n\n3. The introduction of a new generalisation metric based on JS divergence to evaluate the distribution gap between training and testing data is both correct and plausible, providing a reliable way to assess whether the policy produces OOD actions that may lead to OOD states.\n\n4. The experimental results are overall convincing, and the inclusion of detailed algorithm descriptions enhances the reproducibility and transparency of the work.\n\nWeaknesses\n\n1. The paper only employs three commonly used data augmentation methods in the first stage, without exploring potential novel augmentation techniques or analyzing the relative contributions of different methods. This limits the novelty and depth of the work.\n​\n2. While the diffusion model is used to generate latent vectors for augmentation, a comparison with methods that directly generate states using diffusion models would better demonstrate the benefits of latent space augmentation.\n\n3. Although the goal is to synthesize diverse states in transitions, the paper does not experimentally investigate whether incorporating information from actions or rewards could further improve the synthetic data’s effectiveness for policy generalisation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a new data augmentation method tailored for ​visual offline reinforcement learning (RL), addressing the challenge of generalisation to unseen states. In offline RL, the absence of online exploration often leads to policies or value functions that fail to generalise beyond the pre-collected dataset. This issue is exacerbated in visual offline RL due to the presence of noise, distractions, and spurious correlations in image-based states. The proposed augmentation method specifically targets this problem by assuming that unseen states primarily arise from modifications to the states in the pre-collected dataset.\n\nThe augmentation algorithm operates in two stages: 1) Techniques such as image rotation, color jittering, color cutout, and background image overlay are applied to augment the original dataset, enhancing its diversity and robustness, 2) A diffusion model is trained to approximate the high-dimensional latent vector distribution of state pairs in the dataset transitions. \n\nThe proposed method, which combines standard image augmentation with diffusion model-based latent vector generation, is validated across both ​continuous and ​discrete action spaces. Experimental results demonstrate that the synthetic data generated by this approach significantly reduces the generalisation gap at test time, thereby improving the performance of policies and value functions in visual offline RL tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, the proposed augmentation method is ​correct and the experimental results are ​convincing. While the method’s validity and empirical success are clear, the inclusion of additional experiments, as suggested in the Weaknesses section, would further strengthen the paper’s contribution and address potential concerns. However, the absence of these experiments does not undermine the core contributions of the work, as the augmentation method itself is well-founded and the results are compelling. So while the additional experiments would enhance the paper, they are not a ​major issue for acceptance.", "broader_impact_concerns": "N.A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)