GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning
Authors: Jaewoo Lee
Sujin Yun
Taeyoung Yun
Jinkyoo Park
Venue: NeurIPS 2024 poster (2024)
Keywords: Offline Reinforcement Learning
Data Augmentation
Diffusion Models.
Primary area: reinforcement_learning
TL;DR: Novel data augmentation method for offline RL using conditional diffusion model.

Abstract: Offline Reinforcement Learning (Offline RL) presents challenges of learning effective decision-making policies from static datasets without any online interactions. Data augmentation techniques, such as noise injection and data synthesizing, aim to improve Q-function approximation by smoothing the learned state-action region. However, these methods often fall short of directly improving the quality of offline datasets, leading to suboptimal results. In response, we introduce GTA, Generative Trajectory Augmentation, a novel generative data augmentation approach designed to enrich offline data by augmenting trajectories to be both high-rewarding and dynamically plausible. GTA applies a diffusion model within the data augmentation framework. GTA partially noises original trajectories and then denoises them with classifier-free guidance via conditioning on amplified return value. Our results show that GTA, as a general data augmentation strategy, enhances the performance of widely used offline RL algorithms across various tasks with unique challenges. Furthermore, we conduct a quality analysis of data augmented by GTA and demonstrate that GTA improves the quality of the data. Our code is available at https://github.com/Jaewoopudding/GTA

GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning
Jaewoo Lee' Sujin Yun
Taeyoung Yun' Jinkyoo Park
KKIIT Omelet
(jaewoo, yunsj0625, 99yty, jinkyoo (ackyyo..rrk)).... Ckaist. ac kr.
Abstract
Offline Reinforcement Learning (Offline RL) presents challenges of learning effee- tive decision-making policies from static datasets without any online interactions. Data augmentation techniques, such as noise injection and data synthesizing, aim to improve Q- Q-function approximation by smoothing the learned state-acts action region. However, these methods often fall short directly improving the quality offline datasets, leading to suboptimal results. In response, introduce GTA, Generative Trajectory Augmentation, a novel generative data augmentation approach designed enrich offline data by augmenting trajectories to be both high-rewarding and dynamically plausible. GTA applies a diffusion model within the data augmen- tation framework. GTA partially noises original trajectories and then denoises them with classífier-free guidance via conditioning on amplified return value. Our results show that GTA, general data augmentation strategy, enhances the per- formance of widely used offline RL algorithms across various tasks with unique challenges. Furthermore, we conduct quality analysis of data augmented by GTA and demonstrate that GTA improves the quality of the data. Our code is available at
comp www./aawoopudddingg/oooooooooo
1 Introduction
Learning a decision-making policy through continual interaction with real environments is challenging when online interaction costly or risky. Offline Reinforcement Learning [Offline RL, [ emerges as a solution, focusing on training effective decision-makie making policies with static dataset gathered through an unknown policy (2). However, offline data often does not provide enough coverage of state-action space, resulting in extrapolation error addressed as overestimation of Q-value [3] 4 To mitigate extrapolation error, many previous works rely on adding explicit regularization terms 10 and have shown significant progress.
Moving beyond these mainstream methodologies, there exists underexplored approach, data augmentation methods: traditional data augmentation and generative data augmentation. Traditional augmentation methods [2] inject minimal noise to the state, preserving environment dynamics. Generative data augmentation [13 [4] builds a data synthesizer with a generative model to upsample offline data. Employing generative data augmentation approaches broadens the support of the data, thereby improving Q-function approximation [12).
While previously proposed data augmentation methods enhance the performance of offline RL, they still have limitations. As illustrated in Figure [] traditional augmentation methods have difficulties in discovering novel states or actions beyond the existing offline data, limiting their role only to smoothing small local regions of the observed state. In the case of generative data augmentation, the reward distribution of generated data is constrained by the support of offline data, resulting in the
Equaa contribution authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Original Distribution
Begh-Rewadding Distribution
Original Datapoint
Augmented Datapoint
(a) RAD. S4RL
(b) SynthER
(c) GTA (Ours)
Figure I: Comparison of noise injection
generative data augmentation
13 and GTA.
generation of suboptimal data. These issues make the existing methods not align with the objective of offline RI., which aims learn the most effective decision-making policy from the static dataset.
We introduce Generative Trajectory Augmentation (GTA), a novel approach that applies the con- ditional diffusion model to data augmentation, aiming address the aforementioned limitations. GTA is designed to generate novel and high-rewarding trajectories while minimizing degradation of dynamic plausibility. Our approach consists of three main stages: (1) Training conditional diffusion model that generates trajectory- levell data, (2) Augmenting the offline data via partial noising with diffusion forward process and denoising with amplified return guidance, and (3) Training any offline
RL algorithm with augmented data.
We train a conditional diffusion model that approximates the conditional distribution of trajectory given its return. Once the diffusion model trained, we sample trajectories from the offline dataset and partially noise the trajectory with the diffusion forward process. Then, denoise the noised trajectory with amplified return guidance, directing trajectories to the high-rewarding region. To this end, we can orthogonally integrate data augmented by GTA into any offline RL algorithm without any modification.
Our contributions. In this paper, we introduce GTA, novel data augmentation framework that utilizes conditional diffusion model generate nigh-rewarding, novel, and dynamically plausible data. Through extensive experiments commonly studied benchmarks [15 [6]. we demonstrate that GTA significantly improves performance across various tasks with unique challenges, such as sparse reward tasks and high-dimensional robotics tasks. We thoroughly examine the impact of core design components of GTA, trajectory generation, partial noising with diffusion forfisa process and denoising with amplified return guidance. Furthermore, we assess the GTA augmented dataset with data quality metries, proving its alignment with the objective of offline RL. These findings underscore the capability of GTA to efficiently augment trajectories, resulting in high-quality samples and improving the performance of offline RL algorithms.
2 Related Works
Data Augmentation for Reinforcement Learning. In Reinforcement Learning (RL), data aug- mentation enhances sample efficiency and improves Q-function approximation. In pixel-based RL. methods like CURL [17], RAD II and DrQ [8] have leveraged image augmentations such as
of RL. For the proprioceptive observations. S4RL [22 introduces variations into states by adding small-scale noise or adversarial gradients of Q-function under the similarity prior, that similar states yield similar rewards. AWM [19] improves robustness of augmented data by applying simple transformations to the learned dynamics model, enabling zero-shot generalization to unseen dynamics without requiring multiple test-time rollouts.
cropping and translation address sample-efficiena
Recent strides in generative models have led to their adoption in generative data augmentation, as seen in SynthER [3], MTDiff-S [14. and PGD 20 Among these, GTA, MTDiff-S. PGD, and SynthER all generate synthetic data using diffusion models but with distinct approaches. While GTA, MTDifT-S, and PGD generate the trajectory level, MTDiff-S is designed for multi-task settings, focusing on generating trajectories for unseen tasks. PGD generates synthetic trajectory with classifier guidance of policy, similar to model-based offline RL. GTA, on the other hand, focuses on generating high-rewarding trajectories using return guidance. Focusing single tasks, GTA and SynthER differ in what they generate. SynthER generates individual transitions, in contrast to the trajectory-level generation of GTA. This makes GTA compatible with any kind of offline RL model. Additionally, GTA initializes augmentation from the original trajectory, utilizing a conditional diffusion model, while SynthER starts augmentation from the Gaussian noise without conditional guidance.

---

Diffusion Models for Offline Reinforcement Learning. In focus on the diffusion planners, Diffuser 2T] devises the RL problem as trajectory-level generation with cumulative reward classifier guidance. Following this, the Decision Diffuser [22] replaces the classifier guidance of the Diffuser with classifier-fres guidance [23]. Adaptdiffuser 24 alternates between generating trajectories with
diverse reward functions and training the model using self-generated trajectories. However, diffusion planners require extensive time to sample actions, making their practical application challenging. GTA shifts the computational burden of the diffusion model from the decision-making step to the data-preparation step. This transfer allows GTA to leverage the advantages of diffusion models while avoiding extensive time costs during decision-making. making.
Model Based Offline Reinforcement Learning. Model- Offline RL focuses on learning environment dynamics and reward functions from data, creating simulated trajectories to train both the critic and policy (21-28] While both GTA and model-based offline RI. generate synthetic trajectories, in the case of GTA, key difference is that data generation and policy learning are separated and not done in an alternative cycle. This separation ensures that the quality of generated data is not affected by the training progress of critics or policies. It also mitigates the accumulating errors associated with single-step dynamics rollouts [22
3 Preliminaries
3.1 Offline Reinforcement Learning
Reinforcement Learning (RL) modeled with the Markov decision process (MDP) described by the tuple (S, A, T, R, 7). consisting of state space S, action space A, transition function T S x A - S, reward function R: S A S R. and discount factor of future reward y E [0. 29 At each timestep t, the agent selects an action ar according to the policy и given the state st. Consequently, the agent receives reward r, for the action a, taken in the state st, leading to the next state s s1+1. The goal of RL is to learn policy π²,, which maximizes expected discounted return, ((T) E, 12t= rrre).
In the offline RL setting, can only have access the fixed dataset D, which has been collected using unknown behavior policy тz. With insufficient and suboptimal offline data, offline RL aims to learn effective decision-making policy that surpasses the behavior policy.
3.2 Diffusion Models
Score-based diffusion models. Diffusion models are family of generative models that approximate the data distribution p(æ) with (o()). When considering the data distribution as p(æ) and the standard deviation of noise as then the distribution of data with added noise is denoted p(æ; o)).
The diffusion reverse process involves sequentially denoising from noise K that randomly sampled from a Gaussian distribution (00 I) following noise level sequence оg a max σK-1 > Oo = O. Consequently, the endpoint x. of this process aligns with the original data distribution.
Considering the probability flow ordinary differential equations (probability flow ODE), noise is continuously added to data during the forward process and reduced in the reverse process. By scheduling the noise level at time k, represented as o(k), the reverse probability flow ODE is formulated as follows [30]:
dx = - log okk))dk
(I)
where Vz log p(æ; (k:) denotes the score function, signifies the direction towards the data for a given noise level, and the dot represents the time derivative. The score function Va log p(æ:; a (k)) is trained via denoising score matching. The denoising score matching loss for given denoiser Do(2;; a)) is given by
C(De; o) -E.-p.c-N00.= 1y Do + c; - x11
(2)
When the denoiser Dg (ax; a) is optimally trained, the score is calculated as
V logp(z; = (Do(z; x)/0².
(3)
We sample data via solving Equation I with the learned denoising network.
3

---

Stgge A: Train Diffoiion Model
Stage B: Angment Trajuciory Dana
Stage C: Tram Offinne RL.
AHUM
Trajectory Generation
Co 5p
TD3BC
IQL
@ @
?
Denolinng Nerwork
Partial Noising
g a p
Amplitied Return Conditioning
MCQ
200 ',f:5]'
Foont Pasceed
r
Reweighted Sampling
Offinee Dustanel
Angmenitd Daizeat
D
DA
Figure 2: Overall framework of the GTA comprises 3 major stages. In the first stage, we train a conditional diffusion model designed for generating trajectories. Following this, We perturb the original trajectory and subsequently denoise it using the trained diffusion model, conditioned by amplified return. Lastly, employ the augmented dataset to train various offline RL algorithms.
Conditional score-based diffusion model. In the domain of conditional diffusion models, two primary strategies are recognized: classifier guidance [31] and classifier- free guidance (23 Classifier free guidance sets its guidance distribution po as po (a: yy) x po(z)yy) po (y/ x) Subsequently, using the implicit classifier po po(ylaz 2)) x po (2l)///p / (a:) and the equivalence relationship between score
matching and denoising process, described as V. log po xx ep (a. y) 32 23), the classifier free
guidance score éy is formed follows:
c (y) = (w + 1) cs .)) w ea ca Ø)
(4)
where w controls the strength of the guidance. The training objective of the classifier free guidance is to concurrently train the conditional score function and the unconditional score function as follows:
[||c co (x*, B)y + 32) [1]
(()) E,.
(5)
where y condition and B Bern(A) (A) is binary variable with dropout rate of condition y.
4 Method
In this section, we introduce GTA, Generative Trajectory Augmentation, which leverages the conditional diffusion model generate high-rewarding, novel, and dynamically plausible trajectories for augmentation. As shown in Figure 2 our method is divided into three stages: (1) Train a conditional diffusion model the trajectory level. (2) Augment trajectories using partial noising and demoising frumework with amplified return guidance. (3) Train any offline RL algorithm with the augmented dataset. We will discuss how each component works and their roles in effective data augmentation.
4.1 Stage A: Train Diffusion Model
Trajectory-leve generation. The diffusion model for GTA is designed generate the subtrajectory T. This subtrajectory is a consecutive transition sequence of the state, action, and reward sampled from a trajectory (81, a1, 71.... ... ST, 0T, r.). We represent subtrajectory 7 with horizon H follows:
St 8t-1
St+H-1
04 0++1
0++H-1
(6)
Pt Ttt1
Tt+
We train a conditional diffusion model to approximate the conditional distribution ((T(y(7)) with offline dataset, where the condition y(T) = Σ - denotes the sum of the discounted return. The parameter 0 is updated maximize the expected log- likelihooh of the data:
6* f arg max Ey-D [log po po(T)y(T)))

---

Using diffusion model to generate trajectories offers multiple advantages. First, the diffusion model leverages sequential relationships between consecutive transitions while generating trajectories. allowing it to minimize the degradation of dynamic plausibility. Second, the diffusion model captures long-term transition dynamics, which beneficial for environments with sparse rewards.
Diffusion Model Implementation. Integrating sequential dependencies within the subtrajectory into the model architecture is essential for trajectory-level generation. Therefore, we choose denoising network that combines Temporal-Unet [2I] with MLP-mixer [33] to exploit both local and global sequential information. We adopted the Elucidated Diffusion Model (30,, known for its powerful performance in recent work 13 The implementation details are elaborated in Appendix A.2
4.2 Stage B: Augment Trajectory-level Data
We propose a novel approach to augment trajectories, partial noising and denoising framework with amplified return guidance. Partial noising modifies the original trajectory using the forward process of the diffusion model, thus providing exploration opportunities. The exploration level is adjusted by the nosing ratio μ.. Following this, denoising with amplified return guidance refines the noised trajectory. During the denoising process, we guide the trajectory towards reghrrewarding regions, promoting the exploitation of learned knowledge about the environment. We introduce the multiplier for conditioned return a to control the exploitation level. Figure 3 augmentation via partial noising and denoising framework.
outlines the principle of data
Partial noising with forward process.
Original Trijecncy
Munifold of Feavible Trajectories
Noiiin
Let r = T represent the original trajectory, and k denotes the diffusion timestep. We in- troduce a noising ratio 1(0 < p K S 1) to determine the extent of erasing informa- tion of the trajectory for exploration. We add noise to the original trajectory 70 cre- ating noised trajectory denoted as Tl- ~ N(T:7 (u I)).). The parameter μ controls the level of exploration. Small μ results in minimal exploration, thereby pre- serving much of the original information of the trajectory. Large i facilitates broader exploration, potentially leading to generat- ing novel trajectories while losing original information significantly.
Denoising Process
High Revanding
Origiaality Procrving Region
11
7"2
Figure 3: Mechanism of the partial noising and de- noising framework. The extent of exploration increases with p (μ) < #2 < Ha). During denoising, amplified return guidance shifts trajectories towards the high- rewarding region.
Denoising with amplified return guidance.
After noising the trajectory, we reconstruct the trajectory with classifier-free guidance to push the trajectory towards the high-rewarding region enhance optimality. We introduce amplified return guidance, which sets the conditioning value as the multiplied return of the original trajectory. It can prevent adverse effects that occur when significantly larger return values are conditioned during generation. Further analysis on this phenomenon is claborated in Appendix F.]
The amplified return is formally defined as follows:
0(7 .)) a
T
(8)
where a is the control parameter for the exploitation level. We set a > 1 to make the conditioning value higher than the return of the original trajectory. a close to I induces mild conversion towards high-rewarding trajectory region, while large a promotes significant drift, substantially exploiting the diffusion model. Our partial noising and denoising framework can be summarized as follows:
Partial Noising (7 + -. 7 poised
(9)
Denoising: (T - ... - 40 T denoised)
(10)
5

---

4.3 Stage C: Offline RL Policy Training
The final stage of GTA utilizing high-quality trajectories generated through previous stages for policy training. The trajectories generated by GTA can seamlessly integrate with existing offline RL methods. By introducing GTA, we can orthogonally integrate the expressiveness and controllability of diffusion models with existing offline RI. algorithms.
4.4 Additional Technique: Reweighted Sampling
To concentrate on samples in high-rewarding regions, we additionally adopt a reweighting strategy during the sampling process. This approach prioritizes the sampling of subtrajectories with higher returns. Consequently, augmented trajectories are more densely distributed in high-rewarding regions. The complete specification of our reweighting strategy, including implementation details and parameter configurations, is provided in Appendix A.3
R4 35
5 Experiments
In this section, we present extensive experiments conducted across commonly studied benchmarks [15 [6] to evaluate the augmentation capability of GTA. Through experiments, we aim address the following questions: 1) How much performance gain does GTA exhibit across various types of tasks? 2) What impacts do the design elements of GTA have on performance? 3) Are trajectories augmented via GTA high quality? 4) What is the preferable μ and a setting for the new tasks?
5.1 Experimental Setup
Datasets and environments. We demonstrate the versatility of GTA on the D4RL benchmark from the standard continuous control tasks to the challenging tasks with unique difficulties. validate that GTA enhances offline RL under the sparse rewards environments and overcoming high-dimensional, complex robotics tasks. Additionally, we demonstrate that GTA is effective with suman-demonstratee small-scale datasets, showing its adaptability to realistic settings. Finally, we extend GTA to pixel-based environments, specifically within VD4RL benchmark [16].
115.
Data augmentation baselines. We compare GTA with existing augmentation methods. For tra- ditional augmentation, we choose S4RL [12], which introduces Gaussian noise into states. In the domain of generative augmentation, we choose SynthER 13]. which employs an unconditional diffusion model for transition-lever generation.
Offline RL algorithms. To demonstrate the general efficacy of GTA proprioceptive observations, we select four widely used offline RL algorithms: TD3BC (7]. CQL [8] IQL. (36], and MCQ 44. For particularly challenging tasks, such as Maze2d, Antmaze, Adroit, and FrankaKitchen, we employ IQL, which provides all hyperparameter configurations across all tasks and consistently demonstrates stable performance. For pixel-based observations, we utilize DrQ+BC as the bascline algorithm, as established in the VD4RL benchmark by Lu et al. [16].
Data quality metrics. We propose data quality metrics analyze whether GTA provides high-quality data. Following prior works 13 37), we conduct a thorough analysis with three metrics: oracle reward, novelty, and dynamic MSE. Oracle reward, computed with the true reward of generated data, represents optimality. Novelty measures the ability of the augmentation method to discover novel states and actions not existing in the offline data. Finally, dynamic MSE evaluates how well the generated trajectories adhere to the dynamics of the environment. Formally, dynamic MSE and novelty are defined as follows:
Dynamic MSE(DA)
1
Σ
(s) 8'2
(11)
[DA]
Novelty(DA, D) [D (s.,,
Σ
min (.,.F,)")
((s, (), - (8,a))
(12)
where DA represent the augmented dataset and original offline dataset. (s,a, denotes single transition, and denotes the true dynamic model of the environment.

---

Table 1: Normalized average scores Gym locomotion and maze tasks, with the highest scores highlighted in bold. Each cell displays the mean and standard deviation across 8 seeds.
Alzo
Helfehetthh repla
molliate expert
modian
Hopper recdier ceplay
madlian
expert
WolkerBd mediureerppl repla
mallian espert
Nosc
3.30
804408
2,00
90.75
110:21
77 24
SyILER
413
2,39
45.57
9:3.4
$5.47
62
= 3
76.51 13:30
95,99
11.21
$5.45
9067 8: 1.30
106.55 2.33
78
2:330
CTA
57.34
=2 0.51
50.04 =2 0364
93.83
3.7
09.57
4.85
$9.31
16354
110.40
4.84
86.00
6.99
93.82 8.74
110.86 0.34
$4.63 2:2.0
CQU
SAHIER
S4ILL
47.00
47:21
5:14
5.23
44:82
46.05 1:040
8:0 42
13. 09:399100
95.23 29
0345
1
14.05 0 6:75
12.72 8 7:40
12.14
78:32 8 9.89
17:40
168.51
001.56
3.3.5
3.00
$1.91
$1.46 2.28
0.33
9081 6 6.35
$5.01 3:303
196.55
100.16
0:33
0:332
91: 11:20-0 0: 1,46
09.87-
1.42
0.31
$1.36
0.27
94.33
5.21
74.20 7.42
98.99 1.55
110.90
$0.40 4.55
91.57
5:315
110.44 0.36
85:27
1.83
48.05
0.19
45.35
1.0.5
94..5 1.65
16.35
7.09
05-76 4.0
01.69
35.07
$6.36 1.31
69
10.30
112.31
0.000
13 51 3.552
IQL
SynthEL
S4IL
68.5%
0.330
66.91
43.57
6.055
0.26
54.22
91-141 3.75
1.50
16. 5.64
21 4.55
101.77
06.12
23.01
04.08
19.82
51.44
3.19
90.15
64.58
16.17
6.36
9061
70033
à 7.90
112.20
112.12 0.043
0.070
51:97 3.21
73.35 3.255
Nova
10.96
0.73
14:00 9.996
$8.51 9.45
7337 1 9.57
101 06 63
00.85 12.35
10:70- 716
97.19 + 1884
114.55 3,10
54: 17 1.30
M03
S1R1.
N1.55 0-88
51:12
11-12-1-0.4 1:3.4
19.00:1 14.15
71:55:1 10:54
171 1.44
14
90:11
85:00 1-544
107.6111
81 69
CITA
0.91
19.00
104.24 1.90
111.53
55.90 1.85
Algo
mediom
Marr26
1ang
medtinggty
torg-glay
AntManee
medium- -Bserw
bege-dhore
9.16
IQL
SydTER
$4R1
39.00
2.36
34.82
01.76 01 4.51
62:33
55,00
17 665
41
80.85
1038
37 50 k.d
42.88
33:84 + 11:3
51.63
40 14:5
14,00 4:12
37 50 8.30
53 3.22
35 99
- 2.00
CTT
41.00
1.41
37.78 = 1.66
76.56 4.4
06.00 KM
$1.88 4:15
41.38 4.66
$7.80 9,51
78 738
47.75 6.0
62.75 1.03
Table 2: Normalized average scores on complex robotics tasks, with the highest scores highlighted in bold. Each cell displays the mean and standard deviation across 8 seeds.
Adroit
Franka mixed
Algo.
Aug.
peu-human
door-human
Average
partial
complete
Average
None
69.52 1: 5.48
3.34 4: 1.16
36.43 :: 2.97
38.06 3.15
54.88 i: 2.68
$7.75 4.33
50.23 :: 1.45
IQL
S4RL
72.52 5.79
3,22 1 0.80
37,87
: 3.22
36.78
2.30
54.25
1: 3,26
55.06
1: 3.52
48.7 4: 1.40
SynthER
72.13 4.48
3.77
: 0.72
37.95
2.13
37.38
1.55
56.13 0.61
59.04 + 6,98
50.85 6.30
GTA
76.11 + 9.54
9.35
1448
42.73
4.26
45.91 = 6.41
56.22 1 1.88
$7.78 3.58
53.3 :: 3.23
5.2 Benchmark Evaluations
We conducted experiments to evaluate whether GTA can provide a performance boost when applied alongside existing offline RL algorithms in various tasks with its own unique challenges.
Gym locomotion. We experimentally demonstrate on Table that offline RL algorithms with GTA outperform all other baselines across all algorithms on the average score with statistically significant margin. This result highlights the versatility of GTA with various offline RL algorithms. We provide p-value for our experiments to validate the statistical significance of the results in Appendix F2
Maze tasks. To assess GTA in sparse reward tasks, we test GTA on Maze2d and AntMaze tasks. Table D reveals that GTA notably enhances the performance of IQL policy on the sparse reward tasks While SynthER often degrades the performance. This result suggests two insights. First, the trajectory-level generation of GTA helps capture long-term dynamics, effectively leveraging that information while augmenting sparsely rewarded trajectories. Second, augmenting the dataset with high-rewarding trajectories strengthens the goal-reaching ability by enriching the demonstrations with more successful trajectories.
Complex robotics tasks. We evaluate the effectiveness of the GTA on realistic, high dighddimessionll, challenging control tasks. Adroit-human datasets comprise 25 gamaan-genrrted trajectories. The FrankaKitchen dataset, which involves multitasking behavior trajectories, requires generalization ability to stitch trajectories. According to the results in Table 2 GTA effectively boosts the per- formance in nigh-dimensional, complex robotics tasks. The results on the Adroit-human dataset demonstrate that GTA effectively augments small-scale human demonstration data through the ex- pressiveness of the diffusion model. Additionally, the performance improvements in kitchen tasks indicate that GTA aids offline RL in trajectory stitching 21
Pisel-based Observations. We extend our approach to paxel-based observations by following the experimental setup detailed in [3. Initially, we pretrain the policy and extract its visual encoder. Subsequently, we project offline pixel-based observations into the embedding space, followed by augmenting these embedded states with GTA. More detailed experiment setups are elaborated in Appendix C.4 The results presented in Table 3 show performance improvements on DrQ+BC across the environments and dataset qualities. indicates that GTA can be further extended to pixel-based observations beyond proprioceptive observations.
7

---

Table 3: Normalized average highlighted in bold. Each cell displays the mean and standard deviation across 6 seeds.
txee--based observation tasks, with the highest scores
Algo.
Aug.
medium
medium replay
Cheetahb-an
medium- expert
expect
medium
medium replay medium expert
Walkerr-walkk
expent
Average
None
53.3 + 3.0
44.8 36
50.6 8.3
54.5 833
40.1 :M1
13.4 =40
43.3 14.1
65.3 1:5
42.7 #4.1
DrQ+BC
SynhER
50.4 1:1.
25.6 2.9
36.8 6.60
11.9 1:
29.1 24.
7.6 #2.
30.2 =2.
28.7 233
21.5 23.3
GTTA
$3.3
38.1 2.5
61.0 9:2
46. 82
38.0 $50
13,6 =15
45.3 =5.
67.4 110.
45.4 :5,7
μ 0.25
==
0.75
1.0
Anglitedd concitionnss
Fissi conditziing
Cormallzzell scort
Probubility demityy
Urroediinnig
Offine duta
Normalized
60-
8.22
0.11
0.7
Train steps
Train steps
Ornile rewands sum of subtrajectory
60
(b)
(c)
Figure 4: (a), (b) D4RL normalized score across different noise levels over the course of training TD3BC on halfcheetah-mediums and saachheetah-meiuu---ppeee (c) Comparison oracle reward sum of subtrajectory between conditioning strategy halfchectah-mediume
5.3 Ablation Studies
We carry out thorough ablation studies to assess the effectiveness of each component within GTA. These studies are evaluated based on the performance scores of D4RL tasks and the aforementioned data quality metrics.
What are the benefits of trajectory-level generation? To investigate the impact of utilizing sequential relationships through trajectory-level generation, we examine performance differences as the trajectory length varied. We find that when H = 1, the dynamic MSE becomes five times or more higher compared trajectory-level generation with H longer than 16. Additionally, when training TD3BC with augmented data, the normalized score increased by 21% with trajectory generation compared to transition generation, where H = 1. This result highlights sequential relationship between consecuive transitions in trajectory does help generation, minimizing degradation of dynamic plausibility. Detailed experiment results are in Appendix E.
Is preserving original information via partial noising essential? We conduct experiments by varying the noising ratio on datasets of different optimality explore the effect of the exploration on performance. In Figure 4a seffcheetaa-meddium higher noise levels enhance performance, while in Figure 4 nalfcheetah-medium-eee medium-expert, optimal performance occurs noise level of μ = 0.25, with declines at higher levels. We speculate that if the original data already contains high-rewarding trajectories, even minimal modifications can highly effective, and excessive exploration might lead to unexpected outcomes. Therefore, preserving information of original trajectory by partial noising crucial achieving effective data augmentation while avoiding potential negative impacts from excessive exploration. Additionally, the terminal signal, which may be very sparse, could be lost significantly if the original information is not preserved. We further conduct experiments about the impact of partial noising preserving terminal state information in Appendix F3
Does amplified return guidance improve the optimality? Figure 4 illustrates the superiority of the amplified return guidance on augmentation with respect to unconditioning [13), and fixed conditioning (22] Augmented trajectories without any condition tend to replicate the original reward distribution, thereby failing to enhance data optimality. Fixed conditioning concentrates rewards near maximum return offline dataset. Notably, amplified return conditioning demonstrates a superior ability to cover broader range of rewards, especially on unseen high rewards, while leading on the average reward and D4RI. score. Further analysis of the impact of the conditioning method on data quality can be found in Appendix F.]
8

---

Oracle Reward
Novelty
Dynamic MSE
bellcaotebreee
Lalichctabmne
suppereter
happeritt
lopparytte
Ielfblessbbnn
Talobhetikk-r it
halkfecthh.c a
hepprres c
hupporn 2
hepperimi
wille-tte
valkerm...
S4RL
SynthER
GTA
Figure 5: Data quality analysis of S4RL, SynthER, and GTA. GTA augmented data exhibits superior optimality and novelty across gym locomotion datasets while maintaining dynamic plausibility.
5.4 Data Quality Analysis
We introduce quality metrics specifically suitable for offline data analysis in Section We compare GTA with other augmentation methods regarding not only how much it expands the coverage of offline data but also its optimality and dynamic MSE. Figure metric across baseline augmentation methods gym locomotion tasks. Data augmented by GTA shows significantly higher oracle reward and novelty while maintaining a comparably low dynamic MSE. These outcomes indicate that the conditional diffusion model of GTA expands data coverage under dynamic constraints and discovers novel states and actions with higher rewards, effectively integrating environmental knowledge. We add further analysis in Appendix F.4
the relative scale of each
5.5
Guideline for Selecting μ and
Table 4: D4RL normalized score on locomotion environments with fixed a and The experiments conducted with TD3BC.
Mpp
mediomn
Hollchwtah modarmeppia
medom expert
modeam
Hopper maddan--mppo
modeam oxpert
modium
Walker2e medier-nplay
mediam coper
Aserage
Nass
42.42 0.62
44.64 0.71
50 a 5.50
60.44
4:1.8
63.09 24.1
106.56 5.81
$4.59 6 1.2
B4.13 4.12
110.23 2 0.57
76.92 s 2:50 23544 2:18
SyuthER
0.30
4557 0.34
65.47
11.35
63.20 2:300
78.11 = 15,00
$8,99 11:27
$5.43
90,7 1.56
19955
0.32
66.15
Table 5: D4RL normalized score medium and medium-replay quality locomotion environments with fixed a and μ.. The experiments are conducted with TD3BC.
Algo.
Halfcheclah
Medium Hopper
Walker2d
Hulfcheeiah
Medium-replay Enviromment Hopper
Walker2d
Average
None SynthER
48.42 + 0.62
61.04 + 3.18
84.58 + 1.92
44.64 0.71
65.69 + 24.1
84.11 4.12
64.75 = 4.15
49.16 0.39
63.70
3.69
85.43
1.14
45.57
2: 0.34
78.81 : 15.80
90.67
1,56
68.89 2.62
GTA (1 0.5, 1.3)
57.92
68.46 1.32
88.38
2.70
48.23
77.17 + 22.17
91.12 6.45
72.17 = 11.23
GTA 0.75, 1.3)
57.85 = 0.27
61.58
5.00
87.14
17.14110
1.73
48.99 1: 2.16
97.26 2 3,38
90.89
3.29
79.05 = 2.90
In this section, propose a guideline for selecting hyperparameters based on dataset quality and environment characteristics. First, when the known information about the dataset is limited and minimizing exploration risk is essential, low exploration and exploitation approach can enhance performance without losing stability. As illustrated in Table 4 GTA with low exploration and exploitation (μ 0.25, 1.1) surpasses baseline methods in gym locomotion tasks.
However, if know that the offline dataset is suboptimal and there exists considerable room for improvement, a higher exploration and exploitation setting can yield superior performance by uncovering broader range of high-quality behaviors. fable/5sshow that GTA with (μ 0.5, a 1.3) and (μ 0.75, a = 1.3) significantly boosts performance on suboptimal datasets. These findings indicate that while conservative parameters with low μ and a offer stable gains, we can elevate values of and a by leveraging prior knowledge about dataset quality to achieve superior outcomes. Notably, GTA consistently outperforms baselines across all configurations without dataset-specific hyperparameter tuning.

---

5.6 Futher Experimental Results
We conduct extensive additional experiments explore the further potential of the GTA.
GTA under realistic settings. We examined the augmentation ability of GTA for two realistic settings: one with a mixed dataset consisting of few expert datasets and the majority of low trajectories in Appendix F.2 and the other with a small amount of offline dataset in the Appendix [E.3 In both cases, We observed that GTA significantly enhances the performance, improving sample efficiency of the offline RI..
Larger batch size and training epochs. Since GTA offers more training data, it is worth exploring whether increasing batch size or training epochs can lead to additional performance gain. The results in Appendix G.1 demonstrate that training epochs while baselines do not benefit from the batch size and training epochs increment.
enhance the performance by increasing batch size and
Extending to sequence modeling approach. We proved that GTA can be applied to sequence modeling approaches such as Decision Transformer [38] as GTA augments data at the trajectory level. We confirmed that GTA improves the performance of the sequence modeling approach, especially in sparse reward tasks. The detailed experiment setup and results are Appendix E.4
Sensitivity tests. We conduct experiments on the sensitivity of parameters a in Appendix G and in Appendix G.3 The sensitivity of GTA with the amount of augmented dataset is in Appendix G.4 Our results indicate that GTA maintains similar performance beyond one million augmented transitions, demonstrating robustness to the number of transition samples.
Additional ablations on design choices. We explore the impact of reweighted sampling introduced
on Section 4.4in in Appendix E.5 For exploring alternative design choices, we adopt conditioning-by: inpainting as an alternative conditioning approach instead of classifier-free guidance in Appendix [..6 We also test the model-based approach by replacing the reward of the generated trajectory with the prediction of the reward proxy model in Appendix E.7
Conclusion
We propose GTA: Generative Trajectory Augmentation, novel generative data augmentation approach for enriching offline data with high-rewarding and dynamically plausible trajectories. GTA combines data augmentation with the diffusion sampling process by partially adding noise to original trajectories and subsequently denoising them under amplified return guidance. Our extensive experiments on 31 datasets across environments exhibit considerable performance improvements in four distinct offline RL algorithms, demonstrating the versatility and effectiveness of the GTA. We show that GTA successfully creates high-quality datasets from the sub-optimal offline dataset, which leads to learning effective cadisionnmaking policy.
Limitations and future works. In the GTA framework, we do not train additional transition model and reward model for simple augmentation framework. Therefore, in tasks where dynamic violations have a critical impact, we may not expect significant performance boost. However, as shown in our experiments, dynamic violations are generally minimal. A Additionally, while the main focus of GTA on the offline setting, future work could explore off-to oof-io- and online settings.
Acknowledgement
We thank the anonymous reviewers for their insightful comments and suggestions, which signifi- cantly improve our manuscript. This work was supported by the National Research Foundation of Korca(NRF) grant funded by the Korea (MVI (No. RS-2024-00410082.,
10

---

References
[1] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. arXiv preprint arXiv:2005.01643 2020.
[2] Sascha Lange, Thomas Gabel, and Martin Riedmiller. Batch reinforcement learning. In Reinforcement learning: State-of-the-ars pages 45-73. Springer, 2012.
[3] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy q-learning via bootstrapping error reduction. Advances in Neural Information Processing Systems, 32, 2019.
[4] Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711 1-1171111724, 2022.
[5] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv:1911.11300 2019.
[6] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[7] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances neural information processing systems, 34:20132-20144 2021.
[8] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 13:1179- 1191, 2020.
[9] Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2 40, 2021.
[10] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:20066.09359, 2020.
[11] Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Reinforcement learning with augmented data. Advances in neural information processing systems, 33:19884 19895, 2020.
[12] Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4RL: Surprisingly simple self supe-suino for offline reinforcement learning robotics. In Conference Robot Learning, pages 907-917. PMLR, 2022.
[13] Cong Lu, Philip J Ball, and Jack Parker-Holder. Synthetic experience replay. arXiv preprint arXiv:2303.06614, 2023.
[14] Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang, Bin Zhao, and Xuelong Li. Diffusion model is effective planner and data synthesizer for multi-task reinforcement learning. arXiv preprint arXiv:2305. 188.005.884499 18459, 2023.
[15] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.072199 2020.
[16] Cong Lu, Philip J Ball, Tim GJ Rudner, Jack Parker- Holder, Michael A Osborne, and Yee Whye Teh. Challenges and opportunities in offline reinforcement learning from visual observations. arXiv preprint arXiv:2206. 04700, 2022.
[17] Michael Laskin, Aravind Srinivas, and Pieter Abbeel. Curl:
Contrastive unsupervised represen- tations for reinforcement learning. In International Conference on Machine Learning, pages 5659- PMLR, 2020.

---

[18] Denis Yarats, Rob Fergus, and Ilya Kostrikov. Image augmentation is all you need: Regular- izing deep reinforcement learning from pixels. In 9th International Conference on Learning Representations, ICLR 2021, 2021.
[19] Philip J Ball, Cong Lu, Jack Parker-Holder, and Stephen Roberts. Augmented world models facilitate zero-shot dynamics generalization from a single offline environment. International Conference on Machine Learning, pages 619-629. PMLR, 2021.
[20] Matthew Thomas Jackson, Michael Tryfan Matthews, Cong Lu, Benjamin Ellis, Shimon Whiteson, and Jakob Foerster. Policy-guided diffusion. arXiv preprint arXlv:2404.06356, 2024.
[21] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205. 199919 2022.
[22] Anurag Ajay, Yilun Du Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision arXiv preprint arXiv:2211. 15657, 2022.
[23] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
[24] Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdif- fuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877, 2023.
[25] Tianhe Yu, Garrett Thomas. Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 33:14129-14144 14142, 2020.
[26] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning. Advances in neural information processing systems, 33:21810-218 823, 2020.
[27] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline basel policy optimization. Advances neural information processing systems, 34:28954 28967, 2021.
[28] Mare Rigter, Bruno Lacerda, and Nick Hawes. Rambo-rl: Robust adversarial model-based offline reinforcement learning. Advances in neural information processing systems, 35:16082- 16097, 2022.
[29] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[30] Tero Karras, Mïla Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems, 35:26565-26577, 2022.
[31] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
[32] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1. 13456, 2020.
[33] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung. Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, al. Mlp- mixer: An mlp architecture for vision. Advances in neural information processing systems, 34:24261-24272, 2021.
[34] Aviral Kumar and Sergey Levine. Model inversion networks for model-based optimization. Advances in Neural Information Processing Systems, 13:5126-5137, 2020.
[35] Siddarth Krishnamoorthy, Satvik Mehul Mashkaria, and Aditya Grover. Diffusion models for black-box box optimization. arXiv preprint arXiv:2306.071800 2023,
12

---

[36] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-learning. arXiv preprint arXiv:2110.0619,, 2021.
[37] Minsu Kim, Federico Berto, Sungsoo Ahn, and Jinkyoo Park. Bootstrapped training of score-conditioned generator for offline design of biological sequences. arXiv preprint arXiv:2306.03111, 2023.
[38] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
[39] Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-orientee deep offline reinforcement learning library. arXiv preprint arXiv:2210.07105, 2022.
[40] Yihao Sun. Offiinerl-kit: elegant pytorch offline reinforcement learning library. https httphhhassnn1/////11112 12t//ff1ieff 2023.
[41] Cong Lu, Philip Ball, Jack Parker- Michael A Osbome, and Stephen J Roberts. Revisiting design choices in offline model-based reinforcement learning. arXiv preprint urXiv:2110004.335 2021.
[42] Moksh Jain, Emmanuel Bengio, Alex Hemaanezz-Gariia, Jarrid Rector-Brooks, Bonaventure FP Dossou, Chanakya Ajit Ekbote, Jie Fu, Tianyu Zhang, Michael Kilgour, Dinghuai Zhang, et al. Biological sequence design with gflownets. In International Conference on Machine Learning. pages 9786-9801. PMLR, 2022.
[43] Zhang-Way Seaag-WWiiin Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Harness- ing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint a30iv:2306. /3085, 2023.
[44] Bernard L Welch. The generalization of 'student's 's' problem when several different population varlances are involved. Biometrika, 34(1-2):28-33 33,, 1947.
[45] Kerong Wang, Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Boot- strapped transformer for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:34748-34761, 2022.
[46] Shitong Luo, Yufeng Su, Xingang Peng, Sheng Wang, Jian Peng, and Jianzhu Ma. Antigen- specific antibody design and optimization with reffusiionbbssee generative models for protein structures. Advances in Neural Information Processing Systems, 35:9754 9767, 2022.
[47] Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.0117 2021.
[48] Brandon Trabucco. Kyle Doherty, Max Gurinas, and Ruslan Salakhutdinov. Effective data augmentation with diffusion models. arXiv preprint arXiv2200.007444 2023.
13

---

Appendix
We provide further details of our paper in the appendix. Our code implementation can be found in httpe//ggthh...com/leesppddddd/p
Methodology Details
A.1 Subtrajectory Generation Details
In this section, we provide further details for our trajectory-level generation. generating a whole episode that includes more than hundreds of timesteps once is infeasible, we generate subtrajectory with horizon H, following prior works [21 22]. Each subtrajectory consists of a sequence of state, action, and reward.
Note that we need full enviromment transitions, i.e., states, actions, rewards, next states, and terminals (if MDP is finite) for training conventional Q-learning baaringg-based Offline RL algorithms. To achieve this, we actually model the subtrajectory length H + and use generated state 81, H for the next state of Hth transition.
8t
S141
H1+H-1
8t+H
dt+1
d++H-1
/
rt r++1
T+#H-1
{(t, Ot St+1, rt), (81+ S1 (++ Se
In environments like Walker2d and Hopper, which have terminal transitions, each trajectory may have different episode length. It is necessary to incorporate terminal information when setting the conditioning value for guiding generative augmentation. When comes to directly generating terminal value, modeling the terminal variable is hard due to its binary nature and sparseness, as pointed out [3]. The situation is worse in trajectory-Ievel generation as it should be forbidden that the terminal variable is set to I in intermediate transitions. To address this, follow the implementation [22] by adjusting the return calculation and generation process: we subtract 100 from the reward of terminal transitions. This modification allows for the encoding of terminal information without the need for binary variable terminal generation.
A.2 Model Architecture Details
We represent the denoising network model with temporal U-Net [22 with repeated residual blocks. We choose MLP-Mixer 33 architecture for modeling each block instead of temporal convolution layers. In MLP-Mixes we first concatenate each component (states, actions, and rewards) and project it into the MLP layer dimension-wind wise, then apply MLP across temporal locations and repeat the process. By doing this, we can efficiently capture both temporal and component-wise relationships.
A.3 Reweighted Sampling
We adopt reweighted sampling, which is suggested in 34 35
First, segment the condition y. which is the return value of subtrajectory, into Nn equal intervals. Each interval is weighted according to its value and the number of points in the interval, with the specific weight vy calculations applied as follows:
[B, By B +u
10
yby
exp
(13)
where i represents the maximum return value in the offline dataset DD |B denotes the count of data points in the jth bin, and yb, refers to the midpoint of the interval associated with bin By. In the equation, u and serve as smoothing parameters. The parameter y determines the weighting between high-score bins and low-score bins. Lowering & results in reduced weights for the low-score bins, and conversely, increasing g assigns higher weights them. The detail setting of the u and g is provided in Appendix C,3
14

---

B Hyperparameters
B.1
Diffussion Models
Size of the diff'usion model and training details. We use the same size of network and hyperparam- eters for modeling conditional diffusion model across all experiments for case of implementation. The hyperparameters we used for modeling and training are listed in Table The number of train- able parameters of our diffusion model is about 29. 29.6M, which is comparable with the size of the diffusion model used in SynthER = 25.6M. When using the default network model, training takes approximately 30 hours for 1M training steps with batch size 256 on RTX3090 NVIDIA GPU.
Table 6: Hyperparameter setting for backbone of diffusion model for GTA
Parameters
Values
Batch size
256
Optimizer
Adam
Learning Rate
3 x 10
Training
Learning Rate Schedule
Cosine Annealing Warmup
Training Steps
le6
Conditional dropout (A)
0.25
Trajectory sampling from the diffusion model. For the sampling process, we use the Stochastic Differential Equation (SDE) sampler suggested in EDM (30] with the default hyperparameter setting used in SynthER [13] As we employ classifier-free guidance, an additional hyperparameter guidance scale w has been introduced. We use the same guidance scale w 2.0 for all experiments. Detailed hyperparameters are listed in Table 7
Table 7: Hyperparameter setting of diffusion model of GTA for trajectory sampling
Parameters
Values
Number of Diffusion Steps
128
min
0.002
Omax
80
EDM
Schum
80
Smin
0.05
S Smax
50
Spoise
1.003
Conditioning
Guidance scale (w)
2.0
To sample 5M transitions, takes approximately 3.5 hours with 128 diffusion timesteps on a single RTX 3090 NVIDIA GPU. Note that our sampling time be reduced adopt the partial noising and denoising techniques. For example, with μ = 0.5, we only require 64 denoising steps and can accelerate the sampling process.
15

---

B.2 Detailed parameter settings in GTA
We provide hyperparameter search spaces for partial noising level μ,, multiplier a, horizon length H, and reweighting parameter used for GTA in Table g
Table 8: Hyperparameter setting for partial noising level μ and guidance multiplier
Environment
H
Reweighting
Locomotion
[1.1, 1.3, 1.4)
(0.5, 0.25)
[32. 128)
{Ø, .B 20, 50, 100)
Maze2d
[1.25, 1.3]
[0.5]
(32)
{c= 10)
AntMaze
(0.05, 0.1, 0.25)
(32)
{2, 10, 30)
Adroit
(1.1,1.3)
(0.1, 0.25)
(32)
{Ø]
FrankaKitchen
(1.1, 1.51
(0.1, 0.25, 0.5)
(32)
c 10)
Note that in sparse reward settings where the reward a binary variable, the reweighted strategy tends predominantly sample subtrajectories the goal location. It might not be helpful since makes the distribution of generative data cover a narrow region. To address this, we sample episodes instead of subtrajectories with reweighting sparse reward settings. We assign the weight times the bigger value to the successful episodes. Once the episode is sampled via weighted sampling, subtrajectories for augmentation is sampled from the uniform distribution. For dense reward environments like Gym locomotion, we put details of reweighting parameters in Table
Table Hyperparameter setting for reweighted sampling
Environment
Reweighting
NB
u
Halfcheetah maalichetta-mmedim--y
X
malichhctta-mmediu--rge
50
0.001
5.0
Halichetth-mmedium-eeprrr..
0
20
0.001
5.0
Hopper-medium-v2
O
50
0.001
5.0
Hopper-mediuu--rpplay-222
O
50
0.001
5.0
Hopper-medduum-exxpr--222
0
50
0.001
5.0
Walker2d-medium--22
100
0.001
5.0
Walker&2-meediun--ppaay--2
0
50
0.001
5.0
Walker2d Walker2d-medium
50
0.001
5.0
16

---

C Experiment Details
C.1 Data Augmentation Baselines
In this subsection, we describe the implementation details of the data augmentation baselines. Note that we try to follow the official implementation provided by authors, including hyperparameters, and re-im re-implement only when public code is unavailable.
S4RL 12: S4RL adds zero-mean Gaussian noise to imput states as augmented data, i.c, &t
8t +G ~ ((0, We choose a 0.0003, following the original paper.
SynthER [13 SynthER trains unconditional diffusion model that generates transition- level data (state, action, reward, next state, and terminals). We build SynthER based on code provided by authors" For a fair comparison. RI with a larger diffusion network for all environments and sample the same number of transitions. The diffusion model of SynthER contains 25.6M trainable parameters. We only use generated data from SynthER to train offline RL methods for evaluation, following the original paper. Their code opened http::tt////e /egggggttuuu...o wtt.tt/...o//ooogglll/////tttttt
re-implement the setting for the offline
C.2
Offline RL Algorithms
We employed Clean Offline Reinforcement Learning (CORL) library 33 and OfflineRL-Kit library to adapt implementation code and hyperparameters the Offline RL algorithms. import CQL
40
8
IQL (36), TD3BC [7] and DT [38] from the CORL library, and MCQ [4] from the OfflineRL-
Kit. The codebase of the CORL can be found at / com/tinkofi -afff-as ai/CORL and OfflineR-KKi from
Kit
C.3 Dataset Version and Evaluation Procedure
We employ v2 dataset for the Gym locomotion and AntMaze tasks, vl for the Maze2d tasks, and vO for the Adroit and FrankaKitchen tasks. We follow the standard evaluation procedure of D4RL benchmarks. For all the tasks we examined, we train TD3BC, IQL, CQL, MCQ for 1M training steps and report the score over the final evaluations. Especially, ease of the DT in Appendix [F.4] we trained it for 100k steps. We reported the final evaluation scores of the experiments, using TO evaluation steps for Gym locomotion tasks and 100 steps for Maze2d, AntMaze, Adroit, and Franka Kitchen tasks.
We normalize the score where 0 corresponds random policy, and 100 corresponds an expert policy as proposed at D4RL benchmark [15]. . All experiments of the main sections are conducted over 8 different random seeds. We set the batch size as 1024 across all Q-learning based offline RL algorithms. The justification for batch size configuration is covered at Appendix G.1
C.4 Pixel-based Environments
We conducted experiments on pixel-based environments using the VD4RL benchmark 16]. We follow the experiment protocol of the pixel-based experiment of Synther [13, firstly pretrain the CNN encoder with image observations for 1M steps and then fine-tune policy and Q-network with the augmented latent observations for 300k steps. We reported the mean and standard deviation of the last evaluation score. We calculate the standard deviation of the average score by averaging the standard deviation of each dataset, not identical with the standard deviation calculate protocol of Table [ and Table 2 which utilize the standard deviation of mean performance of each seed. We used the DrQ+BC codebase and the offline dataset on the https: : /ttgttttttuuub/ com/ woomm/cogglu1977/-488000 and re- re-implement the fine-tuning.
We used 2.0 and 0.1 for the medium, medium- and expert dataset and 1.3 and μ = 0.1 for the medium-replay dataset. For the reweighted sampling, we choose NB = 50, 0.001, and q 5.0 across all datasets.
17

---

D Data Quality Metrics
In this section, we present details about the quality metrics of the augmented dataset. We sampled IM of transitions among 5M generated transitions to effectively test these metrics.
D.1 Dynamic Plausibility
Dynamic plausibility, or Dynamic Mean Squared Error (Dynamic MSE), assesses the congruence of generated subtrajectorics with the environment dynamics, following methodologies from prior works 41 [3]. Let DA represent the augmented dataset. r')) denotes single transition, and f denotes the true dynamic model of the environment. Then, dynamic MSE is computed by the error between the generated next state s' and the true next state f* Prior to error computation, we normalize each state.
Dynamic MSE(DA)
1
E
(s)
(14)
[A
D.2 Novelty
Novelty metrie, inspired by 42 quantifies the uniqueness of the augmented trajectories compared to the original offline data. Novelty is calculated as the average 12 distance between each generated trajectory and its nearest offline data point. To gain a deeper understanding of the generated data, we categorized novelty into state novelty, action novelty, and state-action novelty. The calculation formula as follows:
Dovelty(DA,DD) 1
Σ
min ((s,a) (5)
(15)
[DAl (s,D.
(5
where D offline dataset.
D.3 Optimality
Lastly, optimality reflects the ability of generated trajectories to achieve high rewards. We quantify optimality an oracle reward, which is computed as the average of the reward values obtained by querying the enviromment with the generated states and actions (s, a). This metric serves as an indicator of the effectiveness of the generated trajectories in terms of reward maximization.
E
Additional Experimental Results
In this section, we present additional experiment results to deepen our understanding on GTA.
E.1
Ablation on trajectory-level generation
Table 10 shows that reducing the trajectory generation to single transition significantly raises the Dynamic MSE and degrades policy performance. Once the generation target becomes trajectory, horizon length marginally impacts both performance and Dynamic MSE. These results emphasize the importance of leveraging sequential relationships within transitions when augmenting trajectories.
Table 10: Ablations on the length of horizon H in malfchhettah-mediu--V
metric
16
32
128
D4RL
46.59 0.24
55.85
0.54
57.41 + 0.53
58.40
0.69
56.93 0.89
Dynamic MSE (x10-2)
5.12
1.11
0.85
0.66
0.65
E.2 Mixed-quality Dataset
widely known that the performance of offline RL. significanlly varies depending on the perfor- mance of the behavior policy used to collect the data [15]. However, offline RL algorithms often fail
18

---

to fully exploit the experience from figh-performing trajectories 43 when the offline dataset is mixed-quality dataset, which consists of few performmng trajectories and many wwwpforomning trajectories. Here, we demonstrate that GTA can effectively address mixed-quality dataset issues by augmenting high-performing trajectories using information from low-performing trajectories.
In this experiment, we mix the hatphhhtttt--miiii---ppaaaaaaeee and halfchectah-expertt data with ratios of 1:20 and 1:10. We choose these two datasets the performance gap between them the dataset consists of
largest among the Gym locomotion tasks. The pullchectah-mddum-rpppp 200k transitions, and the expert transitions in the mixed -uvilty dataset are 10k and 20k each.
We augment the -uulity dataset with various methods, including naive duplication, S4RL, SynthER, and GTA. Naive duplication is the setting where perffrerfoninn trajectories duplicated to 5M transitions without any modification.
To prove that GTA can exploit information on low-performing trajectories for augmenting high- performing trajectories, we additionally trained TD3BC on the expert-only configuration where the dataset consists of only tegh-perfomming trajectories used for comprising mixed-quality datasets. We employ S4RL, SynthER, and GTA as an augmentation method for expert-only configuration. We hypothesize that If the result of the GTA on the mixed-quality outperforms GTA expert-only, GTA exploits the information of low-performing trajectories while augmenting high- nigh-performing trajectories.
We train GTA on the mixed-quality or expert-only dataset and augment only the -erffornrooi trajectories. Similarly, we train SynthER on both datasets and generate the new transitions from the diffusion model. By the partial noising and denoising technique, GTA can choose the specific data to augment, while SynthER cannot because SynthER starts generating the transition from the Gaussian noise without any guidance.
Table IT demonstrates that GTA shows a significant performance boost on mixed-quality config- urations. The result also exhibits that GTA on the mixed -uxlity configuration outperforms GTA on the expert-only configuration. From these results, we suggest that GTA can effectively augment the high-performing trajectories while exploiting the information on low performing trajectories. Additionally, the exceptionally superior performance of GTA on expert-only configuration implies that GTA can enhance the sample efficiency considerably.
Table 11: D4RL score of the TD3BC on sparse expert dataset. The results are calculated with 10 evaluations over 8 seeds.
Expert Ratio
Data Configuration
Aug.
1:20
1:10
Baseline
None
44.64 + 0.71
None
42.67 4.44
44.47
4.45
Naive Duplication
29.23 + 5.52
41.17 + 12.26
Mixed-quae quality
S4RL
38.43 7.79
41.95 : 4.91
SynthER
2.00 + 0.09
2.02 1: 0.05
GTA
69.30 6.79
59.54 17.69
None
-0.83 + 1.12
1.82 2.19
S4RL
-0.73 +: 1.92
1.84 1: 1.04
Expert-only
SynthER
-0.91 + 0.83
2.31 1.08
GTA
36.26 8.20
48.49 + 3.81
E.3 Effectiveness of GTA Under The Data Sparsity
To assess the effectiveness of GTA when the number of transitions in the offline data is small, we designed experiments that utilized only 5%, 10% 15%, and 20% of the total offline data for the training diffusion model. We then evaluated GTA with TD3BC in the environment. Table ( baseline implemented with 100% of the data. This implies that GTA can efficiently improve the sample efficiency of offline RL algorithm when the volume of the offline dataset is limited.
that Even with only 5% of the data, GTA outperforms the TD3BC
19

---

Table 12: D4RL score of the TD3BC on small dataset. The results are calculated with 10 evaluations over seeds.
metric
5%
10%
15%
20%
100%
TD3BC
46.96 0.34
48.28 0.21
48.36
0.25
48.33 # 0.21
48.65+0.31
TD3BC + GTA
52.81 # 0.29
53.49 + 0.38
53.63 # 1.92
55.39 = 0.61
57.41 # 0.53
E.4
Effectiveness of GTA sequence modeling-basee approaches
Unlike prior works that augment data at the transition level, GTA generates trajectories and therefore can be applied to sequence modeling approaches such as Decision Transformer (DT, (38. For training DT, we should assign returns-to-pe token for each timestep i, which is calculated Rg Σ Pk- To model return-to-go for our generated subtrajectory, we set the retur-(o-go of the first timestep of the subtrajectory as Rg, where Rg can be computed from the original data. For subsequent timesteps, we deduct the reward that we generated.
St
814-1
H-
ar+1
H-+
-T
dg d++1
(16)
rt Tt+1
T+#H-1
T/ T+t1 1+H-1
R' = Re R r
E ---
(17)
To test the efficacy of GTA on sequence modeling, we train DT on Gym locomotion tasks and Maze2d environments. The result in Table demonstrates that DT with GTA outperforms baselines in terms of average performance at Gym locomotion tasks. And Table shows that DT with GTA augmented data exhibits superior performance with respect to average score. These results highlight the effectiveness of trajectory-leve data augmentation in preserving sequential relationships and leveraging the learned long-term dynamics.
Table 13: Normalized average scores on D4RL MuJoCo locomotion tasks, with the highest scores highlighted in bold. The results are calculated with from 100 evaluations using 4 seeds.
Alges
mesiium
Halfcheotah meliumeeeplay
medium-ce
malium
Hogper
meduum-cpps
maliur
Walkerld medium-cppla
mediat--coent
Noue
42.43 0.14
29.44 1.2
92.43 0:0
63.00 2.49
81.81 333
105.5 2012
11,64 0.59
62.06 = 1.85
105.38
74.47 1.40
Table 14: Normalized evaluation scores of Decision Transformer D4RL Maze 2d tasks. The results are calculated with 100 evaluations over 4 seeds.
maze2d medium
Algo.
Aug.
Average
umazze
large
None
+ 13.49
55.24 +
DT
S4RL
39.38 4 13.51
55.05 : 31.79
18.69 : 24/46
37.71 :: 23.25
GTA
50.19
5.59
25.49
E.5 Ablation Reweighted Sampling
We conduct an ablation study on reweighted sampling, which is additionally introduced to make the diffusion model focus high-reward regions. We conduct experiments (Hopper-medium and Walker2d-medium) to verify the effectiveness of the reweighted sampling. Table 15 presents evaluation scores and oracle rewards with and without reweighted sampling. As shown in the table, reweighted sampling achieves higher D4RL score compared to its counterpart, showcasing its effectiveness. We also observe that the oracle reward of the generated dataset is also increased, indicating that our reweighted sampling strategy successfully positions more points in higher reward regions.
environments
E.6 Ablation Different Conditioning Strategies
We explored alternative design choices for GTA that exploit the inpainting ability of the diffusion model. GTA condition amplified return value to improve the optimality of the offline dataset.
20

---

Table 15: Ablations on the reweighted sampling technqiues
D4RL score
Oracle reward
Env
X
O
X
Hopper-medium--2
57,45 + 4.54
56.95 + 6.30
3.61
3.60
Walker2d-medium-v2
88.26 + 0.80
86.43 + 4.60
4.35
4.17
However, another design choice is to condition the diffusion model by inpainting each step of reward with amplified reward. We named this method as amplified reward inpainting. We implement reward inpainting by multiplying the reward of each trajectory by a multiplier a = 1.3 and continually replacing generated rewards with multiplied rewards of the original trajectory during the denoising process the diffusion model. Note that we do not apply the return conditioning originally used in GTA when amplified reward inpainting is applied. Table 16 demonstrates that the trajectories augmented with reward inpainting yield lower returns than GTA, especially a significant performance drop the Hopper-medee medeiu dataset.
Table 16: D4RL score comparison between two conditioning strategies, amplified return guidance of GTA and amplified reward inpainting
strategy
Halfcheetah-medium
Hopper-medium
Walker2d-medium
amplified return guidance (GTA) amplified reward inpainting
57.41 0.53
60.34 3.27
87.07 + 0.45
55.13 + 0.80
50.68 5.29
87.30 + 1.25
Additionally, we conducted a comparative analysis of the augmented trajectories with different conditioning strategies, examining both the dynamic MSE and the oracle reward. As illustrated in Table [7] the dynamics MSE for data augmented through reward inpainting is significantly higher compared to the one augmented with GTA. Oracle reward of data augmented through reward inpainting is also lower than GTA. We hypothesize that reward inpainting introduces constraints that reduce the flexibility of states and actions, resulting in poor dynamic MSE and degrading the underlying offline RL policy.
Table 17: Data quality comparison between two conditioning strategies, amplified return guidance of GTA and amplified reward inpainting
strategy
Dynamic MSE (L) (x10-2)
Oracle Reward (t)
amplified return guidance (GTA) amplified reward inpainting
0.92
6.54
8.51
6.26
E.7
Ablation on utilizing reward proxy model
Instead of using augmented reward, we additionally train the reward model to replace it. Table 18 compares performance using a reward function instead of augmented rewards.
Although the method that replaces augmented rewards with the predictions of the proxy reward model shows comparable performances, incorporating the reward model into the augmentation process would additionally involve reward labeling step for augmented states and actions before training the offline RL algorithm. Therefore, we opted to generate all states, actions, and rewards at once to achieve more straightforward yet more effective augmentation.
Table 18: Normalized return on halfcheetah- mediettttaammiiuu-2222e with TD3BC.
metric
Reward proxy
GTA
halfcheetah-medium
57.96 + 0.49
57.69 + 0.73
21

---

F
Additional Analysis
In this section, we present additional analysis that are not included in the main section due to the page limit.
F1 Comparing Conditioning Strategies
common approach for providing classifier-free guidance is to condition on fixed value, repre- senting the desired return according to the environment [22] [4]. However, this strategy can lead to significant issues when jointly implemented with partial noising. In the scenario where trajectory is denoised from the partially noised trajectory, conditioning on fixed return, which is excessively greater than the return of the original trajectory, collides with the pre-existing context.
When the partially noised trajectory denoised with guidance towards the fixed excessively high return value, the guidance collides with the pre-existing context. Such forceful conditioning can lead to the generation of invalid trajectories. Furthermore, even without applying partial noising and denoising, using fixed value tends to decrease the diversity of the samples because of the narrow extent the conditioning value.
To this end, analyze the fixed conditioning method in more detail. Table 19 shows the quality measurements of the dataset generated with a fixed conditioning. We observe that fixed-value conditioning is capable of generating novel and high-rewarding samples. However, the dynamic plausibility significantly increases due to the forceful conditioning.
Table 19: Data quality comparison on halfcheetah- v2 between distinct conditioning strate-
gies: fixed conditioning, unconditioning, and amplified return conditioning.
Amplified (ours)
metric
Baseline
Fixed
Unconditioned
Dynamic MSE (x10-2)
0.00
4.88
0.91
0.85
Oracle reward
4.77
4.89
4.84
6.08
Novelty
1.41
1.42
1.58
F2 Statistical Test Results for The Performance Boosting Effects of GTA
We investigate the statistical meaningfulness of the performance improvement compared to the ones of the SynthER. To compute the standard deviation for the average performance of each offline RL algorithm, we first organize the final performance by seed and calculate the mean score of each seed across the tasks of the table. Next, we calculate the standard deviation of these mean scores for each seed and report this value as the standard deviation of the algorithm's 's performance. This protocol for calculating standard deviation is applied at Table T and Table
Based on the t-test results shown Table 20, we suggest that GTA induces a statistically significant performance boost over SynthER, achieving p-values below 0.05 across all algorithms. The p- p-value result on Table is calculated by the Welch's t-test for t-statistic [44] and the experiment results are sourced from the Table
Table 20: Welch's t-test result demonstrates statistically significant performance improvements of GTA over SynthER
TD3BC
IQL
CQL
MCQ
SynthER
78.64 2.38
81.20 1.46
81.97 + 2.21
82.81 =: 3.21
GTA
84.63 :: 2.20
85.27 : 1.02
86.11 + 0.94
85.91 :: 1.05
p-value
0.0001
0.0000
0.0008
0.0303
F.3 Impact of Partial Noising and Denoising Framework on Terminal States
We further explore the effects of partial noising and denoising frumework, particularly focusing on how well they preserve original trajectory information, especially terminal states. Halfcheetah contrasts with hopper and walker2d environments in terms of the existence of episode termination.
22

---

While hopper and walker2d have terminals, the sparsity of terminal states in datasets, exemplified by their 0.0001% occurrence walker2d transitions, underscores the challenge of preserving this critical information.
Our findings, illustrated in Figure 6 show that in Walker2d, high noise level (n 1) leads to unstable learning due to excessive deviation from the original trajectory. In contrast, a lower noise level (μ 0.5) maintains more of the original trajectory data, resulting in more stable learning outcomes. This highlights the importance of an approach to the balance of retaining original information environments with terminal states.
hulfchectal-mellien
hoppersttolium
walker 34-meiimmm
70-
63-
68)
Steps
Training
Figure 6: Normalized average scores on D4RL locomotion environments. Walker2d underperforms when the noise level is set to μ leading to the loss of crucial terminal information
Data Quality Table for the Figure 5
In this section, we provide the exact results in addition to the analysis of the data quality metric presented Section Table[[] Table[22 and Table 2ablee23exhhibl the optimality, novelty, and dynamic MSE of GTA and baseline augmentation methods. For clear presentation of Figure we take the logarithm the Dynamic MSE. For novelty and oracle reward, we adjust the axis scale for each environment.
GTA consistently achieves higher values than other augmentation methods at both the oracle reward and novelty across all locomotion environments. Simultaneously. the dynamic MSE remains comparably small. These results suggest that GTA successfully generates data that is novel, high-rewarding, and dynamically plausible.
As shown in Table 21 and Table 22
Beyond comparing the novelty of state-action pairs, we measured the novelty by separating state and action to assess the ability of GTA to discover novel states as well as novel actions. As seen in Table 25 and Table 24 GTA consistently outperforms other methods in generating novel states and actions.
Table 21: Comparison of oracle rewards across gym locomotion tasks.
Optimality (t)
S4RL
Synther
GTA
Halfcheetah-meddine
4.77
4.78
6.08
Halfchhettah-mediun-nnee
3.10
3.09
3.99
Halfcheethh-mechuueeexprrrr
7.73
7.7
10.07
Hopper-medium-v2
3.11
3.10
3.28
Hoppe--nedium--eppaay-22
2.38
2.37
2.70
Hopper--eddium-exxpertt
3.36
3.35
3.87
Walker24-meddium-22
3.39
3.40
3.51
liilerrddmmdium-rreee
2.47
2.46
3.35
Walker2d-medium-xxee
medium-expert-v2
4.15
4.17
4.81
23

---

Table 22: Comparison of the novelty of state and action across gym locomotion tasks.
Novelty (t)
S4RL
Synther
GTA
Hallcheetah--mcciun--2
1.39
1,41
1.58
Halfchhetall-medduu----ppyyyy
1.78
1.88
1.91
Haffchettallmmmdiiu---xpprrr---
1.22
1.22
1.3
1.36
1.37
2.05
Sopper-medium-regee
2.03
2.10
2.28
cooper-medium-epperr
1.45
1.46
1.79
1.26
1.26
1.31
palleerddmmdduu--reelaay-y
1.84
1,93
2.00
Walkerrddmeddum-exxpr.r-..
1.16
1.18
1.30
Table 23: Comparison of Dynamic MSE across gym locomotion tasks.
Dynamic MSE (Q) (x10-3)
S4RL
Synther
GTA
Halfcheetah-medium--22
0.0
0.73
0.85
Halfchheaah-medduu---paaaaaa
0.01
1.58
1.25
Halfiheetta-mmdiiuu-cccrrrrr
0.0
0,91
0.79
Hopper-medium--v2
0.07
1.56
0.08
Hopper-meddium--pply--
0.06
1.35
0.12
Hopper-medium-exxip
0.04
0.76
0.08
Walker2d -ilikrru-mmmium-2222222
1.90
2.09
2.98
Malker2d-medium-reee
2.59
3.37
2.77
Walkerrddmediiu-eexpetr..
1.63
1.73
3.49
Table 24: Comparison of the novelty of state across gym locomotion tasks.
Novelty (t)
S4RL
Synther
GTA
Halfcheetah-mediuu--22
1.12
1.14
1.28
Haffchhetall-medduu--eppyya-y
1.36
1.44
1.47
Haffchettal-meddiu--erperry
0.95
0.94
1.01
Hopper-medium--2
0.38
0.39
0.69
Hopper-medium-repla--22
0.63
0.66
0.74
Hopper-medium--xpprr-y
0.41
0.41
0.54
Walker2d- 2ilkkerr2dmeiium--22
0.81
0.81
0.85
Walker2d meroddd-mmiuu-lppaaaa
1.21
1.28
1.35
Walker2d Walker2d-medium medium-expert-ve
0.71
0.73
0.81
Table 25: Comparison of the novelty of action across gym locomotion tasks.
Novelty (t)
S4RL
Synther
GTA
Halfcheetah-mediim--22
0.22
0.22
0.23
Haffchhetah-mmdduu-rrrrlayyy
0.28
0.29
0.3
Halfchetaal-mmdiu--ccpprrrz
0.24
0.23
0.25
Hopper-medium--v
0.07
0.07
0.08
Hopper-meddum--eeelay-->9
0.07
0.07
0.08
Hlopper-medium-expe
0.07
0.07
0.08
0.35
0.35
0,36
Walker2d
0.41
0.42
0,44
0.36
0.36
0.38

---

F,5
Comparisons of GTA with Offline Model-based RL and Diffusion Planners
In this section, we compare the performance and computational cost of GTA with offline model-based RL methods and diffusion planners. While GTA shares commonalities with both approaches as discussed Section 2 it introduces key approaches that lead to improved performance and test-time computation efficiency.
Table 26 demonstrates superior performance of GTA compared to both model-based RI. algorithms and diffusion planners in locomotion tasks. The results suggest that trajectory augmentation via diffusion models can be more effective than traditional single-transition dynamic models used in offline model-based RL. Additionally, GTA outperforms diffusion planners, which require time- intensive sampling process during decision-make making.
We also analyze the computational efficiency GTA compared to baseline methods. As shown in Table 27 GTA exhibits significant computational advantages over diffusion planners during test-time evaluation. This advantage stems from shifting the computational burden from the sampling phase to the data preparation stage.
The model training process for both GTA and offline model-based RL algorithms [25] 27 consists of two stages: dynamic model training and policy training. GTA takes longer for dynamic model training due to its diffusion model with millions of parameters. However, offline model-based RL algorithms require significantly
time for policy training because model rollouts slow down this step. Overall, the total training time for COMBO and GTA is comparable, but GTA demonstrates superior performance, as shown in Table 26
Table 26: Comparison with diffusion planners and offline model-based RL baselines with GTA. For baselines, we take results from their original papers. For GTA, we report score of TD3BC and CQL as base algorithms.
Diffusion Planners
Model-based
GTA
Dataset Type
Diffuser
DD
AdapdDiffuser
MOPO
MORel.
COMRO
TD3BC
CQL
halfehectah-medione
42.8
49.1
44.2
42.3
42.1
54.2
$7.84 0,51
54.14 0.31
halfehoetah- balfehoetall-medidm-reee medium-replay
37.7
39.3
38.3
53.1
55.1
50.04
$1.36
halfcheetali medium expert
88.9
90.6
89.6
63.3
53.3
90.0
93.13
3.79
hopper-medium medium
74.3
79.3
92.2
28.0
95.4
97.2
69.57 a: 4.05
74,80 7.42
hopper medium medium-replay
93.6
16.84
hopper-medie medium-expert
103.3
111.8
111.6
23.7
108.7
111.1
110.40 + 4.04
110.90 3.44
79.6
82.5
84.7
17.8
77.8
81.9
86.69 = 0.89
80.40 4,98
walker2d medium medium-replay
70.6
1.74
walker2d medium expert
106.9
108.8
108.2
44.6
95.6
103.3
110.86 0.33
110.44 + 0.28
locomation average
77.5
81.8
83.4
42.60
72.72
82.03
84.63 2.20
85.27 1.02
Table 27: Comparison of computational cost between Diffusion planners, offline model-based RL baselines, and GTA. The elapsed time was measured using a single RTX 3090 on the HalfCheetah- medium-v2 environment.
Diffusion Planners
Model-based RL
GTA
DD
AdaptDiffuser
MOPO
COMBO
TD3BC
Model Training (H)
32.0
45.3
1.7
1.7
53.0
RL Policy Training (H)
20.8
45.8
2.0
Synthetic Data Generation (H)
2.4
3.5
Policy Evaluation (s)
1.3
1.4
6x10-4
7x10-4
3x10-4
F.6 Correlation on Conditioning Value and Generated Reward
To evaluate the controllability of GTA in trajectory generation, we analyze the Pearson correlation between the sum of rewards for generated subtrajectories and the conditioning value. As shown Figure 7, the correlation coefficients of Halfcheetah medium, Halfcheetah medium-replay and Halfcheetah medium-expert datasets are 0.55, 0.91, and 0.99, respectively. These results demonstrate that our diffusion model can effectively generate trajectories that align with the conditioned return values.
25

---

30
20
240
300
(a) sarchhetah-mecduu-rrllla
(b) haldcheetah-medium
(c)
Figure 7: Correlation on conditioning value and generated reward
F.7 Oracle Reward Distribution Generated Trajectories
We visualize the reward of the original dataset, generated reward from GTA, and real reward computed from the environment using state and action generated from GTA to visualize that GTA can really shift the data distribution to high reward region while preserving environmental dynamics. We present the results conducted on Halfcheetah-mediumg below. As shown in the figures, GTA mostly generates valid and high-reward trajectories across various environments and datasets.
the main section. Other results are presented
hulfcheatt-mmdiimn
Iselichorrds
deveeyy
0.000
01000
0015
0 000
0m0
01000
0104
0.066
0142
0.000
30
(10)
231
0.000
141
300
0.000
1(0)
301
300
400
Iopper-meliam
hopps-medun- replay
hopei--eedun--oe
0.04
0.06
Dubabity
0.00
0.006
0.01
0100
01000
300
25
125
wilkeerd medium
walktrrd medum replxy
walker3d medumm expen
0.005
0.010
0.000
0.00
Phokhhin
0.000
0.004
0.006
0.000
0.01
Ocadle
rowards
Anpliiied
Figure 8: Oracle reward sum of subtrajectories on D4RL Gym locomotion
G GTA Sensitivity Test
In this section, we conduct a sensitivity test on hyperparameters introduced GTA. To demonstrate the sensitivity of GTA on these hyperparameters, we choose Halfcheetah-medium-V dataset with TD3+BC as baseline Offline RL algorithm. For all experiments, we train the algorithm with IM gradient steps with 4 random seeds.
G.1 Batch Size and Training Epochs for Offline RL Training
We further verify that increasing batch size and training epochs affect the performance of our method. Most prior works fix batch size 256 and training epochs as 1M, which is a reasonable choice where
26

---

the original dataset size is quite small (100K to 2M). However, as we upsample the dataset into 5M, it is natural to check whether increasing batch size or training epochs might lead an improvement in the performance. To validate this, we perform experiments with the upsampled dataset by varying batch size and training epochs.
Figure 9 shows the performance of the TD3BC algorithm trained with an augmented dataset from GTA by varying batch size. We observe that increasing batch size significantly improves the performance. Table 28 illustrates the performance of offline RL algorithms trained with the original offtine dataset. As illustrated in the table, there is no big difference in performance when we just naively scale up the batch size. We attribute this to the natural phenomenon of neural networks to generalize well with a larger batch size if there is more data available for training. To this end, we set the batch size as 1024 for all methods in the main experiments,
As shown in the Figure 10 we observe that as increase the training epochs, GTA can achieve more performance gain. We also observe that using both large batch size and more training epochs continually improves the performance. After 10M training steps with batch size of 1024, we achieve significant improvement compared to the TD3BC trained with the original dataset, trained IM steps with batch size of 1024.
halfeheetah medium
hollichetaah medium replay
halfehoetah aandim-edimmuggyo expert
07.5
Nomadizzt Ncom
$5.0
62.5
10-
Training (M)
Training Staps
Performance of TD3BC on Halfcheetah environments with different batch sizes. Training with larger batch size achieves higher scores.
Figure 9:
Table 28: Normalized average scores on D4RL MuloCo locomotion tasks. Training with larger batch size makes no difference without modification on the offline dataset.
Algarithm
Batch Size
saddiass
sucdiate-ryyls replay
Hallcheetah
moliias-cspent
reddar
Happer medinn- eqpuay
uediars-cnpe expert
median
Walker2d mediun-eclay
meddsn-esppet
Average
TD3BC
H04
1131
44277 1.05
9282 2:44
60.99 333
69.09 a 21.45
H66
$5.00
1:000
85.85 230
100 0:4
18:29 4:15
EXpL
256
47.04
45:04 1:033
99.08
$5.11
09.6 10
80-35 178
73.09
T7.76
IQI.
1004
236
48.73 8 10.00
48.31 1.22
43.50 0:45
44.45 0:22
94.3 052
0410 2:71
6475 3.809
4:371:4:37
97.43 2:39
95.35 2:435
102002 & 10399
607.42 730
B099 3:3
$1.56
2:22
71,93 8 1515
$2.15 1.00
111 72 0.96
112 0.56
81.65 2,0
19.45 2:477
halfcheetah-
halfcheetah-medium medium
600
D4RL Normulized Score
58
-66
batch sizc: 256 busch size:
0.0
2.0
4,0
6.0
K.
10.0
Training Steps (M)
Figure 10: Ablations training with more epochs. Training more epochs leads to improvement in performance.
27

---

G.2 Scale of Multiplier a
The scale of multiplier determines the strength of model exploitation towards the high reward region. If we set a high, GTA tries to generate high-reward trajectories compared to the original trajectories, which might lead generating dynamically infeasible trajectories. If leans towards I, it would generate trajectories with similar return values to original trajectories and cannot push data distribution towards high-reward regions. Table 29 shows the performance of TD3+BC trained with generated data across different a values. We observe that there is a significant drop in performance where a too high (e.g., 2.0). Also, we find that1.1 S I 1.5 generally exhibits good performance.
G.3 Scale of Partial Noising 1
The scale of partial noising ratio μ controls the balance between generating in-distribution and OOD trajectories. If we μ = 0, GTA generates trajectories similar to the original dataset and stays close to the original data distribution. If we set = 1, it can generate novel trajectories via exploration while possessing the risk of violating environment dynamics. Intuitively, the high value of μ might lead to a significant performance drop due to the over-exploration of unknown regions. However, we observe that even when we choose relatively high jr (even set μu = 1), occasionally generates only novel but also dynamically plausible trajectories. Table 29 shows the D4RL score by varying The table shows that data generated with relatively high exhibits higher performance. However, the aforementioned trend is not always true for all environments, as we have already discussed in Section 5.3
Table 29: Sensitivity test on partial noising level and guidance multiplier a
x1.1
x1.2
x1.3
x1.4
x1.5
x2.0
0.1
48.56
0.44
48.43
0.35
48.54 + 0.51
48.69 0.35
48.57 + 0.20
48.85
0.41
0.25
48.62
= 0.29
48.97 = 0.3
49.48 + 0.53
50.21 + 0.3
51,30 0.49
53.32+ 0.85
0.5
53.55
= 0.82
57.08 0.34
57.94 + 0.49
58.01 + 0.42
57.80 4: 0.53
53.39 +: 0,65
0.75
54.39
0.57
56.89 0.53
57.85
+ 0.27
58. 0.59
58.09 = 0.29
54.19 0.52
1.0
54.42
0.13
57.08 0.61
56.58 + 2.84
58.21 0.22
58.02 + 0.34
53.84 0.80
G.4 Size of Augmented Dataset
In our main experiment section, we generate 5M transitions with the conditional diffusion model. We investigate the performance of our method in terms of the size of the upsampled dataset. To this end, we choose four different levels of size from the range [M.IM, 10M]. As shown Figure [T] the performance gain achieved by the upsampled dataset increases as the size of the augmented dataset becomes larger and converges after 5M samples.
halfehectah-
58.5
35.0
D4RL Normalized
57.0
56.
55.5
0.1
1.0
5.0
10.0
Size of Upsampled Dataset (M)
Figure 11: Ablation the size of the upsampled dataset.
28

---

H Extended Related Works
H.1 Related Works in Offline RL
In this section, we discuss related works with our work but just briefly discuss them in the main paper clearly define the position of work in offline RL literature.
BooT: BooT 45 suggests novel algorithm train Transformer-basee model for offline RL by incorporating the idea of bootstrapping. BooTT is a bootstrapping- methodology that builds upon the Trajectory Transformer, reusing self-generated trajectories within the sequence model itself.
AdaptDiffuser: AdaptDiffuser 24 is planning method with a diffusion model that can self-evolve by training with self- geferetteeedd diverse synthetic data using guidance from reward gradients. The AdaptDiffuser generates trajectories guided by various reward functions and trains the diffusion planner itself, shifting the data distribution learned by the diffusion model towards high reward regions.
Main differences: Our focus is building a high-quality augmented dataset that can be used for training any offline RL algorithms in plug-and-play manner. We develop algorithn-agnostcc method to boost the overall performance of offline RL, not confined to specific kinds of model. Unlike work, BooT and AdaptDiffuser utilize generated trajectories only for training itself and thereby are categorized as planners, not data augmentation, which is the reason why they are not included baselines of our approach. While our method also utilize bootstrapping. i.e., further training diffusion model with generated trajectories, we leave as a future work.
H.2 Related Works in Diffusion Models
We also notice that there several approaches that resemble approach, denoising from intermediate state instead of pure random noise to preserve originality various domains.
DifrAb: DiffAb [46] diffusion model optimize existing antibodies by first perturbing the CDR sequence and denoise it. They observe that optimizing antibodies with this procedure can improve binding energy while keeping the sequence similar the original one.
SDEdit: SDEdit [47] highlights the key challenge of image editing as balancing the faithful- ness and realism of synthetic images and proposes methodology that leverages diffusion models to augment images. To enhance the realism of the generated images, it introduces partial noising and denoising, similar the approach in GTA, where the editing is guided by the input image itself.
DA-Fusion:
DA-Fusion 48 conducted data augmentation utilizing pretrained diffusion model. They generate synthetic images using the partially noised images and then denoise them with guidance toward the original image. DA-Fusion significantly improved few-shot classification performance the Pascal and COCO datasets.
I Broader Impacts
This paper for the advancement of offline Reinforcement Learning. While it opens up various potential applications, any specific societal impacts immediately apparent and require further exploration.
29

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justifiiaton right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]", ", it is perfectly acceptable to answer "[No] provided proper justification is given (e.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clearly state the main claims in the abstract and introduction, accurately reflecting the paper's 's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
30

---

Justification:
We discuss limitations in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
We do not include theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide detailed experiment settings in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
31

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We provide open access the data and code to reproduce the paper.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. /ttppp.cc/pblic/ggideeee http://oweesimiiiooooooooooooooe
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
32

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: All of these be found in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We conduct all experiments with multiple random seeds and report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide information on the computer resources in the Appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
33

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///aaaarggcccaaaaaaaaaaatae
Answer: [Yes]
Justification: content.
We use D4RL, and VD4RL which does not contain harmful or offensive
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact of the paper in Appendix !
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We do controversial dataset.
Guidelines:
34

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We mention the license the README.dd of our code.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- ce-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We offer codebase of GTA via Anonymous github.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
35

---

Justification:
GTA does not involve any crowdsoureing and human- -unnant elements.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: GTA is about data augmentation without any crowdsourcing or human subject- related experiments.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
36


Reviews:
[{"review_id": "sXqakcUcZA", "reviewer": "Reviewer_cNng", "summary": "The paper presents Generative Trajectory Augmentation (GTA) that is aimed at improving offline reinforcement learning (RL). GTA uses a diffusion model conditioned on high returns to generate high-rewarding trajectories. These trajectories are used to augment the static datasets used to train offline RL algorithms. The augmentation process involves partially adding noise to existing trajectories and then refining them using a denoising mechanism guided by an amplified return value. The authors demonstrate that GTA improves the performance of popular offline RL algorithms across tasks in the D4RL benchmark. They also analyze the quality of the augmented trajectories, showing improvements in both data optimality and novelty over two baselines (S4RL and SynthER).", "questions": "1. Why is IQL the only chosen baseline (and not TD3-BC, CQL, MCQ) for the tasks listed in Table 2 (Adroit and FrankaKitchen)?\n2. Why is DrQ+BC the only chosen baseline and Cheetah-run the only chosen pixel-based observation task in Table 3? Why were results on the other baselines (IQL, TD3-BC, CQL, and MCQ) not presented for the pixel-based task(s)?\n3. How compute-intensive (GPU or hours) is running GTA on the pixel-based observation tasks?", "limitations": "1. As the authors have mentioned, while empirical results on the D4RL benchmark tasks show that GTA has low dynamic MSE, in tasks where dynamic violations have a critical impact, the performance boost may not be significant.\n2. The current evaluation is limited to simulated tasks, and the effectiveness of GTA in real-world offline RL tasks remains to be validated.\n3. As shown in Figures 4 (a)(b) and Table 25, the performance of GTA is sensitive to the hyperparameters ($\\mu$ and $\\alpha$) which might limit its applicability.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Originality:** The use of a conditional diffusion model for trajectory augmentation is a novel approach that adds computation overhead on the data creation stage instead of the policy learning stage.\n\n**Quality:** The paper includes experiments across tasks from the D4RL benchmark, demonstrating the effectiveness of GTA in various settings, including dense and sparse reward tasks, high-dimensional robotics tasks, and a pixel-based observation task.\n\n**Clarity:** The paper is well-organized and presents the methodology, experiments, and results clearly and logically. The authors provide an anonymized link to their code enhancing the reproducibility and transparency of the proposed method.\n\n**Significance:** GTA is shown to be compatible with offline RL algorithms, making it a flexible solution for data augmentation in offline RL. By adding novel high-rewarding trajectories to the datasets in the D4RL benchmark, GTA shows improvement in the performance of offline RL algorithms.", "weaknesses": "1. As shown in Figures 4 (a)(b) and Table 25, the performance of GTA is sensitive to the choice of hyperparameters, such as the noising ratio ($\\mu$) and the multiplier for conditioned return ($\\alpha$) which might require extensive tuning for different tasks.\n2. While empirical results show that the generated trajectories using GTA are dynamically plausible (Figure 5), it is not explicitly enforced in GTA and might be a byproduct of generating high-rewarding trajectories. The reviewer believes that further evaluation of GTA on real-world data is needed to make claims regarding GTA's dynamic plausibility such as in lines 6-9 (“In response, we introduce … and dynamically plausible”), lines 39-40 (“GTA is designed … dynamic plausibility”).\n3. Related to weakness #2, while the paper shows GTA's effectiveness on standard benchmarks, it lacks validation in real-world applications where the dynamics and data distributions may differ significantly from simulated environments. This poses a question on the real-world applicability of GTA.\n4. The paper does not include any experiments comparing GTA with model-based RL baselines. Although the authors state that both approaches perform data augmentation at different stages (data generation vs policy learning), the reviewer believes that a comparative study should be included. This is because both approaches learn a separate model to generate augmented data.\n\nMinor:\n- Line 48: “... any offline RL algorithms …” should be “... any offline RL algorithm …”.\n- A pink curve is present in Figure 4 (b), but its legend is missing. The blue curve denoting $\\mu$ = 1.0) is missing in this figure,", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful review!\n\n> **(Weakness 1, Limitation 3)** The performance of GTA is sensitive to the choice of hyperparameters, which might require extensive tuning for different tasks. \n\nAs we illustrated in tables in general response, we demonstrate that GTA outperforms other data augmentation baselines with a single set of hyperparameters across locomotion tasks. This result indicates that we may not need to extensively tune such hyperparameters. Furthermore, we also provide a general recipe for selecting hyperparameters for a new task: choosing high $\\mu$ and $\\alpha$ for low-quality datasets to promote exploration.\n\n> **(Weakness 2-1)** Dynamic plausibility is not explicitly enforced in GTA and might be a byproduct of generating high-rewarding trajectories.\n\nAs you mentioned, GTA does not explicitly enforce dynamic plausibility. Instead, it implicitly learns the dynamics through the trajectories which inherit true dynamics of environment, resulting in the generation of plausible trajectories.\n\nWe want to clarify that dynamic plausibility is not a byproduct of generating high-rewarding trajectories. Table 1 compares the dynamic MSE of total augmented trajectories to that of high-rewarding trajectories (top 10% in terms of reward). We found that the dynamic MSE of both cases are similar, validating our claim.\n\nTable 1. Dynamic MSE of data generated with GTA using halfcheetah-medium-v2\n||total trajectories|high rewarding trajectories (top 10%)|\n|-|-|-|\n|Dynamic MSE($\\times 1e-2$)|0.843|0.846|\n\n\n\n> **(Weakness 2-2, 3, Limitation 2)** GTA lacks validation in real-world applications where the dynamics and data distributions may differ significantly.\n\n\nThank you for suggesting problem setting that enhaces our manuscript . To verify the effectivenes of GTA in real-world data, we conduct further experiments on the NeoRL benchmark [1]. NeoRL is composed of datasets with narrower data distributions, enforced stochasticity, and aleatoric uncertainty to evaluate how offline RL algorithms behave in realistic environments.\n\nWe train GTA on the halfcheetah-v3-medium-noise-1000 and halfcheetah-v3-low-noise-1000 of NeoRL benchmark to evaluate the performance. As shown in table 2, GTA enhances the performance of the base algorithm, indicating that GTA can also generate trajectories with real-world datasets.\n\nTable 2. Experiment results on NeoRL benchmark. Experiments are conducted with 4 random seeds. Base algorithm is CQL.\n| Env | Original | S4RL | GTA |\n|:- |:- |:- |:- |\n| Halfcheetah-v3-L-1000 | 4352.58 ± 79.81| 4320.42 ± 76.65| **4359.08 ± 46.74**|\n| Halfcheetah-v3-M-1000 | 6162.49 ± 713.11 | 5759.13 ± 224.87 | **6173.46 ± 739.10** |\n\n[1] Qin, Rong-Jun, et al. \"NeoRL: A near real-world benchmark for offline reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 24753-24765.\n\n\n> **(Weakness 4)** The paper does not include any experiments comparing GTA with model-based RL baselines.\n\nThank you for your insightful comment. As illustrated in the general response, we compare offline model-based RL methods, such as MOPO, MOReL, and COMBO, with GTA. We take the results of baselines from their original papers. For GTA, we report the score of TD3BC and CQL as base offline RL algorithms.\n\nAs shown in Table 1 of the attached PDF, GTA outperforms offline model-based RL algorithms in locomotion tasks. It suggests that using a diffusion model to augment trajectories is more beneficial than learning and exploiting a single-transition dynamic model [2, 3].\n\n[2] Janner, Michael, et al. \"Planning with Diffusion for Flexible Behavior Synthesis.\" International Conference on Machine Learning. PMLR, 2022.\n\n[3] Jackson, Matthew Thomas, et al. \"Policy-guided diffusion.\" arXiv preprint arXiv:2404.06356 (2024).\n\n> **(Question 1)** Why is IQL the only chosen baseline for the Adroit and FrankaKitchen?\n\nFor Adroit and FrankaKitchen tasks, IQL is the most reliable baseline among offline RL algorithms we adopted (TD3BC, MCQ, CQL). TD3BC exhibits near-zero performance on these tasks. MCQ does not provide any hyperparameter setting for these tasks, while the main hyperparameter $\\tau$ in MCQ significantly influences the performance of the algorithm. Although CQL offers hyperparameter settings for these tasks, its training is notoriously unstable, leading to significant performance fluctuations.\n\n> **(Question 2)** Why is DrQ+BC the only chosen baseline and Cheetah-run the only chosen pixel-based observation task in Table 3? Why were results on the other baselines (IQL, TD3-BC, CQL, and MCQ) not presented for the pixel-based task(s)? \n\nTo evaluate GTA on more diverse algorithms and environments, we additionally conduct experiments on walker-walk environment and add BC algorithm for baseline as done in [4]. As shown in table 3, we achieve better performance than other baselines across both environments and algorithms, indicating the generalizability of our method.\n\nTable 3. Experiment results on pixel-based observation tasks. The mean scores of table are calculated with 3 seeds.\n||BC|||DrQ+BC|||\n|:-|:-|-|-|-|-|-|\n|| Original | SynthER | GTA| Original | SynthER | GTA |\n| Cheetah-run | 40.0 | 30.0| **41.1** | 45.8 | 31.2| **49.8**|\n| Walker-walk | 37.8 | 24.4| **40.0** | 39.7 | 23.8| **41.1**|\n\n[4] Lu, Cong, et al. \"Synthetic experience replay.\" Advances in Neural Information Processing Systems 36 (2024).\n\n\n> **(Question 3)** How compute-intensive (GPU or hours) is running GTA on the pixel-based observation tasks?\n\nPlease note that the diffusion model for pixel-based observation tasks generates a latent vector rather than raw pixel observation. In a single RTX 3090, training GTA in a pixel-based environment for 500k gradient steps requires 50 hours.\n\nWe promise to fix minor issues raised by the reviewer of our manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kOYEg5A9yY", "reviewer": "Reviewer_WvGz", "summary": "The paper introduces Generative Trajectory Augmentation (GTA), a data augmentation approach for Offline Reinforcement Learning (RL) that enhances the quality of static datasets by generating high-rewarding and dynamically plausible trajectories using a conditional diffusion model. GTA partially noises original trajectories and then denoises them with classifier-free guidance via conditioning on amplified return value. The authors demonstrate that GTA improves the performance of various offline RL algorithms across several benchmark tasks.", "questions": "- How do you ensure that the generated trajectories are plausible and adhere to the environment's dynamics?  What mechanisms to pick &alpha to handle cases where the return used for high-reward guidance is not reasonable? How do you ensure that the generated trajectories remain valid and useful? \n- Does each task require a new &mu, &alpha hyperparameter? How does this impact the generalizability and practical application of GTA?\n- What is the proportion of augmented trajectories to original ones in the final dataset? Is the entire offline dataset kept before training, with augmented trajectories added on top? Do all the offline datasets have the same size? Are 5 million augmented tranitions added regardless of the original dataset's size?\n- Could you clarify if Figure 3 is based on a real example or if it is just an illustration of the method's intuition? If it is the latter, how can you ensure it accurately represents the method's practical application?", "limitations": "The authors discuss the limitations and potential impacts in their paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- GTA integrates seamlessly with existing offline RL methods. It builds on previous advancements, and new offline RL algorithms could also benefit from it, given its agnostic nature towards the specific RL method used.\n- The ability to generate high-return trajectories that do not exist in the logged data is a significant advantage. This feature potentially improves the performance of offline RL algorithms by enriching the dataset with valuable transitions, as supported by the experimental results.\n- The experimental results presented in the paper show significant improvements across various offline RL algorithms and environments. Additionally, thorough ablations are provided, highlighting the impact of different components and hyperparameters on the overall performance. This comprehensive evaluation demonstrates the practical effectiveness of GTA.", "weaknesses": "- The claim that GTA-generated data adheres to the dynamics of the environment (lines 145-148) seems unfounded. Is there a principled argument on why diffusion models would learn the dynamics of the environment well? Especially where the goal is to create transitions outside the dataset. Figure 5 does not conclusively show that GTA is dynamically plausible, especially for tasks like Cheetah compared to methods like S4RL.\n- Compared to S4RL, the augmented trajectories appear less plausible. This issue is evident in tasks like HalfCheetah, yet GTA still achieves better rewards. It is unclear why this does not affect the final results.\n- Based on Appendix B.2, the performance of GTA appears to depend heavily on finely-tuned hyperparameters, as suggested by the different values used for different environments in Table 6. This raises concerns about the generalizability of the method, indicating that it might rely on online manual tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and valuable feedback!\n> **(Weakness 1-1)**  Is there a principled argument on why diffusion models would learn the dynamics of the environment well? Especially where the goal is to create transitions outside the dataset. **(Question 1-1)** How do you ensure that the generated trajectories are plausible and adhere to the environment's dynamics? \n\nAs you correctly pointed out, diffusion model of GTA does not explicitly ensure dynamic plausibility. Instead, it implicitly learns the dynamics governing the trajectory through learning the data distribution.\n\nGTA concentrates on enhancing the optimality of the dataset while reducing the degradation of the environment's dynamics of the augmented dataset to improve the performance of offline RL algorithms. Therefore, we will adjust our statement from \"ensuring dynamic plausibility\" to \"minimizing the degradation of dynamic plausibility\". We apologize for the overclaiming of our proposed method.\n\nHowever, our method still has the potential to generate high-rewarding and dynamically plausible trajectories by virtue of two novel strategies: *partial noising and denoising framework* and *amplified return guidance*. As discussed in Appendix F.1, amplified return conditioning prevents the partially noised trajectories denoise conditioning on excessively high value, yielding lower dynamic MSE. Additionally, table 1 shows the dynamic mse of the generated dataset according to the differing level of $\\mu$. The result demonstrates that low $\\mu$ does help to lose dynamic information less.\n\nTable 1. Dynamic MSE of generated data on walker-medium-v2.\n|$\\mu$|Dynamic MSE($\\times 1e-2$)|\n|-|-|\n|0.1|2.80|\n|0.25|3.02|\n|0.5|3.04|\n|0.75|3.34|\n|1.0|3.32|\n\nTo sum up, even though GTA does not explicitly enforce the model to generate dynamically plausible dataset, we introduce *partial noising and denoising framework* and *amplified return guidance* to minimize the degradation of dynamics of generated data.\n\n> **(Weakness 1-2, Weakness 2)** Figure 5 does not conclusively show that GTA is dynamically plausible, especially for tasks like Cheetah compared to methods like S4RL, yet GTA still achieves better rewards. It is unclear why this does not affect the final result.\n\nAs the reviewer noted, the trajectories generated by GTA exhibit larger dynamic MSE than S4RL in some cases like Halfcheetah. However, we observe that the dynamic plausibility of GTA is still better than other baselines such as SynthER. More importantly, GTA augments trajectories with higher rewards, which significantly affects the performance of offline RL policies. \n\n> **(Weakness 3)** The performance of GTA appears to depend heavily on finely-tuned hyperparameters. **(Question 2)** Does each task require a new &mu, &alpha? \n\nTo address the concern that the reviewer mentioned, we compared GTA with other baselines on single set parameter($\\alpha=1.1, \\mu=0.25$). The results demonstrate that fixed parameter configurations generally show performance gain across locomotion tasks, indicating that it may not require extensive online tuning. Furthermore, we propose a guideline for determining the appropriate $\\mu$ and $\\alpha$ for new tasks to boost performance. Please refer to the general response for a more detailed description.\n\n> **(Question 1-2)** What mechanisms to pick &alpha to handle cases where the return used for high-reward guidance is not reasonable? How do you ensure that the generated trajectories remain valid and useful? \n\nAs you mentioned, excessively guiding towards high rewards can result in invalid trajectories. This phenomenon occurs, as shown in Table 17 in Appendix F.1, when we generate trajectories conditioned on the maximum return that can be achieved from the environment. To prevent conditioning on unreasonable return, we introduced amplified return guidance, which involves multiplying the original trajectory by $\\alpha$ for conditioning. It controls the exploitation level of GTA, and we observe that our method empirically generates valid and useful trajectories to improve the performance of RL algorithms.\n\n> **(Question 3)** What is the proportion of augmented trajectories to original ones? Is the entire offline dataset added augmented trajectories on top? Do all the offline datasets have the same size? Are 5M augmented transitions added regardless of the original dataset's size?\n\nGTA augments 5M transitions for D4RL, and 1M transitions for VD4RL regardless of different sizes of original datasets. We follow the procedure of the prior method SynthER for a fair comparison. After the augmentation, we added augmented trajectories to the original dataset. There is no reason to exclude the original dataset, as it preserves the true dynamics of the environment. Please refer to Table 10 of Appendix G.4 for more discussion on the size of the augmented dataset.\n\n> **(Question 4)** Could you clarify if Figure 3 is based on a real example or if it is just an illustration of the method's intuition? If it is the latter, how can you ensure it accurately represents the method's practical application?\n\nWhile Figure 3 of section 4.2 is presented for intuitive comprehension, our empirical results of table 2 clearly demonstrate the relationship between $\\mu$, oracle reward, and the deviation from the original trajectory. Table 2 shows that both oracle reward and deviation from the original dataset tend to increase as $\\mu$ becomes larger. These results align well with Figure 3, which illustrates that trajectories deviate from the original trajectories and towards high-rewarding regions as $\\mu$ gets larger.\n\nTable 2. Analysis of dataset generated by GTA with different $\\mu$.\n|$\\alpha$ = 1.4|Oracle Reward|Deviations($\\times 1e-1$)|\n|-|-|-|\n|offline data|4.77|0.00|\n|$\\mu$ = 0.1|4.84|0.01|\n|$\\mu$ = 0.25|5.06|0.24|\n|$\\mu$ = 0.5|7.21|11.31|\n|$\\mu$ = 0.75|7.23|21.50|\n|$\\mu$ = 1.0|7.24|21.99|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zMX9F128gQ", "reviewer": "Reviewer_ABdH", "summary": "This paper builds on ideas from SynthER but adds classifier free guidance to boost the returns of the generated trajectories. This makes sense as the resulting data has higher quality and the paper is a totally sensible next step in the series of works building upon SynthER, which is an exciting direction for research. The paper is well executed and a solid contribution.", "questions": "NA", "limitations": "Nice to see this in the main body rather than the Appendix. It would be good to add a discussion on scalability if possible.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* A highly relevant area building on synthetic data generation for offline RL. Out of all the current active areas in RL research this one benefits the most from the current foundation model/large data regime, and scaling offline RL has been shown to be highly impactful in areas such as Robotics (E.g. RTX). \n* The method is not overly complicated and builds upon the recent SynthER paper, presented at NeurIPS 2023. This is an example of a simple idea that makes a great deal of sense, and it is well executed. \n* The paper reads well.\n* The experiments go beyond what was done in SynthER, including some new environments. It is great to see this - as RL needs to keep pushing for more complex benchmarks and not just sticking to D4RL and Atari \"because thats what the previous paper did\".", "weaknesses": "There are no major weaknesses here, it is a solid paper taking a nice step in an exciting general direction of research. The below points are fairly minor:\n* The authors could cite Ball et al 2021 \"Augmented World Models\" as another example of data augmentation in offline RL.\n* The authors could discuss how this relates to another paper building on SynthER, Policy-Guided Diffusion by Jackson et al. This is very recent work so makes sense it is not needed as a baseline, but the comparison may be worth including in text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and valuable feedback! \n\n> **(Weakness 1)** The authors could cite Ball et al 2021 \"Augmented World Models\" as another example of data augmentation in offline RL.\n\nThank you for pointing out crucial paper. The paper [1] augments learned model with simple transformations and approximate the augmentation for unseen environment in a self-supervised fasion for zero-shot generalization. While we focus on a single task offline RL problems, it is highly related in terms of data augmentation methods for offline RL problems. We will cite the paper in the related work section.\n\n> **(Weakness 2)** The authors could discuss how this relates to another paper building on SynthER, Policy-Guided Diffusion by Jackson et al. This is very recent work so makes sense it is not needed as a baseline, but the comparison may be worth including in text.\n\nThank you for highlighting an exciting concurrent work. Policy-Guided Diffusion (PGD) [2] generates synthetic trajectories for augmentation with classifier guidance from the target policy. On the other hand, GTA generates trajectories from the original trajectory, and introduce partial noising and denoising framework. To generate high-rewarding trajectories, GTA uses amplified return guidance. While the method is slightly different, both papers share ultimate objective, generating high-rewarding trajectories while retaining low dynamics error to improve offline RL algorithms. We will add comparison with PGD in the manuscript.\n\n[1] Ball, Philip J., et al. \"Augmented world models facilitate zero-shot dynamics generalization from a single offline environment.\" International Conference on Machine Learning. PMLR, 2021.\n\n[2] Jackson, Matthew Thomas, et al. \"Policy-guided diffusion.\" arXiv preprint arXiv:2404.06356 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gLLTEUV9Qz", "reviewer": "Reviewer_hVee", "summary": "To improve the quality of offline datasets, in this paper, a generative data augmentation approach is proposed by leveraging the diffusion models. Moreover, with the adoption of partial noising and denoising framework with amplified return guidance, the trajectory can be guided towards high-rewarding region. Finally, with the generated trajectories, existing offline RL methods can be utilized to learn the optimal policy.", "questions": "1. In section 4.2, by denoising with amplified return guidance, the authors claim the trajectory can be guided towards high-rewarding regions, are there any theoretical guarantee for this statement?\n2. How about the whole computation and time cost of the proposed approach when it is compared with other diffusion-free baselines? \n3. As shown in the experiments, for different environments, different noise levels are required for the optimal performance, then how can we determine the noise level for a new task? \n4. The proposed approach aims to deal with trajectories with reward signals, however, in reality, it may be difficult to obtain such signals, can the proposed approaches be extended to the cases where no reward signals can be provided?", "limitations": "The authors have addressed some limitations, but for some other limitations, e.g., the computation and time cost of the proposed approach, the application of the proposed approach in some real-world applications with no reward signal provided should be further stated.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is clearly organized and the experimental results show the effectiveness of the proposed approach. Also, the separation of the data generation with diffusion models and the policy learning mitigates the possible time and computation cost in the policy learning. Moreover, the trajectory-level data generation can capture the transition dynamics, which is beneficial for environments with sparse rewards.", "weaknesses": "The proposed approach is mainly a direct combination of current techniques, e.g., the diffusion model for data generation, the adding noise for the exploration and the existing offline RL techniques for policy learning, the theoretical contribution of the paper is trivial. Also, experimental results are still not sufficient, e.g., the lack of the comparations of time and computation costs between different methods, and only two data augmentation baselines are compared. Moreover, the proposed approach is limited to trajectories with reward signals, which is not the case in many real-world applications. Furthermore, it is unclear to me whether there are any theoretical guarantees to push the generated trajectory toward the high-rewarding region.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your critical reviews and valuable feedback.\n\n> **(Weakness 1)** The proposed approach is mainly a direct combination of current techniques.\n\nPlease note that we propose the hypothesis that augmenting the offline dataset with feasible and high-rewarding trajectories would boost the performance of offline RL algorithms. To achieve this, we made a novel combination of various techniques. GTA involves (1) generating trajectory-level data to capture sequential relationships between transitions and long-term transition dynamics, (2) introducing partial noising to control the exploration level of generated trajectories, and (3) guiding diffusion model with amplified return to find high-rewarding trajectories. We effectively balance exploration and exploitation with two strategies carefully tailored for diffusion models, as both strategies can seamlessly be integrated with the forward and reverse process of diffusion models. Our method consistently outperforms several baselines across different environments, highlighting the effectiveness of our proposed method. We want to emphasize that numerous machine learning studies continue to explore innovative combinations of established techniques. \n\n> **(Question 1)** In section 4.2, by denoising with amplified return guidance, the authors claim the trajectory can be guided towards high-rewarding regions, are there any theoretical guarantee for this statement?\n\nThere is existing work [1] proposing theoretical bounds between conditioning value and generated value. However, these proofs rely on stringent assumptions, making it challenging to directly apply them to GTA. Meanwhile, related works with return guidance [2, 3, 4] show that the conditionally sampled trajectories yield high rewards, demonstrating the effectiveness of return-guidance. \n\nTo verify that our conditional sampling indeed generates high-reward trajectories, we conducted analysis on the correlation between the conditioning value and the rewards of generated trajectories. As illustrated in Figure 1 attached in pdf, we observed Pearson correlation coefficients of 0.55, 0.91, and 0.99 for halfcheetah-medium, medium-replay, and medium-expert dataset, respectively. This indicates that diffusion model effectively samples trajectories aligned with the conditioned return.\n\n[1] Reward-directed conditional diffusion: provable distribution estimation and reward improvement\n\n[2] Janner, Michael, et al. \"Planning with diffusion for flexible behavior synthesis.\" arXiv preprint arXiv:2205.09991 (2022).\n\n[3] Ajay, Anurag, et al. \"Is Conditional Generative Modeling all you need for Decision Making?.\" The Eleventh International Conference on Learning Representations.\n\n[4] Liang, Zhixuan, et al. \"AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners.\" International Conference on Machine Learning. PMLR, 2023.\n\n> **(Question 2)** How about the whole computation and time cost of the proposed approach when it is compared with other diffusion-free baselines?\n\nAs mentioned in general response, we compare the whole computation and time cost of the proposed approach with other diffusion-free baselines. For model-based methods, we need to train dynamic model and generate synthetic trajectories during policy training.\n\nThe table 2 attached in PDF summarizes the whole training and evaluation time cost of algorithms. While training dynamic models takes shorter time than training diffusion models, model-based RL takes much longer time for training policy as it rollouts synthetic trajectories during training. \n    \n\n> **(Question 3)** As shown in the experiments, for different environments, different noise levels are required for the optimal performance, then how can we determine the noise level for a new task?\n    \nAs mentioned in the general response, we find that even if we fix $\\mu$ across different environments, we achieve higher performance than other baselines. We also observe that we can improve the performance further if we increase $\\mu$ for exploration when we have low-quality dataset. Therefore, we recommend selecting $\\mu$ based on dataset quality for a new task instead of extensive online tuning.\n\n> **(Question 4)** The proposed approach aims to deal with trajectories with reward signals, however, in reality, it may be difficult to obtain such signals, can the proposed approaches be extended to the cases where no reward signals can be provided?\n\nNumerous offline RL methods [5, 6, 7, 8, 9] aim to learn decision-making policy that maximizes expected cumulative rewards. In this context, we want to emphasize that GTA focuses on enhancing the optimality of offline datasets while minimizing the degradation of dynamic plausibility to improve the performance of offline RL algorithms. Therefore, generating trajectories without any reward signal is beyond our scope, as they cannot be augmented toward high-rewarding trajectories and are not the objective of RL problems. Please acknowledge that we conduct extensive experiments on realistic and challenging tasks such as AntMaze, FrankaKitchen, Adroit, and pixel-based environments.\n\n[5] Fujimoto, Scott, and Shixiang Shane Gu. \"A minimalist approach to offline reinforcement learning.\" Advances in neural information processing systems 34 (2021): 20132-20145.\n\n[6] Kumar, Aviral, et al. \"Conservative q-learning for offline reinforcement learning.\" Advances in Neural Information Processing Systems 33 (2020): 1179-1191.\n\n[7] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline Reinforcement Learning with Implicit Q-Learning.\" International Conference on Learning Representations.\n\n[8] Lyu, Jiafei, et al. \"Mildly conservative q-learning for offline reinforcement learning.\" Advances in Neural Information Processing Systems 35 (2022): 1711-1724.\n\n[9] Yu, Tianhe, et al. \"Combo: Conservative offline model-based policy optimization.\" Advances in neural information processing systems 34 (2021): 28954-28967.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TyQkSSYIcz", "reviewer": "Reviewer_UiuM", "summary": "This paper tackles the challenges of Offline RL, which involves learning effective decision-making policies from static datasets without online interactions. The authors introduce Generative Trajectory Augmentation (GTA), a novel approach that uses a diffusion model to enhance offline data by augmenting trajectories to be both high-rewarding and dynamically plausible. GTA partially noises original trajectories and denoises them with classifier-free guidance based on amplified return values. The experiment results indicate that GTA, as a general data augmentation strategy, enhances the performance of widely used offline RL algorithms in both dense and sparse reward settings.", "questions": "I will adjust my rating score based on the rebuttal results.\n\n(1) How to select the hyperparameter \\alpha in the denoising with amplified return guidance in equation (8). Can you also provide ablation study results for different selections of \\alpha?\n\n(2) In the generation phase, do you generate whole trajectories or generate subsequences of trajectories?\n\n(3) Is the proposed method sensitive to the hyperparameter \\mu? Based on the description in Figure 4 and Appendix F.3, when \\mu = 1, it only performs well on the dataset with low quality. For high-quality datasets with abundant expert demonstration, does your method have trouble preserving and reconstructing the high-rewarding trajectories?\n\n(4) Can you provide comparison results with diffusion-based offline RL methods? For example, Decision Diffuser [1] and AdapDiffuser [2].\n\n(5) In the ablation study on reweighted sampling, what base offline RL do you use for D4RL score calculation?\n\n(6) Since the reweighted sampling can implicitly change the dataset distribution, can you also apply the reweighted sampling technique as well for other baseline methods?\n\nReference:\n\n[1] Ajay, A., Du, Y., Gupta, A., Tenenbaum, J., Jaakkola, T., & Agrawal, P. (2022). Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657.\n\n[2] Liang, Z., Mu, Y., Ding, M., Ni, F., Tomizuka, M., & Luo, P. (2023). Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877.", "limitations": "Yes, the authors discussed the limitations in their paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "(1) Interesting and important topic: The guided data augmentation for offline RL is interesting. The success in this domain will further benefit the real-world applications of offline RL algorithms.\n\n(2) Extensive experiments: the authors provide extensive experiments to validate their proposed method.", "weaknesses": "(1) Lack of theoretical guarantee to support the algorithm: The authors only provide empirical results, lacking theoretical analysis of the performance of the proposed method.\n\nFor other potential weaknesses, please see my questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback, which can enhance our manuscript.\n\n> **(Weakness 1)** Lack of theoretical guarantee to support the algorithm: The authors only provide empirical results, lacking theoretical analysis of the performance of the proposed method.\n\nWhile existing work [1] proposes theoretical bounds between conditioning value and generated value, these proofs rely on stringent assumptions, which may not align with complex tasks in Offline RL benchmarks. Empirically, on GTA, we conducted an analysis of the correlation between the conditioning value and the rewards of generated trajectories. As illustrated in Figure 1, attached in pdf, we observed Pearson correlation coefficients of 0.55, 0.91, and 0.99 for halfcheetah-medium, medium-replay, and medium-expert datasets, suggesting that GTA can generate trajectories aligned with the conditioned return.  Augmented datasets with high-rewarding trajectories can aid offline RL policies in reaching higher performance. \n\nGTA has demonstrated consistent performance gains through extensive experiments across various environments. Thorough ablation studies have further proven the effectiveness of our proposed method. We would like to emphasize the empirical results presented in our paper, which underscore the effectiveness of our approach.\n\n\n[1] Yuan, Hui, et al. \"Reward-directed conditional diffusion: Provable distribution estimation and reward improvement.\" Advances in Neural Information Processing Systems 36 (2024).\n\n    \n> **(Question 1)** How to select the hyperparameter $\\alpha$ in the denoising with amplified return guidance in equation 8? **(Question 3-1)** Is the proposed method sensitive to the hyperparameter $\\mu$?\n    \nAs discussed in the general response, GTA outperforms other baselines in locomotion tasks even with a single hyperparameter configuration ($\\mu=0.25, \\alpha=1.1$), demonstrating the robustness of our method. We also provide a general recipe for selecting hyperparameters for new tasks: Increasing $\\mu$ and $\\alpha$ for datasets with low quality to promote further performance gain.\n\n> **(Question 2)** In the generation phase, do you generate whole trajectories or generate subsequences of trajectories?\n\nInstead of generating the entire trajectory, we focus on generating subtrajectories. Please refer to Table 8 in Appendix E.1 for more details about subtrajectory generation.\n\n> **(Question 3-2)** Based on the description in Figure 4 and Appendix F.3, when \\mu = 1, it only performs well on the dataset with low quality. Does GTA have trouble preserving and reconstructing the high-rewarding trajectories?\n\n\n    \nIn addition to the dynamic MSE and oracle reward presented in Appendix F.4, we include data quality metrics for trajectories generated with a higher $\\mu$ in the table below. The results indicate that while a $\\mu$ of 0.75 results in a higher dynamic MSE than a $\\mu$ of 0.25, it still maintains a lower value than SynthER. This demonstrates that GTA does not encounter significant issues in reconstructing high-rewarding trajectories.\n    \nTable 1. Dynamic MSE of generated data on halfcheetah-medium-expert-v2.\n||SynthER|GTA($\\mu=0.25, \\alpha=1.3$)|GTA($\\mu=0.75, \\alpha=1.3$)|\n|-|-|-|-|\n|Dynamic MSE($\\times 1e-2$)|0.91|0.79|0.90|\n|Optimality|7.7|10.07|11.56|\n    \n    \n\n\n> **(Question 4)** Can you provide comparison results with diffusion-based offline RL methods? For example, DecisionDiffuser and AdaptDiffuser.\n\nWe provide a comparison between diffusion planners [2, 3, 4] and GTA in the table 1, 2 attached in PDF. As shown in the table, GTA outperforms diffusion planners in locomotion tasks and demonstrates significant efficiency in test-time evaluation.\n\nAs mentioned in related work section, our method resembles diffusion planners in terms of generating trajectories with high return guidance. However, adopting diffusion as an augmentation method and utilize the model-free offline RL policies can transfer extensive computational burden of diffusion models from the evaluation stage to the data preparation stage. It can bypass the crucial challenge of diffusion planners, which requires extensive cost for sampling action.\n    \n[2] Janner, Michael, et al. \"Planning with diffusion for flexible behavior synthesis.\" arXiv preprint arXiv:2205.09991 (2022).\n    \n[3] Ajay, Anurag, et al. \"Is conditional generative modeling all you need for decision-making?.\" arXiv preprint arXiv:2211.15657 (2022).\n    \n[4] Liang, Zhixuan, et al. \"Adaptdiffuser: Diffusion models as adaptive self-evolving planners.\" arXiv preprint arXiv:2302.01877 (2023).\n\n\n> **(Question 5)** What is the baseline algorithm in the ablation study of reweighted sampling?\n\nWe choose TD3BC for the ablation study of reweighted sampling because TD3BC requires minimal computational cost along the offline RL baselines (IQL, CQL, MCQ).\n\n>**(Question 6)** Can you also apply the reweighted sampling techniques for other baseline methods?\n\nWe conduct experiments on the effect of reweighted sampling for other baselines. Specifically, we sample transition from the buffer with probability proportional to its reward during policy training. Table 1 describes the performance of reweighted sampling on other baselines. We find that baselines with reweighted sampling underperform GTA. \n\nTable 1. Experiment results of reweighted sampling on baselines.\n||Original|S4RL|SynthER|GTA|\n|-|-|-|-|-|\n|Halfcheetah-medium|48.52±0.39|48.70±0.31|48.73±0.83|57.84±0.51|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank the review committee for their detailed feedback. We appreciate the recognition of our paper's strengths, highlighted by the reviewers: **Originality** (CNng, ABdH), **Significance** (CNng, WvGZ, ABdH, UiuM), and **Extensive experiments** (CNng, WvGz, ABdH, UiuM). In response to the reviewers' feedback, we provide a summary of conducted additional experiments:

- **Comparison with offline model-based RL and diffusion planners (Table 1)**

We present a table that compares the performance of GTA with model-based RL and diffusion planners in locomotion tasks. We observe that GTA outperforms recent baselines in terms of average performance.

- **Analysis on computational cost (Table 2)**

We present a table that shows total computational time for training GTA and other methods. We find that while training cost of GTA is relatively high, the evaluation time is much faster than other methods, especially when compared with diffusion planners.

- **Guidance for selecting $\mu$ and $\alpha$ for a new task**

We would like to address the concerns regarding the generalizability of GTA to new tasks, specifically on hyperparameter settings for noise level $\mu$ and guidance multiplier $\alpha$.


We conducted experiments across gym locomotion environments using a single set of hyperparameters ($\mu=0.25, \alpha=1.1$). As shown in the table, GTA outperforms SynthER even with a single hyperparameter setting, highlighting the generalizability of our method. 


Table 1. D4RL normalized score on locomotion environments with fixed $\alpha$ and $\mu$. The experiments are conducted with TD3BC.

| Env             | None          | SynthER       | GTA($\mu = 0.25, \alpha=1.1$) |
| --------------- | ------------- | ------------- | ----------------------------- |
| halfcheetah-m-r | 44.64 ± 0.71  | 45.57 ± 0.34  | **46.48 ± 0.39**              |
| halfcheetah-m   | 48.42 ± 0.62  | 49.16 ± 0.39  | **49.22 ± 0.52**              |
| halfcheetah-m-e | 89.48 ± 5.50  | 85.47 ± 11.35 | **94.98 ± 1.66**              |
| hopper-m-r      | 65.69 ± 24.41 | **78.81 ± 15.80** | 72.86 ± 28.42             |
| hopper-m        | 61.04 ± 3.18  | 63.70 ± 3.69  | **66.16 ± 4.89**              |
| hopper-m-e      | 104.08 ± 5.81 | 98.99 ± 11.27 | **107.09 ± 2.69**             |
| walker-m-r      | 84.11 ± 4.12  | **90.67 ± 1.56**  | 86.02 ± 8.98              |
| walker-m        | 84.58 ± 1.92  | **85.43 ± 1.14**  | 85.42 ± 1.30              |
| walker-m-e      | 110.23 ± 0.37 | 109.95 ± 0.32 | **110.67 ± 0.89**             |
| Average         | 76.92 ± 2.66  | 78.64 ± 2.38  | **79.88 ± 3.35**              |

We also provide a general recipe for selecting hyperparameters on new tasks. While setting $(\mu=0.25, \alpha=1.1$) generally works, we observe that for low-quality datasets, increasing $\mu$ and $\alpha$ leads to further improvements by promoting exploration, as shown in the tables below.



Table2. D4RL normalized score on medium quality locomotion environments with fixed $\alpha$ and $\mu$. The experiments are conducted with TD3BC.
| Env           | None         | SynthER      | GTA($\mu = 0.5, \alpha=1.3$) | GTA($\mu = 0.75, \alpha=1.3$) |
| ------------- | ------------ | ------------ | ------------------------------ | ------------------------------- |
| halfcheetah-m | 48.42 ± 0.62 | 49.16 ± 0.39 | **57.92 ± 0.48**                   | 57.85 ± 0.27                    |
| hopper-m      | 61.04 ± 3.18 | 63.70 ± 3.69 | **68.46 ± 1.32**                   | 61.58 ± 5.00                    |
| walker-m      | 84.58 ± 1.92 | 85.43 ± 1.14 | **88.38 ± 2.70**                   | 87.14 ± 1.73                    |
| Average       | 64.68     | 66.10      | **71.59 ± 0.45**                   | 68.42 ± 2.22                    |                |


Table3. D4RL normalized score on medium-replay quality locomotion environments with fixed $\alpha$ and $\mu$. The experiments are conducted with TD3BC.
| Env | None| SynthER | GTA($\mu = 0.25, \alpha=1.1$) |
| - | - | - | - |
| halfcheetah-m-r | 44.64 ± 0.71| 45.57 ± 0.34| **46.48 ± 0.39**|
| halfcheetah-m | 48.42 ± 0.62| 49.16 ± 0.39| **49.22 ± 0.52**|
| halfcheetah-m-e | 89.48 ± 5.50| 85.47 ± 11.35 | **94.98 ± 1.66**|
| hopper-m-r| 65.69 ± 24.41 | **78.81 ± 15.80** | 72.86 ± 28.42 |
| hopper-m| 61.04 ± 3.18| 63.70 ± 3.69| **66.16 ± 4.89**|
| hopper-m-e| 104.08 ± 5.81 | 98.99 ± 11.27 | **107.09 ± 2.69** |
| walker-m-r| 84.11 ± 4.12| **90.67 ± 1.56**| 86.02 ± 8.98|
| walker-m| 84.58 ± 1.92| **85.43 ± 1.14**| 85.42 ± 1.30|
| walker-m-e| 110.23 ± 0.37 | 109.95 ± 0.32 | **110.67 ± 0.89** |
| Average | 76.92 ± 2.66| 78.64 ± 2.38| **79.88 ± 3.35**|


In summary, we propose a general configuration of hyperparameters in GTA that mostly enhances baseline performance and also provides a recipe for selecting hyperparameters to allow researchers to apply GTA for new tasks without extensive tuning.

Meta-review: Reasons to accept:
- Offline RL and specifically data augmentation in RL is a research problem of practical significance.
- Paper clearly articulates a specific problem in data augmentation for offline RL and develops a method to address the problem.
- Empirical evaluation is appropriate (show casing the proposed benefit).

Reasons to reject:
- All concerns raised by the reviewers were addressed during paper discussions.

Summary: This paper identifies a significant problem for offline RL applications and develops a method for data augmentation that addresses it. An empirical study confirms the proposed benefit. The paper is clear and accessible to the RL audience at NeurIPS.
Decision: Accept (poster)