GTA: Generative Trajectory Augmentation with Guidance for Offline Reinforcement Learning

Authors: jaewoo  lee
sujin  yun
taeyoung  yun
jinkyoo  park
  omelet
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 18:45:44

Abstract: Offline Reinforcement Learning (Offline RL) presents challenges of learning effective decision-making policies from static datasets without any online interactions. Data augmentation techniques, such as noise injection and data synthesizing, aim to improve Q-function approximation by smoothing the learned state-action region. However, these methods often fall short of directly improving the quality of offline datasets, leading to suboptimal results. In response, we introduce GTA, Generative Trajectory Augmentation, a novel generative data augmentation approach designed to enrich offline data by augmenting trajectories to be both high-rewarding and dynamically plausible. GTA applies a diffusion model within the data augmentation framework. GTA partially noises original trajectories and then denoises them with classifier-free guidance via conditioning on amplified return value. Our results show that GTA, as a general data augmentation strategy, enhances the performance of widely used offline RL algorithms across various tasks with unique challenges. Furthermore, we conduct a quality analysis of data augmented by GTA and demonstrate that GTA improves the quality of the data. Our code is available at https://github.com/Jaewoopudding/GTA

Main paper: 
 introduction: 
 Learning a decision-making policy through continual interaction with real environments is challenging when online interaction is costly or risky. Offline Reinforcement Learning [Offline RL, 1] emerges as a solution, focusing on training effective decision-making policies with static dataset gathered through an unknown policy [2] . However, offline data often does not provide enough coverage of state-action space, resulting in extrapolation error addressed as overestimation of Q-value [3, 4] . To mitigate extrapolation error, many previous works rely on adding explicit regularization terms [3] [4] [5] [6] [7] [8] [9] [10] and have shown significant progress.Moving beyond these mainstream methodologies, there exists an underexplored approach, data augmentation methods: traditional data augmentation and generative data augmentation. Traditional augmentation methods [11, 12] inject minimal noise to the state, preserving environment dynamics. Generative data augmentation [13, 14] builds a data synthesizer with a generative model to upsample offline data. Employing generative data augmentation approaches broadens the support of the data, thereby improving Q-function approximation [12] .While previously proposed data augmentation methods enhance the performance of offline RL, they still have limitations. As illustrated in Figure 1 , traditional augmentation methods have difficulties in discovering novel states or actions beyond the existing offline data, limiting their role only to smoothing small local regions of the observed state. In the case of generative data augmentation, the reward distribution of generated data is constrained by the support of offline data, resulting in the [11, 12] , generative data augmentation [13] and GTA.generation of suboptimal data. These issues make the existing methods not align with the objective of offline RL, which aims to learn the most effective decision-making policy from the static dataset.We introduce Generative Trajectory Augmentation (GTA), a novel approach that applies the conditional diffusion model to data augmentation, aiming to address the aforementioned limitations.GTA is designed to generate novel and high-rewarding trajectories while minimizing degradation of dynamic plausibility. Our approach consists of three main stages: (1) Training a conditional diffusion model that generates trajectory-level data, (2) Augmenting the offline data via partial noising with diffusion forward process and denoising with amplified return guidance, and (3) Training any offline RL algorithm with augmented data.We train a conditional diffusion model that approximates the conditional distribution of trajectory given its return. Once the diffusion model is trained, we sample trajectories from the offline dataset and partially noise the trajectory with the diffusion forward process. Then, we denoise the noised trajectory with amplified return guidance, directing trajectories to the high-rewarding region. To this end, we can orthogonally integrate data augmented by GTA into any offline RL algorithm without any modification.Diffusion Models for Offline Reinforcement Learning. In focus on the diffusion planners, Diffuser [21] devises the RL problem as a trajectory-level generation with cumulative reward classifier guidance. Following this, the Decision Diffuser [22] replaces the classifier guidance of the Diffuser with classifier-free guidance [23] . Adaptdiffuser [24] alternates between generating trajectories with diverse reward functions and training the model using self-generated trajectories. However, diffusion planners require extensive time to sample actions, making their practical application challenging. GTA shifts the computational burden of the diffusion model from the decision-making step to the data-preparation step. This transfer allows GTA to leverage the advantages of diffusion models while avoiding extensive time costs during decision-making.Model Based Offline Reinforcement Learning. Model-based Offline RL focuses on learning environment dynamics and reward functions from data, creating simulated trajectories to train both the critic and policy [25] [26] [27] [28] . While both GTA and model-based offline RL generate synthetic trajectories, in the case of GTA, a key difference is that data generation and policy learning are separated and not done in an alternative cycle. This separation ensures that the quality of generated data is not affected by the training progress of critics or policies. It also mitigates the accumulating errors associated with single-step dynamics rollouts [22] .3 Preliminaries 
offline reinforcement learning: 
 Reinforcement Learning (RL) is modeled with the Markov decision process (MDP) described by the tuple (S, A, T , R, γ), consisting of state space S, action space A, transition function T : S ×A → S, reward function R : S × A × S → R, and discount factor of future reward γ ∈ [0, 1) [29] . At each timestep t, the agent selects an action a t according to the policy π given the state s t . Consequently, the agent receives a reward r t for the action a t taken in the state s t , leading to the next state s t+1 . The goal of RL is to learn policy π * , which maximizes expected discounted return,J(π) = E π [ ∞ t=0 γ t r t ].In the offline RL setting, we can only have access to the fixed dataset D, which has been collected using unknown behavior policy π β . With insufficient and suboptimal offline data, offline RL aims to learn effective decision-making policy that surpasses the behavior policy.Score-based diffusion models. Diffusion models are a family of generative models that approximate the data distribution p(x) with p θ (x). When considering the data distribution as p(x) and the standard deviation of noise as σ, then the distribution of data with added noise is denoted as p(x; σ). The diffusion reverse process involves sequentially denoising from noise x K that randomly sampled from a Gaussian distribution N (0, σ 2 max I), following a noise level sequence σ K = σ max > σ K-1 > • • • > σ 0 = 0. Consequently, the endpoint x 0 of this process aligns with the original data distribution.Considering the probability flow ordinary differential equations (probability flow ODE), noise is continuously added to data during the forward process and reduced in the reverse process. By scheduling the noise level at time k, represented as σ(k), the reverse probability flow ODE is formulated as follows [30] :EQUATIONwhere ∇ x log p(x; σ(k)) denotes the score function, signifies the direction towards the data for a given noise level, and the dot represents the time derivative. The score function ∇ x log p(x; σ(k)) is trained via denoising score matching. The denoising score matching loss for given denoiser D θ (x; σ) is given byEQUATIONWhen the denoiser D θ (x; σ) is optimally trained, the score is calculated asEQUATIONWe sample data via solving Equation (1) with the learned denoising network. Conditional score-based diffusion model. In the domain of conditional diffusion models, two primary strategies are recognized: classifier guidance [31] and classifier-free guidance [23] . Classifier free guidance sets its guidance distribution pθ as pθ (x|y) ∝ p θ (x|y) • p θ (y|x) w . Subsequently, using the implicit classifier p θ (y|x) ∝ p θ (x|y)/p θ (x) and the equivalence relationship between score matching and denoising process, described as ∇ x log p θ (x|y) ∝ ϵ θ (x, y) [32, 23] , the classifier free guidance score εθ is formed as follows:EQUATIONwhere w controls the strength of the guidance. The training objective of the classifier free guidance is to concurrently train the conditional score function and the unconditional score function as follows:EQUATION)where y is a condition and β ∼ Bern(λ) is a binary variable with dropout rate λ of condition y. 
method: 
 In this section, we introduce GTA, Generative Trajectory Augmentation, which leverages the conditional diffusion model to generate high-rewarding, novel, and dynamically plausible trajectories for augmentation. As shown in Figure 2 , our method is divided into three stages: (1) Train a conditional diffusion model at the trajectory level. (2) Augment trajectories using a partial noising and denoising framework with amplified return guidance. (3) Train any offline RL algorithm with the augmented dataset. We will discuss how each component works and their roles in effective data augmentation.Trajectory-level generation. The diffusion model for GTA is designed to generate the subtrajectory τ . This subtrajectory is a consecutive transition sequence of the state, action, and reward sampled from a trajectory (s 1 , a 1 , r 1 , ..., s T , a T , r T ). We represent subtrajectory τ with horizon H as follows:EQUATIONWe train a conditional diffusion model to approximate the conditional distribution p(τ |y(τ )) with offline dataset, where the condition y(τ ) = T i=t γ i-t r i denotes the sum of the discounted return. The parameter θ is updated to maximize the expected log-likelihood of the data:EQUATIONUsing a diffusion model to generate trajectories offers multiple advantages. First, the diffusion model leverages sequential relationships between consecutive transitions while generating trajectories, allowing it to minimize the degradation of dynamic plausibility. Second, the diffusion model captures long-term transition dynamics, which is beneficial for environments with sparse rewards.Diffusion Model Implementation. Integrating sequential dependencies within the subtrajectory into the model architecture is essential for trajectory-level generation. Therefore, we choose a denoising network that combines Temporal-Unet [21] with MLP-mixer [33] to exploit both local and global sequential information. We adopted the Elucidated Diffusion Model [30] , known for its powerful performance in recent work [13] . The implementation details are elaborated in Appendix A.2.We propose a novel approach to augment trajectories, partial noising and denoising framework with amplified return guidance. Partial noising modifies the original trajectory using the forward process of the diffusion model, thus providing exploration opportunities. The exploration level is adjusted by the nosing ratio µ. Following this, denoising with amplified return guidance refines the noised trajectory. During the denoising process, we guide the trajectory towards high-rewarding regions, promoting the exploitation of learned knowledge about the environment. We introduce the multiplier for conditioned return α to control the exploitation level. Figure 3 outlines the principle of data augmentation via partial noising and denoising framework.Guided Denoising ProcessOriginality Preserving Region Let τ = τ 0 represent the original trajectory, and k denotes the diffusion timestep. We introduce a noising ratio µ(0 < µ = k K ≤ 1) to determine the extent of erasing information of the trajectory for exploration. We add noise to the original trajectory τ 0 , creating a noised trajectory denoted as τ µ•K ∼ N (τ ; τ 0 , σ(µ • K) 2 I). The parameter µ controls the level of exploration. Small µ results in minimal exploration, thereby preserving much of the original information of the trajectory. Large µ facilitates broader exploration, potentially leading to generating novel trajectories while losing original information significantly.! ! ! ! !!⋅# ! !"⋅# ! !#⋅# " # ! " " # # " " # $ " Original TrajectoryDenoising with amplified return guidance.After noising the trajectory, we reconstruct the trajectory with classifier-free guidance to push the trajectory towards the high-rewarding region to enhance optimality. We introduce amplified return guidance, which sets the conditioning value as the multiplied return of the original trajectory. It can prevent adverse effects that occur when significantly larger return values are conditioned during generation. Further analysis on this phenomenon is elaborated in Appendix F.1.The amplified return is formally defined as follows:EQUATION)where α is the control parameter for the exploitation level. We set α > 1 to make the conditioning value higher than the return of the original trajectory. α close to 1 induces mild conversion towards a high-rewarding trajectory region, while large α promotes significant drift, substantially exploiting the diffusion model. Our partial noising and denoising framework can be summarized as follows:EQUATIONDenoisingEQUATIONThe final stage of GTA is utilizing high-quality trajectories generated through previous stages for policy training. The trajectories generated by GTA can seamlessly integrate with existing offline RL methods. By introducing GTA, we can orthogonally integrate the expressiveness and controllability of diffusion models with existing offline RL algorithms.To concentrate on samples in high-rewarding regions, we additionally adopt a reweighting strategy [34, 35] during the sampling process. This approach prioritizes the sampling of subtrajectories with higher returns. Consequently, augmented trajectories are more densely distributed in high-rewarding regions. The complete specification of our reweighting strategy, including implementation details and parameter configurations, is provided in Appendix A.3. 
experiments: 
 In this section, we present extensive experiments conducted across commonly studied benchmarks [15, 16] to evaluate the augmentation capability of GTA. Through experiments, we aim to address the following questions: 1) How much performance gain does GTA exhibit across various types of tasks? 2) What impacts do the design elements of GTA have on performance? 3) Are trajectories augmented via GTA high quality? 4) What is the preferable µ and α setting for the new tasks?Datasets and environments. We demonstrate the versatility of GTA on the D4RL benchmark [15] , from the standard continuous control tasks to the challenging tasks with unique difficulties. We validate that GTA enhances offline RL under the sparse rewards environments and overcoming high-dimensional, complex robotics tasks. Additionally, we demonstrate that GTA is effective with human-demonstrated, small-scale datasets, showing its adaptability to realistic settings. Finally, we extend GTA to pixel-based environments, specifically within VD4RL benchmark [16] .Data augmentation baselines. We compare GTA with existing augmentation methods. For traditional augmentation, we choose S4RL [12] , which introduces Gaussian noise into states. In the domain of generative augmentation, we choose SynthER [13] , which employs an unconditional diffusion model for transition-level generation.Offline RL algorithms. To demonstrate the general efficacy of GTA on proprioceptive observations, we select four widely used offline RL algorithms: TD3BC [7] , CQL [8] , IQL [36] , and MCQ [4] . For particularly challenging tasks, such as Maze2d, Antmaze, Adroit, and FrankaKitchen, we employ IQL, which provides all hyperparameter configurations across all tasks and consistently demonstrates stable performance. For pixel-based observations, we utilize DrQ+BC as the baseline algorithm, as established in the VD4RL benchmark by Lu et al. [16] .Data quality metrics. We propose data quality metrics to analyze whether GTA provides high-quality data. Following prior works [13, 37] , we conduct a thorough analysis with three metrics: oracle reward, novelty, and dynamic MSE. Oracle reward, computed with the true reward of generated data, represents optimality. Novelty measures the ability of the augmentation method to discover novel states and actions not existing in the offline data. Finally, dynamic MSE evaluates how well the generated trajectories adhere to the dynamics of the environment. Formally, dynamic MSE and novelty are defined as follows:EQUATIONEQUATIONwhere D A represent the augmented dataset and D is original offline dataset. (s, a, r, s ′ ) denotes a single transition, and f * denotes the true dynamic model of the environment. We conducted experiments to evaluate whether GTA can provide a performance boost when applied alongside existing offline RL algorithms in various tasks with its own unique challenges.Gym locomotion. We experimentally demonstrate on Table 1 that offline RL algorithms with GTA outperform all other baselines across all algorithms on the average score with a statistically significant margin. This result highlights the versatility of GTA with various offline RL algorithms. We provide p-value for our experiments to validate the statistical significance of the results in Appendix F.2.Maze tasks. To assess GTA in sparse reward tasks, we test GTA on Maze2d and AntMaze tasks. Table 1 reveals that GTA notably enhances the performance of IQL policy on the sparse reward tasks while SynthER often degrades the performance. This result suggests two insights. First, the trajectory-level generation of GTA helps capture long-term dynamics, effectively leveraging that information while augmenting sparsely rewarded trajectories. Second, augmenting the dataset with high-rewarding trajectories strengthens the goal-reaching ability by enriching the demonstrations with more successful trajectories.Complex robotics tasks. We evaluate the effectiveness of the GTA on realistic, high-dimensional, challenging control tasks. Adroit-human datasets comprise 25 human-generated trajectories. The FrankaKitchen dataset, which involves multitasking behavior trajectories, requires a generalization ability to stitch trajectories. According to the results in Table 2 , GTA effectively boosts the performance in high-dimensional, complex robotics tasks. The results on the Adroit-human dataset demonstrate that GTA effectively augments small-scale human demonstration data through the expressiveness of the diffusion model. Additionally, the performance improvements in kitchen tasks indicate that GTA aids offline RL in trajectory stitching [21] .Pixel-based Observations. We extend our approach to pixel-based observations by following the experimental setup detailed in [13] . Initially, we pretrain the policy and extract its visual encoder. Subsequently, we project offline pixel-based observations into the embedding space, followed by augmenting these embedded states with GTA. More detailed experiment setups are elaborated in Appendix C.4. The results presented in Table 3 show performance improvements on DrQ+BC across the environments and dataset qualities. It indicates that GTA can be further extended to pixel-based observations beyond proprioceptive observations. We carry out thorough ablation studies to assess the effectiveness of each component within GTA.These studies are evaluated based on the performance scores of D4RL tasks and the aforementioned data quality metrics.What are the benefits of trajectory-level generation? To investigate the impact of utilizing sequential relationships through trajectory-level generation, we examine performance differences as the trajectory length varied. We find that when H = 1, the dynamic MSE becomes five times or more higher compared to trajectory-level generation with H longer than 16. Additionally, when training TD3BC with augmented data, the normalized score increased by 21% with trajectory generation compared to transition generation, where H = 1. This result highlights sequential relationship between consecuive transitions in trajectory does help generation, minimizing degradation of dynamic plausibility. Detailed experiment results are in Appendix E.1Is preserving original information via partial noising essential? We conduct experiments by varying the noising ratio on datasets of different optimality to explore the effect of the exploration on performance. In Figure 4a halfcheetah-medium, higher noise levels enhance performance, while in Figure 4b halfcheetah-medium-expert, optimal performance occurs at a noise level of µ = 0.25, with declines at higher levels. We speculate that if the original data already contains high-rewarding trajectories, even minimal modifications can be highly effective, and excessive exploration might lead to unexpected outcomes. Therefore, preserving information of original trajectory by partial noising is crucial in achieving effective data augmentation while avoiding potential negative impacts from excessive exploration. Additionally, the terminal signal, which may be very sparse, could be lost significantly if the original information is not preserved. We further conduct experiments about the impact of partial noising on preserving terminal state information in Appendix F.3.Does amplified return guidance improve the optimality? Figure 4c illustrates the superiority of the amplified return guidance on augmentation with respect to unconditioning [13] , and fixed conditioning [22] . Augmented trajectories without any condition tend to replicate the original reward distribution, thereby failing to enhance data optimality. Fixed conditioning concentrates rewards near the maximum return offline dataset. Notably, amplified return conditioning demonstrates a superior ability to cover a broader range of rewards, especially on unseen high rewards, while leading on the average reward and D4RL score. Further analysis of the impact of the conditioning method on data quality can be found in Appendix F.1.Figure 5 : Data quality analysis of S4RL, SynthER, and GTA. GTA augmented data exhibits superior optimality and novelty across gym locomotion datasets while maintaining dynamic plausibility.We introduce quality metrics specifically suitable for offline data analysis in Section 5.1. We compare GTA with other augmentation methods regarding not only how much it expands the coverage of offline data but also its optimality and dynamic MSE. Figure 5 presents the relative scale of each metric across baseline augmentation methods on gym locomotion tasks. Data augmented by GTA shows significantly higher oracle reward and novelty while maintaining a comparably low dynamic MSE. These outcomes indicate that the conditional diffusion model of GTA expands data coverage under dynamic constraints and discovers novel states and actions with higher rewards, effectively integrating environmental knowledge. We add further analysis in Appendix F.4. In this section, we propose a guideline for selecting hyperparameters based on dataset quality and environment characteristics. First, when the known information about the dataset is limited and minimizing exploration risk is essential, a low exploration and exploitation approach can enhance performance without losing stability. As illustrated in Table 4 , GTA with low exploration and exploitation (µ = 0.25, α = 1.1) surpasses baseline methods in gym locomotion tasks.However, if we know that the offline dataset is suboptimal and there exists considerable room for improvement, a higher exploration and exploitation setting can yield superior performance by uncovering a broader range of high-quality behaviors. Table 5 shows that GTA with (µ = 0.5, α = 1.3) and (µ = 0.75, α = 1.3) significantly boosts performance on suboptimal datasets. These findings indicate that while conservative parameters with low µ and α offer stable gains, we can elevate values of µ and α by leveraging prior knowledge about dataset quality to achieve superior outcomes. Notably, GTA consistently outperforms baselines across all configurations without dataset-specific hyperparameter tuning.We conduct extensive additional experiments to explore the further potential of the GTA.GTA under realistic settings. We examined the augmentation ability of GTA for two realistic settings: one with a mixed dataset consisting of few expert datasets and the majority of low-performing trajectories in Appendix E.2, and the other with a small amount of offline dataset in the Appendix E.3. In both cases, We observed that GTA significantly enhances the performance, improving sample efficiency of the offline RL.Larger batch size and training epochs. Since GTA offers more training data, it is worth exploring whether increasing batch size or training epochs can lead to additional performance gain. The results in Appendix G.1 demonstrate that we can enhance the performance by increasing batch size and training epochs while baselines do not benefit from the batch size and training epochs increment.Extending to sequence modeling approach. We proved that GTA can be applied to sequence modeling approaches such as Decision Transformer [38] as GTA augments data at the trajectory level. We confirmed that GTA improves the performance of the sequence modeling approach, especially in sparse reward tasks. The detailed experiment setup and results are in Appendix E.4.Sensitivity tests. We conduct experiments on the sensitivity of parameters α in Appendix G.2 and µ in Appendix G.3. The sensitivity of GTA with the amount of augmented dataset is in Appendix G.4. Our results indicate that GTA maintains similar performance beyond one million augmented transitions, demonstrating robustness to the number of transition samples.Additional ablations on design choices. We explore the impact of reweighted sampling introduced on Section 4.4 in Appendix E.5. For exploring alternative design choices, we adopt conditioning-byinpainting as an alternative conditioning approach instead of classifier-free guidance in Appendix E.6. We also test the model-based approach by replacing the reward of the generated trajectory with the prediction of the reward proxy model in Appendix E.7.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provide open access to the data and code to reproduce the paper.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 We propose GTA: Generative Trajectory Augmentation, a novel generative data augmentation approach for enriching offline data with high-rewarding and dynamically plausible trajectories. GTA combines data augmentation with the diffusion sampling process by partially adding noise to original trajectories and subsequently denoising them under amplified return guidance. Our extensive experiments on 31 datasets across 9 environments exhibit considerable performance improvements in four distinct offline RL algorithms, demonstrating the versatility and effectiveness of the GTA. We show that GTA successfully creates high-quality datasets from the sub-optimal offline dataset, which leads to learning effective decision-making policy.Limitations and future works. In the GTA framework, we do not train additional transition model and reward model for simple augmentation framework. Therefore, in tasks where dynamic violations have a critical impact, we may not expect a significant performance boost. However, as shown in our experiments, dynamic violations are generally minimal. Additionally, while the main focus of GTA is on the offline setting, future work could explore off-to-online and online settings.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: All of these can be found in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes]Justification: We conduct all experiments with multiple random seeds and report error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We provide information on the computer resources in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We mention the license in the README.md of our code.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: We offer codebase of GTA via Anonymous github.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: GTA does not involve any crowdsourcing and human-subject elements. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Size of the diffusion model and training details. We use the same size of network and hyperparameters for modeling conditional diffusion model across all experiments for ease of implementation. The hyperparameters we used for modeling and training are listed in Table 6 . The number of trainable parameters of our diffusion model is about 29.6M, which is comparable with the size of the diffusion model used in SynthER ≈ 25.6M. When using the default network model, training takes approximately 30 hours for 1M training steps with batch size 256 on a 4 RTX3090 NVIDIA GPU. Trajectory sampling from the diffusion model. For the sampling process, we use the Stochastic Differential Equation (SDE) sampler suggested in EDM [30] with the default hyperparameter setting used in SynthER [13] . As we employ classifier-free guidance, an additional hyperparameter guidance scale w has been introduced. We use the same guidance scale w = 2.0 for all experiments. Detailed hyperparameters are listed in Table 7 . Note that in sparse reward settings where the reward is a binary variable, the reweighted strategy tends to predominantly sample subtrajectories near the goal location. It might not be helpful since it makes the distribution of generative data cover a narrow region. To address this, we sample episodes instead of subtrajectories with reweighting in sparse reward settings. We assign the weight c times the bigger value to the successful episodes. Once the episode is sampled via weighted sampling, subtrajectories for augmentation is sampled from the uniform distribution. For dense reward environments like Gym locomotion, we put details of reweighting parameters in Table 9 Table 9 : Hyperparameter setting for reweighted sampling Environment Reweighting N B u q In this subsection, we describe the implementation details of the data augmentation baselines. Note that we try to follow the official implementation provided by authors, including hyperparameters, and re-implement only when public code is unavailable.Halfcheetah-medium-v2 X - - -• S4RL [12] : S4RL adds zero-mean Gaussian noise to input states as augmented data, i.e, st ← s t + ϵ, ϵ ∼ N (0, σI). We choose σ = 0.0003, following the original paper.• SynthER [13] : SynthER trains an unconditional diffusion model that generates transitionlevel data (state, action, reward, next state, and terminals). We build SynthER based on code provided by authors 2 . For a fair comparison, we re-implement the setting for the offline RL with a larger diffusion network for all environments and sample the same number of transitions. The diffusion model of SynthER contains 25.6M trainable parameters. We only use generated data from SynthER to train offline RL methods for evaluation, following the original paper. Their code is opened at https://github.com/conglu1997/SynthERWe employed Clean Offline Reinforcement Learning (CORL) library [39] and OfflineRL-Kit library [40] to adapt implementation code and hyperparameters of the Offline RL algorithms. We import CQL [8] , IQL [36] , TD3BC [7] and DT [38] from the CORL library, and MCQ [4] from the OfflineRL-Kit. The codebase of the CORL can be found at https://github.com/tinkoff-ai/CORL, and OfflineRL-Kit from https://github.com/yihaosun1124/OfflineRL-Kit.We employ v2 dataset for the Gym locomotion and AntMaze tasks, v1 for the Maze2d tasks, and v0 for the Adroit and FrankaKitchen tasks. We follow the standard evaluation procedure of D4RL benchmarks. For all the tasks we examined, we train TD3BC, IQL, CQL, MCQ for 1M training steps and report the score over the final evaluations. Especially, in case of the DT in Appendix E.4, we trained it for 100k steps. We reported the final evaluation scores of the experiments, using 10 evaluation steps for Gym locomotion tasks and 100 steps for Maze2d, AntMaze, Adroit, and Franka Kitchen tasks.We normalize the score where 0 corresponds to a random policy, and 100 corresponds to an expert policy as proposed at D4RL benchmark [15] . All experiments of the main sections are conducted over 8 different random seeds. We set the batch size as 1024 across all Q-learning based offline RL algorithms. The justification for batch size configuration is covered at Appendix G.1We conducted experiments on pixel-based environments using the VD4RL benchmark [16] . We follow the experiment protocol of the pixel-based experiment of Synther [13] , firstly pretrain the CNN encoder with image observations for 1M steps and then fine-tune policy and Q-network with the augmented latent observations for 300k steps. We reported the mean and standard deviation of the last evaluation score. We calculate the standard deviation of the average score by averaging the standard deviation of each dataset, not identical with the standard deviation calculate protocol of Table 1 and Table 2 , which utilize the standard deviation of mean performance of each seed. We used the DrQ+BC codebase and the offline dataset on the https://github.com/conglu1997/v-d4rl and re-implement the fine-tuning.We used α = 2.0 and µ = 0.1 for the medium, medium-expert, and expert dataset and α = 1.3 and µ = 0.1 for the medium-replay dataset. For the reweighted sampling, we choose N B = 50, u = 0.001, and q = 5.0 across all datasets.In this section, we present details about the quality metrics of the augmented dataset. We sampled 1M of transitions among 5M generated transitions to effectively test these metrics.Dynamic plausibility, or Dynamic Mean Squared Error (Dynamic MSE), assesses the congruence of generated subtrajectories with the environment dynamics, following methodologies from prior works [41, 13] . Let D A represent the augmented dataset. (s, a, r, s ′ ) denotes a single transition, and f * denotes the true dynamic model of the environment. Then, dynamic MSE is computed by the error between the generated next state s ′ and the true next state f * (s, a). Prior to error computation, we normalize each state.EQUATIONNovelty metric, inspired by [42, 37] , quantifies the uniqueness of the augmented trajectories compared to the original offline data. Novelty is calculated as the average l2 distance between each generated trajectory and its nearest offline data point. To gain a deeper understanding of the generated data, we categorized novelty into state novelty, action novelty, and state-action novelty. The calculation formula is as follows:EQUATIONwhere D is offline dataset.Lastly, optimality reflects the ability of generated trajectories to achieve high rewards. We quantify optimality as an oracle reward, which is computed as the average of the reward values r obtained by querying the environment with the generated states and actions (s, a). This metric serves as an indicator of the effectiveness of the generated trajectories in terms of reward maximization.In this section, we present additional experiment results to deepen our understanding on GTA.Table 10 shows that reducing the trajectory generation to a single transition significantly raises the Dynamic MSE and degrades policy performance. Once the generation target becomes trajectory, horizon length marginally impacts both performance and Dynamic MSE. These results emphasize the importance of leveraging sequential relationships within transitions when augmenting trajectories. It is widely known that the performance of offline RL significantly varies depending on the performance of the behavior policy used to collect the data [15] . However, offline RL algorithms often fail to fully exploit the experience from high-performing trajectories [43] when the offline dataset is a mixed-quality dataset, which consists of few high-performing trajectories and many low-performing trajectories. Here, we demonstrate that GTA can effectively address mixed-quality dataset issues by augmenting high-performing trajectories using information from low-performing trajectories.In this experiment, we mix the halfcheetah-medium-replay-v2 and halfcheetah-expert-v2 data with ratios of 1:20 and 1:10. We choose these two datasets as the performance gap between them is the largest among the Gym locomotion tasks. The halfcheetah-medium-replay-v2 dataset consists of 200k transitions, and the expert transitions in the mixed-quality dataset are 10k and 20k each.We augment the mixed-quality dataset with various methods, including naive duplication, S4RL, SynthER, and GTA. Naive duplication is the setting where high-performing trajectories are duplicated to 5M transitions without any modification.To prove that GTA can exploit information on low-performing trajectories for augmenting highperforming trajectories, we additionally trained TD3BC on the expert-only configuration where the dataset consists of only high-performing trajectories used for comprising mixed-quality datasets. We employ S4RL, SynthER, and GTA as an augmentation method for expert-only configuration. We hypothesize that If the result of the GTA on the mixed-quality outperforms GTA on expert-only, GTA exploits the information of low-performing trajectories while augmenting high-performing trajectories.We train GTA on the mixed-quality or expert-only dataset and augment only the high-performing trajectories. Similarly, we train SynthER on both datasets and generate the new transitions from the diffusion model. By the partial noising and denoising technique, GTA can choose the specific data to augment, while SynthER cannot because SynthER starts generating the transition from the Gaussian noise without any guidance.Table 11 demonstrates that GTA shows a significant performance boost on mixed-quality configurations. The result also exhibits that GTA on the mixed-quality configuration outperforms GTA on the expert-only configuration. From these results, we suggest that GTA can effectively augment the high-performing trajectories while exploiting the information on low-performing trajectories. Additionally, the exceptionally superior performance of GTA on expert-only configuration implies that GTA can enhance the sample efficiency considerably. To assess the effectiveness of GTA when the number of transitions in the offline data is small, we designed experiments that utilized only 5%, 10%, 15%, and 20% of the total offline data for the training diffusion model. We then evaluated GTA with TD3BC in the halfcheetah-medium-v2 environment. Table 12 demonstrate that Even with only 5% of the data, GTA outperforms the TD3BC baseline implemented with 100% of the data. This implies that GTA can efficiently improve the sample efficiency of offline RL algorithm when the volume of the offline dataset is limited. Unlike prior works that augment data at the transition level, GTA generates trajectories and therefore can be applied to sequence modeling approaches such as Decision Transformer [DT, 38] . For training DT, we should assign returns-to-go token for each timestep t, which is calculated as Rt = T k=t r k . To model return-to-go for our generated subtrajectory, we set the return-to-go of the first timestep of the subtrajectory as Rt , where Rt can be computed from the original data. For subsequent timesteps, we deduct the reward that we generated.EQUATION)EQUATION)To test the efficacy of GTA on sequence modeling, we train a DT on Gym locomotion tasks and Maze2d environments. The result in Table 13 demonstrates that DT with GTA outperforms baselines in terms of average performance at Gym locomotion tasks. And Table 14 shows that DT with GTA augmented data exhibits superior performance with respect to average score. These results highlight the effectiveness of trajectory-level data augmentation in preserving sequential relationships and leveraging the learned long-term dynamics. We conduct an ablation study on reweighted sampling, which is additionally introduced to make the diffusion model focus on high-reward regions. We conduct experiments on two environments (Hopper-medium and Walker2d-medium) to verify the effectiveness of the reweighted sampling. Table 15 presents evaluation scores and oracle rewards with and without reweighted sampling. As shown in the table, reweighted sampling achieves a higher D4RL score compared to its counterpart, showcasing its effectiveness. We also observe that the oracle reward of the generated dataset is also increased, indicating that our reweighted sampling strategy successfully positions more points in higher reward regions.We explored alternative design choices for GTA that exploit the inpainting ability of the diffusion model. GTA condition on amplified return value to improve the optimality of the offline dataset. However, another design choice is to condition the diffusion model by inpainting each step of reward with amplified reward. We named this method as amplified reward inpainting. We implement reward inpainting by multiplying the reward of each trajectory by a multiplier α = 1.3 and continually replacing generated rewards with multiplied rewards of the original trajectory during the denoising process of the diffusion model. Note that we do not apply the return conditioning originally used in GTA when amplified reward inpainting is applied. Table 16 demonstrates that the trajectories augmented with reward inpainting yield lower returns than GTA, especially a significant performance drop on the Hopper-medium dataset. Additionally, we conducted a comparative analysis of the augmented trajectories with different conditioning strategies, examining both the dynamic MSE and the oracle reward. As illustrated in Table 17 , the dynamics MSE for data augmented through reward inpainting is significantly higher compared to the one augmented with GTA. Oracle reward of data augmented through reward inpainting is also lower than GTA. We hypothesize that reward inpainting introduces constraints that reduce the flexibility of states and actions, resulting in poor dynamic MSE and degrading the underlying offline RL policy. Instead of using augmented reward, we additionally train the reward model to replace it. Table 18 compares performance using a reward function instead of augmented rewards.Although the method that replaces augmented rewards with the predictions of the proxy reward model shows comparable performances, incorporating the reward model into the augmentation process would additionally involve a reward labeling step for augmented states and actions before training the offline RL algorithm. Therefore, we opted to generate all states, actions, and rewards at once to achieve a more straightforward yet more effective augmentation. In this section, we present additional analysis that are not included in the main section due to the page limit.A common approach for providing classifier-free guidance is to condition on a fixed value, representing the desired return according to the environment [22, 14] . However, this strategy can lead to significant issues when jointly implemented with partial noising. In the scenario where a trajectory is denoised from the partially noised trajectory, conditioning on fixed return, which is excessively greater than the return of the original trajectory, collides with the pre-existing context.When the partially noised trajectory is denoised with guidance towards the fixed excessively high return value, the guidance collides with the pre-existing context. Such forceful conditioning can lead to the generation of invalid trajectories. Furthermore, even without applying partial noising and denoising, using a fixed value tends to decrease the diversity of the samples because of the narrow extent of the conditioning value.To this end, we analyze the fixed conditioning method in more detail. Table 19 shows the quality measurements of the dataset generated with a fixed conditioning. We observe that fixed-value conditioning is capable of generating novel and high-rewarding samples. However, the dynamic plausibility significantly increases due to the forceful conditioning. We investigate the statistical meaningfulness of the performance improvement compared to the ones of the SynthER. To compute the standard deviation for the average performance of each offline RL algorithm, we first organize the final performance by seed and calculate the mean score of each seed across the tasks of the table. Next, we calculate the standard deviation of these mean scores for each seed and report this value as the standard deviation of the algorithm's performance. This protocol for calculating standard deviation is applied at Table 1 and Table 2 .Based on the t-test results shown in Table 20 , we suggest that GTA induces a statistically significant performance boost over SynthER, achieving p-values below 0.05 across all algorithms. The p-value result on Table 20 is calculated by the Welch's t-test for t-statistic [44] and the experiment results are sourced from the Table 1 . We further explore the effects of partial noising and denoising framework, particularly focusing on how well they preserve original trajectory information, especially terminal states. Halfcheetah contrasts with hopper and walker2d environments in terms of the existence of episode termination.While hopper and walker2d have terminals, the sparsity of terminal states in datasets, exemplified by their 0.0001% occurrence in walker2d transitions, underscores the challenge of preserving this critical information.Our findings, illustrated in Figure 6 , show that in Walker2d, a high noise level (µ = 1) leads to unstable learning due to excessive deviation from the original trajectory. In contrast, a lower noise level (µ = 0.5) maintains more of the original trajectory data, resulting in more stable learning outcomes. This highlights the importance of an approach to the balance of retaining original information in environments with terminal states. In this section, we provide the exact results in addition to the analysis of the data quality metric presented in Section 5.4. Table 21 , Table 22 , and Table 23 exhibit the optimality, novelty, and dynamic MSE of GTA and baseline augmentation methods. For a clear presentation of Figure 5 , we take the logarithm of the Dynamic MSE. For novelty and oracle reward, we adjust the axis scale for each environment.As shown in Table 21 and Table 22 , GTA consistently achieves higher values than other augmentation methods at both the oracle reward and novelty across all locomotion environments. Simultaneously, the dynamic MSE remains comparably small. These results suggest that GTA successfully generates data that is novel, high-rewarding, and dynamically plausible.Beyond comparing the novelty of state-action pairs, we measured the novelty by separating state and action to assess the ability of GTA to discover novel states as well as novel actions. As seen in Table 25 and Table 24 , GTA consistently outperforms other methods in generating novel states and actions. In this section, we compare the performance and computational cost of GTA with offline model-based RL methods and diffusion planners. While GTA shares commonalities with both approaches as discussed in Section 2, it introduces key approaches that lead to improved performance and test-time computation efficiency.Table 26 demonstrates superior performance of GTA compared to both model-based RL algorithms and diffusion planners in locomotion tasks. The results suggest that trajectory augmentation via diffusion models can be more effective than traditional single-transition dynamic models used in offline model-based RL. Additionally, GTA outperforms diffusion planners, which require a timeintensive sampling process during decision-making.We also analyze the computational efficiency of GTA compared to baseline methods. As shown in Table 27 , GTA exhibits significant computational advantages over diffusion planners during test-time evaluation. This advantage stems from shifting the computational burden from the sampling phase to the data preparation stage.The model training process for both GTA and offline model-based RL algorithms [25, 27] We visualize the reward of the original dataset, generated reward from GTA, and real reward computed from the environment using state and action generated from GTA to visualize that GTA can really shift the data distribution to high reward region while preserving environmental dynamics. We present the results conducted on Halfcheetah-medium-v2 in the main section. Other results are presented below. As shown in the figures, GTA mostly generates valid and high-reward trajectories across various environments and datasets. In this section, we conduct a sensitivity test on hyperparameters introduced in GTA. To demonstrate the sensitivity of GTA on these hyperparameters, we choose Halfcheetah-medium-v2 dataset with TD3+BC as a baseline Offline RL algorithm. For all experiments, we train the algorithm with 1M gradient steps with 4 random seeds.We further verify that increasing batch size and training epochs affect the performance of our method. Most prior works fix batch size as 256 and training epochs as 1M, which is a reasonable choice where the original dataset size is quite small (100K to 2M). However, as we upsample the dataset into 5M, it is natural to check whether increasing batch size or training epochs might lead to an improvement in the performance. To validate this, we perform experiments with the upsampled dataset by varying batch size and training epochs.Figure 9 shows the performance of the TD3BC algorithm trained with an augmented dataset from GTA by varying batch size. We observe that increasing batch size significantly improves the performance. Table 28 illustrates the performance of offline RL algorithms trained with the original offline dataset.As illustrated in the table, there is no big difference in performance when we just naively scale up the batch size. We attribute this to the natural phenomenon of neural networks to generalize well with a larger batch size if there is more data available for training. To this end, we set the batch size as 1024 for all methods in the main experiments.As shown in the Figure 10 , we observe that as we increase the training epochs, GTA can achieve more performance gain. We also observe that using both large batch size and more training epochs continually improves the performance. After 10M training steps with a batch size of 1024, we achieve significant improvement compared to the TD3BC trained with the original dataset, trained 1M steps with a batch size of 1024. The scale of multiplier α determines the strength of model exploitation towards the high reward region. If we set α high, GTA tries to generate high-reward trajectories compared to the original trajectories, which might lead to generating dynamically infeasible trajectories. If α leans towards 1, it would generate trajectories with similar return values to original trajectories and cannot push data distribution towards high-reward regions. Table 29 shows the performance of TD3+BC trained with generated data across different α values. We observe that there is a significant drop in performance where α is too high (e.g., 2.0). Also, we find that1.1 ≤ α ≤ 1.5 generally exhibits good performance.The scale of partial noising ratio µ controls the balance between generating in-distribution and OOD trajectories. If we set µ ≈ 0, GTA generates trajectories similar to the original dataset and stays close to the original data distribution. If we set µ ≈ 1, it can generate novel trajectories via exploration while possessing the risk of violating environment dynamics. Intuitively, the high value of µ might lead to a significant performance drop due to the over-exploration of unknown regions. However, we observe that even when we choose relatively high µ (even set µ = 1), it occasionally generates not only novel but also dynamically plausible trajectories. Table 29 shows the D4RL score by varying µ.The table shows that data generated with relatively high µ exhibits higher performance. However, the aforementioned trend is not always true for all environments, as we have already discussed in Section 5.3. In our main experiment section, we generate 5M transitions with the conditional diffusion model. We investigate the performance of our method in terms of the size of the upsampled dataset. To this end, we choose four different levels of size from the range [0.1M, 10M]. As shown in Figure 11 , the performance gain achieved by the upsampled dataset increases as the size of the augmented dataset becomes larger and converges after 5M samples. In this section, we discuss related works with our work but just briefly discuss them in the main paper to clearly define the position of our work in offline RL literature.• BooT: BooT [45] suggests a novel algorithm to train a Transformer-based model for offline RL by incorporating the idea of bootstrapping. BooT is a bootstrapping-based methodology that builds upon the Trajectory Transformer, reusing self-generated trajectories within the sequence model itself. • AdaptDiffuser: AdaptDiffuser [24] is a planning method with a diffusion model that can self-evolve by training with self-generated diverse synthetic data using guidance from reward gradients. The AdaptDiffuser generates trajectories guided by various reward functions and trains the diffusion planner itself, shifting the data distribution learned by the diffusion model towards high reward regions.Main differences: Our focus is building a high-quality augmented dataset that can be used for training any offline RL algorithms in a plug-and-play manner. We develop an algorithm-agnostic method to boost the overall performance of offline RL, not confined to specific kinds of model. Unlike our work, BooT and AdaptDiffuser utilize generated trajectories only for training itself and thereby are categorized as planners, not data augmentation, which is the reason why they are not included as baselines of our approach. While our method can also utilize bootstrapping, i.e., further training diffusion model with generated trajectories, we leave it as a future work.We also notice that there are several approaches that resemble our approach, denoising from an intermediate state instead of pure random noise to preserve originality in various domains.• DiffAb: DiffAb [46] use diffusion model to optimize existing antibodies by first perturbing the CDR sequence and denoise it. They observe that optimizing antibodies with this procedure can improve binding energy while keeping the sequence similar to the original one. • SDEdit: SDEdit [47] highlights the key challenge of image editing as balancing the faithfulness and realism of synthetic images and proposes a methodology that leverages diffusion models to augment images. To enhance the realism of the generated images, it introduces partial noising and denoising, similar to the approach in GTA, where the editing is guided by the input image itself. • DA-Fusion: DA-Fusion [48] conducted data augmentation utilizing pretrained diffusion model. They generate synthetic images using the partially noised images and then denoise them with guidance toward the original image. DA-Fusion significantly improved few-shot classification performance on the Pascal and COCO datasets.This paper is for the advancement of offline Reinforcement Learning. While it opens up various potential applications, any specific societal impacts are not immediately apparent and require further exploration.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.https://github.com/conglu1997/SynthER 


Decision: Reasons to accept:
- Offline RL and specifically data augmentation in RL is a research problem of practical significance.
- Paper clearly articulates a specific problem in data augmentation for offline RL and develops a method to address the problem.
- Empirical evaluation is appropriate (show casing the proposed benefit).

Reasons to reject:
- All concerns raised by the reviewers were addressed during paper discussions.

Summary: This paper identifies a significant problem for offline RL applications and develops a method for data augmentation that addresses it. An empirical study confirms the proposed benefit. The paper is clear and accessible to the RL audience at NeurIPS.

Reviews: review: paper_summary: paper_summary: The paper presents Generative Trajectory Augmentation (GTA) that is aimed at improving offline reinforcement learning (RL). GTA uses a diffusion model conditioned on high returns to generate high-rewarding trajectories. These trajectories are used to augment the static datasets used to train offline RL algorithms. The augmentation process involves partially adding noise to existing trajectories and then refining them using a denoising mechanism guided by an amplified return value. The authors demonstrate that GTA improves the performance of popular offline RL algorithms across tasks in the D4RL benchmark. They also analyze the quality of the augmented trajectories, showing improvements in both data optimality and novelty over two baselines (S4RL and SynthER).
strength_weakness: strength_weakness: 1. As shown in Figures 4 (a)(b) and Table 25, the performance of GTA is sensitive to the choice of hyperparameters, such as the noising ratio ($\mu$) and the multiplier for conditioned return ($\alpha$) which might require extensive tuning for different tasks.
2. While empirical results show that the generated trajectories using GTA are dynamically plausible (Figure 5), it is not explicitly enforced in GTA and might be a byproduct of generating high-rewarding trajectories. The reviewer believes that further evaluation of GTA on real-world data is needed to make claims regarding GTA's dynamic plausibility such as in lines 6-9 (“In response, we introduce … and dynamically plausible”), lines 39-40 (“GTA is designed … dynamic plausibility”).
3. Related to weakness #2, while the paper shows GTA's effectiveness on standard benchmarks, it lacks validation in real-world applications where the dynamics and data distributions may differ significantly from simulated environments. This poses a question on the real-world applicability of GTA.
4. The paper does not include any experiments comparing GTA with model-based RL baselines. Although the authors state that both approaches perform data augmentation at different stages (data generation vs policy learning), the reviewer believes that a comparative study should be included. This is because both approaches learn a separate model to generate augmented data.

Minor:
- Line 48: “... any offline RL algorithms …” should be “... any offline RL algorithm …”.
- A pink curve is present in Figure 4 (b), but its legend is missing. The blue curve denoting $\mu$ = 1.0) is missing in this figure,
questions: questions: 1. Why is IQL the only chosen baseline (and not TD3-BC, CQL, MCQ) for the tasks listed in Table 2 (Adroit and FrankaKitchen)?
2. Why is DrQ+BC the only chosen baseline and Cheetah-run the only chosen pixel-based observation task in Table 3? Why were results on the other baselines (IQL, TD3-BC, CQL, and MCQ) not presented for the pixel-based task(s)?
3. How compute-intensive (GPU or hours) is running GTA on the pixel-based observation tasks?
limitations: limitations: 1. As the authors have mentioned, while empirical results on the D4RL benchmark tasks show that GTA has low dynamic MSE, in tasks where dynamic violations have a critical impact, the performance boost may not be significant.
2. The current evaluation is limited to simulated tasks, and the effectiveness of GTA in real-world offline RL tasks remains to be validated.
3. As shown in Figures 4 (a)(b) and Table 25, the performance of GTA is sensitive to the hyperparameters ($\mu$ and $\alpha$) which might limit its applicability.
paper_summary: paper_summary: The paper introduces Generative Trajectory Augmentation (GTA), a data augmentation approach for Offline Reinforcement Learning (RL) that enhances the quality of static datasets by generating high-rewarding and dynamically plausible trajectories using a conditional diffusion model. GTA partially noises original trajectories and then denoises them with classifier-free guidance via conditioning on amplified return value. The authors demonstrate that GTA improves the performance of various offline RL algorithms across several benchmark tasks.
strength_weakness: strength_weakness: - The claim that GTA-generated data adheres to the dynamics of the environment (lines 145-148) seems unfounded. Is there a principled argument on why diffusion models would learn the dynamics of the environment well? Especially where the goal is to create transitions outside the dataset. Figure 5 does not conclusively show that GTA is dynamically plausible, especially for tasks like Cheetah compared to methods like S4RL.
- Compared to S4RL, the augmented trajectories appear less plausible. This issue is evident in tasks like HalfCheetah, yet GTA still achieves better rewards. It is unclear why this does not affect the final results.
- Based on Appendix B.2, the performance of GTA appears to depend heavily on finely-tuned hyperparameters, as suggested by the different values used for different environments in Table 6. This raises concerns about the generalizability of the method, indicating that it might rely on online manual tuning.
questions: questions: - How do you ensure that the generated trajectories are plausible and adhere to the environment's dynamics?  What mechanisms to pick &alpha to handle cases where the return used for high-reward guidance is not reasonable? How do you ensure that the generated trajectories remain valid and useful? 
- Does each task require a new &mu, &alpha hyperparameter? How does this impact the generalizability and practical application of GTA?
- What is the proportion of augmented trajectories to original ones in the final dataset? Is the entire offline dataset kept before training, with augmented trajectories added on top? Do all the offline datasets have the same size? Are 5 million augmented tranitions added regardless of the original dataset's size?
- Could you clarify if Figure 3 is based on a real example or if it is just an illustration of the method's intuition? If it is the latter, how can you ensure it accurately represents the method's practical application?
limitations: limitations: The authors discuss the limitations and potential impacts in their paper.
paper_summary: paper_summary: This paper builds on ideas from SynthER but adds classifier free guidance to boost the returns of the generated trajectories. This makes sense as the resulting data has higher quality and the paper is a totally sensible next step in the series of works building upon SynthER, which is an exciting direction for research. The paper is well executed and a solid contribution.
strength_weakness: strength_weakness: There are no major weaknesses here, it is a solid paper taking a nice step in an exciting general direction of research. The below points are fairly minor:
* The authors could cite Ball et al 2021 "Augmented World Models" as another example of data augmentation in offline RL.
* The authors could discuss how this relates to another paper building on SynthER, Policy-Guided Diffusion by Jackson et al. This is very recent work so makes sense it is not needed as a baseline, but the comparison may be worth including in text.
questions: questions: NA
limitations: limitations: Nice to see this in the main body rather than the Appendix. It would be good to add a discussion on scalability if possible.
paper_summary: paper_summary: To improve the quality of offline datasets, in this paper, a generative data augmentation approach is proposed by leveraging the diffusion models. Moreover, with the adoption of partial noising and denoising framework with amplified return guidance, the trajectory can be guided towards high-rewarding region. Finally, with the generated trajectories, existing offline RL methods can be utilized to learn the optimal policy.
strength_weakness: strength_weakness: The proposed approach is mainly a direct combination of current techniques, e.g., the diffusion model for data generation, the adding noise for the exploration and the existing offline RL techniques for policy learning, the theoretical contribution of the paper is trivial. Also, experimental results are still not sufficient, e.g., the lack of the comparations of time and computation costs between different methods, and only two data augmentation baselines are compared. Moreover, the proposed approach is limited to trajectories with reward signals, which is not the case in many real-world applications. Furthermore, it is unclear to me whether there are any theoretical guarantees to push the generated trajectory toward the high-rewarding region.
questions: questions: 1. In section 4.2, by denoising with amplified return guidance, the authors claim the trajectory can be guided towards high-rewarding regions, are there any theoretical guarantee for this statement?
2. How about the whole computation and time cost of the proposed approach when it is compared with other diffusion-free baselines? 
3. As shown in the experiments, for different environments, different noise levels are required for the optimal performance, then how can we determine the noise level for a new task? 
4. The proposed approach aims to deal with trajectories with reward signals, however, in reality, it may be difficult to obtain such signals, can the proposed approaches be extended to the cases where no reward signals can be provided?
limitations: limitations: The authors have addressed some limitations, but for some other limitations, e.g., the computation and time cost of the proposed approach, the application of the proposed approach in some real-world applications with no reward signal provided should be further stated.
paper_summary: paper_summary: This paper tackles the challenges of Offline RL, which involves learning effective decision-making policies from static datasets without online interactions. The authors introduce Generative Trajectory Augmentation (GTA), a novel approach that uses a diffusion model to enhance offline data by augmenting trajectories to be both high-rewarding and dynamically plausible. GTA partially noises original trajectories and denoises them with classifier-free guidance based on amplified return values. The experiment results indicate that GTA, as a general data augmentation strategy, enhances the performance of widely used offline RL algorithms in both dense and sparse reward settings.
strength_weakness: strength_weakness: (1) Lack of theoretical guarantee to support the algorithm: The authors only provide empirical results, lacking theoretical analysis of the performance of the proposed method.

For other potential weaknesses, please see my questions.
questions: questions: I will adjust my rating score based on the rebuttal results.

(1) How to select the hyperparameter \alpha in the denoising with amplified return guidance in equation (8). Can you also provide ablation study results for different selections of \alpha?

(2) In the generation phase, do you generate whole trajectories or generate subsequences of trajectories?

(3) Is the proposed method sensitive to the hyperparameter \mu? Based on the description in Figure 4 and Appendix F.3, when \mu = 1, it only performs well on the dataset with low quality. For high-quality datasets with abundant expert demonstration, does your method have trouble preserving and reconstructing the high-rewarding trajectories?

(4) Can you provide comparison results with diffusion-based offline RL methods? For example, Decision Diffuser [1] and AdapDiffuser [2].

(5) In the ablation study on reweighted sampling, what base offline RL do you use for D4RL score calculation?

(6) Since the reweighted sampling can implicitly change the dataset distribution, can you also apply the reweighted sampling technique as well for other baseline methods?

Reference:

[1] Ajay, A., Du, Y., Gupta, A., Tenenbaum, J., Jaakkola, T., & Agrawal, P. (2022). Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15657.

[2] Liang, Z., Mu, Y., Ding, M., Ni, F., Tomizuka, M., & Luo, P. (2023). Adaptdiffuser: Diffusion models as adaptive self-evolving planners. arXiv preprint arXiv:2302.01877.
limitations: limitations: Yes, the authors discussed the limitations in their paper.

Comments: title: Notice of correction to Table 3 in general response
SGTM
comment: We would like to express our gratitude for your thorough review and thoughtful comments on our rebuttal.

Following your suggestion, we will tone down our statement from "ensuring dynamic plausibility" to "minimizing the degradation of dynamic plausibility". We apologize for the overclaiming of our proposed method.

If you have any additional suggestions or considerations, please feel free to share them with us. Once again, we sincerely appreciate your thoughtful feedback.
Thank you for your detailed response and for providing additional supporting and clarifying empirical results. In line with reviewer **WvGz**, I am still skeptical about the paper's claims regarding GTA ensuring "dynamic plausibility". I would be more comfortable if such a claim is toned down; for instance, augmented trajectories generated from benchmark datasets using GTA exhibit dynamic plausibility.

I am satisfied with the author's responses to the rest of my questions and concerns and have raised my rating from 5 to 6.
Thank you for your thoughtful response and detailed review of our rebuttal. Following your suggestion, we will expand the ablation study presented in Appendix G by conducting additional experiments across various tasks and incorporating these results into the revised manuscript.

If you have any further suggestions or considerations, please feel free to share them with us. Once again, we sincerely appreciate your insightful feedback.
I appreciate the detailed response, most of my questions have been addressed. Regarding the ablation study against \alpha and \mu, and the comparison results with reweighted sampling, I believe they should be evaluated more comprehensively with more testing values and tasks. Considering the limited time of the rebuttal phase, I understand that the authors can not provide the full results. However, I suggest to have a more detailed discussion regarding this part in their revision.

After reading the reviews from other reviewers and the authors' responses, I decided to raise my score from 5 to 6 in favor of acceptance.
Dear Reviewers,

We hope this message finds you well.

We have noticed that reviewers **cNng** and **UiuM** have not yet participated in the discussion. As the author-reviewer discussion period is set to conclude in two days, we kindly wish to remind you of the opportunity to provide your valuable feedback.

Your insights and perspectives are incredibly important to us and will greatly contribute to improving our work. If you have any questions or require further information, please do not hesitate to reach out.

We sincerely appreciate your time and consideration, and we look forward to your input.

Best regards,
The Authors
We recently noticed that Table 3 was mistakenly uploaded as a duplicate of Table 1 in our general response. We sincerely apologize for any confusion this may have caused the reviewers and the area chair. We have now uploaded the correct Table 3, which shows a performance boost on medium-replay quality datasets, and we kindly ask you to review the updated table. Once again, we apologize for the inconvenience.


Table 3. D4RL normalized score on medium-replay quality locomotion environments with fixed $\alpha$ and $\mu$. The experiments are conducted with TD3BC.

| Env             | None          | SynthER       | GTA($\mu = 0.5, \alpha=1.3$) | GTA($\mu = 0.75, \alpha=1.3$) |
| --------------- | ------------- | ------------- | ------------------------------ | ------------------------------- |
| halfcheetah-m-r | 44.64 ± 0.71  | 45.57 ± 0.34  | 48.23 ± 5.42                   | **48.99 ± 2.16**                    |
| hopper-m-r      | 65.69 ± 24.41 | 78.81 ± 15.80 | 77.17 ± 22.17                  | **97.26 ± 3.38**                    |
| walker-m-r      | 84.11 ± 4.12  | 90.67 ± 1.56  | **91.12 ± 6.45**                   | 90.89 ± 3.29                    |
| Average         | 64.81         | 71.68         | 72.17 ± 11.23                  | **79.05 ± 2.90**                    |
Dear Reviewer UiuM, thanks for giving the authors a detailed list of questions. Did the authors answer these to your satisfaction or did they raise any concerns for you about accepting this paper? If so, it would be great to clarify while we can still interact with the authors. Thanks!
Thank you for your kind response and for thoroughly reviewing our rebuttal. While most of your concerns have been addressed, if there are any remaining issues or points of discussion, please feel free to share them with us. We are always ready to engage in further discussion. Once again, we appreciate your thoughtful feedback.
I appreciate the authors' detailed response and additional experimental results，most of my concerns have been addressed, I'd like to raise my score to 6, best.
Thank you for your kind response! Please let me know if you have any further suggestions or adjustments you would like us to consider.
We would like to express our gratitude once again for your feedback, which led our paper to a more progressive and positive direction. In response to your comments, we will revise our paper to include additional details about Figure 5 and provide practitioner guidance for the hyperparameter settings.

We are fully prepared to respond to any additional discussions or inquiries you may have, so please feel free to reach out at any time. Thank you.
Thank you for your answers.
Thanks for the response!