Trajectory Generation with Conservative Value Guidance for Offline Reinforcement Learning
Authors: Tieru Wang
Kunbao Wu
Guoshun Nan
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Sequential Decision Making
Primary area: reinforcement learning
TL;DR: We propose TGCVG, a simple yet effective offline RL framework that synthesizes high-quality trajectories via conservative value-guided generation.

Abstract: Recent advances in offline reinforcement learning (RL) have led to the development of high-performing algorithms that achieve impressive results across standard benchmarks. However, many of these methods depend on increasingly complex planning architectures, which hinder their deployment in real-world settings due to high inference costs. To overcome this limitation, recent research has explored data augmentation techniques that offload computation from online decision-making to offline data preparation. Among these, diffusion-based generative models have shown potential in synthesizing diverse trajectories but incur significant overhead in training and data generation.
In this work, we propose Trajectory Generation with Conservative Value Guidance (TGCVG), a novel trajectory-level data augmentation framework that integrates a high-performing offline policy with a learned dynamics model. To ensure that the synthesized trajectories are both high-quality and close to the original dataset distribution, we introduce a value-guided regularization during the training of the offline policy. This regularization encourages conservative action selection, effectively mitigating distributional shift during trajectory synthesis.
Empirical results on standard benchmarks demonstrate that TGCVG not only improves the performance of state-of-the-art offline RL algorithms but also significantly reduces training and trajectory synthesis time. These findings highlight the effectiveness of value-aware data generation in improving both efficiency and policy performance.

Published as a conference paper at ICLR 2026
TRAJECTORY GENERATION WITH CONSERVATIVE VALUE GUIDANCE FOR OFFLINE REINFORCEMENT LEARNING
Tieru Wang Kunbao Wu
Guoshun Nan"
Beijing University of Posts and Telecommunications, China (wangtieru, wukunbao,
ABSTRACT
Recent advances in offline reinforcement learning (RL) have led the develop- ment of high-performing algorithms that achieve impressive results across stan- dard benchmarks. However, many of these methods depend increasingly complex planning architectures, which hinder their deployment in real-world set- tings due high inference costs. To overcome this limitation, recent research has explored data augmentation techniques that offload computation from on- line decision- makinn to offline data preparation. Among these, diffusion-basee generative models have shown potential in synthesizing diverse trajectories but incur significant overhead in training and data generation. In this work, we pro- pose Trajectory Generation with Conservative Value Guidance (TGCVG), novel trajectory-level data augmentation framework that integrates high-performing offline policy with learned dynamics model. To ensure that the synthesized tra- jectories are both high-quality and close to the original dataset distribution, we in- troduce value-guiae guudedd regularization during the training of the offline policy. This regularization encourages conservative action selection, effectively mitigating dis- tributional shift during trajectory synthesis. Empirical results standard bench- marks demonstrate that TGCVG not only improves the performance of state-of- the-art offline RL algorithms but also significantly reduces training and trajectory synthesis time. These findings highlight the effectiveness of value-aware data gen- eration in improving both efficiency and policy performance. Our code available at /nttps://githhi com/
INTRODUCTION
Offline Reinforcement Learning (RL) (Kumar al. 2020 Fujimoto Gu 2021, Kostrikov al, 2021b), which focuses on training policy from the static dataset, is emerging as the critical solution for environments where online interaction costly or risky. The main challenge of offline RL the distributional shift which would cause extrapolation error (Fujimoto al. 2019). To mitigate this problem, many previous works add the constraints to estimated value functions (Kumar et al., 2020 Lyu et al. 2022) and policy networks (Wang al.) 2020; Fujimoto & Gu 2021), learn without querying out-of-distribution (OOD) samples (Kostrikov et al. (021b) Hansen-Estruch et al., works have introduced strong generative architectures like Diffusion and Transformer (Wang et al. 2022) Chen et al.) 2024 Kim et al. 2023 Hu et al.) 2024) for richer
2023 Moreover,
representation learning to train the policy.
However, these methods often involve increased model and algorithmic complexity, leading to longer inference time and raising concerns about their practicality in real-world deployment. To address this issue, recent studies have explored data augmentation techniques that aim to broaden the dataset distribution and improve the performance of simpler offline RL algorithms (Kumar al.) 2020 Fujimoto & Gu 2021 Kostrikov et al.] 2021b). Early works primarily enrich the data dis- tribution by adding noise states (Laskin et al. 2020, [iinaa et al.) (022), but such perturbations are limited in diversity due to their constrained range. More recent approaches introduce generative models, particularly Diffusion models (Lu et al. 2023
et al. 2024 Lee et al.) 2024; Yang &
"Corresponding author.

---

Published as a conference paper at ICLR 2026
which offer stronger mechanism for synthesizing diverse and dynamics-consisteee samples. However, Diffusion models are computationally expensive to train and require multi-step denoising for accurate generation. Moreover, they typically lack explicit guidance toward high- value regions, resulting in limited improvements in data quality and marginal gains in offline policy performance.
Wang
2025
To address these challenges, we propose a simple and effective data synthesizer for offine RL, termed Trajectory Generation with Conservative Value Guidance (TGCVG). Our method uses Transformer-based policy network trained with Conservative Q-learning (CQL) to generate high- quality actions. dynamics model from model-based RL (Lin al.) 2024) is then employed to predict the corresponding rewards and next states, yielding set of synthetic transitions. Intuitively, this process mimies the original dataset collection procedure, where an online policy interacts with the environment to generate trajectories (Fu et al., 2020) Hatner et al. 2020). Similar to prior works (Li al. 2024 Lee al., 2024), our method generates sequential data, but replaces the Diffusion module with the Transformer to significantly reduce computational overhead. Transformer has been shown to be effective in offline RL tasks (Chen et al. 2021] Hu et al. 2024; Kim et al.) 2024), making them practical and efficient alternative. The key component of TGCVG is conservative value guidance, which constrains each generated (st, dg $t+1) tuple to lie within the dataset distribution. This limits OOD risk single step each model interaction and prevents its accumulation across rollouts. As a result, TGCVG produces dynamically consistent trajectories that are more stable and reliable for offline policy training.
Our Contributions are summarized as follows. (1) We propose TGCVG, a novel value-guided data synthesizer that generates high-quality trajectories while significantly reducing computational over- head. (2) We demonstrate that TGCVG consistently achieves state-of-the-art performance across wide range of offline RL benchmarks, validating the effectiveness of our method. (3) We provide a comprehensive ablation study to analyze the impact of key design choices and demonstrate the robustness of our pipeline.
2 RELATED WORK
2.1 MODEL-FREE OFFLINE REINFORCEMENT LEARNING
Model-free offline RL algorithms aim to train policy networks directly from static datasets. Several methods (Fujimoto & Gu 2021: Kostrikov al. 2021b Lyu et al., 2022] adopt the value-based paradigm from online RL by learning Q-functions to guide behavior improvement. To address dis- tributional shift,
approaches typically incorporate explicit policy constraints (Kumar et al. 2019: Wu et al. (2019) Fujimiuu & Gu Gul 2021). penalize overestimated Q-values on OOD sam- ples Kumar al. 2020; Kostrikov et al.] 20.1a Wu et al., 202T). or avoid querying OOD actions Wang et al.) 2018 Kostrikov et al. 202 Tb, Hansen- et al. (023). Beyond these methods, generative models such as Transformer (Chen et al.) 2001 Kim et al.) (023) and Diffusion models (Janerr et al. 2022]] Ajay et al.) 2022) have been introduced for modeling offline RL tasks. Recent works further explore hybrid approaches that combine generative modeling with value-based tech- niques (Wang al. 2022 2024: Hu et al., 2024 Gao et al.) 2025 to enhance policy improvement. However, as algorithmic architectures grow increasingly complex, the computational cost of policy evaluation rises substantially, limiting the practical deployment of offline RL algorithms.
2.2 MODEL-BASED OFFLINE REINFORCEMENT LEARNING
Model-based offline RL algorithms aim learn dynamics model from static datasets and leverage it to derive or improve policies. A central challenge in this setting is how to effectively utilize the learned model. Prior works address this by quantifying model uncertainty (Yu et al. 2020 Sun et al,) 2023). learning conservative value functions Yu et al., 2021 Jeong et al. (022). transforming one-step models into multi-step transition models (Machado et al. 2023 Lin al. 2024). While data augmentation methods (Lu et al.) 2023 Lee et al., 2024h also generate synthetic trajectories, they decouple data generation from policy learning. This modularity enables the synthesized data to be reused by model-free algorithms, offering greater flexibility and generalization. Notably, such methods demonstrate superior scalability in high-dimensional environments, where model-based approaches often suffer from compounding model errors and poor generalization.
2

---

Published as a conference paper at ICLR 2026
2.3 DATA AUGMENTATION OFFLINE REINFORCEMENT LEARNING
Data augmentation has emerged as an important technique for improving the performance of offline RL algorithms. Traditional approaches Yaarats et al. 2021 Laskin et al. 2020 Sinha et al. 2022) apply simple transformations such as noise injection random translations to pixel-based or state- based observations. With the rise generative models, recent works have explored generative data augmentation. Several studies (Lu ct al. 2023 He et al. 2023 Li ct al. 2024 Yang & Wang 2025)
leverage Diffusion models to approximate the data distribution and synthesize dynamically plausible transitions. Building on Diffusion-ber based data augmentation, GTA (Lee et al. 2024) further proposes generating high-return trajectories to improve overall dataset quality. This reward-guided design has been shown to enhance the performance of offline RL algorithms. Our TGCVG differs from prior methods in two aspects: (1) it replaces costly Diffusion models with a lightweight Transformer- based policy for efficient sequential data generation; (2) it leverages conservative value guidance to steer synthesis toward high-return trajectories while remaining within the dataset distribution.
PRELIMINARIES
3.1 OFFLINE REINFORCEMENT LEARNING
Reinforcement learning typically formalized as a Markov Decision Process (MDP) (Bellman
1957 defined by the tuple (Po, S, A, P. 7), where Po denotes the initial state distribution, $ and A are the state and action spaces, P(s"|s, a) is the transition probability, R(s, a) is the reward function, and y is the discount factor. The objective of standard RL is to learn policy x' (als) that maximizes the expected return E " az)] through direct interaction with the environment. In contrast, offline RL focuses on learning from fixed dataset D {(s, a, r, s')} collected by behavior policy TH. without further environment interaction. This setting poses unique challenges, the agent must generalize from static data without the ability to explore.
3.2 TRANSFORMER IN OFFLINE REINFORCEMENT LEARNING
Transformer based methods typically frame offline RL as a supervised learning problem to improve training stability. Among them, Decision Transformer (DT) (Chen al. 2021) is seminal approach that models trajectories as sequences of states, actions, and returns-to-ge (RTGs), where the RTG at timestep t is defined as the sum of future rewards: R, Ei Tr. At each timestep f, DT takes (Rt-L+1. 9t- d ... Rt-1, St-1, ar-1. Re. st), where L denotes the sequence length, and predicts the next action dz. During training, the model is optimized to match the ground- trutt actions from the dataset. In the evaluation stage, since true RTGs are unavailable, DT conditions on pre-specified target RTG that represents the desired return. At each step, the received reward is subtracted from the target RTG until the episode ends or the maximum trajectory length is reached.
as input a sequence
While Transformer-basee based methods benefit from leveraging long-term historical information for decision-making, their lack of explicit value guidance can hinder effective trajectory stitching. To overcome this limitation, recent work has explored integrating value-based learning with sequence modeling to improve policy optimization (Yamagata et al., 2023 Gao et al. 2024 Hu et al. 2024 Kim et al., 2024), effectively improving decision quality and leading to stronger empirical perfor- mance.
4 METHOD
In this section, we propose Trajectory Generation with Conservative Value Guidance (TGCVG), Transformer-based generative framework that interacts with a pretrained dynamics model to pro- duce high-quality offline trajectories. We first describe how to train the Transformer policy using conservative Q-value guidance. Then, we introduce the pipeline for trajectory-Ievel data genera- tion using the learned policy and the dynamics model. The overall architecture is summarized in Figure I Finally, we employ standard offline RL algorithms on the augmented dataset to improve performance.
3

---

Published as a conference paper at ICLR 2026
Value-Guided Data Augmentation
Originl Destibuttoon
Conservative Fatimote
True Valu
Low Reward Region
Value/Poliey PDE
Leamed Palicy
High//oower Region
Detaset samples
Orignnl Datepoint
Datapoint
Action
Offline Augmented
Train Models
Generate Trajectory Data
Trans former
Policy Model Training
Dynomics
Dynamics Model Training
Transformer
Dynamics Model
Figure 1: Overview of the TGCVG Architecture. the first stage, we train conservative policy network and a dynamics model to facilitate data generation. In the subsequent stage, we exploit the interaction between these components to synthesize new trajectories that align with the high-reward distribution of the offline dataset.
4.1 CONSERVATIVE Q-LEARNING BASED TRANSFORMER
Building upon Transformer-basee architectures, explore the integration of return-conditioned supervised learning (RCSL) (Brandfonbrener et al.] 2022) with Q-learning to optimize the policy network. To streamline training, we eliminate action and RTG tokens according DD (Ajay al.) 2022) and our ablation study in Appendix [.I which demonstrates that RTGs are unnecessary when employing Q-value regularization. The resulting trajectory representation, consisting solely of states, is defined as:
Tt -S1-Ltt,, St-1, St),
where I. represents the trajectory length. Adopting the conservative value estimation technique from CQL (Kumar et al. 2020), employ ensemble of five networks: Q-networks Q Q
two target networks Qay dy Qøy and the policy network ma. The Q-networks are trained by solving the following optimization problem:
min AE, ABer 81) log Σ eepp(Qo az)) Far (86s a.),
+ E
Σ
|@ Qo. (Sm,, amill
m
where Qm i-mmy + y + min (8541,,,
1=1
j=m
In this formulation, p((-(s:) is used to match the marginal distribution the dataset, âe41 is sampled from the policy Te, (-)s)): denotes the behavior policy, 7 is the discount factor, and À is weighting coefficient. We adopt the n-step Bellman backup to estimate the Q-value function, which has been shown to outperform the 1-step approximation in recent studies (Hu et al. 2024). The first term in
4

---

Published as a conference paper at ICLR 2026
Algorithm Transformer--QL
Input: Sequence horizon I., offline datasets D, coefficient p.
Initialize policy network тæ, critic networks Qs, and QΦ and target networks Qot and Qos. for t 1 to T" do
Sample mini-batch B {[s}. oj T;) ~ D.
// Q-value function learning
Sample de+tr. TT(ār+ r. 11t+.
Update Qo and Qøz by Equation
I/ Policy learning
for iori=00 0 to L do
Sample â To ro(0e+ilsaa end for.
Update policy by minimizing Equation I1 Update target networks
$ - pat (1 p)d; for 21,
end for.
the objective serves as a regularization component, penalizing Q-values for OOD state-action pairs while preserving values for those within the dataset distribution.
The policy learning objective is defined as:
(()) = min D (1))) a log (ā(()))))) - min Qo, (54, (s)) 1=12
(3)
where a arg min E B..
loga (log No (ā, (s)) + Hurret))
where Hurget is predetermined by the environment's action space. The policy loss follows the SAC (Haarnoja et al.) 2018) framework to promote policy improvement while encouraging suffi- cient exploration. We detail the learning procedure Algorithm [ In addition our conserva- tive value-based optimization, an alternative Transformer-baset Q-learning approach exists, which incorporates value improvement into the supervised learning objective by adopting the policy con- straint paradigm from TD3BC (Hu et al. 2024). For clarity, we refer to these two approaches as TrL and Transtormer-TDD TD3BC, respectively. this work, we ultimately adopt Transformer-CQU and the rationale for this choice is discussed in Section 5.3
4.2 GENERATING SEQUENCE DATA WITH THE DYNAMICS MODEL
In the data generation phase, we mimic the online data collection process, where an agent interacts with the environment to collect (s. s') tuples. Specifically, interact with pretrained dynamics model (Lin et al.) 2024). To generate synthetic transitions, we first sample state sequence of length K from the original dataset:
the learned policy network to
[8t-K+1,'". 1, 8t).
(4)
We then take the first I. states from this sequence, denoted as T (st-K+1, 8t- K+L), and feed them into the policy network to obtain the action åz-K+L- The next state and reward are predicted by the dynamics model fw as:
Št- f.l---t--ftttt-flf-----------
(5)
The predicted state 94-K+41 is appended to the state sequence, forming new window Tt-K+L+1 (s:- 8 ++11), which is used as input to the Transformer- based policy. This autoregressive process is repeated until we construct the full generated trajectory:
8t- K+L
$t-1, St. 6
=
ûr- K4L
Φ de: 0
6
ft-K+L
fr-1, ft.
Other auxiliary information (e.g., terminal indicators) is directly aligned with and inherited from the corresponding timesteps in the original sequence, thereby preserving the original episode termina- tion signals. The generated trajectories are fused with original trajectories, leading to our augmented dataset.
5

---

Published as a conference paper at ICLR 2026
4.3 TRAINING OFFLINE RL ALGORIHTMM WITH AUGMENTED DATASET
Following the data generation phase, we obtain a set of new, high-quality trajectories. Consistent with prior works (Lu al. 2023 Lee al. 2024), we combine the generated trajectories with the original dataset via random shuffling. The resulting augmented dataset is then used to train standard offline RL algorithms.
EXPERIMENTS
In this section, we present series of experiments to evaluate the effectiveness of TGCVG. We begin by outlining the experimental setup and demonstrate that our method consistently enhances the performance offline RL algorithms across range of environments. We then perform ablation studies to assess the contribution of each component and provide rationale for our design choices. Finally, we investigate key characteristies of TGCVG to gain deeper insights into its behavior.
5.1 EXPERIMENTAL SETUP
Datasets. We conduct experiments on the D4RL benchmark (Fu al. 2020), including MuJoCo Gym, Maze2D, and AntMaze tasks. The MuJoCo locomotion suite consists of widely used bench- mark tasks offline RL, characterized by smooth reward functions and high proportion of near- optimal trajectories. The Maze2D datasets are designed to evaluate an algorithm's ability to effec- tively stitch sub-trajectories in navigation tasks. AntMaze presents more challenging maze nav- igation environment, featuring sparse 0-1 rewards and higher-dimensional state and action spaces compared to Maze2D.
Baselines. We compare our method against several data augmentation baselines: (1) S4RL (Sinha et al. al.] 2022), which perturbs states with Gaussian noise; (2) Synther (Lu et al. (023). which employs diffusion model to learn the transition distribution from offline data and generate new samples; (3)
GTA (Lee et al. 2024), which augments offline data via a partial noising and denoising process guided by amplified return signals. The performance scores for these baseline methods mainly sourced from results published in (Lee et al., 2024), ensuring a fair comparison.
Offline RL Algorithms. We evaluate our TGCVG using several representative algorithms: (1)
CQL (Kumar et al.] 2020), which penalizes Q-values on unseen actions to enforce conservatism; (2) TD3BC (Fujimoto & Gul 202T), which constrains the learned policy stay close to the behavior policy; IQL (Kostrikov et al. (2021b), which performs implicit value regularization by querying only in-distribution actions; and (4) DT (Chen et al., (021). which takes trajectory data as input and applies Transformer to model the distribution of trajectories in the offline dataset. Following GTA, for
we employ only IQL. for the Maze2D and AntMaze datasets, as provides complete hyperparameter settings and demonstrates stable performance in these environments.
5.2 MAIN RESULTS
Tables D and 2 report the normalized scores across three task domains, following the protocol of (Fu
al. 2020). We analyze results by domain.
Gym Domain. Our TGCVG consistently boosts the performance of all offline RL algorithms, with
CQL and TD3BC achieving results comparable recent state-of-the-art methods (Hu et al., 2024 Kim et al. 2024). This highlights the effectiveness of the policy used for data synthesis and demon- strates that the conservative value guidance effectively mitigates distributional shift. Gains are es- pecially notable on suboptimal datasets (e.g., medium, medium-replay), where trajectory stitching is more critical, indicating the advantage of our Transformer- CQL design in imperfect data regimes.
Maze2D and AntMaze Domains. In Maze2D, which evaluates trajectory stitching capabilities, our TGCVG generates useful data across both simple and complex environments. In the more challenging AntMaze tasks, which are characterized by sparse rewards and high-dimensional state spaces, TGCVG still enables model-free learners to perform competitively, even though it employs similar dynamics models those used in previous model-based approaches (Yu et al. 2020 Sun cet al. 2023 Lin et al. 2024). These results suggest that placing greater emphasis on policy learning may be more beneficial than solely refining dynamics models.

---

Published as a conference paper at ICLR 2026
Table I: Normalized results of TGCVG and baselines the MuJoCo datasets. Average and standard deviation scores are reported over 5 seeds and the best average values are marked in bold. The dataset names are abbreviated as follows: medium 'm', meplum-replay to 'm-r', and medium-expert to 'm-e".
halfchootah
waller?d
Average
Nane
48.42-4-6.2
44.64:0.71
53.48-4-.50
3.18
65 24.44
104.08::5.0
84.58=1199
$4.11:4.112
110.23-40
76.512
S4RL
110.21
77.24
TD3BC
SynthER
19.16:10.399
0..37
15.47.1:1..5
63.70:133.0
78.81
1:
98.99
11.27
15.43:1114
90.67
1.55
109.95
1:0.32
78.64
GIA
57.54+00.51
50.04
93.13+-3007
09.57-4-.05
19.31
110.40::4404
86.69 0.89
93.824
1.24
110,%
84.63
TGCVG
08.144-0.58
57.76-0.32
94.67+0067
99.47
00.47:44.006
94,08
15:35
106.75--6.74
87.50
110.124-0.22
89,30
Nane
46.96-20.30
44.20:00:11
95.00:0.522
61.134:3.20
82.31-116.37
67
104.38:7.33 30
62.26=1111
79.74::5119 19
109:0022:43
78.55
SIRL
47.00:1::23
05:00-1::45
62:72:1346
98.82:19.89
80882:8.35
60:00:2
CQI
SynthER
47.31::00.44
46.03:20.40
05:29:::000 1.00
64:65:44788
1340
107 07766626660
81.91
8667233001
109 36:::600033
81.20
GTA
54.14::0331
51. 51.36220.27
94.943371
7480:27422
98.84233551
110.90
80. 80.40=
91.57::5.115
110.44:::022
85.27
TOCVG
68.31::0:27
59.03:00.28
97.40:10:47
28.25:1:999
101.80:0:.13
111.36::1330
05.69:0.34
9.291:11.22
109.79::0030
90.50
Nane
48.65--0.119
43.35-+0.50
94.57-+1188
56.35 46005
95.76--4.01
91.69
34.34
09.00 10.80
112.37 112.57++00.00
78.52
S4RI
48.5840.29
43.57:00.5
66 5.94
26.722222011
99.82
84.58
70:33:
112.29::007
78.35
IQL
Syameg
29.764-227
46.91-00.28
91.00--3225
45215585
102.97 1:66
94.08
80,15-16.44 16.47
9063-44..6
112.12+40.58
81,97
GIA
54.824-035
46.89-33.00
05550-
71.46-33.22
102.11:11511
107.78-44.66
84.20=1.32
93.77 66.75
112.8) +-0.66
86.11
TOCVG
59.63=0.22
95.99:0..99
81.55+:5.42
10099:00.7
97.30 10:96
81 21,00=2955
86 34.32 10.94
112.99:007
87.09
Nane
42.43:1:.14
3.234::..22
92.41:1:.50
63:09:12.49
81.81:3.39
109055:2.02
71.64:0.69
62.06:1:188 1.88
108.38
74,47
S4R1.
42 42.44::0339
38. 08.71
91 77:40777
64.49:1:.70
19.4
110.7:0001
72.22=2499
50.6744991
108.40:0.334
72.75
43,83:1:013
37.98114.77
91.78.11888
78.43:19933
110.5110:18
74.95:1.72
67.90:133411
006.211:0622
75.36
TOCVG
62.48:10052
35.79:29.95
92.76:10.74
60.11:1:4.99
73.36231.59
104.87:77:33
82.18:0099
82.96:1.322
108.061:0000 59
78.09
Table 2: Normalized results of TGCVG and baselines on the Maze2D and AntMaze datasets. Aver- age and standard deviation scores are reported over seeds and the best average values are marked in bold. The dataset names are abbreviated follows: umaze to 'u', medium to 'm', large to T, play to 'p', and diverse to 'd'.
Algo. Aug
muneld m
mipp
antmaae
u-d
m-d
1-1
Avergee
Nows
37.411283
1,
98.99
58.7548.900
18:13:33.44
40.63188755
17.38
65.50:19:46
45.75:66.3
$2.04
S4R1
37.6913.36
34.82:3316
63.93-1147
55.00=10.47
10.47
50.88:2.17
42.8818771
11.6211677
7400:1:9:72
46.1328.54
$1.99
IQL
SynhER
39.00:12226
34.27122511
61.75:1.51
17.1316.45
11.00:120..8
37.50116.48
13.9121183
40.8-1114.15
37.5018.37
36,99
41.68:1141
37.78:1666
76.56:2470
66.50:6691
81.88:14119
44:381:4.66
57.882951
78:13:7885
47.75:6.69
TGCVG
53.7617.50
48.1123044
103.97212.85
41.20:2:.71
23.20:2:144
55.2026668
57:20-2440
84.01114331
57.20:2426
5.3 ABLATION STUDIES
How does Transformer-CQI. perform compared the original CQL? We compare Transformer-CQL with the original CQL, which employs a simple MLP as the policy network. As shown in Table the Transformer-based action synthesizer outperforms the original architecture in CQL. consistent with observations from RCSL methods (Chen et al.) 2021 Wang et al. 2024] Hu et al. 2024). We attribute this performance gap to the superior representation capacity of the Transformer backbone.
How does Transformer-CQL. perform compared to Transformer-TDDBBC As discussed in the method section, design conservative value-guided policy for data generation, in contrast to policies constrained by behavior cloning. We compare two augmentation strategies: Transformer- CQL, which represents the conservative Q-value paradigm, and Transforme:--DDBCC representa: tive method of policy constraint. As shown in Figure 2(a)) Transformer-CQU. yields stable perfor- mance when its augmented data used train both CQL and TD3BC. In contrast, Transformer- TD3BC also converges well when paired with TD3BC, but results in a performance collapse when its augmented data is used to train CQL. To further investigate this discrepancy, we visualize the data distributions in Figure [( The figure shows both portion of the original dataset and the corresponding generated samples. We observe that samples generated by Transformer-CQL remain well-aligned with the original distribution, whereas Transforme:-TDDBCC produces dense clusters of outlier points (highlighted with green circles). According to (Lyu et al. 2022), policy regulariza- tion methods (such TD3BC) typically exert less constraint on Q-values than calue-penalization methods (such as CQL). As a result, Transformer-TD3B. may generate actions that fall outside the support of the original dataset. In the subsequent offline RL. training phase, these dense outlier
7

---

Published as a conference paper at ICLR 2026
Table 3: Ablation the action synthesizer. The final scores trained with TD3BC. Average and standard deviation scores are reported over 5 seeds. The best average values are marked in bold.
Action Synthesizer
hopper-medium
walker2d-medium
halfcheetah--medium
CQL
60.66+:3.83 90.47+4.006
84.98=0.82 87.50=10.20
48.43::0.2 68.14=0.588
Cransformer-CQL
TranfomenCC
TaanoomenCy TDBCC Thandome-TD -DD1C-
Tandsmm-
Transformer-Co COL Augmentation
Trannformen- IIC Augmentation
halfrhectah-mediu raplay
halfthestah-medde medium
Diginal
Diggaal
Gene and
Gererated
Training Stops (x10f)
Training Stops (X10")
(a)
(b)
Figure 2: (a) Ablation study comparing Transformer-CQL- and Transformer-TD3BB (b) t-SNE visu- alization of generated data distributions on hulfcheetah-medium. Each point represents state-action pair.
clusters are incorrectly treated in-distribution data due to CQL's conservatism, while the original sparse samples are mistakenly penalized as OOD data. This mismatch leads to inaccurate value estimation and poor performance.
The effect of X. The tradeoff coefficient À plays a critical role in value penalization. We conduct experiments by varying À on different datasets to investigate how the level of conservatism affects performance. As shown in Figure ((a)] changing À has limited impact on the valker-medium dataset but significantly influences performance on hopper-mediune
smaller X, corresponding to lower degree of conservatism, tends to improve the performance of the downstream offline algorithms. Fig- ure (3b) illustrates that the effect of manifests through influence on the decision-making capacity of Transformer-CC rectly impacts the final offline learning performance. This insight provides a practical guideline for applying our TGCVG: the quality of synthesized trajectories is highly dependent on the decision- making ability of the action synthesizer. Therefore, improving the policy model's ' capability during the early stage of training can lead to higher-quality data generation.
which in turn determines the quality of the synthesized data and indi-
5.4 FURTHER ANALYSIS
Data Quality Analysis. We adopt the data quality metrics introduced in GTA (Lee et al. 2024 to evaluate our generated datasets. Specifically, we consider three metries: (1) novelty, which quantifies the ability of an augmentation method to explore novel state-action pairs; (2) optimality, which measures the actual rewards of the generated data; and (3) dynamic MSE, which computes the
normalized discrepancy between synthesized and real states after applying the transition dynamics. Although baselines provide code for data generation, the implementations of the evaluation metrics are not publicly available. To ensure a fair and consistent comparison, we reproduce the datasets using the official codebase of each method and evaluate all datasets using our own implementation of the evaluation pipeline. Table reports reportss the results standard Gym tasks. While our TGCVG does not achieve the best score in every individual metric, consistently yields the best overall performance, Compared to GTA, our TGCVG achieves better dynamic MSE despite slightly lower optimality, which may imply that dynamic consistency a more important prerequisite than reward magnitude when evaluating the quality of generated data. Notably, although our TGCVG constrains trajectory generation within the original data distribution via value penalization, the novelty scores indicate that the generated samples not mere replicas of existing ones. This suggests that the
8

---

Published as a conference paper at ICLR 2026
walker22dme dumm
hopser-edilum
walker2d medmmm
hoppormeduum
Nurnaked Scare
Nomatoed
Training Staps (x109)
Truining Steps (xilf))
Training Steps (x10f))
Training Staps (X110))
(a)
(b)
Figure 3: Ablation on the X. (a) Final performance of TD3BC trained our augmented datasets. Average and standard deviation scores are reported over 5 seeds. (b) Evaluation scores of the Transformer-Co policy networks, before interacting with dynamics models to generate synthetic trajectories. Average and standard deviation scores are reported over 5 seeds.
Table 4: Data quality analysis of TGCVG and baselines across Gym locomotion tasks. Arrows 1/4 indicate whether higher/lower values are better. The best and the second-best results of each setting are marked as bold and underline, respectively.
Novelty (1) Synther
Optimality (t)
Dynamics MSE (L)
Task
S4RI.
GTA
TGCVG
S4RL.
Synther
GTA
TGCVG
S4R1,
Synther
TGCVG
hulfcheetah medium
0.00
3.72
4.21
5.80
4.77
4,78
6.26
6.25
0.05
2.04
2.64
1.29
0.00
10.91
9.47
13.68
3.09
3.10
3,21
4.70
0.05
4.82
8.05
4,80
haffchoctal-mciiu---ppp
0.00
3.16
6.00
3.63
7.71
7.69
9.39
8.06
0.03
2.54
5.57
0.86
hopper-medinee
0.09
0.11
3.74
3.22
hopper medium replay
0.00
0.32
0.19
0.46
2.37
2.37
2.59
2.57
0.09
0.24
0.10
0.12
0.00
0.08
0.11
0.08
3,36
3.35
3.84
3.48
0.08
0.13
0.10
0.08
walker2d medium
0.00
0.84
1.07
1.12
3.39
3.40
3.61
3,46
4.27
4.34
5.87
4.24
walker2d meriium -eppla
0.00
3.49
2.53
3.64
2.47
2.46
2.74
2.65
7.92
8.73
8.90
6.33
walker2d-nn Miikerrd-mdduu--pppee
0.00
0.62
0.84
0.77
4.29
4.27
4.68
5.07
4.35
interaction between the policy and dynamics models enables degree of generalization beyond the training data.
Training and Generation Time Comparison for TGCVG and GTA. We compare the training time of our TGCVG and GTA, the Diffusion-basse method guided by scaled returns. All exper- iments are conducted on single NVIDIA RTX TITAN GPU us- ing the halfcheetall-eediinnv dataset, with 2 10 training steps and 5 x 10th synthesized data points. As shown in Figure 4 our TGCVG enables parallel training of the dynamics model and the policy generation model, leading to significantly reduced training time compared the diffusion process in GTA. Moreover, our TGCVG achieves much faster data generation by avoiding the time-consuming partial noising and denoising stages required by GTA. These results demonstrate the substantial reduction in time cost achieved by our approach.
Dynamic Mudii Training Aug Rodal Traiting Goneration
1200
Time Overhead (min)
GTA
TGCVG
Figure 4: Compare the time over- head of our TGCVG and GTA.
CONCLUSION
We propose Trajectory Generation with Conservative Value Guidance (TGCVG), simple yet effec- tive data augmentation framework for offline RI. The Q-learning module encourages above-average actions compared to the dataset, while value-based penalization constrains generated samples within the data distribution, mitigating OOD errors. Experiments on standard offline benchmarks show that our TGCVG not only outperforms state-of-the-an baselines but also significantly reduces compu- tational overhead. These results demonstrate the potential of leveraging the interaction between an offline policy and learned dynamics model to synthesize high-quality data, mimicking the data collection process in online RI.

---

Published as a conference paper at ICLR 2026
7
REPRODUCIBILITY STATEMENT
We have made every effort to ensure the reproducibility of our work. A detailed description of the proposed method provided in Section and all hyperparameter settings are included in Appendix
A.1 The datasets used in our experiments are publicly available through the D4RL benchmark (Fu
et al. 2020).
ACKNOWLEDGMENTS
This work is supported in part by the Guangxi Key Research and Development Program under Grant FN2504240005; in part by the National Natural Science Foundation China under Grant 62471064; in part by the Fundamental Research Funds for the Beijing University of Posts and Telecommunications under Grant 2025A14S02.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint vrXiv:2211.15657. 2022.
Richard Bellman. markovian decision process. Journal of mathematics and mechanics, pp. 679- 684, 1957.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-condition supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, 35:1542-111 1553, 2022.
Xinye Cao, Hongcan Guo, Jiawen Qian, Guoshun Nan, Chao Wang, Yuqi Pan, Tianhao Hou, Xiao- juan Wang, and Yutong Gao. Videominer: Iteratively grounding key frames of hour-long videos via tree-based group relative policy optimization. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 23773-27722 23783, 2025.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Tianyu Chen, Zhendong Wang, and Mingyuan Zhou. Diffusion policies creating a trust region for offline reinforcement learning. arXiv preprint 2rXiv:2405. 19690, 2024.
Zhuoran Duan, Guoshun Nan, Rushan Li, Zijun Wang, Lihua Xiong, Chaoying Yuan, Guorong Liu, Hui Xu, Qimei Cui, Xiaofeng Tao, al. Agile orchestration will: An entire smart service-based security architecture towards 6g. JEEE Wireless Communications, 2025.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint arXiv:2004.07219 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. International conference on machine learning, pp. 2052-2062. PMLR, 2019.
Chen-Xiao Gao, Chenyang Wu, Mingjun Cao, Rui Kong, Zongzhang Zhang, and Yang Yu. Act:
Empowering decision transformer with dynamic programming via advantage conditioning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 12127-12135. 2024.
Chen-Xiao Gao, Chenyang Wu, Mingjun Cao, Chenjun Xiao, Yang Yu, and Zongzhang Zhang. Behuaior-regularized diffusion policy optimization for offline reinforcement learning. arXiv preprint arXiv:2502.04777 2025.
10

---

Published as a conference paper at ICLR 2026
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International confer- on machine learning, pp. 1861-1870. Pmir, 2018.
Danijar Hafner, Timothy Lillicrap. Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193, 2020.
Philippe Mlansen-Estrude Ilya Kostrikov, Michael Janner, Jakub Grudzien Kuba, and Sergey Levine. Idql: Implicit q-learning as actor-critic method with diffusion policies. arXiv preprint arXiv:2304. 10573, 2023.
Haoran He, Chenjia Bai, Kang Xu, Zhuoran Yang, Weinan Zhang, Dong Wang. Bin Zhao, and Xue- long Li. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning. Advances neural information processing systems, 1194896-649177 2023.
Shengchao Hu, Ziqing Fan, Chaoqin Huang. Li Shen, Ya Zhang, Yanfeng Wang, and Dacheng Tao. Q-value regularized transformer for offline reinforcement learning. arXiv preprint arXiv:2405.17098 2024.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999 2022.
Jihwan Jeong, Xiaoyu Wang, Michael Gimelfarb, Hyunwoo Kim, Baher Abdulhai, and Scott San- ner. Conservative bayesian model-based value expansion for offline policy optimization. arXiv preprint arXiv:2210.03800 2022.
Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making. arXiv preprint arXiv:2310.03022 2023.
Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Adaptive q-aid for conditional supervised learning in offline reinforcement learning. Advances in Neural Information Processing Systems, 37:87104 10-0001351,,111 2024.
Diederik P Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Ilya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:2110.06169, 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 13:1179-1191 2020.
Misha Laskin, Kimin Lee, Adam Stooke, Lerrel Pinto, Pieter Abbeel, and Aravind Srinivas. Rein- forcement learning with augmented data. Advances neural information processing systems, 33: 19884-19895, 2020.
Jaewoo Lee, Sujin Yun, Taeyoung Yun, and Jinkyoo Park. Gta: Generative trajectory augmentation with guidance for offline reinforcement learning. arXiv preprint arXiv:2405. 16907, 2024.
Guanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long, and Weinan Zhang. Diffstitch: Boost- ing offline reinforcement learning with diffusion-hasee trajectory stitching. arXiv preprint arXiv:2402.02439, 2024.
Haoxin Lin. Yu- Yan Xu, Yihao Sun, Zhilong Zhang, Yi-Chen Li, Chengxing Jia, Junyin Ye, Jiaji Zhang, and Yang Yu. Any-step dynamics model improves future predictions for online and offline reinforcement learning. arXiv preprint arXiv:2405. 17031, 2024.

---

Published as a conference paper at ICLR 2026
Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. Ad- vances in Neural Information Processing Systems, 36:46323-46344 2023.
Jiafei Lyu, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711-1724, 2022.
Marlos C Machado, Andre Barreto, Doina Precup, and Michael Bowling. Temporal abstraction in reinforcement learning with the successor representation. Journal of machine learning research, 24(80):1-69, 2023.
Chenyang Qiu, Guoshun Nan, Caiyi Zhang, Chenrui Liang, Ruiqi Dai, Hongchen Yang, Shiyuan Liu, Zhili Zhou, Changhua Pei, and Xiaofeng Tao. Mining multi-scale spatial-frequency clues for unsupervised intrusion detection. IEEE Transactions on Information Forensics and Security, 2025.
Samarth Sinha, Ajay Mandlekar, and Animesh Garg. S4r1: Surprisingly simple self-supervision for offline reinforcement learning in robotics. Conference on Robot Learning, pp. 907-917. PMLR. 2022.
Yihao Sun, Jiaji Zhang, Chengxing Jia, Haoxin Lin, Junyin Ye, and Yang Yu. Model-bellman inconsistency for model-based offline reinforcement learning. International Conference on Machine Learming. pp. 33177-33194, PMLR, 2023.
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. Corl: Research-oriented deep offline reinforcement learning library. Advances in Neural Infor- mation Processing Systems, 36:30997-31020, 2023.
Qing Wang, Jiechao Xiong, Lei Han, Han Liu, Tong Zhang, et al. Exponentially weighted imitation learning for batched historical data. Advances in Neural Information Processing Systems, 31, 2018.
Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu, and Yu Qiao. Critic-guided decision transformer for offline reinforcement learning. Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15706-15714, 2024.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint XXXi:22008006993. 2022.
Ziyu Wang, Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-77777 2020.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning.
arXiv preprint arXiv: / 1361, 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint arXiv:2105.08/ 2021.
Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In Inter- national Conference on Machine Learning, pp. 38989-39007. PMLR, 2023.
Qianlan Yang and Yu-Xiong Wang. Rtdiff: Reverse trajectory synthesis via diffusion for offline reinforcement learning. In The Thirteenth International Conference on Learning Representations, 2025.
Denis Yarats, Ilya Kostrikov, and Rob Fergus. Image augmentation is all you need: Regularizing deep reinforcement learning from pixels. In International conference on learning representations, 2021.
12

---

Published as a conference paper at ICLR 2026
Tianhe Yu, Garrett Thomas, Lantao Yu, Stefano Ermon, James Y Zou, Sergey Levine, Chelsea Finn, and Tengyu Ma. Mopo: Model-based offline policy optimization. Advances in Neural Information Processing Systems, 15:14129-144422 2020.
Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offfine model-based policy optimization. Advances in neural information processing systems, 34:28954 2296677, 2021.
13

---

Published as a conference paper at ICLR 2026
A EXPERIMENTAL DETAILS
A.1 DETAILS OF TGCVG
We implement TGCVG based on the official GTA codebase (https://githin) com/ Jaewoopudd ng/ GTA), replacing the original Diffusion module with our Transformer-based synthesizer. For policy network training, most hyperparameters of the Transformer backbone follow those used in DT (Chen et al. (202T) as detailed in Table For the context length I. on the Gym domain, set L 5 for halfcheetah-meddiee remaining tasks, following (Hu et al. 2024). On the Maze2D and AntMaze domains, we consis- tently set 5. Our conservative Q-value guidance is adapted from CQL (Kumar et al.) 2020). with modifications to the tradeoff coefficient to balance conservatism and exploration. During the data augmentation phase, we adopt a similar pipeline to GTA, and dynamics model is im- ADMPC),
and malfcheetal-mmediun-rrpeeee
and 20 for the
plemented using the official ADMPO codebase (https: : //ggt htg.tttb.bb...... com/ The sample length K chosen based on the complexity of each environment. The generated dataset includes 5 million transitions for each task. Complete hyperparameter settings for all evaluated tasks are summarized in T 6
Table 5: Hyperparameters of TGCVG in our experiments.
Parameter
Value
Number of layers
3
Number attention heads
Embedding dimension
256
Nonlincarity function
GeLU
Batch size
256
Dropout
0.1
Optimizer
Adam KKngmaa 2014
Learning rate
le-4
Table 6: Hyperparameter settings of all datasets.
Datasets
K
halfchectah-mediim-v2
0.1
60
malichhetah-mediiu--rpphee
0.1
60
10.0
60
hopper-medium-y
2.0
60
hopper-mecium--rppaa--22
2.0
40
hopper-meddium-expprre22
5.0
60
walker2d4-medium-222
5.0
40
waller22-mmediuu--cllaa--a
5.0
40
waller22dmmccium--pperyyy
10.0
40
maze2d-umazee-v
0.5
maze2d-medium-vi
0.3
7
maze2d-large-vl
0.4
7
antmaze-umaze-v2
5
6
antmarze-mediun-play-222
7
antmaze-large-play-22
1
7
untmazz-umazz--ivvrreeww
7
antnaze-medium-divrse-w-
7
antmazelarge-diverre--Z
2
7
14

---

Published as a conference paper at ICLR 2026
A.2 OFFLINE RL ALGORITHM
We utilize the Clean Offline Reinforcement Learning (CORL) library (Taasovv et al. 2023) as the implementation base for all offline RL algorithms, available at (https://gi hub. com/ tinkoff-at ai/COR)). For the Gym domain, we adopt the default hyperparameters for CQL, IQL., TD3+BC, and DT. For the Maze2D and AntMaze tasks, IQL hyperparameters are configured ac- cording to the original paper.
A.3 DATASET AND EVALUATION
We use the v2 datasets for the Gym locomotion and AntMaze tasks, and the v1 datasets for Maze2D, consistent with GTA. For all tasks evaluated, we train TD3+BC, IQL, and CQL for million steps and report performance based on final evaluations. For DT, we train for 200,000 steps due to its faster convergence. We report final evaluation scores averaged over 10 episodes for Gym locomotion tasks and 100 episodes for Maze2D and AntMaze tasks. All scores are normalized following the D4RL protocol (Fu al. 2020), where score of 0 corresponds random policy and 100 corresponds to an expert policy.
B ADDITIONAL EXPERIMENTS
B.1 ABLATION STUDY RETURN CONDITION
Transformer- based methods offline RL typically formulate the task sequence modeling prob- lem, using the RTG condition generate high-return trajectories. Recent approaches such as QT (Hu et al. 2024) incorporate Q-learning into the Transformer architecture and have shown effec- tive policy improvement. However, we investigate whether the RTG input remains necessary when the Transformer is already equipped with a Q-lcarning module. To this end, we conduct an ablation study RTG using the official QT codebase (https: / thub. com/ charl eshsc/QT). As shown in Table 7 removing RTG has little impact on performance, suggesting that the Q-learning component is the primary driver of policy improvement. Based this observation, we design Transformer-CQU model to take only state sequences as input, which reduces input tokens and im- proves computational efficiency.
Table 7: Ablation on RTG conditioning in QT. Average and standard deviation scores are reported over seeds. The best average values marked in bold.
Dataset
QT w/o RTG
QT w/ RTG
Hopper-medium
73.29+1.115 99.24
77.28 100.99:0.51
hopper- mediu---ediu
B.2 ABLATION STUDY ON SAMPLE LENGTHI
We investigate the effect of sample length K in the data synthesis process. As shown in Table g increasing K improves trajectory coverage and allows the policy model explore longer, higher- return behaviors, particularly in stable environments such as halfcheetah. However, larger K also increases the likelihood of sampling trajectories that contain terminal states. Since our method in- herits terminal flags from the original trajectories, this can result in inconsistencies: the generated portion may not actually reach terminal condition, but the trajectory still forcibly terminated. Such mismatches between true and inherited termination become more pronounced as K increases, especially in environments with well-defined terminal conditions like hopper. Conversely, smaller values of K reduce the chance of terminal mismatch but may lead to insufficient trajectory explo- ration, as the generated segment shorter and less likely to achieve high returns. These results highlight tradeoff in choosing the appropriate sample length K during synthesis.
15

---

Published as a conference paper at ICLR 2026
Table 8: Ablation the number of sampled steps K. The final scores trained with TD3BC. Average and standard deviation scores are reported over 5 seeds. The best average values are marked in bold.
Dataset
K 25
K 40
K 60
K = 80
K 100
halfcheetablmmcdum
65.691:0.60
66.9910.72 93.9414.60
67.42++2.11 68.14 90.47:4.00
69.21 87.24:44.66
hopper-medium
91.06:t4.07
86.5011.02
B.3 ABLATION STUDY ON ORIGINAL DATA SIZE
To evaluate the robustness of our method in low-data regimes, we conduct experiments using only 1% and 10% of the original trajectories from the D4RL datasets. As shown in Table our method demonstrates exceptional data efficiency. Notably, on halfcheetah-mediim., TGCVG achieves a score of 49.90 using only 1% of the data, surprisingly surpassing the original TD3BC trained on the full dataset (100%) (48.42). Furthermore, our method exhibits superior scaling capabilities com- pared to the strongest augmentation baseline, GTA. While both methods perform comparably hopper-medium at 1% data, TGCVG achieves a massive performance jump at 10% data (reaching 86.58), whereas GTA stagnates (58.16) This highlights the effectiveness of generative augmen- tation in extracting and amplifying useful patterns even from extremely limited data.
Table 9: Ablation study on the size of the original dataset. We compare our method against the strongest augmentation baseline (GTA) and the original TD3BC algorithm (No Aug). The final scores are trained with TD3BC. Average and standard deviation scores are reported over seeds. The best average values are marked in bold.
Datasct
Method
1% Data
10% Data
100% Data (Full)
Original (No Aug)
9.26 = 7.90
48.38 = 0.21
48.42 + 0.62
halfcheetah-medium
GTA
46.24 + 0.11
53.21 0.23
57.84 0.51
TGCVG (Ours)
49,90 + 0.23
57.00 + 0.30
68.14 0.58
Original (No Aug)
35.57 + 2.28
54.98 + 3.94
61.04 3.18
hopper-medium
GTA
56,56 = 3.64
58.16 = 7.97
69.57 + 4.05
TGCVG (Ours)
54.04 + 12.13
86.58 + 5.13
90.47 4.06
B.4 ABLATION STUDY ON SYNTHETIC DATA SIZE
We further investigate the impact of the volume of generated synthetic data on downstream per- formance. We vary the synthetic dataset size from 0.01M to 5M transitions. Table 10 presents the comparison between our TGCVG and the GTA baseline. The results indicate that our method is highly data-efficient. On haffcheetah-mmddimm TGCVG trained with only 0.IM synthetic data (63.08) already outperforms GTA trained with 5M data (57.84). Furthermore, our method exhibits superior scalability: while GTA's performance tends to plateau increase slowly, TGCVG shows a continuous and substantial performance gain as the synthetic data size increases, particularly on hopper-medium, where it reaches a score of 90.47 with 5M samples.
B.5 PERFORMANCE COMPARISON BETWERN TGCVG AND NODEL-BASED METHODS
Model-based methods also generate data, though their data generation is typically not decoupled from the learning process, unlike data augmentation approaches. We are interested in whether data generated by model- methods can be repurposed to train offline model-free algorithms for per- formance improvement. Since our dynamics model is adapted from ADMPO (Lin et al.. 2024), we compare our method with ADMPO to investigate this question. Table I1 compares ADMPO+IQL. which reuses data from ADMPO training for IQL, with the original ADMPO scores and our TGCVG results. The results suggest that data generated by model- based methods is not well-suited for direct use in offline model- free training, demonstrating that TGCVG provides a more general and effective data generation strategy for offline RI.
16

---

Published as a conference paper at ICLR 2026
Table 10: Ablation study on the size of the synthetic dataset (generated transitions). We compare the scalability of our method versus GTA. The final scores are trained with TD3BC. Average and standard deviation scores are reported over 5 seeds. The best average values are marked in bold.
Dataset
Method
0.01M
0.1M
IM
5M (Default)
GTA
49.46 0.28
54.03
0.45
$7.62 0.37
57.84 1 0.51 68.14 + 0.58
halfcheetah-medium
TGCVG
51.21 + 1.03
63.08 1,85
67.92 + 0.64
GTA
57.00 + 1.64
56.36 3.13
63.91 2.88
69.57 + 4.05
Hopper-medium
TGCVG
59.00 + 3.02
60.53 4.47
66.08 + 5.38
90.47 + 4.06
Table 11: Normalized results on halfcheetah-medium and antmace-large-divere Average and stan- dard deviation scores are reported over 5 seeds. The best average values are marked in bold.
Algo.
halfcheetah-mediimg lattmaze-large-iivrre
ADMPO+IQL
21.90113.91
0.00
ADMPO
72.20=00.0
0.00=00000
TGCVG+IQL
68.61
47.20:14.26
B.6 FURTHER STUDY ON COMPOUNDING ERROR
Compounding eπrоrs remain a major challenge for model-based methods in offline RL. To assess this issue, we follow the evaluation protocol used in ADMPO (Lin et al. 2024) and plot the pre- diction error curves over rollout steps. We compare TGCVG and GTA using generated trajectories with rollout lengths corresponding to those used in ADMPO. For fair comparison, we evaluate prediction errors over the shared rollout horizon, defined as the minimum trajectory length between the two methods. As shown in Figure []] the prediction errors of TGCVG and GTA remain close and increase slowly with rollout length, indicating that the dynamic plausibility of the generated sequences is well preserved.
B.7 FURTHER STUDY ON DATA QUALITY
We further analyze how data quality evolves with increasing rollout length. As shown in Figure 6 both novelty and optimality metrics improve as the rollout horizon increases. This suggests that the policy network becomes less dependent on the original dataset and gradually shifts toward gener- ating more diverse and higher-reward samples. Meanwhile, the dynamic MSE remains relatively stable across different rollout lengths, indicating that the interaction between the policy and dynam- ies models consistently produces transitions that are dynamically plausible.
C HYPERPARAMETER TUNING GUIDELINES
We provide principled guideline for selecting the conservatism weight À and the generation length K. To address concerns regarding the search space, we propose a systematic "Initialization + Local Fine-tuning" workflow that leverages established community knowledge minimize tning over- head.
Conservatism Weight (A): A Systematic Workflow. Instead of global search, selection of À inherits heuristics from the backbone algorithm (CQL) and aligns with the "BC + Q-value Regularization" paradigm (e.g.. TD3BC (Fujimoto & Gu 202T). We recommend the following two-step process:
1. Initializatonn (Anchor): Since our method builds upon the CQL backbone, practitioners can initialize À using standard settings recommended in the original CQL literature (c.g.,
X R 5.0 10.0 starting baseline).
17

---

Published as a conference paper at ICLR 2026
GTA
TGCVG
hopper-medium
walker2d-medium
75
Prediction Error
Prediction Error
5
50
25
0
-25
5
10
15
20
0
5
10
15
20
Rollout Length
Rollout Length
hopper-medium-replay
walkerzdd-eddium-relly
Prediction Error
Prediction Error
5
100
50
0
-5
15
20
0
5
10
15
20
Rollout Length
Rollout Length
Comparison among TGCVG and GTA, in terms of the growth curve of the compounding error rollout length increases, after data generation.
Figure 5:
halfcheetah-medlum
hopper-medlum
walker2d-medlum
helfcheetah--medlum replay halfcheetah- -eppert
hopper-medlum-replly hopper-me eeem--eeet
walker2d-medium-reell
milker/d-meddum-expert
Optimaltty
25 30 Lengtth
50 35
30 Length
24
Lengtth
Figure 6: Data quality analysis of TGCVG rollout length increases.
2. A djustment the return distribution of the dataset, following inverse logic to TD3BC:
Tuning):
We then refine this anchor based on
For datasets concentrated around a single return mode (e.g., medium or medium-replan repplam,-ee we adjust À to be lower. A reduced conservative penalty allows the policy leverage Q-value guidance for extrapolation, improving upon the average behavior.
Unimodal Concentrated Distributions:
Multimodal Distributions: For datasets with conflicting modes (e.g.. medium- expert), adjust À to be higher. Stronger conservatism is necessary to restrict the
18

---

Published as a conference paper at ICLR 2026
policy high-density regions, preventing invalid transitions that interpolate between distinct behavioral modes.
3. Local Fine-tuning: cal neighborhood of the adjusted anchor, significantly reducing the search space compared to a blind sweep.
Finally, practitioners only need to perform a minimal search in the
Generation Length (K): Termination-Aware Selection. The choice of K is deterministic based on the strictness of the environment's termination conditions:
Lenient Termination: environments where early termination rare (e.g., halfcheetah), longer K is preferred. This enables the policy to explore more extensively and stitch together longer, higher-quality trajectories.
Striet Termination: In environments with strict termination conditions or sparse rewards (e.g., animaaz)) shorter K is crucial. This prevents "terminal mismatch", scenario where high-quality generated trajectory might trigger premature termination in the simulator that is inconsistent with the original long-horizon plan.
D LIMITATIONS
Our work proposes novel data augmentation framework for offline RL that improves the quality of generated trajectories while reducing computational overhead. However, since our method builds upon the implementation of CQL, it inherits certain limitations of CQL itself. In particular, tasks that are inherently challenging for CQL remain difficult for our Transformer- CQL variant, even though policy model is more robust and expressive. When the policy fails to produce high-quality actions, the benefit of incorporating generated data into offline RL. training is limited.
E FUTURE WORK
Several promising directions remain for future exploration. First, iterative grounding mechanisms inspired by tree-based key frame extraction (Cao et al. 2025) could enable selective augmentation
of high-value trajectory segments. Second, unsupervised anomaly detection (Qiu et al. 2025) may filter low-quality synthetic trajectories. Finally, service-oriented architectures (Duan al. 2025 would facilitate flexible deployment across diverse robotic platforms.
F THE USE OF LARGE LANGUAGE MODELS (LLMs)
In accordance with the ICLR policy on the use of large language models (LLMs), we disclose that LLMs are employed during the preparation of this paper. Their use is limited to assisting with language editing, improving readability, and polishing the presentation of the manuscript. LLMs are not involved formulating the research questions, designing the methodology, conducting ex- periments, analyzing results, or drawing scientific conclusions. We take full responsibility for all technical content, claims, and conclusions presented in this work.
19


Reviews:
[{"review_id": "7CpfEbI62r", "reviewer": "Reviewer_mtk1", "summary": "The paper proposes TGCVG, a framework that combines CQL transformer with a dynamics model to generate high-quality, in-distribution trajectories for offline reinforcement learning.\nBy guiding trajectory synthesis with conservative value estimates, TGCVG improves both stability and sample efficiency.", "questions": "1. How is the dynamics model trained in this work? Please clarify the training objective, data source, and other details.\n\n2. How does the model handle terminal indicators when generating or evaluating trajectories? This part is not clearly presented in the main paper.\n\n3. In Table 2, why does SynthER perform worse than the version without augmentation? \n\n4. Can the authors provide more explanation on why restricting trajectories to be in-distribution leads to better performance, especially when the dataset contains medium-expert level data because in-distribution seems to bring similar information with the original dataset.\n\n5. Have the authors tested TGCVG on visual RL environments to examine whether the proposed framework generalizes beyond state-based tasks?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses an important problem in offline reinforcement learning—how to generate high-quality trajectories that remain in-distribution to improve policy learning stability.\n\n2. The proposed TGCVG framework is both novel and solid, effectively leveraging conservative Q-learning to guide trajectory generation and ensure distributional consistency.\n\n3. The experimental results are strong and impressive, showing clear and consistent improvements across multiple D4RL benchmarks.\n\n4. The paper is well-written and clearly presented, making the motivation, methodology, and findings easy to follow.", "weaknesses": "1. The experimental section could be more comprehensive. In particular, an ablation study comparing conservative trajectory generation with standard generation would help clarify the effectiveness of the proposed mechanism.\n\nSee quetions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PkAnc6eu9u", "reviewer": "Reviewer_kHAm", "summary": "This paper proposes TGCVG (Trajectory Generation with Conservative Value Guidance), a novel data augmentation framework for offline reinforcement learning (Offline RL). TGCVG consists of transformer based policy network trained using Conservative Q-learning(CQL), learned dynamics model that predicts next states and rewards to generate synthetic transitions. TGCVG guides generation process with conservative value guidance, keeping them close to data distribution while favoring high-value regions. The augmented trajectories are then mixed with the original dataset and used to train standard offline RL. Extensive experiments on D4RL benchmarks (MuJoCo, Maze2D, AntMaze) show that TGCVG improves baseline performance across domains. And produces high-quality, dynamically consistent synthetic data according to novelty, optimality, and dynamic MSE metrics.\n- An LLM was used to improve writing.", "questions": "Could the authors explicitly define the metrics used for novelty, optimality, and dynamic MSE?\nFor example, novelty seems to be measured via L2 distance, but is this metric appropriate given the domain characteristics of the datasets? A short justification or alternative metric discussion would strengthen the paper.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Low computational cost: The proposed method achieves significantly lower training and data-generation overhead compared to diffusion-based approaches such as GTA — up to 10× faster in both stages.\n\n2. Strong empirical performance: Comprehensive evaluation on D4RL benchmark tasks demonstrates consistent and robust performance gains against a wide range of baselines.\n\n3. Clarity and presentation: The paper is well-written and well-organized, with clear motivation, sound theoretical reasoning, and a coherent algorithmic presentation that makes the method easy to follow.", "weaknesses": "1. Lack of evaluation on high-dimensional or robotics domains: The paper does not include results on more complex D4RL tasks such as Adroit or Kitchen, which are higher-dimensional and closer to real-world robotics scenarios.\n\n2. Ablation on value guidance and policy choice: It would be valuable to see how performance changes with or without the conservative value guidance, or when using Decision Transformer (DT) as the policy for augmentation.\n\n3. Missing experimental details: In Table 3, results for HalfCheetah are not presented, and experiments where Transformer-CQL itself serves as the policy appear insufficiently explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XrXHWCf45V", "reviewer": "Reviewer_KvRm", "summary": "The paper proposes a novel data augmentation method for offline RL, TGCVG, using the transformer-based augmentation policy with the dynamics model. In contrast to diffusion-based augmentation methods such as GTA [1], TGCVG incurs lower overhead during both training and dataset augmentation. Using the TGCVG-augmented dataset, offline RL policies achieved better performance than with any other augmentation baseline.\n\n[1] Lee, Jaewoo, et al. \"Gta: Generative trajectory augmentation with guidance for offline reinforcement learning.\" Advances in Neural Information Processing Systems 37 (2024): 56766-56801.", "questions": "Because the hyperparameters are the pair $(\\lambda, K)$, the search space is combinatorial. To make offline RL algorithms viable in the real world, where online interaction is unavailable, we must be able to select good hyperparameter combinations purely offline; hence, simplicity in offline RL methods is crucial [4]. I therefore argue that a thorough practical guideline for joint tuning of $(\\lambda, K)$ is essential for this paper.\n\n[4] Fujimoto, Scott, and Shixiang Shane Gu. \"A minimalist approach to offline reinforcement learning.\" Advances in neural information processing systems 34 (2021): 20132-20145.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Well motivated\n\nThe paper is well motivated and clearly constructed. Without increasing the complexity of the policy algorithm, TGCVG improves the performance of offline RL policies by generating high-quality data guided by the conservative Q-function.\n\n2. Improved efficiency for training and augmentation\n\nBecause the augmentation process is fully offline, its impact on decision-time efficiency is limited; nevertheless, Figure 4 indicates that TGCVG is 4× more efficient for training and augmenting the offline dataset.\n\n3. Fresh insight about the conservatism quality of TD3BC and CQL\n\nThe authors show that Transformer-CQL generates in-distribution samples, whereas Transformer-TD3BC does not; consequently, the CQL policy cannot learn a conservative policy from Transformer-TD3BC’s augmented data.", "weaknesses": "1. Concerns about the hyperparameter sensitivity\n\nWhile the authors conduct an ablation study on $\\lambda$, they vary its value from 0.1 to 1, suggesting that TGCVC requires extensive hyperparameter tuning. In addition, Table 6 lists another hyperparameter, $K$, but lacks an ablation, further increasing the tuning burden for practitioners.\n\n\n2. Results on larger-scale benchmarks would be beneficial\n\nRecently, the scalability of offline RL has emerged as an important theme [2]. OGBench [3] provides a diverse suite of offline tasks with higher-dimensional state spaces and larger datasets. I understand the rebuttal window limits time and compute, but if TCGVC were corroborated by strong results on OGBench, its impact and practical value would be substantially enhanced.\n\n[2] Park, Seohong, et al. \"Horizon Reduction Makes RL Scalable.\" arXiv preprint arXiv:2506.04168 (2025).\n\n[3] Park, Seohong, et al. \"OGBench: Benchmarking Offline Goal-Conditioned RL.\" The Thirteenth International Conference on Learning Representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8pe5LmdfoX", "reviewer": "Reviewer_Cmj7", "summary": "The paper proposes TGCVG, a novel generative data augmentation method designed for offline RL algorithms. TGCVG tackles two limitations of prior works: computational overhead and distribution shift. TGCVG trains Transformer instead of Diffusion models and applies conservative value guidance to mitigate those issues. Experiment results show TGCVG outperforms prior baselines.", "questions": "- It would be better to specify the number of generated transitions for each round, the ratio of transitions from the original offline dataset and the generated dataset during policy training, and the update-to-data (UTD) ratio for a clearer understanding.\n\n- It would be better to visualize t-SNE visualization of generated data distributions on diverse tasks for more comprehensive understanding of the behavior.\n\n- At first, I understood the conservative value guidance as using conservative RTG values for generating trajectories with the Transformer policy. However, it seems that we train Transformer-CQL and directly use the network for generating an augmented dataset. Is there any reason why the current method is preferred over the aforementioned method? I'm not asking about the experiment, just curious. I think we can achieve similar results when we carefully tune the hyperparameter for choosing a conservative RTG value.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea is easy to follow\n- Limitation of prior works and How to mitigate the limitations are clearly stated\n- Strong empirical results against several baselines across diverse tasks", "weaknesses": "- It seems that $\\lambda$ and $K$ should be heavily tuned across different environments and datasets. I'm not sure the method can be generalized to unseen tasks without extensive hyperparameter tuning, which is crucial in offline RL.\n\n- It would be better to add ablation studies on different sizes of the dataset, which is crucial for evaluating the performance of generative data augmentation.\n\n- It would be better to add ablation studies on different sizes of the generated dataset, which is also crucial for evaluating the performance of generative data augmentation. I'm also struggling to find how many transitions are augmented during the training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)