Value-Guided Decision Transformer: A Unified Reinforcement Learning Framework for Online and Offline Settings
Authors: Hongling Zheng
Li Shen
Yong Luo
Deheng Ye
Shuhan Xu
Bo Du
Jialie Shen
Dacheng Tao
Venue: NeurIPS 2025 poster ()
Keywords: Offline RL
Conditional Sequence Modeling
Decision Transformer
Primary area: reinforcement_learning
TL;DR: In this paper, we propose Value-Guided Decision Transformer (VDT), which employs progressively optimized value functions  to guide the Decision Transformer (DT) in making optimal decisions.

Abstract: The Conditional Sequence Modeling (CSM) paradigm, benefiting from the transformer's powerful distribution modeling capabilities, has demonstrated considerable promise in Reinforcement Learning (RL) tasks. However, much of the work has focused on applying CSM to single online or offline settings, with the general architecture rarely explored. Additionally, existing methods primarily focus on deterministic trajectory modeling, overlooking the randomness of state transitions and the diversity of future trajectory distributions. Fortunately, value-based methods offer a viable solution for CSM, further bridging the potential gap between offline and online RL. In this paper, we propose Value-Guided Decision Transformer (VDT), which leverages value functions to perform advantage-weighting and behavior regularization on the Decision Transformer (DT), guiding the policy toward upper-bound optimal decisions during the offline training phase. In the online tuning phase, VDT further integrates value-based policy improvement with behavior cloning under the CSM architecture through limited interaction and data collection, achieving performance improvement within minimal timesteps. The predictive capability of value functions for future returns is also incorporated into the sampling process. Our method achieves competitive performance on various standard RL benchmarks, providing a feasible solution for developing CSM architectures in general scenarios. Code is available at here.

Value-Guided Decision Transformer: A Unified Reinforcement Learning Framework for Online and Offline Settings
Hongling Zheng'
Li Shen
Yong Luo Deheng Ye
Shuhan Xu²
Bo Du
Jialie Shen Dacheng Tao
School of Computer Science, National Engineering Research Center of Multimedia Software and Hubei Key Laboratory of Multimedia and Network Communication Engineering,
Wuhan University. Wuhan, China
Shenzhen Campus of Sun Yat-sen University, Shenzhen, China Teencent Inc., Shenzhen, China *City St George's, University of London, UK *Nanyang Technological University, Singapore {hlzheng, luoyong, xushuhan, dubo)e@hu.. edu. cn dericye@tencent. com (mathshenli, dacheng. @gmail com jerry. shen@citystgoorges ac. uk
Abstract
The Conditional Sequence Modeling (CSM) paradigm, benefiting from the trans- former's powerful distribution modeling capabilities, has demonstrated consider- able promise in Reinforcement Learning (RL) tasks. However, much of the work has focused applying CSM single online offline settings, with the general architecture rarely explored. Additionally, existing methods primarily focus on deterministic trajectory modeling, overlooking the randomness of state transitions and the diversity of future trajectory distributions. Fortunately, value-based meth- ods offer viable solution for CSM, further bridging the potential gap between offline and online RL. In this paper, we propose Value-Guides Decision Trans- former (VDT), which leverages value functions to perform advantage-weighhe and behavior regularization on the Decision Transformer (DT), guiding the policy toward upper-bound optimal decisions during the offline training phase. the online tuning phase, VDT further integrates value-based policy improvement with behavior cloning under the CSM architecture through limited interaction and data collection, achieving performance improvement within minimal timesteps. The predictive capability of value functions for future returns is also incorporated into the sampling process. Our method achieves competitive performance vari- ous standard RL benchmarks, providing a feasible solution for developing CSM architectures in general scenarios. Code is available at here
Introduction
Offline reinforcement learning (Offline RL) [T] aims to develop wwward-maximizing RL strategy using offline data. This approach is highly valuable in real-world scenarios where online data collection expensive, time-consuming, or impractical. Transformer 2 is widely regarded for its capacity to capture complex data distributions and long-term temporal dependencies, becoming a foundational architecture in fields such as Natural Language Processing 3 and Computer
Vision (5) (). Inspired by this success, Decision Transformer (DT) 7 and its variants [8] 9 introduce transformer to the field of offline RL, demonstrating its powerful capabilities Conditional Sequence Modeling (CSM) 10 Specifically, DT integrates cumulative rewards, states, and actions
"ooresppnniing authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

into a tuple and trains on offline datasets autoregressively to output appropriate actions. This approach relaxes the MDP assumption by considering multiple historical steps, allowing the model to handle long sequences and avoid stability issues associated with bootstrapping [11.
The CSM method is essentially goal-conditioned behavior cloning, which optimistically treats the highly incidental performance gains resulting from the randomness in offline data as a general expectation. As result, while performs well in deterministic environments, it struggles to achieve good performance in stochastic environments or when faced with suboptimal data. Some works [3] have combined value functions trained on offline data with CSM to generate suboptimal trajectory stitching, guiding the agent's 's robust learning. However, these approaches typically introduce the value function by simply re-labeling the retum-to-go (RTG) or directly using as penalty terms, with little consideration given to how to explore further the optimization upper bound of the value function in the CSM setting and effectively integrate with the DT to curb its overly optimistic behavior cloning.
Bridging the gap between offline and online reinforcement learning remains central challenge. One promising direction that has emerged in recent years is offline-to-online RL [14] 15]. Within the CSM framework, commonly used method is the Online Decision Transformer (ODT) [6], which extends DT training into the online phase while maintaining the same supervised learning paradigm used in offiine RL. However, ODT faces challenges in achieving expert-level performance in online scenarios with limited or suboptimal data, primarily due to its inability to compose or integrate suboptimal trajectories effectively. Furthermore, ODT demonstrates sigmificant advantages only after online fine-tuning, while its performance in offline scenarios remains suboptimal. There is still limited research on how the CSM architecture can generalize across offline and online settings.
To remedy these drawbacks, we propose the Value-Guided Decision Transformer (VDT), a unified RL framework for online and offline settings. (1) Offline Training Phase: Based on the DT training framework, we combine a multi-step Bellman-optimized Q-function and state-value function to explore the upper bound of value estimation. Value guidance is integrated with the behavior eloning of Decision Transformer (DT) through advantage- weentage-weighttd weighted learning, while the maximum estimated value is concurrently used as a penalty term to regularize the expected value of the current action distribution. The coupling and regularization effectively mitigate the overly optimistic estimates of CSM in stochastic environments and enable trajectory stitching from suboptimal data. (2) Online Tuning Phase: VDT refines the value function and the policy through limited interactions. Integrating the trajectory-level replay buffer and RTG alignment achieves significant performance improvements within minimal timesteps. (3) Sampling Process: The Q-function evaluates the expected future return of each action the policy generates under different RTGs within a predefined evaluation horizon and selects the optimal decision. The introduction of the value function significantly improves the DT's performance in both pure offline and offfiieeto-oninee settings, regardless of data quality or reward sparsity, and further bridges the potential gap between offline and online RL.
The main contributions of this work are as follows:
We incorporate the value function into the CSM architecture and enhance behavior cloning with advantage-weighted learning and regularization constraints. These components enable VDT to stitch together suboptimal trajectories under based guidance and achieve robust performance across varying- -ualittyyaayyy datasets. We further provide theoretical guarantee of its superior performance.
We leverage the inherent strengths of the value function to fine-tune the policy with a limited number of interactions in the online phase. By introducing the trajectory-level replay buffer and return-to-go alignment, we bridge the gap between offline training and online tuning, offering insights into the design of generalizable architectures.
We demonstrate the effectiveness of VDT across a broad spectrum of benchmarks, exhibiting superior performance in pure offline and offinnett-online settings.
2
Related works
CSM for Offline RL. In contrast to online RI, offline RL focuses on training models and performing trial-and-error using offline data without environmental interaction to arrive at appropriate strategies. Recently, CSM for RL [18 19, represented by the transformer architecture, has further
2

---

demonstrated the advantages of data data-driven policy learning. DT [7] is trained on an offline dataset of triplets encapsulating return-to-ge to-go f, state S;, and action ar. and outputs the optimal action. The & token quantifies the cumulative reward from the current time step to the end of the episode. During training, DT processes trajectory sequence Tr in an auto-regressive manner, which encompasses the most recent K Ststep historical context:
(fi-K +1. K+, 4t-k1
Pr. St,
(I)
The prediction head associated with a state token 8, is trained to predict the corresponding action ag Regarding continuous action spaces, the training objective is to minimize the mean-squared loss Lpr-
K Σ
1 K
LDT = ET
-a
no (Tt).)²
1=-- K+1
Subsequent work has made various improvements to DT, including prompt tuning 200,, trajectory concatenation [2I]. and value regularization [22]. These approaches often involve more complex
modifications of DT to adapt it to specific tasks.
Pifiineto-Onnlie RL. Offfine-to-online RL focuses fine-tuning a policy pre-trained on offline data using& a limited number of online interactions to improve performance and narrow the gap between offiine and online learning. Key challenges in this setting include effectively leveraging offline data for initializing the policy and mitigating distributional shifts during the online adaptation process. Early approaches focus on pre-training policies with offline data (23) 24] and use techniques such as balanced sampling (25), adaptive conservatism (15), and actor-critic alignment [26] to stabilize the transition the online phase. For efficient online fine-tuning, optimistic exploration strategies are employed, utilizing Qeeneembles [27], uncertainty-guiil
exploration (28). or model based
uncertainty estimation [29] 30). While most existing methods follow Q-learning paradigms, the
use of CSM architectures such as DT for offline-to-onling reinforcement learning remains largely underexplored.
Value-based Offline RL. The value-based method is one of the most prominent categories for addressing the distribution shift problem in offline RL. Primarily previous works generally address this problem in one of three ways: (I) constraining the learned policy to the behavior policy 223 31
(2) constraining the learned policy by making conservative estimates of future rewards 124 22
introducing model-based methods, which learn a model of the environment dynamics to generate more data for policy training and perform pessimistic planning in the learned MDP 32 33
The most relevant work to ours is ODT [16], the first to establish a pipeline for transitioning from offline to online RL within the DT framework. However, since ODT adopts strategy that mirrors its offline training phase, it struggles to effectively handle suboptimal data and adapt to dynamic stochastic environments. Moreover, tuning, ODT performs significantly worse than most baselines in offline settings. TD3+ODT (34), which is primarily based on the ODT algorithm pipeline, further incorporates the RL gradient from the critic as an additional penalty term in the loss function. Our method is distinguished by the effective optimization of the value function and its integration with the DT, enabling the policy to achieve robust performance across varying data quality in offline and online settings. In addition, the evaluation mechanism we design during the sampling process further encourages exploration, facilitating efficient trajectory stitching.
few-shot method primarily focused online fine-
3 Preliminary
Markov Decision Process. Reinforcement Learning is typically formulated as a Markov Decision Process (MDP), defined by tuple (S,A, P, 7), where S represents the state space, A is the action space, P is the transition function, r is the reward function, and y E [0, 1] is the discount factor.
In offline RL, the objective is to optimize the RL policy using previously collected dataset D {(s), ai, ri, st 1 consisting of N trajectories. The key distinction in offline RL is that the agent does not interact directly with the environment during the learning phase but instead learns from the historical data. This offline dataset represents the sate-action-reward-ee agent experienced during earlier episodes. The agent's 's task to extract useful information from these trajectories to improve its decision decision-making policy without further environmental exploration.
transitions the
3

---

Offline Training & Online Tuning
Sampling Process
Trajectories
Evallation Horizon E
Q
Tuning
Linear Predictor
Avarage Weightine & Regul arizatioon
Transformer Layer
OF
Transformer
Embedding Layer
QF
Update
RTG Alignment
Replay Buffer
Offiine Training& Online Tuninn Policy
Value-function
Q-function
Q-function
Learning
arg max
Figure 1: The architecture of Guided Decision Transformer, Left: Offline Training & Online Tuning. VDT is trained offline under value guidance and interacts with the environment online to generate trajectories for updating the replay buffer. These trajectories are then used to tune VDT further. Right: Sampling Process. At a specific timestep f, the policy generates candidate actions within the evaluation horizon E under predefined RTGs, and evaluates them using the Q-function to obtain the optimal action dz.
Online Decision Transformer. Online Decision Transformer (ODT) operates in two stages: offline pretraining and online finetuning. In the offline phase, the model learns from static trajectories using supervised learning, following the standard DT paradigm. In the online phase, it collects new trajectories and refines the policy with supervised updates. During rollouts, the action a, timestep l is either computed deterministically as a, (DD((T)), ), or sampled stochastically from the policy a (Dr Tr). The policy initialized using the offline dataset and is updated during finetuning as newly collected online trajectories gradually replace the old data buffer.
4 Method
This section provides comprehensive description of the proposed VDT. We break down VDT into three parts: offline training, online tuning, and the sampling process. The offline training details how the value function is integrated into the conventional CSM architecture using advantage-weighte learning and regularization terms. The online tuning focuses on the collection and processing of online data. Although the model has already incorporated the value guidance during training, due to the inherent randomness the environment, we perform value evaluation on the predicted actions during the sampling process to maximize decision-making performance.
4.1 Offline Training
We incorporate value guidance into the DT architecture to enable the CSM architecture to perform robustly in highly stochastic environments and with suboptimal trajectories, and to achieve effective trajectory stitching. As a first step, we learn the Q-function. Specifically, we aim to estimate an upper bound of the Q-function within the support of the dataset's action distribution. We implement the nentwwrrks Qo, (s, a) and Qo, a), along with their corresponding target networks Qú, (s, a) and Qa (s,a), a), as multi-layer perceptrons (MLPs) with two hidden layers. This architecture provides sufficient representational capacity to approximate complex value functions while maintaining stability during offline training. Inspired by Implicit Q-Learning (IQL) (24), we adopt an expectancy regression technique, which leverages an asymmetric L2 loss to provide an unbiased estimation of the conditional expectile. For a given threshold > 0.5, this loss function adaptively emphasizes high-return actions by down- wown-weighting the influence of actions with returns below the threshold, thereby approximating the conditional upper bound of the f-fucction. To support this, we use an independent state-valus function Vs(s), implemented as simple two-layer MLP, and optimize

---

using the following objective:
Ly (1b) (( [T -V (+)))
where L5(u)= = l < 0)/u². After obtaining the Vi with average properties, given that the input to the DT consists of historical trajectories, we choose the n-step Bellman equation to estimate the double Q-networks, which has been shown to provide better stability in practical training compared to single-step optimization [3]. To update Q0 E [1, we use the following formula:
E (on
Σ 7 Trts + 7"h ( Qo, (st,
(4)
where y is the discount factor. We modify the TD learning procedure learn an approximation to the optimal Q-function on an offline dataset. In designing the learning objective, we weigh the DT's trajectory modeling loss by computing the advantage function min;=1,2 Qù. (st, at) - Vi (s:).
which helps the model identify high-return action trajectories in the dataset and assign them higher importance. As proposed by [36]. this advantage-
method performs policy optimization within the action distribution supported by the dataset, stabilizing the learning process. Additionally, to further guide the policy towards high-value directions, we compute the policy's 's action output at the current state and evaluate its value using the target Q-networks, which is then incorporated into the loss function as a regularization term.
E TED (exp(n( (S..4))TT
(imp min Qú, (s), az) V (()))))))) a 1 X 3
min Qá, (St, (Dr (5) ==1,2
where 17 is an inverse temperature, which we set to 3. The parameter À is a hyperparameter that balances the weight between the two loss terms. Through this loss design, the combination of advantage weighting and the regularization term optimizes the stability of trajectory modeling while guiding the value function to break through the local optima within the data distribution.
Theorem 4.1. Let br be the optimal policy of Equation For any 8 E S. we have that Vabr (s) (s) and x (a s) = 0 given B(a s) =0.
Theorem 4.2. For any initial state distribution p. we have that V* (H) (=br(p) S
2v
(I-7)²
Ex maxxgAn(s) Q= (s, a) maxaesupp(8(1s)) Q" (s,a)).
Building on the theoretical foundations established by Theorem [.] and the new upper bound provided by Theorem 4.2] we further confirm the effectiveness of Equation[5] These theoretical insights suggest that a policy guided by the value function is likely to outperform the behavior policy. In particular, errantage-weighteee
and regularization play crucial role by prioritizing high-value actions, thus steering the learning process towards optimal returns and ensuring consistent improvement over the baseline behavior policy B. Complete proofs are provided in Appendix and Appendix B
4.2 Online Tuning
During the online tuning stage, while following almost the same pipeline as the offline training phase, we also introduce several distinct components achieve the most significant performance gains with the fewest interaction steps.
Trajectory-Leve Replay Buffer. In the online tuning phase, we employ a replay buffer similar to that used in ODT [6],, where the buffer consists of entire trajectories rather than individual transitions. The replay buffer is initially populated with the trajectories that yield the highest returns in the offline dataset. Each time the policy interacts with the environment, we fully roll out an episode using the current policy, then refresh the replay buffer by adding the collected trajectory in a first-in-firs-ouut manner. Afterwards, we update the policy and proceed with another rollout. We use the two-step sampling procedure ensure that the sub-trajectories of length K in the replay buffer T Treplay sampled uniformly. We first sample a single trajectory with probability proportional to its length, then uniformly sample sub- crajtrajcttory of length K. Our sampling strategy is akin to importance sampling for environments with non nen-negative, dense rewards. In those cases, the length of a trajectory is highly correlated with its return.
Return-to-go Alignment. During the offline training phase, the RTG at the current step is accumu- lated from subsequent rewards, and the policy learns conditioned this RTG. However, in the online
5

---

tuning phase, the policy interacts with the environment in real-time based on a predefined RTG, and the induced RTG may differ from the predefined RTG. This discrepancy leads mismatch between the expected and actual returns, further affecting the effectiveness of the value function in guiding the policy. To address this, we modify the RTG token at each step of the rolled-out trajectory using the achieved returns, such that the RTG token at time step ! is set as RTG, = Σ} "j where [r] denotes the trajectory length and ry denotes the reward at each step. Specifically, for the last timestep of trajectory, the RTG token equals the immediate reward obtained by the agent upon trajectory termination, which more accurately reflects the return at that step and helps to align the expected and actual returns. Note that, depending on the properties of the environment, the immediate reward at the final step is not necessarily zero.
4.3 Sampling Process
Traditional DT uses multiple RTGs for completely independent evaluations and typically selects the trajectory with the highest cumulative return under a specific RTG as the evaluation result, neglecting the potential guidance from alternative RTGs at each timestep. We incorporate mul- tiple RTGs at each trajectory step to overcome this limitation. The value function evaluates the candidate actions, and the optimal action is chosen as the unified decision across all RTG- RTG-guided trajectories at that step. This strategy guarantees a unique evaluation outcome, eliminates manual trajectory selection, and effectively integrates the strengths of all RTGs. Specifically, we prede- fine m candidate RTGs (f8- fo.... fi") and maintain m parallel trajectories. At each timestep t, the policy simultaneously generates m candidate actions (af: at a.....,,ay a."), where af is generated under the guidance of RTG rb. To evaluate these candidates, we introduce an evaluation horizon E. For specific candidate action at the model autoregressively predicts the subsequent E-step trajectory Tf = (St, at s4+1: aa=1- 1f-1..... 54+ Æ²) under the corresponding RTG it. and then computes the cumulative action- sum over the horizon:
E
Qf -x Q(sitina
(6)
==0
where Q denotes the action-value function and is the discount factor. The optimal action a is selected as a, arg max,ę Qr. This optimal action then appended to all m trajectories and used to interact with the environment to obtain the next state and reward shared across the m parallel trajectories. Crucially, the parallel nature of trajectory prediction across candidates ensures computational efficiency- despite evaluating m trajectories, the batched computation on modern GPUs results in latency comparable to single-trajectory inference. By employing a value function to evaluate and select the optimal action, we effectively integrate guidance from different RTGs, achieving optimal decision- decision-making at each step while maintaining constant inference time. We concisely outline the VDT pipeline in Appendix[D
Our sampling procedure shares similarities with both CEM 37 and SBBC 38 Like CEM, we generate multiple candidate actions at each step, evaluate them with value function over short planning horizon, and select the best one esse-ssaty a single-step population- SIBC, which samples from behavior policy and selects by Q-value, VDT uses RTGs to guide candidate actions and evaluates them in parallel. This integration of RTG guidance and batch evaluation allows VDT to combine the strengths of population search and candidate selection while maintaining efficient inference.
search. Unlike
5 Experiment
In this section, extensively evaluate our proposed Value-Guided Decision Transformer (VDT) using the widely recognized D4RL benchmark [39]. As an integrated framework, VDT focuses on performance in offline and offline-to-online (hereafter referred to as "online" for simplicity, without causing ambiguity) settings in the main experiments to ensure the model's generality. Since the offline and online pipelines are nearly identical, we conduct ablation studies under the offline setting to evaluate the shared components.
Datasets.
We consider five different domains of tasks in the widely used D4RL benchmark: Gym, Adroit, Kitchen, AntMaze and Maze2D. A detailed introduction to these five environments is presented in Appendix E

---

Baselines. In the offline traing phase, we compare VDT with representative offline RL algorithms from value-based and CSM methods. For value-based methods, including BEAR 40) BCQ [4]. 43 and COMBO [32 For CSM methods, including DT, DD [44]
CQL 23 MoRel
42 O-RL
EDAC 45 D-QL [46, MPPI 477. StAR 48 GDT 49 and CGDT 12]. the online tuning
phase, we compare VDT with ODT, IQL (24), AWAC 50), CQL [23] and PDT (9). SAC [5]] and
TD3+BC [52]. The performance scores for these baseline methods are sourced from the best results published in respective papers or from our runs, ensuring fair comparison.
Implementation details. All experiments are carried out on a server with 8 NVIDIA 3090 GPUs, each with 24GB of memory. The experimental hyperparameter configurations of VDT are shown in Appendix C
5.1
Main Experiment
Table I: Offline training performance of VDT and state-of-the-art baselines on D4RL tasks. For VDT, results are reported as the mean and standard error of normalized rewards over 30 random rollouts (3 independently trained models with 10 trajectories each), generally showing low variance.
Dataset
Value-Based Methods
Conditional Sequence Modeling Methods
Gym Tasks
BEAR
BCQ
CQL
IQL
MoRel
BC
DT
StAR
GDT
CGDT
DC
VDT
malfcheetab-mmeiun--eplagg
38.6
34.8
37.5
44.1
40.2
36.6
36.6
36.8
40.5
40.4
41.3
39.4 +2.0
hupper-mediam-rellayye
33.7
31.1
95.0
92.1
93.6
18.1
82.7
29.2
85.3
93.4
94.2
96.021.
walker2d mefeerdddmmiiimpppppa----.
19.2
13.7
77.2
73.7
49.8
32,3
79.4
39.8
77.5
78.1
76.6
82.3 12.1
halfcheetah- medium-x2
41.5
42.1
42.6
42.6
42.9
43.9
hopper-mediom-42
52.1
65.1
58.5
63.8
95.4
52,9
67.6
59.5
77.1
96.9
92.5
98.3 20.1
walker2d mediumm-22
59.1
52.0
72.5
79.9
77.8
75.3
74.0
73.8
76.5
79.1
79.2
81.6.1 .7
hulfcheetah medium expert-v2
69.6
55.2
86.8
93.0
93.9.c.1
pamper-medium-expee
96.3
109.1
105.4
91.5
108.7
52.5
107.6
111.1
111.1
107.6
110.4
111.5
walker2d-nen mxduum-xcemmum-
40.1
67.3
108.8
109.6
95.6
107.5
108.1
109.0
107.7
109.3
109.6
110.4.n9
Average
53.8
77.6
76.5
72.9
52.6
76.2
66.2
79.1
82.4
82.2
84.1
Adroit Tasks
BEAR
BCQ
CQL
IQL.
MoRel
EDAC
BC
D-QL
StAR
GDT
VDT
pen-human-vl
-1.0
66.9
37.5
71.5
-3.2
$2.1
63.9
79.5
72.8
77.9
92.5
126.7
hammer-bunan--i
2.7
0.9
4.4
1.4
2.3
0.8
1.2
3.7
0.2
3.7
5.5
3.2 10.5
door humannu vl
9.9
10.7
2.0
19.7,03
pen-cloned-vl
-0.2
50.9
39.2
37.3
-0.2
68.2
37.0
75.8
57.3
33.1
86.2
148.6
clonedr-
2.3
0.4
2.1
2.1
2.3
0.3
0.6
3.0
3.1
0.3
8.9
19.6. A
door-cloned-vl
2.3
0.01
0.4
1.6
2.3
9.6
0.0
16.3
0.0
0.0
19.8
30.6
Average
1.0
19.8
15.6
19.7
1.0
23.6
17.5
32.2
22.2
19.4
38.9
57.6
Kitchen Tusks
BEAR
BCQ
CQL
IQL
O-RL
BC
DT
DD
SLAR
GDT
DC
VDT
kilchen-complete-D
0.0
8.1
43,8
62.5
2.0
65.0
50.8
65.0
40.8
43,8
40.9
65.9.c.
kitchen-partial-v0
33.8
57.9
57.0
12.3
73.3
66.8
1200
Average
6.6
13.5
46.8
54.4
18.8
51.5
54.4
61.0
26.6
58.6
58.7
71.0
Maze2D Tasks
BCQ
CQL
IQL
COMBO
BC
MPPI
QUT
GDT
DC
VDT
maze2d-umao-vv
65.7
49.1
86.7
42.1
76.4
85.7
33.2
31.0
57.3
50.4
20.1
88.0.
maze2--mediumxll
25.0
17.1
41.8
34.9
38.5
38.3
10.2
8.2
13.3
7.8
38.2
60.3gc.5
Average
45.35
33.1
64.3
38.5
72.5
63.6
21.7
19.6
35.3
29.1
$7.6
74.2
AntMaze Tasks
BEAR
BCQ
CQL
IQL
O-RL
BC
DT
RvS
StAR
GDT
DC
VDT
antmaze- uu
78.9
54.6
59.2
65.4
51.3
85.0
100.0.s:
nataaz-umaazedivesse--
61.0
55.0
84.0
64.4
60.7
45.6
66.2
60.9
45.6
69.0
78.5
100.0.4.
antmaze medium diverse vO
8.0
0.0
53.7
70.0
0.0
0.0
7.5
67.3
0.0
0.0
0.0
30.0uzs
Average
33.4
44.3
75.0
32.3
48.3
$4.5
76.7
Offline Training Performance. VDT consistently achieves or approaches thee---a- art perfor- mance across all datasets in the pure offline setting. demonstrating the effectiveness of our architecture. The Gym and Adroit environments are characterized by a limited scope of human demonstrations, which leads to extrapolation errors that particularly challenge offline RI. This is precisely why VDT's excellent performance across all tasks can be attributed to its high expressiveness and more effective value guidance. The results of Kitchen tasks requiring generalization to unseen states and long-term term
7

---

Table 2: Offline-to-online performance of each method, with average rewards reported before (left of arrow) and after (right of arrow) online tuning.
Dataset
TD3+BC
AWAC
COL
IQL
PDT
ODT
VDT
44.6 - 48.1
24.3
39.0
45.5 44.3
44.1 + 44.0
314
42.88 39.9 + 40.4
39.4 - 49.2
hupper-medium--eplyy-y22
60.9 90.7
77.3 79.6
95.0 95.3
92.1 - 93.5
84.5 94.8 86.6 - 88,9
96.0 - 119.2
walker2dd moduuzd--mmddum--ppaa--2222
77.2
73.7
76.9
halfchectal--ncddum-v22
48.3 + 50.9
-7.4 41.1
44.0 -4.0-24 29.1
47.4 - 48.0
39.4 - 69.5
42.7-+42.3
43,9 - 53.5
hopper--eddum--22
59.3 64.6
72.0 - 91.0
58.5 95.7
63.8 44.3
74.4 100.2
66.9 - 97.5
98.3 108.1
walker2d-meciimm-22
72.5
76.8
halfcheetah- medium expert-v2
90.7 - 92.1
36.8 - 41.0
91.6 99.9
86.7 95.3
82.6 93.3
36.8
100.9
93.9 101.7
bogge--mdddum--xxpptt2222
- 42.7 - 78.3
106.3
- 92.9
111.5
walker2d medium expert-v2 v2
110.1 - 110.1
108.8 - 110.1 109.6 -> 109.6
99.1 - 108.9 62.0 - 78.7
110.4 - 112.7
animaze umaze-vt v0
78.6 79.1
56.7 59.0
70.1 99.4
86.7 96.0
48.6 66.8
53.1
88.5
100.0
110.0
animaze uwaze diverse-Ve
71.4
49.3 1- 49.0
31.1
- 56.0
100.0
attnaaz--mcdiuu--drrreo-00
0.0 - 56.7
0.7 - 0.3
23.0 - 32.3
68.3 + 72.0
8.0 - 63.4
0.8
55.6
20.0 - 75.0
Average
81.6
75.78
80.51
75.13
94.38
Table 3: Ablation study on model components during offline training, We have abbreviated some task names for simplicity, which does not affect understanding. All experiments are repeated three times, and the average value is taken.
Advantage Weighting Regularization Sampling hopper-m walker-m-c pen-eloned mazc2d-m antmaze-u
90.3
99.9
86.1
12.1
75.1
88.9
78.1
99.3
30.5
60.9
78.6
80.3
82.0
19.3
0.0
95.6
103.6
131.8
40.5
95.9
98.3
110.4
145.6
60.3
100.0
value optimization demonstrate that VDT can learn useful data features from offline trajectories, enhancing generalization and stability. For the Maze2d environment, which serves benchmark to evaluate the capacity of algorithms to stitch segments of disparate trajectories effectively, the performance of VDT significantly outperforms other methods, demonstrating the advantage of the value functions in stitching high-quality trajectories. The AntMaze environment is characterized by sparse rewards and many suboptimal trajectories, which present an The performance results of VDT demonstrate the effectiveness and generalizability of the architecture designed, particularly in antmaze- wettaaze-umzz--eee diver-e tasks.
significant challenge.
Online Tuning Performance. We conduct online tuning VDT and observe from the results that VDT achieves optimal or competitive performance across nearly all tasks (as shown in Table (2) These results demonstrate that value-guided methods retain their advantage even in online settings. We attribute the strong performance of value-guided methods in the online setting to their ability to rapidly extract and leverage the Markovian structure of the environment through interaction. This inductive bias allows value-based approaches to adapt efficiently with limited data. In contrast, CSM lacks explicit mechanisms for modeling state transitions and thus may struggle settings that require fast generalization from sparse interactions.
Compared methods like ODT, which are restricted to a single setting, VDT achieves optimal performance in offline and offline-to-onling scenarios. This powerfully demonstrates the inherent potential of value- guided strategies.
5.2 Ablation Study
Role of Different Components. As shown in Table 3 we conduct an ablation study on the three key components involving the value guidance in VDT. The results show that using either advantage weighting or individual regularization can improve baseline performance. However, the effectiveness varies significantly across tasks. When both components are applied together (as shown in the fourth row), performance improves substantially, suggesting that advantage weighting and regularization complement policy learning. Specifically, advantage weighting is important form of sampling

---

hoppri-mediu--22
wa ko22d-mmeduu--mmppaa222
antmaze-urAAz--00
100
125
100
Average Rowards
Orline
80
Oniine
100
80
40
50-
Offline Online
20
25
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
0.4
0.6
0.8
2
Figure Ablation on the hyperparameter 6.
that encourages the policy to favor high-return behaviors. At the same time, the regularization term penalizes low-value actions, effectively constraining the policy within high-value regions. Furthermore, incorporating Q-value-guided sampling enhances the stability and robustness of policy execution. When three components are combined (as in the last row), the model achieves the best overall performance and consistency across tasks.
Computational complex- ity.
Table 4: Ablation the computational complexity.
IQL, ODT. and VDT regarding memory usage, parameter count, and training time during offline training and online tuning. Although IQL has the lowest compu- tational cost, inevitably suffers from performance limitations. VDT has a slightly higher parameter count than ODT due the introduction of the value function. However, the value-guided training process leads to faster convergence and better performance in both offline and online stages. There- fore, the slight increase in parameters is considered acceptable. VDT balances model capacity and computational efficiency well, maintaining strong representational power while significantly reducing time costs.
Offline Training
Online Tuning
Complexity
IQL
ODT
VDT
IQL
ODT
VDT
Memory 4
960 M
3968 M
4024 M
960 M
3968 4024 M
Params 1
0.60 M
5.01 M
5.24 M
0.60 M
5.01 M
5.24 M
Clock Time +
1.0
R h.0
2 5.0
1.0
R3 4.5
4.0
Impact of evaluation horizon E. We investigated the impact of the evaluation horizon during the sampling process (shown in 3). Specifically, we analyzed how varying the evaluation horizon under the guidance of the value function affects model performance. It is well known that a short evaluation horizon may lead myopic policies, while overly long horizon can significantly slow down evaluation. Ablation studies show that a horizon length of 5 yields the best performance. Further increasing the horizon results in varying performance trends depending on the task, which is related to the dataset's quality and the rewards' sparsity. For simplicity, we set the evaluation horizon for all tasks.
152
Homaads
232
hoppan peppermman mediumi
129
wekenmeduumeceett
pen-steredvv
310
100
Figure 3: Ablation: Effect of Evalu- ation Horizon Length on Offline Training Performance of VDT.
Impact of context lengths K. As shown in Table 5 we observed that VDT per- formance improved vary- ing degrees with increasing context length, indicating that VDT exhibits excellent ex- tendability. When
Table Offline training performance VDT with different context lengths (K) on Gym tasks.
Datasets
VDT (8)
VDT (20)
VDT (60) VDT((10)
halfehcetah-medde
28.6
43.9
44.6
43.0
hopper-medium
77.0
98.3
99.1
65.4
walker2d-medium
52.6
81.6
79.9
80.5
60, VDT achieved highest aver- age performance. However, increasing context length un- questioningly would signifi- cantly increase model com- plexity and computational cost. Considering the trade-off between performance, complexity, and ensuring fair comparison, we set K 20.
halfcheetah aaafichetth-mmediu-eeppeee
89.5
93.9
93.9
77.0
hopper-medium-exprr.
109.3
111.5
112.7
111.2
walker2d-meddum-exper
100.6
110.4
110.4
103.8
Average
76.3
89.9
90.1
80.2

---

Impact of hyperparameter e. In expectile regression, the parameter controls the value function's s preference over different TD targets. Such preference plays crucial role in sparse reward settings or high-variance environments, as it helps prevent the policy from being misled by "lucky" samples. As shown in Figure 2 setting = 0.7 is generally effective across most tasks in both offline and online scenarios. When approaches larger values, performance improvements are observed only on the AntMaze tasks, which we attribute to their reliance on trajectory stitching.
Comparison with stitching baselines. We conducted further comparison between VDT and EDT [53], which specializes in trajectory stitching. The core idea behind EDT's ability to stitch trajectories lies in adaptive history truncation. During inference, EDT dynamically selects the optimal history length to maximize the maximum achievable return from the current state. EDT does not use Q-function to guide action selection. Instead, it forgets unsuccessful histories, which allows the model to escape from low-return trajectories and connect more optimal trajectory branches. In addition, we have included a comparison with
QDT (54) which combines the dynamic pro- gramming capabilities of Q-learning with the sequence modeling strengths of DT, enabling re- labeling of return targets and thus improving DT's ability to stitch suboptimal trajectories. VDT differs in that it explicitly introduces O-function to provide value- value-guided guided weighting for action selection. At the same time, it em- plays multi-step Bellman equations and double Q-networks to stabilize value function training, thereby extending VDT to be applicable both offline and online settings, and enabling paral- lel decision-making during sampling to leverage the evaluation capability of the Q-function fully, As shown in Table 6 VDT demonstrates clear advantage over EDT and QDT across a wide range of tasks.
Table 6: Performance comparison of EDT, QDT,
and VDT on Gym tasks.
Datasets
EDT
QCS
VDT
halfcheetah-mmeding
42.5
42.3
43.9
hopper-medium
63.5
66.5
98.3
walker2d-medium
72.8
67.1
81.6
halfchhecah-mmddiu--rpllpg
37.8
35.6
39.4
hopper-mediim--eplay
89.0
52.1
96.0
walker2d (iiiu22--mciium-rpplaa
74.8
58.2
82.3
Average
63.4
53.6
73.6
Performance comparison in Atari environments. We evaluate the proposed VDT on the image-based Atari dataset, with results averaged over three random seeds (Table VDT achieves the highest average score across all considered games, substantially outperforming the prior methods CQL, DT, and DC. Specifically, VDT sets new benchmarks in the majority of tasks, evidencing its enhanced ability to handle high-dimensional visual observations. These results demonstrate that VDT pos- sesses superior generalization capability and robustness, validating its effectiveness not only on text-based envi- ronments but also in challenging visual domains. The consistent improvements highlight the benefit of our model design in leveraging sequential and high-level representations, thereby providing unified solution for diverse decision-making scenarios.
Table Performance Comparison in Atari Environments.
Game
CQL
DT
DC
VDT
Breakout
211.1
242.4
352.7
420.8
Qhert
104.2
28.8
67.0
69.4
Pong
111.9
105.6
106.5
113.9
Seaquest
1.7
2.7
2.6
3.9
Frostbite
9.4
25.6
27.8
28.9
Average
87.7
81.0
111.3
127.4
Conclusion
In this work, we propose the Value-Guided Decision Transformer (VDT), which organically integrates policy improvement with behavior cloning, enabling efficient trajectory stitching and decision- decision-makin making through components such advantage-weighted learning and value regularization. Experiments on diverse RL benchmarks demonstrate that VDT achieves competitive performance in offline and online settings, particularly excelling in stochastic scenarios and suboptimal data regimes. This work establishes a unified CSM architecture for generalizable RL, paving the way for scalable and robust transformer-based policies in weal-world applications.
Limitation. VDT relies on parallel trajectory evaluation during sampling, which introduces some additional computational cost, though this is generally manageable with GPU batching.
10

---

Acknowledgments and Disclosure of Funding
This work is supported by the National Key Research and Development Program of China (2033FF27005000) the National Natural Science Foundation of China (Grant No. 62225113, U23A20318, 62576364 and 62276195), the Foundation for Innovative Research Groups of Hubei Province (Grant No. 2024AFA017), the Science and Technology Major Project of Hubei Province (Grant No. 2024BAB046), the Shenzhen Basic Research Project (Natural Science Foundation) Basic Research Key Project (NO. JCYJ202412021 12244244444444 the CCF-Tencent Rhino-Bird Open Research Fund (NO. CCF-Tencent RAGR202501 14) and Tencent JR2025TEG002. Dr. Tao's research partially supported by NTU RSR and Start Up Grants. The numerical calculations in this paper have been done on the supercomputing system the Supercomputing Center of Wuhan University.
References
[1] Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives open problems. arXiv preprint 2xXiv:005500164 2020.
[2] A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems,
2017.
[3] Tom B Brown. Language models are few-shot learners. arXiv preprint (00.vv:2005.411660 2020.
[4] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 0rX10:2303.08774, 2023.
[5] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 10012-10022, 2021.
[6] Chenyang Si, Weihao Yu, Pan Zhou, Yichen Zhou, Xinchao Wang, and Shuicheng Yan. In- ception transformer. Advances in Neural Information Processing Systems, 35:23495-23509, 2022.
[7] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling, Advances in neural information processing systems, 14:15044-15097, 2021.
[8] Linghui Meng, Muning Wen, Chenyang Le, Xiyun Li, Dengpeng Xing, Weinan Zhang, Ying Wen, Haifeng Zhang, Jun Wang, Yaodong Yang, al. Offline pre-trained multi-agent decision transformer. Machine Intelligence Research, 20(2):233-248, 2023.
[9] Zhihui Xie, Zichuan Lin, Deheng Ye, Qiang Fu, Yang Wei, and Shuai Li. Future-conditioned unsupervised pretraining for decision transformer. In International Conference on Machine Learning, pages 38187-38203. PMLR, 2023.
[10] Hongling Zheng, Li Shen, Yong Luo, Deheng Ye, Bo Du, Jialie Shen, and Dacheng Tao. Decision mixer: Integrating long-term and local dependencies via dynamic token selection for recision-making In Forty-second International Conference on Machine Learning, 2025.
[11] Longyang Huang, Botao Dong, and Weidong Zhang. Efficient offline reinforcement learning with relaxed conservatism. IEEE Transactions on Pattern Analysis and Machine Intelligence, 46(8):5260-5272, 2024.
[12] Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu, and Yu Qiao. Critic-guided decision trans- former for offline reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, pages 15706-15714, 2024.
[13] Chen-Xiao Gao, Chenyang Wu, Mingjun Cao, Rui Kong, Zongzhang Zhang, and Yang Yu. Act: empowering decision transformer with dynamic programming via advantage conditioning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 12127-12135, 2024.
11

---

[14] Philip Ball, Laura Smith, Ilya Kostrikov, and Sergey Levine. Efficient online reinforcement learning with offline data. In International Conference on Machine Learning, pages 1577-1594. PMLR, 2023.
[15] Mitsuhiko Nakamoto, Simon Zhai, Anikait Singh. Max Sobol Mark, Yi Ma, Chelsea Finn, Aviral Kumar, and Sergey Levine. Cal-ql: Calibrated offline rl pre-training for efficient online fine-tuning. Advances in Neural Information Processing Systems, 36:6224446222222 2023.
[16] Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning, pages 27042-27059. PMIR, 2022.
[17] Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An optimistic perspective on offline reinforcement learning. In International conference on machine learning, pages 104-114 14. PMLR, 2020.
[18] Shengchao Hu, Li Shen, Ya Zhang, Yixin Chen, and Dacheng Tao. On transforming reinforce- ment learning with transformers: The development trajectory. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[19] David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does reuun-conditionee supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, 35:1542-1553 1553, 2022.
[20] Hongling Zheng. Li Shen, Yong Luo, Tongliang Liu, Jialie Shen, and Dacheng Tao. Decomposed prompt decision transformer for efficient unseen task generalization. Advances in Neural Information Processing Systems, 37:122984 0119984-130000 123006, 2024.
[21] Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances Neural Information Processing Systems, 36, 2024.
[22] Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan, Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, al. Q-transformer: Scalable offline reinforcement learning via autoregressive q-functions. In Conference on Robot Learning, pages 3909-3922 PMLR, 2023.
[23] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in neural information processing systems, 33:1179- 1191, 2020.
[24] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q-leaming, arXiv preprini arXiv:21 2021.
[25] Ying Li and De Xu. Skill learning for robotic insertion based on one-shot demonstration and reinforcement learning. International Journal of Automation and Computing, 18(3):457-467,
2021.
[26] Shenzhi Wang, Qisen Yang, Jiawei Gao, Matthieu Lin, Hao Chen, Liwei Wu, Ning Jia, Shiji Song, and Gao Huang. Train once, get family: State-adaptive balances for offline-to-nnline reinforcement learning. Advances in Neural Information Processing Systems, 36:47011-47104, 2023.
[27] Kai Zhao, Yi Ma, Jinyi Liu, Jianye Hao, Yan Zheng, and Zhaopeng Meng. Improving offline- to-online reinforcement learning with q-ensembles. ICML Workshop on New Frontiers in Learning, Control, and Dynamical Systems, 2023.
(28) Qingyang Zhang, Hongming Zhang, Dengpeng Xing, and Bo Xu. Latent landmark graph for efficient exploration-exploitatiii balance in hierarchical reinforcement learning. Machine Intelligence Research, 22(2):267-288, 2025.
[29] Chi Zhang, Wei Zou, Ningbo Cheng, and Shuomo Zhang. Towards jumping skill learning by target-guided policy optimization for quadruped robots. Machine Intelligence Research, 112 2024.
12

---

[30] Zhongjian Qiao, Jiafei Lyu, Kechen Jiao, Qi Liu, and Xiu Li. Sumo: Search-basee uncer- tainty estimation for model-based offline reinforeement learning. In Proceedings of the AAAI Conference Artificial Intelligence, pages 20033-2004) 2025.
[31] Yan Song, He Jiang, Zheng Tian, Haifeng Zhang, Yingping Zhang, Jiangcheng Zhu, Zonghong Dai, Weinan Zhang, and Jun Wang. An empirical study on google research football multi-agent scenarios. Machine Intelligence Research, 21(3):59--770, 2024.
[32] Tianhe Yu, Aviral Kumar, Rafael Rafailov, Aravind Rajeswaran, Sergey Levine, and Chelsea Finn. Combo: Conservative offline model-based policy optimization. Advances in neural information processing systems, 34:28954 28967, 2021.
[33] Danijar Hafner, Jurgis Pasukonis, Jimmy Ba, and Timothy Lillicrap. Mastering diverse control tasks through world models. Nature, pages 1-7, 2025.
[34] Kai Yan, Alex Schwing, and Yu-Xiong Wang. Reinforcement learning gradients as vitamin for online finetuning decision transformers. Advances in Neural Information Processing Systems, 37:38590-38628, 2024.
[35] Brett Daley, Martha White, and Marlos C. Machado. Averaging n-step returns reduces variance reinforcement learning. Proceedings of the 41st International Conference on Machine Learning, IC.L'24. JMLR.org, 2024.
[36] Eric Mitchell, Rafael Rafailov, Xue Bin Peng, Sergey Levine, and Chelsea Finn. Offline meta- reinforcement learning with advantage weighting. In International Conference on Machine Learning, pages 7780-7791. PMLR, 2021.
[37] Kurtland Chua, Roberto Calandra, Rowan McAllister, and Sergey Levine. Deep reinforcement learning in a handful of trials using probabilistic dynamics models. Advances in neural information processing systems, 31, 2018.
[38] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. Offline reinforcement learning via high-fidelity generative behavior modeling. In The Eleventh International Conference on Learning Representations, 2023.
[39] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint wxXiv:2004.07211 2020.
[40] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- policy g-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
[41] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[42] Rahul Kidambi, Aravind Rajeswaran, Praneeth Netrapalli, and Thorsten Joachims. Morel: Model-based offline reinforcement learning, Advances in neural information processing systems, 33:21810-21823. 2020.
[43] David Brandfonbrener, Will Whitney, Rajesh Ranganath, and Joan Bruna. Offline rl without off-policy evaluation. Advances in neural information processing systems, 34:4933-4946, 2021.
[44] Anurag Ajay, Yilun Du Abhi Gupta, Joshua Tenenbaum. Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-malee making? arXiv preprint 1rXiv:2211.16577 2022.
[45] Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty- based offline reinforcement learning with diversified qeensemble. Advances in neural information processing systems, 34:743677447, 2021.
[46] Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.06199 2022.
13

---

[47] Jintasit Pravitra, Kasey A Ackerman, Chengyu Cao, Naira Hovakimyan, and Evangelos A Theodorou. Ll-adaptive mppi architecture for robust and agile control of multirotors. In 2020 IEEE/RSI International Conference on Intelligent Robots and Systems (IROS), pages 7661-7666. IEEF, 2020.
[48] Jinghuan Shang, Kumara Kahatapitiya, Xiang Li, and Michael Ryoo. Starformer: Trans- former with state-action-rewwe reward representations for visual reinforcement learning. In European conference on computer vision, pages 462-479. Springer, 2022.
[49] Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer. arXiv preprint w333v:2303.03777,, 2023.
[50] Ashvin Nair, Abhishek Gupta, Murtaza Dalal, and Sergey Levine. Awac: Accelerating online reinforcement learning with offline datasets. arXiv preprint arXiv:2006.09359, 2020.
[51] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. Pmlr, 2018.
[52] Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances neural information processing systems, 34:20132-22 32-20122-,,4,,, 2021.
[53] Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in neural information processing systems, 36: 1532--88500 2023.
[54] Taku Yamagata, Ahmed Khalil, and Raul Santos- Roorioo-.d Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. In Interna- tional Conference on Machine Learning, pages 38989-39009 PMLR, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We introduce the scope of this paper in the abstract and the Introduction section, and summarize the contribution the Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We state the limitation in Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
15

---

Justification:
We include the full set of assumptions and complete proofs in the appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a detailed experimental description in the main text and the appendix. All the datasets, code, and model checkpoints are publicly available.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer:
[Yes]
Justification:
All the datasets, code, and model checkpoints are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/
sttpaiisssooooiocyyy. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, both in the main paper and in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide standard deviations in numerical form in Table Additionally, the visualization results in the ablation study section include standard deviations represented as shaded areas.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information in ablation study section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics http::
Answer: [Yes]
Justification:
The research conducted in the paper fully conforms to the NeurlPS Code of
Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly credited the creators and original owners of all assets used in the paper through citations and other appropriate means, ensuring that the licenses and terms of use are respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
19

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy buctps://neure wwwparippps..coooeeeeeeeee for what should or should be described.
21

---

Appendix
The appendix organized into several sections, each providing additional insights and details related to different aspects of the main work.
Proof of Theroem
22
Proof of Theroem 4.2
23
Hyperparameters Configuration
24
D Algorithm Pseudocode
24
Environment Details
25
More Experiments
27
Broader Impact
30
A Proof of Theroem 4.1
Let bhr be the optimal policy obtained by solving Equation For any 8 S, we have:
1. Support Constraint: (s) E sup)()())s).
2. Value Improvement: V (s) (s),
where supp($(-1s)) = {a A E E D). and (a s) = whenever 3(a s) =
Lemma A.1 (Implicit Support Constraint). The policy gradient update satisfies:
V Ep 2e (T (T.) ar) XV AV.Qu(s: a) 1- (B)
where all gradient components vanish outside supp( Lemma A.2 (Policy Improvement). The advintage-weightted value function [36] satisfies:
(s) (=bi((s)
1-9 ED min Qo, (s, (s))
(d+ (067"(())))
(8)
Proof. Define the policy optimization sequence ( (T ) where denotes training iterations, with (0)
DT B. We prove each statement in turn.
(1) Support Constraint: From the gradient expression in Lemma A.1 the policy update rule is given
by:
V Ep
2 (x (T.) ar) XV V. (s a) l= (2) (r))
where all terms are evaluated only on trajectories (st, az) from the dataset D. The gradient thus vanishes outside sups()(-
Moreover, the behavior cloning component implicitly restricts (d) to lie within the convex hull of actions observed in the dataset:
(T) lim ProjAn()) d
(DT
(10)
where Ap(s) {a (8) E DJ. In addition, the Q-regularization term enforces vanishing gradients near the boundary of this set, preventing the policy from drifting outside. Hence, the optimal policy satisfies the support constraint.
22

---

(2) Value Improvement: Define the value gap at iteration d as:
( V
(4) 1. (s),
(11)
where VTh denotes the advantage errantag--eeighte surrogate value used the optimization.
From Lemma A.2 the policy update guarantees monotonic improvement:
(d=1) VTD
(s) 2
(TD
1-7 ED min Qo, (s. a "
(12)
Applying the n-step Bellman operator T yields:
(l++1) S ,yn² AED min Qa, (s, DT
=D (s)) S 5mnc(a) - X >(vT (d+1)
(s) - ( v))=))
(13)
where we used that min; Qs lower bounds the true VTDr and that V the value under the behavior policy.
(d12)
Telescoping this recurrence and assuming convergence as d oo, we obtain:
VDbT(s) 2 (s).
(14)
The inequality is strict unless the Q-functions are constant over Ap(s), in which case the behavior policy B is already optimal within the dataset support.
B Proof of Theroem 4.2
Let x² denote the globally optimal policy and TDr denote the optimal policy constrained to dataset support. Define (s(s) = {a (8, a) D) to be set of actions observed dataset at state 8. For any we have:
27 v" (M) VTb S (1 7)2
E Bander
max QF (s.a)
max (ESupp((((s))
QT (3)
(15)
(gAA((s)
Proof. Step 1: Performance Difference Lemma
Recall for any two policies π,, ní, the performance difference lemma states:
(T) V= (w) Egnd" (a-r(-1s) |0o (s, V vT(s)]
(16)
where (s)(s) (s) is the normalized discounted state distribution under æ.
Step 2: Comparing x² and Tbr
Applying the lemma yields:
V (p) V (1) = 1
Exndr BanTr((a)) (s) v
(17)
1
<
Byndt max (s.
max (xbT(s,a) aEAD(s)
(18)
since nT may select actions a' & Ap(s) that Tbr cannot, incurring value gap.
Step 3: Relating " to Q""
Since Q (s, a) s Q=' (s, for all a, we may further bound:
(*br(s) max o (s,a) S max Q= (s, aEAp(s)
(19)
aEAp(.)
and
V"" (s)
max QT (s, 0)
(20)
B
23

---

At each state s, the maximal loss incurred is thus
8(8) = V (s)
max
Q" (s) max Q" (8,
max aEAp(()
(3,
(21)
aE.A((s)
Step 4: Cumulative suboptimality by recursion
Because Dr can only select actions within the dataset support at each step, the loss 0(s,) compounds over all timesteps. By recursive expansion and geometric series, we obtain:
V (w) V (p)
LE Er [6(s)] + 1 1- E3 ann [ō(s)] +...
(22)
Summing the geometric series yields:
V V= (N) S (1 y1- snnd (8(s)
(23)
By refinement [41] accounting for effect propagation across steps, a factor 27 appears:
27 E Band [8(s)
V (1) V (1) S (1 7)2
(24)
Thus, we have proven
V" (4) Vair (1) -
27
max Q (s, aEmup((((-1a)
max
Q (s.2))
(25)
(1 7)2
aggAn(s)
which bounds the loss in optimal value due to dataset support constraints.
C Hyperparameters Configuration
Table 8: Hyperparameter configurations for offline training and online tuning.
Hyperparameter
Offline Training
Online Tuning
Context Length K
20
20
Batch Size
512
512
Training Steps
10000
25000
Learning Rate
3e-4
le-4
Weight Decay
le-4
Number of Layers
6
6
Attention Heads
4
4
Embedding Dimension
256
256
Activation
GeLU
GeLU
Dropout
0.1
0.1
Discount 7
0.99
0.99
Threshold
0.7
0.7
Inverse Temperature
3
3
Balance Coefficient À
0.5
0.5
pet_traj
1
1
Updates between Rollouts
300
Gradient Norm Clip
0.25
0.25
Replay Buffer Size
1000
Q-network Layers
2
2
Q-network Width
256
256
D Algorithm Pseudocode
24

---

Algorithm Offline Training with VDT
Input: Offline dataset Doffine Context length K, Expectile Discount 7, X, Initialize: TDT. Vas Q8,, Q02 Qū, Q0,
for iteration to T do
Sample batch B D Doffine
(Value network update)
Update V using: (( (t.a))=lL5((gg3..) - (s){
(Q-network update}
Compute n-step targets: y En= rhe + 7"h "/V (8t44)
Update Qo, using: Eslyy Qa, (st, ax)) for i = 1,2 { [Advantage-weighte. policy update}
Compute advantages: At min; Qi (st, (s((s)
Update DD using:
Es (e)p((A))(==xe
aella - X mmn, Qa, (&e, (DT(T)))
(Target network )pdate}
0, < p0; + (1 - p)0; for i = 1,2
end for
Algorithm 2 Online Tuning with Trajectory Replay
Input: Pretrained DD. Qa, Online RTG goalines Buffer size N, Context length K Initialize replay buffer Treplay Top-N(Damiing for round 1 to R do
Generate trajectory with FDT using Jonline Relabel RTG: gr = E} FJ for T
Update Treplay (FIFO)
for gradient step 1 to 7 do
Sample trajectories {Ty} ~ Troplay with p(T) x |T|
for each Tj do
Sample sub-trajectory ($114K a.atK (gxgKKK Compute A, min, Qú, (8t, a+) - Vo (82)
end for
Update DD using:
Σ exp(()A.) (lT 9:) a. min; Qa, (Be, (DI 0+))
Update Qo. using online transitions
0, + p0, + (1 -)),
end for
end for
E
Environment Details
Gym tasks: The Gym-MuJoCo tasks (hopper, halfcheetah, walker2d) are popular benchmarks used in offline deep RL. They are relatively straightforward and characterized by datasets with significant proportion of near-optimal trajectories and smooth reward functions. The "medium" " dataset generated by first training policy online using Soft Actor-Critic, early-stopping the training, and collecting 1M samples from this partially-trainet policy. The "random" " datasets are generated by unrolling randomly initialized policy these three domains. The "medium-replay" dataset consists of recording all samples in the replay buffer observed during training until the policy reaches the "medium" level of performance. Datasets similar to these three have been used in prior work, but in order to evaluate algorithms on mixtures of policies, we further introduce a mediium-expens dataset by mixing equal amounts of expert demonstrations and suboptimal data, generated via a partially trained policy or by unrolling a uniform-a ciiim-at-rrann at-random policy.
Adroit tasks: The Adroit domain involves controlling a 24-DoF simulated Shadow Hand robot to perform tasks such as hammering a nail, opening a door, twirling a pen, or picking up and moving ball. This domain is chosen to study the impact of narrow expert data distributions and human
25

---

Algorithm 3 Sampling Process
Input:
Initial state so, candidate RTGs (f....., )0)) Evaluation horizon E, Discount 7 Policy
NDT, Q-networks Qè Qbz
Initialize: Current state <- Active trajectories {T*) + {(80. (0) Target Q- networks Qú.
while not termination condition do
// Parallel candidate action generation
for
m in parallel do Sample action of TD (T*) end for
// Batched trajectory prediction for = m in parallel do
Initialize predicted trajectory "pred f- (st, a
Initialize cumulative Q-value Qrocal +- for i = 0 to F 1 do
Predict next state: st st++1 EnvModel pred)
Sample next action: a.+i+1 ~ TTD pred
Compute Q-value: of minj=1,2 Qô, (s (44))
Accumulate: Qt <- Q +
Append (af+141> s(+++1) to pred
end for end for
// Optimal action selection
Select optimal index: k* <- arg maxiskem Qu
Execute action: ag + af
// Environment interaction & trajectory update
Observe reward rt, next state 8t+1 from environment for to m do
Update RTG: F < Pt Pt Append transition: t t- 7h @ (ar, Te. S141, f411) end for + +++1 end while
Figure 4: Illustration of Gym environments 39
Figure 5: Illustration of Adroit environments 39
26

---

demonstrations on sparse-reward, sigg--imensional robotic manipulation tasks. Since these tasks are primarily derived from human behavior, they exhibit a limited state-action space, requiring robust policy regularization to ensure consistent agent performance. The Adroit domain has several unique properties that make it qualitatively different from the Gym tasks. First, the data collected from human demonstrators. Second, each task is difficult to solve with online RI due to sparse rewards and exploration challenges, which make cloning and online RL alone insufficient. Lastly, the tasks are high-dimensional, presenting representation learning challenge.
Kitchen tasks: The Kitchen domain involves controlling 9-DoF Franka robot in a kitchen environment with everyday household items such as microwave, kettle, overhead light, cabinets, and an oven. The goal is to interact with these items to achieve desired state configuration. This domain bench- marks the impact of multitasking behaviour in realistic, non- navigation environment, where the "stitching" challenge arises from complex paths through the state space. Consequently, al- gorithms must generalize to unseen states rather than rely solely on training trajectories. The environment requires the agent to complete multiple sequential sub-tasks, further emphasizing the need for robust generalization. The "complete" dataset consists of the robot performing all the desired tasks order. This provides data that is easy for imitation learning method to solve. The "partial" dataset consists of undirected data, where the rohot performs subtasks that are not necessarily related to the goal configuration. In the "partial" dataset, subset is guaranteed to solve the task, meaning imitation learning agent may learn by selectively choosing the proper subsets the data.
Figure 6: Illustration of Kitchen en- vironments [39].
The Maze2D domain is navigation task in which 2D agent must reach fixed goal location. tests offline RL algorithms" ability to stitch together previously collected sub- trajectories to find the shortest path to the goal. Three maze layouts are provided: the "maze", "medium", ", and "large" mazes. These tasks evaluate the algorithm's 's capability to effectively combine sub- uub-trajectories and identify the shortest path to the set goal. The data is generated by selecting goal locations randomly and then using a planner that generates sequences of waypoints, followed by PD controller. The trajectories in the dataset visualized in Appendix G. Because the controllers memorize the reached waypoints, the data collection policy is non-Markovian.
Maze2D tasks:
Figure 7: Illustration of Maze2D and AntMaze environments 39
AntMaze tasks: The AntMaze domain extends the Maze2D task by replacing the 2D ball with more complex 8-DoF "Ant" quadruped robot, presenting more demanding navigation challenge. This domain is introduced to test the stitching challenge with morphologically complex robot, better representing real-world robotic navigation tasks. The task uses sparse 0-1 reward, activated upon reaching the goal. The data is generated by training a goal-reaching policy and using it with the same high-level waypoint generator from maze2d to provide subgoals that guide the agent to the goal. As in Maze2D, the controllers for this task are non-Markovian as they rely tracking visited waypoints.
More Experiments
Impact of model size. We investigate the impact of model size on various behavior cloning and decision transformer variants, including BC, DT, ODT, and VDT, under both offline and online
27

---

Table 9: The size and the average and standard deviation of the normalized reward in our experiments.
Dataset
Size
Normalized Reward
balichetalhmmciun--rpaayav
202000
27.17 = 15.79
hopper-mediuu--epllyy-22
402000
14.98 + 16.32
walkerzdd-mcdduu--eeepay-y2
302000
14.84 + 19.48
halfcheetahh-medium--2
1000000
40.68 + 5.12
hopper-medium-v2
999906
44.32 + 12.27
walker2d-mediuun-22
999995
62.09 + 23.83
hallcheectah-expert-V2
1000000
80.30 = 35.82
hopperr-medduu--xxeerttz2
999906
100.59 31.66
walker2d-mmediuu--xxpennz22
999995
112.09 + 23.83
pen-human-vl
4800
202.69 154.48
hammer-human-vl
10948
23.80 33.36
door-human-vl
6504
28.35 + 13.88
pen-cloned-vl
499886
108.63 + 122.43
hammer-cloned-vl
999872
8.11 + 23.35
door-cloned-vl
999939
12.29 + 18.35
kitchen-comppete--v)
999800
62.25 + 19.83
kitchen-partial-v0)
999800
89.49 + 14.15
maze2d-umaze-vi
999869
12.55 + 9.82
maze2d-mediun-vl
1999733
-3.46 + 3.95
antmaze-umaze-vo
998573
86.14 34.55
unthaaze-mediuun-diveee--
999930
6.36 - 10.07
antmaze- sianazz--umaazz-diveee-----
999000
3.48 + 18.32
settings. As shown in Table 10 performance generally improves with increased model capacity. For example, across all three datasets haaasstts-alffhettalll hopper, and walker2d-VDD (online) consistently achieves the best performance for each model size, with scores improving we move from the smallest configuration (3,1,256) the largest (12.12,768). The performance saturates drops slightly on walker2d in the largest model, suggesting that model complexity must be matched with task difficulty and data availability. Similarly, other variants like ODT and DT also benefit from larger models, though the gain is more moderate than VDT. Interestingly, the offline methods also demonstrate strong performance with moderate-sine models, especially VDT (offline), which performs competitively or better than its online variant in smaller models. For instance, in the (6,4,256) configuration, VDT (offline) achieves 98.3 on hopper, matching the best result at this size. These results suggest that while increasing model size generally boosts performance, particularly for VDT, the returns diminish and may even reverse if the model becomes too large relative to the dataset, likely due to overfitting optimisation difficulty in reinforcement learning scenarios.
Impact of inverse temperature 77. Table I1 shows the impact of the inverse temperature r) on policy performance. As 1 increases from to 3, performance consistently improves, indicating that more substantial advantage weighting helps the model focus on high-value trajectories. In this context, n amplifies the difference between actions, guiding the policy toward more optimal behavior. However, setting v too high (e.g.. = 10) leads to performance degradation, likely due overfitting to small subset of high-advantage samples and reduced generalization. performance offline and online. This trend highlights the role of v in controlling the selectivity the learning process, where moderate value strikes good balance between stability and performance.
3 achieves the best average
28

---

Table 10: Ablation on the model size. Model size is denoted as (x, y, 2) for number of layers. attention heads, and embedding dimension. Bold indicates the best result overall, and underline highlights the best among offline methods.
Model Size
Method
-atiieetta--meiuum-222 hopper-medium--2 walker2d-mediumt
BC (offline)
34.0
44.8
72.3
DT (offline)
42.7
60.3
70.6
ODT (offline)
18.3
62.9
57.2
(3,1,256)
VDT (offline)
43.6
98.3
79.0
ODT (online)
23.6
70.1
52.8
VDT (online)
49.0
105.9
89.8
BC (offline)
42.6
52.9
75.3
DT (offline)
42.6
67.6
74.0
ODT (offline)
42.7
66.9
72.2
(6,4,256)
VDT (offline)
43.9
98.3
81.6
ODT (online)
42.2
97.5
76.8
VDT (online)
53,5
108.1
89.8
(offline)
42.9
67.3
69.1
DT (offline)
39.3
74.9
75.2
ODT (offline)
42.7
79.1
66.0
(12,12,768)
VDT (offline)
44.0
96.0
74.1
ODT (online)
43.9
99.8
82.1
VDT (online)
54.9
108.9
84.2
Table 11: Ablation on the hyperparameter
Offline Training
Online Tuning
Dataset
=1
")
=3
5
=10
=1
")
=3
5
=10
halfchhetah-meduum--xpprr.
34.2
93.9
88.65
59.1
77.3
101.7
10.9
103.9
hopper-medium-v2
18.3
98.3
97.6
70.0
29.6
108.1
93.1
100.3
walker2d- mallerrddmmdduum-eeelay-22
65.9
82.3
82.1
46.9
58.3
95.5
96.9
22.1
antmaze-umaze-vo
50.7
100.0
90.0
88.3
39.1
110.0
110.0
75.8
Average
42.3
93.6
89.1
66.1
51.1
103.8
77.7
75.5
Impact of RTG alignment. Table presents the ablation study RTG alignment during online tuning. Without RTG alignment, VDT performs worse than offline training on certain tasks, indicating that RTG alignment effectively leverages trajectory signals from online interaction by correcting reward guidance. Incorporating RTG alignment consistently improves performance across all tasks. The average score increases from 75.6 to 94.2, indicating substantial performance gain. In particular, benefit significantly, demonstrating
tasks such as salfcheeeah-mediinne that RTG alignment is especially effective when dealing with suboptimal trajectories.
and
29

---

Table 13: Ablation study model components during online tuning. All experiments repeated
three times, and the average value is taken.
Advantage Weighting Regularization hopper-medium-v2 walkerr20-meddium-eepeett-t anlmaze-umaze-v0
104.1
110.2
90.5
107.0
94.3
100.0
108.1
112.7
110.0
Table 12: Ablation study on RTG alignment. Online Tuning-W/o refers to VDT during online tuning without RTG alignment.
Datasets
Offline Training
Online Tuning-w/o
Online Tuning
39.4
36.0
49.2
hopper-meddium-ccelaa-222
96.0
99.4
119.2
walker2ddmmddium--eppaay-2
82.3
85.6
95.5
halfchhettah-meciim--22
43.9
11.7
53.5
hopper-medium-v2
98.3
99.6
108.1
walker22d-mediun--22
81.6
44.0
89.8
halffhhetaa-mmcduunn-eperev.yy
93.9
80.6
101.7
hopper-nediuum-expertt22
111.5
111.0
117.8
walkerr2-mmcdium-eepptt22
110.4
112.3
112.7
Average
84.1
75.6
94.2
Role of different components. As shown in Table 13 we also conduct ablation study advantage weighting and regularization in online fine-tuning for VDT. Since the sampling stage is independent of the training phase, and we have already demonstrated the advantage of value guidance during sampling in Table we do not perform an additional ablation for the sampling process. We observe that the components of VDT exhibit similar effectiveness in the online setting in the offline one. Using either component individually leads to performance improvement while combining both yields the best results. This demonstrates that value-guided methods effective in both online and offline scenarios.
G Broader Impact
The Value-Guided Decision Transformer advances decision-making automation by improving adapt- ability across offline and online settings, offering potential benefits in areas like robotics, logistics, and personalized Al assistance. While designed enhance efficiency and scalability, its adoption invites considerations around balancing human-Al collaboration -uchh as ensuring human oversight critical decisions and avoiding over-reliance on automated systems. By prioritizing transparency in value-guided objectives and fostering partnerships between developers and domain experts, VDT's deployment can support human-centric innovation while addressing practical challenges responsibly.
30


Reviews:
[{"review_id": "GwD1x8pNlk", "reviewer": "Reviewer_yr9t", "summary": "This paper introduces value-function guidance into the decision transformer for trajectory sequence prediction (conditional-sequence-modeling) problems in reinforcement learning. This value-function guidance enables the model to (1) stitch together sub-optimal trajectories during offline training and (2) keep improving rapidly with only a few online interactions. Experiments show that the proposed value-guided decision transformers (VGDT) outperform prior methods in both offline and offline-to-online settings.", "questions": "- Line 153: Why is the expectile loss function an unbiased estimate of the conditional expectation? The IQL [1] paper has already shown that the value function will converge to the maximal Q-function within the support of the dataset.\n\n- Line 160: Why do we prefer the n-step TD loss over the simpler one-step TD loss? \n\n- Eq. 5: Any empirical evidence to show the effect and significance of $\\eta$ and $\\lambda$?\n\n- Line 173 - 175: This sentence is confusing. How does this loss function optimize the stability of trajectory modeling? How is it able to prevent the local optima within the data distribution?\n\n- Line 195 - 196: Why is the two-step sampling procedure uniform? From the sentence, it sounds like the method is prioritizing trajectory segments from longer trajectories. \n\n- Line 199 - 206: In the online learning phase, does the method use the current ongoing trajectory to update the DT?\n\n- The sampling process in Sec. 4.3 is similar to both the cross-entropy method in model-based RL [2] and the Selecting from Behavior Candidates (SfBC) [3] policy extraction method in offline RL. Including some discussions of these two categories of methods in this section could be helpful.\n\n- Line 215: How do we predefine the m candidate RTG? Are they varying across different tasks?\n\n- Fig 3: From this figure, it looks like the effect of the evaluation horizon is minor – the performance for $E = 1$ and $E \\geq 1$ is not significantly different. Then, why do we not prefer a simpler method?\n\n[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline Reinforcement Learning with Implicit Q-Learning.\" In International Conference on Learning Representations.\n\n[2] Chua, Kurtland, Roberto Calandra, Rowan McAllister, and Sergey Levine. \"Deep reinforcement learning in a handful of trials using probabilistic dynamics models.\" Advances in neural information processing systems 31 (2018).\n\n[3] Chen, Huayu, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. \"Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling.\" In The Eleventh International Conference on Learning Representations.", "limitations": "The limitation section only discussed the computation cost of the algorithm. Including some discussions about the theoretical issue or missing properties of the current method could be helpful.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- This paper uses value-function guidance to unify the learning of the decision transformer in both offline and online RL settings. By leveraging the Q-function learned by an n-step implicit Q learning loss, the model is able to use an advantage-weighted regression loss to predict the action sequences, enabling stitching from sub-optimal trajectories and value-guided action selection during inference.\n\n- Through experiments on D4RL benchmarks, the paper shows that VGDT achieves competitive empirical performance, compared to prior offline reinforcement learning methods and other decision transformer variants. Additional results ablate different components of the algorithm and show the effects of them in the complete VGDT.\n\n**Weaknesses**\n\n- Theorem 4.1 seems to state that the optimal decision transformer derived from the proposed method results in *one-step* policy improvement, which does not provide a guarantee to converge to the optimal policy. In practice, there has been success in using *one-step* policy improvement to derive an RL method. But some connections between the one-step value function and the optimal value function will still be helpful. \n\n- For the experiments, ablation studies of some hyperparameters are missing. Also, including some results on benchmarks other than D4RL or tasks with image-based observations can potentially make the experiment more convincing.", "was_revised": true, "final_justification": "The presented theoretical characterization of the gap between one-step and optimal value functions is helpful, the hyperparameter ablations substantiate the proposed components, and the image-based experiments further attest to VDT’s effectiveness. Accordingly, I will raise my rating to 4.", "rebuttal": "Thanks for reviewing our work attentively. We'll answer your questions in the following.\n\n**W1: Connections between the one-step value function and the optimal value function will still be helpful.**  \nWe can indeed provide further theoretical guarantees for the strong empirical performance of VDT. To this end, we propose Theorem 2, which establishes an upper bound for the one-step value function and quantifies its difference from the optimal value function.\n\n**Theorem 2.** For any initial state distribution $\\mu$, we have\n\n$$V^{ \\pi^* }( \\mu )-V^{ \\pi_{DT}^* }( \\mu) \\leq \\frac{ 2 \\gamma }{(1- \\gamma )^2} \\cdot \\mathbb{ E }_{ s \\sim d^{ \\pi^* }}\\left[ \\max _{ a \\notin \\mathcal{ A }_D(s)} Q^{ \\pi^* }(s, a)- \\max _{a \\in  \\operatorname{ supp }( \\beta( \\cdot \\mid s)) } Q^{\\pi^*}(s, a)\\right ]$$\n\nHere, $\\mathcal{A}_D(s)$ denotes the action support of the dataset at state $s$, the set $\\mathrm{supp}(\\beta(\\cdot \\mid s))$ denotes the set of actions observed in the dataset at state $s$. Due to the strict word limit during the rebuttal phase, we will include the detailed proof in the appendix. Theorem 2 complements Theorem 1 by further elucidating the quantitative relationship between VDT and the optimal policy $\\pi^*$. When the dataset has good coverage, our policy is nearly optimal; even with limited coverage, VDT stays conservatively optimal within the supported regions.\n\n**W2&Q3: Empirical evidence of hyperparameterss $\\eta$ and $\\lambda$ and image-based observations.**  \nWe have already conducted an experimental analysis of the range of $\\eta$ in Table 9 of the Appendix E. Additionally, we have included an ablation study on $\\lambda$ in the offline setting. We found that the value of $\\lambda$ does not significantly affect the experimental results (except $\\lambda = 0$). This further demonstrates that VDT is highly robust to the choice of $\\lambda$.\nGame|$\\lambda$=0|$\\lambda$=0.3|$\\lambda$=0.5|$\\lambda$=1\n-|-|-|-|-\nhopper-medium|89.0|96.9|98.3|98.4\nwalker2d-medium-expert|100.9|110.4|110.4|109.5\nantmaze-umaze|84.5|100.0|100.0|98.0\n\nWe evaluated VDT on the Atari dataset with image-based observations, averaging results over three random seeds. Compared to existing methods, VDT achieves superior performance, demonstrating its generalization ability across diverse tasks.\nGame|CQL|DT|DC|VDT\n-|-|-|-|-\nBreakout|211.1±15.2|242.4±31.8|352.7±44.7|420.8±29.4\nQbert|104.2±8.7|28.8±10.3|67.0±14.7|69.4±5.0\nPong|111.9±3.1|105.6±2.9|106.5±2.0|113.9±1.5\nSeaquest|1.7±0.2|2.7±0.4|2.6±0.3|3.9±0.7\nAsterix|4.6±0.5|5.2±1.2|6.5±1.0|8.7±1.1\nFrostbite|9.4±1.0|25.6±2.1|27.8±3.7|28.9±0.6\nGopher|2.8±0.9|34.8±10.0|52.5±9.3|55.3±0.8\n\n**Q1: Why is the expectile loss function an unbiased estimate of the conditional expectation?**  \nWe apologize for our spelling mistake. The expectile loss function is an unbiased estimator of the conditional expectile, not the conditional expectation, except when the expectile parameter $\\tau = 0.5$.\n\n**Q2: Why do we prefer the n-step TD loss over the simpler one-step TD loss?**  \nWe use n-step TD loss instead of one-step TD loss because n-step targets better balance bias and variance, leading to faster and more stable learning, especially with sparse or delayed rewards. Prior work has shown the benefits of n-step methods in offline RL [1][2][3]. Our ablation experiments confirm that n-step generally outperforms one-step, but using too many steps can also have negative effects. This is likely because longer returns amplify errors from distributional shift, offsetting the benefits of n-step learning.\n||1-step|5-step|8-step\n-|-|-|-\nhopper-medium-v2|90.0|98.3|97.6\nwalker2d-medium-expert-v2|87.2|110.4|113.8\nmaze2d-medium|34.9|60.3|60.4\nantmaze-umaze|88.0|100.0|90.0\naverage score|75.0|92.3|90.4\n\n[1] Reinforcement learning: An introduction.\n[2] Fixed-horizon temporal difference methods for stable RL.\n[3] Adaptive temporal-difference learning for policy evaluation with per-state uncertainty estimates.\n\n**Q4: How does this loss function optimize the stability of trajectory modeling and prevent the local optima.**  \nThese two advantages correspond to the two terms in the loss function. The advantage-weighted term promotes imitation of high-value actions, reducing the impact of low-value or outlier actions and improving stability—an idea also discussed in the IQL paper. The regularization term encourages the policy to explore actions with higher Q-values, even if they are rare in the data, helping the policy escape local optima within the data distribution. The results in Tables 1 and 2 empirically verify that VDT can indeed learn optimal actions that surpass most previous methods. Furthermore, the visualisation results in Tables 4 and 8 in the appendix also demonstrate, from the perspectives of convergence time and reward curves, that VDT exhibits a certain degree of competitiveness in terms of stability.\n\n**Q5: Why is the two-step sampling procedure uniform?**  \nThis is because longer trajectories contain more sub-sequences. By sampling trajectories in the first step with probability proportional to their length, and then uniformly selecting a sub-sequence from within each trajectory in the second step, every possible sub-sequence in the entire dataset has an equal probability of being chosen. In other words, the higher chance of picking a long trajectory is balanced out by the fact that there are more sub-sequences to choose from within it, resulting in uniform sampling across all sub-sequences.\n\n**Q6: In the online learning phase, does the method use the current ongoing trajectory to update the DT?**  \nIf \"current ongoing trajectory\" refers to the complete trajectory obtained after the model interacts with the environment once, then answer is yes. After each full episode is collected using the current policy, we perform return-to-go alignment on the trajectory and add it to the replay buffer. The DT is then updated using trajectories sampled from this replay buffer, which now includes the most recent trajectory collected from the environment. Therefore, during the online fine-tuning phase, we follow an interact-sample-update loop, which ensures that the model continually incorporates the latest experiences. The training procedure for the online fine-tuning phase is presented in Algorithms 1 and 2 in Appendix C.\n\n**Q7: Including some discussions of these two categories of methods in this section could be helpful.**  \nWe will add the following content to this section:\n\"Our sampling process is similar to CEM [2] and SfBC [3]. Like CEM, we generate multiple candidate actions at each step, evaluate them with a value function over a short planning horizon, and select the best one—essentially a single-step population-based search. Unlike SfBC, which samples from a behavior policy and selects by Q-value, VDT uses RTGs to guide candidate actions and evaluates them in parallel. This integration of RTG guidance and batch evaluation allows VDT to combine the strengths of population search and candidate selection while maintaining efficient inference.\"\n\n**Q8: How do we predefine the m candidate RTG?**  \nThe candidate RTGs vary across different tasks, and the selection of RTGs is largely consistent with that of the DT. Our main focus is on utilizing candidate RTGs effectively, specifically leveraging the guidance provided by different candidate RTGs. As can be seen, in some tasks, we use only a small number of candidate RTGs, and we did not deliberately increase the number or variety of RTGs (although this could potentially lead to performance improvements). The primary reason for this is to ensure a fair comparison and to minimize the need for manual design. Due to space limitations, we present the candidate RTGs used in some tasks below. A more detailed table will be reintroduced in the main text.\nEnvironment|Targets\n-|-\nhopper|[7200,3600,1800]\nwalker2d|[5000,4000,2500]\npen|[12000,6000]\nhammer|[12000,6000,3000]\ndoor|[2000,1000,500]\nrelocate|[3000,1000]\nmaze2d|[300,200,150,100,50,20]\n\n**Q9: Fig 3 looks like the effect of the evaluation horizon is minor?**  \nSince the evaluation horizon is decoupled from other innovations in VDT, it is indeed possible to omit this component for the sake of simplicity. However, we are still eager to explore opportunities for optimizing the sampling process. Although the performance improvements shown in Figure 3 are not substantial enough to be considered groundbreaking for the field (partly due to the strong baseline performance of the model itself), we believe this direction remains valuable. As mentioned in Q8, we did not carefully select the candidate RTGs. We further expanded the range of candidate RTGs used for hopper-medium, as shown in the table below. Surprisingly, we found that increasing the evaluation horizon leads to different changes in model performance across different RTG ranges. This suggests that effective collaboration between the evaluation horizon and RTG selection can potentially lead to further performance gains. Moreover, due to the parallelism in the evaluation process, we believe that the minor additional computational cost introduced by the evaluation horizon is entirely acceptable.\nHorizon|[7200,3600,1800]|[18000,7200,3600,1800]|[72000,36000,18000,7200,3600,1800,720]\n-|-|-|-\n1|96.8|96.9|94.0\n3|98.3|98.4|100.3\n5|98.3|101.2|104.3\n10|98.6|104.3|104.9\n\n**L1: Including some discussions about the theoretical issue or missing properties could be helpful.**  \nWe have reintroduced the analysis of the gap between the single-step value function used in VDT and the optimal value function, providing an upper bound and further improving the theoretical framework of VDT. In addition, following the valuable suggestions from the reviewers, we have refined many details of existing methods, including ablation studies on components and hyperparameters, literature comparisons, and discussions of potential advantages. All of these improvements will be incorporated into the revised manuscript."}, {"review_id": "dSrJh3sTP8", "reviewer": "Reviewer_x4AC", "summary": "The paper proposes Value-Guided Decision Transformer (VDT), such that it integrates value-based learning into Decision Transformer (DT) by incorporating **advantage-weighting** on behavior cloning (BC) during offline training. Also, VDT proposes to unify offline training with online fine-tuning with trajectory-level replay buffers and return-to-go (RTG) alignment leading to robust performance. During action selection, Q-function based sampling process is used to choose more optimal actions.", "questions": "1. I am curious about the ablation study on how performance improves as we increase the range of RTGs together with Q-value based sampling, e.g. are we able to query higher RTGs than that of in the training dataset?\n    \n2. In online tuning stage, how much the performance depends on the initial offline training? how about ablating the different data ratios? how does VDT perform if the offline dataset has very poor coverage while including high-value trajectories?", "limitations": "- As the authors also discussed, one primary limitation of the proposed method is the computational overhead of the Q-value guided sampling process during inference, especially when scaling to larger-scale, complex environments.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths\n\n1. **Integrate Value-based Methods**: The integration of advantage-weighted learning and Q-function-based sampling is promising and bridging the gap between DT and other offline RL methods.\n    \n2. **Empirical Strength**: VDT achieves on-par or better results across diverse benchmarks, including Adroit, Kitchen, Maze2D, and AntMaze.\n\n\n### Weaknesses\n\n1. **Computational Overhead in Sampling**: Although possibly alleviated by GPU parallelization, the parallel sampling with multiple RTGs and evaluation horizons could become unaffordable for larger scale environments. Further development trading off between optimality and efficiency needs to be addressed.", "was_revised": true, "final_justification": "The further experiments answered my questions regarding:\n* Whether expanding RTG at inference will further enhance the performance: YES\n* Ablations on various data ratios does illustrate the performance differences.", "rebuttal": "Thank you very much for your careful review of our work. We'll answer your questions one by one in the following, including some misunderstandings and some essential academic questions worth exploring.\n\n**W1: Computational Overhead in Sampling**  \nThe trade-off between exploration performance and computational cost is a common challenge faced by all deep learning methods. VDT requires parallel trajectory evaluation during sampling, which introduces some additional computational overhead. However, as shown in our experimental results in Table 4, this overhead is generally manageable due to efficient GPU batching. Compared to other methods, it does not become a bottleneck in terms of actual runtime or computational cost, while providing significant performance improvements. Therefore, we consider this additional overhead to be justified.\n\n**Q1: I am curious about the ablation study on how performance improves as we increase the range of RTGs together with Q-value based sampling, e.g. are we able to query higher RTGs than that of in the training dataset?**  \nIn our original work, we did not attempt to expand the range of RTGs mainly to avoid introducing excessive manual effort and to maintain the simplicity of our method. Intuitively, expanding the RTG range could further improve the algorithm's performance across different tasks. To investigate this, we conducted experiments in the offline setting on hopper-medium-v2, walker-medium-expert-v2, and pen-cloned-v1. In fact, during sampling on hopper-medium-v2, the maximum RTG used for querying the model in VDT already exceeded the maximum RTG in the training set, which is consistent with the usage in DT and aims to ensure fairness. However, we further explored whether expanding the RTG range could bring breakthroughs. For each task, we designed three additional RTG ranges, referred to as Ext. 1, Ext. 2, and Ext. 3.\n\n---\n- **hopper-medium-v2** (max RTG in dataset: 3322)\n    - **Original range:** [7200, 3600, 1800]\n    - **Ext. 1:** [18000, 7200, 3600, 1800]\n    - **Ext. 2:** [72000, 36000, 18000, 7200, 3600, 1800, 720]\n    - **Ext. 3:** [72000, 36000, 18000, 7200, 3600, 1800]\n---\n- **walker-medium-expert-v2** (max RTG: 5011)\n    - **Original range:** [5000, 4000, 2500]\n    - **Ext. 1:** [6000, 5000, 4000, 2500]\n    - **Ext. 2:** [8000, 6000, 5000, 4000, 2500, 1000]\n    - **Ext. 3:** [8000, 6000, 5000, 4000, 2500]\n---\n- **pen-cloned-v1** (max RTG: 11271)\n    - **Original range:** [12000, 6000]\n    - **Ext. 1:** [15000, 12000, 6000]\n    - **Ext. 2:** [200000, 15000, 12000, 6000, 3000]\n    - **Ext. 3:** [200000, 15000, 12000, 6000]\n---\n\nThe purpose of Ext. 1 is to include higher RTGs, while Ext. 2 further increases the upper bound and adds a lowest RTG to provide a lower limit, aiming to maximize the RTG space available to the model without being excessive. Ext. 3 removes the lowest RTG to verify whether it affects sampling performance. To control variables, the evaluation horizon was fixed at 5. The experimental results demonstrate that expanding the RTG range consistently enhances performance, which implies that there is still room for improvement in the strategy used for RTG selection. In addition, the lowest RTG in the range has little impact on sampling performance, suggesting that the model may learn to utilize the largest possible RTG for decision-making without being affected by the lowest value.\nTask|Original Range|Ext. 1|Ext. 2|Ext. 3\n-|-|-|-|-\nhopper-medium-v2|98.3|99.1|101.6|102.0\nwalker-medium-expert-v2|110.4|112.8|112.8|113.1\npen-cloned-v1|145.6|149.5|148.3|149.2\n\nFurthermore, we observe that the RTG range and the evaluation horizon are highly correlated, and together they exhibit a synergistic effect that jointly influences the upper bound of performance during the sampling process. A possible explanation is that increasing the evaluation horizon allows the model to leverage better the guidance provided by RTGs. Conversely, if either the evaluation horizon or the RTG range is insufficiently large, this process may be hindered. We present the relevant experimental results on hopper-medium-v2.\n|Horizon|Original Range|Ext. 1|Ext. 2\n-|-|-|-\n1|96.8|96.9|94.0\n3|98.3|98.4|100.3\n5|98.3|101.2|104.3\n10|98.6|104.3|104.9\n\n**Q2: In online tuning stage, how much the performance depends on the initial offline training? how about ablating the different data ratios? how does VDT perform if the offline dataset has very poor coverage while including high-value trajectories?**  \nWe believe that the impact of offline training on online tuning can be analyzed from aspects such as the quality and quantity of offline data. For data quality, we have included datasets of varying quality within the same task. For example, in the hopper task, hopper-medium-replay represents the lowest quality, containing trajectories from random to medium policies; hopper-medium is of medium quality; and hopper-medium-expert is of the highest quality, consisting of expert policy trajectories. As shown in the experimental results in Table 1, the performance of VDT trained on datasets of different quality is highly correlated with the data quality. \n\nIn the online tuning stage, we fine-tune VDT models that were pre-trained on offline datasets of varying quality. As shown in Table 2 of the main text, we observe that after fine-tuning, the performance of VDT pre-trained on different datasets is further improved, and the overall performance remains linearly correlated with the quality of the offline data utilized. However, in certain scenarios, the VDT pre-trained on the lowest quality dataset (hopper-medium-replay) outperforms those pre-trained on the higher quality datasets (hopper-medium and hopper-medium-expert). We hypothesize that this is mainly because under-trained models may be more sensitive to new gradient signals in some cases, making it easier for online fine-tuning to escape local optima and achieve greater improvements [1].\n\n[1] Levine S, Kumar A, Tucker G, et al. Offline reinforcement learning: Tutorial, review, and perspectives on open problems.\n\nTo investigate the impact of offline data quantity on the final online fine-tuning performance, we further subsampled the task datasets in the Mujoco benchmark by randomly selecting trajectories and retaining only 5%, 10%, and 50% of the original data. Notably, the various Mujoco datasets contain an average of approximately 2,000 trajectories, which is relatively small compared to typical vision and language datasets. After subsampling, we can evaluate the performance of VDT under the most challenging data-scarce scenarios.\nIt is worth noting that subsampled expert-level datasets can effectively represent scenarios with limited data coverage but containing high-value trajectories. This is because the significant reduction in data volume inevitably leads to lower coverage, while the inherent characteristics of expert datasets ensure the presence of sufficiently high-quality trajectories. \n\nAs shown in the experimental results below, we observe that the performance degradation caused by reducing the data quantity is not substantial. When the data is reduced to only 5% of the original amount, the performance on most tasks remains at around 90% of the initial level. Furthermore, since the number of training steps is kept constant in our experiments, it is possible to further improve performance under reduced data conditions by increasing the number of training steps.\nTask|5%|10%|50%|100%\n-|-|-|-|-\nhalfcheetah-medium-v2|49.8|53.6|52.1| 53.5\nhopper-medium-v2|102.9| 107.1|107.9|108.1\nwalker-medium-v2|84.6|88.0|89.9|89.8\nhalfcheetah-medium-expert-v2|80.3|97.6|101.9|101.7\nhopper-medium-expert-v2|109.6|110.0|118.7|117.8\nwalker-medium-expert-v2|107.2|106.0|112.7|112.7\n\nWe also investigated the impact of the number of offline training steps on the final performance of online tuning. In the original experiments, we used 10,000 offline pre-training steps, after which the model had almost converged. Keeping all other hyperparameters unchanged, we varied the number of offline pre-training steps to 1,000, 5,000, and 10,000 to simulate VDT models that had not fully converged during pre-training. The experimental results are shown in the table below. It can be seen that the effect of the number of offline training steps on final performance is highly similar to the effect of insufficient data on final performance. Both can be attributed to underfitting scenarios. As the number of offline pre-training steps increases, there is still room for performance improvement, which is consistent with intuition.\nTask|1000|5000|10000\n-|-|-|-\nhalfcheetah-medium-v2|40.3|51.9|53.5\nhopper-medium-v2|88.6|100.2|108.1\nwalker-medium-v2|33.1|76.0|89.8\nhalfcheetah-medium-expert-v2|79.9|88.3|101.7\nhopper-medium-expert-v2|65.6|99.0|117.8\nwalker-medium-expert-v2|48.0|101.5|112.7\n\n**L1: The proposed method is the computational overhead of the Q-value guided sampling process during inference.**  \nIn Table 4 of the main text, we compare the VDT algorithm with other methods. We observe that the average computational cost of VDT remains lower than that of the commonly used ODT, and this overhead is relatively manageable. When scaling VDT to large and complex environments, the cost can be further mitigated by adopting more efficient batching strategies or model pruning techniques. As discussed in Q1, compared to the potential performance gains brought by VDT through the integration of value functions and parallelized sampling, this additional overhead may be relatively acceptable. More importantly, our approach is currently the only method that demonstrates a significant performance advantage in both offline and online settings. We hope that our work can also provide the offline RL community with some inspiration for designing more general-purpose RL algorithms from a new perspective."}, {"review_id": "FTNM3EwBh4", "reviewer": "Reviewer_a7nS", "summary": "The paper extends the Decision Transformer (DT) architecture by integrating value-based guidance into it. The approach is applicable both for offline learning and for offline->online setups (where one first trains offline, then fine-tunes). In the offline learning part, the approach uses value functions (coupled with an advantage calculation) to guide the DT. In the online phase, the approach uses a replay buffer and the value function to fine-tune the approach. The evaluation shows improvement over the Online DT baseline, as well as other approaches.", "questions": "The authors are welcome to address my comments above.", "limitations": "YEs", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1) Advantage-Weighted Learning and regularization - while this approach is not the first to incorporate value functions into DT (e.g., CGDT, which is a baseline), the use of advantage functions and learning is a novelty.\n2) Effective offline-to-onilne setup - the proposed process has novel aspects. In addition, it seems to be (relatively) efficient and effective.\n3) Robust evaluation: the appraoch is evaluated on a large number of baselines.\n\nWeaknesses:\n1) Evaluation - while the evaluation consists of multiple baselines, the authors only include the standard deviation for their approach. There are several \"close calls\" where seeing the standard deviation of the baselines would have been very helpful in assessing the contribution of the approach.\n2) Relevant baselines - while CGDT is a very relevant baseline, a comparison to additional DT-based baselines, especially ones that are known to be good at stitching (Elastic DT is a strong candidate in my opinion, also because it also evaluated multiple options) could be helpful. Such a comparison would enable the readers to understand how the appraoch stands in comparison to other enhancements of DT.", "was_revised": true, "final_justification": "The authors' response addressed most of my concerns. I have updated the score.", "rebuttal": "Thank you very much for your careful review of our work. We'll answer your questions one by one in the following, including some misunderstandings and some essential academic questions worth exploring.\n\n**W1: Seeing the standard deviation of the baselines would have been very helpful in assessing the contribution of the approach.**  \nWe sincerely thank the reviewers for their suggestions. We agree that standard deviations are important for reflecting the stability of each method and can further highlight the advantages of VDT over other approaches. In response, we have now added the standard deviations for all methods in Tables 1 and 2. Some of these values are directly taken from the original papers. For cases where the original papers did not report standard deviations, we reproduced the results using the available open-source code. We calculated the average standard deviation over the same three random seeds as VDT (123, 321, 132). The updated Tables 1 and 2 with standard deviations are shown below. As can be seen, VDT demonstrates better stability compared to most baseline methods, which further illustrates the advantages of our approach. Moreover, as the only method that achieves optimal performance in both offline and online settings, we believe that the VDT pipeline can also provide valuable insights for the offline RL community.\n## Offline\nGym Tasks|BEAR|BCQ|CQL|IQL|MoRel|BC|DT|StAR|GDT|CGDT|DC|VDT\n-|-|-|-|-|-|-|-|-|-|-|-|-\nhalfcheetah-medium-replay-v2|38.6±2.1|34.8±1.7|37.5±0.3|**44.1**±0.5|40.2±2.0|36.6±2.33|36.6±1.6|36.8±1.8|40.5±2.2|40.4±2.0|41.3±3.8|39.4±2.0\nhopper-medium-replay-v2|33.7±2.0|31.1±1.6|95.0±5.2|92.1±8.1|93.6±4.7|18.1±2.07|82.7±3.9|29.2±1.5|85.3±4.3|93.4±4.7|94.2±5.9|**96.0**±1.9\nwalker2d-medium-replay-v2|19.2±1.2|13.7±0.8|77.2±13.2|73.7±7.0|49.8±3.2|32.3±10.15|79.4±11.2|39.8±2.0|77.5±5.4|78.1±5.6|76.6±1.5|**82.3**±2.1\nhalfcheetah-medium-v2|41.7±2.1|41.5±2.1|44.0±0.2|**47.4**±0.3|42.1±2.1|42.6±0.19|42.6±2.2|42.9±2.1|42.9±2.1|43.0±2.2|43.0±5.6|43.9±0.7\nhopper-medium-v2|52.1±2.6|65.1±3.3|58.5±3.77|63.8±5.7|95.4±4.8|52.9±1.76|67.6±2.2|59.5±3.0|77.1±3.9|96.9±4.8|92.5±0.9|**98.3**±0.1\nwalker2d-medium-v2|59.1±3.0|52.0±2.6|72.5±3.28|79.9±1.8|77.8±3.9|75.3±16.24|74.0±1.2|73.8±3.7|76.5±3.8|79.1±4.0|79.2±0.6|**81.6**±1.7\nhalfcheetah-medium-expert-v2|53.4±2.7|69.6±3.5|91.6±0.42|86.7±2.5|53.3±2.7|55.2±7.35|86.8±1.4|93.7±4.7|93.2±4.7|93.6±4.7|93.0±2.4|**93.9**±0.1\nhopper-medium-expert-v2|96.3±4.8|109.1±5.5|105.4±10.91|91.5±10.0|108.7±5.4|52.5±4.01|107.6±1.9|111.1±5.6|111.1±5.6|107.6±5.4|110.4±6.0|**111.5**±3.8\nwalker2d-medium-expert-v2|40.1±2.0|67.3±3.4|108.8±0.39|109.6±0.8|95.6±4.8|107.5±15.98|108.1±0.3|109.0±5.5|107.7±5.4|109.3±5.5|109.6±1.1|**110.4**±0.9\n\nAdroit Tasks|BEAR|BCQ|CQL|IQL|MoRel|EDAC|BC|DT|D-QL|StAR|GDT|VDT\n-|-|-|-|-|-|-|-|-|-|-|-|-\npen-human-v1|-1.0±0.2|66.9±6.7|37.5±3.8|71.5±28.5|-3.2±0.3|52.1±5.2|63.9±6.4|79.5±2.1|72.8±7.3|77.9±7.8|92.5±3.6|**126.7**±4.3\nhammer-human-v1|2.7±0.3|0.9±0.1|4.4±0.4|1.4±0.2|2.3±0.2|0.8±0.1|1.2±0.1|3.7±0.4|0.2±0.02|3.7±0.4|**5.5**±0.9|3.2±0.3\ndoor-human-v1|2.2±0.2|-0.05±0.01|9.9±1.0|4.3±0.4|2.3±0.2|10.7±1.1|2.0±0.2|14.8±1.5|0.0±0.0|1.5±0.2|18.6±2.9|**19.7**±0.5\npen-cloned-v1|-0.2±0.02|50.9±5.1|39.2±3.9|37.3±29.9|-0.2±0.02|68.2±6.8|37.0±3.7|75.8±3.3|57.3±5.7|33.1±3.3|86.2±2.5|**145.6**±4.0\nhammer-cloned-v1|2.3±0.2|0.4±0.04|2.1±0.2|2.1±0.2|2.3±0.2|0.3±0.03|0.6±0.06|3.0±0.3|3.1±0.3|0.3±0.03|8.9±1.5|**19.6**±1.6\ndoor-cloned-v1|2.3±0.2|0.01±0.001|0.4±0.04|1.6±0.2|2.3±0.2|9.6±1.0|0.0±0.0|16.3±1.6|0.0±0.0|0.0±0.0|19.8±1.7|**30.6**±0.7\n\nKitchen Tasks|BEAR|BCQ|CQL|IQL|O-RL|BC|DT|DD|StAR|GDT|DC|VDT\n-|-|-|-|-|-|-|-|-|-|-|-|-\nkitchen-complete-v0|0.0±0.0|8.1±0.4|43.8±2.2|62.5±3.1|2.0±0.1|65.0±3.3|50.8±2.5|65.0±3.3|40.8±2.0|43.8±2.2|40.9±0.3|**65.9**±0.2\nkitchen-partial-v0|13.1±0.7|18.9±0.9|49.8±2.5|46.3±2.3|35.5±1.8|33.8±1.7|57.9±2.9|57.0±2.9|12.3±0.6|73.3±3.7|66.8±5.9|**76.1**±10.8\n\nMaze2D Tasks|BEAR|BCQ|CQL|IQL|COMBO|BC|MPPI|DT|QDT|GDT|DC|VDT\n-|-|-|-|-|-|-|-|-|-|-|-|-\nmaze2d-umaze-v1|65.7±3.3|49.1±2.5|86.7±4.3|42.1±2.1|76.4±3.8|85.7±4.3|33.2±1.7|31.0±1.6|57.3±2.9|50.4±2.5|20.1±6.9|**88.0**±4.6\nmaze2d-medium-v1|25.0±1.3|17.1±0.9|41.8±2.1|34.9±1.7|38.5±1.9|38.3±1.9|10.2±0.5|8.2±0.4|13.3±0.7|7.8±0.4|38.2±2.4|**60.3**±0.5\n\nAntMaze Tasks|BEAR|BCQ|CQL|IQL|O-RL|BC|DT|RvS|StAR|GDT|DC|VDT\n-|-|-|-|-|-|-|-|-|-|-|-|-\nantmaze-umaze-v0|73.0±3.7|78.9±4.0|74.0±3.7|87.1±4.4|64.3±3.2|54.6±2.7|59.2±3.0|65.4±3.3|51.3±2.6|76.0±3.8|85.0±4.9|**100.0**±5.5\nantmaze-umaze-diverse-v0|61.0±3.1|55.0±2.8|84.0±4.2|64.4±3.2|60.7±3.0|45.6±2.3|66.2±3.3|60.9±3.0|45.6±2.3|69.0±3.5|78.5±9.7|**100.0**±4.7\nantmaze-medium-diverse-v0|8.0±0.4|0.0±0.0|53.7±2.7|**70.0**±3.5|0.0±0.0|0.0±0.0|7.5±0.4|67.3±3.4|0.0±0.0|0.0±0.0|0.0±0.8|30.0±2.8\n\n## Online\nDataset|TD3+BC|AWAC|CQL|IQL|PDT|ODT|VDT\n-|-|-|-|-|-|-|-\nhalfcheetah-medium-replay-v2|48.1±1.2|39.0±2.1|44.3±1.5|44.0±0.3|42.8±1.0|40.4±1.6|**49.2**±1.3\nhopper-medium-replay-v2|90.7±3.5|79.6±4.2|95.3±2.8|93.5±4.4|94.8±3.1|88.9±6.3|**119.2**±5.7\nwalker2d-medium-replay-v2|82.0±2.7|44.0±3.8|78.0±6.1|60.9±5.8|79.0±2.5|76.9±4.0|**95.5**±3.9\nhalfcheetah-medium-v2|50.9±1.0|41.1±0.7|29.1±6.2|48.0±0.2|**69.5**±2.0|42.2±1.5|53.5±1.8\nhopper-medium-v2|64.6±2.2|91.0±3.1|95.7±2.5|44.3±4.1|100.2±2.7|97.5±2.1|**108.1**±4.2\nwalker2d-medium-v2|85.2±5.0|79.1±8.8|89.4±2.3|68.9±2.3|88.1±2.6|76.8±8.3|**89.8**±2.7\nhalfcheetah-medium-expert-v2|92.1±1.5|41.0±2.0|99.9±1.7|95.3±1.2|93.3±1.8|100.9±1.6|**101.7**±1.9\nhopper-medium-expert-v2|110.2±2.8|111.9±3.0|106.3±2.5|92.9±2.1|80.0±2.7|99.1±2.2|**117.8**±3.5\nwalker2d-medium-expert-v2|110.1±7.6|78.3±7.9|110.1±9.4|109.6±0.2|108.9±2.5|78.7±2.8|**112.7**±2.9\nantmaze-umaze-v0|79.1±7.9|59.0±4.7|99.4±8.1|96.0±5.4|66.8±4.0|88.5±5.9|**110.0**±4.2\nantmaze-umaze-diverse-v0|78.1±3.9|49.0±3.2|99.4±7.2|84.0±6.4|79.3±3.7|56.0±5.7|**100.0**±5.9\nantmaze-medium-diverse-v0|56.7±2.8|0.3±0.1|32.3±4.0|72.0±8.9|63.4±3.5|55.6±6.2|**75.0**±2.7\n\n**W2: Relevant baselines**  \nThe core idea behind Elastic DT's ability to stitch trajectories lies in adaptive history truncation. During inference, EDT dynamically selects the optimal history length to maximize the maximum achievable return from the current state. Specifically, EDT estimates the maximum return for different history lengths using expected quantile regression and then chooses the history length to feed into the Transformer accordingly. EDT does not use a Q-function to guide action selection; instead, it essentially \"forgets\" unsuccessful histories, allowing the model to escape from low-return trajectories and stitch onto more optimal trajectory branches. We will further add a comparison of Elastic DT in both the methods and experiments sections of the main text. In addition, we have included a comparison with QDT [2], which combines the dynamic programming capabilities of Q-learning with the sequence modeling strengths of DT, enabling relabeling of return targets and thus improving DT's ability to stitch suboptimal trajectories.\n\nVDT differs in that it explicitly introduces a Q-function to provide value-guided weighting for action selection. At the same time, it employs multi-step Bellman equations and double Q-networks to stabilize value function training, thereby extending VDT to be applicable in both offline and online settings, and enabling parallel decision-making during sampling to leverage the evaluation capability of the Q-function fully. In the experimental section, as shown in the table below, we present a comparison of VDT with EDT and QCS. VDT demonstrates a clear advantage over EDT and QCS across a wide range of tasks.\n\n||EDT|QCS|VDT\n-|-|-|-\nhalfcheetah-medium|42.5|42.3|43.9\nhopper-medium|63.5|66.5|98.3\nwalker2d-medium|72.8|67.1|81.6\nhalfcheetah-medium-replay|37.8|35.6|39.4\nhopper-medium-replay|89.0|52.1|96.0\nwalker2d-medium-replay|74.8|58.2|82.3\naverage score|63.4|53.6|73.6\n\n[1] Elastic decision transformer, NeurIPS 2023.  \n[2] Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. PMLR 2023."}, {"review_id": "YHbdItx5tc", "reviewer": "Reviewer_8M1K", "summary": "This paper proposes Value-Guided Decision Transformer (VDT), which leverages value functions for offline-to-online finetuning for decision transformers. In the offline stage, a double MLP critic network with TD(n) IQL loss is trained, and the decision transformer is trained via advantage-weighted supervised loss with a Q-value regularizer. For the online stage, the training process is almost the same, except that the replay buffer is trajectory-based, and multiple active trajectories with different RTGs are maintained. On a variety of environments, the proposed method outperforms several baselines.", "questions": "I have one question: In Tab. 4, the authors compared the efficiency of their method with IQL. I am a bit curious about the settings of IQL here: why is IQL taking comparable amount of parameters to decision transformer-based method and is much slower during online finetuning? My expectation is that IQL should be a lot faster as it does not need to deal with the heavy (compared to MLP similar to your critic) decision transformer architecture.", "limitations": "Yes (though I would say a limitation is lack of evaluation on image-based environments; I do not feel this to be a ground for rejection but it is a limitation).", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper is well-written and very easy to follow; the design of value-augmented decision transformer loss and online stage designs are clearly conveyed. It is also very well-rounded in the sense that it provides detailed hyperparameters, discussions on broader impact, environment specifications and pseudocode that helps the understanding of VDT.\n\n2. The idea is intuitive: adding RL gradients for supervised learning loss, and is supported by theoretical analysis (Thm. 4.1). It is also very interesting to see the idea of sampling from different RTGs with parallel trajectories which tries to address a crucial concern of decision transformers that it could be sensitive to different RTGs.\n\n3. The experiment results seems quite solid. Authors compare their method to a large number of baselines on a variety of environments, which I feel sufficient for proving the effectiveness of the proposed method. There are also extensive ablations on computational complexity (Tab. 4), algorithm components (Tab. 3) and hyperparameters such as context length, architecture, $\\eta$ and $\\epsilon$ (Fig. 3, Tab. 5, Tab. 8, Tab. 9).\n\n**Weaknesses**\n\n1. While the idea of sampling parallel trajectories is very interesting, it limits the proposed method largely within simulation environments. The paper does not seem to maintain a standalone dynamic / world model to predict future states, nor is there any loss term on predicting states or state prediction output by the decision transformer itself. In this case, when you \"predict\" next state from \"EnvModel\" (Alg. 3), you are generating a sequence of $E$-step rollout for each of the $m$ candidate RTGs, and always cancel your actions afterwards until the optimal action is selected. This is a major limitation in real-life scenarios; for example, if you cook a steak, there is no way to \"uncook\" it.\n\n2. It seems that the implementation of \"alignment of RTGs\" is not explained in Sec. 4.2 where it is proposed. More specifically, how is the RTG token in the last timestep \"matching\" the reward obtained by the agent? Ideally the RTG in the last step should be $0$, which should not match the reward obtained by the agent. The mechanism seems somehow explained in Sec. 4.3 and its related content in the appendix; it would be better if the authors can make this clearer when it is introduced.\n\n3. There are some literature in online finetuning decision transformers missing. For example, Yan et al. [1] conducts offline-to-online RL with online decision transformer + TD3; the idea of attaching RL gradients from Q-value to the training objective and trajectory-level replay buffer in its implementation are similar to this paper, and thus is more relevant to this paper than ODT. Another related work that uses RL values for offline decision transformer is Wang et al. [2].\n\n4. The proposed method seems to be sensitive with respect to $\\epsilon$.\n\n**Minor Weaknesses** \n\n1. On page 25 in the appendix, a space is missing between . and $\\eta=3$.\n\n2. in Fig. 1, the subscripts are out of the circle and almost overlap with other parts (e.g. $s_{t-K+1}$ and $a_{t-K+1}$).\n\n**References**\n\n[1] K. Yan et al. Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers. In NeurIPS, 2024.\n\n[2] Y. Wang et al. Critic-Guided Decision Transformer for Offline Reinforcement Learning. In AAAI, 2024.", "was_revised": true, "final_justification": "The authors addressed my concerns with ample experiment results, which exceeds my expectation. Therefore, I am willing to increase my score to 5.", "rebuttal": "Thank you very much for your careful review of our work. We'll answer your questions one by one in the following, including some misunderstandings and some essential academic questions worth exploring. \n\n**W1: The paper does not seem to maintain a standalone dynamic model to predict future states. In this case, when you \"predict\" next state from \"EnvModel\" (Alg. 3), you are generating a sequence of $E$-step rollout for each of the candidate RTGs, and always cancel your actions afterwards until the optimal action is selected.**  \nWe must acknowledge that the approach of sampling parallel trajectories is difficult to implement in scenarios where future states are completely unpredictable. In such cases, we can only rely on a \"single-shot\" sampling strategy, i.e., single-step sampling with Evaluation Horizon = 1. As shown in the table below, we find that VDT still maintains a advantage over other methods, which is intuitive: the advantage-weighting and penalty mechanisms provided by the value function can further enhance the model's decision-making ability, even when multi-step parallel sampling is not possible. \n||DT|DC|VDT (Evaluation Horizon=1)\n-|-|-|-\nhalfcheetah-medium|42.6 |43.0|42.9\nhopper-medium|67.6|92.5|96.8\nwalker2d-medium|74.0|79.2|80.0\nhalfcheetah-medium-expert|86.8|93.0|93.6\nhopper-medium-expert|107.6|110.4|111.2\naverage score|75.7|83.6|84.9\n\nOn the other hand, although most tasks have or can be trained with a simulation environment, it is indeed important to consider our sampling methods in extreme scenarios where no explicit simulator is available. One intuitive approach is to directly add an environment prediction head to VDT, enabling the model to perform parallel sampling during inference autonomously. Another approach is to train a world model to provide future states for VDT, which has been explored in many previous works and is relatively feasible [1][2]. In general, training a dynamic model to predict states and rewards is often slightly easier than training a decision model. If we do proceed in this way, it could be argued that, to some extent, we are essentially equipping the current task with a simulation environment based on the existing data. Due to the limited time for rebuttal, we regret that we are unable to explore these two designs now. We will include experiments and discussions on both methods in the appendix of the revised version.\n\n[1] Ha D, Schmidhuber J. Recurrent world models facilitate policy evolution. NeurIPS, 2018.  \n[2] Hafner D, Pasukonis J, Ba J, et al. Mastering diverse control tasks through world models. Nature, 2025.\n\n**W2: The implementation of \"alignment of RTGs\" is not explained. More specifically, how is the RTG token in the last timestep \"matching\" the reward obtained by the agent? Ideally the RTG in the last step should be 0, which should not match the reward obtained by the agent.**  \nIn line 5 of Algorithm 2 in Appendix C, we illustrate the alignment of RTGs, where the RTG at time step $t$ for the current trajectory is calculated as: $RTG_t = \\sum_{j=t}^{|\\tau|} r_j$, with $|\\tau|$ denoting the trajectory length. During trajectory sampling, a trajectory may terminate either due to reaching the interaction limit (1000 steps) or by actually achieving the task goal. Therefore, the reward at the final step (i.e., the reward upon trajectory termination) is more accurately the immediate reward returned by the environment at that step, which may or may not be zero. Even if the goal is achieved, the immediate reward is not necessarily zero. Thus, in the main text, we refer to the RTG of the last token as the reward obtained by the model at the current step. For example, in the maze task, the reward for the final step after reaching the goal is +1. We will further clarify the details regarding the alignment of RTGs in the main text to avoid any ambiguity.\n\n**W3: There are some literature in decision transformers missing[1][2].**  \nWe have already cited and compared CGDT [2] in the experimental section of the main text. CGDT, designed for offline reinforcement learning, also adopts the DT loss as its foundation, but further introduces the value estimation from an asymmetrically pre-trained Critic as a penalty term, encouraging the policy to select optimistic actions that the Critic evaluates as exceeding the target return. TD3+ODT [1], which largely follows the ODT algorithm pipeline, additionally incorporates the RL gradient from the Critic as a penalty term in the loss. Both [1] and [2] introduce gradient information as penalty terms for offline or online fine-tuning in reinforcement learning, but are only applicable to a single setting. In contrast, VDT leverages advantage-weighted and penalty losses, together with parallel sampling techniques, and is the first to achieve competitive performance in both offline and online fine-tuning scenarios. We will further elaborate on the comparison between VDT and [1][2] in the related work section. In addition, the table below presents a performance comparison of the methods, where it can be seen that VDT consistently achieves strong results across different settings.\noffline|CGDT|VDT\n-|-|-\nhalfcheetah-medium|43.0|43.9\nhopper-medium|96.9|98.3\nwalker2d-medium|79.1|81.6\nhalfcheetah-medium-expert|93.6|93.9\nhopper-medium-expert|107.6|111.5\nwalker2d-medium-expert|109.3|110.4\naverage score|88.2|89.9\n\nonline|TD3+ODT|VDT\n-|-|-\nhalfcheetah-medium|66.9|53.5\nhopper-medium|89.1|108.1\nwalker2d-medium|90.8|89.8\nhalfcheetah-medium-expert|99.8|101.7\nhopper-medium-expert|112.6|117.8\nwalker2d-medium-expert|111.9|112.7\naverage score|95.1|97.3\n\n**W4: The proposed method seems to be sensitive with respect to $\\epsilon$.**  \nIn Figure 2, we tested four values of $\\epsilon$: 0.3, 0.5, 0.7, and 0.9. When $\\epsilon$ is greater than 0.5, the value function tends to optimistically estimate the value of actions, while for $\\epsilon$ less than 0.5, the value function tends to be pessimistic. Since the range of values used in our ablation study was relatively narrow, it was not easy to fully assess the model's sensitivity to $\\epsilon$. Therefore, we further extended the experiments in Figure 2 by expanding the range of $\\epsilon$ to [0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]. The table below shows the results of varying $\\epsilon$ on both offline and online experiments. We found that the model's performance is relatively stable and optimal when $\\epsilon$ is in the range of approximately 0.6 to 0.8. This observation is also generally consistent with the findings in the original IQL paper. Beyond a certain range, performance drops rapidly, which is related to the inherent properties of the value function. When $\\epsilon$ is less than 0.5, the value function tends to underestimate values, making it difficult to effectively guide the model to achieve performance beyond the expert data. The same issue also arises when $\\epsilon$ is set too high.\nEnv|$\\epsilon$|Offline|Online\n-|-|-|-\nhopper-medium-v2|0.2|43.8|83.6\n||0.3|34.6|91.6\n||0.4|32.0|85.0\n||0.5|35.3|80.8\n||0.6|75.8|105.0\n||0.7|98.3|108.1\n||0.8|96.0|106.0\n||0.9|84.7|40.6\n|walker2d-medium-replay-v2|0.2|0.3|8.9\n||0.3|0.5|18.9\n||0.4|0.4|12.0\n||0.5|5.6|10.6\n||0.6|65.3|88.9\n||0.7|82.3|95.5\n||0.8|81.0|92.0\n||0.9|19.7|79.4\n\n**MW1: On page 25, a space is missing.**  \nWe apologize for this typesetting error. We have reinserted the missing space and updated it in the subsequent version.\n\n**MW2: the subscripts are out of the circle and almost overlap with other parts in Fig1**  \nWe have updated Figure 1 by enlarging the circles to accommodate the subscripts and slightly adjusting the layout of longer subscripts to prevent the circles from becoming excessively large. As images cannot be directly displayed during the rebuttal period, we will present the updated figure in the main text in subsequent versions.\n\n**Q1: why is IQL taking comparable amount of parameters to decision transformer-based method and is much slower during online finetuning?**  \nWe apologize for this potentially confusing point. To evaluate performance, we replaced the policy network in the IQL algorithm with a transformer architecture whose prediction head outputs probabilities. This led to a rapid increase in the number of parameters and a noticeable decrease in convergence speed, although the performance remained comparable to the original IQL. We aimed to demonstrate that simply replacing the policy architecture in IQL with a transformer does not achieve the same effect as VDT. While the original IQL algorithm has a lower computational cost compared to VDT, this inevitably results in a loss of performance. To avoid such misunderstandings, we will revise the relevant table to include the original IQL algorithm with an MLP architecture, as shown in the table below.\n||Memory↓|Params↑|ClockTime↓|Memory↓|Params↑|ClockTime↓\n-|-|-|-|-|-|-\n||**Offline Training**|||**Online Tuning**||\nOriginal IQL|960M|0.6M|≈1.0h|960M|0.6M|≈1.0h\nIQL (Transformer policy)|2128M|3.31M|≈6.0h|2128M|3.31M|≈10.0h\nODT|3968M|5.01M|≈9.0h|3968M|5.01M|≈4.5h\nVDT|4024M|5.24M|≈5.0h|4024M|5.24M|≈4.0h\n\n**L1: lack of evaluation on image-based environments**  \nWe thank the reviewers for this suggestion. We have evaluated the performance of VDT on the image-based Atari dataset, reporting the average performance across three random seeds, as shown in the table below. VDT also demonstrates competitive performance on the Atari dataset, indicating that our method is well-suited for both text and image-based datasets. This success was expected, given the precedents set by DT.\nGame|CQL|DT|DC|VDT\n-|-|-|-|-\nBreakout|211.1±15.2|242.4±31.8|352.7±44.7|420.8±29.4\nQbert|104.2±8.7|28.8±10.3|67.0±14.7|69.4±5.0\nPong|111.9±3.1|105.6±2.9|106.5±2.0|113.9±1.5\nSeaquest|1.7±0.2|2.7±0.4|2.6±0.3|3.9±0.7\nAsterix|4.6±0.5|5.2±1.2|6.5±1.0|8.7±1.1\nFrostbite|9.4±1.0|25.6±2.1|27.8±3.7|28.9±0.6\nGopher|2.8±0.9|34.8±10.0|52.5±9.3|55.3±0.8"}]

Author rebuttal:

Meta-review: This paper incorporates guidance from value functions into the decision transformer framework for offline learning, and for online tuning of offline-trained policies.  In empirical evaluations, the approach is competitive (either matching or outperforming) a large variety of baselines.  All of the reviewers found the experiments to be both convincing and extensive; one reviewer did suggest an additional baseline (Elastic DT), which the authors will include in the final revision.

Overall, the technique appears to be both well motivated and empirically well performing; I think it is likely to have significant impact.
Decision: Accept (poster)