Title: Value-Guided Decision Transformer: A Unified Reinforcement Learning Framework for Online and Offline Settings
Venue: NeurIPS 2025
Accepted: yes
Keywords: Offline RL

Conditional Sequence Modeling

Decision Transformer
Abstract: The Conditional Sequence Modeling (CSM) paradigm, benefiting from the transformer's powerful distribution modeling capabilities, has demonstrated considerable promise in Reinforcement Learning (RL) tasks. However, much of the work has focused on applying CSM to single online or offline settings, with the general architecture rarely explored. Additionally, existing methods primarily focus on deterministic trajectory modeling, overlooking the randomness of state transitions and the diversity of future trajectory distributions. Fortunately, value-based methods offer a viable solution for CSM, further bridging the potential gap between offline and online RL. In this paper, we propose Value-Guided Decision Transformer (VDT), which leverages value functions to perform advantage-weighting and behavior regularization on the Decision Transformer (DT), guiding the policy toward upper-bound optimal decisions during the offline training phase. In the online tuning phase, VDT further integrates value-based policy improvement with behavior cloning under the CSM architecture through limited interaction and data collection, achieving performance improvement within minimal timesteps. The predictive capability of value functions for future returns is also incorporated into the sampling process. Our method achieves competitive performance on various standard RL benchmarks, providing a feasible solution for developing CSM architectures in general scenarios. Code is available at here.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Value-Guided Decision Transformer (VDT), which leverages value functions for offline-to-online finetuning for decision transformers. In the offline stage, a double MLP critic network with TD(n) IQL loss is trained, and the decision transformer is trained via advantage-weighted supervised loss with a Q-value regularizer. For the online stage, the training process is almost the same, except that the replay buffer is trajectory-based, and multiple active trajectories with different RTGs are maintained. On a variety of environments, the proposed method outperforms several baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The paper is well-written and very easy to follow; the design of value-augmented decision transformer loss and online stage designs are clearly conveyed. It is also very well-rounded in the sense that it provides detailed hyperparameters, discussions on broader impact, environment specifications and pseudocode that helps the understanding of VDT.\n\n2. The idea is intuitive: adding RL gradients for supervised learning loss, and is supported by theoretical analysis (Thm. 4.1). It is also very interesting to see the idea of sampling from different RTGs with parallel trajectories which tries to address a crucial concern of decision transformers that it could be sensitive to different RTGs.\n\n3. The experiment results seems quite solid. Authors compare their method to a large number of baselines on a variety of environments, which I feel sufficient for proving the effectiveness of the proposed method. There are also extensive ablations on computational complexity (Tab. 4), algorithm components (Tab. 3) and hyperparameters such as context length, architecture, $\\eta$ and $\\epsilon$ (Fig. 3, Tab. 5, Tab. 8, Tab. 9).\n\n**Weaknesses**\n\n1. While the idea of sampling parallel trajectories is very interesting, it limits the proposed method largely within simulation environments. The paper does not seem to maintain a standalone dynamic / world model to predict future states, nor is there any loss term on predicting states or state prediction output by the decision transformer itself. In this case, when you \"predict\" next state from \"EnvModel\" (Alg. 3), you are generating a sequence of $E$-step rollout for each of the $m$ candidate RTGs, and always cancel your actions afterwards until the optimal action is selected. This is a major limitation in real-life scenarios; for example, if you cook a steak, there is no way to \"uncook\" it.\n\n2. It seems that the implementation of \"alignment of RTGs\" is not explained in Sec. 4.2 where it is proposed. More specifically, how is the RTG token in the last timestep \"matching\" the reward obtained by the agent? Ideally the RTG in the last step should be $0$, which should not match the reward obtained by the agent. The mechanism seems somehow explained in Sec. 4.3 and its related content in the appendix; it would be better if the authors can make this clearer when it is introduced.\n\n3. There are some literature in online finetuning decision transformers missing. For example, Yan et al. [1] conducts offline-to-online RL with online decision transformer + TD3; the idea of attaching RL gradients from Q-value to the training objective and trajectory-level replay buffer in its implementation are similar to this paper, and thus is more relevant to this paper than ODT. Another related work that uses RL values for offline decision transformer is Wang et al. [2].\n\n4. The proposed method seems to be sensitive with respect to $\\epsilon$.\n\n**Minor Weaknesses** \n\n1. On page 25 in the appendix, a space is missing between . and $\\eta=3$.\n\n2. in Fig. 1, the subscripts are out of the circle and almost overlap with other parts (e.g. $s_{t-K+1}$ and $a_{t-K+1}$).\n\n**References**\n\n[1] K. Yan et al. Reinforcement Learning Gradients as Vitamin for Online Finetuning Decision Transformers. In NeurIPS, 2024.\n\n[2] Y. Wang et al. Critic-Guided Decision Transformer for Offline Reinforcement Learning. In AAAI, 2024.\n\nQUESTIONS:\nI have one question: In Tab. 4, the authors compared the efficiency of their method with IQL. I am a bit curious about the settings of IQL here: why is IQL taking comparable amount of parameters to decision transformer-based method and is much slower during online finetuning? My expectation is that IQL should be a lot faster as it does not need to deal with the heavy (compared to MLP similar to your critic) decision transformer architecture.\n\nLIMITATIONS:\nYes (though I would say a limitation is lack of evaluation on image-based environments; I do not feel this to be a ground for rejection but it is a limitation).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper extends the Decision Transformer (DT) architecture by integrating value-based guidance into it. The approach is applicable both for offline learning and for offline->online setups (where one first trains offline, then fine-tunes). In the offline learning part, the approach uses value functions (coupled with an advantage calculation) to guide the DT. In the online phase, the approach uses a replay buffer and the value function to fine-tune the approach. The evaluation shows improvement over the Online DT baseline, as well as other approaches.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) Advantage-Weighted Learning and regularization - while this approach is not the first to incorporate value functions into DT (e.g., CGDT, which is a baseline), the use of advantage functions and learning is a novelty.\n2) Effective offline-to-onilne setup - the proposed process has novel aspects. In addition, it seems to be (relatively) efficient and effective.\n3) Robust evaluation: the appraoch is evaluated on a large number of baselines.\n\nWeaknesses:\n1) Evaluation - while the evaluation consists of multiple baselines, the authors only include the standard deviation for their approach. There are several \"close calls\" where seeing the standard deviation of the baselines would have been very helpful in assessing the contribution of the approach.\n2) Relevant baselines - while CGDT is a very relevant baseline, a comparison to additional DT-based baselines, especially ones that are known to be good at stitching (Elastic DT is a strong candidate in my opinion, also because it also evaluated multiple options) could be helpful. Such a comparison would enable the readers to understand how the appraoch stands in comparison to other enhancements of DT.\n\nQUESTIONS:\nThe authors are welcome to address my comments above.\n\nLIMITATIONS:\nYEs", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes Value-Guided Decision Transformer (VDT), such that it integrates value-based learning into Decision Transformer (DT) by incorporating **advantage-weighting** on behavior cloning (BC) during offline training. Also, VDT proposes to unify offline training with online fine-tuning with trajectory-level replay buffers and return-to-go (RTG) alignment leading to robust performance. During action selection, Q-function based sampling process is used to choose more optimal actions.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Integrate Value-based Methods**: The integration of advantage-weighted learning and Q-function-based sampling is promising and bridging the gap between DT and other offline RL methods.\n    \n2. **Empirical Strength**: VDT achieves on-par or better results across diverse benchmarks, including Adroit, Kitchen, Maze2D, and AntMaze.\n\n\n### Weaknesses\n\n1. **Computational Overhead in Sampling**: Although possibly alleviated by GPU parallelization, the parallel sampling with multiple RTGs and evaluation horizons could become unaffordable for larger scale environments. Further development trading off between optimality and efficiency needs to be addressed.\n\nQUESTIONS:\n1. I am curious about the ablation study on how performance improves as we increase the range of RTGs together with Q-value based sampling, e.g. are we able to query higher RTGs than that of in the training dataset?\n    \n2. In online tuning stage, how much the performance depends on the initial offline training? how about ablating the different data ratios? how does VDT perform if the offline dataset has very poor coverage while including high-value trajectories?\n\nLIMITATIONS:\n- As the authors also discussed, one primary limitation of the proposed method is the computational overhead of the Q-value guided sampling process during inference, especially when scaling to larger-scale, complex environments.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces value-function guidance into the decision transformer for trajectory sequence prediction (conditional-sequence-modeling) problems in reinforcement learning. This value-function guidance enables the model to (1) stitch together sub-optimal trajectories during offline training and (2) keep improving rapidly with only a few online interactions. Experiments show that the proposed value-guided decision transformers (VGDT) outperform prior methods in both offline and offline-to-online settings.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- This paper uses value-function guidance to unify the learning of the decision transformer in both offline and online RL settings. By leveraging the Q-function learned by an n-step implicit Q learning loss, the model is able to use an advantage-weighted regression loss to predict the action sequences, enabling stitching from sub-optimal trajectories and value-guided action selection during inference.\n\n- Through experiments on D4RL benchmarks, the paper shows that VGDT achieves competitive empirical performance, compared to prior offline reinforcement learning methods and other decision transformer variants. Additional results ablate different components of the algorithm and show the effects of them in the complete VGDT.\n\n**Weaknesses**\n\n- Theorem 4.1 seems to state that the optimal decision transformer derived from the proposed method results in *one-step* policy improvement, which does not provide a guarantee to converge to the optimal policy. In practice, there has been success in using *one-step* policy improvement to derive an RL method. But some connections between the one-step value function and the optimal value function will still be helpful. \n\n- For the experiments, ablation studies of some hyperparameters are missing. Also, including some results on benchmarks other than D4RL or tasks with image-based observations can potentially make the experiment more convincing.\n\nQUESTIONS:\n- Line 153: Why is the expectile loss function an unbiased estimate of the conditional expectation? The IQL [1] paper has already shown that the value function will converge to the maximal Q-function within the support of the dataset.\n\n- Line 160: Why do we prefer the n-step TD loss over the simpler one-step TD loss? \n\n- Eq. 5: Any empirical evidence to show the effect and significance of $\\eta$ and $\\lambda$?\n\n- Line 173 - 175: This sentence is confusing. How does this loss function optimize the stability of trajectory modeling? How is it able to prevent the local optima within the data distribution?\n\n- Line 195 - 196: Why is the two-step sampling procedure uniform? From the sentence, it sounds like the method is prioritizing trajectory segments from longer trajectories. \n\n- Line 199 - 206: In the online learning phase, does the method use the current ongoing trajectory to update the DT?\n\n- The sampling process in Sec. 4.3 is similar to both the cross-entropy method in model-based RL [2] and the Selecting from Behavior Candidates (SfBC) [3] policy extraction method in offline RL. Including some discussions of these two categories of methods in this section could be helpful.\n\n- Line 215: How do we predefine the m candidate RTG? Are they varying across different tasks?\n\n- Fig 3: From this figure, it looks like the effect of the evaluation horizon is minor – the performance for $E = 1$ and $E \\geq 1$ is not significantly different. Then, why do we not prefer a simpler method?\n\n[1] Kostrikov, Ilya, Ashvin Nair, and Sergey Levine. \"Offline Reinforcement Learning with Implicit Q-Learning.\" In International Conference on Learning Representations.\n\n[2] Chua, Kurtland, Roberto Calandra, Rowan McAllister, and Sergey Levine. \"Deep reinforcement learning in a handful of trials using probabilistic dynamics models.\" Advances in neural information processing systems 31 (2018).\n\n[3] Chen, Huayu, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu. \"Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling.\" In The Eleventh International Conference on Learning Representations.\n\nLIMITATIONS:\nThe limitation section only discussed the computation cost of the algorithm. Including some discussions about the theoretical issue or missing properties of the current method could be helpful.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}