Offline Reinforcement Learning with Adaptive Feature Fusion
Authors: Tieru Wang
Kunbao Wu
Guoshun Nan
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement Learning
Trajectory Stitching
Primary area: reinforcement learning
TL;DR: We propose the Q-Augmented Dual-Feature Fusion Decision Transformer (QDFFDT), which adaptively integrates global sequential and local immediate features, achieving state-of-the-art results on D4RL benchmark tasks.

Abstract: Return-conditioned supervised learning (RCSL) algorithms have demonstrated strong generative capabilities in offline reinforcement learning (RL) by learning action distributions based on both the state and the return. However, many existing approaches treat RL as a conditional sequence modeling task, which can lead to an overreliance on suboptimal past experiences, impairing decision-making and reducing the effectiveness of trajectory synthesis. To address these limitations, we propose a novel approach, the Q-Augmented Dual-Feature Fusion Decision Transformer (QDFFDT), which adaptively combines both global sequence features and local immediate features through a learnable fusion mechanism. This model improves generalization across different tasks without the need for extensive hyperparameter tuning. Experimental results on the D4RL benchmark show that QDFFDT outperforms current methods, establishing new state-of-the-art performance and demonstrating the power of adaptive feature fusion.

Published as a conference paper at ICLR 2026
OFFLINE REINFORCEMENT LEARNING WITH ADAP- TIVE FEATURE FUSION
Tieru Wang Kunbao Wu
Guoshun Nan'
Beijing University of Posts and Telecommunications, China fwangtieru, wukunbao, /ebuuupt.ooeebpp........
ABSTRACT
Return-condttoned supervised learning (RCSL) algorithms have demonstrated strong generative capabilittes in offline reinforcement learning (RL) by learn- ing action distributions based on both the state and the return. However, ex- isting approaches treat RL as a conditional sequence modeling task, where ac- tions are predicted from historical context and a target return. This leads a critical flaw: the policy can overfit to the specific, often suboptimal, actions found within those historical contexts. Consequently, a high target return, the model struggles to synthesize a correspondingly high- quality action sequence, which fundamentally limits its ability to perform effec- tive trajectory stitching and outperform the behavioral policy. To address these limitations, we propose a novel approach, the Q-Augmented Dual-Feature Fu- sion Decision Transformer (QDFFDT), Our key innovation a learnable fu- sion mechanism that explicitly separates and then adaptively combines global, history-aware sequence features with local, immediate Markovian features. This introduces a structural bias that prioritizes single-step dynamics while still lever- aging long-term context, improving generalization without the need for exten- sive hyperparameter tuning. Experimental results on the D4RL benchmark show that QDFFDT outperforms current state-of-the-an methods, demonstrating the power adaptive feature fusion for robust offline RL, Our code available at https::/gittuut com http:/wwww...oooeeeeneeeere/erpe
when conditioned on
INTRODUCTION
Offline reinforcement learning (RL) is data- driven paradigm that learns exclusively from
happer
kd
110
110
fixed dataset of previously collected experi- ences (Levine et al. 2020). This paradigm can be used in scenarios where online interaction is impractical, either because data collection expensive or dangerous (Kiran et al., 2021). However, the inability to interact with the en- vironment brings some challenges, like out-of- distribution (OOD) problems. Previous works on offline RL generally address this problem by compelling the learned policy to be close to the behavior policy (Fujimoto et al. 2019) Kumar al. 2019 Wu et 2019; Wang et al. 2020 Fujimoto & Gu 1 or constraining the value function assign low values OOD actions
70-
70-
60
DT
ODFFOT
60
DT
DFFOT
QT
CDFFDT
Figure 1: Normalized returns on medium datasets. By adaptively fusing two feature streams, our methods leverage suboptimal trajectories and achieve improved performance.
(Kumar et al., 2020 Kostrikov et al. [021aa Wu et al. 2021) In recent years, with the success
of Transformer in CV (Dosovitskiy) 2020: Liu al. 2021 1) and NI.P (Devlin) 2018 Radford al.) 2019), the architecture has also been introduced to offline RL to model trajectories (Chen et al. 2021 Janner et al. 202I). Based on Transformer, the offline RL problem is viewed as a distribution of supervised learning (RCSL) task whose core idea is to learn the return-ondiitonall
"Corresoonding author.

---

Published as a conference paper at ICLR 2026
actions in each state (Brandfonbrener et al. 2022). By framing RL as a supervised learning problem, RCSL improves both data efficiency and training stability, demonstrating superior performance in offline settings.
However, existing RCSL methods, by treating RL as a pure sequence modeling problem, suffer from fundamental limitation: they tend to overfit to historical sub-trajectories and thus hinder effective trajectory stitching. A sequence generation model is usually required to reliably reproduce trajectories found within the training dataset, but this objective is inherently misaligned with the goal of RL, which is to discover novel policies that may outperform any single trajectory in the dataset by combining the best segments from multiple different trajectories. Some studies have recognized this issue and employ various approaches to shorten the dependency on long sequences Wu et al.) 2024 Kim et al. 2023; Wang et al. 2025). However, these prior works either lack the flexibility to effectively balance global and local information or require extensive hyperparameter tuning to specific datasets to achieve optimal performance.
Building upon these insights, propose the Q-Augmented Dual-Feature Fusion Decision Trans- former (QDFFDT), a novel architecture that explicitly addresses the tension between global context and local optimality. Our core idea is to learn a state-dependent fusion weight that adaptively bal- ances two distinct feature streams: (1) a global sequential modeling stream, which uses a standard self-attention module to capture long-term context, and (2) a local immediate modeling stream, which employs simple MLP to extract features from the current state, respecting the Markov property. This dual-feature design introduces structural bias that helps the model prioritize high- quality local decisions while not ignoring valuable long-term information, eliminating the need for brittle hyperparameter tuning across different datasets. Furthermore, inspired by (Hu et al.) 2024), we augment the RCSL framework with a Q-learning module to enable explicit policy improvement through value-based guidance. As previewed in Figure [ our method significantly outperforms pure sequence modeling baselines across several suboptimal datasets. Extensive experiments on the D4RL benchmark (Fu et al., 2020) demonstrate that our QDFFDT achieves state-of-the-ar perfor- mance. Ablation studies further illustrate the flexibility and stability of our method across diverse datasets.
We summarize our contributions follows: (1) We identify and demonstrate how pure sequence modeling hinders the trajectory stitching ability of the RCSL paradigm. (2) We propose QDFFDT, an adaptive method that integrates global sequential features with local immediate features, pro- viding the necessary inductive bias for offline RL. (3) We demonstrate QDFFDT's superiority over prior methods on D4RL. tasks, underscoring its potential to be employed in various scenarios.
2 MOTIVATION: WHEN PURE SEQUENCE MODELING HINDERS TRAJECTORY STITCHING
Some prior works (Wu et al. 2024; Kim et al. 2023 Wang et al., 2025 have acknowledged the challenges associated with Long-sequenee modeling in the context of RL. However, it remains un- clear when sequence modeling specifically hinders trajectory stitching in practice. To explain the limitation of sequence modeling, consider simple task in Figure 2 A sequence modeling method is capable of finding the optimal sub-trajectory "ABCD" in the initial stages. However, after reach- ing state 'D', the context information can easily mislead the model, causing it to follow trajec- tory from the training data towards state 'K', thereby missing the optimal trajectory: "ABCDE". In contrast, a single-step modeling method based on the Markov property unaffected by historical sub-tre cub-trajectories and can directly select the action that yields the highest return. We set the sequence length in DT 1, resulting in the single-step Decision Transformer (SSDT). We then conduct exper- iments in the simple benchmark environment and measure the success rates of achieving the optimal trajectory "ABCDE". shown in Table SSDT almost always obtains optimal solutions, whereas DT tends to converge to suboptimal outcomes.
The results of this experiment highlight the inherent difficulty of achieving effective trajectory stitch- ing solely through sequence modeling, Several studies (Yamagata et al. 2023 Wang et al. 2024 Hu et al. 2024) propose augmenting sequence models with value learning to reduce their depen- dence historical trajectories. However, whether such hybrid approaches can fully overcome the limitations inherent in sequence modeling remains an open question. To investigate this, we extend case study by testing the QT (Hu et al. 2024) method. For simplicity, omit the value learning
2

---

Published as a conference paper at ICLR 2026
B
B
E
10
10
10
Trajectories
agent
SSDT agent
Figure 2: A simple example explains the issue of sequence modeling (DT) to find the tra- jectory with max return. contrast, the single-step method (SSDT) finds the optimal path. The numbers on the arrows are rewards and uppercase letters inside the circles represent states. There are two trajectories, "ABCDK" and "AIJDE".
Table 1: The performance of DT, SSDT, and QT with different settings the simple case. K and 7 are hypeparameters in related methods. Success rates are averaged over 1000 random rollouts (10 independently trained models and 100 trajectories per model).
Method
K
"
Success Rate (%)
DT
3
0.0
SSDT
100.0
3
0.01
0.1
0.1
0.1
QT
3
1.0
0.6
10.0
96.2
process and instead directly assign action values based on prior knowledge. Reconsidering the ample in Figure ald setting a discount factor 7 = 1.0, we can compute the Q-values with Dynamic Programming (DP): QD, >) 10, QDD, )) 1, Q,Q y) 10 + max(Q(D, /). Q(D, x)) = 20. The values of other actions are computed in similar manner. Table shows that even if with the introduce of the Q-function guide policy optimization, the model remains susceptible
the influ- ence of prior sub-trajectories. We hypothesize that this is due to the design of QT's policy objective, which seeks simultaneously optimize action values while constraining the policy to close the behavior distribution. As result, the policy optimization term cannot fully counteract the adverse effects introduced by the behavior cloning term
RELATED WORK
3.1 OFFLINE REINFORCEMENT LEARNING
Offline RL aims to learn effective policies from static datasets, where a central challenge is miti- gating the distributional shift that arises from evaluating policies different from the data-collection policy. A dominant line of work addresses this by constraining the learned policy. For example. CQL (Kumar et al. 2020) learns conservative, lower-bound Q-function to penalize out-of-distributtion actions. Similarly, TD3+BC (Fujimoto & Gu 2021) incorporates behavioral cloning loss to reg- ularize the policy, while IQL (Kostrikov et al. 2021b) uses expectile regression to implicitly guide policy improvement. More recently, various generative models have been employed for their ca- pacity to model complex action distributions. These range from A2PO (Qing et al., 22024) which uses conditional VAEs model advantages, diffusion models like D-QL (Wang et 2022)) and DTQL (Chen et al. 2024) that have shown promise in capturing the underlying behavior policy.
3.2 RETURN-CONDITTONED SUPERVISED LEARNING
Another line work tries to solve the offline RL problem by the return-conditioned supervised learning method Brandfonbrener et al. (2022), which aims to learn the return-conittioaal distribu- tion of actions in each state, and then define a policy by sampling from the distribution of actions that receive high returns. This approach is pioneered by Decision Transformer (DT) (Chen al.
3

---

Published as a conference paper at ICLR 2026
2021). which utilizes the sequence model to capture temporal dependencies within entire trajecto- ries. Subsequent works have explored variations on this theme; for instance, DC (Kim et al. 2023) investigate convolution- baseed backbones, while EDT (Wu et al., 2024) introduce mechanism to dynamically adjust the model's context length. While powerful, pure return-conditoonng can be an unreliable signal for optimality. This limitation has spurred the development of hybrid models that integrate RCSL with explicit value learning. Consequently, ACT (Gao et al. (024) generates actions conditioned on learned advantages. QT (Hu al. 2024) combines trajectory modeling with explicit Q-value predictions, and QCS (Kim et al. (2224) adaptively integrates Q-function guidance into the RCSL training objective.
4 PRELIMINARIES
4.1 OFFLINE REINFORCEMENT LEARNING.
A reinforcement learning problem is typically represented as Markov decision process (MDP) Bellman 1957) (Po. S, A. P. R. 7), where Po is the distribution of initial states, S is the state space, A is the action space, P(s'|s, a) the transition probability, R(s, a) is the reward function, and y is the discount factor. The goal of traditional RL is to train the agent to interact with the environ- ment for policy n² (alss) that maximizes the expected return IB| [= ('r(St, a.)]. Offline RI. different from conventional online RI, we can only train an agent based on the fixed offline dataset
D {(s, a, r, s'}} instead of exploring in a simulation environment. The dataset is collected by a behavior policy TB from online interaction.
4.2 RETURN-CONDITIONED SUPERVISED LEARNING
RCSL methods formulate RL as a supervised learning problem to improve stability. Decision Transformer (DT) (Chen et al. 2 one of the important works in RCSL methods. DT mod- els the trajectory as a sequence comprising states, actions, and RTGs. At timestep 4, DT accepts
= (Rt-K+1, K+1, 9t-K 11,, Rt-K+1,--. .... Rr-1; 1t-11, St-1, at-1, st) as input and outputs ag, where K is the sequence length. The goal of the training stage is to update a neural network so that it converges to the true in the dataset. In the evaluation stage, the real RTG is unavailable. Therefore, DT specifies a target RTG as the desired score. Each timestep, the reward obtained is subtracted from the target RTG until the end of the episode or the maximum length of the trajectory.
While RCSL methods are capable of leveraging historical information for decision- makinn, their lack of accurate value guidance often hinders their ability to perform effective trajectory stitching. To address this limitation, recent works have proposed to combine value-based learning with sequence
modeling for policy optimization (Yamagata et al. 2023 Gao et al. 2024 Hu et al. 2024: Kim
et al. 2024).
METHOD
To address the trajectory stitching challenges inherent in sequence modeling, we consider introduc- ing single-step inputs. However, entirely discarding sequence modeling is impractical, particularly in high-dimensional, pixel-based environments like Atari (Mnih et al. (013). where compressed state representations often lack the necessary information for optimal decision-making. In such cases, temporal context remains essential. To balance these demands, we propose hybrid architec- ture. hual-Feature Fusion Decision Transformer (DFFDT), which adaptively integrates sequential and immediate contextual features using learnable weighting mechanism. This design allows the model dynamically modulate the influence of each feature branch, improving generalization while reducing the burden of manual hyperparameter tuning. Furthermore, to address the known limita- tion that RTG signals may not reliably reflect action values, we incorporate the Q-learning module, enabling more accurate value estimation and more effective policy improvement. The resulting al- gorithm is the Q-Augmented Dual-Feature Fusion Decision Transformer (QDFFDT). In this section, first describe the optimization process for learning the weighting coefficient. We then present novel RCSL architecture that fuses sequence-level and single-step state features. Finally, we illus- trate our value-based policy improvement mechanism.
4

---

Published as a conference paper at ICLR 2026
5.1 SEAL-EEAURE FUSION DECISION TRANSFORMER
We investigate how to balance sequential and immediate decision information in RCSL. While se- quence modeling excels reproducing behaviors from offline datasets, often prioritizes the con- tinuation of previously observed sub- sub-trajectories This can lead the model to favor actions that are consistent with historical patterns, even better options exist. On the other hand, single-step mod- eling makes decisions solely based on RTGs, enabling more flexible trajectory stitching by directly choosing actions associated with higher desired returns. Formally, given a state with candi- date actions {a' M and corresponding RTG values {f( at ) single-step model selects a = arg maxg: (3, a²). In contrast, a sequence model, conditioned on preceding sub-trajectorics, tends choose the next action from within trajectory in the dataset that closely resembles the observed history. To address this issue, we adaptively modulate the influence of sequential features based on whether the training RTGs for & align with the optimal return max; Rt
5.1.1 LEARNING ADAPTIVE FUSION WEIGHTS
For evaluating the degree of alignment between the training returns and the optimal return, we employ expectile regression (Newey & Powell 1987 Koenker Hallock 2001) to approximate the maximum RTG for given state. This method is usually used in applied statistics and econometrics. Notably, it has also been integrated into offline RL like IQL (Kostrikov et al. 2021b). We utilize it to estimate the supremum of {R' (s, at )MM 1 for state s. The a E (0, 1) expectile of a random variable X is defined as solution to the asymmetric least squares problem:
arg min Ex [L m.)],
where (z( = l0 1(u < 0)/2². When > 0.5, the asymmetric loss assigns more weight to larger than my and less weights to x smaller than Ma. Then we can approximate max; R' using separate value function:
(v) = (., [L((R V=)()))
The state-value function (s)s) is trained to approximate the empirical upper bound of RTG values in the dataset. Notably, our application of expectile regression differs from that in IQI., which uses it to approximate the optimal Q-value. The purpose of (s) is thus to provide a reliable signal for identifying suboptimal data. Specifically, when (s)ss significantly exceeds an observed RTG R(s)), it indicates the subsequent trajectory is suboptimal. This signal prompts our model to reduce reliance on sequence features, as they might replicate the suboptimal behavior, and instead prioritize Markovian information through an Alpha network:
(s) max - R 0)
La (w) (e,
T
where T' is a temperature hyperparameter, and (s) (0, 1) produced via final Sigmoid layer. This final fusion coefficient for sequential features is computed as â(s) ammin + (1 (1-cl amin) (s), where Ormin is a predefined minimum sequential weight. Correspondingly, Markovian features are weighted by - ás). To encourage strong reliance on sequence modeling at the beginning of training, we initialize the final linear layer of the Alpha network with zero weights and a large positive bias (e.g., 5), ensuring (s) initially. As training progresses, the loss La (w) penalizes ow as(s) (s) when (s) > Rł, driving (s) (and thus â(s)) downwards. Conversely, when Vs((s) < R, the loss term vanishes, allowing (s) and â(s) to remain high, thereby preserving the contribution from sequence modeling for high-quality trajectories.
5.1.2 CONTEXTUAL. AND MARKOVIAN FEATURE INTEGRATION
Our Dual-Feature Fusion block consists of two key components, as illustrated in Figure Following the design of DD (Ajay al. 2022). we exclude action tokens from the input resulingg in the following trajectory representation:
7t (R. k+1 8--+1, Re-1, 81-1. Re. 8t)
5

---

Published as a conference paper at ICLR 2026
42
Linear
R2
ConCat
MLP
a(e)
a(e))
glo
Casual Transformer
Figure 3: An overview our Dual-Feature Fusion Decision Transformer architecture. Both states and returns are simultaneously input into the Transformer and MLP components. The resulting intermediate features are adaptively fused via the learned fusion coefficient and then projected into the action space.
Each RTG-state pair (Re, $t) encoded into the local representation hioe using a lightweight MLP. In parallel, we compute the global representation hifh using the self-attention mechanism, as in DT, to capture long-range temporal dependencies. The two feature streams are then fused through the learned coefficient å follows:
(1- ā(s:)) h1 + ā(s;) hglh.
The fused representation is finally projected into the action space. To enable effective fusion, both hose and hi are normalized using Layer Normalization (LN) (Ba al. 2016) align their feature scales.
During training, consistent with standard RCSL methods, we minimize the Mean Squared Error (MSE) between predicted and ground-trute actions:
LDFFDT (0) K(rL)
1
Σ (a: mo(s)))
K
i=t- K+1
where denotes the ground-truth action and To TT(T)) is the i-th action predicted by the policy net- work.
5.2 Q-AUGMENTED RCSL OPTIMIZATION
During training, the RTG often fails accurately represent the true value of given state-action pair. Furthermore, the mismatch between the target RTG and the optimal RTG can further degrade the performance of RCSL methods, preventing convergence to the theoretical optimum (Brandfon- brener et al.) 2022). Therefore, leveraging the value functions from DP methods to implement policy improvement is necessary. We develop five networks, including two Q-networks Q Qáy two tar- get networks Q Q06, and one target policy network TØ. For detailed implementation, we draw inspiration from QT (Hu al. 2024) and make minor modifications, leading to in the following Q-function loss:
| Qo. (8m: am))
B K4aat------------------- K+HIND
K+1
E p + 1+1 1-
min Qøy (S1+1, det1), (n-step)
where Qm =
j=m
1=12
I' + min Qøy (sm+1 am+1), 1=1.2
(I-step)

---

Published as a conference paper at ICLR 2026
where y is the discount factor. The target actions 0m+1 and u,+1 sampled from the target policy Tg.. In practice, we manually choose either the n-step or 1-step Bellman equation to estimate the Q-value function. While QT suggests that the n-step Bellman equation demonstrates improvement over the 1-step approximation, we find that this observation does not always hold in experiments. We provide detailed discussion of this discrepancy in Appendix [D.4]
To combine policy improvement with supervised learning, we define the policy loss as a weighted sum of the DFFDT behavior cloning loss and the Q-value guided improvement term:
(8) where À is weighting coefficient that balances the supervised learning and the value-based policy improvement. Following TD3+BC (Fujimoto & Gu (02T), the Q-function is normalized to mitigate the scale mismatch across offline datasets and prevent gradient imbalance between the two learning objectives. The detailed pseudo-code is provided by Algorithm[] in Appendix AA.
L = À (DFFDD (0) + (o) X LDFFDT (0) Bra-plEsrr (8 (T(T)))
Convergence Guarantees. Our approach relies on two well-understood principles: the DFFDT (RCSL) component, whose convergence to the behavior policy has been verified in prior work (Hu et al. 2024), is preserved in our method despite certain architectural adjustments; the Q- augmentation component follows standard dynamic programming, guiding the policy toward higher- value regions with established convergence properties. While a formal proof for the combination is non-trivial, the components theoretically grounded and validated empirically.
6 EXPERIMENTS
Our experiments aim to answer the following questions: (1) How does our QDFFDT perform on the offline benchmarks compared with other state-of-the-art approaches? (2) What advantages does the feature fusion architecture offer compared to modeling with either sequential features or immediate features alone?
(3) How does the dynamic fusion of the two components in our QDFFDT differ from static fusion in terms of performance?
Datasets. We evaluate our approach on the D4RL benchmark suite (Fu al. 2020). including tasks from Gym-MuJoCo, Maze2D,, AntMaze, Kitchen, and Adroit. Gym-MuloCo tasks relatively simple. containing many near-optimal trajectories and smooth reward functions. Maze2D evaluates an agent's ability stitch sub-trajectories to reach goal. AntMaze introduces sparse rewards and higher complexity by replacing the 2D ball with an 8-DoFF quadruped. Kitchen involves multi-task sequential objectives, emphasizing generalization to unseen states. Adroit is derived from human demonstrations, with a limited state-action space that requires strong policy regularization to ensure stability.
Baselines. We compare the proposed QDFFDT with several traditional value-based methods, where we select CQL (Kumar al., 2020). IQL (Kostrikov et al. 20216), BEAR Kumar al. (2019)
AWR (Peng et al.) 2019), TD3+BC (Fujimoto & Gu, 2021). EQL (Xu et al. 2023). LAPO Chen al. 2022), and CQL+AW (Hong et 2023). For RCSL and derivative methods, the evaluated algorithms include DT Chen et 20.211 DC (Kim et 2023). LSDT (Wang et al. 2025), RvS 2023). EDT (Wu et al. POR (Xu et al. (2022). CGDT (Wang et al. 2024). ACT Gao et al. 2024), QT (Hu et al. 2024). and QCS (Kim et al. 2024). Additionally, we also compare baEebased and iiffusion-based methods, including BCQ (Fujimoto et al.) 2019)) A2PO (Qing et al.;
(Emmons et al. 202T). GDT (Hu et al. 2023). QDT (Yamagata et al.
2024). Reinformer (Zhuang | 2004).
2024], Diffuser (Janner et al. 2022), DD (Ajay et al., 2022), and D-QL (Wang et al. 2022). The
scores of these baselines are sourced from their respective papers or from our experiments to ensure
a fair comparison.
6.1 MAIN RESULTS
We evaluate our QDFFDT against a suite strong baselines, reporting normalized scores on the D4RL benchmark following the standard protocol (Fu et al. 2020). As shown in Table 2 our QDFFDT demonstrates significant performance gains, achieving state-of-thh-ar or competitive re- sults across all evaluated benchmarks. On suboptimal datasets with Markov property, such as the medium and medium-replay tasks, our method outperforms QT, which lacks sufficient emphasis on local dependencies for policy training. This advantage extends to the challenging navigation tasks
7

---

Published as a conference paper at ICLR 2026
The performance of our QDFFDT and baselines on D4RL. tasks. Results for QDFFDT correspond to the mean and standard deviations of normalized scores over 50 random rollouts (5 independently trained models and 10 trajectories per model) for Gym tasks, and over 500 random rollouts (5 independently trained models and 100 trajectories per model) for the other tasks. The dataset names are abbreviated as follows: medium to 'm', replay to 'r,, expert to 'e', umaze to 'u', large to II, diverse to 'd', and play to 'p'. The best average values marked in bold.
Table 2:
Gym Tasks
CQLL IQL
TD3+BC
DT
DC
LSDT ACT
CGDT
QT
QCS]
QDFFDT
halfcheetah-m-el
91.6 86.7
90.7
86.8
93.0
93.2
96.1
93.6
96.1
93.3
97.5 1 0.6
hopper-m-e
105.4 91.5
98.0
107.6
110.4
111.7
107.6
113.4
110.2
112.3 1 0.4
walker2d-m-e
108.8 109.6
110.1
108.1
109.6
109.8
113.3
109.3
112.6
116.6
111.6 1 0.4
halfchectah-m
49.2 47.4
48.4
42.6
43.0
43.6
49.1
43.0
51.4
59.0
65.7 1.3
hopper-m
69.4 66.3
59.3
67.6
92.6
87.2
67.8
96.9
96.9
96.4
101. 0.3
walker2d-m
83.0 78.3
83.7
74.0
79.2
79.1
88.8
88.2
94.9 +
halfcheetah-m-rr
45.5
44.2
44.6
36.6
41.3
42,9
43.0
40.4
48.9
54.1
62.0 +
hopper-m-r
95.0
94.7
60.9
82.7
94.2
93.9
98.4
93.4
102.0
100.4
102.7 + 0.5
walker2d-m-r
77.2
73.9
81.8
66.6
76.6
74.7
56.1
78.1
98.5
94.1
101.0
1.3
Average
80.6
77.0
75.3
74.7
82.2
82.0
79.6
82,4
89.8
90.3
94.3
Maze2D Tasks
CQLL
IQL
Difffuser
DD
EQL
DT
QDT
EDT
Reinformer
QT
QDFFDT
maze2d-u
94.7 42.1
113.9
116.2
56.5
$7.3
35.8
$7.2
92.3
123.3 1 19.7
maze2d-m
41.8 34,9
121.5
122.3
36.3
8.2
13.3
18.3
85.6
156.5
157.1 1 4.0
maze2d-1
49.6 61.7
123.0
125,9
57.0
2.3
31.0
26.8
47.4
214.5
198.9 1 13.0
Average
62.0 46.2
119.5
121.5
49.9
13.8
33.9
28.6
63.4
154.4
159.8
AntMaze Tasks
CQL
IQL
TD3+BC
DT
DC
LSDT
RvS
POR
QT
QCS|
QDFFDT
antmaze-u
74.0
87.5
78.6
59.2
85.0
80.0
65.4
90.6
78.3
92.5
99.8 + 0.4
antmaze-u-d
84.0
62.2
71.4
53.0
78.5
83.2
60.9
71.3
72.7
82.5
78.4 - 8.8
antmaze-m- m-p
61.2
71.2
0.0
33.2
85.5
84.6
28.7
84.8
94.8 + 2.0
antmaze-m-d
53.7
70.0
3.0
0.0
27.5
75.8
67.3
79.2
26.7
75.2
97.0 - 1.4
antmaze-l-p
15.8
39.6
0.2
0.0
4.8
32.4
58.0
1.7
70.0
79.6
+ 3.3
antmaze-l-d
14.9
47.5
0.0
0.0
12.3
36.9
73.4
25.0
77.3
80.0
3.5
Average
50.6
63.0
27.3
18.7
40.2
53.5
76.2
38.9
80.4
88.3
Kitchen Tasks
CQL
IQL
BCQ
BEAR
AWR D-QL LAPO CQL+AW
DT
QT
QDFFDT
kitchen-partial
49.8
46.3
18.9
13.1
15.4
60.5
53.7
36.0
57.9
68.1
65.9 + 1.0
kitchen-mixed
51.0 51.0
8.1
47.2
10.6
62.6
62.4
50.5
42.5
53.0
63.9 1 3.4
Average
50.4 48.7
13.5
30.2
13.0
61.6
58.1
43.3
50.2
60.6
64.9
Adroit Tasks
CQL
IQL
D-QL
A2PO
DT
GDT
QT
QCS)
QDFFDT
pen-human
37.5 71.5
72.8
68.9
62.9
74.2
66.7
92.5
94.6
83.9
94.1 d: 7.9
pen-cloned
39.2
37.3
57.3
85.0
28.7
50.0
42.8
86.2
85.6
66.5
94.8 + 8.2
Average
38.4 54.4
65.1
77.0
45.8
62.1
54.8
89.4
90.1
75.2
94.5
of Maze2D and AntMaze, which require long-horizon reasoning. In these domains, QDFFITT again demonstrates superior performance, leveraging sequence modeling for long-term dependen- cies while incorporating local signals to mitigate the impact of suboptimal sub-trajectories. Notably, our architecture achieves this without requiring explicit goal-conditioning, prerequisite for meth- ods like LSDT and QCS. The significant gains over purely return-conme conn-cooniiionee models like LSDT and DT underscore the importance of integrating value learning, particularly in environments with sparse rewards. Finally, our QDFFDT demonstrates strong performance on the high-dimensional Kitchen and Adroit domains. This success in complex, manipulation-focusee environments highlights the robustness and generalization capabilities of our hybrid approach.
6.2 COMPONENT ABLATION STUDY
To assess the effectiveness of hybrid architecture, conduct an ablation study the individual feature extraction branches across D4RL and Atari domains (Mnih et al.) (003). For RCSL meth- ods, compare pure sequence modeling baseline (DT), purely Markovian return-onnditioned
8

---

Published as a conference paper at ICLR 2026
Mujace 6ym
AntMaae
Atrrl
QTC Terulamm
Turnforma
120
140-
50-
mad medimmnaplyy
mepuumespptt
MmmAre-- Fp
artiiak--1d
Obert
Fong
Senguet
(a) Mujoco Gym performance
(b) Antmaze performance
(c) Atari performance
Figure 4: Ablation study on the contributions of the sequential and immediate feature branches. We report the performance of the sequential-only, immediate-one -oneeiittteoll,, and fused architectures. Average normalized scores are computed over 5 seeds for the Gym and AntMaze datasets, and 3 seeds for the Atari datasets.
MLP (RC-MLP), and our proposed DFFDT. For the value-augmented variants, evaluate three models built upon unified codebase to ensure fair comparisons. First, QRC-Transformer, our re- implementation of QT, serves the pure sequence modeling baseline. Second, to establish purely Markovian baseline, we introduce QRC-MLP, which replaces the Transformer backbone of QRC- Transformer with a simple MLP, thereby relying only on the current state. Finally, we present our QDFFDT, which integrates both approaches. As shown in Figure on fully observable tasks such as Gym and AntMaze, pure sequence modeling tends suffer from suboptimal segments within historical trajectories. This issue degrades both trajectory stitching and local feature extraction. In contrast, MLP-based methods that emphasize immediate information show stronger performance. On the Atari benchmark, high-dimensional visual observations need to be compressed into latent feature vectors for downstream RL. However, this compression process often results in significant information loss, making the tasks partially observable. In this scenario, sequence modeling be- comes especially effective by leveraging historical context to compensate for the missing informa- tion. Although sequence modeling may be affected by suboptimal sub-trajectories in historical data, its strong capability to recover lost signals significantly outweighs this drawback. As result, demonstrates superior feature extraction compared methods that rely solely on local observations. Our hybrid approach integrates both sequence modeling and local feature extraction, resulting in consistently strong performance across all benchmarks.
Table 3:
The effects in our QDFFDT. Average and standard deviation scores are reported over 5 seeds for Maze2D and AntMaze tasks. The best average values are marked in bold.
Maze2D
AntMaze
Average
u
m
m-p
m-d
1.p
I-d
QDFFDT(Á 0)
25.9 55.3
144.5 + 17.3
94.2 +
2.9
95.4 94.4 76.6
82.2
5.3
79.2
6.3
86.9
QDFFDT(Ā = 0.25)
17.5 + 41.3
117.1 22.8
9.44
5.5
82.2 2 4.3
4.3
80.4
QDFFDT( = 0.5)
19.0 + 25.8
132.5
+ 22.0
94,6
3.0
95.2
84,8
8.4
85.0
QDFFDT(6 = 0.75)
70.7 + 32.3
145.8 9.9
96.0
1.
94.2
80.6
2.3
94.9
QDFFDT(6 1)
117.4 7.0
156.6 + 2.5
95.0
1.6 94.8
1.5
71.8
7.16
101.2
QDFFDT
123.3 + 19.7
157.1 4.0
194.8 +
2.0
97.0
1.4 79.6
80.0 3.5|
105.3
6.3
DYNAMIC VS. FIXED COFFFICIENT &
We have demonstrated the importance of integrating both sequential and Markovian features to sup- port effective decision-making. this subsection, we further examine the influence of dynamic versus static fusion coefficients. Within our dual-feature fusion framework, setting the fusion co- efficient & to forces the policy to rely exclusively on sequence modeling, whereas setting & to makes it depend entirely immediate information derived from the current state and return. We conduct experiments on our QDFFDT variants, evaluating static fusion settings by fixing á to the set (0, 0.25, 0.5, 0.75, 1), and report the results for Maze2l) and AntMaze tasks in Table The results

---

Published as a conference paper at ICLR 2026
indicate that, with appropriate tuning, static coefficients can yield strong performance specific datasets. However, achieving optimal performance requires careful manual tuning for each dataset, which reduces the general upplicability of this approach. In contrast, sion coefficient consistently achieves robust performance across all datasets and obtains a higher average score than any fixed setting. These results highlight the effectiveness of the proposed adap- tive fusion mechanism in combining global and local information without the need for extensive hyperparameter tuning.
dynamically learned fu-
7 CONCLUSION
In this work, we propose the Q-Aummented Dual-Feature Fusion Decision Transformer (QDFFDT), which combines sequential and immediate information through a learnable fusion mechanism. By dynamically adjusting the contributions of global and local features, our QDFFDT effectively allevi- ates the trajectory stitching limitations of conventional sequence modeling. Additionally, the integra- tion of the Q-learning module further enhances value estimation and policy optimization. Extensive experiments on the D4RL benchmarks demonstrate that our QDFFDT consistently achieves supe- rior performance across range of tasks, including those with sparse rewards and high-dimensional state space, underscoring the effectiveness of value-guided feature fusion in offline RL.
REPRODUCIBILITY STATEMENT
We have made every effort to ensure the reproducibility of our work. Pseudocode for the proposed method is presented in Appendix A.]] and all hyperparameter settings are included in Appendix A.33 The datasets used in our experiments are publicly available through the D4RL benchmark (Fu al, 2020
ACKNOWLEDGMENTS
This work is supported in part by the National Natural Science Foundation of China under Grant 624710644 in part by the Guangxi Call-for-Bid Science and Technology Program JB2504240002; in part by the Fundamental Research Funds for the Beijing University of Posts and Telecommunica- tions under Grant 2025A14S02.
REFERENCES
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.15633 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 16X.vv1607.06450 06.60,006450, 2016.
arXiv preprint
Richard Bellman. markovian decision process. Journal of mathematics and mechanics, pp. 679 684, 1957.
Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. When should prefer decision transformers for offline reinforcement learning? arXiv preprint arXiv:2305.1455m0 14550, 2023.
David Brandfonbrener, Alberto Bietti, Jacob Buckman, Romain Laroche, and Joan Bruna. When does return-conditioned supervised learning work for offline reinforcement learning? Advances in Neural Information Processing Systems, 35:1542-1553, 2022.
Xinye Cao, Hongcan Guo, Guoshun Nan, Jiaoyang Cui, Haoting Qian, Yihan Lin, Yilin Peng, Diyang Zhang, Yanzhao Hou, Huici Wu, et al. Advancing compositional Ilm reasoning with structured task relations in interactive multimodal communications. IEEE Journal on Selected Areas in Communications, 2025a.
10

---

Published as a conference paper at ICLR 2026
Xinye Cao, Hongcan Guo, Jiawen Qian, Guoshun Nan, Chao Wang. Yuqi Pan, Tianhao Hou, Xiao- juan Wang, and Yutong Gao. Videominer: Iteratively grounding key frames of hour-long videos via tree-based group relative policy optimization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 23773-23783, 2025b.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34: 5084-15097, 2021.
Tianyu Chen, Zhendong Wang. and Mingyuan Zhou. Diffusion policies creating trust region for offline reinforcement learning. arXiv preprint arXiv:2405.1969/ 2024.
Xi Chen, Ali Ghadirzadeh, Tianhe Yu, Yuan Gao, Jianhao Wang, Wenzhe Li, Bin Liang, Chelsea Finn, and Chongjie Zhang. Latent-variable advantage-weighted policy optimization for offline rl. arXiv preprint arXiv:2003.0mm 08949, 2022.
Jacob Devlin. Bert: Pre-training of deep bidirectional transformers for language understanding.
arXiv preprint
2018.
Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1199 /929, 2020.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint arXiv:2112.107511 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep
data-driven reinforcement learning. arXiv preprint urXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In International conference on machine learning, pp. 0052-2062. PMLR, 2019.
Chen-Xiao Gao, Chenyang Wu, Mingjun Cao, Rui Kong, Zongzhang Zhang, and Yang Yu. Act:
Empowering decision transformer with dynamic programming via advantage conditioning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 12127-12135, 2024.
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193, 2020.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
Stfl:1600.08415. 2016.
Zhang-Wa Wei Hong, Pulkit Agrawal, Rémi Tachet des Combes, and Romain Laroche. Harness- ing mixed offline reinforcement learning datasets via trajectory weighting. arXiv preprint arXiv:2306. 13085, 2023.
Shengchao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer. arXiv preprint arXiv:2303.03747, 2023.
Shengchao Hu, Ziqing Fan, Chaoqin Huang. Li Shen, Ya Zhang, Yanfeng Wang, and Dacheng Tao. Q-value regularized transformer for offline reinforcement learning. arXiv preprint 2rXiv:2405.17099 2024.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-1286, 2021.
Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for ficxible behavior synthesis. arXiv preprint arXiv:2205.0999) 2022.
Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making. arXiv preprint arXiv:2310.03022 2023.

---

Published as a conference paper at ICLR 2026
Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Adaptive q-aid for conditional supervised learning in offline reinforcement learning. Advances in Neural Information Processing Systems, 37:87104 11771144877131 2024.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6990, 2014.
Ravi Kiran, Ibrahim Sobh, Victor Talpaert, Patrick Mannion, Ahmad A Al Sallab, Senthil Yoga- mani, and Patrick Pérez. Deep reinforcement learning for autonomous driving: A survey. IEEE Transactions on Intelligent Transportation Systems, 23(6):4909-4926 2021.
Roger Koenker and Kevin Hallock. Quantile regression. Journal of economic perspectives, 15(4): 143-156, 2001.
Nya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofir Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774-5783. PMLR, 2021a.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning, arXiv preprint arX:v:2110006169, 2021b.
Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off-policy q-learning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1179-1191 2020.
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tuto- rial, review, and perspectives on open problems. arXiv preprint arXiv:2005.00644; 2020.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10012-10022, 2021.
Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint 1rXiv:1312.5602, 2013.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10), pp. 807-814, 2010.
Whitney Newey and James L Powell. Asymmetric least squares estimation and testing. Econo- metrica: Journal of the Econometric Society, pp. 819-847, 1987.
Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Advantage-weighted regression: Simple and scalable off-policy reinforcement learning. arXiv preprint 1910:0:110000/177 2019.
Yunpeng Qing, Shunyu Liu, Jingyuan Cong, Kaixuan Chen, Yihe Zhou, and Mingli Song. A2PO: Towards effective offline reinforcement learning from an advantage-aware perspective. In The Thihty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL
https: http::/:///peeeviie...oom net/ www.:neeetootteeeee forummid-dhio..o RmGqg5ee
Chenyang Qiu, Guoshun Nan, Caiyi Zhang, Chenrui Liang, Ruiqi Dai, Hongchen Yang. Shiyuan Liu, Zhili Zhou, Changhua Pei, and Xiaofeng Tao. Mining multi-scale -petial-frequency clues for unsupervised intrusion detection. IEEE Transactions on Information Forensics and Security, 2025.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models unsupervised multitask learners. OpenAl blog, 188)99 2019.
12

---

Published as a conference paper at ICLR 2026
Jincheng Wang. Penny Karanasou, Pengyuan Wei, Elia Gatti, Diego Martinez Plasencia, and Dim- itrios Kanoulas. Long-short decision transformer: Bridging global and local dependencies for generalized decision-makee making. In The Thirteenth International Conference Learning Represen- fations, 2025.
Yuanfu Wang, Chao Yang. Ying Wen, Yu Liu, and Yu Qiao. Critic-guided decision transformer for offline reinforcement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 15706-15714, 2024.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint arXiv:2208.00 06793, 2022.
Ziyu Wang, Alexander Novikov, Konrad ZoIna, Josh S Merel, Jost Tobias Springenberg. Scott Reed, Bobak Shahriari, Noah Siegel, Caglar Gulcehre, Nicolas Heess, al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
Jialong Wu, Haixu Wu, Zihan Qiu, Jianmin Wang, and Mingsheng Long. Supported policy opti- mization for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:31278 31291, 2022.
Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint 1911:1911..136.1 2019.
Yue Wu, Shuangfei Zhai, Nitish Srivastava, Joshua Susskind, Jian Zhang, Ruslan Salakhutdinov, and Hanlin Goh. Uncertainty weighted actor-critic for offline reinforcement learning. arXiv preprint
arXiv:2105.081400 2021.
Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in Neural Information Processing Systems, 36, 2024.
Haoran Xu, Li Jiang, Li Jianxiong, and Xianyuan Zhan. A policy-guided imitation approach for offline reinforcement learning. Advances in neural information processing systems, 35:4085- 4098, 2022.
Haoran Xu, Li Jiang, Jianxiong Li, Zhuoran Yang, Zhaoran Wang, Victor Wai Kin Chan, and Xi- anyuan Zhan. Offline rl with no ood actions: In-sample learning via implicit value regularization. arXiv preprint 2r3.v:2303. 15810, 2023.
Taku Yamagata, Ahmed Khalil, and Raul contos-Rodriguez, Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. Inter- national Conference on Machine Learning, pp. 38989-39007, PMLR, 2023.
Zifeng Zhuang, Dengyun Peng, Jinxin Liu, Ziqi Zhang, and Donglin Wang. Reinformer: Max-return sequence modeling for offline rl. arXiv preprint 4rXiv:2405.08740, 2024.
13

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
A.1 QDFFDT
Our QDFFDT follows the framework of DT (Chen et al. 2021). as shown in Algorithm
Algorithm QDFFDT: Q-Augmented Dual-Feattur Fusion Decision Transformer
Initialize V network V& and Alpha network Ow-
Train the V network for each iteration do
Sample transition mini-batch B = {(R, ~ D.
Update Vp by Equation 2 end for.
Train the Alpha network for each iteration do
Sample transition mini-batch {())) D.
Update aw by Equation 3 end for.
Initialize policy network and critic networks Qai Qøs-
Initialize target networks TT' Qal Qa4- Train the QDFFDT for each iteration do
Sample sequence transition mini-batch B {(Rg, Sy, sy) }
D.
# Q-value function learning
Sample target actions à via one of two methods:
(n-step): Sample ā ~ To (ā++ RooekK|BetttKK... s1+1:++)
(I-step): For each step E [1KK,, sample 0141444 ~ Te (0c+1:444 (ee 56-1:04))
Update Qú, and Qøz by Equation 7 # Policy learning do Sample ac:4i Te. (ama4i Reati, Stat end for.
for 0 to
Update policy by minimizing Equation # Update target networks
0' pp' + p)0, o', pσl, (1 p)dt for i [1,
end for.
Inference with QDFFDT Given multiple target return-to-go choice fm and initial state 8p- repeat
Sample multiple actions with different return-to-oo aj To Pi K4 9t-K-11) for i I,..., m.
Compute Q value with candidate state-action pair (st, at) for i 1....,
m Sample the action ag from action set {at) with the max Q value by arg max min Qo! (st, at).
a: im1,2
Execute the action a, and collect the reward and next state 81+1.
Update current retur-toogo R6 ft; r for 1,.... m. until Done true.
A.2 NETWORK ARCHITECTURES
Network. We build the network as 3-layer MLP with ReLU (Nair & Hinton 2010) activation functions and 256 hidden units in each layer.
Alpha Network. We build the Alpha network as a 2-layer MLP with a ReLU activation and Sigmoid output layer, using 256 hidden units.
14

---

Published as a conference paper at ICLR 2026
Q Networks. We build two Q networks with the same architecture, each consisting of 4-layer MLP with Mish activation functions and 256 hidden units in each layer.
Conditional MLP Policy. We build our RTG-conditio MLP branch 3-layer MLP with GeLU (eendrycks & Gimpel activation functions and 256 hidden units in each layer.
Conditional Transformer Policy. We build our RTG-conditioned Transformer branch following the architecture DT (Chen et al., 2021). To simplify integration and reduce implementation com- com plexity, utilize the codebase from https: wom/tinooff-aa//oo....... instead of the official DT implementation.
A.3 HYPERPARAMETERS
Table Hyperparameters of QDFFDT in our experiments.
Parameter
Value
Number of layers
3
Number of attention heads
1
Embedding dimension
512 Kitchen and Adroit
256 other tasks
Nonlinearity function
GeLU
Grad norm clip
5.0
Batch size
256
Dropout
0.1
Optimizer
Adam (Kingma 2014)
Learning rate
3e-4
Temperature coefficient
le5
mmnn
0.3
For the Gym domain, we use context length of 20, while a context length of 5 is used for the other domains. The discount factor is set to y 0.99 by default, and increased to 7 0.995 for AntMaze tasks, following the design choice in SPOT (Wu et al. 2022). The expectile level is typically set to a = 0.7, except for the untmaze-large datasets, where we set to 0.9. For Q-value updates, we employ the 1-step Bellman equation halfcheetah datasets and the n-step Bellman equation on all other tasks. Additional detailed hyperparameter settings for actor training are given in Table 4 Finally, À is adjusted according to the characteristics of different datasets. For example, medium-exper expert datasets usually require more policy regularization due to their high quality, while medium and medium-replay tasks require more Q-learning improve the policy. Based on these considerations, the choice of X for each dataset is shown in Table 5
B BASELINE DETAILS
We evaluate the performance of QDFFDT with many baselines. These methods consist of traditional
value-based methods: CQL (Kumar et al. 2020), IQL (Kostrikov et al. 2021bb, BEAR Kumar et al.
2019), AWR (Peng al. 2019), TD3+BC (Fujimoto Gu (021). EQL (Xu et al. 2023). LAPO
Chen et al.) (022). and +OW+AW (Hong et al. 2023); RCSL and its derivative methods: DT (Chn
al.] 2021). DC (Kim et al. 2023). LSDT (Wang et 2025).
RvS (Emmons al. 2021). GDT
(Hu et al. 2023), QDT (aaaagata et al,, (023). EDT (Wu et al. 2024). Reinformer (Zhuang et al.
2024). POR (Xu et al. 2022). CGDT (Wang et al. 2024), ACT (Gao al. 2024]. QT (Hu et al,
methods: BCQ (Fujimoto ctt al. 2019), A2PO Qing et al., 2024), Diffuser (Janner et al. 2022), DD (Ajay et al. 20022) and D-QL (Wang et al.) 2022). The performance results for these baselines are taken from their original publications, with the exception of QT. In the original QT paper, scores on the AntMaze and Adroit domains averaged over random seeds with 10 evaluations per dataset, which is inconsistent with the evaluation protocol used by other basclines. To ensure fair comparison, we re-evaluate QT using the official codebase, computing average performance 100 evaluations for every seed.
2024. and QCS (Kim et al. 2024); as well as sederrtive-moeel-eeee
15

---

Published as a conference paper at ICLR 2026
Table 5: Hyperparameter settings of all selected tasks.
Tasks
À
nalecheeall-mmdiiu-eeennr----2
0.1
hopper- pepper-medium-expeee
1.0
mulkker2d-meiium-ene
0.5
haffcheetah-mediun--n
0.01
hopper-medium-y-
0.2
molker2d-medium-ii
0.2
malffcheetal--ediuu--eee
0.01
Supper-medium--eeplay-aaoo
0.2
walker2d walker2d-mediumee medium-replay---22
0.2
maze2d-umaze-vl
0.3
maze2o-medium-v
0.2
maze2d-large-vl
0.1
antmaze-umazz-vv
0.5
auttmaz--umaze-divree---0
0.5
antmaaze-medium-play---a
0.3
anthnaze-meduum--dvvee---00
0.3
antmazeelarge-play-vo
0.2
antimaze-laree-diverse--0
0.2
kitchen-parrial-v00
50.0
kitchen-mixed-vv
20.0
pen- gen-human-vl pen-cloned-vl
8.0
9.0
Mujoco Gym
120
Oftenign K0: Oftiremve actons, K=
$507
Normalized Score
40
hopper-m
hopper-m-r
walker2d-m
walker2d-m-r
halfcheetan-m halfchestah-m-r
Figure 5: Evaluation results for DT, DT (remove actions), and SSDT on the MuloCo Gym subopti- mal datasets. We record the normalized scores averaged across 3 seeds, 10 evaluations each.
C DISCUSSIONS OF AUXILIARY EXPERIMENTS
C.1 COMPARATIVE ANALYSIS SSDT DT
To further investigate the role of sequence length, particularly the weaknesses of long-tem context on suboptimal datasets as discussed in Section [ we conduct a comparative analysis between SSDT and DT on the MuJoCo Gym benchmarks. Since single-step models make decisions based solely
16

---

Published as a conference paper at ICLR 2026
on the current state and target return without action inputs, we establish a fair baseline by using a DT variant with action tokens removed. To ensure equivalent data exposure across models with varying sequence lengths, adjust the number of training iterations: 25,000 for sequence length of 20 and 500,000 for a sequence length of (e.g.. SSDT). All other hyperparameters, including the batch size of 256, align with the official DT implementation. As shown in Figure SSDT consistently outperforms the standard DT across all tested tasks. This result is particularly insightful because it challenges the conventional assumption that longer temporal context always beneficial. The strong performance of SSDT stems from the fully observable, Markovian nature of the Gym environments. In such a setting, the current state theoretically contains all information required for optimal decision-making Consequently, for DT, conditioning on long context can introduce restrictive inductive bias, potentially forcing the model to replicate suboptimal historical patterns rather than selecting the optimal action from the current state. SSDT, with its context length of 1, effectively avoids this bias. It is crucial to note that this finding does not diminish the importance of sequence modeling in general. In Partially Observable MDPs (POMDPs), such Atari from pixels, history vital for inferring the latent state from ambiguous single observations (Bhargava et al, 2023
C.2 EXPERIMENTAL DETAILS OF DFFDT
Our DFFDT framework (Section 6.2) implemented on top of the original DT, and therefore inherits the majority of its hyperparameter settings. The major modifications concern the introduction of additional training hyperparameters for the dynamic weighting coefficient. A complete list of these settings is provided in Table
Table 6: Hyperparameters of DFFDT in our experiments.
Parameter
Value
Number of layers
Number of attention heads
Embedding dimension
128
Nonlinearity function
ReLU
Grad norm clip
0.25
Batch size
64
Context length
20
Dropout
0.1
Learning rate
le-4
Temperature coefficient
le5
O
0.3
C.3 EXPERIMENTAL DETAILS ON ATARI
To illustrate the impact of sequence-leved representations in pixel-based environments, we conduct comparative study on the Atari domain between the sequence modeling approach QRC-Transformeer, the single-step modeling variant QRC-MLP, and our hybrid method QDFFIT in Section 6.2 Be- cause the performance gains afforded by sequence modeling in image-based scenarios substantially outweigh drawbacks introduced by suboptimal historical sub-trajectories, we maintain high pro- portion of sequence modeling by manually tuning the feature- weettrr- coefficient á. Additionally, to mitigate the burden of computational resourees, reduce both the number of attention heads and the number of layers in the sequence- sequence-modeling module. Final scores are normalized following (Hafner al. 2020). The complete hyperparameter configurations are detailed in Table 7
17

---

Published as a conference paper at ICLR 2026
Table 7: Hyperparameters of QDFFDT for Atari experiments.
Hyperparameter
Value
Number of layers
3
Number of attention heads
1
Batch size
128 Pong
64 other tasks
Context length
50 Pong
30 other tasks
Embedding dimension
256
Nonlinearity
ReLU, encoder
GeLU, otherwise
Encoder channels
32, 64, 64
Encoder filter sizes
8 x 8, x 4,3 x 3
Encoder strides
4,
Max epochs
10
Dropout
0.1
Learning rate
x 10-4
Adam betas
(0.9.0.95)
Grad norm clip
1.0
Weight decay
0.1
Learning rate decay
Linear warmup and cosine decay
Warmup tokens
512 20
Final tokens
2 500000 K
&
0,9 Pong
1.0 other tasks
D SUPPLEMENTARY STUDIES
D.1 COMPARISON BETWEEN ADDITION-BASEI FUSION AND CONCATENATION FUSION
To evaluate our adaptive addition-bant based fusion module (Add) against the concatenation- based fusion module (Cat), we combine contextual and Markovian features by concatenation followed by linear projection, yielding
= Concat ((1 - á(s))) 1/o á(st) hggl),
where ho and hglo denote the single-step and contextual representations (see Section 5.1.2). Un- der identical hyperparameters on the wtwp:/ddddiiii--pp-p------ and hopper-medium-replay datasets, Add and Cat achieve closely matched performance (Table 8). Since Add requires slightly fewer parameters, we adopt as our preferred fusion strategy.
Table 8: Performance comparison between addition- basted fusion and concatenation based fusion. Average and standard deviation scores reported over 5 seeds. The best average values are marked in bold.
Dataset
Add
Cat
walker2d www.ediiummeepa--2222
101.0 + 1.3 102.7 0.5
100.3 + 1.8 103.9 + 0.3
hopper-mediummrrplaye
D.2 ABLATION STUDY ON EXPECTILE LEVEL Ø
We incorporate expectile regression to estimate an upper bound of RTGs and use the discrepancy between this estimate and the observed RTGs during training to dynamically adjust the weighting between sequential and single-step modeling. To understand the effect of the expectile level a, we conduct ablation study. As illustrated in Figure varying influences the relative contribution
18

---

Published as a conference paper at ICLR 2026
antmaze-u-d
antmaze-I-p
Normalized
Figure 6: Ablation on the a. Each line represents the variation the average scores reported over seeds, and the shaded represents the corresponding standard deviation.
0.13
a 0.54
RTG
Action
Action
Figure 7: Visualization of the correlation between the a and the RTG distribution. Each subfigure corresponds a different fixed state. The x-axis represents the ID projection of action vectors via PCA..
of sequential features. Higher values of reduce the weight assigned to sequential modeling. which can help mitigate the negative impact of suboptimal sub-trajectories historical data. However, in partially observable environments such as AntMaze, sequential information provides valuable context for decision-making. Therefore, selecting an appropriate value a is critical for achieving favorable trade-off between leveraging historical context and mitigating the influence of suboptimal behavioral trajectories.
D.3 RELATIONSHIP BETWEEN & AND RTG DISTRIBUTION DURING TRAINING
Single-mode approaches face fundamental limitations. Pure sequence models reproduce subop- timal sub-trajectories by overemphasizing historical information, whereas purely local models tend to fit the average of expert actions. To address these issues, introduce the dynamically adjusted weighting factor å to control the fusion of global and local features. The final fusion coefficient å obtained by scaling the output of the Alpha Network (Section 5.I.I), and the raw output a is pos- itively correlated with á. Analyzing thus provides an effective lens to understand feature fusion behavior. Figure ( visualizes the distribution of on the 2itiker-ddmdiuum-22 dataset. For RTG dis- tributions with larger or more frequent fluctuations, tends to be lower, indicating reduced reliance on sequence modeling. Our fusion mechanism mitigates the limitations of single-mode models by preserving sequence stability high-reward trajectories while adaptively amplifying local signals in suboptimal regions, thereby enabling more effective trajectory stitching.
D.4 ANALYSIS OF THE V-VUL UPDATE STRATEGY
As described in Appendix A.3 we apply the 1-step Bellman equation to the halfcheetah datasets. This choice is motivated by our observation that model performance degrades when using the n-step Bellman backup in combination with longer context lengths. Since incorporate Markovian local features into the fused representation, which are independent of sequence length, we hypothesize that this degradation stems from the n-step Bellman backup's dependence trajectories. To verify
19

---

Published as a conference paper at ICLR 2026
Table 9: Performance with varying context length K on the halfcheetah--mediun-- Average and standard deviation scores are reported over 5 seeds.
K=5
K=10
K=20
n-step Bellman
65.2 + 1.1
59.6 + 0.6
55,6 + 0.9
1-step Bellman
66.9 + 1.3
64.7 + 1.5
65.7 1.3
Table 10: Performance with varying context length K on the 2a Aver- age and standard deviation scores are reported over 5 seeds.
K=5
K=10
K=20
n-step Bellman
55.8 1.7
48.3 + 0.7
38.4 + 1.1
1 1-step Bellman
64.2 + 0.8
62.8 + 1.8
62.0 + 1.1
halfcheetah-medium-2
wallerr2b-medduu--repayy22
MAc mit.m
Frequency
150
50-
50
20
80
Normalized Return
Normalized Return
(a) salfchectah-meddime
(b) walker2d walker2d-medium mediim--rpplayye mediuum-replay-v2
Figure The distribution of returns of trajectories for seffiheetth-mediume and walker2d- medium-replay-v2 datasets. The dashed lines represent the maximum trajectory returns.
this, we conduct an ablation study on hulfcheetah- and hatfcchetub-mmedium---paaayy Table Table [o demoonstrate that increasing sequence length impairs trajectory stitching when using the n-step Bellman equation, whereas performance remains stable under the 1-step Bellman equation, thereby exposing a limitation of n-step methods in certain settings. However, QT (Hu et al.) 2024) demonstrates that n-step updates outperform 1-step updates walker2d- 22 To illustrate this discrepancy, we plot the normalized return distributions for halfcheetah-men -eddimm-- v2 and walker22d-medium--eplaay
in Figure X On halfcheetah malfcheetah-mmedium-22 medium-v2, the return distribution is concentrated around 40, indicating a large number of medium-return trajectories. These dense, moderate-reward sequences provide sufficient local learning signals that reduce the impact of bias in 1-step Bellman updates. However, due to the absence of long, contiguous high-return trajectories, the n-step Bellman targets are often underestimated, as the cumulative rewards over longer horizons remain limited. This leads to degraded performance when n-step updates are applied. In contrast, presents a long-tailed return distribution with a predominance of low- return trajectories and sparse but critical high-return segments. In this setting, the limited coverage of high-return regions prevents I-step updates from accurately propagating value signals, resulting in pronounced estimation bias. N-step backups, by aggregating rewards across extended time spans, help bridge these gaps and anchor credit assignment in high-return regions, thereby mitigating the shortcomings of 1-step updates and enabling more effective policy learning. These observations underscore the distinct advantages of each strategy in different scenarios: 1-step updates are well- suited for environments with locally concentrated return distributions, whereas n-step backups
20

---

Published as a conference paper at ICLR 2026
more effective in long-taile long-tailed settings where high-return trajectories sparse and require longer temporal credit assignment.
E TRAINING TIME
We compare the training efficiency of our proposed methods, DFFDT and QDFFDT, against the baseline methods, DT and QT. All experiments conducted single NVIDIA RTX 2080 Ti GPU. In terms training time, DFFDT requires 1.19 hours, while DT takes 0.79 hours. QDFFDT takes 3.88 hours compared to 3.59 hours for QT. Regarding memory usage, the peak GPU memory consumption is 1.226 GB for DFFDT, 1.216 GB for DT, 2.836 GB for QDFFDT, and 2.742 GB for QT. To ensure fairness, QDFFDT and QT use identical hyperparameters wherever applicable, and the same configuration is adopted for DFFDT and DT. These results show that the additional computational cost introduced by our feature fusion design is limited, while the performance gains are substantial.
F LIMITATIONS
Our QDFFDT adaptively fuses global and local features based on the estimated quality of future tra- jectories, with Q-learning component to guide policy improvement. Experimental results across range of tasks demonstrate its effectiveness in trajectory planning. However, the overall architecture introduces additional structural complexity compared to standard sequence modeling approaches. Reducing this complexity while maintaining improving performance remains important direc- tion for future work.
G FUTURE WORK
Several promising directions remain for future exploration. First, incorporating multi-scale tem- poral modeling (Qiu et al., 2025) could enhance credit assignment across varying time horizons. Second, investigating compositional task structures (Cao et al. 2025a) may improve generalization in multi-task offline settings. Finally, exploring tree-based policy optimization (Cao al. 2025b) for hierarchical decision making remains a promising direction.
H THE USE OF LARGE LANGUAGE MODELS (LLMS)
In accordance with the ICLR policy on the use large language models (LLMs), we disclose that LLMs are employed during the preparation of this paper. Their is limited to assisting with language editing, improving readability, and polishing the presentation of the manuscript. LLMs are not involved formulating the research questions, desigming the methodology, conducting ex- periments, analyzing results, or drawing scientific conclusions. We take full responsibility for all technical content, claims, and conclusions presented in this work.
21


Reviews:
[{"review_id": "MnDVBCA4ny", "reviewer": "Reviewer_qJYv", "summary": "This paper combines long-term sequential modeling and local immediate modeling via an adaptive feature fusion mechanism for efficient offline return-conditioned supervised learning (RCSL). Previous pure sequence modeling methods tend to suffer from replicating suboptimal trajectories in the training dataset, and additional Q-value guidance cannot eliminate this drawback. By estimating an approximated max-return for each state, the proposed method DFFDT can adaptively rely more on the local one-step action prediction when suboptimal RTG signals are observed (after comparing them with the estimated V-values). With the aid of the Q value, the variant QDFFDT achieves superior performance compared to strong baselines on D4RL benchmarks and is claimed to be flexible in balancing the use of global and local features.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written and easy to follow. Figure 3 efficiently captures the main idea of this paper. Notations are clean and consistent.\n- The core component, feature fusion, is well-motivated through a necessary didactic example in Section 4 and demonstrates that the Q-value guidance alone is insufficient for counteracting the “momentum”  of replicating the suboptimal trajectories.\n- QDFFDT performs well on D4RL benchmarks with an acceptable increase in training complexity (as shown in Appendix E).\n- Appendix includes enough experimental details for reproduction purposes.", "weaknesses": "- **In terms of the didactic example,** why does QT perform worse when $\\eta=10.0$ than when $\\eta=1.0$? Increasing $\\eta$ gradually eclipses the effect of the DT loss and enables QT to choose actions with the highest Q values (which is true for $\\eta=0.01-1.0$). Intuitively, setting $\\eta$ to a very large number should completely erase the effect of the BC term, and thus QT should be able to achieve a higher success rate.\n- In my opinion, the core contribution of this paper is the content in Sections 4 and equations (3) and (5), which trains an Alpha network to adaptively trade off between global sequential features and local immediate features. **However, I have the following concerns about the effectiveness of feature fusion, despite QDFFDT’s strong performance shown in Table 2:**\n    1. In terms of the training objective of the Alpha network, a naive solution to minimize it could predict $\\alpha_\\omega (s) = 0 $ for most of the state. And when $V_\\psi (s) \\leq \\hat{R}$, there is no gradient signal to control the training behavior. I suspect that this adaptive mechanism would eventually bias towards learning from local features more than from global features. This suspicion is amplified, especially when the authors have to manually set $\\hat{\\alpha}=1$ for 3 of the 4 evaluated Atari games to ensure the model fully relies on the sequence modelling (since it benefits largely for these partially observed environments). Moreover, in Appendix D.3, it seems to be true that the learned $\\alpha$ is small in general. Could the authors comment on this? What is the average learned $\\alpha_\\omega(s)$ for these datasets?\n    2. In terms of the ablation study, Figure 4 gives me a feeling that (1) for environments with the Markovian property, local modeling alone is enough; and (2) for partially observed environments, global modeling alone is enough. The performance gain of feature fusion is not as prominent as in Table 2, if we take the variance into consideration. Given the question (a) above, it makes sense for (Q)DFFDT working relatively well for Gym and Antmaze, since it might predict low $\\alpha$ most of the time. And what is the performance for Atari, if $\\hat{\\alpha}$ is not manually set?\n    3. In summary, I feel that it is a bit far-fetched to conclude that “These results highlight the effectiveness of the proposed adaptive fusion mechanism in combining global and local information without the need for extensive hyperparameter tuning.” (Lines 473-475). The authors should provide more evidence to defend this core argument.\n- **Some experimental details need more clarification:**\n    1. In Table 2, why does QT perform poorly for antmaze-l, while the reimplementation of QT, QRC-Transformer, performs much better in Figure 4(b)?\n    2. I am aware that most of the offline methods fail on Adroit-door, Adroit-hammer, and Adroit-relocate. However, QT has made some progress on those challenging ones. I wonder how QDFFDT performs in those environments.\n    3. For papers claiming SOTA results, it is common for them to tune hyperparameters. However, I found that the authors reduce context length from 20 to 5 for tasks outside the Gym domain, which influences the analysis since this hyperparameter also controls the strength of global feature modeling. If the proposed adaptive feature fusion works well, it should be able to handle longer context for those environments with a Markovian property. How does the performance of QDFFDT change with different context lengths?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YkO9Hq41dE", "reviewer": "Reviewer_EsME", "summary": "This paper proposes QDFFDT (Q-Augmented Dual-Feature Fusion Decision Transformer), an offline reinforcement learning framework that adaptively combines global sequence modeling and local single-step (Markovian) modeling through a learnable fusion mechanism. The method extends return-conditioned supervised learning (RCSL) by learning a state-dependent fusion coefficient based on the expectile regression of return-to-go values, balancing the influence of contextual and immediate features. Additionally, a Q-learning module is integrated to improve policy optimization and mitigate the overreliance on suboptimal trajectory patterns common in sequence-based methods like Decision Transformer (DT) and QT. Experiments on the D4RL benchmark across Gym-MuJoCo, Maze2D, AntMaze, Kitchen, and Adroit show that QDFFDT achieves consistent state-of-the-art performance, improving generalization without extensive hyperparameter tuning.", "questions": "1. How sensitive is QDFFDT to the expectile level σ and the temperature parameter T in α-learning?\n\n2. Does α̂(s) exhibit meaningful variation across states or converge to near-constant values? Can this be visualized?\n\n3. What is the runtime overhead compared to QT or DT in wall-clock training?\n\n4. Would replacing the Transformer with a lightweight recurrent or convolutional encoder yield similar benefits?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- Well-motivated insight: The paper identifies a real weakness in RCSL methods—overfitting to suboptimal sequence fragments that hinder trajectory stitching—and directly targets it with an adaptive fusion approach.\n\n- Elegant hybrid design: The fusion of sequential and immediate features through a learned state-conditioned weight is conceptually neat and grounded in the Markov property. It’s a balanced architectural innovation rather than an ad hoc ensemble.\n\n- Integration of value learning: The Q-augmentation complements RCSL’s generative bias, yielding both theoretical plausibility and empirical strength.\n\n- Comprehensive experiments: The authors evaluate on five major domains (Gym, Maze2D, AntMaze, Kitchen, Adroit) against more than 15 baselines, consistently outperforming DT, QT, and RCSL variants. Ablations (static vs. adaptive α̂, sequence-only vs. single-step) are insightful and well-documented.\n\n- Clarity and reproducibility: The paper is clearly written, with structured explanations, pseudo-code, and full hyperparameter tables. The method is easy to reimplement.", "weaknesses": "- Limited theoretical contribution: Despite clear empirical validation, the method is largely architectural. The theoretical motivation for how expectile-based weighting improves trajectory stitching remains heuristic.\n\n- Overcomplexity relative to gain: The approach introduces five networks and multiple losses (expectile, α-loss, Q-loss, MSE) for modest algorithmic advancement; simplicity compared to QT or ACT is reduced.\n\n- Incremental relationship to prior work: The approach closely resembles QT (Hu et al., 2024) with an extra fusion branch and expectile weighting. The novelty over these strong baselines feels evolutionary rather than revolutionary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uYtWhMwDOs", "reviewer": "Reviewer_n7e3", "summary": "This paper introduces QDFFDT and DFFDT, novel offline RL models that mitigates the limitations of purely sequence-based RCSL. DFFDT fuses global sequential and local immediate features through a learnable fusion weight, enabling effective balance between long-term context and Markovian decision signals. QDFFDT further integrates Q-learning to enhance value estimation and policy improvement. Experiments on the D4RL benchmark demonstrate that QDFFDT achieves SOTA across diverse domains.", "questions": "1. I suggest that the authors improve the Abstract and Introduction to define the problem and present their motivation more clearly. For example, in the Abstract, they state that existing approaches “lead to an overreliance on suboptimal past experience,” which is not immediately clear to the reader. In the Introduction, the authors should describe the problem more explicitly in the second paragraph and clarify the method and its rationale in the third paragraph. They may also consider moving Section 4 into the Introduction.\n\n2. It would be helpful to include the Q-component in Figure 3. With this component, the architecture corresponds to QDFFDT; without it, it represents DFFDT.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper presents a clear illustration of the limitations of prior return-conditioned sequence models and introduces an adaptive transformer architecture that empirically achieves strong performance across diverse offline RL tasks.", "weaknesses": "The paper’s contribution primarily lies in modifying the architecture of the Decision Transformer for sequential decision modeling. The authors aim to learn a mapping $f(a_t \\mid \\hat{R}_{\\leq t}, s_{\\leq t})$ and introduce a dual-feature fusion mechanism to encourage attention to immediate Markovian information. However, this design raises a theoretical concern:\n\nThe input information remains unchanged, and a sufficiently expressive Transformer should, in principle, be capable of learning any desired feature weighting — including the adaptive combination $((1-\\hat{\\alpha}(s_t))h^{loc}_t + \\hat{\\alpha}(s_t)h^{glo}_t)$ — directly through its self-attention and learned parameters. The additional fusion module, therefore, seems architecturally redundant unless the authors can demonstrate a theoretical advantage.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)