DRDT3: Diffusion-Refined Decision Test-Time Training Model
Authors: Xingshuai Huang
Di Wu
Benoit Boulet
Venue: Accepted by TMLR ()

Abstract: Decision Transformer (DT), a trajectory modelling method, has shown competitive performance compared to traditional offline reinforcement learning (RL) approaches on various classic control tasks. However, it struggles to learn optimal policies from suboptimal, reward-labelled trajectories. In this study, we explore the use of conditional generative modelling to facilitate trajectory stitching given its high-quality data generation ability.  Additionally, recent advancements in Recurrent Neural Networks (RNNs) have shown their linear complexity and competitive sequence modelling performance over Transformers. We leverage the Test-Time Training (TTT) layer, an RNN that updates hidden states during testing, to model trajectories in the form of DT. We introduce a unified framework, called Diffusion-Refined Decision TTT (DRDT3), to achieve performance beyond DT models. Specifically, we propose the Decision TTT (DT3) module, which harnesses the sequence modelling strengths of both self-attention and the TTT layer to capture recent contextual information and make coarse action predictions. DRDT3 iteratively refines the coarse action predictions through the generative diffusion model, progressively moving closer to the optimal actions. We further integrate DT3 with the diffusion model using a unified optimization objective. With experiments on multiple tasks in the D4RL benchmark, our DT3 model without diffusion refinement demonstrates improved performance over standard DT, while DRDT3 further achieves superior results compared to state-of-the-art DT-based and offline RL methods.

Published in Transactions on Machine Learning Research (09/2025)
DRDT3: Diffusion-Refined Decision Test-Time Training Model
Xingshuai Huang
ringshuai Thngshuiihuungg@mail.meee
Deportment of Electrical and Computer Engineering McGill Universily
Di Wu
di. uru @magill.ccc
Department Electrical and Computer Engineering McGill University
Benoit Boulet
benoit. mrnoit.boulee@@mcgll.
Department of Electrical and Computer Engineering
McGill University
Reviewed on OpenReview: h/pp::://penreeteee net/ forum? 162gLhIzgh
Abstract
Decision Transformer (DT), trajectory modelling method, has shown competitive per- formance compared to traditional offline reinforcement learning (RL) approaches on vari- ous classic control tasks. However, it struggles to learn optimal policies from suboptimal, reward-labelled trajectories. In this study, we explore the use of conditional generative modelling to facilitate trajectory stitching given its high-quality data generation ability, Additionally, recent advancements in Recurrent Neural Networks (RNNs) have shown their linear complexity and competitive sequence modelling performance over Transformers. We leverage the Test-Time Training (TTT) layer, an RNN that updates hidden states during testing, to model trajectories in the form of DT. We introduce a unified framework, called Iiffusion-Refince Decision TTT (DRDT3), to achieve performance beyond DT models. Specifically, we propose the Decision TTT (DT3) module, which harnesses the sequence modelling strengths of both self-attention and the TTT layer to capture recent contextual information and make coarse action predictions. DRDT3 iteratively refines the coarse tion predictions through the generative diffusion model, progressively moving closer to the optimal actions. We further integrate DT3 with the diffusion model using a unified opti- mization objective. With experiments on multiple tasks in the D4RL benchmark, our DT3 model without diffusion refinement demonstrates improved performance over standard DT, while DRDT3 further achieves superior results compared state-of-the-art DT-based and offline RL methods.
1 Introduction
Reinforcement learning aims train agents through interaction with the environment and receiving reward feedback. It has been widely applied in areas such robotics (Jia al. 2023
Guo et al. 2023 autonomous transportation systems (Zaii
driving (Liu et al. 2024a), energy systems (Zhang et al. 2022 Bui et al.
2025
ct al. 2025), and games [Barros e Sá & Madcira 2025). Offline RL, also known as batch RL, climinates
the reliance on ouline interactions with the environment (Wang el. al. 2022) by directly learning policies from offline datasets composed historical trajectories. Therefore, is promising for applications where interactions are risky and costly. Directly applying dynamic programming- offline RL problems typically learns poor-quality policy due overestimated nut-of-distribution actions caused by distribution shift. (Yamagata et al. 2023 Conventional offline RT generally addresses such issues
online RL methods

---

Published in Transactions on Machine Learning Research (09/2025)
by regularizing policy to stay close to behavior policy (yy et al. 2022 constraining values of out-of- distribution actions Kostrikov et al., 2021 Liu et. al. 2024b).
DT (Chen et al. 2021], an innovative offline RL approach that formulates policy learning as sequence modelling problem, has shown superior performance over conventional offline RL methods on some bench- marks. By leveraging the strong sequence modelling capacity of Transformer structures (Vaswani et al. 2017 Radford al. 2019), , DT models trajectories autoregressively, bypassing the necessity for bootstrapping and learning a value function for given state. Fed with the recent context, historical subtrajectory composed of states, actions, and return-to-go (RTG) (cumulative rewards from the current step), DT predicts actions autoregrossively. However, the abandonment of dynamic programming in DT makes it lack stitching ability (Yamagata et al. 2023). Some recent work studies this problem. For instance, Q-learning DT (Yamagata) jet al.) 2023) relabels RIG with a value function learned from Q-learning, enhancing DT to learn better policies from sub-optimal datasets. Elastic DT (Wu et al.] 2023) adjusts the context length according the evaluated optimality of the context to achieve trajectory stitching. Our work instead addresses such a problem using generalive modelling.
Recently, diffusion models (Ho et al. 2020 Cao et al. 2024 Yan et al. 2024 have gained recognition for their extraordinary ability to generate high-quality complex data, such as images and texts. Additionally, recent. works have also introduced the diffusion model to oflline RT. (Janner et al. 2022 Ajay et al.] 2023 Ho et al.] 2020). Similar to DT, Diffuser (Janner et al. 2022) also works as a trajectory generator, using expressive diffusion models instead of Transformers. Unlike existing diffusion-based offline RL methods, our work leverages diffusion models a refinement tool.
We propose DRDT3, a framework that combines novel DT-stylc trajectory modelling method and conditional diffusion model. Specifically, we first introduce a DT3 module, which harnesses both self-attention and the TTT layer to achieve reduced complexity and improved sequence modelling ability. This module predicts coarse action reprosentations based on recent context, which are then used as prior knowledge or conditions for a denoising diffusion probabilistic model (DDPM) (Ho et al. 2020 Additionally, we present a novel gated MLP noise approximator, designed to capture essential information from both noisy input and coarse action predictions, facilitating the denoising process. To integrate the DT3 module and diffusion model effectively, employ unified optimization objective with two key components: term, which ensures that the DT3 module generates coarse action conditions close to the optimal action distribution, and an action refinement term that constrains the diffusson model to sample actions within the dataset distribution. This approach enables the predicted actions from DT3 to be iteratively refined with Gaussian noise through the denoising chain within the diffusion model, facilitating trajectory stitching.
action representation
The contributions of this work are summarizod as follows:
(i) We introduce the DT3 model, which combines the self-attention mechanism and the TTT layer for improved action generation and enhanced performance over DT.
(i) We further propose DRDT3, diffusion-refined DT3 algorithm that leverages coarse action representa- tions from the DT3 module as priors and uses diffusion models to iteratively refine these actions. Additionally, we present a gated MLP noise approximator for more effective denoising. DRDT3 integrates the DT3 and diffusion model into a cohesive framework with a unified optimization objective.
(iii) Experiments on extensive tasks from the D4RL benchmark (Fu et al., 2020 demonstrate the superior performance of our proposed DT3 and DRDT3 over conventional offline RL and DT methods.
The following sections are organized as follows: Section 2 provides an overview of recent related work on DT, sequence modeling, and diffusion models in RI. We then present preliminaries in Section 3 covering offline RL, DT, the TTT layer, and diffusion models. Our DRDT3 methodology is detailed in Section [ followed by experiments evaluating the proposed models Section 5 Finally, we present conclusions and discussions in Section i
2

---

Published in Transactions on Machine Learning Research (09/2025)
2 Related Work
2.1 Decision Transformer
Offline RL learns policies from historical trajectories pre-collected using other behavior policies. Different from classic offline RL (Fujimoto & Gu 2021
Kostrikov et al. 2021 where dynamics programing is employed formulates the policy learning as a sequence modelling problem by autoregressively generating trajectorics with Generative Pre-trained Transformer (GPT) architectures (Radford et al., 2019). Many variants of DT have been studied and shown competitive performance on some RL benchmarks. Online DT (Zheng et al., 2022) equips DT with entropy regularizers, enabling online fine- tuning with efficient. exploration. Prompting DT (Xu et al., 2022) and Hyper-DT (Xu et al.) 2023) make DT easily transferable to novel tasks using trajectory prompt and an adaptation module initialized with a hyper-network, respectively. Hierarchical DT (Correia & Alexandre! 2023) frees DT from specifying RTG while using sub-goal selection instead. Q-learning DT (Yamagata et al., (023) and Elastic DT (Wu et al, 2023) improve the stitching ability of DT by introducing a dynamic programming-derivee value function and varying-length context, respectively. DT has also been applied to some real-world applications, such as robotics (Corrcia & Alexandre 2023) and traffic signal control (Huang ct al. 2023).
for policy optimization, DT (Chen et al. 2021
2.2 Sequence Modeling
Transformer-based methods leveraging the attention mechanism have demonstrated remarkable performance
across various sequence modelling domains (Vaswani et al. 2017 Radford et al. 2019 Achiam et al. 2023
Peebles & Xie 2023). However, their quadratic time complexity during inference (Gu Dao 2023
poses
challenges for long-soquence modelling. Recently, structured state space models (SSMs) (Gu et 2021
Gu & Dao 2023 have introduced a compelling architecture for long-sequence processing, distinguished by their linear computational complexity. Among SSM- based methods, MAMBA has proven effective and is
widely applied in various research areas (Zhang et al. 2024 Wang et al. 2024). MAMBA (Gu & Dao 2023
introduces selection mechanism to adapt SSM parameters based on input, along with parallel scanming, kernel fusion, and recomputation techniques for efficient computation. Other recent. sequence modelling approaches, such as RWKV (Peng et al.) 2023 2024), Gated Linear Attention (GLA) (Yang et al.) 2023
and xLSTM (Beck el al.) 2024), build on RNN architectures with enhanced expressiveness by utilizing matrix hidden states, unlike the vector hidden states of traditional RNNs. Additionally, TTT layers (uun et al. 2024) further enhance hidden state expressiveness by updating them through self-supervised learning during both training and inference. TTT-Linear and TTT-MLP, whose hidden states are linear layer and an MLP, respectively, achieve linear time complexity with performance that is comparable to, or even surpasses, Transformers and MAMBA.
2.3 Diffusion Models in Reinforcement Learning
Diffusion models (Ho al. 2020) have been appcaling for their strong capacity to generate high-dimensional image text data. In light of the expressive representation and strong multi-modal distribution modelling ability of the diffusion model, some researchers have introduced the diffusion model to RL paradigms. Diffuser (Janner et al. 2022) first employs diffusion models as a planner for generating trajectories in model-based offline RL, which alleviates the severe compounding errors of conventional planners. AdaptDiffuser (Liang et al. 2023 improves the diffusion model-based planner by self-evolving with filtered high-quality senth demonstrations. In addition to employing diffusion models as planners, SynthER (Lu al.] (023). DiffStich et al. 2024), GODA Huang et al. (2024), and PRIDE Feng el. al. (2025) further exploit diflusion models data synthesizers to directly augment either offline or online training data with higher diversity and quality,
Diffusion models have also been adopted to represent policies. Decision Diffuser (Ajay et al. 2023) formulates sequential decision-making problems as conditional generative modeling and introduces constraints and skill conditions. Diffusion-Q2 QL (Wang et al. 2022 explores representing policy as a diffusion model and employing Q-value function guidance during training. It overcomes the over-conservatism of policies Icarned from conventional offline RL. Efficient Diffusion Policy (EDP) (Kang et al. 2023) extends diffusion models policy gradient methods and accelerates training by approximating actions from corrupted samples in single
3

---

Published in Transactions on Machine Learning Research (09/2025)
step. Diffusion Actor-Critic (DAC) Fang al. 2024) adopts an actor-critic training framework, training critic network alongside diffusion-based KT-constrained policy, where soft Q-guidance is used to regularize the policy to remain close to the behavior policy. Ma et al. (2025) develop two diffusion- based online RL algorithms, i.c., Diffusion Policy Mirror Descent (DPMD) and Soft Diffusion Actor-Critic (SDAC), built on two novel reweighted score matching methods. MaxEntDP (Dong et al., 2025 and DIME (Celik et al. 2025) incorporate maximum entropy RI. to enhance online exploration with diffusion-repreeentee policies. DRCORL (Guo et al. 2025) applies diffusiionbased policies constrained RL and introduces gradient operation to balance reward and safety constraints. FDPP (Chen et al., 2025) and PRIDE (Feng et al.) 2025 further explore the application of diffusion models in preference-based RL. Moreover, Clean (Dong et al. 2024) offers a diffusion model library that facilitates the development of diffusion--bar basedd decision peeision-making methods.
Our DRDT3 uses unified framework that aims to enhance a DT--sylee trajectory modeling method by refining the predictions iteratively within the expressive DDPM. In contrast to the above methods, DRDT3 employs the diffusion model solely as action refinement module, directly operating the DT output without multi-stage training. It remains modelicg approach without dynamic programming, avoiding the compounding errors of dynamics models and value cstimation.
3
Preliminaries
3.1 Offline Reinforcement Learning and Decision Transformer
RL trains an agent by interacting with an environment that is commonly formulated as a Markov decision process (MDP): M = (S, A, P. The MDP consists of state s E S, action a E A, reward r = R(s, a): state transition (s a), and discount factor 7 E (0,1) (Sutton & Barto 2018 The objective of RT. is learn policy æ that maximizes the expected cumulative discounted rewards:
J = Ex En'R
''R (sx, at)
(1)
where 8, and ag denote state and action at time f, respectively. Offline RL typically learns a policy from offline datasets composed of historical trajectories collected by other behavior policies, eliminating the need for online interactions with the environment. A trajectory is a sequence of states, actions, and rewards up to time step T:
T (81, 01, .i, 8T, ST,0T,
Or, rT).
(2)
Different from conventional dynamic programming bassad based offline RL methods, DT (Chen et al. 2021
b formu-
lates the policy return-condition sequence model represented by GPT-like architectures (Radford et al. 2019). Instead of generating the action based on a single given state, DT autoregressively predicts actions based on the recent context, a K-step sub-trajectory т including historical tokens: states, actions, and RIGs 9: = = re, where the RTG specifies the desired future return. The trajectory generation process can be formulated
+' = (gt- St-K1, 0- St,
(3)
Specifically, at time step DT predicts a, based on (Q - 9r, st) and repeats the procedure till the end.
3.2 Test-Time Training Layer
Due to the quadratic complexity of sell-attention in processing long-sequence inputs, some recent work leverages the RNN structure for sequence modeling, given its linear complexity. However, RNNs are often constrained by their limited expressive power hidden states. The TTT layer (Sun et al., 2024) aims address this challenge by updating the hidden state on both training and testing sequences with self- supervised learning. Concretely, each hidden state can be expressed W,, the weights of model f that
4

---

Published in Transactions on Machine Learning Research (09/2025)
produces the output f(x; Wt). The update rule for W, is expressed as step of gradient descent certain self-supervised loss &:
W W VVVe
(4)
where y is the learning rate and xe is the input token. loss of Zy.
TTT layer (unn et al. 2024 sets & to reconstruction
e (W; - II (Z,; - xell"
(5)
where ス, = 2x denotes a low-rank projection corrupted input with a learnable matrix 0x. To sclectively remember information in 24, TTT reconstructs another low-rank projection 0yz, instead. Therefore, the self-supervised loss becomes:
& <W; = 11f (0xIι: W) 2v
(6)
To make the dimensions consistent, the output rule becomes 2, = f(0gzz; W.). A sequence model with TTT layers consists of two training loops: the outer loop for updating the larger nelwork and hyperparameters 0q.0k and fy, and the inner loop for updating W within each TTT layer. TTT further makes updating parallelizable through mini-batch TTT and accclerates computation using the dual form technique (Sun et al. 2024).
3.3 Diffusion Model
typically learns exprossive representation of : pe (x"*)) dx1:N, where latent variables x! x'....,>' from timestep = 1 till i = N have the same dimensionality as data x9 ~ q(x"). Diffusion models typically consist of two processes: diffusson or forward process, and reverse process. The forward process is defined as a Markov chain, where the data x" g(x²) is gradually added with Gaussian noise following a variance schedule BI,..,,N:
2020 Diffusion models (Sohl-Dickstein al. 2015 Ho et al.
the data distribution g(x") from dataset in the form Pa (x")
q ((x' x)-1)= = N (x:vV- 3xtt11,111
(7)
The Gaussian transition can be alternatively formulated as x vo + v1 a'6', where at = and et ~ (0, The simplified transition from x0 x² can be inferred that
x' = =oxx VI a'c(x',
(8)
where a' = - a and E(x', N' (0,1) denotes the cumulative noise accumulated over timesteps. The reverse process denoises starting noise sampled from (x^) - (x~;0, back to the data distribution following a Markov chain:
Pe (x--1 x²) = (x'-1: uo (x),1) Eo (x²
(9)
where 0 denotes the learnable parameters Gaussian transitions. Generating data using diffusson models involves sampling starting noises, N p (x*), and implementing reverse process from timestep i N till =.
4 Methodology
In this part, we present the details our DRDT3, Within the unified framework of DRDT3, a diffusion model refines the coarse action predictions from DT3 module by injecting as condition and processing it using gated MLP noise approximator. A unified objective proposed to jointly optimize DT3 and the diffusion model.
5

---

Published in Transactions on Machine Learning Research (09/2025)
DT3 Module
Diffusion Module
a
Noise Approximator
Attention TTT Block
1 x²
f
v1
Val
Attention TTT Block
cK+1
t-++1
Figure 1: An illustration of the inference procedure of DRDT3. The left and right parts show the structures of DT3 and diffusion modules, respectively. During inference, the coarse action representation predicted by the DT3 module serves as the condition and refined iteratively within the diffusson module.
Attention TTT Block
Gated MLP Noise Approximator
Layer
Noise
Linear and Reshape
Prediction Layer
Add & Norm
7
Zt-1
Layer
Scale
Wo
W
We-1
W
Linear
M
Add & Norm
Activation
N
X₁
Xt-1
Xt
Linear
Linear
Masked
Multi-head Attention
2 = (xy; W/)
Scale, Shift
7.8
Layer Norm
W; = W - -V((W--ii))
Concatenate
Input
Input
Timestep
Coarse action
Figure 2: Structure illustration of Attention TTT block, TTT layer, and gated MLP noise approximator.
4.1 Decision TTT
The architecture of the DT3 module is depicted in the left part of Figure 1 We modify the standard DT model by replacing Transformer blocks with proposed Attention TTT blocks for trajectory generation. The structure of the Atlention TTT block is detailed in Subsection 4.1.1 In DT3, each input token from the context is projected into the embedding space through a linear layer, and timestep embeddings are subsequently added. Input contexts with length smaller than & are zero-padded for better generation. With the structure of DT3, actions can be generated autoregressively with known context.
4.1.1 Attention TTT Block
Given the competitive performance and linear complexity of TTT layers (Sun et al. 2024 on sequence modelling, we leverage it for trajectory generation the form of DT. Instead using the GPT-2 model to process a sequence, we propose an Attention TTT block that exploits both self-attention and the TTT layer. As shown in the left part of Figure 2 we apply masked multi-head attention capture correlations between input tokens, followed by an Add & Norm layer. We further employ the TTT layer to extract

---

Published in Transactions on Machine Learning Research (09/2025)
additional information beyond what is captured by the attention mechanism. The structure of the TTT layer is illustrated in the middle part of Figure 2 In our TTT layer, we use a linear function to represent the weights W, resulting in f(z) = Wa, where W is a learnable square matrix. The TTT layer is also followed by an Add & Norm layer.
4.1.2 Coarse Action Prediction
Besides directly generating the final actions autoregressively using DT3, in our DRDT3, we also leverage the DT3 as one of the modules to predict coarse representation of the action, which serves as an action condition for the subsequent diffusion model:
K, := k= To 1 K, Kk,)),
(10)
where denotes learnable parameters of DRDT3, a-k,t is the predicted sequence of actions, s-K,t and g-K,: are the latest K-step RTGs and states, respectively. We use to denote the predicted coarse action representation for the current timestep, which is also the final term in á_
During implementation, initialize the RTG the return of the entire trajectory, and the next RTG is calculated using J: -- - Tt-1- When evaluating our DRDT3 online, we introduce an RTG scale factor to adjust the initial RTG. This is achieved by multiplying the maximum positive relurn from the oflline trajectories by while dividing the negative return by it. This defines the desired higher return for the upcoming evaluation.
4.2 Conditional Diffusion Policy
We adopt DDPM (Ho et al. 2020), a well-known generative model, as an enhancement module to refine the coarse actions predicted from DT3. To achieve this, we extend DDPM a conditional diffusion model and employ coarse action predicted from DT3 as prior, depicted in Figure 1 Consequently, the final action is sampled through reversed process:
To(ala) = po (a (a (a") 11 pe (a'-1 a')
(11)
where the final denoised sample a denotes the sampled action to be implemented. It worth noting that we use subscripts t f1..... T} to denote timesteps trajectorics while using superscripts E fl,..., N) represent diffusion timesteps. We formulate pe (a'-1 a',a) using a Gaussian transition
Po (a i-1 laaaa : = (a - 140 (a), a. Eo (at, a))),
(12)
where the covariance matrix is set as Eg (a a) = I.I. Following DDPM (Ho et al. 2020 we can infer the mean as
a''- 1_1-- a Eo (a', a.1)
po (a', ā, (.:i)== = V
(13)
where the noise prediction model Eg is represented using ncural networks. Therefore, during inference, as shown in Algorithm 2
we first sample N ~ N (0,I). The coarse action can be refined iteratively following the formulation below till we get the final action a
d
1 a' a- Vo
Eo (a', id..a.)) B'c.
(14)
v1- a
where e ~~ ~ (0. when i> 1 while e 0 when 1 for better sample quality (Ho et al., to predict the noise e using neural networks. Therefore, according to DDPM (Ho et al. 2020 the noise approximator eg can be optimized with simplified loss function
2020 :. We choose
Laiff = E B.a. [|c- (Vora vT- viae++,))aa;;)"
(15)

---

Published in Transactions on Machine Learning Research (09/2025)
Algorithm Training of DRDT3
1:
Initialize the parameters of policy
2: repeat
3: Sample ((B-K,SKK, a-k)) from foffline
4: Predict coarse action representations a-k = To (8_K and extract a Compute Las by Equation 16
> DT loss
Sample i ~ ((11NN
Sample ~ -V (0,1)
8: Compute Laiff by Equation 15
Diffuusion loss Unified objective
9: Update policy by minimizing LDRDTS = Lairs + (0)
10: until converged
Algorithm 2 Inference of DRDT3
Sample a (0,1)
for i= N....1 do Predict coarse action representation ă with the DT3 module 4: Sample ~ N (0, if > else E =0 1 a' 1-a' co (a', + 3'c Vo
Coarse action prediction
5:
a'-l=
Vi-ai
b Action refinement
end for Return a the selected action
Final action
where N) and a Doffline-
4.3
Gated MLP Noise Approximator
To fully integrate coarse action predictions into the diffusion model and improve noise prediction accuracy, we introduce a gated MLP noise approximator, as depicted in the right part of Figure 2 The gated MLP noise approximator utilizes adaptive layer normalization (adaLN) (Peebles & Xie 2023) to capture condition- specific information. specifically learns the scale and shift parameters, y and B, for layer normalization, as well scaling parameter a based on the concatenated condition of the diffusion timestep and coarse action prediction. Additionally, we employ a gated MLP block (Dauphin et al.) 2017 De et al. 2024 extract information from the input, enhancing the model's noise reconstruction capabilities. This architecture splits the output from the Scale & Shift layer into two separate pathways, each expanding the feature dimensionality by a factor of M using a linear layer. The GeLU (Hendrycks & Gimpel 2016) lonelineay is applied in one branch, after which the two streams are combined through element multiplication. A final linear layer reduces back the dimensionality, and a residual connection is incorporated facilitate better gradient flow.
4.4 Unified optimization objective
The diffusion loss (0) facilitates refinement for the coarse action representation à and behavior cloning towards the dataset, while it fails to constrain the DT3 module, potentially leading to deviations in coarse action predictions from the data distribution. To alleviate such issues and jointly optimize DT3 and the diffusion model, we propose a unified optimization objective. This involves introducing a DT3 loss as additional guidance during the training of the diffusion model
Lat3 = K Kamax Bla.s.g) [|la_ - K, 7
(16)
where a_ is the ground truth of the latest K-step actions, and amax: the maximum action in the action space, serves as a scaling factor to render the L1 loss dimensionless. We choose L₃ loss over L2 loss since the former is more robust to outliers and provides stable gradients within the unified objective in Equation
8

---

Published in Transactions on Machine Learning Research (09/2025)
without weakening the inffuence of the diffusion loss. The directional information provided by L1 loss further enhances the effectiveness of coarse-to-fiee refinement. It. is worth noting that the diffusion model only conditions on the last term ă within the sequence a-K,t; whereas the DT3 loss is computed for the entire sequence.
17
The final unified loss for DRDT3 is a linear combination of the DT3 loss (action representation term) and diffusion loss (action refinement term)
CDRDTS == +CarsO))(Es(()))
(17)
where Ç is a loss coefficient for balancing the two loss terms. Note that the DT3 and diffusion modules are jointly trained in single stage by minimizing LDRDTS.
4.5 Implementation Details
Algorithm and 2 illustrate the training and inference procedures of DRDT3. When implementing our proposed DRDT3, we train it for 50 epochs with 2000 gradient updates per epoch. The learning rate and batch size are designated as 0.0003 and 2048, respectively. To procoed with historical subtrajectorics with DT3 module, we set. the context length as 6. The Atlention TTT block used in the DT3 module consists of L-layer self-attention and L-layer TTT with embedding dimensions of 128. We use a linear layer to output deterministic coarse representation of the action. To enhance computational efficiency without sacrificing performance, the number of diffusion timesteps is set to 5. We set the variance schedule according to Variance Preserving (VP) SDE (iiao et al. 2021 ::
=1 - (x) Bmin)) N
(18)
Most of the hyperparameters are selected through the Optuna hyperparameter optimization framework (Akiba et al. 2019).
5
Experiments
We conduct experiments evaluate our proposed DRDT3 the commonly used D4RL benchmark (uu
et al. 2020 using an AMD Ryzen 7 7700X 8-Core Processor with single NVIDIA GeForce RTX 4080 GPU.
5.1
Experimental Settings
5.1.1
Tasks and Datasets
We adopt four Mujoco locomotion tasks from Gym, which are HalfChoctah, Hopper, Walker2D, and Ant, and a goal-reaching task, AntMaze. We further evaluate our methods on more challenging tasks, including the Pen and Door manipulation tasks from the Adroit benchmark, which involve a 24-DoF robot hand, as well as Humanoid locomotion tasks from the Gym environment featuring 348-dimensional observations and 17-dimensional actions. For the Gym tasks, dense rewards are provided, and use three different dataset settings from D4RI. (Fu et al.) 2020): Medium (m), Medium-Replay (mr), and Mediun-Expert (me). Specifically, Medium datasets contain one million samples collected using a behavior policy that achieves 1/3 scores of an expert policy; Medium-Reply datasets contain the replay buffer during training a policy till reaching the Medium score; Bodium--Expert datasets consist of two million samples evenly generated from medium agent and an expert agent. The AntMaze task aims to move an ant robot to reach target location with sparse rewards (1 for reaching the goal, otherwise 0). Following experimental settings in DT variant (Zheng et al. 2022). the Pen and Door tasks, we use the human and cloned datasets from D4RL. For the Humanoid tasks, we adopt three offline datascts provided by Bhargava et al. (023): Medium, Medium-Expert and Expert.
adopt the umaze and umaze-diverse datasets for evaluation. For
5.1.2 Baselines
To verify the effectiveness of our proposed DRDT3, compare it with several baseline methods.

---

Published in Transactions on Machine Learning Research (09/2025)
Table 1: Normalized scores of different offline RL methods on Gym and AntM tasks from D4RL bench-
mark. Results of DRDT3 represent the mean and variance across three seeds. The best scores are highlighted in bold.
Dataset
Environment
BC
TD3+BC
CQL
IQL
DT
ConDT
QDT
EDT
DRDT3 (ours)
HalfCheetah
42.6
48.3
41.0
47.4
42.6
43.0
42.3
42.5
44.121.3
Medium
Hopper
$2.9
59.3
58.5
66.3
67.6
74.5
66.5
63.5
92.718.8
Walker2d
75.3
83.7
72.5
78.3
74.0
72.8
67.1
72.8
8..2+1188
HalfCheetah
42.310..5
Medium-Replay
Hopper
18.1
60.9
95.0
94.7
82.7
95.1
52.1
89.0
92.7++2.1
Walker2d
26.0
77.2
73.9
66.6
72.5
58.2
85.0#1.88
HalfCheetah
55.2
90.7
91.6
86.7
86.8
92.6
79.0
89.1
94.2:22.1
Kodium-Expert
Hopper
52.5
105.4
91.6
107.6
110.3
108.7
112.5+1.5
Walker2d
107.5
110.1
108.8
109.6
108.1
109.1
101.7
106.2
1-4.5
Average
51.9
75.3
77.6
77.0
74.7
79.0
66.3
76.0
83.6
Medium
99.9
97.9
103.3:55.8
Medium Medium-Replay Replay
Ant
91.2
89.1
92.0
96. 98.7:55.1
Average
95.6
91.1
95.0
100.0
Umaze
Antmaze
54.6
78.6
74.0
87.5
59.2
67-2
76.6+
Umaze- Diverae
Antmaze
15.6
71.4
84.0
62.2
53.0
62.1
82, 5.5..3
Average
50.1
75.0
79.0
74.9
56.1
64.7
79.6
Classic offtine R.L algorithms:
Behavior Cloning (BC): A supervised learning approach that trains a policy to mimic the behavior of expert demonstrations by learning from state-action pairs.
TD3+BC (Fujimoto & Gu 2021 : It adapts the TD3 method (Fujimoto et al. 2018 by incorporating
behavior cloning regularization term into the policy objective. This modification aims to prevent the policy from generating actions that deviate significantly from the offline dataset distribution.
penalizes the Q-values associated with out-of-distributiin actions, thereby encouraging more conservative Q-function estimates.
Conservative Q-Lcarning (CQL) (uumar et al. 2020
Implicit Q-Learning (IQL) (Kostrikov et al. 2021 IQL approximates the upper expectile of Q-value distributions, allowing it to learn policies focusing actions within the dataset distribution.
DT-based algorithms:
DT CCen et al. 2021): DT formulates the policy learning as sequence modelling problem by autore- gressively gencrating trajectorics using GPT models.
Contrastive DT (ConDT) (Konan et al. 2023): Tt harnesses the power of contrastive representation
learning and return-dependent transformations cluster the input embeddings based on their associated returns.
Q-learning DT (QDT) Yamagata et al.
2023): It improves the stitching ability of DT by introducing
dynamic programming-derivee value function.
Elastic DT (EDT) (Wu et al. 2023 the quality of the previous trajectory.
dynamically adjusts the length of the input context according
Diffusion model-based RL methods:
Synther Lu et al.
2023 A diffusion model-based data augmentation method that directly samples
synthetic offline data. We use TD3+BC as the evaluation algorithm on the augmented datascts.
Diffuser (Janner et al. 2022 A diffusion-based amgorithm that adopts diffuusio models as a planner for generating trajectories in the form of model-based offline RL.
Decision Diffuser (DD) (Ajay et al., 2023): A sequence modeling method that samples future states with diffusion model and infers actions inverse dynamics.
10

---

Published in Transactions on Machine Learning Research (09/2025)
Table 2: Performance comparison DRDT3 and diffusion model- based methods on Gym tasks. While DRDT3 does not surpass the baselines on individual tasks, it achieves the highest average normalized score across all tasks.
Dataset
Environment
SynthER
Diffuser
DD
AdaptDiffuser
EDP
DRDT3 (ours)
HalfCheetah
48.3+0.0
43.820.1
45.3=0.3
41.30022
50,8=0.0
41.11.3
Medium
Hopper
51.910.1
89.510.7
98.2.10.1
95.5.11.1
72.610.2
92.718.8
Walker2d
86.620.0
79.4+1.0
79.6=0.9
83.81
86.520.2
84.24 1.8
HalfCheetah
43.410.0
36.010.7
42.9
36.710..8
44.910.4
Medium-Replay
Hopper
24.7=0.1
91.8+0.5
99.2:1..2
91.24:0.1
83.0+
92.7422.1
Walker2d
88.6#0.4
68.341.8
76.6=0.6
82.9+1.3
87.0+2.6
86.04188
HalfCheetah
94.8:00.0
90.340.11
88.9-1.9
90 00.4-10.1
95.8
94.2412.1
Medlum-Expee Expertt
Hopper
76.6#0.4
107. 240.9
110.4+0.6
0.9
110.8:0..4
112.5+1.5
Walker2d
110.0:10.0
107.4:0..1
108.4.0.1
107.740.1
110.4=0.0
6.4
Average
69.4
78.2
83.2
82.4
82.4
83.6
Table 3: Performance comparison on challenging robot manipulation tasks, i.e., Pen and Door, and the Humanoid task with high-dimensional observations.
Tasks
BC
CQL
DRDT3 (ours)
pen-human-vl
25.8
35.2
72.4
78.3#5.7
pen-cloned-vl
38.3
27.2
40.2
57.0119..8
door-human-vl
0.5
19.8+6..3
door-cloned-vl
-0.1
0.4
3.7
16.6:15.2
Average
16.1
18.2
31.4
42.9
humanoid-medium
13.2
49.5
23.7
45.9:14.4
humsnoid-medlum-eeee
13.7
54.1
52.6
02.2:44.2
humanoid-expert
39.1+22.0
Average
17.0
55.8
43.2
55.7
AdaptDiffuser (Liang et al., 2023):
improves the diffusion model-based planner by self-evolving
with filtered high-quality synthetic demonstrations.
EDP (Kagg et al., 2023) : A diffusion-based policy gradient method that accelcrates training by approximaling actions from corrupted samples in single step.
The best scores reported in papers of these baselines are adopted. For ConDI,, we calculate the normalized score according to the returns reported in (Konan et al. 2023 For diffusson- based methods, we adopt reproduced results in CleanDifluser (Dong el al. 2024).
5.2 Performance Comparison
Table presents the normalized scores Fu et al. 2020) of our DRDT3, classic offlnne RL and DT-based baselines on several tasks from the D4RT. benchmark. As can see, DRDT3 significantly improves the average scores on both Gym and AntMaze tasks. Specifically, DRDT3 outperforms other offline RL methods, including the state-of-the-art DT variants, on out of 11 Gym tasks. For the other 4 Gym tasks, DRDT3 is still comparable over the best baselines. Particularly, the Medium--Fxper dataset not reported for the Ant task, given the lack of comparing basclines evaluated on For AntMaze tasks, DRDT3 shows comparable performance over the best baselines for each single task while achieving the best overall results. When compared with the vanilla. DT, DRDT3 presents significant improvements 12 out of 13 tasks, especially on tasks using Medium and Modium-RRplan datascts with sub-optimal trajectories, which demonstrates the enhanced stitching ability of our DRDT3 over DT.
We further evaluate DRDT3 by comparing with recent diffusion model-basee methods. As shown in Table g
DRDT3 achieves superior performance on only one individual task, yet maintains consistently strong results across all tasks, with only a small gap the best-performing method. Overall, it attains the highest average normalized score among all diffusion-based methods, underscoring its robustness. This outcome may
11

---

Published in Transactions on Machine Learning Research (09/2025)
halrchhetah-medilu-eepertty
hegpermellum-experrty
waller2d--eellmmeepprrr
Normalized Score
100
75
50
w/
25
Attention MAMBA
DRDT3
0
30 40
50
40
Training Step (# 2k steps)
Training Step 2k steps)
Training Step (# 2k steps)
Figure 3: Ablation study sequence model.
Table 4: Performance comparison between DT, DT3 and DRDT3 on Gym locomotion tasks. The underlined values indicate performance better than DT, and bold values are the best results. the': halfcheetah; 'hp': hopper; 'wk': walker2d.
he-m
hp-m
wk-m
he-mr
hp-mr
wk-mm
ho-me
hp-me
wk-me
Average
42.6
67.6
36.6
82.7
86.8
107.6 6
108.1
DT3 (ours)
12 6+0.77
73.0
79.3+1.1
39.9#0.1
75.9=4.4
63.
93.4=2.7
109.9
101.348.0
75.8
DRDT3 (ours)
44.1#113
92.7=8.8
84.2=118
42.3=0.5
92.7#2.1
85.0+1.8
94.2#2.11
112.5=1..5 101.566..4
83.6
stem from the fact that DRDT3 is inherently constrained by the performance of its underlying DT module, as it directly refines DT-generated actions without incorporating dynamic programming. Conscquently, DRDT3's performance is steady rather than outstanding individual tasks.
Table presents the evaluation performance of DRDT3 on more challenging tasks, including the Pen and Door tasks from the Adroit benchmark and the Humanoid task from Gym. We report results for BC, CQL, and DT as baselines, since other methods are rarely evaluated on these tasks. As shown, DRDT3 consistently outperform the baseline methods on most Pen and Door datasets, with DRDT3 achieving an average improvement of 36.6% over DT. For the Humanoid task, DRDT3 attains performance comparable to the best baseline, CQL, while still delivering a 28.9% improvement over DT. The underperformance observed certain tasks likely stems from inherent limitation of sequence modeling methods, which lack dynamic programming capabilities to effectively extrapolate toward higher performance.
Our primary goal, however, is to demonstrate that diffusion models can effectively refine DT DT-generated actions and enhance DT's trajectory-stitching capability- an improvement our experiments confirm across a wide range of DT-based baselines and evaluation tasks. particular, DRDT3 achieves average improvements of 11.9%, 9.4%, 41.9%, 36.6%, and 28.9% over DT on Gym locomotion, Ant, AutMaze, Adroit, and Humanoid tasks, respectively.
5.3 Ablation on Sequence Model
We compare our DRDT3 framework with two variants: one where the sequence model is replaced by a GPT-2 model (Rafford et al. 2019) and another where the TTT layer is replaced by an MAMBA layer (Gu & Dao 2023 The GPT-2 model relies entirely on the attention mechanism, while MAMBA is a selective state space model based on the RNN structure, where parameters are dynamically dependent on the inputs. For the two variants w/GPT-2 and w/ Attention MAMBA, we use one block per model as our DRDT3. To ensure fair comparison, adjust the number of layers in both the GPT-2 and MAMBA variants to match
Table 5: Performance comparison between DT3 and DT on Ant and AntMaze tasks.
Ant-m
ant-mr
Average
antmaze-umaze
aatmase-umaze-divese
Average
DTT DT3 (ours)
93.6
80.1
91.4
50.2
53.0
56.1
96.7 7.3
5.2
62.4=3.3
63.0=:4.8
57.7
DRDT3 (ours
103.325..8
96.7
100.0
76.641..
82.5:16.3
79.6
12

---

Published in Transactions on Machine Learning Research (09/2025)
Table 6: Performance comparison of DT, DT3, and DRDT3 across Adroit and Humanoid tasks.
Method
pen-h
pen-c
door-h
door-c
Avg.
human--
humant-ne
human-c
Avg.
DT
72.4
40.2
3.7
23.7
52.6
53.4
DT3 (ours)
79.0:2222
38. 08.91120.11
12.746.2
3.74
33.6
5..
51.54:2.2
54.01:2.3
44.9
DRDT3 (ours)
78.3
57.0.119.4 19.8
19.8
16.6.5.2
42.9
45.911.4
62.2 4.2
59.1122
55.7
halltheetalh-eddimm-2
hopger-necium-22
walker2d-medium-2
Nomalized Score
100
DT3
DRDT3
50
25
10
20
30
40
50
10
20
30
40
50
0
10
20
30
40
50
Training Stap (# 2k steps)
Training Stop (# steps!
Training Stop (# 2k steps)
hosper-mediun-replgee
wallee2dd-edum--elllya
Normalized Score
100
75
50
50
20 Training Stop steps)
10 Training Stop stops)
Taining Step stops)
haltcheetaa--mediimeeeerrr
hogper-meeduum-expnnt222
wellerr2-mmediumexxprtt-
Normalized Score
50
25
10
20
30
40
50
10
20
30
40
50
0
10
70
30
50
Training Step (# 2k steps)
Training Stop (# steps)
Training Stop (# 2k steps)
Figure 4: Learning curves DT, DT3, and DRDT3 on Gym tasks. For clarity, each curve is plotted with a moving average (window size 10) to highlight the trend.
the model size of our DRDT3. Specifically, the total number of learnable parameters for our DRDT3, the variant w/GPT-2, and the variant w/ Allention MAMBA are 235K, 218K, and 329K, respectively.
As shown in Figure 3 our DRDT3 with Attention TTT blocks outperforms the other variants across three Medium-Exxee Expert tasks. The variant with Attention MAMBA shows slight performance degradation and lacks the stability of DRDT3. The variant with GPT-2 performs the worst across all tasks, which might be attributed to the typical reliance larger model sizes of GPT-2, When constrained to similar size as the other two variants, GPT-2 experiences performance degradation. These results indicate that our proposed DT3 module with Attention TTTT blocks enhances performance without increasing model size.
5.4 Ablation on Diffusion
To test whether the diffusion module plays significant role in action refinement and verify the effectiveness of our DT3 module, we train DT3 model along with the same settings DRDT3 by minimizing the Lats in Equation 16. Table 5 and 0 show the performance comparison between DT and our DT3 model on Gym, AntMaze, and challenging robotic (Adroit and Humanoid) tasks. Figure illustrates the learning curves of the three hods on Gym locomotion tasks.
As we can see, our proposed DT3 trained independently can outperform DT on out of Gym locomotion tasks, all AntMaze tasks, out of Adroit tasks, and 2 out of Humanoid tasks. This demonstrates the superiority of our proposed DT3 with the Attention TTT blocks the vanilla DT with only the attention
13

---

Published in Transactions on Machine Learning Research (09/2025)
Table 7: Model size and average training and inference time comparison. # param. denotes the number of learnable parameters. Training time refers to the average time required for one epoch of model training, while inference time indicates the time needed evaluate the policy for single episode.
Model Size # param. (K)
Hopper-Medium
MMIChhetaa--Mdiiue
Malker2d-Medingg
Training (8) Inference (s)
Training (*)
Inference (8)
Training (s)
Inforence (s)
DT
226.8 234.6
162.2
1.7
258.7
2.6
239.1
2.6
DT3
134.3
2.6
245..
4.2
223.9
3.1
w/o adaLN
w/o gated MLP
w/o both
Noise
Noïse
Prediction Layer
Prediction Layer
Prediction Layer
Scale
Linear
Linear
Linear
Activation
x N
Activation
Activation
x N
Linear
Linear
Linear
Linear
Y.8
Scale, Shift
Layer Norm
Layer Norm
Layer Norm
MLP
Concatenate
Concatenate
Concatenate
Input
Timestep
Coarse action
Input
Timestep
Coarse action
Input
Timestep
Coarse action
Figure 5: Structure illustration of three noise approximators. Left: w/o adaLN; Middle: w/o gated MLP; Right: w/o both.
mechanism. Furthermore, our proposed DRDT3, which enhances DT3 through a diffusion module, further boosts performance across all tasks with different datasets, achieving average improvements of 10.3%, 7.6%, 38.0%, 27.7%, and 24.1% on Gym locomotion, Ant, AntMaze, Aroit, and Humanoid tasks, respectively. This demonstrates the substantial impact of incorporating diffusion refinement in DRDT3.
However, underperformance is observed in the walkor2d-me task. As shown in Figure 4 we degradation in the learning curve training iterations increase, indicative of overfitting to the dataset's 's mixed-quality dynamics. Specifically, as training progresses, the DT3 module might memorize noisy or erratic details from medium-quality trajectories rather than generalizing to the smooth, energy-efficient patterns of expert demonstrations. This leads to increasingly misaligned coarse actions, which the diffusson module cannot fully correct. These may result degraded policy performance on this task.
slight
5.5 Computational Efficiency
We evaluate the computational efficiency of our proposed DT3, which consists of Attention-TTT blocks, by comparing against DT, which composed solely of Attention blocks. Table reports the model sizes along with the average training and inference times. As shown, DT3 consistently achieves faster training but lails to surpass DT in inference speed. This is consistent with the findings the original TTT paper (Sun) et al. 2024), which reported that TTT layers offer clear advantage over Transformers during training and become more efficient at inference when processing long contexts. In our case, the sequence length is only 6, relatively short context, which prevents DT3 from benefiting from TTT's inference-tine speed advantage.
5.6 Ablation on Noise Approximator
We also verify the effoctiveness of our gated MLP noise approximator by comparing DRDT3 with several variants, shown in Figure 3 one where the gated MLP structure replaced by a simple MLP, another where the adaLN is replaced with in-context conditioning (Peebles & Xie 2023), and third variant that
14

---

Published in Transactions on Machine Learning Research (09/2025)
Table 8: Ablation studies on the noise approzimator (left side) and unified loss function (right side).
Dataset
w/o adaLN
gated
w/o both
DRDTS (ours)
L2 Lasa w/o Cass
ballcheetab-m
13.1216
11.01.5
43.841.2
41.1#1.3
11.1#1.4
12.541.3
hopper-m
91.3214.9
77.315.2
75.416.0
92.718.8
81.218.2
92.015.2
walker2d-m
$1.0#110
$3.1+1.1
82.5+1.8
81.211.8
80.3+2.1
79.8+1.2
hallcheetab-mr
42.720.8
42.820.9
44.410.8
12.340.5
42.720.9
38.24:0.9
hopper-mn
92.71:1.3
86.9111.5
85.9.15.1
92.71:2.1
84.24:5.4
91.5:11.77
walker2d-mr
85.542.8
82.2+2.77
80.7+00.9
85.0+188
85.1+11.3
86.0+t2.9
balfcheetah-mee
93.8#2.5
89.8:3.11
90.3+2.5
94.2#2.1
92.3+2.6
93.3+3.0
hopper-me
111.91177
106.611.5
103.941.3
112.5+1.5
105.121.6
110.321.5
waller2d-me
102.611.3
101.111.1
98.415.2
104.516.4
100.7:55.6
103.011.3
Average
83.1
79.3
78.6
83.6
79.5
81.8
HalfCheetah-Medllm-.
HoppeerMedllm--2
Walker2D-Medium-V2
100
Normalized Score
60
40
zeta = 0.05
zeta = 0.1
zeta = 0.2
zeta 0.3
zeta 0.5
zeta m 0.8
zeta = 1.0
Figure Sensitivity analysis on the loss coefficient Ç. We empirically choose 0.2 according the performance comparison.
removes both structures. As shown in T the removal of adaLN causes slight performance degradation, while removing the gated MLP results in significant. reduction in performance.
5.7 Ablation on Loss Function
To verify the effectiveness of our proposed unified optimization objective, compare our loss function with the vanilla diffusion loss function (w/o Las) and unified loss function incorporating L2-based DT3 loss term (w/ L2 Las). As shown in Table 8 the variant w/o Las shows worse performance than the variant with an extra T.2-based DT3 constraint. Our DRDT3, utilizing an L1-based DT3 loss term instead, can further enhance performance beyond the L2 variant. These results demonstrate that incorporating DT3 loss into the final loss improves performance, with the L1 Cas offering additional gains over the L2 variant.
5.8 Sensitivity Analysis on Loss Coefficient
We further vary the loss coefficient < across different values to select the best one empirically, Figure illustrates the normalized seores of our DRDT3 with varying Ç and trained on three Gym tasks with Medium datasets. We empirically set ( = 0.2 based on the results.
6 Conclusion and Discussion
This work introduces the DRDT3 algorithm to enhance the stitching ability of the D'T-based method. We propose novel DT3 modulc with attention TTT blocks to process historical information and predict a better coarse action representation. The diffusion module then conditions this coarse representation and iteratively refines the action using a novel gated MLP noise approximator, improving performance on sub-optimal datascts. To achieve joint lcarning of the DT3 and diffusion modules, we introduce unified optimization objective, consisting of both an action representation term and an action refinement term. Experiments on a variety of tasks with different dataset configurations from the D4RL benchmark demonstrate the effectiveness
15

---

Published in Transactions on Machine Learning Research (09/2025)
and superior performance of DRDT3 compared to conventional offine RL methods and DT-based approaches. Experiment results also demonstrate that DT3 can outperform the vanilla DT.
Nonetheless, we acknowledge that there remains performance gap between our DRDT3 and certain state- of-the-art dynamic programming- assisted diffusion-based policies. This gap likely arises because DRDT3 is inherently constrained by the performance of the underlying DT3 module, as directly refines the actions generated by DT3. Therefore, we plan further investigate strategies narrow this gap in the future, aiming to bring DT-based methods closer to the performance of their dynamic programming based counterparts.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lgge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 1:Xtt:2308.08774, 2023.
Amurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is condi- tional generative modeling all you need for decision-making? ar preprint 0rXiv:2211.1663 2023.
Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A next- generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD interna- tional conference on knowledge discovery & data mining, pp. 2623 2631, 2019.
Gabriel Caldas Barros e Sá and Charles Andrye Galvão Madeira. Deep reinforcement learning in real-time strategy games: systematic literature review. Applied Intelligence, 55(3):243, 2025.
Maximilian Beck, Korbinian Pöppel, Markus Spanring, Andreas Auer, Oleksandra Prudnikova, Michael Kopp, Günter Klambauer, Johannes Brandstetter, and Sepp Hochreiter. xlstm: Extended long short-term memory, arXiv preprint 2rXiv:2405.04511 2024.
Prajiwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. When should we prefer decision transformers for oflline reinforcement learning? ar preprint or arxi::2305. 14550, 2023.
Van-Hai Bui, Sina Mohammadi, Srijita Das, Akhtar Hussain, Guilherme Vieira Hollweg, and Wencong Su, A critical review of safe reinforcement learning strategics in power and energy systems. Engineering Applications of Artificial Intelligence, 143:1 14:110099, 2025.
Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng- Ann Heng, and Stan Z Li. A survey on generative diffusion models. IEEE transactions on knowledge and data engineering, 36(7): 2814 2814-2830, 2024.
Onur Celik, Zechu Li, Denis Blessing, Ge Li, Daniel Palenicek, Jan Peters, Georgia Chalvatzaki, and Gerhard Neumann. Dime: Diffusion-based maximum entropy reinforcement learning, arXio preprint orXiv:2502.02333 2025.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Ar- avind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15044-15099 2021.
Yuxin Chen, Devesh K Jha, Masayoshi Tomizuka, and Diego Romeres. Fdpp: Fine-tune diffusson policy with human preference. arXiv preprint 2xf:22501..0250, 2025.
André Correia and Luís A Alexandre. Hierarchical decision transformer. In 2023 TEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pp.
IEEE, 2023.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolu- tional networks. In International conference on machine learning, pp. 933 941. PMI.R, 2017.
16

---

Published in Transactions on Machine Learning Research (09/2025)
Soham De, Samuel L Smith, Amushan Fernando, Aleksandar Botev, George Cristian-Muraru, Albert Gu, Ruba Haroun, Leonard Berrada, Yutian Chen, Srivatsan Srinivasan, et al. Griffin: Mixing gated linear recurrences with local attention for efficient language models, feb. 2024. URL http://arzie. org/abs/24 20.s/2202.199 19427 v1, 2024.
Xiaoyi Dong, Jian Cheng, and Xi Sheryl Zhang. Maximum entropy reinforcement learning with diffusion policy. orXiv preprint arXiv:2502. 1 11612, 2025.
Zibin Dong, Yifu Yuan, Jianye Hao, Fei Ni, Yi Ma, Pengyi Li, and Yan Zheng. Cleandiffuser: An easy-to-use modularized library for diffusion models in decision making. Advances in Neural Information Processing Systems, 37:86899-869988 2024.
Linjiajie Fang, Ruoxue Liu, Jing Zhang, Wenjia Wang, and Bing- Yi Jing. Diffusion actor-critic: Formulating constrained policy itcration as diffusion noise regression for offline reinforeement learning. arXiv preprint arXiv:2405.20533 2024.
Jingjing Feng, Lucheng Wang, Alona Tenytska, and Bei Peng. Sample-effciiett preference-based reinforce- ment learning using diffusion models. In The Seventeenth Workshop on Adoptive and Learning Agents, 2025.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data- driven reinforcement learning. arXiv preprint arXiv:2004.07219, 2020.
Scott Fujimoto and Shixiang Shane Gu. A minimalist approach to offline reinforcement learning. Advances in neural information processing systems, 34:20132-20145, 2021.
Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pp. 1587-1596. PMLR, 2018.
Albert Gu and Tri Dao. Mamba: Linear- time sequence modeling with selective state spaces, arXiv preprint arXiv:2312.00752 2023.
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. Xiv preprint 0Xxx:2111.00390 2021.
Junyu Guo, Zhi Zheng, Donghao Ying, Ming Jin, Shangding Gu, Costas Spanos, and Javad Lavaei. Reward- safety balance in offline safe rl via diffusion regularization. arXio preprint arXiv:2502.125: 2025.
Siyuan Guo, Lixin Zou, Hechang Chen, Bohao Qu, Haotian Chi, Philip S Yu, and Yi Chang, Sample efficient offlne--o-online reinforcement learning. IEEE Transactions on Knowledge and Data Engineering, 36(3): 1299 1310, 2023.
Dan Hendryeks and Kevin Gimpel. Gaussian error lincar units (gelus). arXiv preprint arXiv: 10bo:666..00 08415. 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in newral 6851, 2020.
information processing systems,
Xingshuai Huang, Di Wu, and Benoit Boulet. Traffic signal control using lightweight transformers: An offlie--ooonline rl approach. arXiu preprint orXiv:2312.077000 2.0795, 2023.
Xingshuai Huang, Di Wu Member, and Benoit Boulet. Coal-conditioned data augmentation for offline reinforcement learning. arXiv preprint 12. 205101 2024.
Michael Janner, Yilun Du, Joshua. B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arviv preprint arXic:2205.000 099999 2022.
Yuanyuan Jia, Podro Miguel Uriguen Eljuri, and Tadahiro Taniguchi. bayesian reinforcement learning method for periodic robotic control under significant uncertainty. In 2023 IEEE/RSJ International Con- ference on Intelligent Robots and Systems (TROS), pp. 384-30 491. IEFF, 2023.
17

---

Published in Transactions on Machine Learning Research (09/2025)
Bingyi Kang, Xiao Ma, Chao Du, Tianyu Pang, and Shuicheng Yan. Efficient diffusion policies for offiine reinforcement learning. Advances in Neural Information Processing Systems, 36:67195-67212, 2023.
Sachin G Konan, Esmaeil Seraj, and Matthew Gombolay. Contrastive decision transformers. In Conference on Robot Learning, pp. 2159-2169. PMLR, 2023.
Nya Kostrikov, Rob Fergus, Jonathan Tompson, and Ofrr Nachum. Offline reinforcement learning with fisher divergence critic regularization. In International Conference on Machine Learning, pp. 5774 5783. PMLR, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offiiee reinforce- ment learning. Advances in Newral Information Processing Systems, 33:1179-1191, 2020.
Guanghe Li, Yixiang Shan, Zhengbang Zhu, Ting Long, and Weinan Zhang, Diffstitch: Boosting oflline reinforcement learning with diffusion based trajectory stitching. arXiv preprint 0rXiv2440.00438, 2024.
Zhixuan Liang, Yao Mu, Mingyu Ding, Fei Ni, Masayoshi Tomizuka, and Ping Luo. Adaptdiffuser: Diffusion models as adaptive self-cvolving planners. arXiv preprint orXic:2302.01877 2023.
Jia Liu, Yunduan Cui, Jianghua Duan, Zhengmin Jiang, Zhongming Pan, Kun Xu, and Huiyun Li. Rein- forcement learning- based high-speed path following control for autonomous vehicles. IEEE Transactions on Vehicular Technology, (3(6))760337611 2024a.
Jie Liu, Yinmin Zhang, Chuming Li, Yaodong Yang, Yu Liu, and Wanli Ouyang. Adaptive pessimism via target v-value for offline reinforcement learning. Neural Networks, 180:106588, 2024b.
Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. Advances in Newral Information Processing Systems, 36:46323 06-46333-46344, 2023.
Jiafei Lyn, Xiaoteng Ma, Xiu Li, and Zongqing Lu. Mildly conservative glearning for offline reinforcement learning. Advances in Neural Information Processing Systems, 35:1711 1724, 2022.
Haitong Ma, Tianyi Chen, Kai Wang, Na Li, and Bo Dai. Efficient online reinforcement learning for diffusion policy, arXiv preprint arXiv:2502.0033 2025.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 4195 4205, 2023.
Bo Peng, Eric Alcaide, Quentin Anthony, Alon Albalak, Samuel Arcadinho, Stella Biderman, Huanqi Cao, Xin Cheng, Michael Chung, Matteo Grella, et al. Rwkv: Reinventing rmns for the transformer era. arXiv preprint Xiv:2300 13048, 2023.
Bo Peng, Daniel Goldstein, Quentin Anthony, Alon Albalak, Eric Alcaide, Stella Biderman, Eugene Cheah, Toddy Ferdinan, Haowen Hou, Przemyslaw Kazienko, et al. Eagle and finch: Rwkv with matrix-valued states and dynamic recurrence. arXiv preprint XXi::244400589, 2024.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Tlya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pp. 2256-2265. PMLRR, 2015.
Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang. Sanmi Koyejo, al. Learning to (learn at test timc): Rnns with expressive hidden states. arXiv preprint orXio:2407.0400 2024.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
18

---

Published in Transactions on Machine Learning Research (09/2025)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informalion processing systems, 30: 6000 6010, 2017.
Zhendong Wang, Jonathan J Hunt, and Mingyuan Zhou. Diffusion policies as expressive policy class for offline reinforcement learning. arXiv preprint orXiv:2208.0611 2022.
Zihan Wang, Fanheng Kong, Shi Feng, Ming Wang, Xiaocui Yang, Han Zhao, Daling Wang, and Yifei Zhang, Is mamba effective for time series forecasting? arXiv preprint arXiv:208.11114, 2024.
Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in neural information processing systems, 36:18532 18550, 2023.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arxi2:1 20n.21122000 11..010000 2021.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In international conference on machine learning, pp. 24631-24645. PMLR, 2022.
Mengdi Xu, Yuchen Lu, Yikang Shen, Shun Zhang, Ding Zhao, and Chuang Gan. Hyper-decision transformer for efficient online policy adaptation. arXiv preprint arXie:2304.08459 2023.
Taku Yamagata, Ahmed Khalil, and Raul Gantos-Rodrighe Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in oflline rl. In International Conference on Machine Learning, pp. 38989-39339 39007. PMLR, 2023.
Jing Nathan Yan, Jiatao Gu, and Alexander M Rush. Diffusion models without attention. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognation, pp. 8239-8249, 2024.
Songlin Yang, Bailin Wang, Yikang Shen, Rameswar Panda, and Yoon Kim. Gated linear attention trans- formers with hardware-officient training. arXiv preprint arXio:2312.0660 2023.
Ziyang Zhai, Ruru Hao, Boyang Cui, and Siyi Wang. Hgat and multi-agent rl-based method for multi- intersection traffic signal control. IEEE Transactions on Intelligent Trunsportation Systems, pp. 6848 6864, 2025.
Hanwei Zhang, Ying Zhu, Dan Wang, Lijun Zhang, Tianxiang Chen, Ziyang Wang, and Zi Ye. A survey on visual mamba. Applied Sciences, 14(13):5683, 2024.
Huiliang Zhang, Di Wu, and Benoit Boulet. Metaems: A meta reinforcement learning based control frame- work for building energy management system. arXie preprint arXiv:2210. 100:2200. 2022.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning, pp. 27042 27059. PMLR, 2022.
19


Reviews:
[{"review_id": "tXwXdz9AN9", "reviewer": "Reviewer_Kk15", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. DRDT3 addresses the key limitation of DT—its poor stitching ability—through a clear and technically justified two-stage approach.\n2. Integrating TTT layers into a DT-like model is an effective and original idea for improving sequence modeling under compute constraints.\n3. Using diffusion models not as full trajectory generators but as action-level refiners is an elegant way to leverage their generative capacity while maintaining interpretability.\n4. DRDT3 consistently outperforms several strong baselines (DT, Diffuser, IQL, QDT) across diverse datasets and settings.\n5. The paper carefully evaluates the effect of each component (TTT, diffusion, gated MLP, loss functions), enhancing the credibility of the claims.\n\nWeaknesses: \n1. While the combination is new, the core ideas (DT, TTT, diffusion, gated MLP) exist independently and are only moderately novel when integrated.\n2. All experiments are on benchmark synthetic tasks; additional results on real-world datasets (e.g., robotics) would enhance practical impact.\n3. The paper discusses the reduced complexity of DT3 and fast diffusion refinement, but does not report inference time or memory usage, which is important for applications.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces DRDT3, a novel framework for offline RL that improves the DT through two complementary enhancements:\n1. DT3 Module: A new sequence modeling component that combines self-attention with Test-Time Training (TTT) layers to better capture temporal dependencies with reduced inference complexity. This module produces coarse action representations.\n2. Conditional Diffusion Refinement: A conditional denoising diffusion model is applied to refine the coarse actions predicted by DT3, enabling better stitching of sub-optimal trajectories.\n3. Unified Optimization Objective: The paper proposes a joint training objective that integrates both coarse action prediction loss and diffusion refinement loss to ensure consistency between the DT3 module and the diffusion policy.\n4. Gated MLP Noise Approximator: A novel noise prediction module is introduced to improve the effectiveness of the diffusion refinement, incorporating both adaptive layer normalization and gated MLPs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. The paper would benefit greatly from enhanced visualization to better communicate the effectiveness of the proposed DRDT3 framework. Currently, the paper includes minimal qualitative or visual analysis, making it difficult for readers to intuitively grasp how DRDT3 improves over DT and other baselines. The authors should include step-by-step visualizations of the action refinement process through the diffusion iterations. Furthermore, adding training curves comparing convergence behavior across DT, DT3, and DRDT3 on representative tasks could further validate the claimed improvements in optimization dynamics.\n2. The authors should more explicitly position DRDT3 with respect to Decision Diffuser, Diffuser, and QDT in terms of technical differences and advantages.\n3. Add a table or discussion reporting the wall-clock inference time and memory usage of DT3, DT, Diffuser to validate the claim of improved complexity.\n4. Include a discussion on failure cases where DRDT3 underperforms or behaves suboptimally (e.g., walker2d-me).\n5. Consider releasing code or providing implementation pseudocode for the diffusion refinement module and gated MLP block, to facilitate reproducibility and follow-up research.", "broader_impact_concerns": "No significant ethical concerns are raised by this work. The method targets general offline reinforcement learning and could potentially benefit fields such as robotics or healthcare decision-making where safe and efficient offline learning is needed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JfbEH51wjJ", "reviewer": "Reviewer_7Lmj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**: \n- This article is well written.\n- The chosen baselines are both comprehensive and appropriate for the evaluation of the proposed method.\n- The ablation study validates the effectiveness of the proposed method.\n\n**Weaknesses**: \n- Generalization to high-dimensional observations remains unexplored. The tasks in the experiments are relatively simple with low-dimensional state and action spaces. The authors are recommended to add experiments on more complex Mujoco tasks, such as Humanoid and Humanoid Standup, which have a a state-space dimension of 348 and an action-space dimension of 17.\n- The authors are recommended to provide training curves of the baselines to compare the method performance during the learning phase.\n- As an offline RL approach, the stability of the proposed training strategy is suggested to be discussed theoretically.\n- The normalized scores of the methods show little improvement on the walker2d-medium-expert-v2 task, as depicted in Figure 3. The paper could benefit from further discussion of this phenomenon.\n- Several references require page numbers to meet standard citation practices:\n    -   Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama. Optuna: A nextgeneration hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.\n    -  Hanqun Cao, Cheng Tan, Zhangyang Gao, Yilun Xu, Guangyong Chen, Pheng-Ann Heng, and Stan Z Li. A survey on generative diffusion models. IEEE Transactions on Knowledge and Data Engineering, 2024.\n   -   Siyuan Guo, Lixin Zou, Hechang Chen, Bohao Qu, Haotian Chi, S Yu Philip, and Yi Chang. Sample efficient offline-to-online reinforcement learning. IEEE Transactions on Knowledge and Data Engineering, 2023.\n   -  Jia Liu, Yunduan Cui, Jianghua Duan, Zhengmin Jiang, Zhongming Pan, Kun Xu, and Huiyun Li. Reinforcement learning-based high-speed path following control for autonomous vehicles. IEEE Transactions on Vehicular Technology, 2024a.\n   -  Cong Lu, Philip Ball, Yee Whye Teh, and Jack Parker-Holder. Synthetic experience replay. Advances in Neural Information Processing Systems, 36, 2024.\n   -  Hamid Taghavifar, Chuan Hu, Chongfeng Wei, Ardashir Mohammadzadeh, and Chunwei Zhang. Behaviorally-aware multi-agent rl with dynamic optimization for autonomous driving. IEEE Transactions on Automation Science and Engineering, 2025.\n   -  Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.\n   -  MaonanWang, XiXiong, YuhengKan, Chengcheng Xu, andMan-OnPun. Unitsa: Auniversal reinforcement learning framework for v2x traffic signal control. IEEE Transactions on Vehicular Technology, 2024a.\n   -  Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. Advances in Neural Information Processing Systems, 36, 2024.\n   -  Ziyang Zhai, Ruru Hao, Boyang Cui, and Siyi Wang. Hgat and multi-agent rl-based method for multiintersection traffic signal control. IEEE Transactions on Intelligent Transportation Systems, 2025.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript considers offline RL problems and introduces two components to enhance algorithm performance: a novel DT-style trajectory modelling method and a conditional diffusion model. This approach enables the predicted actions from DT3 to be iteratively refined with Gaussian noise through the denoising chain within the diffusion model, facilitating trajectory stitching. Experiments on extensive tasks demonstrate the effectiveness of the proposed methodology.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the above weaknesses.", "broader_impact_concerns": "NO.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YkvdHe1H6b", "reviewer": "Reviewer_2Ywh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- [s1] The paper is well-written and is easy to follow.\n- [s2] The architectural changes in Decision Transformer, replacing Transformer block with Attention TTT block, sounds interesting. This change makes the model leverage the historical information by RNN-like structure.\n- [s3] Two-staged approach, incorporating diffusion models to refine the action, can boost the performance.\n- [s4] The paper conducts extensive ablation studies on architecture and loss functions, which would provide insightful information for practitioners.\n\n### Weaknesses\n- [w1] I think that the discussion and comparison around the diffusion policy is quite insufficient. While there is a lot of works using diffusion policy in offline RL, no baseline, except for Diffuser, is included in Table 1. For instance, even with my coarse survey, these three papers proposed offline RL methods with diffusion policies, and some achieved better performance than DRDT3 (https://arxiv.org/abs/2305.20081; https://arxiv.org/abs/2406.09509; https://arxiv.org/abs/2405.20555). I understood TMLR is not a venue aiming for SoTA, and not achieving SoTA should not be a reason for rejection, but that does not mean we can ignore the previous works in the literature. I highly recommend to update the related work section and experiments substantially, and discuss the relevant to this work and why the performance gap happens there.\n- [w2] it is unclear if DT module needed to be pre-trained, and then optimized through the unified objective in Section 4.4, or if we can train the policy from scratch just by optimizing the unified objective.\n- [w3] are there any reasons to use L1 loss in equation 16?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces DRDT3, a novel offline reinforcement learning framework designed to enhance the Decision Transformer (DT) by improving its ability to learn from suboptimal data, a concept known as trajectory stitching. The core idea is a two-stage process where a base model first predicts a coarse action, which is then iteratively refined by a diffusion model. The base model, named Decision TTT (DT3), replaces the standard transformer blocks in DT with a novel \"Attention TTT block.\" This block combines self-attention with a Test-Time Training (TTT) layer, an RNN-like structure that updates its hidden states at test time, aiming to better capture recent context. This DT3 module generates a coarse action prediction. This coarse action then serves as a condition for a conditional diffusion model. The diffusion model, using a novel gated MLP noise approximator, iteratively denoises a random sample, guided by the coarse action, to produce a final, more refined action. The main contributions are the DT3 model, which improves upon the standard DT, and the full DRDT3 framework, which uses diffusion for action refinement. The authors propose a unified optimization objective to train both the DT3 and diffusion components jointly. Through extensive experiments on D4RL benchmarks, the paper shows that DRDT3 achieves best results among the conventional offline RL methods and other DT-based approaches.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I highly recommend to update the related work section and experiments substantially, and discuss the relevant to this work and why the performance gap happens there. At least please consider to include the comparison and discussion against the following three (https://arxiv.org/abs/2305.20081; https://arxiv.org/abs/2406.09509; https://arxiv.org/abs/2405.20555). Please also note that these three are not all the relevant works for offline RL with diffusion policy, and survey the prior works extensively.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)