Long-Short Decision Transformer: Bridging Global and Local Dependencies for Generalized Decision-Making
Authors: Jincheng Wang
Penny Karanasou
Pengyuan Wei
Elia Gatti
Diego Martinez Plasencia
Dimitrios Kanoulas
Venue: ICLR 2025 Poster (2025)
Keywords: Deep Learning
Reinforcement Learning
Transformer
Decision Transformer
Long-Short Decsion Transformer
OfflineRL
Primary area: reinforcement learning

Abstract: Decision Transformers (DTs) effectively capture long-range dependencies using self-attention but struggle with fine-grained local relationships, especially the Markovian properties in many offline-RL datasets. Conversely, Decision Convformer (DC) utilizes convolutional filters for capturing local patterns but shows limitations in tasks demanding long-term dependencies, such as Maze2d. To address these limitations and leverage both strengths, we propose the Long-Short Decision Transformer (LSDT), a general-purpose architecture to effectively capture global and local dependencies across two specialized parallel branches (self-attention and convolution). We explore how these branches complement each other by modeling various ranged dependencies across different environments, and compare it against other baselines. Experimental results demonstrate our LSDT achieves state-of-the-art performance and notable gains over the standard DT in D4RL offline RL benchmark. Leveraging the parallel architecture, LSDT performs consistently on diverse datasets, including Markovian and non-Markovian. We also demonstrate the flexibility of LSDT's architecture, where its specialized branches can be replaced or integrated into models like DC to improve their performance in capturing diverse dependencies. Finally, we also highlight the role of goal states in improving decision-making for goal-reaching tasks like Antmaze.

Published as a conference paper at ICLR 2025
LONG-SHORT DECISION TRANSFORMER: BRIDGING GLOBAL AND LOCAL DEPENDENCIES FOR GENERAL-
IZED DECISION-MAKING
Jincheng Wangb
Penny Karanasou" Pengyuan Wei
Elia Gatti
Diego Martinez Plasencia Dimitrios Kanoulas!
University College London University of Cambridge
ucabj46fucl. a.c.ukk uk, pk407ēcam.aco (pengyuan. wei. .2, elia.gatti, d.plasencia, d.kanoulasj@ucl.ce
ABSTRACT
Decision Transformers (DTs) effectively capture long-range dependencies us- ing self-attention but struggle with fine-grained local relationships, especially the Markovian properties in many offline-RL datasets. Conversely, Decision ConvFormer (DC) utilizes convolutional filters for capturing local patterns but shows limitations in tasks demanding long-term dependencies, such as Maze2d. To address these limitations and leverage both strengths, we propose the Long- Short Decision Transformer (LSDT), a general-purpose architecture to effectively capture global and local dependencies across two specialized parallel branches (self-attention and convolution). We explore how these branches complement each other by modeling various ranged dependencies across different environ- ments, and compare it against other baselines. Experimental results demonstrate our LSDT achieves state-of-the-arr performance and notable gains over the stan- dard DT in D4RL offline RL benchmark. Leveraging the parallel architecture, LSDT performs consistently on diverse datasets, including Markovian and non- Markovian. We also demonstrate the flexibility of LSDT's architecture, where its specialized branches can be replaced or integrated into models like DC to improve performance in capturing diverse dependencies. Finally, we also highlight the role of goal states in improving decision-making for goal-reaching tasks like Antmaze.
INTRODUCTION
Transformers Vaswani et al. 2017) have emerged as predominant model, due their simplic- ity and capability modeling high-dimensional distributions (Yu et al. 2022). They serve as the foundational model in various domains, such as the Transforanf (Liu et al, 2021) in Com- puter Vision (CV) or GPT (Brown et al.) (2020) in Natural Language Processing (NLP). NLP, they significantly enhance the machine's s ability to understand and generate natural language by adeptly capturing long-distance dependencies between words. Also, Transformers have been explored as a powerful tool in Reinforcement Learning (RL). The Decision Transformer (DT) (Chen al.) 2021) and its variants (Zheng et al. 2022), conceptualize RL problems as conditional sequence modeling rather than conventional bootstrapping methods (Mao et al. 2022 Liu et al. 2023), drawing par- allels with sequence-tor coaennceett-sqqueee to-sequence tasks in NLP. DT employs causal transformer to predict future actions. Leveraging the self-attention mechanism, can model wide distribution of behaviors, achieving superior performance in many offline RL tasks (Paster al. 2022).
Although Transformers have demonstrated impressive performance, its architecture shows limita- tions in extracting fine-grained local patterns in datasets from other domains(Gullan et al., 2020 Shen et al., 2023), such as the correlations of neighboring pixels in CV (Wu et al., 2021). In the offline RL domain, this architecture struggles to effectively capture inherent Markovian patterns (a local association between two consecutive time-steps) in many RL datasets (Kim et al. 2023). An effective way to address this limitation is to integrate Convolutional Neural Networks (CNNs) into the transformer architecture, as CNNs are good at extracting fine- imne-rained local features (Alzubaidi
al. 2021 Chua & Roska 1993). Kim al. (2023) propose Decision ConyFormer (DC), which

---

Published as a conference paper at ICLR 2025
67.5
50.1
N
Long-tarm
Shot-term
Long-term
Shot- term
Maznzó mediu
Nano2o lrge
Inpet
Inpuc
Input
Figure 2: Results in Maze2d. Benefiting from our Long- Short structure, DC gains no- table improvement.
Decision Transformer (DT)
Convolution-only (CDT, e.g,, DC)
Long-Short Short Decision Transformer (LSDT)
Figure I: The overall architecture of DT,CDT and LSDT.
replaces the attention module with causal convolution filters to extract local Markovian associa- tions. However, DC's architecture has limited adaptability to datasets collected by non-Markovian policies, where long-term dependencies are crucial for optimal decision-making, This limitation is clearly reflected in our results from domains like Maze2d (see Figure 2 These findings highlight the importance of model's ability capture both local and global dependencies, which is cial for enabling agents to learn an optimal policy from diverse datasets. Typically, in offine RL, datasets often composed of trajectories generated by various policies, leading data that is not strictly Markovian and may instead exhibit non- Mon-Markovian or mixed characteristics. To enhance Transformers' capability in capturing both local and global dependencies, dual branch architectures have been proposed in CV and NLP (Peng et al. 2022]] Wu et al. 2020). While these models have proven effective their respective domains, is unclear whether such a dual branch structure can also enhance DT's decision- decision-making capabilities in offline RL or help DT and its variants generalize diverse datasets with different temporal dependencies, such as Markovian and non- non-Markovian.
In this paper, we propose the Long- Short Decision Transformer (LSDT), a generalized and flexible architecture designed to effectively model both long-term and short-term information. LSDT fea- tures two parallel, specialized and customizable branches as shown in Figure [ To demonstrate the effectiveness of LSDT, we adopted two generic approaches as branches: self-attention and Dynamic Convolution. Specifically, the long-term branch leverages the sell-attention mechanism to capture long-range dependencies, whereas the short-term branch employs a parameter-efficient Dynamic Convolution (Wu al., 2019) to model short-range dependencies. Contrary to fixed dimension allocation to each branch (Peng et al. 2022). our short-term branch can be seen as a dedicated head within the multi-headed attention module (Vaswani et al. 2017). By adjusting the dimension proportions fed into the short-term head, the captured global and local dependencies are varied, en- hancing its flexibility aitention-only and convolution-only DT as required by specific scenarios. Moreover, DT struggles in goal-reaching tasks with diverse goals and sparse rewards, where conditioning solely on desired returns is insufficient to guide the agent's 's behavior. To address this, we introduce goal-state condi- tioning that explicitly incorporates goal states into the conditioning information, thereby improving the performance of DTs in these tasks.
adapt diverse task demands. The architecture allows for switching to
We conduct evaluations of our proposed approaches on the standard D4RL benchmark (Pu et al., 2020). Results from our studies demonstrate consistent enhancement and superior performance of our LSDT in dccision-making compared to DT and its variants. LSDT achieves comparable performance compared to state-of-the-an RL methods. Additionally, the goal-state conditioning method significantly enhances the performance Maze2d tasks. To validate the flexibility of LSDT, replaced the Dynamic Conv in the short-term branch with DC's filters, and the observed improvements (e.g., Figure () on non Mor-Markovian datasets confirm that our approach is straightforward and plug-and-play with minimal adjustments.
wranfforme-bbasee RL, as shown in Antmaze and
PRELIMINARIES
Offline Reinforcement Learning. In RL, the learning environment can be considered as Markov Decision Process (MDP), defined by tuples (S, R). Here, represents the set of possible states, A represents the set of actions, reppeseets the state transition probabilities P(s' (, a), and R
2

---

Published as a conference paper at ICLR 2025
represents the reward function r' R(s, a). Unlike online RL, where an agent incrementally leams from interactions with the environment, offline RL involves learning from a static dataset of pre- collected trajectories from diverse behavioral policies 7 (so, ao, "n, $1, , .......... ST, ar, ar,"T) )- where 8t, at, and Tt denote the state, action, and reward at time 4, with T marking the episode's termination. The objective in offline RL is to learn policy that maximizes expected returns, cap- tured by max, E|E;-1 rr]. without the permission for further exploration or data gathering. This setting exposes algorithms to challenges such as model generalization and distribution drift, which are inherent in offine reinforcement learning environments. In our approach, we apply a modified transformer architecture in offline RL to learn optimal policy from suboptimal datasets.
Decision Transformer. Our approach is built based on the Decision Transformer (DT) (Chen et al.) 2021) architecture, which reframes the RL problem as conditional sequence modeling task. Unlike other classical RL methods that primarily focus on estimating value functions or computing policy gradients, DT generates action d, at time t conditioned on the context of the latest k timesteps, current state, and current return-to-go. The input of DT is formulated as: r = (Re 1 1R ht-11 -ot---1100--+1-................. R 4-1, R,, 82)- Rg is return-to-go (RTG) at time t, computed Rg Dr= where is rewards, 8 is states, is action, K is hyperparame- ter and is also referred as the context length. DT parameterized policies by GPT (Radford et al.) 2018) which is built based the decoder of Transformer (Vaswani et al. 2017) architecture. This framework leverages causal mask and stacked self-attention layers to autoregressively generate action sequences, offering novel approach in generating policies in RL. A key feature of DT is its exploitation of the self-attention layers, which enables the model to capture long-range dependen- cies in sequences effectively. This capacity allows DT to directly assign credit actions, thereby bypassing the slower process of reward propagation through Bellman backups.
Dynamic Convolutions. Dynamic Convolution (DynamicConv) (Wu et al. 2019) is variation of Dephhwise Convolution, which performs competitively to the self-attention in transformer architec- ture. Unlike self-attention, which computes attention weights by comparing each context element across all time steps, DynamicConv predicts separate convolution kernels based each time step- This module is simpler and more efficient than self-attention, reducing computational complexity from quadratic to linear with respect to input length. For the f-th element with output channel the computation rule is defined as:
PynamicConv(X,44g)
softmax(f(X)) ((++s- Kt1),
where represents the input to the convolution, is the input dimension, & is the kernel width, f linear module of current timestep only, is the kernel index indicating which kernel element is being used in the convolution and H is hyperparameter of weight sharing. DynamicConv is depth- wise separable, NeftMax-normmlized and share weights across the channel dimension, reducing the number of parameters in the convolutional part from d² x k to H x k compared to regular convo- lutions. The configuration endows the model with better generalization and parameter-efffciiett than standard self attentionio Moreover, exploiting local features dynamically is shown to be beneficial in NLP (Wu al. (2019). Motivated by these advantages, we explore applying dynamic convolution in offline RL. with a transformer, enhancing the model's 's ability to capture fine-grained local features.
METHOD
In this section, we focus on two key improvements. First, we propose Long-Short Decision Trans- former, designed to effectively capture both global and local dependencies. Second, we introduce the goal-state conditioning method to enhance DTs performance goal-reaching tasks.
3.1 MOTIVATION
To address the limitations of attention-only and convolution-ons structures, we propose a long- short structure that leverages the complementary strengths of both, rather than relying a single type of feature extractor. This long-short structure is designed to effectively handle both Markovian (short-term) and non-Markovian (long-term) dependencies across diverse offline datasets. In order
3

---

Published as a conference paper at ICLR 2025
to intuitively illustrate the differences in captured dependencies between these models, conducted experiments on the dataset (Markovian), visualizing the attention scores from the first layer with context length K from the first layer Figure R These scores can be interpreted as alignment measures that indicate how strongly each target token is associated with source tokens (Hu 2020).
dataset Monn-Marovvinn) and the Halfcheetah-medine eexpetttteet
Tellt 10//000
(a) attention-only
(b) convolution-only
(c) long-term (LSDT)
(d) short-term (LSDT)
Figure 3: The heatmap for single sequence Antmaze-medium (upper panel) and Halfcheetah- medium-replay (lower panel). Attention scores of (a) attention-only DT, (b) convolution-only DT, (c) the long-term branch of LSDT and (d) the short-term branch of LSDT
The attention matrix demonstrates the captured associations at timestep t 30, where the elements on the horizontal and vertical axes correspond one-to-one with (Ro: Sp. ap...., Rig. $29, a22).. The shape of these matrices is in lower triangular form because, decision-making model, DT can only rely on historical information to generate future action, with all future information masked out (Figure 3(a). To impose Markov assumption DT, convolution-only DT (e.g.. DC) further narrows the scope of DT's view to adjacent timesteps by setting the convolutional filter size L = 6. This operation forms lower triangular banded pattern Figure 3(b) Self-attention mechanisms are capable of modeling local associations. In D4RL, datasets can be mainly categorized into two types: Markovian and Man-Markovian datasets. Ideally, it should 'pay more attention' to elements between two adjacent timesteps in Markovian offline RL datasets to form concentrated and banded attention scores in both Antmaze and Halfcheetah are dispersed, indicating that DT tends to prioritize long-range dependencies over local patterns. In convolution-only DT, the available historical information restricted, which prevents the model from effectively capturing global dependencies in non-Markovian datasets. While larger kernel size can be used to expand the receptive field of models, CNNs generally struggle to model long- term dependencies (Gulati al. 2020) leading worse performance,
shape similar to Figure 3(b] Unfortunately,
discussed in Appendix 7 The aim of our LSDT is to leverage the unique strengths of both branches, allowing them to complement each other and thereby enhance the model's 's generalization across diverse datasets. Specifically, the long-term branch is designed to capture global dependencies by identifying which distant information the sequence helps predict the next action, while the short-term branch focuses more on local features, including but not limited to Markov patterns. For instance, in Halfcheetah, our short-term branch captured the Markov property (Figure ((d) upper part), where excluding the element's s self-correlation, the next state st+1 only depends on the previous state 8,. Conversely, in the Antmaze environment (Figure lower part), captures more complex local patterns.
3.2 LONG-SHORT DECISION TRANSFORMER
Inspired by the developments of dualhbranch structure in language (Zhu al. 2021 and vision tasks (Peng et al.) 2022), our approach seeks to employ a similar strategy to enhance the capability of exploiting local fine-grained patterns of GPT architecture used in DT. To this end, present
4

---

Published as a conference paper at ICLR 2025
Long torm Branch
0-00
1-1
11-01
Output: Y.
Mutti-oadeed Salvatieniin
Pyruest CaFe
Inputx
Unmit
Short-torm Branch
Figure 4: The architecture of Masked Long- Short block.
Long-Shon Decision Transformer (LSDT). The key innovation in LSDT lies in replacing the stan- dard masked attention block with our Masked Hybrid block, which features simple dual-branch structure as shown in Figure [ By integrating an additional convolution branch into DT, our ap- proach allows the model to better capture local features, while effectively combining it with the original attention branch's 's strength in modeling global dependencies. This comprehensive ability is particularly crucial in RL tasks, where the ability to discern and respond immediate information while maintaining the view of the overall information is essential for decision-making (Chen et al.) 2023 Bakker 2001). Furthermore, LSDT is a general architecture, it can be transformed into origi- nal attention-only DT or convolution-only DT by adjusting the input dimension to each branch. We keep most settings as DT, including training and inference stages, except for the inner architecture.
3.2.1 MASKED LONG-SHORT BLOCK
Our detailed Masked Long-Short block consists of two parallel branches, shown in Figure This structure leverages the complementary strengths of its two branches. Contrasting with methods that either feed the whole input to each branch or evenly split the input for the two branches (Peng al.
2022 Wu et al. (0200, our approach splits the input X along the channel dimension asymmetri- cally. Consider a input vector X with hidden dimension D and sequence length T, we partition the dimension D into two parts according to the dimension ratio 1 - (for long-term branch) and & (for short-term branch), denoted as X1 and X2. This division allows us to flexibly adjust the dimensions according to the specific requirements of different tasks, aiming to achieve better performance. We empirically demonstrate the impact of different dimension ratios & in Section 4.5 The X₁ is fed into the attention branch to focus on capturing complex hidden relationships across sequences, while the other part X2 is processed by a convolution branch along the diagonal to capture local feature patterns. For the short-term branch, we use DynamicConv, detailed in Section [ DynamicConv is characterized by fewer parameters and enhanced performance, dynamically generating weights based on current input instead of using fixed weights after training. These strengths endow the model with greater flexibility and generalization capabilities, beneficial for decision maaiiiinnmmaiiggggg in RL.
3.3 GOAL-STATE CONDITIONING
DT generates optimal paths from sub-optimal trajectories by conditioning on the desired sum of the future reward (return-lo-go, RTG) and contextual information, utilizing original data representations from the D4RL dataset (Chen et al.] 202T; Wa et al., (023). However, DT faces significant path planning challenges in goal-reaching tasks with diverse goals and sparse rewards, where based solely on returns as goals are inadequate for directing agents on where to go or which subpart to stitch. An example of DT failing to generate a path to the target location, even from the expert training dataset, is shown in Figure 5 Consider path planning task to a target position x with reward of 1 at and 0 otherwise. The training data includes the optimal paths to goal x, goal y. and the trajectories that pass through goal on the way to goal x. In DT setup, the RTG for all successful training trajectories are marked as 1. However, such processing can misguide the agent in environments with diverse goals, leading plamning failures. The three potential paths generated by DT. conditioning solely on an RTG of 1 leading to goal x, are depicted as "Generation with dotted lines. Moreover, previous works (Emmons et al. 2021) categorize RL algorithms by different types of conditioning
5

---

Published as a conference paper at ICLR 2025
information. In some experiments, DT is treated as swwrd-conditioons algorithm and directly compared with other goal goal-conditioned methods, which may lead to an unfair comparison.
Original training data in DT
Generation A
graph
Dial X
Training data with goal information
Generation
Figure 5: An example demonstrating the impact of conditioning information on path planning in environments with diverse goals and sparse rewards. The top panel shows how an agent struggles without target information, leading to misguided goal. The bottom panel depicts the agent correctly finding its direction when trained on a dataset where the goal position is incorporated into the state.
To address these, introduce this "Goal-State Conditioning' method which transforms goal- conditioned DT into a model conditioned on both the goal and the goal state. We also demon- strate why relying solely on original reward conditioning is insufficient for handling complex goal- reaching tasks, such as Antmaze-medine in Appendix [. Although this simple idea has been ex- plored in several goal-conditioned MLP-based models (Liu al. 2022 Paster al., 2022; Schaul it al. 2015), it is less researched and discussed in sequence modeling RL. We also extend this approach other transformer- based RL methods (Section 4.3). This integration provides a more
comprehensive basis to model for path planning. The successful path leading to goal x, generated using this approach, is depicted as 'Generation B' in Figure
3.3.1 IMPLEMENTATION OF GOAL-STATE CONDITIONING
During the training stage, we augment each state in the dataset with the corresponding goal state from its trajectory (existing in D4RL dataset), indicating to the agent which segments can be stitched together to generate an optimal path for each distinct goal. We next formally detail the modifica- tions made to the state representation. Let st E R" denote the original state vector at time step t, where represents the dimensionality of the state vector. Let g E R be the goal state's 's coor- dinate vector, with m being the dimensionality of the goal state coordinates. Through Goal-State Concatenation, we obtain augmented state vector St time step i, which can be defined
S = [0t,g] gi" where the augmented state vector S E R=** is formed by stacking the original state vector 8 and the goal state vector g. Furthermore, with the incorporation of goal state in- formation, the representation of modified trajectory Ty in offline training dataset is represented as: Ty = (ro, Sy: ag. .o.....,yy Pr, Sy ar), where T is the length of trajectory, is action and is reward.
During the evaluation, our process mainly adheres to DT with a key modification to enrich condition information. Initially, we specify the desired performance and the starting state, then concatenate the target goal coordinates with the state to create an augmented state representation. This augmented state, along with the desired performance, serves as the conditioning information to initiate trajectory generation. Subsequently, each new observed state S, is concatenated with the goal state g to form the augmented state Sy and insert them into history. We then repeatedly feed the last K historical timesteps into the model to autoregressively generate the next action until episode termination.
EXPERIMENTS
Our experiments systematically assessed the proposed LSDT and gtal-state conditioning approaches around the following questions:

---

Published as a conference paper at ICLR 2025
Table I: MuJoCo Evaluation Results. For locomotion tasks, the mean values of random seeds are reported, with 20 evaluations each. For Antmaze, values are averaged over 4 seeds with 100 evaluations each, using gtal-state conditioning. The values within 5% of the best scores per task are highlighted in bold, in line with IQL (Kostrikov al., (02T) paper. '-m' stands for medium, '-m-r' stands for medium-replay and -m-e' stands for medium-experi dataset. '-d' denotes diverse.
Dataset
TD3+BC
IQL
CQL
RvS
DT
QDT
DC
LSDT
Halfcheetah-m
48.3
47.4
44.0
41.6
42.6
42.3
43.0
43.6
Hopper-m
59.3
66.3
58.5
60.2
67.6
57.2
92.6
87.2
Walker2d-m
83.7
78.3
72.5
71.7
74.0
67.5
79.2
81.0
Halfcheetah-m44
44.6
44.2
37.5
38.0
36.6
30.0
41.3
42.9
Hopper-m-r
60.9
94.7
95.0
73.5
82.7
45.8
94.2
93.9
Walker2d-m-r
81.8
73.9
77.2
60.6
66.6
30.3
76.6
74.7
Halfcheetah-m-cc
90.7
86.7
91.6
92.2
86.8
93.0
93.2
Hopper-m-e
91.5
105.4
101.7
107.6
110.4
111.7
Walker2d-m-e
110.1
109.6
108.8
106.0
108.1
109.6
109.8
Locomotion average
75.3
76.5
77.6
71.7
76.4
82.2
82.0
Antmaze-umaze
78.6
87.5
74.0
65.4
69.8
85.0
80.0
Antmaze-umaze-4
71.4
62.2
84.0
60.9
70.3
78.5
83.2
Antmaze-m-plle
10.6
71.2
61.2
58.1
1.5
85.5
Antmaze-m-d
3.0
70.0
53.7
67.3
0.0
0.0
75.8
How does LSDT perform against DT and its variants? (4.2 4.3
How does the different channel dimension ratio & affect the performance?
(4.5)
Is goal-state conditioning effective for enhancing the path planning capability of DTs?
44.3
Can other algorithms leverage the advantage our Long-Short architecture?
4.4)
We evaluate our approach using continuous control tasks in OpenAI Gym, serving as benchmark for offline RL algorithm performance. Through these experiments, aim to quantify method's enhancements in diverse datasets. Details of benchmarks and datasets can be found in Appendix
4.1 OFFLINE RI. BASELINES
We compare our approach with representative methods value-based offline RL, including TD3+BC (Beeson & Montana 2022). IQL (Kostrikov al.) [202]) and CQL (Kumar al. 2020). Additionally, to demonstrate the enhancement offered by our method, we compare LSDT with three DT-based methods: the standard DT (Chen
2021). Q-learning Decision Transformer (QDT) YYamagata et al., 2023), and Decision ConyFormer (DC) (Wu et al. 2019). Similar to our convolution-only DT, DC achieves superior performance by capturing local features with convolu-
tion filters. Moreover, we include comparisons with the goal-conditioned RvS (Paster et al. 2022) illustrate the role of goal-state conditioning in DT. In Maze2d, we primarily compare with QDT, which significantly outperforms DT.
4.2 LOCOMOTION TASKS WITH MARKOVIAN DATASETS
We evaluate LSDT in Hopper, Walker2d and HalfCheetah with dense rewards. The results of our ex- periments are presented in Table I All reported scores are normalized following the instructions in D4RL (Fu et al. 2020). The valuva
values are reported from (Yamagata et al.) 2023), DC values from (Wu et al. 2019). and other values from IQL (Kostrikov et al. 202 paper. Our LSDT consistently achieves approaches state-of-the-art performance in all tasks, demonstrating the effectiveness of our Long-Short architecture. Notably, LSDT is the only approach that consistently shows advanced performance across all medium-repla and medium-expi expert datasets. Although LSDT does sur- pass value-base algorithms such as TD3+BC and IQL on "Halfcheetah medium' "eeiichheetammmiimmm highest scores among DT' and its variants. Finally, the overall performance of both DC and
achieves the

---

Published as a conference paper at ICLR 2025
Table Maze2d Evaluation Results. We report the mean and standard deviation of random seeds, with 100 evaluations each. CQL, DT and QDT values are reported from QDT paper (Yamagata al., 2023). The best values are highlighted in bold. '-u', 'mm and '-I' denote umaze, medium, and large, respectively.
Dataset
CQL
DT
QDT
DCA
DCA-gs
LSDT
LSDT-gs
Maze2d-u
94.7
31.0
82.9 + 8.8
10.9 + 2.2
68.6 1.4
9.6:2..6
72.31 1.1
Maze2d-m
41.8
8.2
48.5
+ 9.4
16.3 + 0.5
50.1 4.5
17.2+2.33
68.4 3.9
Maze2d-1
49.6
2.3
62.0 + 13.4
57.7+ 12.3
12.4#3.2
116.7= 29.7
LSDT suggests that including specialized modules to focus short-term information can signifi- cantly enhance DT's performance on Markovian datasets.
4.3 MAZE2D AND ANTMAZE TASKS WITH NON-MARKOVIAS DATASETS
To evaluate the stitching and long-term credit signment capabilities of LSDT on non-Markovian datasets, well to demonstrate the effective- ness of "goal-state conditioning' method en- vironments characterized by diverse goals and sparse rewards, we assess our model using the Maze2d and more complex AntMaze environ- ments. To ensure a fair comparison, we conduct experiments on hybrid DC (DC,,) presented in DC (Kim et al.] 2023). It replaces the convolution block with an attention block the final layer, enhancing its ability to manage long-term credit assignment. In tables, "-gs" stands for goal-state conditioning.
95.2
3.5
Figure Ablation study investigating the effects of goal-state concatenation and the long-short structure in DT and DC Antmaze-medium-repla.
Both results from ablation studies in Table 2 and Figure demonstrate that simply offering models with additional conditioning information goal's 's' position can dramatically improve the performance of DT and its variants. By doing so, the agent gains better understanding of the trajectory towards the desired location. This finding validates the effectiveness and generalization of goal-state con- ditioning in transformer- based models. Moreover, we further conduct an empirical experiment to show how goal-state conditioning influences the generated path in Appendix B.I
Across all results in Table I and Table 2 LSDT achieves superior performance environments with non-Markovian datasets. For Maze2d, CQL, and QDT perform better in the umaze environment, benefiting from Q-learning's efficient backward reward propagation in tasks with shorter trajecto- complex Medium and Large environments,
ries. However, in the approaches. In Antmaze tasks, the scores highlight the effectiveness and generalization our long- short structure in complex tasks, particularly in the Medium tasks where DT is minimally effective. The comparison of DT-gs and LSDT-gs in Figure 6 illustrates that integrating an additional local feature extractor into DT can assist the model in making better decisions.
LSDT outperforms other
4.4 ENHANCING OTHER ALGORIHHS COMBINING OUR ARCHITECTURE
In this section, we explore whether our dual-branch structure can be integrated into other approaches to enhance their per-
Dataset
DC₁
LSDC
LSDT
formance. We present the Long-Short Decision ConvFormer
M2d-u
68.6
67.7
72.3
(LSDC) as demonstration, which substitutes the Dynamie- Conv in the short-term branch with DC's convolution filters.
M2d-m
50.1
67.5
68.4
M2d-1
57.7
98.3
101.17
Given that DC, aims to enhance long-term credit assignment over DC, aim to assess whether our architecture outper- forms the alternative of replacing the last block with an atten- tion block for capturing global dependencies.
Table 3: Comparisons of DC,,
LSDT and LSDC in Maze2d domain.
8

---

Published as a conference paper at ICLR 2025
We conduct experiments on Mazeld and Antmaze across 4 seeds. Table and Figure 6 show that LSDC achieves competitive performance, and significantly surpasses the original DC method in non-Markovian datasets. In Mazeld, larger mazes show more significant performance gains, high- lighting our approach's potential to enhance other models by integrating a dual branch structure that effectively captures both global and local dependencies.
4.5 ABLATION STUDY ON DIMENSION RATIO
In our LSDT, we introduce the channel dimension ratio & rather than evenly splitting the dimension, aiming to allow the model to adapt better to different tasks and achieve improved performance. Dif- ferent channel dimension ratios will lead the model to focus different features, thereby affecting dependency modeling. To assess the importance of this setting, we evaluate LSDT with varying dimension ratios ranging from (0%-100%) When the convolution dimension ratio is set 0%, the model is attention-only. reverting to standard DT. When the ratio reaches 100%, all input dimensions processed by the short-term branch, transforming the model into convolutionoonly DT (CDT).
Ruach
12.0% 10.% R05
100
Figure 7: Performance of the locomotion environments across different dimension ratios &. The plots display mean values and standard deviations over 5 random seeds, each with 20 rollouts. From left to right: medium dataset, medium-replay, medium-experr, maze2d.
Figure 7 shows how the ratio influences the final performance across different environments and datasets. Except for the Hopper-medium--eelty dataset and Mazged-Large, the performance of the model enhanced by integrating a local feature extractor into DT across all dimension ratios. On HalfCheetah, there is significant further improvement in performance beyond a convolution di- mension ratio of 12.5% on every dataset. For Walker2d, LSDT achieves its best performance on each dataset convolution dimension ratios of 12.5%, 50%, and 12.5% respectively. The observations from the Hopper task are interesting, LSDT reaches its highest scores at ratio of 100% on medium and medium-replaa. suggesting that the global patterns from the attention branch are totally harmful to decision-making However, on medium-experi dataset, relying solely on the convolution branch leads decreased performance. This finding aligns with the performance gap observed between DC and our LSDT on Hopper tasks in T DC is specifically designed to mainly focus on local patterns, which causes our LSDT to lag slightly behind on the medium and mediun-replay datasets. However, medium-expert, LSDT achieves slightly higher performance by integrating long-term information. This suggests that LSDT performs better on datasets with higher-quality data.
Based on ablation studies conducted Locomotion (Markovian) and Maze2d (non- mon-Maakovian) datasets, we propose general guidelines for selecting the dimension ratio based on the dataset type. Additionally, explore potential alternative eliminate the need for manual tuning of the di- mension ratio, as discussed Appendix B.10, to inspire further enhancements to our approach.
Avoid setting the dimension ratio to 0% when the training datasets are collected by Marko- vian policies, nearly all tasks benefit from integrating the additional short-term branch. For such tasks, we recommend choosing a dimension ratio of 50% or higher (c.g., 75%).
Avoid setting the dimension ratio to 100% when the training datasets are collected by non- Markovian policies, as this limits the model's s ability to access long-term historical infor- mation necessary for accurately predicting the current token. For such tasks, it is advisable to initially conduct experiments with below (c.g., 0%, 12.5%) or equal to 50%.
When computational resources are limited, we recommend 50% as the default, balancing performance and tuning cost, especially without prior knowledge about the dataset type.

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Transformers for RL. Approaches such as Decision Transformer (DT) (Chen et al. 2021) and
Trajectory Transformer (Janner et al. 2021 that abstract the offline RL as sequence modeling problem have been popular (Wu et al. 2023 Compared to the traditional offline RL methods, these approaches are simpler and more robust through doing RL via Supervised learning (RvS) (Emmons et al. (021) rather than learning a Q- Q-function or computing policy gradients (Kumar et al.) [020]] Kostrikov et al., 202T] Beeson& & Montana 2022). DT is among the most successful methods in this field. It generates a sequence of actions through transformer decoder conditioned on desired out-
comes and previous context. Recently, numerous of DT's variants have been developed leveraging the standard architecture. For instance, the Q-learning DT (QDT) (Yamagata ct al. 2023), integrates Dynamic Programming with the DT framework to enhance the optimal path generation ability of DT. Another notable development is Decision ConvFormer (DC) (Kim et al. 2023), which replaces attention blocks with convolution filters to more efficiently capture local associations. In contrast to DC, aim to design general model structure, focusing on enhancing performance across various tasks by balancing local and global dependencies, rather than limiting attention to local information.
Gated -r is a pioneering approach in applying the transformer model RI., particu- larly highlighting the importance of architectural modifications to address optimization challenges inherent in the standard transformer framework (Parisotto et al., 2020). This issue has sparked wave of research focused on designing transformer-based backbones specifically tailored for RL ap- plications (Mao et al., [022] Liu et al., 2023 Hu et al. 2023). In this vein, our work contributes to this field by enhancing the model structure of DT with dral-branch architecture adept at extracting information within different ranges, making it more effective for RL.
Convolution strategies in Transformers. Convolutional Neural Networks (CNNs), known for their efficacy in capturing local feature patterns in vision domain (Kaiser et al. 2018 Yang et al.] 2021 Roy et al. 2018). While the transformer is good at modeling global dependencies, it faces challenges in capturing local feature patterns (Jiang et al. 2020; Wu et al. 2021; Su et al. 2022). Therefore, many approaches have been studied to combine the strengths of transformers and CNNs, notably in Conformer and Branchformer Gulati et al. 2020 Peng et al. 2022). These models have demonstrated improved performance in local feature capture by their success in various tasks across CV and NLP. Similar to the Lite Transformer ( (Wu et al. 2020), our model employs two branches to model global and local dependencies separately, one branch for standard self-attention and another for convolution. Lite Transformer uses Light Convolution to reduce the computation and model size for mobile applications. Differently, our LSDT is specifically designed to address challenges in offline RL tasks. We utilize Dynamic Convolution (Wu et al. 2019). which offers enhanced generalization capability in capturing local features across diverse RL scenarios by dynamically generating the weights at each timestep instead of fixing them after training.
6 CONCLUSION
In this paper, we addressed the challenge of effectively handling both Markovian and mon-Markovian properties offline RL datasets by introducing the Long-Short Decision Transformer (LSDT)) is a general-purposs architecture to effectively capture global and local dependencies across two spe- cialized and customized parallel branches. To demonstrate the effectiveness of this architecture, we adopted two generic approaches (self-attention and Dynamic Convolution) as the long-term branch and the short-term branch. Our experiments on the D4RL benchmarks demonstrate that LSDT consistently achieves strong performance across both Markovian and non-Markovian datasets, val- idating the effectiveness of the proposed parallel architecture. We show that the dimension ratio of LSDT can affect the dependencies captured by the model and performance. Choosing an appropriate dimension ratio can further enhance decision-making. Furthermore, the performance improvement of DC with the integration of the long-short structure demonstrates the transfcrability and adaptabil- ity of our architecture. The results of applying goal-state conditioning to DT and its variants show that simply incorporating explicit goal information can dramatically improve performance in goal- reaching tasks like Antmaze. All these results demonstrate our architecture is promising approach, providing possibilities to design more powerful and parameter-efficient LSDT variants by replacing the current branches with more expressive feature extractors. In future work, we aim to extend our methods to more intricate tasks and explore LSDT's potential in multi task trajectory datasets.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENT
This work was supported by the EPSRC through their Prosperity Partnership program (EP/V037846/1) and by the UKRI Future Leaders Fellowship [MR/V025333/1] (RoboHike). For the purpose of Open Access, the authors have applied CC BY public copyright license to any Author Accepted Manuscript version arising from this submission.
REFERENCES
Laith Alzubaidi, Jinglan Zhang, Amjad Humaidi, Ayad Al-Dujaili, Ye Duan, Omran Al-Shamma, José Santamaria, Mohammed Fadhel, Muthana Al-Amidie, and Laith Farhan. Review of deep learning: concepts, enn architectures, challenges, applications, future directions. Journal of big Data, 8:1-74, 2021.
Gaon An, Seungyong Moon, Jang-Hyun Kim, and Hyun Oh Song. Uncertainty-basel offline re- inforcement learning with diversified g-ensemble. Advances in neural information processing systems, 34:7436-7447, 2021.
Bram Bakker. Reinforcement learning with long short-term memory. Advances in neural informa- fion processing systems, 14, 2001.
Alex Beeson and Giovanni Montana. Improving td3-be: Relaxed policy constraint for offline learn- ing and stable online fine-tuning. arXiv preprint arXiv:2211.18822 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Jinchao Chen, Tingyang Li, Ying Zhang, Tao You, Yantao Lu, Prayag Tiwari, and Neeraj Ku- mar. Global-and-local attention-based reinforcement learning for cooperative behaviour control of multiple uavs. IEEE Transactions on Vehicular Technology, 2023.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in neural information processing systems, 34:15084-15097, 2021.
Leon O Chua and Tamas Roska. The enn paradigm. JEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, 40(3):147-1566 1993.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rvs: What is essential for offline rl via supervised learning? arXiv preprint 1rXiv:2112.107555 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning. arXiv preprint 2,X
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu, et al. Conformer: Convolution-augmennee transformer speech recognition. arXiv preprint urXiv:2005.08/00 2020.
Dichao Hu. An introductory survey on attention mechanisms in nlp problems. In Intelligent Systems and Applications: Proceedings of the 2019 Intelligent Systems Conference (IntelliSys) Volume 2, pp. 432 448. Springer, 2020.
Jie Hu, Shen, and Gang Sun. Squeezz-andd-xxcitale networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Kaizhe Hu, Ray Chen Zheng, Yang Gao, and Huazhe Xu. Decision transformer under random frame dropping. arXiv preprint arXiv:2303.0339/ 2023.
Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. Advances in neural information processing systems, 34:1273-122 2021.

---

Published as a conference paper at ICLR 2025
Zi-Hang Jiang, Weihao Yu, Daquan Zhou, Yumpeng Chen, Jiashi Feng, and Shuicheng Yan. Con- vbert: Improving bert with span-based dynamic convolution. Advances in Neural Information Processing Systems, 33:12837--2848, 2020.
Lukasz Kaiser, Aidan N Gomez, and Francois Chollet. Depthwise separable convolutions for neural machine translation. In International Conference Learning Representations, 2018.
Jeonghye Kim, Suyoung Lee, Woojun Kim, and Youngchul Sung. Decision convformer: Local filtering in metaformer is sufficient for decision making. arXiv preprint arXiv:2310.03022 2023.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. arXiv preprint arXiv:21 10.06169, 2021.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. Advances in Neural Information Processing Systems, 33:1 179-1191, 2020.
Zhi-Hao Lai, Tian-Hao Zhang. Qi Liu, Xinyuan Qian, Li-Fang Wei, Song-Lu Chen, Feng Chen, and Xu-Cheng Yin. Interformer: Interactive local and global features fusion for automatic speech recognition. Network, 100:G2L, 2023.
Bonan Li, Yinhan Hu, Xuecheng Nie, Congying Han, Xiangjian Jiang, Tiande Guo, and Luoqi Liu. Dropkey, arXiv preprint arXiv:2208. 026446 2022.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. coal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.08299, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 10012-10022, 2021.
Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. arXiv preprint arXiv:2302.07351, 2023.
Hangyu Mao, Rui Zhao, Hao Chen, Jianye Hao, Yiqun Chen, Dong Li, Junge Zhang, and Zhen Xiao. Transformer in transformer as backbone for deep reinforcement learning. arXiv preprint arXiv:2212. 145388 2022.
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. Stabilizing transformers for reinforcement learning. In International conference on machine learning, pp. 7487-7498. PMLR, 2020.
Keiran Paster, Sheila Mcllraith, and Jimmy Ba. You can't count on luck: Why decision transformers and rvs fail in stochastic environments. Advances in Neural Information Processing Systems, 35; 38966-38979, 2022.
Yifan Peng, Siddharth Dalmia, Ian Lane, and Shinji Watanabe. Branchformer: Parallel mlp-attention architectures to capture local and global context for speech recognition and understanding. In International Conference Machine Learning, pp. 17627-17643. PMLR, 2022.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language under- standing by generative pre-training. 2018.
Abhijit Guha Roy, Nassir Navab, and Christian Wachinger. Concurrent spatial and channel 'squeeze & excitation in fully convolutional networks. In Medical Image Computing and Computer As- sisted Coarvention-MICCOO 2018: 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part pp. 421 421-429. Springer, 2018.
Tom Schaul, Daniel Horgan, Karol Gregor, and David Silver. Universal value function approxima- tors. In International conference on machine learning, pp. 1312-1320. PMLR, 2015.
12

---

Published as a conference paper at ICLR 2025
Xiang Shen, Dezhi Han, Zihan Guo, Chongqing Chen, Jie Hua, and Gaofeng Luo. Local self- attention in transformer for visual question answering. Applied Intelligence, 53(13):16706 16723, 2023.
Yanfei Su, Weiquan Liu, Zhimin Yuan, Ming Cheng, Zhihong Zhang, Xuelun Shen, and Cheng Wang. Dla-net: Learning dual local attention features for semantic segmentation of large-scale building facade point clouds. Pattern Recognition, 123:108372, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Felix Wu, Angela Fan, Alexei Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with 2019.
lightweight and dynamic convolutions. arXiv preprint
Haiping Wu, Bin Xiao, Noel Codella, Mengehen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 22-31, 2021.
Yueh-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. arXiv preprint arXiv:2307.02488 2023.
Zhanghao Wu, Zhijian Liu, Ji Lin, Yujun Lin, and Song Han. Lite transformer with long-short range attention. arXiv preprint arXiv:2004. / 1886, 2020.
Taku Yamagata, Ahmed Khalil, and Raul Santos-Rodriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline rl. Inter- national Conference on Machine Learning, pp. 38989-390077. PMLR, 2023.
Chenglin Yang, Siyuan Qiao, Adam Kortylewski, and Alan Yuille. Locally enhanced self- attention: combining self-attention and convolution as local and context terms. arXiv preprint arXiv:2107.05637. 2021.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10819-10829, 2022.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online decision transformer. In international conference on machine learning, pp. 27742-270557 PMLR, 2022.
Chen Zhu, Wei Ping, Chaowei Xiao, Mohammad Shoeybi, Tom Goldstein, Anima Anandkumar, and Bryan Catanzaro. Long-short transformer: Efficient transformers for language and vision. Advances in neural information processing systems, 34:17723-17736 2021.
13

---

Published as a conference paper at ICLR 2025
A BENCHMARK AND DATASET DETAILS
OpenAl Gym locomotion. We evaluate LSDT on OpenAl Gym continuous locomotion tasks (Hop- per, Walker2d, HalfCheetah) with dense rewards. Our approach was trained and tested three 'v2' datasets from D4RL (Fu et al. 2020) with different quality levels: medium, medium-replay and medium-expertt datasets. The medium dataset is collected from policy which is third of the per- formance of an expert policy. The medium-replay, sourced from this policy's replay buffer. For medium-expert dataset, it consists of half of the samples (1 million) from the medium policy and the other half (1 million) from the expert policy. Normally, these tasks are used to evaluate the model's performance across different of trajectories of varying quality.
Maze2d. Maze2d is a navigation task, requiring 2D ball to reach a target goal location within a maze. We conduct experiments across three different sizes of environments with sparse rewards: umaze, medium and large, each presenting increasing levels of difficulty. The agent receives a reward of if it successfully reaches the target location and 0 otherwise. The training dataset com- prises trajectories of the agent navigating towards random goal locations.
Antmaze. This domain utilizes the same maze layout and objective as Maze2d but replaces the 2D ball with more complex Ant robot. In D4RL, three types of datasets are included for AntMaze: '-umaze" features a fixed start to a specific goal, '-diverse' involves random starts to random goals, and '-pla'' comprises trajectories originating from set of hand-selected starting points, leading to another set of hand-picked target locations that may differ from the goal locations used during eval- uation. Especially, all of the datasets contain very few near-optimal trajectories, requiring relatively high stitching suboptimal paths capability to generate a path to desired goals.
Adroit. The Adroit domain consists of high-ding sigh-dimensional robotic manipulation tasks using a 24-DoF Shadow Hand for actions like hammering, door opening, and pen twirling. It designed to evaluate fully offline RL under narrow data distributions and sparse rewards. Three types of datasets are used: human demonstrations, expert data from a fine-tuned RL policy, and 'cloned' dataset combining imitation and demonstration data. Unlike standard Gym MuloCo tasks, Adroit data involves human demonstrations, sparse rewards, and complex exploration challenges, making it difficult to solve with traditional RL methods. The results in Adroit for LSDT are shown in Appendix C
B FURTHER DISCUSSION
B. How DOES GoAL-STATE CONDITIONING AFFECT PATH PLANNING IN ENVIRONMENTS WITH DIVERSE GOALS AND SPARES REWARDS?
To illustrate the challenges of path planning that DT encounters in environments characterized by diverse goals and sparse rewards, we consider the AntMaze medium as a representative example. In addition to intuitively demonstrating how goal-state conditioning effectively leads the agent to target, we also empirically show how insufficient conditioning information misleads agents toward incorrect goals, theoretically analysis in Figure 5
Target
(a)
(b)
(c)
Figure 8: Visualization of AntMaze-Medium Task: (a) Distribution of target locations: (b) Zonal di- visions for goal-oriented analysis; (c) Trajectory paths generated using DT; (d) Enhanced trajectory visualization with goal-state concatenation.
14

---

Published as a conference paper at ICLR 2025
First, we mark the goal location of all the trajectories from the D4RL dataset in the environment (Figure B(a)). Second, according to these goals distribution, we divide the environment into three zones as shown in Figure [8b) For Medium environment, the evaluation target location is marked with a white star in Zone B. Through this segmentation, the trajectories are classified into three groups, with each path colored according the zone in which the Ant's final location resides.
We illustrate the trajectories generated by a fully trained DT when conditioned on a desired RTG of I in Figure [[c] It becomes evident that the agent might be incorrectly guided areas or C, which are significantly distant from the intended target area. This misdirection occurs because all RTGs within successful trajectories in the dataset are uniformly labeled as 1, regardless of the actual target location. Therefore, the agent struggles to discern the correct direction during evaluation when conditioned solely on the RTG. Consequently, the termination points of the paths generated by DT are highly dependent on the target locations seen during the training phase. We also show the trajectories generated by DT with our goal-state concatenation approach in Figure [(d) By integrating the location vector of goals into states, DT becomes capable of correctly planning paths that lead the Ant robot to the target location, which underscores the significance including the goal vector additional condition information for effective path planning. We quantify the improvement of DT provided from goal-state conditioning in Table 4
Table 4: Evaluation results of DT with stal-state conditioning. We report the mean and standard deviation of 4 random seeds, with 100 evaluations each. "-gs" denotes goal-state conditioning.
Dataset
DT
DT -gs
antmaze-umazo-v2
69.8 0.2
77.0 1.2
antmaze-umazzeddierse--yy
70.3 5.3
76.3 1.5
antmaze-medium-play--y
0.0
39 = 5.1
altmaaze-mediun-ddvers----
0.0
46.8 + 8.1
B.2 HOW DOES THE CHANNEL DIMENSTON RATIO AFFECT DEPENDENCY MODELING?
To intuitively demonstrate the impact of the dimension ratio & on dependency modeling within each branch, we conducted experiments the halfcheetth-mediim-ee
environment, using hidden dimension of 128, kernel size of 11 and context length of 10. We experiment the dimension ratio for the convolution branch 75%. 50% 25% and 12.5% respectively. Figure 9 shows the learned attention matrix: the convolution branch's 's matrix is shown in the top panel, and the corresponding matrix for the self-attention branch is in the bottom panel. In our model, each timestep consists of a tuple: return-to-go R,, state s and action a. The indices for the x and y coordinates are derived from multiple timesteps, represented as sequences (Ro, 80, ao..... Ro9, S29, Q29))
The dimension ratio plays a crucial role in determining the types of patterns each branch captures. For instance, with a convolution dimension ratio of 75%, the short-term branch finds that each state is related only to itself and its preceding state. Conversely, the attention branch models depen- dency across the sequence of actions. At an attention dimension ratio of 75%, it identifies three dependencies: the current state depends only on previous states, the current return-to-go depends on previous states, and the current actions are closely linked to the return-to-go. The dependency of actions aligns with the concept of DT, which generate sequences of actions based on desired perfor- mance outcomes. These findings demonstrate that varying the dimension ratio results in different dependencies modeled LSDT. We believe that choosing an appropriate dimension ratio can better tailor the model to the specific requirements of each task, thereby achieving better performance as demonstrated in the experiment section.
B.3 HYBRID FUSION MODULE
To dynamically select valuable features and effectively integrate information from the two branches, we designed a hybrid fusion module incorporating a competitive selection process. Since this mod- ule introduces additional parameters without offering significant improvements over direct concate- nation in most tasks and page constraints, we present it as an optional component here to provide interested readers with a potential direction for further exploration. This module explicitly deter-
15

---

Published as a conference paper at ICLR 2025
Convotation Dimensior:
Convolation Dimersion: 64
Convalutio Dimensor::
Convolation Gimersion:
May Indix
Query
Attention Dimension
Attention Dimension: 54
Attenton menesiir:
Aitention Dimension: 112
Kay Indox
R
R
Query Index
Figure 9: Attention scores of Halfchhetta--nedinn--epeet Task. Top: Attention scores of short-term
branch (convolution). Bottom: Attention scores short-term branch (self-attention).
mines the relative importance of different channels through softmax operation, allowing it high- light the most relevant sub-features in the final output. Importantly, by visualizing these importance weights, we can determine which channels and branches contribute the most critical information for decision-making, providing clearer insight into the contribution of each branch.
B.3.1 DETAILS OF HYBRID FUSION MODULE STRUCTURE
Many existing methods (Peng al., 2022 Wu al. 2020 primarily employ direct concatenation to integrate features from different branches. However, the importance of features captured by each branch in each timestep ignored. Inspired by Interformer and SE block (Lai et al. 2023] Hu al., 2018), we introduce the hybrid fusion module (HFM) to select valuable characteristics d dynamically and efficiently fuse local and global representations from two branches. Initially, the outputs from the attention and convolution branches denoted Yatte RTx((I-E) and Yeorr R concatenated, where Ş is the dimension ratio for short-term branch. The concatenated features Y then processed through multi-layer perceptron (MLP) with a ReLU activation, reducing the channel dimension from D to 2
D for refining the feature representation:
Y Concat Yoor)), Ý (e
where W, RDx denotes the MLP layer. Next, restore the refined feature Y through two MLP layers, producing two features with the same size as the long-term branch and short-term branch, and then concatenate them for information aggregation.
Z = (on (Y), (x))
where Wn E R Wy2 E R&D Finally, we employ the softmax to weight the aggre- gation feature 7 time dimension, learning selective weight matrix A E R By multiplying focus important features and filter out
the matrix A with original output Y point- useless information from Y dynamically in each timestep. The output Y, is generated as follow:
Y, Softmarl 7)
16

---

Published as a conference paper at ICLR 2025
B.3.2 COMPARISON OF HFM AND DIRECT CONCATENATION
To the effectiveness of proposed hybrid fusion module (HFM), compare HFM with di- rect concatenation fusion method. We perform experiments on four different tasks using datasets of varying quality: Walker2d-medee medium. Figure IO (left) and Figure I0 (right) compare the learning curves and performance of our HFM with direct concatenation, respectively. We can observe that HFM can stabilize the training process and enhance performance.
Hopper-medine www---meeddeppeeeeee expert, Halficheeaalmmdium-reee medium-replay Mazeld-
w/o
W
Walker2d-m
1..
80.3-10.6
Halfcheetah-m-rr
41.1+-0.2
41.9++0.3
Hopper-m-e
111.5+
111.7+0.1
70
Maze2d-m
6..7
68.4 3,9
wibh HPM HFM
20000
Niumber Uipiiiatee
Figure 10: Ablation HFM. Left: comparison of training curves Walker2d-medium. Right: performance comparison. Experiments conducted across 4 seeds.
Antmaze-meddummale
Sequicc Iroe
Dimensionn Indee
Mazer2d
Carvent Theeseer 20
Oursedt Trmence:
Cavent Threstoe
Coneett Treadee 122
Secuencee ndea
Willker 26-medimm
Secuence Indact
Dinorioon Indoo
Figure 11: The selective weight matrix A for Antmaze-medium-play Walker2d-medium and Maze2d-large at different timesteps. The Dimension ratio & for these examples is 75 %.
We visualize the heat map of the selective matrix A in Figure II the X axis represents the dimension index, the axis represents the timestep, and the model was trained with 128 hidden dimensions, dimension ratio 25% (dimensions 0-95 for long-term, and 96-127 for the short-term branch; dashed line separating both branches). showcases how this module dynamically assigns weights to mensions (branches) based their importance at each timestep. Particularly, the maze2d-large and
17

---

Published as a conference paper at ICLR 2025
walker2d-m datasets, show concentrated weight distributions, meaning only a few dimensions tribute to predicting the current token. contrast, the heatmap for antmaze shows a more dispersed weight distribution, indicating both local (Markovian) and global (non-Markovian) dependencies are important for decision-making in this task. In walker2d-m, branches' outputs are sometimes competitive (one branch carries the highest weights, timesteps 60 and 90), while other times they are cooperative (at timestep 30). This demonstrates how HFM emphasizes valuable features while minimizing the influence of less relevant ones from both long-term and short-term branches, thereby enhancing its flexibility to integrate local and global representations.
B.4 DOES LSDT MAINTAIN THE SAME RTG CONDITIONING AS THE ORIGINAL DT FOR TRAJECTORY GENERATION
To intuitively demonstrate the impact of conditioning on a specified desired RTG on LSDT's perfor- mance, we evaluated it across a broad range and plotted the curve showing the relationship between RTG, and the scores in Figure 12 The yellow band represents the standard deviation of scores. For each environment, we tested the previously trained best model without the hybrid fusion module. This involved evaluating 20 sampled Return-To Go (RTG) values, with each RTG being tested 100 times to obtain comprehensive results. The best RTG of LSDT is reported in Table 5 Consistent observed that the scores achieved
with findings from the DT paper (Chen et al. (02T), related to the initially conditioned RTG, indicating that our LSDT also well models the distribution of returns. Furthermore, our results indicate that the performance of LSDT tends to reach saturation when conditioned on a sufficiently high score. It is noteworthy that for most tasks, conditioning on single RTG value seems sufficient achieve near-optimal scores. This finding particularly intriguing as it suggests that fine-tuning the RTG as hyperparameter may not be necessary. For example, we can obtain near-optimal performance in locomotion tasks by conditioning on a single RTG of 7000 yields comparable results. However, this is an initial hypothesis based on current data, and we will further test it across more environments in our future work.
highly
Men' Scnn
Mas Comt
mullchectah-matkur
Ha HoTCchetah modium replay
Hooser med replay
Via 1ww.ero0
1200
772
3000
2512
130
3TC
680
1363
100
300
700
Figure 12: Normalized D4RL scores achieved by LSDT conditioned on 20 sampled desired RTG in Open AI MuJoCo locomotion tasks.
B.5 Is OUR PROPOSED GOAL-STATTR CONCATENATION THE ONLY WAY TO PROVIDE ADDITIONAL INFORMATION TO DT?
To enhance DT's path planning in environments with diverse goals and sparse rewards, we explored various methods for incorporating the goal position coordinates in AntMaze. Intuitively, we first tried to implicitly incorporate the goal information of each trajectory into each token by adding a learned embedding similar to the time embedding used in DT or position embedding used in NLP. Moreover, we experimented with separately encoding the goal state into each ReturnToo To--u (RTG), action, and state representation. However, these methods did not yield improvement. It appears that the DT struggled to effectively differentiate between time embedding and goal embedding. This suggests that DT might require more explicit differently structured goal information enhance
18

---

Published as a conference paper at ICLR 2025
its decision-making capabilities in complex environments. Therefore, separately experimented with explicitly concatenating the goal information solely to the state, action, or RTG tokens within the trajectories. Through these trials, found that concatenating the goal vector with action or RTG hugely decreases DT's performance. This negative effect may be attributed to the constraints imposed by integrating goal information. When goal data is added action RTG, it potentially restricts the model's s flexibility in stitching trajectories. As result of these findings, we have selected goal-state concatenation our preferred approach enhancing DT's performance. This approach is also aimed at modeling the probability distribution of potential destinations or directions accessible from a given state, thereby providing better understanding of the state's implications.
B.6 How DOES OUR PROPOSED LSDT INFLUENCES THE CONTEXT LENGTH REQUIRED TO ACHIEVE OPTIMAL PERFORMANCE COMPARED THE CONTEXT LENGTH IN THE STANDARD DT?
The context length is a critical hyperparameter in DT that determines how many past tokens will be considered when we generate the next action. In standard DT and its variant QDT, a context length of 20 has been shown to yield better performance on OpenAl MuloCo tasks. However, for our Long-Short Decision Transformer, we observe optimal results with relatively shorter context length the majority of environments like Walker2d and Halfcheetah. This suggests that our model efficiently utilizes available information to generate better actions, potentially due to its enhanced capability to capture fine- reee-rained details through CNNs. The context length of LSDT for training and evaluation are shown in Table 5
Table 5: Context Length and Return-to-go Setting for LSDT.
Environment
Context length K
Return-to-go conditioning
Maze2d (umaze, medium, large)
20
300
AntMaze (umaze, medium)
10
100
Walker2d-medine salfchectah-medinne Hopper-medium
5
7000
10
7000
10
7000
Malker2d-medium--eppla
10
7000
10
7000
Hopper-mediun-repla.
20
7000
Walkee22dmediuum medium-expert
10
10000
Halfcheetah medium medium--eppert
10
15000
cepper-medium-eee
10000
B.7 INSTRUCTIONS FOR SELECTING CONVOLUTION KERNEL SIZF.
In DT and its variants, reinforcement learning problems formulated sequence modeling tasks, where future information is unavailable to the model during action generation. Analogous to masked attention in transformers. the weights corresponding tion kernel to prevent information leaking. In DT, a unit of input to the attention module consists of EecturnTo-CCe (RTG), state (s), and action (a). Our Long-Short LSDT incorporates
future time steps are masked the convolu-
three tokens a bralch structure that aligns with this input format. To get better performance, we find that the convolutional kernel size, particularly the part beyond the mask denoted kn. should also be multiple of 3, which alignment with the format of the input tokens in the attention branch. The kernel size can be computed as (kim * 2) - 1. In our approach, we use a kim value of to capture local patterns in adjacent timesteps.
we suggest choosing an appropriate kernel size based on the context length. The Lite Transformer (Wu et al. 2020) and the study introducing dynamic convolution (Wu et al. 2019) both recommend using larger kernel sizes, such as 15 or 31, which we found to be less effective for the masked attention block in our LSDT's decoder. The primary reason is the context length in model, which considerably shorter than what typically used in NLP models. A large kernel size in our context can lead to overfitting and slow down computation. For example, with context length of 1, we only
19

---

Published as a conference paper at ICLR 2025
deal with three tokens time. Choosing kernel size of 15 in such scenario would mean that the convolution operation extends well beyond the relevant context, processing a significant amount of masked or irrelevant information. This not only increases the risk of inaccurately modeling dependencies due to introducing noise but also unnecessarily burdens the computational resources.
B.8 DROPKEY STRATEGY IN LSDT
In the field of computer vision. Li et al. (2022) introduced the Dropkey technique to improve the transformer's capabilities of capturing global information and alleviating overfitting. Unlike the standard attention mechanisms where dropout is applied after computing the attention weights, Dropkey drops the keys in the attention instead of the weights. Intrigued by its potential, we in- vestigated whether this approach could also enhance the performance of LSDT. We adapted this technique by replacing the standard dropout with Dropkey in our attention branch. Our experiments were conducted in the AntMaze environments over 4 random seeds without HFM module. The re- sults in Tablefojindicate that employing DropKey in our LSDT is not only harmful its performance but also introduces instability. Therefore, we didn't 'tt't report this strategy in our final model. How- ever, we have retained the DropKey implementation exploration those interested in it. This includes the possibility of enhancing model performance through a scheduled decrease in the dropout rate, feature we did not explore in depth, due to its suboptimal performance in our preliminary experiments.
our code, which is accessible for our further
Table 6: Impact of Dropout techniques LSDT Performance.
Environment
Standard Dropout
DropKey
Antmaze-umaze
84.5
1.7
80.25 1.9
Antmaze-umaze-divers
81.5
1.7
76.0 + 3.1
Antmaze-medium-ppla
71.5
6.2
59.3 3.6
Antmaze-mediim--dierr
68.5
5.0
50 17.7
B.9 FILTER SIZE EXPERIMENTS
The convolution-only DT achieves strong performance Markovian datasets due to its ability to capture local dependencies. By setting the kernel size Ke to 6, it effectively models inherent as- sociations between elements in two consecutive timesteps. However, this configuration limits the model's ability to incorporate more historical information, which is essential for learning global patterns in Man-Markovian tasks. To investigate whether a larger kernel size can lead to better per- formance in these tasks, we experiment with the Antmaze Antmaze-mediun medium--lmlllll play task DC in Table The results show that increasing the kernel size provides no further improvement and may even degrade performance.
Table Kernel size influence on Antmaze-medium-pue values are reported over random seeds
Environment
Ke
Ke =12
Ke =18
Antmaze-medium-ppay 69.5 + 3.6
52.3 + 5.9
57.0 + 6.1
B.10 ALTERNATIVE WAYS OF CHANNEL SPLITTING AND INFORMATION ROUTING
In the early stages of our design, we explored various routing mechanisms to enhance the model's expressiveness and improve the overall performance of LSDT. While these alternatives did not out- perform the final structure proposed in this paper, exploring more alternative configurations could provide readers with valuable insights for further optimizing our architecture in the future. Here, we detail one of the most promising alternatives (referred to as weDghweight), which has the potential to eliminate the need for manually setting the hyperparameter dimension ratio Ç. In this configura- tion, the full embedding is fed into both branches instead of subset, and the outputs of the branches
20

---

Published as a conference paper at ICLR 2025
combined through learnable weighted sum, computed using modified version of our HFM module. At the suggestion of a reviewer, we also evaluated an alternative routing mechanism, re- ferred to as LSDT-res, which introduces two separate linear projections with residual connections before feeding inputs to the branches. This design aims to enhance feature extraction and poten- tially improve performance. The performance of these alternatives is reported in Table as the mean and standard deviation over three random seeds, using 20 rollouts for Walker2d-medium and Halfcheetah- siscchetah-medduu--eplllg and 100 rollouts for Maze2d-medium
Table 8: Performance comparison across tasks for LSDT-res, weiht, and LSDT.
Task
LSDT-res
LeDT-
LSDT
Walker2d-medium
78.3 + 3.3
78.4 + 0.5
81.0 + 0.4
merccheeta--meiumm-m -epllay
40.9 + 0.6
41.8 = 0.3
42.9 + 0.3
Maze2d-medium
67.6 - 3.0
64.0 + 4.2
68.4 + 3.9
Our preliminary experiments revealed that these designs currently do result in performance im- provements. On the contrary, ..DDT-weiigh increased the total number of parameters for each branch and extended the training time, as both branches processed the full input dimensionality instead of subset. Nevertheless, its ability to dynamically integrate output features from different branches without relying on a pre-set dimension ratio highlights a potential direction to further improve our LSDT, which warrants further exploration in future research.
C EXTENSIVE EXPERIMENTS.
To further verify the performance of the proposed method, test LSDT more complex Adroit tasks. The CQL and IQL values are reported from IQI. (Kostrikov et al. 2021). ). and EDAC values from the original paper (An et al. 2021). We report the mean and standard deviation of 5 random seeds, with 20 evaluations each.
As shown in Table 9 the general results are consistent with those from our other evaluations on D4RL benchmarks. LDST performs first or second in most cases, showcasing its effectiveness and adaptability across various RL tasks. The one exception this is the celccte-clonedd-v. which is hard task for most offline RL algorithms. Moreover, the performance gap between the '-human' and clloned' datasets suggests that L.SDT performs better with higher-quality data.
Table Adroit evaluation results.
Task
CQL
IQL
EDAC
LSDT
pen-human-v0
37.5
71.5
52.148.6
109.2:16.2
hammer-human-vo
4.4
1.4
0.840.4
12.0=6.0
door-human-vo
9.9
4.3
10.746.8
18.6=8.8
0.2
0.1
0.1+0.1
1.5+0.4
pen-cloned-v0
39.2
37.3
68.247.3
63.6=8.2
hammer-cloned-vo
2.1
2.1
0.3.10.0
9.414.9
door-cloned-v0
0.4
1.6
9.6-8..3
5.043.8
relocate-cloned-
-0.1
-0.2
0.0.0..0
0.010..1
D EXPERIMENTS DETAILS.
Our code can found in the submitted material. The Decision Transformer and our Long-Short Decision Transformer, used in the experiments, are constructed based on the official DT open- source cod and the min-decision transformef Additionally, the Dynamic Convolution imple-
21

---

Published as a conference paper at ICLR 2025
mented in study is developed using the Fairseq toolki The code for Decision Conveormer (DC) is sourced from official codebase Each model trained for 10 updates. The implementa- tion of our model is publicly available at
D.1
DT MODEL DETAILS
The architecture of our DT mainly follows the official DT, with the key modification of substituting the ReLU nonlinearity with GeLU. The hyperparameters were fine-tuned as detailed in Table [1 The context length and RTG for the Antmaze experiments are shown in Table TO
Table 10: Context length and Return-to-go setting for DT.
Environment
Context length
Return-to-go conditioning
AntMaze-umaze
20
AntMaze-umaze-divens
20
AntMaze-medium-
20
AnMMaze-meduum-dierss
20
Table 11: Hyperparameteters of DT.
Hyperparameter
Value
Number of layers
3
Number of attention heads
Batch size
128
Learning rate
4 x 10
Grad norm clip
0.25
Weight decay
3 x 10-4
Embedding dimension
128
Nonlinearity function
GeLU
RTG scale
Dropout
0.1 Linear warmup for first 104 update
Learning rate decay
To compare the performance on the 'Antmaze-medium' environments, we evaluate DC using the same architecture that was implemented for the 'Antmaze-umaze' tasks in their paper. The hyperpa- rameters are sourced from their paper 2 Additionally, we conduct evaluations the Maze2d tasks using DCh, which is introduced in the DC paper to enhance the model's 's long-term credit assignment capability. In the architecture of DCh, the final convolution layer replaced with an attention layer.
D.2 IMPLEMENTATION DETAILS OF LSDT
In architecture of LSDT, notably, the of the input dimensions of the two branches must equal the embedding dimension, as we distribute different parts the embedding dimension to each branch for further processing. The hyperparameters of our LSDT for Maze2d, AntMaze, Adroit, and other locomotion tasks are presented in Table 13 The context length and RTG for training and evaluation are shown the previous section Table
Embedding dimension: Following the suggestions from the DC paper Wu al. (2019). use an embedding dimension of 256 'Hopper-medium' and Hopper-mediun-replan We also observed significant enhancement in performance with this larger dimension. For other environments, we use a dimension of 128. Additionally, we train 'Hepper-medium' without the action information.
22

---

Published as a conference paper at ICLR 2025
Table 12: Hyperparameteters of DC on Antmaze and DCA on Mazeld domain.
Hyperparameter
Value
Number of layers
3 convolution blocks, (DC)
2 convolution blocks attention block, (DC;))
Number of attention heads
1. (DC) (DCr.)
Batch size
64
Learning rate
x 10
Grad norm clip
0.25
Weight decay
1 x 10-4
Embedding dimension
128
Nonlinearity function
GeLU
RTG scale
1
Dropout
0.1
Context length
8 Linear warmup for first 10 update
Learning rate decay
Table 13: Hyperparameteters of LSDT.
Hyperparameter
Value
Number of layers
3
Grad norm clip
0.25
Weight decay
1 x 10-4 GeLU
Nonlinearity function
Convolution type
DynamicConv
Kernel size
3. Maze2d
6, other environments
RTG scale
AntMaze, Maze2d,
1000 Locomotion tasks
Dropout
0.25
Fusion method
HFM (Appendix B.3
Dropout for Convolution
0.2
Linear warmup for first 10 updates
Learning rate decay
followed by single-cycle cosine annealing
Batch Size: We use 256 'Antmaze", 'Adroit' and 'Maze2d' to stabilize the training pro- cess. For locomotion tasks, use 64.
Number of attention heads: For more complex tasks such Antmaze' 'Adroit' and Maze2d', we utilize heads the attention branch, with the exception 'Antmaze-umaze' where we use 1. For other tasks, we use only head.
Learning rate: We train LSDT with learning rate of 1 x 10-4 4 in locomotion tasks and mutmaze-medium-divereee" and x 10-4 in other environments.
Number of kernels: To reduce the number of parameters, use the weights- sharing strategy in our model. The number of kemnels can be computed as: (Number of dimensions in convolution branch) +
The best channel dimension ratio & for the convolution branch on each environment is shown in Table 14
23

---

Published as a conference paper at ICLR 2025
Table 14: The best channel dimension ratio of each task.
Tasks
Dimension ratio
Antmaze-umaze
50%
Antmaze-umaze-divvrr
75%
Autmaze-medium-play
75%
Antmaze-mediun
-ediim--ddeeesee
50%
Maze2d-umaze
50%
Maze2d-medium
12.5%
Maze2d-medium
25%
Walker2d-medium
12.5%
Halfcheetab-medium
50%
Hopper-medium
100%
Walker2d-meedium-rephh
50%
halffhheetah-medium
50%
Hopperr-medium-repla
100%
Walkerrdd-medium-expprrr
12.5%
Halfchheeta--mddiu--cpppe
Halfcheetah
75%
Hoopee--medium-oe expeert
75%
D.3 LSDC MODEL DETAILS
To demonstrate that other approaches can benefit from our Song-Short architecture, we present LSDC. In LSDC, we maintain the same hyperparameters and architecture as our LSDT, except we replace the dynamic convolution with the type of convolution used in DC.
E EXPERIMENTS COMPUTE RESOURCES
To help practitioners in the community evaluate the practical trade-offss of implementing LSDT, we conduct an empirical analysis of the computational efficiency. All models are trained in the Walker2d environment with the same batch size of 64 and context length of 10. The experiments are run on a desktop with the following configuration: an NVIDIA 4080 GPU, an Intel 13700KF CPU, and 32GB of RAM. We present detailed comparison across several metrics, including:
FLOPs, Total parameters comparisons with DT and DC. (Table 15
Training time comparisons with single- brancch architectures (DT and DC) (Table 15
GPU Memory usage analysis across dimension ratios (Table 16
Table 15: Comparison of Compute Resources across DT, DC, LSDT, and LSDC.
Metric
DT
DC
LSDT
LSDC
Flops (MMAC)
17.99
11.01
17.08
16.97
Parameters (M)
1.13
0.9
1.09
1.09
Memory Usage (GB)
0.1
0.1
0.11
0.1
Training Time (s)
381
364
713
473
LSDT currently requires longer training times, primarily due to the computational overhead of dy- namic convolution in the short-term branch. However, our architecture is highly flexible, allowing for improvements through more efficient convolutional computations. As we discussed Section 4.4 and illustrated in Table [5] replacing dynamic convolution with alternatives. such those em- ployed in DC, can significantly reduce runtime. This adaptability highlights the potential for further optimization, offering pathway for future research to balance efficiency and performance.
24

---

Published as a conference paper at ICLR 2025
Table 16: Memory usage of LSDT under different &
configurations.
Memory
12.5%
0.1IGB
25%
0.12GB
50%
0.12GB
75%
0.14GB
F LIMITATIONS
While LSDT demonstrates competitive results across various tasks, there are areas for further im- provement. Currently, the channel dimension of the input is proportionally split and fed into each branch, and the optimal dimension ratio, ç, may vary across tasks. To address this, we conducted experiments identify appropriate values for implementations. Additionally, in Appendix B.10, we introduce a potential alternative architecture that eliminates the need for manual tuning. Although this approach is still exploratory, it shows promise, and plan to further investigate how to efficiently and dynamically select in future research.
25


Reviews:
[{"review_id": "u7xTRndNZU", "reviewer": "Reviewer_wr6s", "summary": "This paper targets two critical limitations in existing Decision Transformers (DTs)-based offline reinforcement learning: (1) DTs' weakness in capturing local dependencies which is associated with Markovian properties in many offline-RL datasets; (2) DTs’ struggles with goal-reaching tasks involving diverse goals and sparse rewards. To address these two issues, the authors propose a dual-branch architecture, Long-Short Decision Transformer (LSDT), that combines self-attention for global patterns and dynamic convolution for local features, along with a goal-state conditioning mechanism. The main contribution is the asymmetric dimension-splitting between branches. Experiments are conducted across diverse environments and the results demonstrate state-of-the-art performance on several D4RL benchmarks, particularly showing significant improvements in complex environments like Antmaze-medium and Maze2d-large.", "questions": "**(Q1)** Could the authors provide training curves showing the convergence behavior of both branches? This analysis would provide insights into how the dual-branch architecture learns to balance global and local features during training. Specifically, it would be valuable to see:\n\n- The loss trajectories for each branch separately;\n- The relative contribution of each branch to the final decision at different training stages;\n- Whether one branch tends to converge faster than the other;\n\n**(Q2)** Have the authors considered using adaptive dimension ratios that change during training? \n\n**(Q3)** To better understand the architecture benefits, I suggest the authors provide t-SNE plots of learned state representations from both branches. This could provide more insights into the technical contribution of this work.\n\n**(Q4)** I suggest the authors conduct an empirical analysis of the computational efficiency. This would help practitioners in the community evaluate the practical trade-offs of implementing LSDT, including:\n\n- FLOPs comparison with baseline models;\n- Memory usage analysis for different dimension ratios;\n- Training time comparison with single-branch architectures;\n\n---\n**Additional Comments:**\n\nI hope my review helps to further strengthen this paper and helps the authors, fellow reviewers, and Area Chairs understand the basis of my recommendation. I also look forward to the rebuttal feedback and further discussions, and would be glad to raise my rating if thoughtful responses and improvements are provided.\n\n---\n## **-------------------- Post-Rebuttal Summary --------------------**\n\nThe additional experiments, discussions, and revised manuscript provided by the authors have significantly strengthened the work and addressed most of my concerns. I suppose this work can provide knowledge advancement to the community, and I look forward to the final version manuscript, which incorporates the additional insights and information presented in the rebuttal stage.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**(S1) The Significance and Clear Presentation of Research Question:**\nThe paper addresses a critical challenge in Transformer-based offline-RL with great clarity and motivation. The key motivation lies in identifying and solving the global and local dependencies modeling, which has been a critical limitation in existing RL architectures. Unlike existing approaches that treated this as an either-or choice (DTs focusing on global, DCs on local dependencies), the proposed LSDT introduces dual-branch parallel processing that captures both types of dependencies simultaneously, and thereby hope to benefit offline-RL tasks.\n\n**(S2) Technical Soundness:** \nThe technical contributions of the proposed LSDT are well-reasoned. Concretely, the Dynamic Convolution in the short-term branch represents an improvement over static convolution methods. By dynamically generating weights based on current input rather than using fixed weights after training, the model may achieve better generalization across diverse environments. The connection between LSDT and the targeted question is clear and reasonable. This is evident in the experiments where LSDT achieves great performance in both Markovian environments (with strong local feature requirements) and the non-Markovian ones (with strong long-range modeling requirements). The asymmetric dimension splitting, controlled by ratio ξ, provides flexibility in adapting the architecture to different tasks. This is demonstrated through ablation studies showing how different ratio values optimize performance across various environments (e.g., ≥50% for Markovian and ≤50% for non-Markovian tasks). Complementing these components, the goal-state conditioning mechanism, while straightforward, addresses the path planning limitations in sparse reward settings. The results on Antmaze tasks (showing improvements from 0% to over 70% success rate) provide strong validation of this method. The optional Hybrid Fusion Module, though not always necessary, presents an approach to dynamic feature selection, with visualization studies clearly showing its ability to weight different features based on task requirements adaptively.\n\n**(S3) Thorough Experiments and Ablation Studies:**\nThe experiments are thorough and well-structured. The authors conduct extensive experiments across diverse environments, including MuJoCo locomotion tasks, Maze2d navigation, AntMaze environments, and Adroit manipulation tasks. The ablation studies are comprehensive, examining the impact of different architecture choices, such as dimension ratio (across 20 different values), kernel size (from 6 to 18), and fusion methods. The performance gains are consistently shown across both Markovian datasets (achieving 82.0 average normalized score in locomotion tasks) and non-Markovian datasets (significant improvements in Maze2d-large and Antmaze-medium). The authors also provide an empirical analysis of failure cases and performance characteristics under different hyper-parameter settings, making the results valuable for practical implementation and future studies in the community.", "weaknesses": "**(W1) Limited Theoretical Analysis and Foundation:** \nWhile the empirical results are impressive, the lack of formal analysis leaves several crucial questions unanswered. First, there is limited theoretical justification for why the combination of self-attention and dynamic convolution in parallel branches should perform better than sequential or alternative operations in RL. Second, the interaction between the two branches lacks theoretical examination. For instance, how information flows between them and potential interference effects, remain unexplored. Third, the paper provides no theoretical bounds or guarantees on the optimal dimension ratio ξ. While empirical guidelines are provided (e.g., ≥50% for Markovian tasks), a theoretical framework supporting these choices would make the approach more principled. \n\n**(W2) Limited Technical Originality and Domain-Specific Analysis:** \nWhile the proposed dual-branch architecture exhibits strong empirical results, its design principle of combining global and local feature extraction has been extensively explored in other domains, particularly Computer Vision and NLP. As such, this work would significantly benefit from a more in-depth analysis of why and how this architecture specifically addresses the unique challenges in reinforcement learning. For example, a deeper analysis to show the relationship between Markovian/non-Markovian properties and global-local feature extraction (how local convolution patterns capture Markovian state transitions, while global attention mechanisms model longer-term dependencies in policy behavior). This could be supported by visualization that expresses how different types of dependencies are processed in each branch for decision-making. \n\n**(W3) Architecture Designs and Computational Efficiency Concerns:** \nSeveral architectural choices raise concerns about practical applicability and scalability. The requirement for manual tuning of the dimension ratio ξ represents a fundamental limitation in the adaptability of LSDT architecture. While the authors provide general guidelines (≥50% for Markovian, ≤50% for non-Markovian tasks), the absence of an adaptive way for determining optimal ratios necessitates extensive experimentation for each new environment. This becomes particularly problematic in real-world applications where task characteristics may not be clearly Markovian or non-Markovian or may shift during deployment. Additionally, the computational complexity of the dual-branch architecture deserves deeper analysis. While the authors mention reduced context length requirements compared to standard DT, they don't provide a comprehensive analysis of the compute-performance trade-off.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "btS3LOLMjK", "reviewer": "Reviewer_vshZ", "summary": "The manuscript proposes the Long-Short Decision Transformer (LSDT), a model that effectively combines the strengths of Decision Transformer (DT) and Decision Convformer (DC) through a dual-branch architecture.\nThis architecture uses a dimension ratio, ξ, to dynamically control the proportion of channels processed by each branch—self-attention for long-range dependencies and convolution for local patterns.\nThe approach is simple yet intuitive, as it leverages complementary feature extraction methods to enhance both global and local dependency modeling, adapting flexibly to diverse reinforcement learning tasks.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The proposed Long-Short Decision Transformer (LSDT) presents an interesting approach by combining attention and convolutional pathways to handle both global and local dependencies effectively.\nThis dual-branch design not only enhances interpretability but also achieves strong performance across diverse reinforcement learning tasks, particularly outperforming benchmarks on the D4RL dataset.\nThe flexibility offered by the dimension ratio, ξ, adds adaptability, allowing LSDT to maintain robust performance in both Markovian and non-Markovian environments.\nAdditionally, the model’s simplicity and modularity enable straightforward integration into existing architectures, further extending its practical utility.", "weaknesses": "The experiments on the routing mechanism in the LSDT are limited, leaving open questions about potential improvements.\nGiven that the proposed method can be viewed as a two-expert MoE (Mixture of Experts) architecture, alternative ways of channel splitting and information routing could be explored.\nFor instance, introducing two separate linear projections with residual connections before feeding inputs to the branches could enhance feature extraction.\nAnother approach might involve feeding the same embedding into both branches and combining the outputs with a learnable weighted sum.\nEvaluating these alternative configurations could offer insights into optimizing the dual-branch structure, potentially enhancing the model’s performance and efficiency.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3YoOHe2nZ2", "reviewer": "Reviewer_NBtN", "summary": "This submission presents a novel architecture for the decision transformer named LSDT that tries to capture both long-term and short range dependencies, such as local spatiotemporal associations, for offline RL tasks. This is achieveed by adding a second, convolutional block next to the self attention block, and the sizes of the two blocks (eg in terms of dimensions) can vary according to the needs of the task to solve.  The paper further proposes to incorporating goal states into the conditioning information for further performance gains in some cases. Experiments are conducted on the D4RL benchmark, where LSDT achieces performance comparable to state-of-the-art RL methods.", "questions": "Q1) Could the CDT subfig in Figure 1 also be depicting the Decision Convformer (DC)? It would be clearer if DT and DC were clearly marked if this is  the case\n\nQ2) How well would the DC perform if one substitutes their conv filters with DynamicConv?\n\nQ3) Are \\xi identical across all datasets in Table 1? \n\nQ4) Is goal conditioning used in Table 1, and if not what is the performance of LSDT-gs in this case?\n\nQ5) IS FIg3 depicting a  single sequence or averages over many?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "S1) The motivation if the proposed approach is clear and the paper is well written. The closest related works are cited and the contributions pf the paper are clearly presented and illustrated both qualitatively and quantitatively.\nS2) Although the technical contributions are incremental, the paper presents clear figures that illustrate the strenghts of the proposed method, and also ablations that study the key points of the proposed architecture\nS3) The method seems to achieve results comaprable to the state of the art.", "weaknesses": "W1) The technical contribution is following other works that propose two-branch transformer blocks: appending a convolutional branch in the architecture, and using a straighforward attention masking scheme to capture long vs short dependences. \n\nW2) The split across the two branches is simple, ie across the channels dimension, and it is manual via hyper parameter \\xi, an adjustment which is non trivial and highly task based. The submission would be stronger if an automatic way of selecting \\xi was proposed. It would be great if the authors discussed potential methods for automating this. \n\nW3) An issue uncovered in Fig7 is the following: this paper is presenting a method for a compromize between DT and CDT, via splitting the dimensions across the two. However, we see in Fig 7 that in most cases there doesnt seem to be a \"peak\" inbetween the already existing edge cases.  Could the authors discuss this further? It would be nice to address this apparent discrepancy between one of the motivationsof their method and the results shown.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)