Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making
Authors: Jeonghye Kim
Suyoung Lee
Woojun Kim
Youngchul Sung
Venue: ICLR 2024 spotlight (2024)
Keywords: MetaFormer
Convolution
Reinforcement Learning
Representation Learning
Primary area: reinforcement learning
TL;DR: We propose Decision ConvFormer, a new decision-maker based on MetaFormer with three convolution filters for offline RL, which excels in decision-making by understanding local associations and has an enhanced generalization capability.

Abstract: The recent success of Transformer in natural language processing has sparked its use in various domains. In offline reinforcement learning (RL), Decision Transformer (DT) is emerging as a promising model based on Transformer. However, we discovered that the attention module of DT is not appropriate to capture the inherent local dependence pattern in trajectories of RL modeled as a Markov decision process. To overcome the limitations of DT, we propose a novel action sequence predictor, named Decision ConvFormer (DC), based on the architecture of MetaFormer, which is a general structure to process multiple entities in parallel and understand the interrelationship among the multiple entities. DC employs local convolution filtering as the token mixer and can effectively capture the inherent local associations of the RL dataset. In extensive experiments, DC achieved state-of-the-art performance across various standard RL benchmarks while requiring fewer resources. Furthermore, we show that DC better understands the underlying meaning in data and exhibits enhanced generalization capability.

Published as a conference paper at ICLR 2024
DECISION CONVFORMER: LOCAL FILTERING IN METAFORMER IS SUFFICIENT FOR DECISION MAKING
Jeonghye Kim Suyoung Lee', Woojun Kim Youngehul Sung KAIST *Carnegie Mellon University
ABSTRACT
The recent success of Transformer in natural language processing has sparked its use in various domains. In offline reinforcement learning (RI., Decision Trans- former (DT) is emerging as a promising model based on Transformer. However, we discovered that the attention module of DT is not appropriate to capture the inherent local dependence pattern in trajectories of RL modeled as Markov de- cision processes. To overcome the limitations of DT, sequence predictor, named Decision ConvFormer (DC), based on the architecture of MetaFormer, which is general structure to process multiple entities in parallel and understand the interrelationship among the multiple entities. DC employs lo- cal convolution filtering as the token mixer and can effectively capture the inherent local associations of the RL dataset. In extensive experiments, DC achieved state- of-the-art performance across various standard RL benchmarks while requiring fewer resources. Furthermore, we show that DC better understands the underly- ing meaning in data and exhibits enhanced generalization capability. Our code available at httpps: /ttt::::/bbanieee hwtp:///...oooccccacciiaooaaaaaa
propose novel action
INTRODUCTION
Transformer (Vaswani et al. 2017) proved successful in various domains including natural language
processing (NLP) (Brown et al. 2020 Chowdhery et al.) 2022). computer vision (Liu et al.) 2021 Hatamizadeh al. (002). Transformer is special instance more abstract structure referred to as MctaFormer (Yu et al. 2022). which is general architecture that takes multiple entities in parallel, understands their interrelationship. and extracts important features for addressing specific tasks while minimizing information loss. As shown in Fig. [ MetaFormer is composed of blocks, where each block contains normalizations, token mixer, residual connections, and a feedforward network. Among these components, the token mixer plays a crucial role in information exchange among multiple input entities. In the case of Transformer, an attention module is used as the token mixer. The attention module has been generally regarded Transformer's 's main success factor due to its ability to capture the information relationship among tokens across a long distance.
With the successes in other areas, Transformer has also been employed in RL, especially in offline RL, and provides an alternative to existing value-base based or policy-gradien gradient methods. The represen- tative work in this vein is Decision Transformer (DT) (Chen al.) 2021). DT directly leverages history information to predict the next action, resulting in competitive performance compared with existing approaches to offline RL. Specifically, DT takes trimodal token sequence of state, action, and return as input, and predicts the next action to achieve target objective. The input trimodal sequence undergoes information exchange through DT's attention module, based on the computed relative importance (weights) between each token and every other token in the sequence. Thus, the way that DT predicts the next action is just like that of GPT-2 (Radford et al.) (019) in NL. with minimal change. However, unlike data sequences in NLP for which Transformer was originally developed, offline RL data has an inherent pattern of local association between adjacent timestep to- kens due to the Markovian property, as seen in Fig. 2 This dependence pattern is distinct from that in NLP and is crucial for identifying the underlying transition and reward function of an MDP (Bell- man 1957). which are fundamental for decision-making in turn. As will see shortly, however,
Youngchul Sung and Woojun Kim are co-corresponding authors.

---

Published as a conference paper at ICLR 2024
Token Mixer Block
Attention Block
Convoluioon Block
8 state
action
R warn-to-go (RTG)
Feed Foward
Feed Foward
Feod Foward
Norm
Norm
Norm
B1
Q2
Token Mixer
Attention
Convolution
R
Ra
Norm
Norm
Norm
Figure 2: The local dependence graph of offline RI. dataset: Blue arrows rep- resent Markov property, red arrows indi- cate the causal interrelation per single timestep, and the gray dotted line shows the correlation of the adjacent returns.
Input Emb.
Input Emb.
MetaFormer
Decision Transformer
Decision ConvFormer (Ours)
Figure I: The network architecture of MetaFormer, DT, and DC.
the attention module of DT is an overparameteriizatoon and not appropriate to capture this distinct local dependence pattern of MDPs.
In this paper, we propose new action sequence predictor to overcome the drawbacks of DT for of- fline RL. The proposed architecture named Decision ConyFormer (DC) is still based on MetaFormer but the attention module used in DT is replaced with new simple token mixer given by three causal convolution filters for state, action, and return order to effectively capture the local Markovian de- provide consistent context for local association and
pendence in RL dataset. Furthermore, tpsk-specific dataset traits, we use static filters that reflect the overall dataset distribution. DC has very simple architecture requiring far fewer resources in terms of time, memory, and the number of parameters compared with DT. Nevertheless, DC has a better ability to extract the local pattern among tokens and inter-modal relationships as we will see soon, yielding superior performance compared to the current state-of-the-art offline RL methods across standard RL benchmarks, includ- ing MuJoCo, AntMaze, and Atari domains. Specifically, compared with DT, DC achieves 24% performance increase in the AntMaze domain, a 39% performance increase in the Atari domain, and a notable 70% decrease in training time in the Atari domain.
MOTIVATION
An RL problem be modeled Markov decision process (MDP) M S,.. where Po is the initial state distribution, S is the state space, A is the action space, (5 ae) is the transition probability. R(st, 0₂) ) is the reward function, and y (0, 1) is the discount factor. The goal of conventional RL is to find an optimal policy π that maximizes the expected return through interaction with the environment.
Offline RL In offline RI, unlike the conventional setting, learning performed without interac- tion with the environment. Instead, it relies on dataset D) consisting of trajectories generated from unknown behavior policies. The objective of offline RL is to learn a policy by using this dataset D maximize the expected return. One approach is to use Behavior Cloning (BC) (Bain & Sammut, 1995), which directly learns the mapping from state to action based on supervised learning from the dataset. However, the offline RL dataset often lacks sufficient expert demonstrations. To address this issue, return-conditioned BC has been considered. Retum-conditioned BC exploits reward in- formation in the dataset and takes target future return as input. That is, based on data labeled with rewards, can compute the true return, referred to return-to-go (RTG), by summing the future rewards from time step / from the dataset: fi Er= Tr. In dataset containing many subopti- mal trajectories, this new label R serves as a crucial indicator to distinguish optimal trajectories and reconstruct optimal behaviors.
Decision Transformer (DT) DT is a representative approach to oeturn-conditioned BC. DT employs Transformer convert RL problem sequence modeling task (Chen et al. 202T). DT treats a trajectory as a sequence of RTGs, states, and actions. At each timestep DT constructs input sequence to Transformer a sub-trajectory of length K timesteps:
2

---

Published as a conference paper at ICLR 2024
T--K+1a (Re (R 8t111 R Re, st), and predicts action de based on T+--+1
In detail, each element of the input s T+-K+ is linearly transformed to a token vector of the same dimension d to compensate for the different sizes of trimodal components R,, &, and ár- Then, the 3K token vectors go through series of blocks, where each block consists of layer normalization, attention module, residual connection, and a feedforward network. In particular, the attention module consists of three matrices Q of size d x d', K of size d x d', and V of size x d. These three matrices generate the query, key, and value vectors from the input token vectors { i =1 3K 1} of size 1 x d, respectively, as follows:
qi = x; K r,K, 4= =
Then, the i-th output of the attention module is given by
3K-1
0j ,,,
3K
with causal masking on the combination weights Oijy i.e., oig = 0, > i. The combination weights aij. also known as attention score, capture the dependence of the i-th output on the j-th input token through the following formula:
azy softmax (f (gt (
(3)
A jayl
A
direct
20
Query
Query
DT
Direct Learning of A
30
68.4
88.2
90
Key j
Key
(a)
(b)
(c)
Motivating results in hopper-medinee (a) attention scores of DT (Ist layer), (b) atten- tion scores of direct learning (1st layer), and (c) performance comparison.
Figure 3:
Attention Score Analysis of DT Our quest begins with the question "Is the attention module initially developed for NLP still an appropriate occl-association identifying structure for data se- quences of MDPs?" To answer this question,
performed an experiment on the widely considered v um dataset with diverse trajectories. Fig. shows the learned at- offline MuJoCo hopper med
tention map of DT with K = 20. The index i (or j) is ordered such that i = corresponds to RTG R₁- 11, i = 2 to state 9.-K+1.
4 to RTG R₂- up until i 59 to the latest state in -11-+ Since causality is applied, au = 0, Vj > i for each i. That is, the attention matrix A [a.y] is lower-triangular. We observe that the attention matrix of DT is in the form of full lower triangular matrix if we neglect the column-wiss periodic decrease in value (these columns correspond RTGs). In the case of the latest s, position of 59, the output depends on up to the past K 20 timesteps. Note that the state sequence forms a Markov chain. From the theory of ergodic Markov chain, however, we know that a Markov chain has a forgetting property, that is, as Markov chain progresses, it soon forgets the impact of past states (Resnick) 1992). Furthermore, from the Markovian property, $t-2: 8t-3.... should be independent of st given St-1 for each l. The result in Fig. Balis not consistent with these facts. Hence, instead of parametriz- ing Q and K and obtaining aij with Eqs. T1 and 33 as in DT,, we directly set the attention matrix A = [a_fireat] as learning parameters together with V, and directly learned {adjreet] and V. The resulting attention matrix A is shown in Fig. 3b Now, it is seen that the resulting attention matrix A is almost a banded lower- trianttri matrix, which is consistent with the Markov chain theory, and its performance is far better than DT as shown in Fig. 3dl Thus, the full lower-triangul structure of
3 to action Mt-K+1s
3

---

Published as a conference paper at ICLR 2024
the attention matrix in DT is an artifact of the method used for parameterizing the attention module, i.., parameterizing Q and K and obtaining dzy with Eqs. { and (3), and does not truly capture the local associations in the RL dataset. Indeed, recent study by Lawson & Qureshi (2023) showed that even replacing the attention parameters learned in one MuloCo environment with those learned in another environment results in almost no performance decrease. One may think that DT can prop- erly extract the local dependency simply by reducing the context length K' to focus on neighboring information and improve its performance. However, this is not the case shown in Appendix G.3 DT with reduced K yields worse performance.
THE PROPOSED METHOD: DECISION CONVFORMER
For our action predictor, we still adopt the MetaFormer architecture, incorporating the recent study of Yu et al. (2022) suggesting that the success of Transformer, especially in the vision domain, stems from the structure of MetaFormer itself rather than attention. Our experiment results in Figs. 3b and 3c guide a new design of token mixer with proper model complexity for MetaFormers as RL. action predictors. First, the lower banded structure of A implies that for each time i, we only need to consider fixed past duration for combination index j. Such linear combination can be accomplished by linear finite impulse response filterinng Second, note that the attention matrix elements [asy] of DT vary over input sequences (T- (T=-K+1:t) } for different l's since they are functions of the token vectors } as seen in Eqs. (1) and (3) although Q and K do not vary. However, the direct attention matrix parameters [agij obtained for Fig. db not vary over input sequences {T--K Fre-+11:e> for different f's. This suggests that can simply use impur-sequence- sequeenc--ndddeeedenn static linear filtering. Then, the so-obtained filter coefficients will capture the dependence among tokens inherent in the whole dataset. The details of our design based on this guidance are provided below.
3.1 MODEL ARCHITECTURE
The DC network architecture adopts a MetaFormer as shown in Fig. I In DC, the token mixer of the MetaFormer is given by convolution module, based on our pre- For every timestep t, the input sequence Ie is formed as 4 RI-1, 8t-1, 0,-1, Re, 8t), where K is the context length. is subjected to a separate input embedding for each of RTG, state and action, yielding Te (
vious discussion.
(Rt-K+1, St-K+1,
(S. +1); Emba (a+
Emb x (Re); Emb) E 1(3
Here, the sequence length is 3K 1, reflecting the trimodal tokens, and d indicates the hid- den dimension. Then, T, passes through the convolution block stacked N times, each compris- ing two sub-blocks. The first sub-block involves layer normalization followed by token mixing through convolution module, expressed as 7, = Conv (LN(T,)) + T, The second sub-block also involves layer normalization followed by a Feed Forward Network, expressed as 2 sub-block FFN (LN sub-block)) )) + 2. sub-block The FFN is realized as a two-layered MLP.
3.2 CONVOLUTION MODULE
The primary purpose of the convolution module is to integrate the time- domain information among neighboring tokens. To achieve this goal with simplicity, we employ ID depthwise convolution on each hidden dimension independently by using filter length L., leaving hidden dimension-wist mixing to the later feedforward network. Considering the disparity among state, action, and RTG, we use three separate convolution filters for each hidden dimension: state filter, action filter, and RTG filter, to capture the unique information for each embedding, Thus, for each convolution block, we have set of 3d convolution kernels with 3dI. kernel weights, which are our learning parameters.
The convolution process is illustrated in Fig. 4 The embeddings T, defined above first go through layer normalization, yielding X, = LN(T.) E (333-1)xd Note that each row of X, corresponds to dddimensional token, whereas each column of X, corresponds
time series of length 3k - for a hidden dimension. The convolution is performed for the time series in each column of X as shown Fig. 4 Specifically,, consider the convolution operation on the q-th hidden dimension column, where q = 1,2,... d. Let w/ wi[l], and [1[[], = 0, 1,..., L - 1, denote the coeffi- cients for the RTG, state and action filters for the q-th hidden dimension, respectively. First, the q-th
4

---

Published as a conference paper at ICLR 2024
Depthwise Convolution
Indices correspond the oldest timestep
X
R
Gj
X
Convolation
modip, 3)
modgy 3)
3K
modiy, 3)
d
X,
C₁
Figure 4: The overall convolution operation of DC.
column of X, appended left by L --1 zeros, i.e., p, q] 0 for p 1,-I,...., -L 2), to match the size for convolution. Then, the convolution output for the q-th column is given by
E'-i w Xz(p - L, gl if mod (p, 3) = 1,
C.lp, =
w X.(w 1,
if mod [p, 3) 2, p- 1,,, 3K-1 -1 (4)
[ Xx(p L.
if (p, =0
for each q = 1.2,.... d. The reason for adopting three distinct filters for (p, 3) 11 (p: RTG position), = 2 (p: state position), or = 3 (p: action position) to capture different semantics when the current position corresponds to RTG, state action. We set the filter length I, = 6 covering the state, action, and RTG values of only the current and previous timesteps, incorporating the Markov assumption. Nevertheless, different filter length be chosen or optimized for given task, considering that the Markov property can be weak for certain tasks. fact, setting = corresponds to imposing inductive bias for the Markov assumption on the locality in association with a dataset. A study on the impact the filter length is available in Appendix G.2
The number of parameters of the token mixer of DC is 3dL, whereas that of Q and K of the attention module of DT is 2dd'. In addition, DT has the matrix of size x d, whereas DC does not have V at all. Since I. < min(d', d), the number of parameters of DC is far less than that of the attention module of DT. The actual number of parameters used for training DT and DC can be found in Appendix
F We conjecture this model complexity is sufficient for token mixers of MetaFormers for most MDP action predictors. Indeed, our new parameterization performs better than even the direct parameterization of A and V used for Fig. 3b The superior test performance of DC over DT in Sec. and especially the result in Sec. 5.3 support our conjecture.
Hybrid Token Mixers For environments which the Markovian property is weak and credit assignment across long range is required, an attention module in addition to convolution modules can be helpful. For this, the hybrid architecture with N MetaFormer blocks composed of the initial N 1 convolution blocks and a final attention block can be considered.
3.3 TRAINING AND INFERENCE
Training In the training stage, K K-length subtrajectory is sampled from offline data D and passes through all DC blocks. Subsequently, the state tokens that have traversed all the blocks undergo final projection to predict the next action. The learning process minimizes the error between the predicted action à Te( (oolf-K44 K+1it, 8t- K4la, at- K+1a-1) and the true action a, for t = 1,..., K, given by
K
2
LDc = ET
ai
8t-K1 -
K+1--1)
(5)
K
Inference In the inference stage, the true RTG is unavailable. Therefore, similarly to Chen et al. (021), as the initial RTG set target RTG that represents the desired performance. During the inference, DC receives the current trajectory data, generates an action to obtain the next state and reward, and subsequently subtracts the reward from the preceding RTG.
5

---

Published as a conference paper at ICLR 2024
4 RELATED WORKS
Return-Conditionel BC Both DC and DT fall under the category of return-condtioned BC,
an active research field of offline RL (Kumar et al. 2019 Schmidhuber [019 Chen et al. 2021 Emmons et al. 2021; David al. 2023). For example, RvS (Emmons et al. 2021) demonstrates comparable performance to DT by modeling the current state and return with a two-layer MLP. This highlights the potential for achieving robust results without resorting to complex networks or long-range dependencies. On the other hand, Decision S4 (David et al., (023) emphasizes the importance of global information in the decision-making process. It resolves the DT's scalability issue by incorporating the S4 sequence model as proposed by [Gu et al. (2022). Unlike the two models, our approach focuses on accurate modeling of local associations and offers flexibility to effectively incorporate global dependence if necessary.
From the context of visual offline RL. Shang et al. 2022 pointed out DT's limitations in com- prehending local associations. They proposed capturing local relationships by explicitly modeling single-step transitions using the Step Transformer and combining ViT-like image patches for better state representation. In contrast, our method does not require training additional models on top of DT. Instead, we replace DT's attention module with a simpler convolution module.
Offline RL with Online Finetuning It is known that the performance of models trained through offline learning is often limited by the quality of the dataset. Thus, finetuning through online inter- actions can improve the performance of offline-pretrainel models (Zhang et al. [2022] Luo et al.) 2023). Overcoming the limitations of DT for online applications, Zheng et al. (2022) proposed an Online Decision Transformer (ODT), which includes stochastic policy and an additional max-
entropy objective in the loss function. A similar method can be applied to DC for online finetuning. We refer DC with online finetuning Online Decision ConvFormer (ODC).
EXPERIMENTS
We carry out extensive experiments to evaluate the performance of DC on the D4RL (Fu et al. 2020) MuloCo, D4RL AntMaze, and Atari (Mnih al.) 2013) domains. More on these domains can be found in Appendix [A] The primary goals of our experiments are 1) to compare DC's performance in offline RL benchmarks with other state-of-the-arr baselines, and especially, to check whether the basic DC model using local filtering is effective in the Atari domain or not, where long-range credit assignment is known to be essential, 2) to determine whether DC can effectively adapt and refine its performance when combined with online finetuning not, 3) to see whether DC can capture the intrinsic meaning of data rather than merely replicating behavior or not, and 4) to evaluate the impact of each design element of DC on its overall performance.
5.1 MUJoCo AND ANTMAZE DOMAINS
We first conduct experiments on the MuJoCo and AntMaze domains from the widely-used D4RL eF al. 2020) benchmarks. MuJoCo features continuous action space with dense rewards, while AntMaze features a continuous action space with sparse rewards.
Baselines We considered seven basclines. These baselines include three value-based methods: TD3+BC (Fujimoto & Gu, 202T), CQL (Kumar et al.] 2020). and IQL (Kostrikov al., 2021).
and four return-conditioned BC approaches: DT (Chen et al., (2021). ODT (Zheng et al. 2022),
RvS (Emmons et al., 2021), and DS4 (David et al., 2023). Further details about each baseline are
provided in Appendix B
Hyperparameters To ensure fair comparison between DC and ODC versus DT and ODT, we set the hyperparameters (related to model and training complexity) of DC and ODC to be either cquivalent to or less than those of DT and ODT. Details on DC's and ODT's hyperparameters are available in Appendix C and Appendix D respectively. Moreover, the impact of context length of DT and DC and be found in Appendix G.2 and G.3 and the examination of the impact of action information on performance is provided in Appendix
Offline Results Table [ shows the resulting performance of the algorithms including DC and ODC in offline settings the MuloCo and AntMaze domains. All the performance scores

---

Published as a conference paper at ICLR 2024
normalized, with score of 100 representing the score of expert policy, indicated by Fu et al. (222)). For DT/ODT and DC/ODC, the initial RTG value for the test period is a hyperparameter. We examine six target RIG values, each being multiple of the default target RIG in Chen et al. (202T) the MuloCo domain, these values reach up to 20 times the default target RTG, while in the AntMaze domain, they reach up to 100 times. We subsequently report the highest score achieved for each algorithm. A detailed discussion on this topic can be found in Section 5.3
Value-Based Method
Return-Conding Conditiondd BC
Dataset
TD3+BC
IQL
CQL
DT
ODT
RvS
DS4
DC
ODC
halfcheetah-m
48.3
47.4
44.0
42.6
43.1
41.6
42.5
43.0
43.6
hopper-m
59.3
63.8
58.5
68.4
78.3
60.2
54.2
92.5
93.6
walker2d-m
83.7
79.9
72.5
75.5
78.4
71.7
78.0
79.2
80.5
halfcheetah-m-r
44.6
44.1
45.5
37.0
41.5
38.0
15.2
41.3
42.4
hopper-m-r
60.9
92.1
95.0
85.6
91.9
73.5
49.6
94.2
94.1
walker2d-m-
81.8
73.7
77.2
71.2
81.0
60.6
69.0
76.6
81.4
halfeheetah-m--c
90.7
86.7
91.6
88.8
94.8
92.2
92.7
93.0
94.8
hopper-m-e
98.0
91.5
105.4
109.6
111.3
101.7
110.8
110.4
111.7
walker2d-m-e
110.1
109.6
108.8
109.3
108.7
106.0
105.7
109.6
108.9
locomotion mean
75.3
76.5
77.6
76.4
81.0
71.7
68.6
82,2
83.4
antmmaz--
78.6
87.1
74.0
69.4
73.5
64.4
63.4
85.0
74.4
antmaze-u-d 71.4
64.4
84.0
62.2
41.8
70.1
64.6
78.5
60.4
antmaze mean
75.0
75.8
79.0
65.8
57.7
67.3
64.0
81.8
67.4
Table 1: The offline results of DC and baselines in MuJoCo and Antamze domains. We report the expert- noreat-ormallzed returns, following Fu et al. (2020). averaged across 5 random seeds. The dataset names are abbreviated follows: 'medium" as 'm', 'medium-replay" as 'm-r', 'medium-expee -epuunteepet'' as 'm-e', 'umaze' as 'u', and 'umaze-diverse" as 'u-d'. The boldface numbers denote the maximum score or comparable one among the algorithms.
In T we observe the following: 1) Both DC and ODC consistently outperform or closely match the state-of-the- performance across all environments. In particular, DC and ODC show far superior performance in the hopper environment compared with other baselines. 3) Our model ex- cels not only MuJoCo locomotion tasks focused on return maximization but also in goal-reaching AntMaze tasks. Considering the sparse reward setting of Antmaze, the competitive performance in this domain highlights the effectiveness DC sparse settings. Through these observations, we can confirm that our approach effectively combines important information make optimal decisions specific to each situation, irrespective of whether the context involves high-quality demonstrations, sub-optimal demonstrations, dense rewards, sparse rewards.
Online Finetuning Results Table shows the online finetuning result obtained with 0.2 million online samples of ODC after offline pretraining. We compare against IQL (Kostrikov et al.) 2021) and ODT Zheng et al. 2022). Like the offline result, all scores are normalized in accordance with Fu al. 2020)). ODC yields top performance across effectiveness of DC in online finetuning. In consistency with the offline result, the performance of ODC stands in the hopper environment. In the case of hopper-met um, ODC achieves nearly maximum scores using sub-optimal trajectories for pretraining and using few samples during online finetuning. The difference between the offline and online performance denoted as 8. ODC shows less fluctuation than other models. This is partly because the offline performance itself is higher than others.
environments, further validating the
5.2 ATARI DOMAIN
In the Atari domain (Mii al. 2013), the setup differs from that of MuJoCo and AntMaze. Here, the action space discrete, and corresponding rewards are not immediately given after an action, and this makes the direct association of specific rewards with states and actions difficult. In addition, the Atari domain is more challenging due to its reliance on image inputs. By testing in this domain, can evaluate the algorithm's capability in credit assignment and managing discrete action space.

---

Published as a conference paper at ICLR 2024
Dataset
IQL (0.2M)
Oror.
ODT (0.2M)
ODC (0.2M)
dopc
halfchcetah-m
47.41
0.04
42.53
-0.62
42.82
-0.81
hopper-m
66.79
2.98
96.33
18.03
99.29
5.64
walker2d-m
80.33
0.44
75.56
-2.89
79.44
-1.09
halfcheetah-m--r
44.14
0.04
40.64
-0.86
41.42
-1.02
hopper-m-r
96.23
4.10
89.26
-2.65
95.23
walker2d-m-r
70.55
-3.12
77.71
-3.32
77.89
-3.14
locomotion mean
67.56
0.75
70.34
1.28
72.68
0.05
antmaze-u
89.5
2.4
86.43
12.88
86.70
12.24
antmaze-u-d
56.8
-7.6
60.26
18.37
61.12
0.68
antmaze mcan
73.2
-2.6
73.35
15.63
73.91
6,46
Table 2: Online finetuning results of DC and baselines after offline pretraining. All models are fine- tuned with 0.2 million online samples. We report the expert-normalized returns averaged across five random seeds. Dataset abbreviations are the same as those used in Table T
Hybrid Token Mixers As the Atari domain requires credit assignment across long horizons, in- corporating module that can capture global dependency in addition to our convolution module, can be advantageous. Therefore, in this domain, addition to experiments with the default DC employ- ing convolution block in every layer, conduct experiments using the hybrid DC mentioned in Section 3.2 composed of MetaFormer blocks with the first N convolution blocks and a final aftention block.
Baselines and Hyperparameters For the Atari domain, we compare DC with CQL (Kumar al. 2020). BC (Bain & Sammut) 1995), and DT (Chen et al., 2021) eight games: Breakout, Obert, Pong, Seaquest, Asterix, Frostbite, Assault and Gopher including the games used in Chen al. (2021) and Kumar et al. (2020). The hybrid DC uses the same hyperparameters used by DT, including the context length K 30 or K = 50. However, we set K 8 for the default DC due to its emphasis on local association. Details of the hyperparameters are provided in Appendix
Game
CQL.
BC
DT
DC
DChhondd
Breakout
211.1
142.7
242.4 3318
352.7 #44.7
416.0 1005.4
Qbert
104.2
20.3
28.8 110.3
67.0 114.7
62.6 19.4
Pong
111.9
76.9
105.6 4-2.9
106.5 +2.0
111.1 +1.7
Seaquest
1.7
2.2
2.7 10.7
2.6 #0.3
#0.04
Asterix
4.6
4.7
5.2 11.2
6.5 #1.0
6.3 =1.8
Frostbite
9.4
16.1
25.6 12.1
27.8 13.7
28.0 =1.
Assault
73.2
62.1
52.1
436.2
73.8 4:20.3
79.0 113.
Gopher
2.8
33.8
34.8 +10.0
52.5 +9.3
51.6 100.7
mcan
64.9
44.9
62.2
86.2
94.7
Table Offline performance results of DC and baselines the Atari domain. report the gamer- normalized returns, following Ye et al. (2021), averaged across three random seeds. We denote the hybrid setting as byyyrdid The boldface numbers denote the maximum score comparable one among the algorithms.
Results Table shows the performance results in the Atari domain. The performance scores are normalized according to Agarwal et al. (2020). such that 100 represents a professional gamer's 'ss pol- icy, and represents random policy. the Atari dataset, four successive frames are stacked to form single observation, capturing the motion over time. Although Although/Chen Chen et al. (202T) proposed that ex- tending the timesteps might be advantageous, our findings indicate that a simple aggregation of local information alone can exceed the performance achieved by the longer-timestep DT setup. Further- more, the hybrid configuration, which integrates an attention module its last layer to balance both local and global information, outperforms the baselines, and the gap is huge in Breakout. These results highlight the importance of effectively integrating local context before incorporating long- term information when making decisions environments that demand long-horizon reasoning.
8

---

Published as a conference paper at ICLR 2024
5.3 DISCUSSION
In this subsection, we examine the mechanisms that allow DC to excel in decision-making by con- sidering the aspects of understanding local associations and model complexity. Please refer to Ap- pendix G for more detailed discussion.
Input Modal Dependency Assessing how the convolution fil- ter gauged the importance of each modal (RTG, state, or action) is challenging because visualizing filters is not as straightforward as
zeroing out action zeroing out states except current
zeroling out RTG
visualizing attention scores. However, performance analysis by zeroing out each modal during inference can reveal their learned significance from training. For instance, if zeroing out RTG dur- ing testing severely impairs performance, indicates its critical role in decision decision-making making, Given the importance of the current state for predicting the next action, we keep it intact when zeroing out states. The results in MuJoCo hopper meedimeiiiii um shown in Fig. reveal that for DT, zeroing out each modal results in a minor per- formance decrease, and the impact is more less the same for each modal except the fact that zeroing out states has a slightly bigger impact. In contrast, for DC, zeroing out action has im- pact on performance, but zeroing out RTG or state causes huge drop over 40%. Indeed, DC found out that RTG and state infor- mation is more important than action, whereas DT seems not.
no zeroing
DT
DC
Average Narmalieed Aeturn
40
20
Figure 5:
Inference perfor-
mance with zeroed modals hopper-mest um.
Generalization Capability:
Out-Of-
Distribution RTG For any given task, there's an optimal model complexity; exceed- ing this point leads to overfitting and larger test or generalization errors (Goodfellow et al. 2016). Thus, one way to check that a model has proper complexity is investigate the generalization errors for samples unseen in the training dataset. For DT and DC, setting the initial target RTG to an out-of-distribution (OOD) value unseen in training effectively tests this. So, we performed experiments by continuously increasing the target RTG from the default value (used in Chen et al. (202T) on
DT
DC
Normalizes, Returr
hopoer-medium
antmaze umaze
100
60
40
107
10"
10a
2-10"
40
10-
10a
10
10a
Normalized Target RTG
Normalized Target RIG
Figure 6: Test performance with respect the target RTG in hopper um and antma ze ze-uzaze.
hepper-med um and antmaze -uttaazze-mezze. and the result is shown in Fig. 6 It is seen that DC has far better generalization capability than DT as the target RTG deviates from the training dataset distribution. This means that DC better understands the task context and better knows how to achieve the unseen desired higher target RTG by learning from the seen dataset than DT. The superior generalization capability of DC to DT implies that the model complexity of DC is closer to the optimal complexity than that of DT indeed.
6 CONCLUSION
In this paper, we have proposed a new decision maker named Decision ConvFormer (DC) for of- fline RL. DC is based on the architecture of MetaFormer and its token mixer is simply given by convolution filters. DC drastically reduces the number of parameters and computational complexity involved token mixing compared with the conventional attention module, but better captures the local associations in RL trajectories so that it makes MetaFormer-basee approaches to RL a viable and practical option. We have shown that DC has a model complexity relevant to MetaFormers as MDP action predictors and has superior generalization capability due to its proper model complex- ity. Numerical results show that DC yields outstanding performance across all the considered offline RL tasks including MuJoCo, AntMaze, and Atari domains. Our token mixer structure for MetaFormers intended for other aspects of MDP problems which were difficult due to attention's heavy complexity, opening up possibilities for more MetaFormer-haar based algorithms for MDP RI..
be used

---

Published as a conference paper at ICLR 2024
REFERENCES
Rishabh Agarwal, Dale Schuurmans, and Mohammad Norouzi. An Optimistic Perspective Of- fline Reinforcement Learning. International Conference on Machine Learning, pp. 104-114. PMLR, 2020.
Anurag Ajay, Yilun Du, Abhi Gupta, Joshua B Tenenbaum, Tommi S Jaakkola, and Pulkit Agrawal. Is Conditional Generative Modeling all you need for Decision Making? International Confer- ence on Learning Representations, 2022.
Michael Bain and Claude Sammut, A Framework for Behavioural Cloning. In Machine Intelligence 15, pp. 103-129, 1995.
Richard Bellman. A Markovian Decision Process. Journal of Mathematics and Mechanics, pD- 679-684, 1957.
Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-Shot Learners. Advances Neural Information Processing Systems, 33:1877-1900 2020.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision Transformer: Reinforcement Learning via Se- quence Modeling. Advances in Neural Information Processing Systems, 34:1 5084-55777, 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling Language Modeling with Pathways. arXiv preprint arXiv:2204.023:111 2022.
Shmuel Bar David, Itamar Zimerman. Eliya Nachmani, and Lior Wolf. Decision S4: Efficient Sequence- Based RL via State Spaces Layers, In International Conference Learning Represen- tations, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Confer- ence of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4171-4186, 2019.
Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. RvS: What is Essential for Offline RL via Supervised Learning? International Conference on Learning Representations, 2021.
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: Datasets for Deep Data-Driven Reinforcement Learning, 2020.
Scott Fujimoto and Shixiang Shane Gu. A Minimalist Approach Offline Reinforcement Learning. Advances in Neural Information Processing Systems, 14:20132-20144 2021.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep Learning. MIT press, 2016.
Albert Gu, Karan Goel, and Christopher Re. Efficiently Modeling Long Sequences with Structured State Spaces. In International Conference on Learning Representations, 2022.
Ali Hatamizadeh, Hongxu Yin, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Global Context Vi- sion Transformers. In International Conference on Machine Learning, pp. 12633-12646, PMLR, 2023.
Dan Hendrycks and Kevin Gimpel. Gaussian Error
Linear Units (GELUs). arXiv preprint
00X.:11606.04415.. 2016.
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline Reinforcement Learning with Implicit Q-Leaming, In International Conference on Learning Representations, 2021.
10

---

Published as a conference paper at ICLR 2024
David Krueger, Tegan Maharaj, Janos Kramar, Mohammad Pezeshki, Nicolas Ballas, Nan Rosemary Ke, Anirudh Goyal, Yoshua Bengio, Aaron Courville, and Christopher Pal. Zoneout: Regularizing RNNs by Randomly Preserving Hidden Activations. In International Conference Learning Representations, 2016.
Aviral Kumar, Xue Bin Peng, and Sergey Levine. Reward-Conditioned Policies. arXiv preprint arXiv: 19146 2019.
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative Q-Learning for Of- fline Reinforcement Learning. Advances in Neural Information Processing Systems, 33:1 179- 1191, 2020.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. arXiv preprint ar 19t.::1999.199000 2019.
Daniel Lawson and Ahmed H Qureshi. Merging Decision Transformers: Weight Averaging for Forming Multi-Taan Task Policies. In Workshop on Reincarnating Reinforcement Learning at ICLR 2023, 2023.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa: A Robustly Optimized BERT Pre- training Approach. arXiv preprint arXiv: /907. 1692, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin Transformer: Hierarchical Vision Transformer using Shifted Windows, Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Yicheng Luo, Jackie Kay, Edward Grefenstette, and Mare Peter Deisenroth. Finetuning from Of- fline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions. arXiv preprint arXiv:2303.17396 2023.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wier- stra, and Martin Riedmiller. Playing Atari with Deep Reinforcement Learning. arXiv preprint arXiv:1312.5610 2013.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Belle- mare, Alex Graves, Martin Riedmiller, Andreas Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforeement learning. Nature, 51 518(7540):529-35 533, 2015.
Vinod Nair and Geoffrey E Hinton. Rectified Linear Units Improve Restricted Boltzmann Machines. In International Conference on Machine Learning, pp. 807-814, 2010.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. OpenAl Blog, 188)9, 2019.
Sidney Resnick. Adventures in Stochastic Processes. Springer Science & Business Media, 1992.
Juergen Schmidhuber. Reinforcement Learning Upside Down: Don't Predict Rewards-Just Map Them to Actions. arXiv preprint arXiv: 19/22.022.025 2019.
Jinghuan Shang, Kumara Kahatapitiya, Xiang Li, and Michael S. Ryoo. StARformer: Transformer Representations for Visual Reinforcement Learning. Computer
with Prate-AAtiin-Reee sion ECCV 2022, pp. 462-479. Springer Nature Switzerland, 2022.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way to Prevent Neural Networks from Overfitting. The Journal of Machine Learning Research, 15(1):199--1988 2014.
Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai Thomas Un- terthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, al. MLP-Mixer: An all-MLP Architecture for Vision. Advances in Neural Information Processing Systems, 34: 24261-24272. 2021.
11

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention All You Need. Advances in Neural Information Processing Systems, 30, 2017.
Mengdi Xu, Yikang Shen, Shun Zhang, Yuchen Lu, Ding Zhao, Joshua Tenenbaum, and Chuang Gan. Prompting Decision Transformer for Few-Shot Policy Generalization. In International Conference on Machine Learning, pp. 24631-24645. PMLR, 2022.
Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, and Yang Gao. Mastering Atari Games with Limited Data. Advances in Neural Information Processing Systems, 34:25476-25488. 2021.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. MetaFormer Actually What You Need for Vision. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 10819-10829, 2022.
Haichao Zhang, Wei Xu, and Haonan Yu. Policy Expansion for Bridging Offline-to-Online Rein- forcement Learning. In International Conference on Learning Representations, 2022.
Qinqing Zheng, Amy Zhang, and Aditya Grover. Online Decision Transformer. In International Conference on Machine Learning, pp. 27042-27059, PMLR, 2022.
12

---

Published as a conference paper at ICLR 2024
Appendix
A DOMAIN AND DATASET DETAILS
A.1 MuJoCo
The MuJoCo domain is a domain within the D4RL (Fu et al. 2020 benchmarks, which fea- tures several continuous locomotion tasks with dense rewards. this domain, we conduct ex- periments in three environments: hal fcheetah, hopper, and walker2d. For each envi- romment, we examined three distinct v2 datasets, each reflecting a different data quality level: medi um, med um-replayy and medi i um um-expert. The medi um dataset comprises 1 million samples from a policy performing at approximately one-third of an expert policy's performance. The med i tmmreepla dataset uses the replay buffer of a policy trained to match the performance of the medium policy. Lastly, the medi um expert dataset consists of million samples from the medium policy and I million samples from expert policy. Therefore, the MuJoCo domain serves as an ideal platform to analyze the impact of diverse datasets from policies at various degrees of proficiency.
A.2 ANTMAZE
AntMaze in the D4RL (Fu et al. 2020) benchmarks consists of environments aimed at reaching goals with sparse rewards and includes maps characterized by diverse sizes and forms. This domain is suitable for assessing the agent's capability to efficiently integrate data and execute long-range planning. The objective of this domain is to guide an ant robot through maze to reach des- ignated goal. Successfully reaching the goal results in reward of I, whereas failing to reach it yields a reward of O. In this domain, we conduct experiments using two v2 datasets: umaze, uma -eee-da verse. In umaze, the ant is positioned at consistent starting point and has a specific goal to reach. On the other hand, umaze-di verse places the ant random starting point with the task of reaching randomly designated goal.
A.3 ATARI
The Atari domain is built upon a collection of classic video games (Mnih et al.) 2013). A notable challenge in this domain is the delay in rewards, which can obscure the direct correlation between specific actions and their outcomes. This characteristic makes the Atari domain an ideal testbed for assessing agent's s skill in long-term credit assignments. In our experiments, datasets provided by Agarwal et al. (2020), constituting 1% of all samples in the replay data gen- erated during the training of DQN agent (Mnih et al. 2015). We conduct experiments in eight games: Breakout, Qbert, Pong, Seaquest, Asterix, Frostbite, Assault, and Gopher.
utilized Atari
B BASELINE DETAILS
B. BASELINES FOR MUJOCO AND ANTMAZE
To evaluate DC's performance in the MuJoCo and AntMaze domains, compare DC with seven
baselines including three balue-based methods: TD3+BC (Fujimoto & Gul 2021). CQL Kumaar
al. 2020). and IQL (Kostrikov al. 2021) and four return ceturn-conditiona. BC methods: DT Chen
et al.
2021 ODT (Zheng et al.) 2022), RvS (Emmons et al.,
2021). and DS4 (David et al. 2023)
We obtain baseline performance scores for BC and RvS from Emmons et al. (2021), for TD3+BC
from Fujimoto & Gul (202I) and for CQL from Kostrikov et al. (2021). Note that cannot directly compare the CQL score from its original paper (Kumar et al. 2020) due to the discrepancies in
dataset versions. For IQL, the score reported in Zheng al. (022) used taking into consider- ation both offline results and online finetuning results. For DT,, ODT, and DS4, we reproduce the results using the code provided by the respective authors.
Specifically, for DT. we use the official implementation available at htttps:/// com/ kz) hell/decian 8 trasonnttttooooooommooo
While training DTT, mainly follow the hyperparameters rec-
13

---

Published as a conference paper at ICLR 2024
ommended by the authors. However, we adjust some hyperparameters as follows, this improves the results for DT:
Activation function: detailed Appendix E.. we replace the ReLUU (Nair & Hinton 2010) activation function with GELU (Hendrycks & Gimpel| 2016).
Embedding dimension: For hoopee-med um and hopper-medi um replay within the Mu- JoCo domain, we increase the embedding dimension from 128 to 256.
Learning rate: Across MuloCo and AntMaze environments, we select learning rate among {10- 10
that yields higher return (the default setting is to use 10 for all environments).
In addition, for ODT, we use the official implementations from https://gitan facebookressarcch wwwrrch/onninedttt... We mainly follow their hyperparameters but switch to the GELU activation function and adjust the learning rate from options of 10-4, 5 x 10-4, and 10-3 For DS4, we use the code provided by the authors as supplementary material available at //tp:s///eeerreieew net forum id=kgiikVVs and apply the hyperparameters as proposed by the authors.
com/
B.2 BASELINES FOR ATARI
In the Atari domain, we compare DC against CQL (Kumar et al. 2020), BC (Bain Sammut 1995). and DT (Chen et al. 202T For the performance score of CQL, follow the scores from Kumar et al. (2020) for games available. For other games such as Frostbite, Assault, and Gopher, conduct experiments using the author-provider code for CQL (https://g' / thub. com/aviral wwwua229000 /CLL) Regarding BC and DT, we conduct experiments using the DT's official implementation (https: github. com/ k com./e///deciiion-teeee/ www.deeiiionnttnnneeeeeee former). When training BC and DT, for the games not in Chen et al. (201)) (Asterix, Frostbite, Assault, and Go- pher), we set the context length K 30 and apply RTG conditioning as per Table Moreover, for all Atari games, the training epochs are increased from epochs to 10 epochs.
C IMPLEMENTATION DETAILS OF DC
We implement DC using the official DT code (https:// ctttt/uuuccomk/// soccisionntraaafoooee and incorporate the convolution module.
C.1 MUJUCO AND ANTMAZE
For our training on MuJoCo and AntMaze domains, the majority of the hyperparameters are adapted from Chen et al. (022). However, make modifications, especially concerning context length, the nonlinearity function, learning rate, and embedding dimension.
Hyperparameter
Value
Number of layers
3
Batch size
64
Context length K
8
Dropout
0.1
Nonlinearity function
GELU
Grad norm clip
0.25
Weight decay
10-4
Learning rate decay
Linear warmup
Total number of updates
10a
Table 4: Common hyperparameters of DC MuJoCo and AntMaze.
Context length: While Chen et al. (202I) suggests a context length of K = 20, we shortened this to 8 for DC, given DC's reliance nearby tokens. Note that the shortened context length is
14

---

Published as a conference paper at ICLR 2024
sufficient for achieving superior performance compared to DT. However, described in Appendix
G.2. extending DC"s context length to match that DT further improves the performance.
Embedding dimension: We use an embedding dimension of 256 in hopper gopper-medi. and hopper-med um seeediu-reppaas and 128 in the other environments. The impact of the embedding dimensions of DT and DC hopper-mee di um and pepperr-meium--eppppe in Table
can be seen
Learning rate: We use a learning rate of 10-4 for training in hopper- medrimeddiiii um, hopper-medium-regy we use 10-3 3
walker2d-mede um, and antMaze. For other environments,
DT
DC
Dataset
128
256
128
256
hopper-medium
63.1
68.4
69.7
92.5
pupper-mediumtiee
83.4
85.6
88.2
94.2
Table 5: The training results of DT and DC in MuJoCo hoppez-medium and heper--ned.u-repllaay with embedding dimensions of 128 and 256 respectively. We report the expert-normalized returns averaged across five random seeds.
C.2 ATARI
Hyperparameter
Value
Number of layers
6
Embedding dimension
128
Batch size
256
Return-to-go conditioning
90 Breakout (+s x max dataset)
2500 Qbert (#= x max dataset)
20 Pong (+s max in dataset)
1450 Seaquest (RS 5 max in dataset)
Game
Context length
520 Asterix (R: 5 x max in dataset)
Breakout
950 Frostbite (sz 5 x max in dataset)
Qbert
780 Assault (=s x max in dataset)
Pong
50
2750 Gopher (As x max dataset)
Seaquest
30
Nonlinearity
ReLU, encoder
Asterix
30
GELU, otherwise
Frostbite
30
Encoder channels
32, 64, 64 8 x 8, x 4, 3
Assault
30
Encoder filter sizes
Gopher
Encoder strides
4, 1
Table 7: The game-specific context length K used when training DChybrid and DT on Atari.
Max epochs
10
Dropout
0.1
Learning rate
x 10
Adam betas
(0.9, 0.95)
Grad norm clip
1.0
Weight decay
0.1
Learning rate decay
Linear warmup and cosine decay
Warmup tokens
512 20
Final tokens
500000 * K
Table 6: Common hyperparameters of DC on Atari.
Similarly to the MuJoCo and AntMaze domains, the DC hyperparameters for the Atari domain mostly follow those from Chen et al. (2021). The only adjustment made is to the context length K, which is decreased to 8, reflecting DC's focus on local information. In this domain, we also con- duct experiments in hybrid manner, combining the convolution module and the attention module.
15

---

Published as a conference paper at ICLR 2024
For the hybrid setup, use the same context length defined in Chen al. 2022 integration with the attention module. Table 6 presents the common hyperparameters used across all Atari games for both DC and DChyhrid, The context length of each game in the hybrid setting is represented in 7
the
D IMPLEMENTATION DETAILS OF ODC
Our ODC implementation builds upon the official ODT code, accessible at httttss:/// //tps::/gtt thub. com/ http://aeboooreeeeeee.a
on ine-dt, by replacing the attention module with convolution module. Table 8 outlines the hyperparameters used for the offline pretraining of ODC in the Mu- JoCo and AntMaze domains. While most of these hyperparameters align with those from Zheng et al. (022). we have modified the learning rate, weight decay, embedding dimension, and nonlin- earity. Regarding positional embedding, DC does not require explicit ones, the convolution with neighboring tokens sufficiently provides positional information. However, akin to the approach of /hheng et al. (022), which determines the use of positional embedding based on specific bench- marks, we make selective decisions regarding the use of positional embedding for each benchmark, as detailed in Table g
Hyperparameter
Value
Number of layers
4
Embedding dimension
256, hopper-mediuu-zzpl...
512, otherwise
Batch size
256
Context length
8
Dropout
0.1
Nonlinearity function
GELU
Learning rate
10-3, wal walker2d-redium
x 10 1 otherwise othhewewses
Grad norm clip
0.25
Weight decay
10-4
Linear warmup for first 10 training steps Learning rate decay
Target entropy
dim(AA)
Total number updates
10a
Table 8: Common hyperparameters of ODC on MuloCo and AntMaze.
Dataset
Positional embedding
(halfchectah, hopper, walker2d)-medee
(halscheetah, hopper, (aaler22))-mmduu--epplll.
yes
{ halfchectthh hopper, (idiuureddymmd
no
antmaze-/umaze umaze-diverse!
yes
Table 9; Usage of positional embedding for ODC by benchmark.
For online finetuning, retain most of the hyperparameters from Table However, specific benchmark-based hyperparameters are outlined in Table [O] Note that, ODC requires an additional target RTG, fontine for gathering additional online data (Zheng al. (022). In Table 10 Geaal de- notes the target RTG for evaluation rollouts, and fonline denotes the exploration RTG for gathering online samples.
16

---

Published as a conference paper at ICLR 2024
Dataset
pretraining updates
buffer size
embedding size
learning rale
weight
geval
Gonline
positional embedding
decay
hopper-m
5000
1000
512
0.0005
0.0001
3600
7200
no
hopper-m-r
5000
1000
512
0.0005
0.00005
3600
7200
no
walker2d-m
10000
1000
512
0.0005
0.0001
5000
10000
no
walker2d-m-
10000
1000
512
0.001
0.0001
5000
10000
no
halfcheetah-mn
5000
1000
0.0005
0.0001
6000
12000
n10
halfcheetah-m-rr
5000
1000
512
0.0001
0.0001
6000
12000
antmaze-u
7000
1500
512
0.001
0
yes
antmaze-u-d
7000
1500
1024
0.0001
0
yes
Table 10:
The hyperparameters employed for finetuning ODC for each benchmark. The dataset names are abbreviated as follows: 'medium" as 'm', 'medium-replay" as 'm-r', 'medium-expert as 'm-e', 'umaze' as 'u', and 'umaze-diverse" as 'u-d'.
E FURTHER DESIGN OPTIONS
E.I ACTIVATION FUNCTION
In the original DT implementation, a ReLU (Nair & Hinton 2010)
activation function used for the 2-layer feedforward network within each block. We conduct experiments by replacing this acti- vation function with the GELU (Hendrycks Gimpel, 2016) function. We observe that this change has no impact on the MuJoCo domain but improves the performance in the AntMaze domain for DT and DC (no improvement for ODT and ODC). GELU is derived by combining the character- istics of dropout (Srivastava et al.) 2014), zoneout (Krueger et al., (016). and the ReLU function, resulting in curve that similar but smoother than ReLU. As a result, GELU has the advantage of propagating gradients even for values less than zero. This advantage has been linked to performance improvements and is widely used in recent models such as BERT (Devlin et al., 2019), ROBERTa (Liu et al. 2019), ALBERT (Lan al., 2019), and MLP-Mixer (Tolstikhin al. (021). When ing the GELU activation, we observe noticeable performance enhancements in some environments with degradation in other environments. Consequently, we conduct experiments by replacing the
ReLU activation function with GELU in DT, DC, ODT, and ODC. The impact of GELU activation in the AntMaze domain is presented in Table
DT
DC
Dataset
ReLU
GELU
ReLU
GELU
antmaze-umaze
66.2
69.4
71.0
85.0
antmaze-umaze-divenee
58.0
62.2
63.6
78.5
Table 11: Expert-normalized returns for DC and
DT on antmaze-umaze and
notmaze-umaze-tee averaged over five random seeds, using ReLU and GeLU.
E.2 INCORPORATING ACTION INFORMATION
In specific environments such as hopper-medie -nmeiiaaa-pplay from MuJoCo, the inclusion of ac- tion information in the input sequence can hinder the learning process. This observation is supported by Ajay al. (2022). which suggests that action information might not always benefit the approach of treating reinforcement learning as a sequence-to-sequens when discussing the application of diffusion models to reinforcement learning, points out that sequence of actions tends to exhibit higher frequency and lack smoothness. Such characteristics can disrupt the predictive capabilities of diffusion models. This phenomenon might explain the challenges observed in hopper -eeeir--eeiiee wemiimmeeeaa wea.m-reel..... Addressing this challenge of high-fire figh-freyuency actions remains an area for future exploration. Comparative training results, with and without the action information in hoppe r-pder-medd um-repl ay, provided in Table 12
learning problem. The same source,
17

---

Published as a conference paper at ICLR 2024
DT
DC
ODT
ODC
Dataset
action
action
action
action
action
action
action
action
O
X
O
X
O
X
0
X
hopper-medium-relayy
82.2
85.6
88.7
94.2
88.9
91.9
90.1
94.1
Table 12: Expert-normalized returns averaged across five random seeds for DT, DC, ODT, and ODC on hopper medi um zeplay, both with and without action information.
E.3 PROJECTION LAYER AT THE END OF TOKEN-MIXER
In the Atari domain, we observe that the utilization of dimension preserving projection layer at the end of each attention or convolution module can affect performance. Therefore, for both DT and DC, we set the inclusion of the projection layer as hyperparameter. The inclusion of the projection layer for each game is listed in Table 13
Jimension-preserving
Projection layer
Game
DT
DC DChyood
Breakout
yes
Qbert
yes yes
Pong
no
no
Scaquest yes yes
yes
Asterix yes no
yes
Frostbite
no
yes
yes
Assault
yes
yes
yes
Gopher
Table 13: The game-specific usage of projection layer when training DT, DC, and DChybrrd on the Atari domain.
F COMPLEXITY COMPARISON
Table 14 15 and 16 present the computation time for one training epoch, GPU memory usage, and the number parameters. These metrics offer comparative analysis of the computational efficiency between DT vs. DC, ODT vs. ODC, all of which are trained on a single RTX 2060 GPU. In the table "#" symbol denotes "number of" and "A%" denotes the reduction ratio of the latter relative to the former, i.c. former-latter x 100. Examining the results, ODC are more efficient than DT and ODT in terms of training time, GPU memory usage, and the number of parameters.
observe that DC and
G ADDITIONAL ABLATION STUDIES
G.1 DISTINCT CONVOLUTION FILTERS
The convolution module in DC employs three separate filters: the RTG filter w state filter w and action filter wy. These designed distinguish variations across the semantics of RTG, state, and action. To assess the contribution of these specific filters, we perform experiments using unified single filter wy RL applicable all position p across various MuloCo and AntMaze environments. Analogous to Eq. 4 for the l-filter DC, the convolution output for the q-th dimension computed as:
Cllp.gl Σ
w X 1,g], p p= 1....... 3K 1.
Results in Table (ibdiictt that except in the walker2d-medium: -m scenario, using three filters enhances performance. Impressively, even when limited to single filter, DC substantially
18

---

Published as a conference paper at ICLR 2024
Complexity
DT
DC
A%
Complexity
ODT
ODC
4%
Training time (8)
426
396
7%
Training time (s)
2147
854
60%
GPU memory usage
0.7GiB
0.6GiB
14%
GPU memory usage
4GiB
L4GiB
65%
All params
1.IM
0.8M
%
params
13.4M
8.1M
40%
Token mixer params
198K
8K
95
Token mixer params
4202K
43K
99%
Table 14: The resource usage for training DT,
Table 15:
The resource usage for training
DC on MuJoCo and Antmaze.
ODT, and ODC on MuJoCo and Antmaze.
Complexity
DT
DC
A%
Training time (8)
764
193
75 %
GPU memory usage
3.7GB
1.8GB
51 %
All params
2.1M
1.7M
19%
Token mixer params
396K
16K
%
Table 16: The resource usage for training DT, DC on Atari.
surpasses DT. This implies that by only capturing local patterns with single filter, there's notable enhancement in decision -maiin..
Dataset
1-filter DC
3-filter DC
DT
hopper-medium
88.3
92.5
68.4
walker2d-medium
77.6
79.2
75.5
hopper-medium-repla.
89.6
94.1
85.6
walker2d walker2d-medium-eee
78,0
76.6
71.2
antmaze-une umzze
81.6
85.0
69.4
antmzze-umazz-ddivess
78.5
62.2
Table Comparison between expert-normalized returns of 1-filter DC, 3-filter DC, and DT, aver- aged across five random seeds.
G.2 CONTEXT LENGTH AND FILTER SIZE OF DC
DC focuses on local information and, by default, employs window size of 6 to reference previous timesteps within the (RTG, a) triple token setup. While enlarging the window size enables decisions that account for a broader horizon, it could inherently reduce the impact of local information. To assess the effect, we conduct extra experiments to validate performance across various filter window sizes L and context lengths
3
6
30
8
83.5
92.5
20
90.1
94.2
93.5
Table 18: Expert- noremazedd returns averaged across five random seeds in hopper -madameeaa di um for different combinations of and
G.3 CONTEXT LENGTH DT
Chen al. 1202 highlights that longer sequences often yield better results than merely consid- ering the previous timestep, particularly in the Atari domain. Consistent with this, we train DT with emphasis on local information, akin how DC is trained on the MuJoCo medium and medium medimrrrpla datasets. For evaluation, we set DT's context length K to parallel DC's configuration and also assess DT with K 2 to prioritize the current timestep and its predecessor.
19

---

Published as a conference paper at ICLR 2024
Examining the outcomes in Table 19 it's evident that in the hopper and walker2d enviromments the performance of DT gradually decreases with reduced context lengths. However, unlike these environments, there's ' almost performance drop in the halfcheet deeper, we conduct experiments by entirely excluding the attention module, and the averaged expert- normalized score is 39.5. In the ha Fcheetah environment, it's apparent that the attention module doesn't It hold a significant role, hence its impact doesn't I seem contingent on context length.
environment. To delve
Dataset
DT (20)
DT (8)
DT (2)
DC (8)
hopper medium & medium-replay
77.0
74.8
72.5
93.4
walker2d medium & medium-repla replay
73.4
72.5
71.6
77.9
halfcheetah medium & medium-replay
39.8
39.3
39.4
42.2
Table 19: Performance of DT across context lengths K: 20, 8, and 2. Expert-normalizer returns are
averaged over the medi um and medi am-replayy MuJoCo benchmarks, and across five random seeds.
H LIMITATIONS AND FUTURE DIRECTION
Although DC offers efficient learning and remarkable performance, has its limitations. Since DC replaces the attention module, it is not immediately adaptable to scenarios demanding long-horizon reasoning, such as meta-learning (Xu al., 2022) or tasks with partial observability. Our proposed hybrid approach might be solution for these scenarios. Exploring further extensions to propagate the high-quality local features DC over long horizons is meaningful direction for future research. Furthermore, crunn-condiiioned BC algorithms, including DC, have not yet achieved the results of the value-based approach in the ha fcheetah environment.
20


Reviews:
[{"review_id": "WdNLEOXkB8", "reviewer": "Reviewer_Eec1", "summary": "The paper revisits the efficacy of the transformer, originally devised for natural language processing, for reinforcement learning (RL). The authors' empirical studies demonstrate that the previous designs of the transformer for RL (e.g., decision transformer) could be an inefficient overparameterization mainly due to the lack of exploiting Markov property, which is a common assumption in RL. As a part of utilizing Markov property, the authors propose a new transformer model (which is a variant of MetaFormer), called Decision Convformer (DC). They empirically show the efficacy of DC in various environments, in particular, when Markov property holds.", "questions": "Can you provide evaluations of the hybrid DC for the benchmark in Table 1 (environments with Markov property)? This would help choose architectures when the prior knowledge of the degree of Markov property is limited. If the hybrid DC is comparable to or better than DC and computational cost is not important, then one may simply consider the hybrid DC for such cases.\n\nCan you report the computational complexity of the hybrid DC as you did for DT ad DC in Table 14,15,16?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The authors have demonstrated a potential risk and inefficiency of using the transformer with a long context length K when Markov property is strong.\n\nAs Markov property can be interpreted as a locality (or local dependence) in the sequence of interactions between the agent and the environment, the authors employ convolution filtering for token mixer in MetaFormer. The convolution filtering helps to reduce the number of model parameters (in particular, the number of token mixer parameters) and provides performance gain in offline RL settings (in particular, in hopper and antmaze datasets).\n\nIn the case of weak Markov property, the authors also propose DC^{hybrid}, which uses both the convolution filtering and the attention model. The hybrid DC showed superiority in Atari datasets, compared to DT.\n\nThe proposed DC and DC^{hybrid} might provide new promising options for model architectures in deep RL.", "weaknesses": "My major concern is \bthe seemingly incomplete justification of the proposed architectures. In my understanding, just DT with a small K (i.e., short context length) could be sufficient and show comparable to or even better than DC. Additional comparisions (in terms of performance and computational complexity) on DC and DT with different choices of K would be helpful. Otherwise, it is unclear whether the gain of DC (or hybrid DC) is mainly from the good combinations of hyperparameters (including the embedding dimension, GELU, K, ...), or indeed the convolution filtering. \n\nIn addition, the advantage of the proposed method (DC) is particularly remarkable in hopper and antmaze datasets. In fact, the gap between DC and DT is not significant in other environments. It seems necessary to clarify the environment-specific gain of DC over DT.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "npHwMoqJBW", "reviewer": "Reviewer_dRz2", "summary": "this paper propose Decison ConvFormer (DC) as an alternative of Decision Transformer (DT). The insight is that most RL task require locality and the particular parameterization of DT seems to not be optimal in learning it. In contrast, they propose to use a depth-wise conv block. The experiment results on both Mojuco and atari shows that it's better in both offline and online finetuning. The discussion section shows that the model generalizes better in RTG and dependes more on state in a meaningful way.", "questions": "1. can you further describe details of the motivating examples? Do you only learn the attention directly of that one layer or all layers?\n2. For the hybrid architecture, what happens if you do attention first then conv?\n3. Can you also test the OOD generalization on novel task with multi-task learning?\n4. Can you visualize the attention of that hybrid model in some atari games?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The paper provides a very good insight about the problem in modelling RL sequence, which is emphasis on local association. By introducing a convolution blocks, it is a very good idea built on insights to the specific problem, and I really like the motivating example in Fig3. \n\nThe method is simple, and I think the community is easy to verify it after few lines of code changes.\n\nThe experiment results are strong, and cover both discrete and continuous domain. The hybrid architecture is a good balance between locality and long-term credit assignment.\n\nThe discussion section is good to see and the generaliation of RTG is an interesting result.", "weaknesses": "There seems not much I can say. But I think to improve, the author could remove the mention of the MetaFormer framework. As someone who has never heard it before, I first though metaformer is a new transformer variant, but then I realized it's just a framework, which is a bit confusing to me.\n\nAlso the the name \"token mixer block\" should be avoided, since it reminds of the token mixing in the MLP-Mixer, which makes me confuse in the beginning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mYt5aC6y2r", "reviewer": "Reviewer_TLiJ", "summary": "The author proposed a new structure, called Decision Convformer, by replacing the token mixing step in MetaFormer with three causal convolution filter for RL tasks. The proposed Decision Convformer achieved better performance on well-known tasks with less training time.", "questions": "See the above section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Improvements with less training computation are achieved. Thus, the proposed DC is efficient.\n\n2. The presentation is easy to follow. The motivation is also described clearly.\n\n3. Extensive experimental results are provided.", "weaknesses": "1. How to compute the embeddings of a float number (reward) in the subsection 3.1? Some explanations might be helpful.\n\n2. The reasons why the propose method is effective are needed to explained. It seems that the self-attention operation is more expressive then the proposed block (three causal convolution filters). Is the proposed DC only suitable for some settings, e.g. the setting with less data?\n\n3. \bWhy the ODC is worse than DC on some tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)