Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL
Authors: Qi Lv
Xiang Deng
Gongwei Chen
Michael Y Wang
Liqiang Nie
Venue: NeurIPS 2024 poster (2024)
Keywords: Decision Transformer
Mamba
Offline Reinforcement Learning
Autoregressive Sequence Modeling
State Space Model
Primary area: reinforcement_learning
TL;DR: We propose Decision Mamba tailored for the offline RL tasks, a novel multi-grained state space model with a self-evolving policy learning strategy.

Abstract: While the conditional sequence modeling with the transformer architecture has demonstrated its effectiveness in dealing with offline reinforcement learning (RL) tasks, it is struggle to handle out-of-distribution states and actions.
Existing work attempts to address this issue by data augmentation with the learned policy or adding extra constraints with the value-based RL algorithm. However, these studies still fail to overcome the following challenges: (1) insufficiently utilizing the historical temporal information among inter-steps, (2) overlooking the local intra-step relationships among return-to-gos (RTGs), states, and actions, (3) overfitting suboptimal trajectories with noisy labels. To address these challenges, we propose $\textbf{D}$ecision $\textbf{M}$amba ($\textbf{DM}$), a novel multi-grained state space model (SSM) with a self-evolving policy learning strategy.
DM explicitly models the historical hidden state to extract the temporal information by using the mamba architecture. To capture the relationship among RTG-state-action triplets, a fine-grained SSM module is designed and integrated into the original coarse-grained SSM in mamba, resulting in a novel mamba architecture tailored for offline RL. Finally, to mitigate the overfitting issue on noisy trajectories, a self-evolving policy is proposed by using progressive regularization. The policy evolves by using its own past knowledge to refine the suboptimal actions, thus enhancing its robustness on noisy demonstrations. Extensive experiments on various tasks show that DM outperforms other baselines substantially.

Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RL
Gongwei Chen' Michael Yu Wang? Liqiang Nie School of Computer Science and Technology, Harbin Institute of Technology (Shenzhen) Schooo of Engineering, Great Bay University lvqiêstu. hitt edu.cnn
Qi Lv
Xiang Deng'
Abstract
While the conditional sequence modeling with the transformer architecture has demonstrated its effectiveness in dealing with offline reinforcement learning (RL) tasks, it is struggle to handle attempts to address this issue by data augmentation with the learned policy or adding extra constraints with the value-based RL algorithm. However, these stud- ies still fail to overcome the following challenges: (1) insufficiently utilizing the historical temporal information among inter-steps, (2) overlooking the local intra- step relationships among return-to-ger (RTGs), states and actions, (3) overfitting suboptimal trajectories with noisy labels. To address these challenges, we propose Decision Mamba (DM), novel multi-grained state space model (SSM) with self- evolving policy learning strategy. DM explicitly models the historical hidden state to extract the temporal information by using the mamba architecture. To capture the relationship among RTG-state-actte designed and integrated into the original coarse- coarse-grained SSM in mamba, resulting in novel mamba architecture tailored for offline RL. Finally, to mitigate the overfitting issue on noisy trajectories, self-evolving policy is proposed by using progressive regularization. The policy evolves by using its own past knowledge to refine the suboptimal actions, thus enhancing its robustness tions. Extensive experiments on various tasks show that DM outperforms other baselines substantially.
states and actions. Existing work
fine-grained triplets, a fine- grained SSM module is
noisy demonstra-
1 Introduction
Offline Reinforcement Learning (RL) 13
27
29
38 has attracted great attention due to its remark-
able successes in the fields of robotic control [
36 and games [ 32 [50] As transformer [49 has 43 and computer
exhibited powerful sequential modeling abilities in natural language processing
25 61 have been made on applying this architecture to off RI. tasks. Transformer-based methods view the reward/return-to-ge (RTG), state, and action sequence, and then predict actions by using the transformer encoder. However, it often fails make correct decisions when encountering nut-of-distributin states or actions, showing limited robustness. Previous work attempts to address this issue from the perspective of data augmentation [51 644 and objective constraints 16, 533 6l]. However, they introduce significant number of noises or the overestimation bias. Thus, how to enhance model robustness remains a highly challenging and insufficiently explored issue.
vision 10) 42 many efforts
In this study, we offer two novel perspectives on improving model robustness through both the model architecture and learning strategy. In terms of the model architecture, (1) although previous
Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

studies have made some modifications to the transformer architecture 233 44 52 they have not fully utilized inter-step information. particularly historical information which is critical for decision- decision-making processes. For example, the robot can adjust its subsequent routes based on the historical information of failed paths for completing the navigation task; (2) furthermore, most existing approaches adopt transformer to model the flattened trajectory as a sequence, while ignoring the structural trajectory patterns of the causal intra-step relationship among RTGs, states, and actions (RSAs). A RL policy typically predicts the next action given the current state based on the RTG. Thus, this kind of fine- grained intrinsic connection among RSAs is intuitively beneficial for policy learning. As regards to the learning strategy, (3) there exists a large number of noisy labels in the suboptimal trajectories which hurt the performance of the policy significantly. Although the existing work that generates pseudo trajectories actions alleviates this problem to some extent [57 64] it also introduces other biases or errors.
To address the above issues, we propose Decision Mamba (DM), a multi-grained state space model with a self-evolving policy learning strategy for offline RL. In order to adequately leverage the historical information, we adopt mamba to explicitly model the temporal state among inter-steps, since mamba architecture [15] 18 40) shows a more effective capability of extracting the historical information. Meanwhile, the causal intra-step relationship is beneficial for the model to understand the common patterns within the local dynamics. Thus, we introduce a fine-grained SSM module to extract the local features of structural patterns among the RSA triplet within each intra-step. Apart from modifying the model architecture and aligning it to the trajectory pattern, we also propose a learning strategy prevent the policy from overfitting noisy labels. This is achieved by progressive self-evolution regularization which leverages the past knowledge of the policy itself to refine and adjust the target label adaptively.
We conduct comprehensive experiments on Gym-Mujoco and Antmaze benchmark, containing tasks with varying levels of noise and difficulties. The performance of DM surpasses other baselines by approximately 8% with respect to the average normalized score on the three classic Mujoco tasks, showing its effectiveness. In summary, the contributions of this paper are summarized as follows:
Different from the existing conditional sequence modeling work for offline RL with the transformer architecture, we propose Decision Mamba (DM), a generic offline RL backbone built on State Space Models, which leverages the historical temporal information sufficiently for robust decision making.
To extract the casual intra-step relationships, we introduce a fine-grained SSM module and integrate it to the original coarse goarse-grained SSM in mamba, which combines the local trajectory patterns with the global sequential features, achieving the multi-grained modeling capability.
To prevent the policy from overfitting the noise trajectories, we adopt self-evolving policy learning strategy to progressively refine the target, which uses the past knowledge of the learned policy itself as additional regularizer constrain the training objective.
2
Related Work
2.1 Offline Reinforcement Learning with Transformer-based Models
Offline Reinforcement Learning (RL) 7 1 22
27
29
38
56
57
67 is widely used for robotic
control and decision-making In particular, transformer- trajectories as a state/action/RTGG sequence, and predict the next action based on the historical trajectories. However, although the sequence modeling methods formulate offline RL in a simplified form, they can hardly deal with the overfitting problem caused by the suboptimial trajectories in offline data [TT 21 59). One line of approaches 34 5T 64 66] focused on exploiting data augmentation methods, such as generating additional data via the bootstrap method, or training an inverse dynamics model to predict actions for the large amount of unlabelled trajectories. Another line
methods 8 25
reformulate the
of work 23, 22
35
44 52 attempted to modify the transformer architecture to explicitly make
use of the structural patterns within the training data. Furthermore, substantial efforts [3] 54 58 62
have also been made on applying regularization terms to learning policies, such as RvS [1] and QDT 61. Nevertheless, previous work simply applies transformer to offline RI. tasks while seldom considering about adapting the architecture trajectory learning. Thus, these methods fail to extract
2

---

the historical information sufficiently and are unable to capture local patterns thoroughly from the trajectories. In this work, we address these issues by proposing DM, tailored mamba architecture for offline RI tasks. A fine-grained SSM module is designed in DM to supply fine-grained intra-step information to the coarse- coarse-grained inter-steps features. Together with the architecture, we also present a self-evolving policy learning strategy to prevent the model from overfitting noise labels.
2.2 State Space Models for Linear-time Sequence Modeling
Recently, State Space Models (SSMs) show high potentials in various domains, including natural
language processing [15 19 20 40 46), computer vision [30 31 B3
41
63 65 and time-scries
forecasting [55]. Stemming from signal processing, SSMs capture global dependencies from sequence more effective lightweight structure and shows advantages compressing the historical information, compared with the transformer architecture. Although SSMs have considerable benefits, still struggles perform contextual reasoning. Mamba [I5] is thus proposed to alleviate this problem. It introduced a time-varying selective mechanism and hardware- -rienddyy design, making as competitive architecture against with transformer. The Mamba architecture then adapted to different downstream tasks by considering the characteristics of these tasks. VIM 665 and VMamba (33 introduced 2D SSMs for image understanding. VideoMamba (30) introduced spatio- temporal scan for video understanding. In this work, we take the fine-grained trajectory patterns into consideration, and introduce multi multi-grained mamba architecture tailored for RL tasks.
3 Method
3.1 Preliminaries
Decision Transformer for Offline RL. The fundamental Markov Decision Process [12 can be represented as M = A,,, 7), where S is the state space, AA is the action space, SxA S the transition function, r ( A R is the reward function, and y (0, 1] is the discount factor. Given an offline dataset D2 collected by the behavior policy su(als), offline RL algorithms aim to maximize the rewards. Formally, the iteration process of learning a policy as below (k denotes the index of the learning iteration):
Q argmin E (e.. a) - (r YEu'N 1610) QK (s', a')))
(1)
T argmax E. an
(s, s.t. EEED, (D(((-ss), ((1ss) se.
(2)
When updating the Q function, (s, a, 8') are sampled from D₁ but the target action a' is sampled from the current policy -
Inspired by the great success of sequence generation models in NLP 9 39 48 Decision Trans- former (8) is proposed to model the trajectory optimization problem as an action prediction procedure. Specifically, it first obtains the return-to-go (RTG) with the reward, i.e., R₁ Σ ri. Then, the
learned policy, which is based the decoder-only transformer architecture 48 predicts the action sequence a; autoregressively, with the offline trajectory т (so, Ro, ....... training objective is as follows:
........ ST, Rr, ar). The
T
minimize [(TT] = E,
Σ 1=1
- log T (at (x-
where Ti-la (s,, Ry aj, ....... ... St. Rτ) (j min(t 0,) is the input trajectory and is the length of context window.
SSMs for Linear-Time Sequence Modeling. The State Space Model (SSM) describes the proba- bilistic dependence between the continuous input signal srtt) and the observed output y(t) via the latent hidden state h(t)) as Eq. (5):
R((I)= = Ah(l) + Bx((),
(4)
y(t) = Ch()).
(5)
3

---

Corrid
BBM
Crarse-Gramed Scan
Decision Mamba Encodor
FG EBM
Norm
Convid
80
8
80
Adtlvtionn
Decission Mamba
9.
A
Group
Decision Mamba Encoder
Fine-Grained Sean
Figure 1: Model Overview. The left: combine the trajectories with position embeddings, and then feed the result sequence to the Decision Mamba encoder which has L layers. The middle: a coarse-grained branch and fine-grained branch are integrated together to capture the trajectory features. The right: visualization of multi nulti-grained scans.
In order to apply the SSM model to the discrete input sequence instead of the original continuous signal, Structured SSM (S4) [8] discretizes it by step size A Eq. (7).
h Ah4-1 + B
(6)
yr = Che.
(7)
I)- AB. To this end, the model can forward- propagate in an efficient parallelizable mode with a global convolution. Due the linear time invariance brought by the SSM model, it lacks the content-aware reasoning ability which is important in sequence modeling. Therefore, mamba [[5] proposes the selective SSM which adds the length dimension to the original parameters (A, B, C), changing it from time-invariant to time-varying, It uses parameterized projection to project size parameter B, C from (D, N) to (B, L, N), and A from (D) to (B, L. D), where D, B, L and N denotes the channel size, batch size, sequence length and hidden size, respectively.
where A = exp(AA), B= (AAA)
3.2 Decision Mamba
The transformer architecture has been well used in offline RL tasks. Despite its strong ability to understand complete trajectory sequence, it shows limited capabilities in capturing historical informa- tion. Thus, we propose a multi-grained space state model to extract the fine-grained local information supply the coarse-grained global information, namely Decision Mamba (DM), for comprehensively learning the trajectory representation. Figure T presents the overall framework of DM.
3.2.1 Multi-Grained Mamba
Trajectory Embeddings Following the sequence modeling 8 we first use multilayer percep- trons (MLPs) to embed the RSAs from the given trajectory = (Rg. Sg. Rr, $T, or). Then, the trajectory embeddings added the absolute step position embeddings to attach the position information, similar to the classic usage in the NLP field. Mathematically, can be formulated as follows:
ef MLP(R.),
MLP((s). & :LP(()))
(8)
= 1c C + broadcast(e).,
(9)
where of B and denotes the concatenate operation.
Coarse-Grained SSM Different from the transformer-based methods K 25
DM models the
historical information before the current i-th step via the latent hidden state h; as shown in Eq. (6). explicitly represents the feature of history information, rather than only learns such information implicitly. As the number of encoder layers increases, historical information is selectively preserved in the representation h; To this end, DM expected to have a better capability to understand the sequential dependencies. It can be formulated as follows:
A,C = InterS3M(A
(10)
where hnG represents the coarsc-grain coarsc-grained hidden state; InterS3M denotes the coarse-grained SSM.

---

Fine-Grained SSM Further, to better discern the dependencies among RSA within each step, we gather the feature of each single step to ob- tain the fine-grained representation via a ID- convolution layer, and then introduce fine- grained SSM module for extracting the local pat- tern among RSA, shown in the middle part and right part of Figure !
Algorithm 1 Decision Mamba
Input: trajectory sequence T (Ro: 80, .o..... R(:81)
Output: action
1: / obtain the embedding trajectory sequence"/
2: a, a: (B, Split(T:-t:r)
3: e" """""," ee: (B, MLP(R), MLP(s) MLP(a)
4: ho: LB, <- FFatten(("" ,."""")
5:
layer
6: h [B, L. - Norm(h.
It can be formulated as follows:
7: he (B, L) + Convld, 1 (h.
hg Conv1D(h:),
(11)
8:
200, (B,
L,, (B,
+ Linear CG (h:
9:
=P (B, D) - Linear ((h-1)
h SILU(Pro((NE"))
(12)
10:
1" process with multi-grained branchs //
11:
for h/ in (h h,5) do
he IntraS3M(45
(13)
12:
b/ (B,L,)
(SanvlCa (h/
where h.t indicates the fine tme-grained hidden state;
13:
: 4) + Parameter
14:
B/ [B, X) + Linear f1 (k/)
IntraS3M means the fine- rreeggriindd SSM.
15:
c/ (B,L,) Linear, /c (h()
16:
A (., <- log(1 oxp (h/)
Fusion Module For gathering both fine- -raiggai local trajectory patterns and coarse- coarse-grained global contextual information, we combine the h with
Parameter (A))
17:
X T (B, discretiio A/. B/)
18:
b/: (B. SSM B (()())
h,CG in each encoder layer and then use the layer normalization to ensure that the multi-grained fea-
19:
end for
20:
h (B,L. 08 @ SiLU(z)
tures have a consistent distribution. In order to remain the important historical information, we
21:
6 (B,L, hec SILU(z)
22:
fusion multi-grained features 7/
23:
1 : L. <- LayerNorm(h b.
add residual connection. The fusion process can
24: h. (B, + Liuear(h MG + hh-1)
be formulated follows:
25: end for
1 MG LN CG + +1150 FG )
(14)
26: a (B, MLP(h)
27: return d,
h;- = Proj(h MG + hh-),,
(15)
where hMG indicates the multi-graing hidden state and LN denotes the layer normalization. The forward propagation procedure of DM is presented in Algorithm
3.2.2 Progressive Self-Evolution Regularization
There are typically amounts of suboptimal trajecto- ries in RL tasks. The previous approaches usually overfit these noisy data and thus lack robustness. Fortunately, the existing literature 26 has shown that deep models learn clean samples (optimal tra- jectories) at the beginning and then overfit the noisy samples (suboptimal tra- jectories). Inspired by this observation, we propose progressive self-evolution regularization (PSER), which uses the knowledge of the past policy to refine the noisy labels supervision for policy learning, thus avoiding fitting the noisy trajecto- ries.
step
@ (1 96) + 94
the training process,
ak
k step
LySBA +
Figure 2: The process of PSER includes: gen- erating action labels with previous step policy, ii) refining target label, iii) computing loss, where the red circle denotes the noise.
Specifically, we obtain a refined target by combin- ing the ground truth and the prediction from the learned policy itself. Let âs denote the prediction about s from the current policy Ta (a|s) at k-th iteration. The refined target at k-th can written as follows:
ã = (1 B) au + 3 3
(16)
where à=-1 ( and 3 is the trade-off weight.
To obtain more insights about the refined targets Eq. (16), we compare the gradients of the training objectives with the original label and the refined label. The standard Mean Square Error (MSE) loss function of Eq. (3) with the original label can be written as:
(ā ax)
l|ā: ag 12..
(17)
5

---

In comparison, the loss function with the refined target of Eq. (16) can be rewritten as:
(sg, 4x) = 1 - axl|² ||ā - (1 - 8) dh 8 0x-1ll?.
(18)
Comparing the objectíves of Eq. (77 and Eq. (88. the gradient of LSF, with respect to the output of policy {a can be derived by:
dLSE. [ (k B (āk-1.; - ak.i ax,)),
(19)
vCx
VR
where VLg indicates the gradient of the loss function Eq. T7 and VR computes the difference between the past predictions and the targets. From the perspective of gradient back propagation, PSER imposes a regularization constraint on the current policy T [ by smoothing the original target action dj.a with the leff--eeneated label 0
Moreover, it is important to determine the value of 3 in Eq. 18 The & controls the learning procedure, where the policy trusts the given actions if 3 is set to a large value. As stated above, the policy tends to gradually from clean patterns to noisy patterns. Thus, we set the to dynamically increased values. As 3 increases, the policy progressively gains more confidence in its own past knowledge. To maintain the learning process stable, we apply the linear growth approach and set a lower boundary. The B at the k-th iteration is computed as follows:
k
B max(3x x K
Pmiu):
(20)
where K is the number of total iterations for training and 8K is the hyperparameter. We replace the original label with the refined target, leading the objective:
minimize E,, [lo (e( RR,, St, T<ce].
(21)
We adopt the MSE loss, and then the objective is converted to:
& (ax: 0x) = ||āg 2 = ||āg - (1 - 3x) au - Bx 2x-111?
(22)
3.2.3 Training Objective
To make the training procedure more robust, we introduce the inverse training goals: predicting the next state and the next RTG. Individuals often assess the feasibility of actions by envisioning their potential outcomes. Therefore, we expect the policy to predict the post-execution state and RTG based on the predicted action, thus improving its robustness. Specifically, given the trajectory = [Ro, So, Qp. R2, st), Decision Mamba originally predicts the next action âz Further, by incorporating the action a, to the original trajectory Tt-La, it is also predicts the next RTG Ri+ and the next state $1+1- Compared to the Eq. (3), the training objective of DM with the refined target can be written follows:
minimize En
Σ Au log no Re, St, St, Tete) +h2 log To (Rey1 (Tga) +hs log No (8(g ]. (23)
PSER
prodicting
prodicting states
where the ä, is computed by Eq. 16). X is the weight hyperparameter, and the sum of X is set to 1. Note, omit the length of context window for simplicity.
4 Experiment
4.1 Settings
Dataset and Evaluation Metrics. We conduct our experiments on Gym-MuJoCo which is one of mainstream benchmarks used in offline deep RL [4 28, (5), including Hopper, HalfCheetah, Walker and Ant tasks. Each task contains medium, medium-expert, medium-replay and expert datasets. To more comprehensively evaluate our proposed method, we also adopt the AntMaze benchmark which is navigation task of aiming reach a fixed goal location, with the 8-DoF "Ant" quadraped robot. We evaluate Decision Mamba by using the popular suite D4RL [33. Following the existing literature 88 233, normalize the score for each dataset roughly for comparison, by
computing normalized score = 100 expert score random
score random score
More details about dataset and
implementation can be found in Appendix

---

Baselines. We compare Decision Mamba with existing SOTA offline RL approaches including
Behavioral Cloning (BC), Conservative Q-Learning (CQL) 22 Decision Transformer (DT) (8)
Reinforcement Learning via Supervised Learning (RvS) I StARformer (StAR) A4], Graph
Decision Transformer (GDT) [23]. Waypoint Transformer (WT) 2 Elastic Decision Transformer
(EDT) [60], and Language Models for Motion Control (LaMo) 45 Among these methods, CQL
stands as a representative of value-based methods, while the other methods belong to supervised learning (SL) approaches. For most of these baselines, we cite the results from the original papers. In addition, we reimplement DT and LaMo for more comparison in different settings by using their repositories. The detailed descriptions of these baselines are presented in Appendix B
Dataset
BC
CQL
DT
RySt
SIAR!
GDT
WT
EDT
LaMo
DM (Ours)
HallChectah-M
42.2
44.4
42.6
41.6
42.9
42.9
43.0
42.5
43.1
43.8 20:23
Hopper-M
55.6
86.6
70.4
60.2
65.8
77.1
63.1
63.5
74.1
98.5 19.19
Walker-M
71.9
74.5
74.0
73.9
77.8
76.5
74.8
72.8
73.3
80.3 10.07
HallCheetthhMM-E
41.8
62.4
87.3
92.2
93.7
93.2
93.2
48.5
92.2
93.5 :0.11
Hopper-M-E
86.4
110.0
106.5
101.7
110.9
111.1
110.9
110.4
109.9
111.9
#1.84
Walker-M- M-E
80.2
98.7
109.2
106.0
109.3
107.7
109.6
108.4
108.8
111.6
#3.31
HalfCheetah-Me
2.2
46.2
37.4
38.0
39.9
40.5
39.7
37.8
39.5
40,8 20043
Hopper-M-R
23.0
48.6
82.7
82.2
81.6
85,3
88.9
89.0
82.5
89.1 #4.32
Walker-M-R
47.0
32.6
66.6
66.2
74,8
77.5
67.9
74.8
76.7
79.3 #1.04
Avg.
50.0
67.1
75.8
71.7
77.4
79.1
78.7
72.0
77.8
83.2 20082
Table I: Overall Performance. M, M-F, and M-R denotes the medium, medium-expert and medium- replay, respectively. The results of the baselines marked with i are cited from their original papers. We report the mean and standard deviation of the normalized score with four random seeds. Bold and underline indicate the highest score and second-hige hiigd- score, respectively.
4.2 Overall Results
For fair comparison, we first conduct experiments on datasets commonly adopted by mainstream approaches. The overall performance is presented in Table It can be observed that Decision Mamba outperforms other baselines in most datasets. On one hand, benefiting from the supervised learning objective, SL-based baselines exhibit strong ability in the high-quality datasets (M-E), but show weakness in the suboptimal datasets (M/M-R). On the other hand, CQL performs well in the suboptimal datasets due to regularizing the Q-values during training, but struggles to perform well in the high-quality datasets.
For DM, it shows significant improvement over the other SL-based methods. Specifically, it outperforms the best of the baselines by 4%+, especially in suboptimal datasets, e.g., on the medium datasets, the performance of DM sur- passes the value-based method CQL and the transformer- based method GDT by around 6% and 9% on average, respectively. This signifi- cant improvement demonstrates the robustness of DM in learning from suboptimal datasets, at- tributed to the multi-grained mamba encoder and PSER module in DM. Note, although DM performs slightly worse than CQL on the Halfcheetah-Me M-R and HalfChectah-M datasets, the difference is not significant. Therefore, the proposed DM shows stronger overall performance, capable of learning from both high-quality and suboptimal datasets simultaneously.
Dataset
BC
DT
LaMo
DM (Ours)
HalfCheetah-E
83.3
90.5
92.0
93.5 20:25
Hopper-E Walker-F
90.2
109.6
111.6
112.5 #0.75
103.2
108.1 I
108.1
108.3 10.13
Ant-M
91.0
95.3
94.6
104.8
Ant-M-E
99.8
129.6
134.8
136.2
Ant-M-R
79.5
81.4
92.7
89.5 11.64
Ant-E
112.6
123.1
134.2
135,9 #0.35
Antmaze-U
63.0
63.0
80.0
100.0 20
Antmazo-U-D
61.0
61.0
70.0
90.0 10.30
Avg.
87.1
95.7
102.0
107.9 13.33
Table 2: Extensive Results. E, and U-D denotes the expert, umazed, and umazed-diverse.
In order to evaluate our method more comprehensively. we also conduct experiments on other datasets. We adopt representative baselines including BC, DT and LaMo, where LaMo leverages extensive additional natural language corpora and knowledge to enhance the model performance. As illustrated in Table 2 all methods perform exceptionally well on the Expert dataset. However, when comes to mixing the suboptimal data into the training set, compared with DT, both LaMo and DM exhibit
7

---

sigmificant superiority, while DM shows a more pronounced overall enhancement, For instance, DM outperforms LaMo by approximately 10% on the Ant-M dataset and around 6% on average. For AntMaze, it requires composing parts of suboptimal trajectories to form more optimal policies for reaching goals. "U-D" is more difficult than "U", and DM shows superiority in these tasks. More comparison results can be found in Appendix C
4.3 Ablation Study
To investigate the effectiveness of each component in DM, we conduct experiments with different variants of DM. In particular, we compare 3 different implementations: (1) w/o MG removes the multi-grained branch, directly using the sequence feature original extracted from mamba; (2) w/o PSER removes the progressive self-evolution regularization, training the model with the labels in the training dataset; (3) w/o ILO removes the inverse learning objective, only predicting the action in the training procedure. As shown in Table 3 the performance of DM drops significantly without either of these components. Notably, the most substantial performance degradation with about 6% occurs when the PSER module removed, especially in the suboptimal datasets. This observation verifies the effectiveness of this module in preventing policy from overfitting and thus enhancing its robustness. MG and II.O are also critical for offline RI. tasks. Once these two modules are excluded, there is noticeable reduction in the model's performance.
Halfcheetah M-E
Hopper M-E
Walker M-E
Avg.
M-R
M
M-R
M-R
DM
43.8
93.5
40.8
98.5
111.9
89.1
80.3
111.6
79.3
83.2
w/o MG
43.3
92.9
40.1
86.2
111.2
77.5
79.2
107.9
74.9
79.2
w/o PSER
42.9
91.0
37.5
85.3
110.4
76.4
76.2
105.6
69.6
77.2
w/o ILO
43.1
92.3
39.4
94.0
110.7
85.3
80.1
108.8
73.5
80.8
Table 3: Ablation Results. "w/o MG//SER/ILO" represents removing the module of multi-grained feature extraction, the progressive self-evolution regularization, and inverse learning objectives, respectively. Best results marked in bold.
4.4 Comparison Results with Different Context Lengths
To validate whether DM can capture the information of inter-step and intra-step, we investigate the performance of our model with different context lengths. We conduct experiments with the context length L = {20, 40, 60, 80, 100, 1203. Figure shows the comparison results. Regardless of different context lengths, the proposed DM consistently achieves high score than other baselines among all datasets, showcasing its superiority in capturing the inter-step dependencies in different lengths.
It is noteworthy that BC shows comparable performance to DT in the Hopper-M and Halfcheetah-M datasets, yet demonstrates a substantial discrepancy in other datasets. We deduce that, in addition to the inherent limitations of the imitation learning paradigm, the BC model that relies solely on MI.P also has significant architectural disadvantages. Consequently, it can achieve scores of only 60-70% at most on the expert datasets. When trained on M-R data, BC evidently struggles to learn effectively, achieving only approximately 30% and less than 4% performance on Hopper-M-A and Halfcheetah-M-1 M-RR respectively. Due to the attention and SSM mechanisms, DT and DM models conspicuously exhibit higher upper bound compared to BC. Among them, our proposed DM shows the best performance across all datasets. This indicates that the specific architecture of DM enables it to extract more useful information from the inter-step and intra-step, leading to a strong performance across different context lengths.
4.5 The Effects of in PSER
We have shown that the proposed PSER in DM enhances the robustness of the policy significantly in learning on suboptimal trajectories. Consequently, we endeavor to delve deeper into the impact of the policy self-evolution throughout the training process. During the training procedure, 8x in PSER determines the upper bound of the policy self-evolution. When 3R set to the policy has the highest dependency on self-learned knowledge; conversely, if 3x is set to O, the policy tends to completely lose its ability to self-evolve. We conduct experiments on DM variants, by removing the lower boundary Smin and selecting 3x from the set {1, 0.75, 0.5, 0.25, 0}. The results are depicted in
8

---

Hopper-M
Hopper-M-E
Hopper-M-R
110
93
Normalzzed Score
102
00-
73
70
93
BC
BC
62
DT
DT
DN (Ours)
-2
49
DT
(Curs)
60
122
100
DO
120
length
Context length
Context length
Halfcheetah-M
Halfcheetah-Me
Halfcheetah-Ma-
44.0
43.5
43.0
Normatteed
42.5
ON (Ouns)
DN (Ours)
(Curs)
40.0
60
120
23
60
1.0
40
50
120
Context length
Context length
Context length
Figure 3: Impact of Context Lengths. We compare the normalized scores of BC, DT and DM with different context lengths. The DM consistently outperforms other baselines.
Table can be observed that we remove the self-evolution capability of DM, i.e., setting 3K to there notable decline in the performance across both the M and M-R correlated datasets, with the poorest overall performance. When increasing the 3 to 0.5, the model obtains the best performance since it reaches good balance between the ground truth and the learned knowledge. To prevent the policy from excessively relying on its past knowledge, DM additionally uses a lower boundary Smin (set to 0.5), achieving the best performance.
Halfchectah M-E
Hopper M-E
Walker M-E
M
M-R
M
M-R
M
M-R
Avg.
DM (5K = =1)
43.5
92.3
38.4
97.9
111.3
82.7
77.8
109.4
74.7
80.9
DM (8xx = -0.75)
42.8
91.9
38.7
98.4
110.5
83.8
77.6
106.2
71.3
80.3
DM (8x =0.5)
43,9
92.1
38.8
98.6
111.1
86.6
77.2
108.6
75.8
81.4
DM (8x =0.25)
43.8
91.5
38.6
97.7
107.0
86.4
76.9
106.2
72.8
80.2
DM (3x =0)
42.9
91.0
37.5
85.3
110.4
76.4
76.2
105.6
69.6
77.2
DM
43.8
93.5
40.8
98.5
111.9
89.1
80.3
111.6
79.3
83.2
Table 4: The effects of ( in PSER.
5
Conclusion
In this paper, we study the offline reinforcement learning from the perspectives of the architecture and the learning strategy. We have accordingly proposed Decision Mamba (DM), a multi multi-grained state space model tailored for RL tasks with a self-evolving policy learning strategy. DM enhances the policy robustness by adapting the mamba architecture to RL tasks by capturing the fine-grained and coarse-grained information. Meanwhile, the proposed learning strategy prevents the policy from overfitting the noisy labels with progressive self-evolution regularization. Extensive experiments demonstrate that DM outperforms other basclines by approximately 4% on the mainstream offline RL benchmarks, showing its robustness and effectiveness.
Limitations and Future Directions. According to 15 18
the mamba structure is more friendly to long sequences than the transformer structure, not only terms of capturing historical information. but also in terms of the computational speed. Benefiting from the structure of SSM, the computational complexity of mamba is On)) while the computational complexity of attention score transformer is Onn²). Thus, the computational efficiency of mamba is higher. Although the exploration of computational efficiency an exciting direction for future research, not within the main scope of this paper.

---

Acknowledgments and Disclosure of Funding
We would like to thank the reviewers for their constructive comments. This work is supported by and National Natu- ral Science Foundation China (Grant No. 62236003). Additionally. it is also supported by National Natural Science Foundation of China (Grant No. 62406092), and partially supported by Research on Efficient Exploration and Self-Evolution of APP Agents Embodied Intelligent Cerebellum Control Model and Collaborative Feedback Training Project (Grant No. TC20240403047).
Shenzhen College Stability Support Plan (Grant No. WXWDDX00202
References
[1] Anurag Ajay, Yilun Du, Abhi Gupta, Joshua Tenenbaum, Tommi Jaakkola, and Pulkit Agrawal. Is conditional generative modeling all you need for decision-making? arXiv preprint arXiv:2211.1567 2022.
[2] Anirudhan Badrinath, Yannis Flet-Berliae, Allen Nie, and Emma Brunskill. Waypoint transformer: Rein- forcement learming via supervised learning with intermediate targets. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, vol- ume 36, pages 78006-78027. Curran Associates, Inc., 2023.
[3] Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemystaw Dębiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, et al. Dota 2 with large scale deep reinforcement learning arXiv preprint arXiv: 191 2.06680 2019.
[4] Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot Iearners. Advances in neural information processing systems, 33:1877-199 1901, 2020.
[5] Yevgen Chebotar, Karol Hausman, Yao Lu, Ted Xiao, Dmitry Kalashnikov. Jake Varley, Alex Irpan, Ben- jamin Eysenbach, Ryan Julian, Chelsea Finn, et al. Actionable models: Unsupervised offline reinforcement learning of robotic skills. arXiv preprint arXiv:2104.077493 2021.
[6] Yevgen Chebotar, Quan Vuong, Karol Hausman, Fei Xia, Yao Lu, Alex Irpan. Aviral Kumar, Tianhe Yu, Alexander Herzog, Karl Pertsch, al. Q-transformer: Scalable offline reinforcement learning via autoregressive g-functions. Conference on Robot Learning, pages 3909-3928. PMLR, 2023.
[7] Huayu Chen, Cheng Lu, Chengyang Ying, Hang Su, and Jun Zhu, Offline reinforcement learning via high-fidelity generative behavior modeling. arXiv preprint arXiv:2209. 14548, 2022.
[8] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. I Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15084-15097. Curran Associates, Inc., 2021.
[9] Jacob Devlin. Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. 2019.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint arXiv:2010.11929 2020.
[11] Scott Emmons, Benjamin Eysenbach, Ilya Kostrikov, and Sergey Levine. Rys: What is essential for offline
rl via supervised learning? arXiv preprint 11X 10751, 2021.
[12] A. Feinherg. Markov decision processes: Discrete stochastic dynamic programming (martin puterman). SIAM Rev, 38(4):689, 1996.
[13] Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4rl: Datasets for deep data-driven reinforcement learning, 2020.
[14] Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without explo- ration. In International conference on machine learning, pages 2052-2062. PMLR, 2019.
[15] Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces, 2023,
10

---

[16] Albert Gu, Karan Goel, Ankit Gupta, and Christopher Ré. On the parameterization and initialization of diagonal state space models. In S. Koyejo. S. Mohamed, A. Agarwal. D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 35971-35983, Curran Associates, Inc., 2022.
[17] Albert Gu, Karan Goel, and Christopher Ré, Efficiently modeling long sequences with structured state spaces. arXiv preprint 11.00996,, 2021.
[18] Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces, 2022.
[19] Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous time models with linear state space layers, Advances in neural information processing systems, 34:572-585, 2021.
[20] Ankit Gupta, Albert Gu, and Jonathan Berant. Diagonal state spaces are as effective as structured state spaces. Advances in Neural Information Processing Systems, 35:22982-22994, 2022.
[21] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1352- 1361. PMLR, 06-11 Aug 2017.
[22] Jonathan Ho and Stefano Ermon. Generative adversarial imitation learning. Advances in neural information processing systems, 29, 2016.
[23] Shengehao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Graph decision transformer. arXiv preprint Stt1:2203.03774,, 2023.
[24] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesss. arXiv preprint 2Xiv:2205.09991, 2022.
[25] Michael Janner, Qiyang Li, and Sergey Levine. Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
[26] Kyungyul Kim, ByeongMoon Ji, Doyoung Yoon, and Sangheum Hwang. Self-Knowledge distillation with progressive refinement of targets. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 6567-6576, October 2021.
[27] Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offfine reinforcement learning with implicit q-Icaming. In International Conference on Learning Representations, 2022.
[28] Aviral Kumar, Justin Fu, Matthew Soh, George Tucker, and Sergey Levine. Stabilizing off- aff-policy g-lcarning via bootstrapping error reduction. Advances in neural information processing systems, 32, 2019.
[29] Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning- In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1179-1191. Curran Associates,
Inc., 2020.
[30] Kunchang Li, Xinhao Li, Yi Wang. Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. arXiv preprint arXiv:2403.06997, 2024.
[31] Xiaojie Li, Yibo Yang, Jianlong Wu, Bernard Ghanem, Liqiang Nie, and Min Zhang. Mamba-fseil: Dynamic adaptation with selective state space model for few-shot class-incremental learning. arXiv preprint Ttl::2407.061366 2024.
[32] Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, and Liqiang Nie. Optimus-1: 1: Hybrid multimodal memory empowered agents excel in long- morizori tasks. In NeurlPS, 2024.
[33] Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model, 2024.
[34] Zuxin Liu, Zijian Guo, Yihang Yao, Zhepeng Cen, Wenhao Yu, Tingnan Zhang, and Ding Zhao. Constrained decision transformer for offline safe reinforcement learning. In International Conference on Machine Learning, pages 2161 1-600 PMLR, 2023.
[35] Yi Ma. Chenjun Xiao, Hebin Liang. and HAO Jianye. Rethinking decision transformer via hierarchical reinforcement learning. 2023.

---

[36] Ajay Mandlekar, Fabio Ramos, Byron Boots, Silvio Savarese, Li Fei-Fei. Animesh Garg, and Dieter Fox. Iris: Implicit reinforcement without interaction scale for learning control from offline robot manipulation data, 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 4414-4420.
IEEE, 2020.
[37] Keiran Paster, Sheila Mcllraith, and Jimmy Ba. You can't count on luck: Why decision transformers and rvs fail in stochastic environments. Advances in neural information processing systems, 35:38966-3897, 2022.
[38] Xue Bin Peng, Aviral Kumar, Grace Zhang, and Sergey Levine. Mevrantage-weighuee regression: Simple and scalable off off-policy reinforcement learning. arXiv preprint arXiv:1910.0011 2019.
[39] Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations, 2018.
[40] Maciej Pióro, Kamil Cichiera, Krystian Król, Jan Ludziejewski, Michat Krutul, Jakub Krajewski, Szymon Antoniak, Piotr Milos, Marek Cygan, and Sebastian Jaszezur. Moe-mamba: Efficient selective state space models with mixture of experts, 2024.
[41] Yanyuan Qiao, Zheng Yu. Longteng Guo. Sihan Chen, Zijia Zhao. Mingzhen Sun, Qi Wu, and Jing Liu. VI-mamha: Exploring state space models for multimodal learning. arXiv preprint arXiv:2403. 13600, 2024.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[43] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9 2019.
[44] Jinghuan Shang, Kumara Kahatapitiya, Xiang Li, and Michael S. Ryoo. Starformer: Transformer with state-action-rewarr representations for visual reinforcement learning. In Computer Vision - ECCV 2022,
pages 462 479.444.6666 Springer Nature Switzerland, 2022.
[45] Ruizhe Shi, Yuyao Liu, Yanjie Ze. Simon Shaolei Du, and Huazhe Xu. Unleashing the power of treitrained language models for offline reinforcement learning. In The Twelfth International Conference on Learning Representations, 2024.
[46] Jimmy TH Smith, Andrew Warrington, and Scott W Linderman. Simplified state space layers for sequence modeling. arXiv preprint arXiv:2208. 04933, 2022.
[47] Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026-5033, 2012.
[48] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polasukhin. Attention is all you need. In I. Guyon, U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems. volume 30. Curran Associates, Inc., 2017.
[49] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, Advances in neural information processing systems, 30, 2017.
[50] Oriol Vinyals, Igor Babuschkin, Wojeicch M Czameckki Michaël Mathicu, Andrew Dudzik, Junyoung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in starcraft using multi-agent reinforcement learning. Nature, 575(7782):350-35,, 2019.
[51] Kerong Wang. Hanye Zhao, Xufang Luo, Kan Ren, Weinan Zhang, and Dongsheng Li. Bootstrapped transformer offline reinforcement learning. Advances in Neural Information Processing Systems. 35:34748-344 34761, 2022.
[52] Yiqi Wang, Mengdi Xu, Laixi Shi, and Yuejie Chi. A trajectory is worth three sentences: multimodal transformer for offline reinforcement learning. Robin J. Evans and Ilya Shpitser, editors, Proceedings of the Thirty-Nimth Conference on Uncertainty in Artificial Intelligence, volume 216 of Proceedings of Machine Learning Research, pages 2226-2236. PMLR, 31 Jul-04 Aug 2023.
[53] Yuanfu Wang, Chao Yang. Ying Wen, Yu Liu, and Yu Qiao. Critic-guie suttic-guiedddde decision transformer for offline reinforeement learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 15706-15714 2024.
12

---

[54] Yuanfu Wang, Chao Yang, Ying Wen, Yu Liu, and Yu Qiao. Critic-guided decision transformer for offline reinforcement learning. In Michael J. Wooldridge. Jennifer G. Dy, and Sriraam Natarajan, ed- itors,
AAAI Conference on Artificial Intelligence. AAAI 2024. Thirty-iixth Conference on Innovative Applications of Artificial Intelligence, IAAI 2024, Fourteenth Symposium on Educational Advances in Artificial Intelligence, EAAI 2014, February 20-27, 2024, Vancouver, Canada, pages 15706- 15714. AAAI Press, 2024.
[55] Zihan Wang, Fanheng Kong, Shi Feng, Ming Wang, Han Zhao, Daling Wang, and Yifei Zhang. Is mamba effective for time series forecasting?, 2024.
[56] Ziyu Wang. Alexander Novikov, Konrad Zolna, Josh S Merel, Jost Tobias Springenberg, Scott E Reed, Bobak Shahriari, Noah Siegel. Caglar Gulcehre. Nicolas Heess, et al. Critic regularized regression. Advances in Neural Information Processing Systems, 33:7768-7778, 2020.
[57] Hua Wei, Deheng Yc, Zhao Liu, Hao Wu, Bo Yuan, Qiang Fu, Wei Yang. and Zhenhui Li. Boosting offiine reinforeement learning with residual generative modeling. arXiv preprint 1tt::::106.1444 2021.
[58] Hua Wei, Deheng Ye, Zhao Liu, Hao Wu, Bo Yuan, Qiang Fu, Wei Yang, and Zhenhui Li. Boosting offline reinforcement learning with residual generative modeling. In Zhi-Hua Zhou, editor, Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence, LICA 2021, Virtual Event / Montreal, Canada, 19-27 August 2021, pages 3574-3580. ijcai.org, 2021.
[59] Yifan Wu, George Tucker, and Ofir Nachum. Behavior regularized offline reinforcement learning. arXiv preprint arXiv: 1. 1361, 2019.
[60] Yuch-Hua Wu, Xiaolong Wang, and Masashi Hamaya. Elastic decision transformer. In Thirty-scventh Conference on Neural Information Processing Systems, 2023.
[61] Taku Yamagata, Ahmed Khalil, and Raúl Santos Santos-Rodriuus Rodríguez. Q-lcaming decision transformer: leveraging dynamic programming for conditional sequence modelling in offline rl. In Proceedings of the 40th International Conference on Machine Learning, ICML'23, JML..org, 2023.
[62] Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Dichotomy of control: Separating what you can control from what you cannot. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023, OpenReview.net, 2023.
[63] Han Zhao, Min Zhang, Wei Zhao, Pengxiang Ding, Siteng Huang, and Donglin Wang. Cobra: Extending mamba multi-modal large language model for efficient inference, 2024.
[64] Qinqing Zheng, Mikael Henaff, Brandon Amos, and Aditya Grover. Semi- supervisd offline reinforcement learning with action-free trajectories. In International conference on machine learning, pages 42339-42362. PMLR, 2023.
[65] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model, 2024.
[66] Tianchen Zhu, Yue Qiu, Haoyi Zhou, and Jianxin Li. Towards long-delayed sparsity: Learning better transformer through reward redistribution. Edith Elkind, editor, Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, IJCA1-23, pages 4693-4701. International Joint Conferences on Artificial Intelligence Organization, 2023. Main Track.
[67] Zhenghang Zhu, Hanye Zhao, Haoran He, Yichao Zhong, Shenyu Zhang, Yong Yu, and Weinan Zhang. Diffusion models for reinforcement learning: A survey, arXiv preprint arXiv:231 1.01223, 2023.
13

---

A Dataset and Implementation Details
A.1 Dataset Details.
We conduct experiments five tasks of Mujoco and Antmaae 47 including Halfcheetah, Hopper, Walker, Ant and Antmaze, as illustrated in Figure Note, all datasets we used is the v2 version. In these tasks, there are totally 5 different datasets which are described below:
Medium: A "medium" policy is trained by using the Soft Actor- 21 with early- stopping the training. and generate 1 million timesteps, achieving about one-third the score of an expert policy.
Mepium-Expert:
million timesteps generated by the medium policy concatenated with million timesteps generated by an expert policy (a fine-tuned RL policy).
Medium Replly:: It involves recording all samples the replay buffer observed during training until the policy achieves "medium" level of performance.
Umaze: It contains the trajectories where the ant to reach specific goal from a fixed start location.
Umaze-diverse: Different from Umaze, it is a more difficult dataset where the start position is also random.
(a) Halfcheetah
(b) Hopper
(c) Walker
(d) Ant
(c) Antmaze
Figure 4: The visualizations of tasks.
A.2 Implement Details
For all our experiments, we utilized the default hyperparameter settings and conducted 100,000 training iterations gradient steps. We implement our method with the official repository of the huggingface. The shared typerparameters are set to the same as those of LaMo, including the batch size, learning rate, overall training steps, and weight decay. The setting of other specific hyperparameters, including 9K. Amin are presented in Table 5 The experiments are conducted on an 8#4090-24G platform, and we run each experiment with four different seeds ensure its reliability.
Dataset
Learning Rate
Weight Decay
Context Length
Return-to-go
Training Steps
8g
Aiii
Halfcheetah
1
x
10-4
10-5
20
1800, 3600
100K
0.85
0.5
Hopper
1
x
10-4
10-5
20
8000, 12000
100K
0.90
0.5
Walker
1 10-4
x 10
20
2500, 5000
100K
0.95
0.5
Ant
10-4
10-5
20
3600, 6000
100K
0.85
0.5
Antmaze
10-4
x 10
20
5,20
100K
0.5
Table 5: Task-specific Hyperparameters.
A.3
Code base
The code bases employed for our evaluations are detailed below.
BC: http:///iiiuboooo///l//ccoooottt
DT: https//ggihbbbcoo/////cccooooooo
EDT: https//gghthhhcoom///sreeyssssst
LaMo: http://ggthbbbcom/ssrrr////22222
14

---

B Details of Baselines
We compare our proposed Decision Mamba with previous strong baselines as follows:
Behavioral Cloning (BC): a representative method of imitation learning. The states and actions collected as the training data first. Then the agent uses classifier regressor to replicate the trajectory when encountering the same state.
Conservative Q- Q-rnninn (CQL.) 29 it encourages policies that are less likely to choose actions with high Q-value estimates that are uncertain or unreliable, thus expecting to address overestimation bias.
Decision Transformer (DT) it flats the trajectory sequence and use conditional sequence modeling method to autoregressively predict actions.
RvS [II] it uses the goal reward the condition to realize the behavior cloning. We use the reward-conditioned BC as comparison.
StARTransformer (StAR) 44 it extracts the image state patches by self-attending mecha- nism, then combining the features with the whole sequence.
it adopts the sequence modeling method, and models the input sequence into a causal graph to capture relationships among states, actions, and return-to-gos
Graph Decision Transformer (GDT) 23
Trasoinnrr (WT) iit integrates intermediate targets and proxy rewards as guidance to steer policy to desirable outcomes.
Elastic Decision Transformer (EDT) 60): it estimates the highest achievable value given certain history, and inputs the traversed trajectory with a variable length to learn the stitching trajectories.
Language Models for Motion Control (LaMo) 45
it adopts the pretrained GPT2 43
model as the backbone, and use the additional NLP corpus to co-training the policy via the parameter-efficienty LoRA method.
C More Comparison
For extensive comparison, we compare DM with baselines, including the diffusion-based model: Diffuser (4], Decision Diffuser (DD) a more complex approaches: Trajectory Transformer
(TT) 25 Critic-Guided Decision Transformer (CGDT) 53
As illustrated Table 6 can be observed DM still has the strongest overall performance, although it did not achieve the best results on some datasets. Diffusion-based models synthesize optimal trajec- tories from a generative perspective, showing a significant advantage on replay datasets. Although CGDT performs well, it requires complex additional training, namely Critic training, which increase convergence difficulty. Overall, DM shows superiority on average.
Dataset
TT
CGDT!
Diffuser!
DD
DM (Ours)
HalfCheetah-M
46.9
43.0
44.2
49.1
43.8 20.23
Hopper-M
61.1
96.9
58.5
79.3
98.5 +8.19
Walker-M
79.0
79.1
79.7
82.5
80.3 10.07
HallCheetah-M--E
95.0
93.6
79.8
90.6
93.5
Hopper-M-E
110.0
107.6
107.2
111.8
111.9 11.84
Walker-M-E
101.9
109.3
108.4
108.8
111.6 #33.51
HalfCheetah-Me -MM
41.9
40.4
42.2
39.3
40.8
Hopper-M-R
91.5
93.4
96.8
100.0
89.1 11.32
Walker-M-R
82.6
78.1
61.2
75.0
79.3 +1.94
Avg.
78.9
82.4
75.3
81.8
83.2
Table 6: More comparison with other baselines. The results are all cited from their original papers.
15

---

D The Result on the Distribution of Returns
We compare the ability of policy understand retur-ttogo tokens by varying the desired target return over a wide range, especially in the out-of-distriuttion range. As illustrated in Figure we can observed in the seen target, i.e., on the left side of the yellow dashed line, the expected target returns and the true observed returns are highly correlated. However, when it comes to the out-of-distribttion target, the score of DM is consistently higher than those of DT. Among them, due to the extreme difficulty of the HalfCheetah dataset, the performance of DM under the OOD target is only slightly surpassing DT. Conversely, on the other two datasets, DM exhibits strong robustness to the OOD target, significantly outperforming DT. The experimental result has illustrated DM has strong robustness.
HalfCheetah
Hopper
Walker2D
Normalized Score
100
50
25
50
75
100
125
75
100
125
25
100
125
Target Return (Normalized)
Target Return (Normalized)
Target Return (Norma Izzmallzzed
DM
DT
Oracle
Best Trajectory in Dataset
Figure 5: The normalized scores of DT and DM when conditioned on the specified target returns.
E
Visualization of Action Distribution
We visualize the action distribution of learned policy. Specifically, we use the policies trained on different level of noisy data predict the next action of the same trajectory, and visualize the hidden layer of the predicted action. As shown in Figure o the distribution obtained by DM is more concentrated. This indicates that even the noise level the training data varies, DM can still learn an approximate distribution, demonstrating its strong robustness.
DT
DM
DT
DM
(a) Halfcheetah
(b) Hopper
Figure 6:
The distributions of action.
F
Impact/Safegard Statements
Impact Statement In this study, we propose DM, which effectively extracts historical information, fuses multi-grained information to predict action, and enhances the effectiveness of conditional sequence modeling for offline RI. tasks. addition, we introduce a self-evolving policy learning strategy to effectively prevent the policy from overfitting the noisy trajectories, and further enhance the robustness of the policy. To this end, this technology is expected to advance the offline RL agent which can assist human beings in working under the dangerous cireumstances. There are many potential societal consequences of developing advanced RL algorithms, none which we feel must be specifically highlighted here.
Safegrad Statement
the paper, we have taken rigorous steps to ensure the responsible release of new offline RL algorithm and any associated models data. Given the potential for misuse
16

---

dual-use of such technology, we have implemented several safeguards to mitigate these risks. Use of our algorithms is subject to the CC BY-NC-SA 4.0 agreement. In addition, we are committed to ongoing monitoring and evaluation of our models' usage to identify any emerging risks or patterns of misuse. We will take prompt action if we detect any unauthorized or inappropriate use of our technology. Finally, we are open to working with regulators, researchers, and industry partners to further refine our safeguards and ensure the safe and ethical use our offline RL algorithm.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions are summarized clearly and accurately in the introduction. More details please refer to Section |
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
We discuss the limitations of our work in the conclusion. More details please refer to Section
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
18

---

Answer: [NA]
This work does not involve hypotheses and proofs of theory. In addition, all formulas are labeled and cross-referenced normally.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All information about reproducing the main experimental results are presented in the Section 4.1 and Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
19

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We have released our code in https://github. ctm.tthoo.com/mmooll.o.... lv/
DecisionMambe
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
public/guides/ Saapleese/CoeeSSSmmmsisioooeee
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee///ddeeebmmmiia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details about all experimental settings presented in the Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We conduct experiments with four different random seeds and report the mean and variance of each result.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
20

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details about compute resources in Appendix A.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We strictly adhere to the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have stated the socictal impact the Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification:
We have stated the safeguard in the Appendix
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We cite the original paper and provide URLs of the code we used.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "g7WhDKTNFZ", "reviewer": "Reviewer_K8tX", "summary": "This paper introduces Decision Mamba, an offline RL backbone based on State Space Models. It enhances policy robustness by integrating a fine-grained SSM module alongside the original coarse-grained SSM in Mamba. Meanwhile, it adopts a progressive self-evolution regularization to prevent the policy from overfitting the noisy labels. Extensive experiments against a broad baseline demonstrate its effectiveness.\n\n-----------Post Rebuttal-----------------\n\nMain concerns resolved. Update rating from BA to WC.", "questions": "- Table 3 indicates a significant impact of PSER on performance. Given its compatibility with other proposed baselines, it would be worth testing whether adding that into them would improve their performance results.\n- The context length in Mujoco environments is relatively short, minimizing the need for extensive historical temporal information. Can you provide experimental data on Atari environments to assess DM's capability in handling longer contextual sequences?\n- More experiments on Multi-Grained SSM modules are requested to bolster the supporting evidence for their effectiveness.", "limitations": "Unable to surpass the performance of the behavior policy, the model performs poorly on a suboptimal dataset.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Mamba have shown good performance across NLP and CV tasks and applying Mamba to offline reinforcement learning is an interesting direction.\n- The paper conducts a comprehensive comparison with SOTA offline RL methods.", "weaknesses": "- The advantages of DM appear somewhat constrained when considering variance. It appears that performance enhancements stem more from the PSER rather than the Mamba architecture. Combining PSER with other architectures, such as transformers, could potentially yield superior performance outcomes.\n- In the experiments investigating various context lengths, the range examined is rather restricted, and there appears to be no notable difference in performance as the context length increases in either DT or DM. It may be beneficial to conduct experiments using longer context lengths to assess potential performance variations.\n- It is not appropriate to assume that readers are very familiar with Mamba related terminology. Enhancing the methodology section with additional details to explain such terms would improve readability. For instance, the term \"group branch\" used in the ablation experiments lacks an earlier clear definition within the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer K8tX for the thoughtful comments. We now address the concerns below.\n\n> **Q1**: (1) The advantages of DM appear somewhat constrained when considering variance. (2) It appears that performance enhancements stem more from the PSER rather than the Mamba architecture. (3) Combining PSER with other architectures, such as transformers, could potentially yield superior performance outcomes.\n\n**A1**: (1) The performance variance mainly stems from the nature of the tasks in Mujoco. As shown in Table T1 in the uploaded pdf in the \"global\" response, the variance of each of the SOTA methods (including DT [1], EDT [2], and LaMo [3]) is quite large on the Mujoco tasks. The average variance of DT, EDT, LaMo, and DM are 1.9, 4.6, 2, and 2.2, respectively. It can be observed that DM has a similar variance comparing with other methods while its performance is substantially higher than them.\n\n(2) Although PSER improves the performance substantially, mamba also brings significant improvements. As shown in Table 1 and Table 3 in the submission, the performance of original DT is 75.8 while the performance of the proposed DM without PSER is 77.2, which shows an unignorable improvement (+1.4). When combing with PSER, the proposed DM is further strengthened.\n\n(3) Following the good suggestion, we conduct extra experiments combining PSER with BC and DT. Please refer to **the common concern in the \"global\" response** for the details. In summary, PSER improves the performances of DT and BC by 1.8 and 1.9, respectively. PSER is also beneficial to other methods, while the best performance is still achieved by the proposed Decision Mamba.\n\n> **Q2**: In the experiments investigating various context lengths, the range examined is rather restricted, and there appears to be no notable difference in performance as the context length increases in either DT or DM. It may be beneficial to conduct experiments using longer context lengths to assess potential performance variations.\n\n**A2**: We conduct additional experiments with longer context. As shown in Table T3 in the uploaded pdf in the \"global\" response, we first increase the context length to 200, both DM and DT experience significant performance degradation. When further enlarging the context length, the GPU (NVIDIA A800-80G) is out of memory. Then, we go to the opposite direction, trying extreme shorter context length, e.g., 5. The performance of DT and DM both drop substantially due to the lack of the contextual information. However, the performance degradation of DM is significantly less than that of DT. This suggests that our proposed DM effectively preserves the information in the input sequences.\n\n> **Q3**: It is not appropriate to assume that readers are very familiar with Mamba related terminology. Enhancing the methodology section with additional details to explain such terms would improve readability. For instance, the term \"group branch\" used in the ablation experiments lacks an earlier clear definition within the text.\n\n**A3**: Thanks for your good suggestion and we will add more background about mamba to the paper, e.g., supplementing the mamba background in the Related Work section 2.2 and the Method section 3.1. We will also add more explanations about the \"group branch\". The group branch denotes the combination of the coarse-grained module branch and the fine-grained module branch.\n\n\n> **Q4**: Table 3 indicates a significant impact of PSER on performance. Given its compatibility with other proposed baselines, it would be worth testing whether adding that into them would improve their performance results.\n\n**A4**: Please refer to **A1(3)** above.\n\n> **Q5**: The context length in Mujoco environments is relatively short, minimizing the need for extensive historical temporal information. Can you provide experimental data on Atari environments to assess DM's capability in handling longer contextual sequences?\n\n**A5**: Regarding the ability to handle longer context sequences, please refer to **A2**. The results show that DM has a significant advantage over DT under the settings of both long and short context. It is a good suggestion to extend experiments in the Atari environment. However, due to the limited rebuttal time, we are unable to conduct comprehensive experiments regarding various context length and methods in Atari. We will supplement this experiment in the future.\n\n> **Q6**: More experiments on Multi-Grained SSM modules are requested to bolster the supporting evidence for their effectiveness.\n\n**A6**: As shown in Table 3 in the submission, DM with Coarse-Grained SSM module (i.e., DM w/o GB) achieves the performance of 79.2, while DM with Multi-Grained SSM module (i.e., the proposed DM) obtains the performance of 83.2. It demonstrates the effectiveness of Multi-Grained SSM modules.\n\n## Reference\n\n[1] Chen L, Lu K, Rajeswaran A, et al. Decision transformer: Reinforcement learning via sequence modeling. NeurIPS 2021.  \n[2] Wu Y H, Wang X, Hamaya M. Elastic decision transformer. NeurIPS 2024.  \n[3] Shi R, Liu Y, Ze Y, et al. Unleashing the power of pre-trained language models for offline reinforcement learning. ICLR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lHSBIGVlLu", "reviewer": "Reviewer_2Wf9", "summary": "The paper proposes a robust method based on Mamba for Offline Reinforcement Learning. Additionally, the paper using the knowledge of the past policy to refine the noisy labels as supervision avoids the model fitting the noisy trajectories. To better train the model, the paper introduce the inverse training goals which simultaneously predict action, state, and RTG for better robustness. Extensive experiments demonstrated the effectiveness of proposed model and training strategy.", "questions": "1.\tHave you considered applying the self-evolution regulation method and inverse training to other transformer-based methods to improve performance? Are these two methods universally applicable?\n\n2.\tThe formula of refined target at k-th is too simple which is a linear formula. Can it be made more complex? Additionally, the parameters of linear formulas are generally learned through learning. Can they also be learned in your work.\n\n3.\tAs mentioned in lines 183-187, there is a lack of explanation as to why Formula 16 does not introduce more historical information such as ak-2, ak-3, etc.", "limitations": "Despite not utilizing Mamba's computationally efficient capability to capture long-range dependencies, a window length was still set.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tIntroducing Mamba from the perspective of capturing historical information is interesting, and there are also some modifications to the Mamba architecture itself.\n\n2.\tUsing easy-to-implement methods to refine noise labels using past policies to avoid overfitting onto suboptimal paths.\n\n3.\tA novel training method that enhances the robustness of the model by incorporating the next stage's state and RTG into the prediction.", "weaknesses": "1.\tLack of explanation for extracting intra step relationships: The paper lacks a specific explanation and verification for how intra-step relationships are extracted, as mentioned in the contributions. \n\n2.\tLack of visual experiments on action changes affected by Formula16 and whether overfitting has truly been avoided: The paper only demonstrates the effectiveness of Formula 16 from the perspective of metrics. Can the effect of Formula 16 be visualized to prove that it preventS the model from overfitting to the suboptimal path and to show the impact it had on the actions compared with actions without Formula 16.\n\n3.\tLack of specific experimental settings for the three hyperparameters in equation 23: The paper mentions improving the robustness of the model by changing the training objectives during the training phase, but there is a lack of experiments on the specific coefficients of the losses for the three parts and the impact of different hyperparameter values.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer 2Wf9 for the efforts in reviewing our work. We have provided detailed explanations and additional experiments to address your concerns.\n\n> **Q1**: Lack of explanation for extracting intra step relationships...\n\n**A1**: Intra-step relationships mean the potential causal relationships among the **S**tate, **A**ction, and **R**eturn-to-go (**SAR**) in a single step. The intra-step relationship is one kind of the fine-grained information. It is well known that convolution neural networks (CNNs) are good at extracting local fine-grained features via small filters and sliding windows. Thus,  we use CNNs to model the  intra-step dependencies among SAR in each step. The extracted fine-grained features are then fused with the original coarse-grained features in each Decision Mamba block, as shown in Figure 1 in the submission.\n\nWe have also conducted experiments to validate the effectiveness of intra-step relationships. As shown in Table 3 in submission,  the performance of DM drops significantly without intra-relationship (i.e., DM w/o GB), which demonstrates its effectiveness.\n\n> **Q2**: Lack of visual experiments on action changes affected by Formula16...\n\n**A2**: In addition to the quantitive metrics as shown in Table 3 in the paper, we actually have already provided the visualization of the action distribution in Figure 6 in Appendix E. We assume the action distribution learned from the medium-expert dataset as the ground truth. Even with high levels of noise in the training data (i.e., the medium-replay dataset), our DM still learns a action distribution highly consistent with the \"ground truth\". In contrast, the action distributions learned by DT are far away from the \"ground truth\". It demonstrates the effectiveness of Formula 16. In addition, we also provide an additional visualization comparing the agent's motion in the Walker-M-R task when trained with DT and DM as shown in the Figure F1 in the uploaded pdf.\n\n> **Q3**: Lack of specific experimental settings for the three hyperparameters...\n\n**A3**: $\\lambda_1$ controls the contribution of the first term that learns the refined action target. As the action target is the primary goal for learning a policy, $\\lambda_1$ is supposed to be set to a great value, i.e., >0.5. $\\lambda_2$ and  $\\lambda_3$ control the contributions of the losses of predicting  the next state and return-to-go (RTG), respectively. Since the state and RTG predictions are not the goal of the policy but serve as the regularizers, we assign $\\lambda_2$ and $\\lambda_3$ with small values. We further conduct experiments with different values for the three hyperparameters  $\\lambda_1$,  $\\lambda_2$, and $\\lambda_3$, i.e., **S1**: $\\{\\lambda_1=1, \\lambda_2=0, \\lambda_3=0\\}$; **S2**: $\\{\\lambda_1=0.9, \\lambda_2=0.05, \\lambda_3=0.05\\}$ (used in our paper); **S3**: $\\{\\lambda_1=0.8, \\lambda_2=0.1, \\lambda_3=0.1\\}$; and **S4**: $\\{\\lambda_1=0.6, \\lambda_2=0.2, \\lambda_3=0.2\\}$.\n\nAs shown in Table T2 in the uploaded pdf in the \"general\" response, the best performance is achieved with the **S2**. The performances of **S2** and **S3** are very close, which demonstrates DM is not sensitive to the hyperparameters. In addition, when $\\lambda_1$ is reduced from 0.9 to 0.6, the performance (i.e., **S4**) drops significantly. The reason is that too small $\\lambda_1$ leads to underfitting the action target. Furthermore, when the weights $\\lambda_2$ and $\\lambda_3$ for the regularizers are set to 0, the performance (i.e., **S1**) decreases by 1.4 compared to **S2**, which validates the effectiveness of these two regularizer terms.\n\n> **Q4**: Have you considered applying the self-evolution regulation method ...\n\n**A4**: Please refer to **the common concern in the \"global\" response** for the details, where we provide more experimental results of the proposed learning strategies, i.e., PSER and ILO, on other methods.\n\n> **Q5**: The formula of refined target at k-th is too simple...\n\n**A5**: $\\beta_k$ controls the contribution of the knowledge from the past policy for refining the action label. As the training progresses, the refined action tends to be more accurate, and thus $\\beta_k$ is supposed to be greater gradully. Therefore, $\\beta$ can be predicted with the loss of fitting the action target.\n\n$\\beta_k=\\exp(-\\lambda\\cdot\\mathcal{L}_{k})=\\exp(-\\lambda\\cdot||\\hat{a}_k-a_k||^2) $\n\nThe experimental result is shown in Table T4 in the uploaded pdf, where DM-Learning denotes predicting $\\beta_k$ using the above strategy, and DM-Linear denotes using the linear formula for $\\beta_k$ as given in the paper. It can be observed that the results of the above two methods do not differ significantly. Therefore, we take the simple linear formula in this paper.\n\n> **Q6**: There is a lack of explanation as to why Formula 16 does not introduce more historical information.\n\n**A6**: Thanks for the good suggestion. We conduct additional experiments to investigate the effects of more historical information. Specifically, we define the $\\tilde{a}_k=\\beta {a}_k+(1-\\beta)\\tilde{a}\\_{k-1}$ following an exponential moving average method, where k goes from 1 to the final step and $\\beta$ is set to 0.9. As shown in Table T6 in the uploaded pdf in the \"global\" response, the performance experiences a slight decrease. We speculate that the knowledge from the policy at the early steps is not accurate, and thus introducing too much historical information may accumulate the errors, hurting its performance.\n\n> **Q7**: Despite not utilizing Mamba's... a window length was still set.\n\n**A7**:  For fair comparison, we follow the existing literature [8, 23, 42, 43, 58] to set the window length. Additionally, we conduct experiments with a larger context length. As shown in Figure 3 of the original submission and Table T3 in the uploaded pdf, the experimental results remain consistent that DM surpasses other transformer-based methods substantially.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KIMX6GHfi3", "reviewer": "Reviewer_vt4f", "summary": "This paper tackles the sequential decision-making problem in an offline RL setting. The authors propose Decision Mamba (DM), an extension of Mamba to adapt to the problem. There are 3 main technical contributions: (a) DM architecture (a mix of fine-grained and coarse-grained SSM modules), (b) progressive self-evolution regularization (PSER) for label robustness, and (c) self-supervised loss (predicting states and rewards as well as action labels). The experiments show that DM outperforms baselines, including transformer-based approaches.", "questions": "Regarding fine-grained and coarse-grained modeling, h^FG contains a conv layer, but h^CG doesn't. Why does the difference come out? What's the motivation?\n\nAre PSER and self-supervised loss also effective against baselines such as decision transformers? Have you conducted any experiments on that?", "limitations": "Limitations are addressed.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Applying Mamba-like models to the offline RL problem sounds natural and an interesting direction.\n1. Empirical results look nice. DM significantly beats baselines. \n1. The paper is clearly written and easy to follow.", "weaknesses": "1. It is somewhat unclear why that design choice on DM was made. \n1. The technical contributions (a,b,c) would be independent and not tightly connected. I mean, (b) and (c) are possibly also effective to other methods like decision transformer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer vt4f for the time in evaluating our work. We now answer the concerns raised in the comments below.\n\n> **Q1**: It is somewhat unclear why that design choice on DM was made.\n\n**A1**: Compared to the transformer architecture, the state space model (used in mamba) has advantages in capturing historical information when modeling a sequence [1,2,3]. This is further confirmed by the supplementary experimental results in Table T5 in the uploaded pdf in the \"global\" response, where the mamba architecture (i.e., mamba_original) surpasses DT significantly. Furthermore, we made a modification to the mamba architecture specifically for Offline RL tasks by proposing the fine-grained SSM module. As shown in Table 3 in the submission, it brings an unignorable improvement (from 79.2 to 83.2). In addition, we further compare the effects of the proposed learning strategies, i.e., PSER and ILO, on mamba and transformer architectures. As shown in **the answer to the common concern in the \"global\" response**, although PSER and ILO improve the performance of DT substantially, DM still benefits more from these two strategies as it outperforms *DT w/ PSER&ILO* by a large margin. Thus, we make these design choices on DM.\n\n> **Q2**: The technical contributions (a,b,c) would be independent and not tightly connected. I mean, (b) and (c) are possibly also effective to other methods like decision transformer.\n\n**A2**: Please refer to **the common concern in the \"global\" response** for the details, where we provide more experimental results of the proposed learning strategies, i.e., PSER and ILO, on other methods.\n\n> **Q3**: Regarding fine-grained and coarse-grained modeling, $h^{\\mathrm{FG}}$ contains a conv layer, but $h^{\\mathrm{CG}}$ doesn't. Why does the difference come out? What's the motivation?\n\n**A3**: Intra-step relationships among the **S**tate, **A**ction, and **R**eturn-to-go (**SAR**) is one kind of the fine-grained information. It is well known that convolution neural networks (CNNs) are good at extracting local fine-grained features via small filters and sliding windows. Thus, we use the convolution layer to obtain the fine-grained features $h^{\\mathrm{FG}}$, modeling the dependencies among each intra-step SAR. Regaring coarse-grained modeling, we use the state space model (gate-based architecture) directly to model the whole sequence to obtain coarse-grained features $h^{\\mathrm{CG}}$. The fine-grained features  $h^{\\mathrm{FG}}$ are then fused with the coarse-grained features $h^{\\mathrm{CG}}$ in each Decision Mamba block, thus obtaining the multi-grained features, as shown in Figure 1 in the submission. We have also conducted experiments to validate the effectiveness of multi-grained information. As shown in Table 3 in the submission,  the performance of DM drops significantly without extracting multi-grained features (i.e., DM w/o GB), which demonstrates its effectiveness.\n\n> **Q4**: Are PSER and self-supervised loss also effective against baselines such as decision transformers? Have you conducted any experiments on that?\n\n**A4**: Please refer to **the common concern in the \"global\" response** for the details, where we provide more experimental results of the proposed learning strategies, i.e., PSER and ILO, on other methods.\n\n## Reference\n\n[1] Gu A, Dao T. Mamba: Linear-time sequence modeling with selective state spaces. Arxiv 2023  \n[2] Zhu L, Liao B, Zhang Q, et al. Vision mamba: Efficient visual representation learning with bidirectional state space model. ICML 2024.  \n[3] Dao T, Gu A. Transformers are SSMs: Generalized models and efficient algorithms through structured state space duality. ICML 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e8Av8qCjS2", "reviewer": "Reviewer_qjYZ", "summary": "This paper introduces a novel application of state space model (SSM) into the offline RL problem. Prior to this, transformer-based architectures were used heavily, but the authors claim that they did not process “historical information,” which is often an important requirement in real world scenarios. In this work, a Decision Mamba encoder is composed of three stages. First, it embeds its trajectory using three types of MLPs along with time embeddings. Second, coarse-grained SSM directly uses hidden states from a structured SSM for understand sequential dependency. Thirdly, fine-grained use an 1D-convolutional layer for encompassing past few embeddings to precisely understand context. For training, this work proposes a progressive self-evolution regularization (PSER). In experiments, DM archives very promising scores in various offline RL MuJoCo tasks.", "questions": "I do not see tendency in Fig. 3. Is there results with very few context lengths? eg. {1, 5, 10}", "limitations": "The authors authors adequately addressed the limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "SSMs are a new class of models that is reported to be successful for various temporal problems; this reviewer believe they need to be tested for various subfields of ML that deal with sequential tasks (presumably equipped with the Markov assumption).  Since SSMs always produce linear-time sequential representation of their states, I think it was reasonable for authors to consider this characteristic as an advantage to solve decision problems. Notably, this paper have following strengths\n- This is a good application of SSM into the offline RL tasks. \n- The proposed architecture was straightforward to understand\n- The experimental results are evaluted in various environments and the proposed model achieved solid numbers.", "weaknesses": "- I find the lack of detailed reasoning of the manuscript why Conv1D module brings “fine-grained” control.\n- From my perspective, the proposed PSER is related to PPO and DPO methods in terms of regularization. It would be great there was similar theoretical justification (formally or informally) for this proposed loss term.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer qjYZ for the valuable comments. We now address the concerns raised in the review below.\n\n> **Q1**: I find the lack of detailed reasoning of the manuscript why Conv1D module brings “fine-grained” control.\n\n**A1**: Intra-step relationships among the **S**tate, **A**ction, and **R**eturn-to-go (**SAR**) is one kind of the fine-grained information. It is well known that convolution neural networks (CNNs) are good at extracting local fine-grained features via small filters and sliding windows. Thus,  we use Conv1D to model the  intra-step dependencies among SAR in each step. The extracted fine-grained features are then fused with the original coarse-grained features in each Decision Mamba block, as shown in Figure 1 in the submission. We have also conducted experiments to validate the effectiveness of fine-grained information. As shown in Table 3 in the submission,  the performance of DM drops significantly without extracting fine-grained features (i.e., DM w/o GB), which demonstrates its effectiveness.\n\n> **Q2**: From my perspective, the proposed PSER is related to PPO and DPO methods in terms of regularization. It would be great there was similar theoretical justification (formally or informally) for this proposed loss term.\n\n**A2**: We attempt to explain the relationship between the PPO algorithm and our method from the gradient perspective. Both the PPO method and our method impose additional constraints on the learning objective. Theoretically, the PPO algorithm limits the gradient magnitude of the policy by maintaining a bounded update on the policy ratio. Our method introduces an additional constraint term $R$ in the loss function to smooth the gradient, by using the refined target label generated from the policy itself as a constraint. Although the forms of the constraints differ, both aim to smooth the gradients, thus ensuring the stability of model training and enhancing model robustness. DPO optimizes the policy directly based on preference information (e.g., human choices) rather than using the old policy to stabilize training, which is intuitively and theoretically different from PPO and the proposed method.\n\n> **Q3**: I do not see tendency in Fig. 3. Is there results with very few context lengths? eg. {1, 5, 10}\n\n**A3**: We run the experiments with the setting of shot context lengths. As shown in Table T3 in the uploaded pdf in the \"global\" response, the performances of DT and DM both drop substantially due to the lack of the contextual information. However, the performance degradation of DM is significantly less than that of DT. This suggests that our proposed DM effectively preserves the information in the input sequences.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# "Global" Response
We thank all the Reviewers, ACs, SACs, and PCs for their efforts and valuable comments. In terms of the idea of this paper, all the reviewers have recognized that exploring mamba on the Offline RL is interesting, and we have also made some modifications to mamba specifically for Offline RL tasks. In terms of the experiments, almost all the reviewers (Reviewer K8tX, vt4f, qjYZ) have recognized that the proposed method beats the state-of-the-art baselines significantly. In summary, both the idea and the experiments have been acknowledged by almost all the reviewers. In this "global" response, we address the common concern raised by the reviewers, while we respond to the other concerns raised by each reviewer in separate rebuttals.

---

Since we improve the effectiveness on offline RL tasks from two novel perspectives, i.e., the model architecture (mamba) and the learning strategies (PSER and ILO), the reviewers have a common concern about **whether the proposed learning strategies, i.e., PSER and ILO, are beneficial to other methods**.

>  **Common Concern: Whether the proposed learning strategies, i.e., PSER and ILO, are beneficial to other methods.**

**Answer**: We conduct additional experiments to evaluate **the effects of the proposed learning strategies (PSER and ILO) on other methods**. Specifically, we apply PSER and/or ILO to two classic Offline RL methods, i.e., Decision Transformer (DT) and Behavior Cloning (BC). As shown in the following table, PSER improves the performances of DT and BC by 1.8 and 1.9 on average, respectively. ILO improves the performance of DT by 0.9 (ILO is not applicable to BC). When combining PSER and ILO, the improvement on DT is further strengthened (+4.2). Therefore, the proposed learning strategies are also beneficial to other methods. However, the best performance is still achieved by the proposed DM, which demonstrates the superiority of DM.

|               | Halfcheetah-M | Hopper-M | Walker-M | Halfcheetah-M-E | Hopper-M-E | Walker-M-E | Halfcheetah-M-R | Hopper-M-R | Walker-M-R | Avg  |
| ------------- | ------------- | -------- | -------- | --------------- | ---------- | ---------- | --------------- | ---------- | ---------- | ---- |
| **DT**        | 42.6          | 70.4     | 74.0     | 87.3            | 106.5      | 109.2      | 37.4            | 82.7       | 66.2       | 75.8 |
| w/ PSER       | 43.1          | 69.3     | 79.8     | 90.4            | 109.3      | 110.3      | 39.3            | 81.3       | 75.6       | 77.6 |
| w/ ILO        | 43.0          | 72.2     | 77.2     | 90.6            | 108.3      | 109.7      | 39.6            | 78.4       | 71.5       | 76.7 |
| w/ PSER & ILO | 43.4          | 83.9     | 82.4     | 92.7            | 111.2      | 110.3      | 41.5            | 79.7       | 75.6       | 80.0 |
| **BC**        | 42.2          | 55.6     | 71.9     | 41.8            | 86.4       | 80.2       | 2.2             | 23.0       | 47.0       | 50.0 |
| w/ PSER       | 43.0          | 57.3     | 75.1     | 42.9            | 88.7       | 83.2       | 2.2             | 28.5       | 46.9       | 51.9 |
| w/ ILO        | N/A           | N/A      | N/A      | N/A             | N/A        | N/A        | N/A             | N/A        | N/A        | N/A  |
| **DM**        | 43.8          | 98.5     | 80.3     | 93.5            | 111.9      | 111.6      | 40.8            | 89.1       | 79.3       | 83.2 |

Meta-review: This paper proposes "Decision Mamba," a novel multi-grained state space model (SSM) with self-evolution regularization tailored for offline reinforcement learning (RL) tasks. The paper addresses three key challenges in offline RL: (1) insufficient utilization of historical temporal information, (2) overlooking intra-step relationships, and (3) overfitting suboptimal trajectories. The proposed method demonstrates robust performance improvements across various tasks by integrating fine-grained and coarse-grained SSM modules and employing a self-evolving policy learning strategy.
This paper received all positive recommendations (three weak accepts and one borderline accept). The major concern from reviewers was about the learning strategies, which was addressed in the rebuttal. The AC agrees with the reviewers’ assessments and recommends accepting the paper.
Decision: Accept (poster)