Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning
Authors: Ruizhe Shi
Yuyao Liu
Yanjie Ze
Simon Shaolei Du
Huazhe Xu
Venue: ICLR 2024 poster (2024)
Keywords: Offline Reinforcement Learning
Decision Transformer
Motion Control
Primary area: reinforcement learning
TL;DR: We leverage the power of pre-trained Language Models for low-level motion control in offline reinforcement learning.

Abstract: Offline reinforcement learning (RL) aims to find a near-optimal policy using pre-collected datasets. Given recent advances in Large Language Models (LLMs) and their few-shot learning prowess, this paper introduces $\textbf{La}$nguage Models for $\textbf{Mo}$tion Control ($\textbf{LaMo}$), a general framework based on Decision Transformers to effectively use pre-trained Language Models (LMs) for offline RL. Our framework highlights four crucial components: (1)  Initializing Decision Transformers with sequentially pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to full-weight fine-tuning, to combine the pre-trained knowledge from LMs and in-domain knowledge effectively, (3) using the non-linear MLP transformation instead of linear projections, to generate embeddings, and (4) integrating an auxiliary language prediction loss during fine-tuning to stabilize the LMs and retain their original abilities on languages. Empirical results indicate $\textbf{LaMo}$ achieves state-of-the-art performance in sparse-reward tasks and closes the gap between value-based offline RL methods and decision transformers in dense-reward tasks. In particular, our method demonstrates superior performance in scenarios with limited data samples.

Published as a conference paper at ICLR 2024
UNLEASHING THE POWER OF PRE-TRAINED LANGUAGE MODELS FOR OFFLINE REINFORCEMENT LEARNING
Ruizhe Shi"
Yuyao Liu''
Yanjie Ze
Simon Du
Huazhe Xu
mss Tsinghua University
Shanghai Qi Zhi Institute
3 University of Washington
'Shanghai Al Lab
ABSTRACT
Offline reinforcement learning (RL) aims find near-optimal policy using pre-collected datasets. In real-world scenarios, data collection could be costly and risky; therefore, offline RL becomes particularly challenging when the in- domain data is limited. Given recent advances in Large Language Models (LLMs) and their few-shot learning prowess, this paper introduces Language Models for Motion Control (LaMo), a general framework based on Decision Transformers to effectively use pre-trained Language Models (LMs) for offline RL. Our frame- work highlights four crucial components: (1) Initializing Decision Transformers with sequentially pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to full-weight fine-tuning, to combine the pre-trained knowledge from LMs and in-domain knowledge effectively. (3) using the non-lineat MLP trans- formation instead of linear projections, generate embeddings, and (4) integrat- ing an auxiliary language prediction loss during fine-tuning to stabilize the LMs and retain their original abilities languages. Empirical results indicate LaMo achieves state-of-the-ae performance in sparse-reward tasks and closes the gap be- tween value-based offline RL methods and decision transformers dense-rewand tasks. In particular, our method demonstrates superior performance in scenarios with limited data samples. Our project website is lamo2023.github.io
Kitchen
Atari
MujaCo
46
Score
150
134
40-
39
20-
67
45
LaMo (Ours)
DT
Wiki-RL RL
CQL
BC
Figure 1: Normalized score on D4RL Fuu al., 2020) dataset of Language Models for Motion Control (LaMo)) Decision Transformer (DT, Chen et al. 2021). Wiki-RL (Reid et al. (022). Con- servative Q-Learning (CQL, Kumar al. 2020) and Behavior Cloning (BC). We average scores over tasks and data sample ratios for each domain. (Medi um for Mujoco and Atari, Comp ete and Partial for Kitchen, of different sample ratios, described in Appendix
INTRODUCTION
Offline reinforcement learning (RL) has gained significant attention in recent years due to its poten- for utilizing pre-collected datasets to improve agent performance (Lange et al., 2012: Prudencio et al. 2023 Levine et al., 2020). Among the prominent algorithms in offline RL. Decision Trans- former (DT) (Chen et al,, 2021) reframes RL as conditional sequence modeling problem and
utilizes the Transformer architecture (Vaswani al. (017), showing the potential of sequence mod- els for decision making (Xu et al.] 2022 Hu et al. 2023a b. Xie et al. 2023 Laskin et al.) 2023). However, Transformers
known be data hungry (Khan et al. [202 Brown al. 2020 OpenAl,
2023), meaning that pre-training on massive amounts of data is usually required to achieve satisfac-
tory model abilty (Touvron al. 202T). One of the most pronounced applications of Transformers
Equual contribution. Order decided by coin flip.

---

Published as a conference paper at ICLR 2024
large language models (LLMs) has achieved significant progress in language understanding recently, such as GPT (Radford & Narasimhan, 20118] Radford et al. 20199 Brown et al., 2020; Ope-
nAl 2023). ChatGPT (Ouyang al. [022). and LLaMA (Touvron et al. (202a)). Pre-trained on
rich and diverse linguistic data, LLMs gain great few-shot and zero-shot learning abilities (Brown
al. 2020 Kojima et al. 2022).
A natural thought to enhance the Transformer-basee sequential decision-makingg methods is thus to introduce the power of pre-trained Language Models (LMs) into them, initially explored by a lot of
2023 Wu et al., 2023; Li et al, recent works (lchter et al. [2022] Huang et al., 2022] Driess et al.
2022 Reed et al., 2022 Lin al. 2023 Brohan et al. 2023b a Tang al.) 2023; Wang et al.,
20236). Among them, Li et al. (2022) propose to encode the environment states with LLMs and
learn policy based on the decoded states, while their environment states are restricted to language descriptions only, making it hard for motion control. |Reid et al. (2022) address this weakness by directly utilizing pre-trained LM as the initialization of DT and processing low-level agent states and actions directly, instead of processing language descriptions. Their architecture thus success- fully utilizes pre-trained LMs in motion control tasks like locomotion (Fu et al. 2020). However, despite the novelty of the proposed method in Reid al. (2022), they still do not fully unleash the power of LMs: their empirical performance is on par with pure DT methods and lags behind CQL Kumar al. 2020). We thus ask,
Can we unleash the power of pre-trained LMs to solve sequential decision-making problems?
In this work, propose Language Models for Motion Control (LaMo), framework to effectively utilize pre-trained LMs for offline RL. While the motivation is straightforward, it takes four crucial designs empower LaMo: 1) pre-trained language model is used as the initial weight of DT: 2) the pre-trained weights are frozen and the model is fine-tuned with parameter-efficient finetuning method LoRA (Hu al., 2022) on 0.7% of the parameters; 3) we replace the input embeddings and the output linear projections with Multi-Layer Perceptrons (MLPs); 4) a language prediction loss function as an auxiliary objective. Consequently, we find that the four components combined can help LaMo preserve the prior knowledge and generalization ability acquired from the pre-training while adapting efficiently to the new domain of offline RL.
We conduct comprehensive experiments across three distinct environments: Kitchen (Gupta et al.) 2019], MuJoCo (Todorov et al.] 2012), and Atari (Bellemare et al.) 2013), spanning 8 tasks alto- gether. These tasks range from sparse-reward to dense-reward, and from state inputs and image inputs. For each task, we evaluate performance under varying data ratios to examine the influence of sample amount on the outcomes. We observe that is shown in Figure [ LaMo surpasses both DT and value-based baselines in sparse-reward tasks; and in dense-reward tasks, our method significantly outperforms DT and closes the gap between value-based methods and DTT-baedd meth- ods. Especially, we find that when the data scale limited (e.g., 1% of the whole dataset), LaMo demonstrates much more powerful learning ability, which could be credited inductive bias within pre-trained LMs.
Our contributions are three-fold:
We propose LaMo, novel offline RL framework that unleashes the power of pre-trained lan- guage models.
To better utilize the cross-domain knowledge from language modeling, we propose 3 additional techniques including LoRA finetuning, non-linear MLP projections, and an auxiliary language loss. Each module is shown to contribute positively to the final results of LaMo.
Through extensive experiments in 8 tasks across diverse domains, dataset scales, and reward densities, we demonstrate the superiority of LaMo over D'T-based and value-based offline RL. algorithms. Specifically, we find that LaMo could successfully handle the challenging low-data regime while DT could not. This highlights the great potential of our cross-domain pre-training for sequential modeling.
2 RELATED WORK
Transformers for decision making, Transformers have dominated the language tasks in the NLP community (Radford & Narasimhan 2018 Radford et al. 2019 Brown al. 2020 Devlin al.
2

---

Published as a conference paper at ICLR 2024
2019) and also started to show potential in other domains, such as decision making. As one initial tral to introduce Transformers into reinforcement learning (RL), Decision Transformer (DT) (Chen et al. 202T) models the elements such as states and actions into sequence, thus framing the RL problem into a sequence prediction problem. There are a lot of following works make improvements
under the framework of DT (Xu et al. [022 Hu et al. 2023b Xie et al., 2023 Yamagata et al., 2023
Liu & Abbeel, 2023). For example, Prompt DT (Xu et al. (002) appends demonstrations into the
sequence achieve generalization in new tasks; Xie et al. (2023) pre-train DT by leveraging future trajectory information; Q-learning DT (Yamagata et al. (023) refinss the return-oogo in training data using Q-values, thereby imbuing DT with QlJearnin's proficiency in handling sub-optimal data. Agentic Transformer (Liu & Abbeel, 2023) addresses the issues of sub-optimality by using chain of hindsight to relabel the target returns, which achieves competitive performance compared with value-based based methods. Trajectory Transformer (Janner et al. (02T) trains on sequences of dis- cretized states, actions, and rewards, indicating a more direct solution. Our work focuses on utilizing the cross-domain knowledge, i.e. language pre-training, as privileged information to enhance DT- based methods, which thus is orthogonal to these works.
Large Language Models (LLMs) have been the most pronounced application of the Transformer architecture in recent years (Radford & Narasimhan 2018 Radford et al.] 2019 Brown et al.) [2200]
OpenAT 2023 Devlin et al.] 2019 Touvron et al. 2023a b). Pre-trained
massive amounts of corpus, L.LMs have shown surprising few-shot and even zero-shot ability in language tasks, such as
GPT series (Radford & Narasimhan 2018 Radford al. 2019 Brown al., 2020 OpenAl 2023).
To personalize LLMs for different downstream user applications with computational efficiency, re- searchers commonly utilize parameter-efffcient finetuning techniques (Hu al.
2022 Zhang al.) 2023a Li & Liang, 2021; Lester et al. 2021 Liu et al. [022] Wang et al. (023a) to finetune LLMs.
In this work, we use the GPT-2 architecture (Radford et al.] (019)) as the backbone due to its afford-
ability and use LoRA (Hu et al. 2022) for downstream finetuning.
LMs for decision making. The great success of L.Ms language tasks also motivates researchers
to explore the potential of I.Ms for decision making problems (Ichter et al. 2022
Huang et al.)
2022
Driess et al.] 2023 Wu et al. 2023). One line of works (Ichter et al. 2022:
Huang et al, Wu et al. 2023 utilizes LMs for high-level task decomposition and task
2022 Driess et al. 2023
planning, while their low-leve execution policy is learned or designed separately. Another line
of works (Li et al. 2022 Reed et al., 2022; Lin et al.] 2023 Brohan et al. 2023a Tang et al.,
2023 Wang et al.) (023b) exploits the representation and generalization power of aire-triied LMs.
Li et al. (2022) adapt pre-trained LMs to generate policies for tasks where the inputs could be converted into word sequences and point out the significance of sequential structure of inputs; Lin al. (2023) use a geometric feasibility planner to encourage LM to generate both mid-level and low- level plans given language instruction; and Tang et al. (2023) design prompts for LMs to encode language instructions. When multi-modal inputs are involved, one solution is transforming them into one common embedding space (Brohan et al.] 2023a Reed et al.) (022). For example, RT- 2 (Brohan et al., (202a)) utilizes a Vision-Language Model pre-trained on massive language and vision-language data, and also represents actions as text tokens on the Robot-Action Fine-tuning stage; GATO (Reed et al. 2022) utilizes a Vision Transformer to encode the image inputs, and learns from large multi-modal, multi-task dataset to perform various tasks all in one model.
The most relevant work to us is Wiki-RL (Reid et al. 2022), which also uses a pre-trained language model as the initialization of DT for offline RL. However, their empirical results are shown to be only close to DT and could not surpass (Kumar et al. 2020). Therefore, our work tries to better unleash the power of pre-trained LMs for offline RL.
3
PRELIMINARIES
3.1 OFFLINE REINFORCEMENT LEARNING
We formulate reinforcement learning (RL) a standard Markov Decision Process (MDP) with tuple (S,A, T, do. 7), where S is the set of states s E S, A is the set of actions a E A, T is the transition distribution of form T( [8t, az), do(so) describes the distribution of states sp.
R R:C. S x A + R is the reward function. = R(s at) is the reward at timestep t. and E (0, 10.1) is the discount factor. The agent in this MDP follows a policy x (a]s)), and the objective is:
[(T) = E E.gordo()) am)l s1), 9111-T ((11-)))))))))))
Σ 7 RR(ss, as)
(1)
3

---

Published as a conference paper at ICLR 2024
In offline RL, the access to interacting with the environment is removed while the objective remains (x). Agents could only learn on pre-collected trajectories D {(( at 11411 rt (())), }] which is generated by unknown behavior policy TB. Here we introduce common properties of the dataset many contexts, T is not an optimal policy, i.e., D would not contain
D: 1) Sub-optimality. the optimal behaviors, and thus simple imitation may exhibit suboptimal performance; 2) Dense- reward or sparse-reward. In the dense-reward environment, agents receive reward signals that correspond to whether agents' behaviors good for each timestep, while in the sparse-reward setting, positive reward signals from the environments might be only given when success achieved, and otherwise zero. The reward setting is thus much more challenging but closer to the real world scenarios.
3.2 DECISION TRANSFORMER
Following Decision Transformer (DT), we frame the RL problem as a sequential modeling problem. We consider each trajectory as a sequence of ordered return-to-go ft action a, and states s, defined follows,
T= (Reg: Sto Otp Roo+1, Sto+1, Oto+1..... Reo+
..o.........KK. K-1 StotK-1 (tt
where return-to-go A is defined the sum of rewards from the current timestep to the future: R₂. = -i= ri T is the episode length, and K is the context length. The learning objective of the model is to predict the future action af given the history sequence and the current state 8t, while the ground truth is dg written as simple squared error term:
fo+
Ldciisoo
Σ
||: d2
METHOD
We propose Language Models for Motion Control (LaMo), an effective framework that incorpo- rates pre-trained Language Models (LMs) into offline Reinforcement Learning. to leverage the rea- soning and few-shot ability of LMs and solve challenging scenarios such as limited data and sparse reward. An illustration of LaMo is given in Figure 2 LaMo encompasses several crucial designs: 1) We adopt pre-trained LM (i.e., GPT-2 (Radford al. 2019)) as the initialization of Decision Transformer (DT) (Chen et al. 2021); 2) We replace the linear embedding projections with MLPs to augment representation learning capabilities for complicated tasks; 3) During training the offline RL agents, we freeze the pre-trained parts and utilize the parameter-efficien fine-tuning technique LoRA (Hu al.] 2022). where the trainable parameters account for only 0.7% of the entire model; 4) We introduce language prediction as an auxiliary objective while finetuning, in order to stabilize the performance and maintain the language ability.
large language model pre-train
downstream offline RL
Actions speak louder with words <eos>
a
a
transformer
transformer + LoRA
<bos>
Actions speak louder with words
R
S
a
F-
The overview of LaMo. LaMo mainly consists of two stages: 1) pre-training LMs on language tasks, 2) freezing the pre-trained attention layers, replacing linear projections with MLPs, and using LoRA to adapt to RL tasks. We also apply the language loss during the offline RL stage as regularizer.
Figure 2:
4.1 PRE-TRAINING ON LANGUAGE TASKS
The initial step involves obtaining pre-trained language models (LMs). Considering the widespread recognition and computational affordability of the GPT-2 architecture (Radford al. 2019)
4

---

Published as a conference paper at ICLR 2024
utilize the commonly available pre-trained weight of GPT-2 from Hugging Facd To further explore the effects of the quality of different pre-trained models on the downstream offline RL tasks, we also pre-train GPT-2 by ourselves in the ablation study, using the corpus dataset WikiText (Merity al. 2017) and the common next-token prediction objective
Clangugge
log (T' (w++1 (wi+1/w1,... (...,,,,......00 w;))
where wy is the ith language token in one sentence, and 1" is the probability distribution of next token predicted by the model. We have explored three variants of models: 7) a model that is pre-trained for fewer steps; 2) model that is pre-trained on randomly shuffled text corpus; 3) a model with randomly initialized weights. Our results in Section 5.35 and Appendix [H] show that high language pre- pre-training quality is helpful for downstream RL tasks, underscoring the importance and necessity of the pre-training.
4.2 FINETUNING FOR OFFLINE REINFORCEMENT LEARNING
Multi-layer perceptrons for embeddings. The pre- pre-trained LMs process the input into latent vectors and decode the latent vectors into the output via simple linear projections. We find that to effectively utilize the pre-trained language model in offiie RI,, replacing the linear projections with MI.s is essential to bridge the domain gap. Extensive ablations are provided in Section 5.5 to support the importance of this non-linear module.
Frozen weights and low rank adaptation. We apply the parameter-efficien training technique LoRA (Hu et al. 2022), which constrains the gradient update process in low-dimension space by rewriting the weight matrix W E Rd Wo + AW Wo + BA, where B E Rd A E R and r < min(d, k). We inject low-rank matrices into the attention weights Q, K, V and freeze all other weights of the Transformer.
Meanwhile, the model is desired to maintain the knowledge of the LMs. The number of trainable parameters only takes up 0.7% of the entire Transformer. We hypothesize that such a mechanism would let the pre-trained model treat the inputs as languages to the maximum extent while maintain- ing adaptivity. Empirically, we find that full-weight finetuning frozen Transformer layers would harm performance, as is shown in Figure 5 More discussions are provided in Section 5.5
Language prediction as an auxiliary objective. To further stabilize the training process and main- tain the knowledge learned from languages, we simultaneously train the model on language predic- tion tasks. The corpus we train on WikiTeet (Merity et al., 2017). same as the pre-training stage. To perform language prediction, we would temporarily replace the input and output projections with the projections of the pre-trained LM. This auxiliary objective is used in Reid et al. (2022). Em- pirically, we find that this term could prominently prevent the model from overfitting. Intriguingly, for reward tasks such as Kitchen, the performance of LaMo is critically enhanced to surpass recent strong baselines, is shown in Figure 6b Besides, this objective could help preserve the language understanding ability, which means we could obtain a model skilled at both language un- derstanding and motion control side effect. A more detailed discussion is in Section 5.5 The overall objective while training the offline RI. agents is then
Ldecision + A Llanguage =
(5)
where À is tunable parameter that set to be in f0, 0.1, 1).
5 EXPERIMENTS
In this work, we delve into solving sequential Recision-making problems while only offline inter- action datasets are available during training, known as the Offline RL. problem. We evaluate the performance of LaMo on the standard benchmark D4RL (Fu et al., (2000) and also evaluate the learning ability of LaMo under the low-data regime. To show the effectiveness of each component LaMo, extensive ablations are also conducted.
5.1 EXPERIMENT SETUP
We conduct our experiments on 8 tasks from 3 domains MuJoCo, Atari, and Kitchen. Detailed task descriptions are provided in Appendix D We use datasets from D4RL (Fu al. 2020) and d4rl-atari
https://hugginnffaa 2
5

---

Published as a conference paper at ICLR 2024
(more details are provided in Appendix C. Due to the limitation of computation resources, each experiment for 3 seeds with numbers 0, 1, 2 to ensure reproducibility.
We compare the performance of LaMo with various powerful basclines in offline reinforcement at
learning: CQL (Kumar et al. 2020). IQL (Kostrikov et al. ).22), TD3+BC (Fujimoto & Gul (02T), BCQ (Fujimoto et al. 2019p. NFQ (Riedmiller 2005), , Behavior Cloning (BC), and DT (Chen al., 2021] Besides, we compare with Wiki-RL (Reid et al. 2022), which also utilizes pre-trained lan- guage model in offline reinforcement learning. To systematically report the performance of all these methods, we compute the average performance over the last 20K training steps out of a total of 100K training steps with evaluations conducted every 2500 training steps. The scores we report are nor- malized scores so that 100 represents an expert policy and O represents a random policy, following the convention of Fu et al. (2020) and Hafnnr et al. (2020).
5.2
SPARSE-REWARD TASKS
Task
Dataset
Ranio
LaMo
DT
Wiki-RI
OQL
IOL.
TD3+BC
BC
Kitchen
Pastial
46.6 53
338 = 14.5
20.4 10.4
0.2 1.0
45.7 3.3
62
11++19 1.9
Kitchen
Complete
64.2 1
528
21.7 6.66
0.0 00:1:00
30,0 13
06 1:0
0.011000
Reacher2d
Medium
22N
29.4
31.5 2
30.4 1.0
4.2
14,0 7.4
Averuge
17.901315)
36.5
23.8
10.6
35,4
13.3
5,0
Thok
Dntaset
Ratio
LaMo
DT
Wiki-RI
OQL
IQU.
TD3-BC
BC
Kithen
Pastial
5.5 1.5
Kitchen
Portial
0.1
95.1
22.6 6.8
27.9
0.0 0.0
19.7 3:3
17.0
4.6.::2.
Kitchan
Complete
0.3
43,9
2.9
31.5 4.5
32.8 3.9
1.7 4: 0.8
29.5
1.2
0.0 4.000
0.0
Kitchen
Complete
0.5
50,6
6.1
36.6 5.
13.9 3.1
17.6 4. 5.0
38.4 2.5
0.1 1.0.
4,5 1.1.
Reacherld
Masium
0.1
12.4
38
2.3 1.5
4.1
2.6
15.8 + 0.2
58 0.8
8.7 +:0.7
2.1 +-2.1
Reacher2d
Medium
0.3
31.2
7.6
6.4 2.6
19.4 14417.7 1.7.4
30,0 1 0.4
10.2 1.1
24.5 1.7
10.3
3.8
Average
51.1(15093
16.7
17.9
11.0
17.7
10.7
3.9
Table 1: Normalized score for sparse-reward tasks. We compare LaMo with DT, Wiki-RL, CQL,, IQL, TD3+BC, and BC. Mean of 3 seeds with number 0, 1, 2. Blue highlight indicates the highest score, orange highlight indicates the second-highest score, and red numbers represent the improve- ment of LaMo over DT.
Results for sparse-reward tasks including Kitchen and Reacher2d are given in Table [ We select strong baselines including CQL, IQL. TD3+BC, BC, DT and Wiki-RL.. We observe that LaMo shows an overwhelming advantage over DT and Wiki-RL across all tasks and datasets, which indi- cates that our approach effectively harnesses the power of the pre-trained model. Overall, LaMo has improved the performance of DT by up to 50%. Compared with value-based methods, our approach also demonstrates significant ad vantages in average performance. We have achieved the best perfor- mance among all strong baselines in tasks and second- second-place results in 2 tasks Kitchen Partial with 1% data and Reacher2d Medi um with 10% data.
Significantly, in Kitchen tasks, CQL initially performs reasonably well, but as training progresses, it faces the issue of overfitting, causing notable drop in its performance, which is shown in Ap- pendix G While for LaMo, such a phenomenon does not oceur, reflecting LaMo's success in pre- venting overfitting.
5.3
DENSE-REWARD TASKS
Tisck
Datsect
Ratio
LaMo
DT
WIG-RI.
CQL.
BCQ
NFQ
BC
Brrakkon
Mediam
4734 + 105.6
4028 1476
1290 105.9
367.8 - 131.9
56..1 19.2
-4. 4.00
201.3+ 11488
Qbert
Medium
19,0 13.1
28.91
766 1 6.5
143
50.8 16.3
0.3.00
51.9 11.2
Pasg
Medium
125.6
1161
98.1 15.6
116.4
16.5
-1.00:00
226.0/12498)
18266
78.2
189.1
114.1
Tack
Dutsset
Rabio
LaMo
DT
Wikh RL
CQL
BCQ
NFQ
BC
Brzakont
Medium
0.1
196.9
91.1
45.0
18.6
94 5.9
5X.1
19.8
150 + 6.5
-3.7 +92
62.5 153
Qbert
Medium
0.1
63.6
17.3
26 L
14.3
6.7
6.1
62.0
20.6
15.0
11.0
-0.6 0.5
0.2 0.1
Pong
Modium
0.1
114.8
à 8.8
87.1 1977
22.7 d: 10.1
199.2 9.6
57.6 à 20.4
-1.0::0000
.100 1.0200 0.1
Average
105.1 109.191998)
52.8
13.0
79.8
29.2
-1.8
205
Table 2: Normalized score for 3 dense-reward tasks in Atari. We compare LaMo with DT, Wiki- RL, CQL, BCQ, NFQ and BC. Mean of 3 seeds with number 0, 1, 2. Blue highlight indicates the highest score, orange highlight indicates the second-highest score, and red numbers represent the improvement of LaMo over DT.
Results for dense reward tasks are given in Table 2 and Table 3 For Atari, Since IQL and TD3+BC do not support discrete control (Seno & Imai 2022), we select CQL, BCQ, and NFQ baselines. We observe that LaMo achieves the highest average scores in Atari and MuJoCo under the low-data regime. However, also notice that in MuJoCo domain, when the data scale is relatively large

---

Published as a conference paper at ICLR 2024
Tink
WiG-RI
TDS+BC
Bopper
Medium
2:33
609 8.3.3
4:.8.:53
Mealium
+.0.5
41..+-0
46.7
42.1+ 1.0
Walker
Malium
57.5
Average
9119)
Tink
Tanvert
Raiis
LaMo
DTT
Wik-RI
106
TD34RC
RC
Hopper
Medium
0.005
$7.0 7:1
35,8
6.6
99.9 5.0
57.9 = 3.9
41.1 2.2.7
40.1 4: 3.6
47.0 1:444.4
Hopper
Madiom
0.1
$2.0
4.6
41.9
90.2 5.0
89.8
51.3 9:4
$1.0 1.9
50.0
Hopper
Madium
0.1
73.7 1.5
57.3
5.8
67.3 4.4
99.8 2.3
50.6 #:11
56.9 4.2.3
44.4 +.7
Talfcheetsh
Medium
0.005
39.0 1.6
22.4 1:5 5.2
37.6
00.5 1.0
34.9
17.3 1:3.0
34,8 1.8
Halicheerhh
Medium
0.01
40.6 1.3
29.5 8: 4.8
38,4
43.9 006
34.8 2
24.3 2.2.
37.2 2.3
Halccheerhh
Medium
0.1
42.3 0.0
41.7 8: 0.8
40.5 2:
45.0 2 0.5
46.7 1:03
48.3 0:0.2
42.0 1.0
Malium
0005
66.9
5.4
16.7
4.8
46.5 20.4
51.9 9.
30.9 9:000-00
34 1.2
34.0+ 12.5
Malium
001
74.5 4.7
36.9 -090--93
60.2 10.5
69.7 + 4.2
44.5 4.4.
12.9 9.4.1
65.3 + 11.2
Walker2d
Medium
70.4 4.2
70.2 1.2.1:7.
75.2
69.5 3.50
66.5 6.6.
66.7
57,4/16663
29.4
51.4
44.9
35.9
45.7
Table 3: Normalized score for dense-reward tasks in MuJoCo. We compare LaMo with DT, Wiki-RL, CQL, IQL, TD3+BC, and BC.
(10%, 100%), LaMo only comes close to DT and falls behind CQL in Halfcheetah and Walker2d. In Obert Medi um (100%) and Pong Medium um (10%), LaMo also does not surpass CQL. We attribute it to the following reasons: unlike sparse-reward tasks, where the Bellman backups would slowly propagate the information of rewards (Chen et al., (021), limiting the performance of aalue-based algorithms, dense-reward tasks are extremely suitable for value-based methods such as CQL while DT is less preferable, which empirically examined by BBhaggaa al. (2023). Our experiments verify the stands and point out that LaMo could further enhance the potential of DT, closing the performance gap between DT and CQL dense-reward tasks.
5.4
ABILITY IN LOW-DATA REGIME
Hopper Medium
Breakout Medium
Kitchen Complete
normalienn acore
275
535
1.7%
2014
45%
12% 2
40%
1/%
105%
55%
55%
LaMo
COL
DT
Figure 3: Normalized score obtained by LaMo, CQL, and DT on various data sample ratios. Mean of 3 seeds with number 0, 1,2. Shaded area [n o, + of interval, where μu is the average and is the standard deviation.
We look into the relationship between the performance of various algorithms and the scale of data. As depicted in the Figure LaMo is capable of achieving excellent performance even with relatively small datasets. For example, in Hopper, LaMo surpasses the performance of CQL and DT when the sample ratio of data is 0.5% and maintains this advantage consistently as the sample ratio increases.
Helthhwrsh Medum
Vatowze Mectum
Medum (10%)
Chet Medamm
Pong Medium (10%)
Reachard Medim (22)
Kichen Partal (195)
Kichan Completa (009)
Figure 4: Ablation on the effectiveness of MLP embeddings. We replace the MLPs in LaMo as embeddings with linear projections, denoted as LaMo w/o. MLP. We compare LaMo with LaMo w/o. MLP and DT across all tasks. Mean of seeds with number 0, Shaded area is [μ- (u-o,pp a.p interval, where is the average and or is the standard deviation.
5.5 ABLATIONS
To show contributions of various designs in LaMo, we conduct extensive ablation experiments.

---

Published as a conference paper at ICLR 2024
Linear projections MLPs. In LaMo, we find that simple linear projections could not fully exploit the cross-domain knowledge from language pre- pre-training, and thus our design to replace linear projections with MLPs is critical. As shown in Figure 4 such design exhibits clear improvements compared to linear projections (termed as LaMo w/o. MLP). It is also observed that in Walker2d task, LaMo with linear projections achieves descent scores after few training steps but suffers from overfitting after more training steps, resulting in sub-optimal convergence.
Hopper Medum (18)
Haflchedthh Medun (1%)
Walkerdd Med
Breakost Medium (10%))
Lob5o
Tul (iiekaring
Traaoiigg
DT
Figure 5: Ablation on the effectiveness of LoRA. (1) We involve all the parameters into fine-tuning. denoted as Full Finetuning. (2) We freeze all parameters in Transformer layers and leave out LoRA, denoted as Freezing. We compare LaMo with Full Fineluning, Freezing, and DT.
Comparing LoRA with full finetuning and frozen parameters. Results are given in Figure 5 Though Hansen et al. (2022); Ze et al. (2023a) show that fully finetuning representations for visual RL tasks is better than adopting the frozen pre-trained models, there are works (Ze et al. 20236) showing that finetuning only a small portion of parameters could outperform frozen and fully fine- tuned models, and we observe that in our settings, freezing the pre-trained parameters and adapting with LoRA could not only improve training efficiency but also address the issue of overfitting that occurs in full finetuning. We attribute this to the internal generalizable knowledge within LMs from large-scale pre-training and transfer it to the domain of motion control. We also conduct experi- ments about removing LoRA and only using the frozen pre-trained LM, which also underperforms LaMo that applies LoRA for in-domain task learning.
Language pre-training v.s. visual pre-training, Furthermore, considering observations in Atari in pixel format, investigate whether the visual pre-training could also be helpful for mo- tion control. We replace the pre-trained model with ImageGPT (Chen et al. (020). a Transformer pre-trained the ImageNet dataset (Russakovsky al. 2015). During pre-training, ImageGPT reshapes two-dimensional images into one- one-dimensional vectors after downsampling, and is trained in an autoregressive manner. The results are presented in Table 4 It is observed across Atari tasks that visual pre-training could be a positive initialization for DT, while since LMs better model the sequence structure, there exists a significant gap between LaMo and ImageGPT. This empirical ev- idence further substantiates our hypothesis that proficiency in sequential modeling is the key to unleashing the potential doosin pre-tr models.
Task
Dataset
Ratio
LaMo
DT
LaMo (ImageGPT Pre-training)
Breakout
Medium
0.1
136.9 i 91.1
45.0 : 18.6
$7.7 56.1
Breakout
Medium
473.4 + 195.6
402.8 : 147.6
454.5 : 219.0
Qbert
Medium
0.1
63.6 + 17.2
26.1
14.3
22.5 + 13.7
Qbeit
Medium
79.0 + 13.1
28.9 + 18.3
29.5 + 17.4
Pong
Medium
0.1
114.8
4 8.8
87.1 + 19.7
0.7 = 1.1
Pong
Medium
125.6
6.6
116.1
10.4
116.7 + 9.4
Average
165.6
117.7
113.6
Table 4: Ablation on the effectiveness of sequential language pre-training. We replace the pre- trained model in LaMo with ImageGPT (Chen et al.] 2020), denoted as LaMo (ImageGPT Pre- training). We compare LaMo with LaMo (ImageGPT Pre-training) and DT across Atari tasks. Blue highlight indicates the highest score.
The relationship between language ability and motion control ability. We found that training on language tasks jointly can prevent overfitting and improve overall performance. For the most challenging one among 8 tasks, Kitchen, as Figure (b) shows, we notice that by adding a simple weighted loss during training, the performance no longer drops significantly in the RL. training stage,
8

---

Published as a conference paper at ICLR 2024
and it consistently outperforms the baselines. This suggests that training with language prediction loss as a regularization jointly can retain the advantages of the pre-trained model while learning from limited decision-making dataset. As presented in Figure [aa we show the curve of cross- entropy loss to approximately demonstrate the change of language ability during training, which remains consistent across all tasks. This empirically validates the ability of language models to simultaneously learn two diffferent sequential modeling tasks. However, whether this term could enhance performance in all cases still requires further investigation.
Kitchen Porttal (10%)
Kithen Partal (1
Medum
Kithhen Complete
Kitchen Complece (30%)
LoMo language loes
LaMo WS. language 1006
Lakoo w. Ianguag loes
LaMo WTo. language loss
DT
(a) Language ability. We use
(b) Motion control ability. We set the weight of language loss as zero and positive respectively to demonstrate the significant improvement in results brought by using the auxiliary language
cross-entropy loss on WikiText to
show effects of the language loss
on the model's language ability.
loss.
Figure 6: Ablations to show effects of the language loss for motion control.
Effects of pre-training qualities of LMs. We conduct a systematic study how pre-training qualities of LMs would affect the performance of downstream offline RL agents. We pre-train several GPT-2 models as follows: 1) carly-stopped pre-trained, which is pre-trained on WikiText for 100K training steps. 2) random corpus, which is pre-trained on randomly shuffled WikiText, so that the token prediction totally disturbed. In this way. we aim investigate whether the performance improvement resulting from pre-training is closely related to the nature of the corpus or solely attributed to the network's 's warm-up. We then replace GPT-2 in LaMo with these models and compare the performance in downstream RL tasks. As Figure 7 shows, while these two pre- trained models achieves competitive results against DT, they still fall short comparison with LaMo in certain tasks. This initial observation verifies our hypothesis that model with stronger language ability could perform more effectively when transferring to the field of motion control. Breakear Medium (10%)
Happer ted (15)
Harideriin Medum (15)
Wateeste Medam
(15)
Obet Medium (10%)
Medium
Resererdd (10%)
12ingg 040
Yeinng regiro
Taiin Mex
Shinng mep
Lotoo
Earl stoppp Pro. rointo
Random Co pus
DT
Figure 7: Ablation on the effects of Qualities of Pre-trained Models and Corpus. We train models with the same architecture GPT-2 from scratch, both on WikiText and shuffled WikiText. Compared with these two models and DT, LaMo shows advantages consistently. CONCLUSION
We propose LaMo, an offline RL framework that leverages the pre-trained Language Models (LMs) for low-level Motion control. On sparse-reward tasks, LaMo achieves strong results and surpasses recent strong algorithms CQL, IQL, TD3+BC, and DT: On dewse-reward tasks, LaMo significantly improves DT and closes the gap between value-based methods and DT-based methods. Notably, in low-data scenarios, our method demonstrates powerful few-shot learning ability, which can be attributed to the inductive bias from pre-trained LMS..

---

Published as a conference paper at ICLR 2024
REFERENCES
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1ith International Joins Conference on Natural Language Processing, ACL/I- JCNLP 2021. (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pp. 7319-7322 7328. As- sociation for Computational Linguistics, 2021. doi: 10. acl ong 568
URL
https: /ttss///ooi.oo .rgg /tt.gg//..000 18653/vllll.
M. G. Bellemare, Y. Naddaf, J. Veness, and M. Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-279, jun 2013. doi: 10. 10.1613/jair.399
https:/:::/// /tte:://dooo. http://ooooooo..o///o// wttp:///....11.22..a....... 3312
Prajjwal Bhargava, Rohan Chitnis, Alborz Geramifard, Shagun Sodhani, and Amy Zhang. Sequence modeling is robust contender for offline reinforcement learning. CoRR, abs/23055,14555 2023. doi: 10445550ARRXIV22055555 145500. URL ltttps:///o// http:///dooloog100444555//////
2305.
14550
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choro- manski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez. Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexan- der Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil J. Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lec, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao. Krista Reymann, Michael S. Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong T. Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-2: lssoonlanguage-acction models transfer web knowledge robotic control. CoRR, abs/2307.15818. 2023a. doi: 0044550/ARXIV.2307..2 15818. URL https: //doi. 10. 48550/ 2307. 15818
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Joseph Dabis, Chelsea Finn, Keerthana Gopalakrishnan, Karol Hausman, Alexander Herzog, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Tomas Jackson, Sally Jesmonth, Nikhil J. Joshi, Ryan Julian, Dmitry Kalash- nikov, Yuheng Kuang, Isabel Leal, Kuang-Huei Lee, Sergey Levine, Yao Lu, Utsav Malla, Deek- sha Manjunath, Igor Mordatch, Ofir Nachum, Carolina Parada, Jodilyn Peralta, Emily Perez, Karl Pertsch, Jornell Quiambao, Kanishka Rao, Michael S. Ryoo, Grecia Salazar, Pannag R. Sanketi, Kevin Sayed, Jaspiar Singh, Sumedh Sontakke, Austin Stone, Clayton Tan. Huong T. Tran, Vin- cent Vanhoucke, Steve Vega, Quan Vuong, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, and Brianna Zitkovich. RT-1: robotics transformer for real-world control at scale. In Kostas E. Bekris, Kris Hauser, Sylvia L. Herbert, and Jingjin Yu (eds.), Robotics: Science and Systems XIX, Daegu, Republic of Korea, July 10-14, 2023, 2023b. doi: 10.15607/RSS.2023.. XIX.022...XXX22....... URL /ttps://ddoi.rro
1ww.15567772 / 2023 XIX. 025
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan- Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL httpsss//pprccceeiiggssssss new.eiss...g.................... ctt cc/ppaper/2220//ath 0tw.0//9900701000000000044440680 Abstseaaaabtttttttte.
Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Misha Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch. Decision transformer: Reinforcement learning via sequence modeling. Advances in newral information processing systems, 34:15084-15099 2021.
10

---

Published as a conference paper at ICLR 2024
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and lly Sutskever. Generative pretraining from pixels. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 1691-1703. PMLR, 13-18 Jul 2020. URL http:::// ptttps://prooeedinggs
m mlr.press/v 9/ chen20s html
Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mírac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gau- rav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling cnarrucctioo-inntunnee language models. ArXiv, abs/2210. 114112 2022. URL https: /api. www.stiiscchooooooeoeeeeooocco
ID: 253018554
Jacob Devlin, Ming-Way Weig-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pp. 4171-4186, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 00.18653/v1/11..11 1423. URL https: http:///aaantthooocoooo org/
N19-1423
Danny Driess, Fei Xia, Mehdi S. M. Sajjadi. Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang. Yevgen Chebotar, Pierre Sermanet, Daniel Duckworth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Mare Toussaint, Klaus Greff, Andy Zeng, Igor Mordatch, and Pete Florence. Palm-e: An embodied multimodal language model. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaiï, USA, volume 202 of Proceedings of Machine Learning Research, pp. mlr. www.ssese0000.. /ttp://2222//rrie22... html
PMLR, 2023. URL httpp::/ppooeeeiiiiii
Justin Fu, Aviral Kumar, Ofir Nachum, George Tucker, and Sergey Levine. D4RL: datasets for deep data-driven reinforcement learning. CoRR, abs/2004.07219, 2020. URL https://arxiv.
org/ hrg/abb/2004 07219
Scott Fujimoto and Shixiang Shane Gu. minimalist approach to offline reinforcement learning- In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wort- man Vaughan (eds.), Advances in Neural Information Processing Systems 34: Annual Confer- ence Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pp. 20142-20145, 2021. URL htttt::: http:////ccoccceeos...e.. www.innngs..errge wwwrneurpps.s. Cc cc/paper/ 2021/hash a8 http://ashhaallscd55eeee 7 aat.aaa0000..a00000222223---eeee Abst html
Scott Fujimoto, David Meger, and Doina Precup. Off-policy deep reinforcement learning without exploration. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.). Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, Cali- formia, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2062- PMLR, 2019.
mlr www.:mmw....p....mmm. wpwpss/vv'. fu wwt.iimmooo..o 9a. html
Abhishek Gupta, Vikash Kumar, Corey Lynch, Sergey Levine, and Karol Hausman. Relay policy learning: Solving long-horizon tasks via imitation and reinforcement learning. In Leslie Pack Kaelbling, Danica Kragic, and Komei Sugiura (eds.), 3rd Annual Conference on Robot Learn- ing, CoRL 2019, Osaka, Japan, October 30 November 1, 2019, Proceedings, volume 100 of Proceedings of Machine Learning Research, pp. 1037. PMLR, 2019. URL http: //proceedings. . mlr. preess/vlui pttpas//100//gppaaaaaaaaa atppgpaa.aaaaa..mmmmaaaa
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML Stockholmsmässaan Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1856-1865 PMLR, 2018. URL http: httt:::/pproceedingg.. m wtr.press/v///heee http/////aaaarra Bb. html

---

Published as a conference paper at ICLR 2024
Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi, and Jimmy Ba. Mastering atari with dis- crete world models. arXiv preprint arXiv:2010.02193, 2020.
Nicklas Hansen, Zhecheng Yuan, Yanjic Ze, Tongzhou Mu, Aravind Rajeswaran, Hao Su, Huazhe Xu, and Xiaolong Wang. On pre-training for visuo-motor control: Revisiting a learning-fron scratch baseline. ICML, 2022.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint WrXiv:1606008415 2016.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In The Tenth Inter- national Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022.
OpenReview.net, 2022.
Kaizhe Hu, Ray Chen Zheng, Yang Gao, and Huazhe Xu. Decision transformer under random frame dropping. In The Eleventh International Conference on Learning Representations, ICLR 2023, Ki- gali, Rwanda, May 1-5, 2023. OpenReview.net, 2023a. httpptt:///eeeeeeeeeeee net./ pdd cdf7id=Nn22v44470
Shengehao Hu, Li Shen, Ya Zhang, and Dacheng Tao. Prompt-tuning decision transformer with preference ranking. CoRR, abs/2305.09648, 2023b. doi: 1048550/ARKXV.2305006666 URL https://doo /ttt::::/doi..og/1.... 48550 /0/5550//xxiv/ 2.05.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero- shot planners: Extracting actionable knowledge for embodied agents. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pp. 9118-9147. PMLR, 2022. URL fetps://proceddinss..LL.. www.spressssseec http:////w1///uua222.aa.a... 2htmmml
Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse. Jornell Quiambao, Peter Pastor, Linda I Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu. Do as can, not as say: Grounding language in robotic affordances. Karen Liu, Dana Kulic, and Jeffrey Ichnowski (eds.), Conference on Robot Learning, CoRL 2022, 14- 18 December 2022, Auckland, New Zealand, volume 205 of Proceedings of Machine Learning Research, pp. 287-318. PML......... 2022. URL ichter23an 2aaar23..00 html
mlr press/v205/
Michael Janner, Qiyang Li, and Sergey Levine, Offline reinforcement learning as one big sequence modeling problem. In Advances in Neural Information Processing Systems, 2021.
Kevin Jarrett, Koray Kavukcuoglu, Marc Aurelio Ranzato, and Yann LeCun. What is the best multi-stage architecture for object recognition? In 2009 IEEE 12th International Conference on Computer Vision, pp. 2146-2153 2009. doi: 10. 111/0/CC..2099555544444
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey, ACM Computing Surveys, 54(10s):1-41, jan 2022. doi: 10.1145/3505244, htRp::/:::///// //doi. owg/olggg///oo 100 1.14522
Takeshi
Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke language models are zero-shot reasoners. In NeurIPS, 2022.
Iwasawa. Large
URL
http: http://papees.npp..cc/pppp http:/pppppp...lsespppppeeep/e/e html
Ilya Kostrikov, Ashvin Nair, and Sergey Levine. Offline reinforcement learning with implicit q- learning. In The Tenth International Conference Learning Representations, ICLR 2022, Vir- tual Event, April 25-29, 2022, OpenReview.net, 2022. URL htttps:/ httttp:://openeview..o net/
forum? farum?id=68mm 68 97.0E8
12

---

Published as a conference paper at ICLR 2024
Aviral Kumar, Aurick Zhou, George Tucker, and Sergey Levine. Conservative q-learning for offline reinforcement learning. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florima Balcan, and assun--Ter Lin (eds.), Advances in Neural Information Processing Systems 33: An- nual Conference on Neural Information Processing Systems 2020, NewrlPS 2020, December 6-12, 2020, virtual, 2020.
ngs neur ips CC cct.papperr/22//aaaiii 182220011625555533211112211 att.aaaassssss0ooaaatttaatta Abstract html
Sascha Lange, Thomas Gahel, and Martin Riedmiller. Batch Reinforcement Learning, pp. 45-73. Springer Berlin Heidelberg, Berlin, Heidelberg, 2012. ISBN 978-3--4--2277655--3 doi: 10.1007/ 978-3-642-27645-3. 2. URL 2tt.a///.//////..o...//0/0000002
Michael Laskin, Luyu Wang, Junhyuk Oh, Emilio Parisotto, Stephen Spencer, Richie Steiger- wald, DJ Strouse, Steven Stenberg Hansen, Angelos Filos, Ethan A. Brooks, Maxime Gazeau, Himanshu Sahni, Satinder Singh, and Volodymyr Mnih. In-context reinforcement learning with algorithm distillation. In The Eleventh International Conference on Learning Represen- tations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL https: // //openreview ew...wwww....................... peet:pddfid=hyy0alt.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter- prompt tuning. In Marie-Francing Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language P Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic. 7-11 November, 2021, pp. 3045- 3059. Association for Computational Linguistics, 2021. doi: 10. 10.118653/1/22221EEE 243. /88
emnlp 1ta.lpppmai...33..
Sergey Levine, Aviral Kumar, George Tucker, and Justin Fu. Offline reinforcement learning: Tutorial, review, and perspectives on open problems. CoRR, abs/2005.01643, 2020. URL
https::/arxa
hwtp://///20 20t.a///200...0 01643
Shuang Li, Xavier Puig. Chris Paxton, Yilun Du, Clinton Wang. Linxi Fan, Tao Chen, De- An Huang, Ekin Akyürek, Anima Anandkumar, Jacob Andreas, Igor Mordatch, Antonio Tor- ralba, and Yuke Zhu. Pre-trained language models for interactive decision-making. In NeurlPS, 2022. URL WtRLhttp: /ttp:////appppappp nips. http:/pppe_.flllsssppaeee/////// ott.:aaeeeots.eaeee2eoooeoo-eoo-
html
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Join! Conference on Natural Language Processing (Volume I: Long Papers), pp. 3582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ httpss//aclanthooooog...o org http:/22222..a1111o11 ael- 333
Kevin Lin, Christopher Agia, Toki Migimatsu, Marco Pavone, and Jeannette Bohg. Tex12motion: From natural language instructions to feasible plans. CoRR, abs/2303. 2023. doi: 10.
48550/ARXIV.2303.22 12153. ht.ps::::: httttt:::/ddo..rogg
www885555xxxee 2303 12153
Hao Liu and Pieter Abbeel. Emergent agentic transformer from chain of hindsight experience. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Re- search, pp. 21362-21374. PMLR, 2023. URL httts:://pxoceeeingg...... .r....rsss v202 1ww.a........... html
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learn- ing. In NewrIPS, 2022. URL http:ttttp////teeetettt itwissiniiiiie nips ec cw/paper fitps//apeseeeeae/2///222
htmi
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 26. 2017, Conference Track Proceedings. OpenReview..et, 2017. htthtt //openrevien ew. net. wwr.://.////o//oo/omm/ wtw.rrummmiiyyyyooee
13

---

Published as a conference paper at ICLR 2024
OpenAL GPT-4 report. CoRR, abtp///3...07744..200000 doi: 1048550/AR2XV.200.00777 URL https:://t...ooeteee...o /ttp://445555///1.222223
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022. URL
wwwppppppr.......ccppppp..s..epe www.:ippppccpicc.. fites/paper//2222 http://222hhhhhah///e - Cewerractt-ooffeeenceee html
Rafael Figueiredo Prudencio, Marcos R. O. A. Maximo, and Esther Luna Colombini. A survey on offline reinforcement learning: Taxonomy, review, and open problems. IEEE Transactions on Newral Networks and Learning Systems, pp. 1-0, 2023. doi: 10. (09//n/ls..03333351121 www.aii..rr.oooooooooo wrgg/0 10. 110982Ftnn1s s. 2023. 3250269.
URI.
https://doi.
Alec Radford and Karthik Narasimhan. Improving language understanding by generative pre- training. 2018. URL http:://ppp/pppp... sttaeet ar..hhllr..r... itr.acooo.....oorrrrrripiii:i::a
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.
Language models are unsupervised multitask learners, 2019. URL https://api.
:6:00225:310
Scott E. Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay. Jost Tobias Springenberg, Tom Recles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. A generalist agent. Trans. Mach. Learn. Res., 2022, 2022.
Machel Reid, Yutaro Yamada, and Shixiang Shane Gu. Can wikipedia help offline reinforcement learning? CoRR, abs/2201.12122. 2022. URL
/ta.bb//2211 12122
Martin A. Riedmiller. Neural fitted Q iteration first experiences with data efficient neural reinforcement learning method. In João Gama, Rui Camacho, Pavel Brazdil, Alípio Jorge, and Lufs Torgo (eds.), Machine Learning: ECML. 2005, 16th European Conference on Ma- chine Learning, Porto, Portugal, October 3-7, 2005, Proceedings, volume 3720 of Lecture Notes in Computer Science, pp. 317-328. Springer, 2005, doi: 10. 10.1007/11564096) _322. URL https://doi.coo waaloi.org//...m 111.0..1.000111150..0..
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJC)), 1 115(3):211255 2015. doi: 10. 100/// 1263-015-0816-y.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, fls/1707.0637,, 2017. URL http:://axxvv..rgg//////
707. 06347
Takuma Seno and Michita Imai, d3rlpy: An offline deep reinforcement learning library. Journal of Machine Learning Research, 23(315):1-20, 2022. URL
22-0017 html
Sheng Shen, Alexei Baevski, Ari S. Morcos, Kurt Keutzer, Michael Auli, and Douwe Kiela. Reservoir transformers. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/IJCNPP 2021, (Volume 1: Long Pupers), Virtual Event, August 1-6, 2021, pp. 4294-4309. Association for Computational Linguisties. 2021. doi: acl -ong. 331
URL https
7/doi. argi..org/1 10. www../.663vvi/1
Yujin Tang, Wenhao Yu, Jie Tan, Heiga Zen, Aleksandra Faust, and Tatsuya Harada. Saytap: Lan- guage to quadrupedal locomotion. CoRR, hbs/22006005880 2023. doi: 10.48555//RXII223066 07580. URL (RIthttss:: :/do:/.////.../
org. 48550 a http://wwwwwv........ 07580
14

---

Published as a conference paper at ICLR 2024
Denis Tarasov, Alexander Nikulin, Dmitry Akimov, Vladislav Kurenkov, and Sergey Kolesnikov. CORL: Research-oriented deep offline reinforcement learning library. In 3rd Offiine RL Work- shop: Offline RL as a "Launchpad", 2022. http:::/ net http://wwwweeeeoorumeaaaaaa forum?id=
SyAS49bBcv 9bBcv
Denis Tarasov, Vladislav Kurenkov, Alexander Nikulin, and Sergey Kolesnikov. Revisiting the minimalist approach to offline reinforeement learning. CoRR, abx/2305 2023. doi: 10. 48850/ARXXV.230000008.
ht.tps::::/ //doi .ww.oio.rgr1... 20210.488500 48550/ http///0.44550//x11..22.22 www.aa.www.........m 09836
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control.
In 2012 IEEE/RSJ International Conference Intelligent Robots and Systems, pp. 5026- 5033,
2012. doi: 10.1109/IROS, 20W..O.........686.1000
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In Ma- rina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine mmlr.
Learning Research, pp. 10347- PMLR, 2021. URL http::/proccediii press/v133ttovroz2220 html
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. 139711 2023a. doi: 10.48550/ARXIV.2302.11 https:://oi..... wtw.iioorg11... 48550/ 2302 13971
URL
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshom, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CoRR, 0atp//377.0928... 2023b. doi: 10.48550ARRXI..230700022 httpss: www.oii.rgg..... /tctp:://aoi.org/00..egg 48550 (ww885550/aiii http//aaxvvvv.. 2307 09288
URL
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. $998-6008, 2017. URL / iew.:rooocoeeddn.........o..... htcppps..ccppeeee wwtpc/ppppp//////aaaa..a. Abst html
Sid Wang, John Nguyen, Ke Li, and Carole-Jean Wu. READ: recurrent adaptation of large trans- formers. CoRR, abs/2305. 153488 2023a. doi: 1044850/ARRXIV.205555555 URL h
Yen-Jen Wang, Bike Zhang, Jianyu Chen, and Koushil Sreenath. Prompt robot walk with large language models. CoRR, 2023b. doi: 10.48550/ARXIV.230999 .09969. URL httpps:// /oi.
org/10. 48550/ 2309 ....399.99996 09969
Zhendong Wang. Jonathan J. Hunt, and Mingyuan Zhou. Diffusion policies as an expressive policy class for offline reinforcement learning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023c. URL httpss://ppenevite www.:wwweeeeeeeeee..ee..eeeeeee net/pdffidd w-w.
15

---

Published as a conference paper at ICLR 2024
Jimmy Wu, Rika Antonova, Adam Kan, Marion Lepert, Andy Zeng, Shuran Song, Jeannette Bohg, Szymon Rusinkiewicz, and Thomas A. Funkhouser. Tidybot: Personalized robot assistance with large language models. CoRR, abs/2305.0565$ 2023. doi: 10.48500ARXXV.230.00065 URL
https:://dos
.rgg /0. 48550 / 2305. 05658
Zhihui Xie, Zichuan Lin, Deheng Ye, Qiang Fu, Yang Wei, and Shuai Li. Future-conditioned unsu- pervised pretraining for decision transformer. Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.). International Confer- ence on Machine Learming, ICMI. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 38187-38203. PMLR, 2023. URL html
ht http::://poceeeingss http:///s......ppe....eeeeeeeeee
Mengdi Xu, Yikang Shen, Shun Zhang. Yuchen Lu, Ding Zhao, Joshua B. Tenenbaum, and Chuang Gan. Prompting decision transformer for few-shot policy generalization. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvári, Gang Niu, and Sivan Sabato (eds.), International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learming Research, pp. 24631-24645. PMLR, 2022. mlr www.sppres/e/uuueepeepep html
URL https::/ppocceeiiiee
Taku Yamagata, Ahmed Khalil, and Raúl Santos- Modros-Rddriguez. Q-learning decision transformer: Leveraging dynamic programming for conditional sequence modelling in offline RL. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference Machine Learning, ICML 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 Proceedings of Machine Learning Re- search, pp. 38989 39007. PMLR, 2023. URL htttps://ppet htttp:://prceeeiigg... press/ html
Yanjie Ze, Nicklas Hansen, Yinbo Chen, Mohit Jain, and Xiaolong Wang. Visual reinforcement learning with supe-sspe 3d representations. IEEE Robotics and Automation Letters (RA-L), 2023a.
Yanjie Ze. Yuyao Liu, Ruizhe Shi, Jiaxin Qin, Zhecheng Yuan, Jiashun Wang, Xiaolong Wang, and Huazhe Xu. H-index: Visual reinforcement learning with hand-informed representations for dexterous manipulation. NeurlPS, 2023b.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efffcient fine-tuning. In The Eleventh Inter- national Conference Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
OpenReview.net, 2023a. /RRL:attts:///peeeettt
http:/..eeeeeeeo.. net/
yiy
Renrui Zhang. Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init at- tention. CoRR, abs/2303. 16199, 2023b. doi: (0.48550/ARXI223333..rr 16199. URL https:
//doi. 48550/ aa..50/////v..... 2303.
16

---

Published as a conference paper at ICLR 2024
A LIMITATIONS
is important to acknowledge the limitations of our work. On dense-reward MuJoCo tasks, we find that CQL is very competitive to LaMo, showing that value-based methods are still very strong in offline RL. Besides, the auxiliary language prediction loss in LaMo has only shown its advantage in very low-horzion tasks, e-g., Kitchen, while in other tasks, it serves the purpose of preserving language capabilities but does not increase the performance significantly. How to better leverage the language reasoning ability to further help offinn RL thus future direction. Lastly, limited by computational resources, we have not looked into utilizing larger language models (Touvron et al., 2023a b Chung al. )022), and we hope our work could motivate the community explore further applications of LLMs in offline RL.
B IMPLEMENTATION DETAILS
Codebase. Our codebase is mainly based on Chen et al. (2021) (nttps://github... com/ trans Eo rmer) and Hu et al. (2023a) (ht / / com/ h DeFog) with minimal modification on implementation details of original models. All codes of the baselines are directly from Tarasov al. (202 CORL) and Senoo Imai http://ghttuubb.cocsreee///eaaaa
Our official
code is released at httpt:tp... /ttpassssllltateee/tttttt.//////
Network architecture for LaMo. Except for the simplest task Hopper, where the observation space and action space of which is of only 11 and dimensions respectively, we replace linear projections after the input with multi-layer perceptrons Mt M,, Ma. and GELU (Hendrycks & Gimpel 2016)
the activation function. With timesteps embedding w()), the embeddings of Ra, S1, at
u u(xe)= = GELU(W/ x;) + w(t), E (Ř, a.).
As for the Transformer, We mainly adopt the architecture of GPT-2 small model, with 124M param- eters. The number of Transformer layers is 12, the number of attention heads is 12, and the hidden size 768. Specifically, for Kitchen, when training on Comp et (30%) dataset and P Part itai (100%) dataset, empirically find that using GPT-2 medium of which the number of layers is 24 and the hidden size is 1024, could enhance the performance.
C DATASET DESCRIPTIONS
For MuJoCo and Atari, we mainly study the Medi um dataset, generated by an agent trained using the SAC (Haarnoja et al.) 2018) algorithm, which was terminated prematurely. The utilization of this dataset aimed at minimizing variations quality among different trajectories. The Atari datasets are taken from d4rl-atari Ihttps://githun com takuseno/ d4rl Marl-ater d4rl-atari).
For Kitchen, we conduct experiments on both the Complete and the Partial dataset. In the Complete dataset, the robot performs all the required tasks sequentially, while the Partial dataset is composed undirected data and ensures that a subset of the dataset can successfully solve the task.
For Reacher2d, which does not belong to D4RL, we train an agent of medium performance ( av- erage normalized score of 36.0 over 50 episodes) by PPO algorithm (Schulman et al.) (017). and then generate trajectories composed of 50 episodes simulated by that agent, referred to as Med um dataset.
To look into the low-data regime, we randomly downsample trajectories from the original dataset for given sample ratio.
D TASK DESCRIPTIONS
Halfcheetah (MuJoCo):: The goal is to make the cheetah move forward (right) as fast as possible by applying torque to its joints.
https::/hugginggfcce....
medi um
17

---

Published as a conference paper at ICLR 2024
Hopper (MuJoCo): The goal is to achieve forward (right) motion through controlled hops.
Walker2d (MuJoCo): The goal is to coordinate movements to achieve forward (right) direction.
Reacher2d: The goal is move the robot's end effector (fingertip) close to randomly spawned target.
For Reacher2d, We compute the average performance over the last 12.5K training steps out of a total of 37.5K training steps with evaluations conducted every 2500 training steps.
Kitchen: The objective in each task is to interact with items reach specific desired configuration.
Breakout (Atari): Players control paddle to hit ball at a brick wall, aiming to break down. Players have five lives.
Qbert (Atarí): The objective to change the color cubes on the pyramid to match "destina- tion' color by hopping each cube while avoiding obstacles.
For Breakout, on which algorithms converge fast, we compute the average performance over the last 10K training steps out of total of 50K training steps with evaluations conducted every 2500 training steps.
Pong (Atari): Players compete to deflect the ball away from their goal and into the opponent's 's goal using paddles.
8 In Figure provide visualizations of each task.
(a) Kitchen
(b) Reacher2d
(c) Hopper
(d) Halfcheetah
DBC
(c) Walker2d
(f) Breakout
(g) Qbert
(h) Pong
Figure Visualization of Tasks from domains: Kitchen, MuJoCo, and Atari.
E TASK SCORE NORMALIZATION
Task Name
Random Score
Expert Score
Kitchen
Reacher2d
0
100
Hopper
-0.33
3234.3
HalfCheetah
-28022
12135.0
Walker2d
1.6
4592.3
Breakout
1.7
31.8
Qbert
163.9
455.0
Pong
-20.7
9.3
Table 5: Scores used for normalization. Scores of each task are linearly normalized by the corre- sponding random score and expert score.
18

---

Published as a conference paper at ICLR 2024
The scores we present normalized using the formula:
score random score
normalized score
x 100;
expert score random score
where the random scores and the expert scores provided in Table 5 that 100 represents the expert score and and 0 represents the score of random policy, following the protocols of et al. (2020) and Hafner (2020).
F HYPERPARAMETERS
In Table 6 and Table 7 we list the task-specific hyperparameters and task-agnostic hyperparameters, respectively. More details can be referred to http://////gtttttbccocoososc/
2023
Task Name / Variable
Leaming Rate
Weight Decay
Context Length
Return-to-go
Training Stops
Kitchen
x 10
10
20
3,4,5
100K
Reacher2d
1x 10
1xX 10
70,76 76
100K
Hopper
1 10
x 10
20
1800, 2200,, 3800
100K
HallCheetaht
1x 10
1 10
20
6000), 5000), 12:00
100K
Walker2d
1x
x
20
2500, 4000, 5000
100K
Breakout
1x
10
30
90, 120
501K
Qbert
1x
14000
100K
Pong
3 10
1
30
10, 20
100K
Table 6: Task-Specific Hyperparameters.
Variable
Value
Number of Transformer Layers
12
Number of MLP Layers (Kitchen & MuloCo)
3
Number of CNN Layers (Atari)
3
Number of CNN Channels (Atari)
32, 64, 64
Dimension of CNN Kemels (Atari)
8,4,3
Hidden Dimension
768
LoRA Rank
(Kitchen & MuloCo), (Atari)
Batch Size
(Kitchen MuJoCo), 128 (Atari) 0.1
Dropout
Table 7: Aano-tgi Hyperparameters.
We follow the common practice of DT that uses multiple values rtg, and reports the best one, as shown in ht/tts:://gtthut com/ ktt///ll//cc////e -ion-ttaasfoonnt The values we adopt are validated by experiments. And one of the reasons that for different sampling ratio, the best rtg would be different.
G MORE RESULTS
Language ability test of models. With the prefix prompt Hello, my name Tom., answers of different models are:
GPT-2: I'm not very funny anymore, and I'm not a comedian, and neither am guy. Oh, I'd like to do some comedy so that we could work together and actually work out some pretty big Reception = Critical response = = number of reviewers praised Tom McNeill's performance, commenting that
Early-ended Pre-trained: Andriaki.
Random Corpus: The, was of the from on to in the on the to for, the, and to that =.. by of for. that the that the
Results on datasets with varying qualities. Tables and present testing results when the models trained datasets of different qualities, Medi um Expert. and Medi waeilum-Rllaaa ay. LaMo shows competitive performance over the baselines, especially on Medi um-Replay (1%) datasets.
Results more tasks. In Tables 10 and I present additional results for tasks in Atari and MuJoCo. Specifically, in the Freeway and Asterix tasks, LaMo demonstrates significant advance- ments over DT and Wiki-RI., effectively narrowing the performance disparity with vasue-based
19

---

Published as a conference paper at ICLR 2024
Tusk
Dutaset
Ratio
Ours
DT
CQL
IQL
TD3+BC
Hopper
Med- Reelay
0.01
29.4
29.3
1.3 = 1.3
16.3 1:
19.7
Halfcheetah
Med Replay
0.01
14.6
4.5
10.0 :: 2.6
-0.2 0.2
17.8 1: 4.5
7.4 2.7
Walker2d
Mod-Replay
0.01
28.1 = 1.0
12.4 = 1.3
31.3 2.6
30.2 4: 1.2
11.0 1.3
Average
24.0K 1400)
17.2
10.8
21.4
12.7
Table 8: Normalized score on Medium-Replay Dataset. Blue highlight indicates the highest score, orange highlight indicates the second -ooondt score.
Dataset
Wik-RRL
Hopper Halfchectah Walker2d
Med-Expert
Med-Expert Expert
92.2 0.7
86.8
91.8
91.6
36.7
90.7
55.2
Med
Med-Experr Expert
108. 3 1 1.1
108.1
108.9
108.8
109.6
110.1
107.5
1-
Average
103,50135)
100.8
103.9
101.9
95.9
99.5
71.7
Table 9: Nomalized score on Medium-Exper Dataset.
Values without standard deviations are
taken from Kostrikov et al. (2022).
based methodologies. Furthermore, in the Ant task, LaMo surpasses the baseline scores, indicat- ing a notable improvement in performance.
Task
Dataset
Ratio
Ours
DT
Wiki-RL
CQL
BCQ
NFQ
Freeway Asterix
Medium
0.1
83.0 = 1.7
70.1 4.0
72.0 + 3.9
101.2 1.7
79.1 6.1
56.2 19.1
Medium
2.8 1.0
0.7 0.7
1.9 = 0.5
-0.1 0.0
Average
42.9(1215)
35.4
36.2
52.7
40.5
28.1
Table 10: More tasks in Atari. We conduct experiments and report normalize scores on another games, Freeway and Asterix, in the domain of Aturi, to present the remarkable improvement of LaMo over DT and Wiki-RL.
comparative analysis is presented between method (LaMo) and the recent powerful diffusion policy, Diffusion Q-learning (Diffusion-QL) (Wang et al.] 2023c), across three distinct tasks. Notably, LaMo outperforms Diffusion-QL. in all three tasks, with particularly remarkable results the low-data regime.
Comparison with Diffusion-QL. In Table 12
Overfitting issues of CQL. In Kitchen tasks, CQL faces the issue of overfitting, causing notable drop in its performance, as presented in Figure
Results based top-k metric. We provide the experiment results of Kitchen using top-k metric, which calculates the average scores over the h checkpoints with the highest testing scores. As is shown in Table 13 LaMo still outperforms other DT-based methods and value-based methods.
Effects of the model size. In Figure TO the influence of language model size on the performance of LaMo depicted. The results indicate that GPT2-small already achieves satisfactory perfor- mance. Additionally, in specific tasks, GPT2-medium demonstrates a slight edge over GPT2-small, showcasing incremental improvements with increased model size.
Hyperparameter tuning for baselines. As demonstrated in Figure II we conduct hyperparame- ter tuning for both DT-based and value-based baselines. As for Wiki-RL, we sincerely follow the authors' statements in their paper (Reid et al. 2022) Section 4.1, to use the same learning hyperpa- rameters DT. These results verify the baselines' performance reported in our study.
Number of parameters of our method and baselines. We have presented Table [4 listing the trainable parameter sizes of LaMo alongside various baselines. For value-based methods, section and the ablation study in section 4.5 in Tarasov et al. (2023) demonstrate that deepening the network structure does not yield improvements, and our experiments shown in Figure [2] reveal similar trend for increasing the network's 's width. Thus adhere to widely accepted model sizes, which are much smaller than Transformer- based methods. Besides, it is important to emphasize that simply increasing the size of the Transformer will not boost the performance, as shown in the results of Reid et al.) (2022) and our ablation studies. Moreover, although our method involves a relatively large model, the number of trainable parameters is fairly small, which is 3.5M, comparable with 7.3M
20

---

Published as a conference paper at ICLR 2024
Task
Dataset
Ratio
Ours
DT
Wiki-RL
CQL
IQL
TD3+BC
Ant
Medium
87.8
87.2
+ 4,9
97.5 = 5.4
Ant
Medium
0.01
86.3 = 6.2
77.8 = 4.8
65.9 = 6.1
76.2 = 10.5
46.4 4: 5.5
3.4
Ant
Medium
0.005
90.2 3.7
76.5 4.9
71.3 5.8
64.5 6.3
65.8
1.2
Average
88.1()9%)
69.2
Table 11: More tasks in MuJoCo. We conduct experiments and report normalized scores other environment, Ant, in the domain of MuloCo, to present the competitive performance of LaMo.
Task
Dataset
Ratio
LaMo
Diffusion- QL
Hopper Hopper
Medium
0.005
57.0
13.2
Medium
0.01
52.0
35.7
Hopper
Medium
0.1
73.7
68.4
Halfcheetah
Medium
0.005
39.0
36.5
Halfcheetah
Medium
0.01
40.6
34.8
Halfcheetah
Medium
0.1
42.1
46.8
Walker2d
Medium
0.005
66.9
32.3
Walker2d
Medium
0.01
74.5
44.7
Walker2d
Medium
0.1
70.4
55.2
Average
57.4
40.8
Table 12: Comparing LaMo with Diffusion-QL. We compare LaMo with the recent strong base- line, Diffusion Q-learning (Diffusion-QU) (Wang et al. 2023ch) across 3 different tasks.
Kitchen Partial (100%)
Kitchen (1.))
Kitchen Partial (1%)
torking
rana stops
baiing saap (0)
COL Ir=3e-4
COL Ir=3e-3
COL Ir=3o-5
Figure 9: Normalized score of CQL. CQL consistently suffers from severe overfitting in Kitchen tasks with various learning rates.
Task
Dataset
Ratio
LaMo
DT
Wiki-RL
CQL
IQL
TD3+BC
BC
Kitchen
Partial
56.4
59.8
39.9
42.5
56.3
17.1
33.8
Kitchen
Complete
73.3
600.
35
45.7
49.4
40.3
33.8
Average
64.9(18%)
60.2
37.5
44.1
52.8
28.7
33.8
Table 13: Normalized score of Kitchen based on the metric of top-3 performance. We present the average normalized scores over the best 3 checkpoints in each run. The results agree with those in D4RL (uu al. 2020). Blue highlight indicates the highest scores, and red numbers represent the improvement of LaMo over DT.
Kitchan Partial (100%)
Kilchen Complete (30%)
Kitchen Completa (50%)
nommlizzdd score
7akiing stops (0)
Iran mian (X)
brining staps 360
LaMo (GPT2-medium)
LaMo (GPT2-small)
Figure 10: Effects of GPT-2 model size. We adopt two language model backbones of different size, GPT2-small' and GPT2-medium GPT2-small already shows satisfying performance. In Kitchen Complete (30%) and Kitchen Partial (100%), GPT2-medium slightly exceeds GPT2-small.
co/gpt2
lttpps:/ http:ss:///gggnggac.ee.. ctlaaace..cc//peee cot medium
21

---

Published as a conference paper at ICLR 2024
Hopcer Medium (1%)
Wollordd Medium(19))
Kitchen Complete (30%)
Ober Midium (105)
del lic
Mecker
(30%))
(198)
Figure 11: Baseline hyperparemeter tuning. We tune the hyperparameters of two strong baseline methods, DT and CQL. We compare the experiment results with different learning rate (Ir) and weight decay (wd) across 3 domains, MuJoCo, Kitchen and Atari.
of Decision Transformer. So the difference of the number of parameters between Transformers and value-based methods does not compromise the fairness the comparisons of performance.
Algorithms
Total Parameters
Trainable parameters
LaMo
128M
3.5M
Wiki-RL
125M
125M
DT
7.3M
7.3M
CQL
0.41M
0.41M
IQL.
0.33M
0.33M
TD3+BC
0.25M
0.25M
Table 14: Number of parameters of each algorithms. We conduct a comparative analysis of the total parameter sizes and trainable parameter sizes of LaMo and various baseline.
Hopper Medium (100%)
Badkeer2dMddiumn (100%)
Halccheethh Medium (100%)
41 buring stope
ring stepe
triiing stapo
IQL hidden size=256
IOL hidden size=512
IOL hidden sizem 1024
Figure 12: Effects of the width of networks on the performance of value-based methods. We train IQL with different hidden size across 3 different tasks, demonstrating that increasing the width does not yield improvements.
22

---

Published as a conference paper at ICLR 2024
H THE SURPRISING EFFECTIVENESS OF FROZEN RANDOM WEIGHTS WITH LORA
To show the help of the pre-trained weight of the LLM,, actively ablation experiments. The only difference is that the weights of the Transformer layers are randomly initialized according to the intialization of GPT2 (Radford et 2019). We present the results in Table [5 and Table We observe that LoRA harnesses the potential of the pre-trained model, enabling it to outperform DT significantly both sparse-reward and dense-reward tasks. Besides, the pre-trained model out- performs the randomly initialized model noticeably.
Although we do not utilize -ellure-aligned functor for the two domains, language and motion control, in LaMo's pipeline, which could be a promising future work, we hypothesize that freezing most parameters and training on language tasks simultaneously could potentially force the feature alignment between two domains, and thus the general knowledge of sequential modelling could be transferred. Besides, Aghajanyan et al. (2021) 's theoretical analysis also backs our claims, which shows the connection between intrinsic dimension and generalization.
Task
Dataset
Ratio
LaMo
LaMo w/o. PT
DT
Kitchen
Partial
0.01
11.6 + 3.0
9,8 2.6
0.9 + 0.9
Kitchen
Partial
0.1
35.1 5.2
24.8
4.3
22.6 + 6.8
Kitchen
Partial
I
46.6 +
43.6 + 7.4
33.8 + 14.5
Kitchen
Complete
0.3
45.9
2.9
42.4 4.7
31,5 + 4.5
Kitchen
Complete
0.5
50.6
6.1
56.8 4.5
36.6 5.1
Kitchen
Complete
I
64.2
5.3
51.8 3.7
52.8 1 3.7
Reacher2d
Medium
0.1
12.4
3.8
6.8 3.6
2.3
1.5
Reacher2d
Medium
0.3
31.2
7.6
20.2 5.4
6.4 1 2.6
Reacher2d
Medium
1
33.0
8.3
28.2 8.4
22.8 1 6.0
Average
36.7
31.6
23.3
Table 15: Ablation on the effectiveness of pre-training for 2 sparse-reward tasks. We replace the pre-trained LM in LaMo with randomly initialized model of the same structure, denoted as LaMo w/o PT. We compare LaMo with LaMo w/o. PT and DT. Blue highlight indicates the highest score.
Task
Dataset
Ratio
LaMo
LaMo w/o. PT
DT
Hopper
Medium
0.005
$7.0 7.1
43.5 + 2.8
35.8
Hopper
Medium
0.01
52.0 =
46.0
41.9 5.2
Hopper
Medium
0.1
73.7 3.5
57.7 +
57.3 3.8
Hopper
Medium
74.1 5.3
64.5 +
60.9 3.3
Halfcheetah
Medium
0.005
39.0 1.6
39.2 + 1.3
22.4 5.2
Halfcheetah
Medium
0.01
40.6 1.3
40.6 - 1.4
29.6 4.8
Halfcheetah
Medium
0.1
42.1 = 0.6
41.4
41.7 0.8
Halfcheetah
Medium
42.5 1 0.4
42.8 1 0.3
42.6 0.5
Walker2d
Medium
0.005
66.9 5.4
$7.0 5.8
16.7 4.8
Walker2d
Medium
0.01
74.5 1
74.2 1 1.9
38.9 9,3
Walker2d
Medium
0.1
70.4 1 4.2
70.9
4.0
70.2 7.5
Walker2d
Medium
73.3 3.1
69.8 1 9.3
70.2 4.3
Average
44.0
Table 16: Ablation on the effectiveness of pre-training for 3 dense-reward tasks. Blue highlight indicates the significantly highest score.
Another interesting phenomenon is that even without pre-training, the model, with enlarged model size, deepened embeddings and LoRA adapting techniques, could still reach higher performance than original DT. [hen et al. (202T) observes the same results, while they use randomly initialized frozen layers as cheap way deepen their model and achieve better performance, albeit their application is in the field of natural language. Jarrett et al. (009)) also proposes Random Filter for object detection, and it has decent performance as feature extractor.
Although we have not looked into the effectiveness of random initialized Transformers deeply, based on the experiment results and previous works, hold belief that freezing random weights with minimal adaption is promising approach.
23

---

Published as a conference paper at ICLR 2024
1 EXPLORATORY FINDINGS AND ADDITIONAL DISCUSSION
1.1 LARGER LANGUAGE MODELS
In this work, we treat the size of GPT-2 as hyperparameterss and observed that in most cases, they did not exhibit significant differences in performance. Furthermore, we explore the pre-trained model LLaMA and apply various techniques, including LLaMA adaptor (Zhang et al., 2023b). lora, and joint training. However, we find that none of these approaches demonstrates substantial improve- ments over the baseline. Constrained by computational resources, we are unable to delve further into this area, but it could serve as intriguing avenue for future research endeavors.
1.2 OTHER EMBEDDING METHODS
We have tried another embedding method, which is inspired by RT-2 (Brohan et al. 2023a). We are intended stimulate the power of pre-trained LM in way that is closer to processing language. We discretize each dimension of each vector into 512 bins after normalization and thus get the index sequence. Indexes are understood by the model as tokens, and the corresponding embeddings of indexes are set independently for returns-to-go, states and actions, while all of them are intialized as pre-trained embeddings for those digit tokens.
While this approach works well in situations with 100% data availability, it exhibits poor perfor- mance in low-data regime, even when various techniques are employed. We suspect that the limita- tion may stem from the lack of generalizability in this method. In the future, addressing this issue could involve exploring higher-level representation learning techniques to improve performance.
24


Reviews:
[{"review_id": "lFwN3eCFOk", "reviewer": "Reviewer_GHKm", "summary": "The authors observe shortcomings in previous applications of pre-trained (transformer) language models (LM) in reinforcement learning (RL) and in particular in control: they are only used as initializations or as interfaces and do not outperform non-pre-trained transformer models such as decision transformers (DT).\n\nThey propose LoMa which adds 3 components to better leverage (or \"unleash\") pre-trained LMs in RL: 1) replacing the linear embedding with trainable MLPs, 2) performing low-rank adaptation (LoRA) for the RL training, and 3) maintaining the original language model loss while training with RL.\n\nThe authors claim that LoMa outperforms competitive offline RL baselines and DT on sparse-reward tasks and in low-data regimes.", "questions": "*(update: all the questions below have been addressed by the authors.)*\n\n**Major:**\n\n- Can the authors point to the specifications of the Reacher2d environment they used? The Reacher I’m familiar with[2] would not be considered a sparse-reward environment. This would also help to confirm the expert score.\nIt would also help to have the score of the policy that generated the medium dataset.\n- Can the authors indicate which Atari offline datasets they have used? D4RL does not seem to provide Atari datasets in its original version.\n- How much hyperparameter tuning has been spent on the value of the language loss hyperparameter and the fraction of parameters to train with LoRA?\n- Are the hyperparameters in Appendix E, used with all transformers in the paper? (LoMa, Wiki-RL, DT)?\n- Do the transformers and non-transformers used in the paper have a comparable number of parameters to ensure a fair comparison of performance?\n\n**Minor:**\n- Figure 6(a): why do the curves not start from the same point at training step 0? How can the authors explain that the cross-entropy loss decreases significantly for an already pre-trained model (red curve)? and also eventually decreases for the ablation (blue curve)\n\n[3] https://gymnasium.farama.org/environments/mujoco/reacher/", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper attempts to overcome important shortcomings of leveraging foundation models (here LMs) in RL in particular offline RL.\n\nOriginality:\n- Adding the sparse-reward experiments compared to the dense-reward environments in the Wiki-RL reference paper [1] to show the benefits of using pre-trained models is original and well-motivated.\n- Using LoRA to fine-tune the LM to a policy is interesting, ~~although not well-motivated.~~\n- The ablation with a model pre-trained on a random corpus to show the importance of sequential modeling is original.\n\n\nClarity:\n- The motivation and the contributions of the paper are very clear. \n- The structure of the paper is easy to follow.\n- Overall the paper is well written.\n\nSignificance:\n- LaMo performs competitively in the low-data regime as opposed to Wiki-RL showing that it potentially leverages/”unleashes” pre-trained LMs in a better way.\n- Overfitting, and other capacity loss phenomena, arising when more gradient steps are performed on the same data are major problems in offline RL. Observing that the auxiliary language loss (and maybe LoRA) introduced in LaMo helps in overcoming this issue is relevant beyond the scope of this paper.\n\n[1] Reid, Machel, Yutaro Yamada, and Shixiang Shane Gu. \"Can Wikipedia help offline reinforcement learning?.\" arXiv preprint arXiv:2201.12122 (2022).", "weaknesses": "*(update: all the weaknesses below have been addressed by the authors.)*\n\nThe points about the experimental results raised below and the questions raised in the question sections make it hard for me to confirm the validity of the claim made in the paper that LoMa outperforms baselines and that its components are crucial.\nClarifications on the experimental setup and results if valid would help raise my soundness and overall score.\n\n**Major:**\n\n1) \nI have a major concern over the validity of the experimental protocol and results presented in the paper from the observation that the performance reported for the baselines differs significantly from their performance as reported in their original papers. In particular\n- BC and CQL in the D4RL paper [2] obtain scores significantly divergent from the scores reported in this paper. Can the authors explain this divergence, and if due to different training budgets/settings explain the motivation behind the different settings?\nE.g. CQL and BC achieve 43.8 and 33.8, respectively, on Franka-complete in D4RL [2], but are reported to achieve 0 scores in this paper. (Overfitting of CQL is mentioned, but does not clarify the discrepancy)\n- Also BC and CQL are reported to perform better on Kitchen with 1% of the datasets (or other fractions) than on Kitchen with full datasets. Same for Wiki-RL and DT on some specific splits.\n- Similarly, Wiki-RL is reported to perform worse than the same model (ChibiT) in its original paper on Atari. Whereas DT is reported to perform better than in the wiki-RL paper. E.g. in Breakout 350 DT  and 130 Wiki-RL in this paper vs 267 DT and 280 Wiki-RL in the Wiki-RL paper.\n\n2) \n The ablation studies are presented each time on a different task&dataset-ratio combination and the full ablation results are not in the appendix (only the pre-training ablation is given). This can lead to a biased evaluation of LaMo if only good results are presented.\nThe combination of components of LaMo is a major contribution of the paper, claimed to \"unleash\" the potential of pre-trained LMs, and ablation experiments are thus crucial to assess the validity of this claim and the significance of the paper.\n\n\n**Significant:**\n- I don't find LoRA to be well-motivated, or at least that its benefits are. The authors mention early overfitting but this is not clear from Figure 5 (LoMa also experiences small drops in performance).\n-  Several elements of the paper can be mistakenly interpreted as original contributions of the paper, such as the language modeling loss (also used in Wiki-RL), the ablation on ImageGPT (also used in Wiki-RL), the selection of the baseline (similar to Wiki-RL).\nWhile not being novel was not the issue, it was often not clear if those contributions were original.\n- It’s hard to assess the significance of an improvement when curves are shown with shaded areas in [μ − 0.5σ, μ + 0.5σ].\n\n**Minor:**\n- Notation is not ambiguous for readers familiar with the topics but would benefit from better presentation. Some variables are undefined in some equations e.g. the expectation distributions are underspecified in equation 1, $a'$ in equation 3, $T$ in equation 4.\n- Citations don’t include the journal/proceedings details which makes it hard to identify which version of a paper was used in some cases.\n\n[2] Fu, Justin, et al. \"D4rl: Datasets for deep data-driven reinforcement learning.\" arXiv preprint arXiv:2004.07219 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0pw73JoDYO", "reviewer": "Reviewer_mYMk", "summary": "This paper studies the use of decision transformer (DT) and pretrained language models for offline RL control. The authors propose a new framework Language Models for Motion Control (LaMo) that improves upon a naive use of a pretrained LM for DT training. The framework 1) uses a pretrained LM model, 2) uses LoRA fine-tuning, 3) use higher capacity input embedding and output networks and 4) use auxiliary language prediction loss during finetune. Empirical results are provided to show the proposed method outperform other transformer-based methods and also offline RL methods in a number of benchmark tasks.", "questions": "It is unclear why language pretraining can help RL tasks which have a large domain gap?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "**originality**\n- Main novelty of the paper is an improved framework to finetune pretrained LM on RL task. Compared to the Reid et al paper, the main additions are the finetuning technique and the increased capacity of the projection layer for input embeddings and output layer. Though the changes are relatively simple, they are shown to provide much stronger performance, can be a novel contribution. \n\n**quality**\n- technical details provided for reproducing the results. \n- good discussion of related works\n\n**clarity**\n- paper is written clearly and easy to follow \n\n**significance**\n- some insights are provided on potential reason that DT methods work better on sparse reward setting. \n- ablations showing the contribution of each component. \n- the proposed changes are not too complex, I appreciate the simplicity.  \n- empirical resutls show significant improvement over DT and DT+Wiki over all 3 benchmarks.", "weaknesses": "Comparisons: \n- Figure 1, in Reid et al paper, they show that a pretrained DT tend to give improved performance, why in your figure DT gets better performance in Kitchen and Atari? \n- Did you finetune the methods you are comparing to on the benchmarks you are studying? \n- The other thing is only some of the tasks in each benchmark are studied, so it is also a bit concerning whether there will still be a big perforrmance gap between proposed method and baseline when other tasks are also tested. \n\nNovelty and significance:\n- Technical novelty of the method is a bit lacking. Essentially compared to DT+Wiki, a different finetuning method is used, and the projection and output layers are made bigger. Neither of these are new techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a1TZllJVvW", "reviewer": "Reviewer_1Dwk", "summary": "The paper proposes a method to utilize the power of pretrained LLMs for offline RL with 3 important design choices: 1) use non-linear MLPs for token embedding and prediction layers; 2) Finetune the pretrained model with LoRA; and 3) regularize the fine-tuned model with a language loss. The experiments show that the proposed method boosts the performance of DT in both sparse and dense reward settings, especially in the low-data regime. The paper conducted several ablation studies to show the importance of each of the proposed design choices.", "questions": "- Can we replace the language loss with a regularization loss so that the fine-tuned model is not too far away from the original pretrained model? For example, we can minimize the discrepancy between the embeddings of the fine-tuned and pretrained models.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Presentation: the paper was easy to follow because of its simplicity and clear writing. \n- The idea is intuitive and simple, and can potentially be adapted to other LLMs as well as other offline RL models and tasks.\n- The empirical results are good and the ablation studies are comprehensive, proving the importance of each proposed component.", "weaknesses": "- Since the paper only studied one LLM (GPT-2) and one RL model (DT), I wonder if the same methodology generalizes to other LLMs and RL models. Since LLMs are improving quickly, it's important that the conclusion in the paper also holds for newer and stronger LLMs.\n- The baselines used for comparison seem a bit outdated. There have been much stronger baselines in offline RL in recent years, especially diffusion-based methods such as Diffusion-QL [1] and DD [2]. The authors should include these as stronger baselines.\n- The experiments only include the medium datasets in D4RL and are missing the medium-replay and medium-expert datasets. I expect LaMO to perform well on medium-replay as these datasets are of low-quality and should highlight the advantage of language pretraining.\n\nMinor comments:\n- The first paragraph of Section 4.1 is a bit confusing, it made me think that the authors pretrained GPT-2 themselves. It should be stated clearly that the authors used the pretrained GPT-2 from HF and only pretrained its special variants.\n\n[1] Wang, Zhendong, Jonathan J. Hunt, and Mingyuan Zhou. \"Diffusion policies as an expressive policy class for offline reinforcement learning.\" arXiv preprint arXiv:2208.06193 (2022).\n\n[2] Ajay, Anurag, et al. \"Is conditional generative modeling all you need for decision-making?.\" arXiv preprint arXiv:2211.15657 (2022).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mqldrZPwTM", "reviewer": "Reviewer_ei7j", "summary": "This paper introduces a novel framework, termed LaMo, aimed at enhancing the utilization of pre-trained Large Language Models (LLMs) in the context of offline Reinforcement Learning (RL). The novelty of the proposed framework is threefold: Firstly, it employs Low-rank Adaptation (LoRA) during the fine-tuning process, targeting a specific subset of model weights and finetuning with offline experience data. Secondly, it innovatively replaces the conventional linear projections for Query (Q), Key (K), and Value (V) within each attention block with Multi-Layer Perceptrons (MLPs). Thirdly, LaMo incorporates a next-word prediction loss in addition to the primary sequence modeling loss to mitigate overfitting concerns.", "questions": "Several questions and suggestions:\n1. In section 4.2 you mentioned that you used LORA to inject low-rank matrices into attention weights Q, K and V only and freeze all other weights inside the Transformer, given that there are other large MLPs inside it, what is the rationale of only applying LoRA to Q, K and V?\n2. In section 5.5, the benchmark tasks you used for comparison change sometimes, I’m curious how you select ablation benchmarks for showing different components in your framework works better?\n3. It would be nice to see how different scales of GPT-2 could affect the performance on your benchmarks.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: The framework introduced in this paper demonstrates novelty through a unique amalgamation of established fine-tuning techniques. It draws inspiration from the wiki-RL paradigm but stands out by effectively addressing limitations that hindered previous work.\nQuality: This paper exhibits a commendable level of quality. The authors meticulously design their experiments to empirically validate the individual components of the framework.\nClarity: The manuscript exhibits a high degree of clarity in its presentation. The conceptual underpinnings and experimental setups are readily comprehensible.\nSignificance: The findings presented in this paper hold great promise. Notably, the results span a diverse array of scenarios, including sparse and dense reward tasks, varying data scales, and a comprehensive ablation study.", "weaknesses": "In section 5.5 Ablations, while empirical results indicate the superiority of the former, the absence of a deeper analysis of the choice to use MLPs warrants consideration. It is advisable to provide further insight into the theoretical basis and motivations for this decision.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)